Delta VACUUM takes hours or frees no space on Azure Databricks: LITE mode, inventory-driven vacuum and ADLS soft delete

VACUUM causes two kinds of tickets. In the first, the maintenance job runs for hours, holds a large cluster and sometimes times out. In the second, VACUUM reports thousands of deleted files but the storage account's capacity graph doesn't move. The causes are different, so work out which one you have before you change anything.

#Symptoms and likely causes

#Check 1: where the time goes

Every VACUUM writes two commits to the table history: VACUUM START and VACUUM END. The gap between them, and the metrics, tell you whether you're listing-bound or delete-bound.

SELECT version, timestamp, operation,
       operationMetrics.numFilesToDelete,
       operationMetrics.numDeletedFiles,
       operationMetrics.sizeOfDataToDelete
FROM (DESCRIBE HISTORY main.sales.orders)
WHERE operation IN ('VACUUM START', 'VACUUM END')
ORDER BY version DESC
LIMIT 10;

Metric names vary by version; run DESCRIBE HISTORY without a projection to see what your runtime emits.

Then open the Spark UI for the run:

• Long stages of tiny tasks that list paths, before the VACUUM START commit: listing-bound. FULL mode walks every directory under the table root. On a table partitioned by date and hour that has run for years, that can be tens of millions of blobs.
• VACUUM START lands quickly and the long wait comes before VACUUM END: delete-bound.

Databricks' sizing guidance: run VACUUM on an autoscaling cluster of 1-4 workers with 8 cores each, and a driver with 8-32 cores. Add workers when finding the files is slow. Increase the driver when deleting them is slow. A 40-worker cluster won't help a delete-bound vacuum.

#Check 2: are you paying for a full listing you don't need?

FULL (the default) removes two kinds of file: files that dropped out of the log longer ago than the retention window, and any file in the directory that the log doesn't reference at all, such as debris from failed writes. LITE (Databricks Runtime 16.1+ for the SQL syntax; the concepts page says 16.4 LTS, Public Preview) reads only the transaction log and skips the listing entirely.

-- Preview first: lists up to 1,000 files
VACUUM main.sales.orders LITE DRY RUN;

VACUUM main.sales.orders LITE;

LITE needs at least one successful VACUUM within the log retention window (30 days by default). If you get DELTA_CANNOT_VACUUM_LITE, run a FULL vacuum once. A practical pattern for very large tables is LITE daily, FULL monthly. LITE handles normal churn cheaply. The monthly FULL run cleans up orphaned files from aborted jobs.

Warning

Never lower retention below the longest-running reader or writer on the table, and don't set spark.databricks.delta.retentionDurationCheck.enabled = false as a way to "fix" a slow vacuum. Deleting files that a running stream or time-travel query still needs causes FileNotFoundException and can break streaming checkpoints.

#Check 3: use a storage inventory instead of listing

If you need FULL semantics on a huge table, you can pass a precomputed file list instead of having Spark list the directory. VACUUM ... USING INVENTORY takes a table or query with path (full URI), length, isDir and modificationTime columns. Runtime support appeared around 15.2. A syntax error means your runtime is too old, so check your build.

Azure Storage blob inventory can produce that list daily as Parquet. Enable a rule scoped to the table's container and prefix:

az storage account blob-inventory-policy create \
  -g rg-data -n stlakeprod \
  --policy @inventory-policy.json

Then map the report to the shape VACUUM expects. Column names in the report depend on the fields you select in the policy; Name, Content-Length and Last-Modified are the usual ones.

CREATE OR REPLACE TEMP VIEW orders_inventory AS
SELECT concat('abfss://lake@stlakeprod.dfs.core.windows.net/', Name) AS path,
       `Content-Length`                                          AS length,
       false                                                     AS isDir,
       unix_millis(to_timestamp(`Last-Modified`))                AS modificationTime
FROM parquet.`abfss://inventory@stlakeprod.dfs.core.windows.net/2026/11/12/*/orders-rule/*.parquet`
WHERE Name LIKE 'sales/orders/%';

VACUUM main.sales.orders USING INVENTORY (SELECT * FROM orders_inventory) DRY RUN;

Caution

The inventory is a snapshot. Files written after it was taken aren't listed, so they're safe. But a wrong prefix or a path written in a different form (abfss:// vs wasbs://, a different container) makes every live file look unreferenced. Always run DRY RUN and compare the count with numFilesToDelete from a recent FULL vacuum before you run the real thing.

#Check 4: VACUUM "worked" but capacity didn't drop

On Azure, a delete is not necessarily a delete. Check the blob service settings on the account behind the external location:

az storage account blob-service-properties show -g rg-data -n stlakeprod \
  --query "{softDelete:deleteRetentionPolicy, versioning:isVersioningEnabled, containerSoftDelete:containerDeleteRetentionPolicy}"

• Blob soft delete keeps every deleted file for the retention period, and you're billed for it. A VACUUM that removes 20 TB adds 20 TB of soft-deleted data for that period.
• Blob versioning turns each overwrite or delete into a previous version that is kept until a lifecycle rule removes it. Delta never overwrites files, but versioning still keeps deleted blobs as versions.
• Use the storage account's capacity metrics, not DESCRIBE DETAIL, to see what's actually billed. sizeInBytes covers only the current snapshot.

Soft delete is a reasonable guard against accidental rm -r. If you keep it, set a short retention period on lake containers and add a lifecycle rule for previous versions. Don't disable it account-wide without agreeing that with whoever owns recovery.

#Check 5: deletion vectors and retention keep the files alive

With deletion vectors, DELETE, UPDATE and MERGE mark rows as deleted without rewriting the data file. The old file is still referenced by the current version, so VACUUM can't remove it. To reclaim the space:

REORG TABLE main.sales.orders APPLY (PURGE);
-- wait longer than delta.deletedFileRetentionDuration, then:
VACUUM main.sales.orders;

Check retention as well: SHOW TBLPROPERTIES main.sales.orders ('delta.deletedFileRetentionDuration'). The default is 7 days. A table set to '90 days' for audit time travel will keep 90 days of rewritten files whatever VACUUM mode you use.

#Check 6: is predictive optimization already doing this?

For Unity Catalog managed tables with predictive optimization enabled, Databricks schedules VACUUM itself. Before you blame or add a job, look at what it ran:

SELECT start_time, end_time, operation_status, usage_quantity, operation_metrics
FROM system.storage.predictive_optimization_operations_history
WHERE catalog_name = 'main' AND schema_name = 'sales' AND table_name = 'orders'
  AND operation_type = 'VACUUM'
ORDER BY start_time DESC
LIMIT 20;

If it's running, remove your own VACUUM job for that table rather than vacuuming twice.

#Prevent it next time

• Schedule LITE vacuums frequently and FULL rarely on any table with more than a few million files.
• Run VACUUM on dedicated job compute sized to the bottleneck you found: more driver for deletes, a few workers for listing.
• Record numDeletedFiles and duration per run from DESCRIBE HISTORY, and alert when duration grows week over week.
• Agree soft delete and versioning retention for lake containers with the storage owner, and document it next to the table's deletedFileRetentionDuration.
• Compact small files (optimized writes, auto compaction) so there's less to list in the first place.