Delta time travel fails for older versions: log retention, file retention and what you can still recover

#Symptom

You need to query or restore an older version of a Delta table, and get one of these:

[DELTA_VERSION_NOT_FOUND] / VersionNotFoundException:
Cannot time travel Delta table to version 12. Available versions: [58, 96]

FileNotFoundException ... part-00003-....parquet
A file referenced in the transaction log cannot be found

Exact error text varies by runtime. The first says that the log entry has gone. The second says that the log entry exists but its data files have gone.

#Two settings, two different jobs

Time travel needs both the log and the files to still exist.

So the default usable window is often about 7 days, not 30, once VACUUM runs. Without VACUUM, old files stay around, and the log limit decides the window. Defaults and behavior can change between releases, so check them on your runtime.

Note the common surprise: DESCRIBE HISTORY can list a version that you can no longer read. History shows log entries, and it doesn't prove the files still exist.

#Step 1: see what's available

DESCRIBE HISTORY main.sales.orders;

SHOW TBLPROPERTIES main.sales.orders ('delta.logRetentionDuration');
SHOW TBLPROPERTIES main.sales.orders ('delta.deletedFileRetentionDuration');

Find the most recent VACUUM:

from pyspark.sql.functions import col

(spark.sql("DESCRIBE HISTORY main.sales.orders")
   .filter(col("operation").startswith("VACUUM"))
   .select("version", "timestamp", "operation", "operationParameters", "operationMetrics", "userName")
   .show(10, truncate=False))

If you don't recognize the user, or there's no job in the record, check whether predictive optimization is running it for you:

SELECT operation_type, operation_status, start_time, end_time
FROM system.storage.predictive_optimization_operations_history
WHERE catalog_name = 'main' AND schema_name = 'sales' AND table_name = 'orders'
ORDER BY start_time DESC LIMIT 20;

Test versions from newest to oldest to find the oldest that's readable. A count(*) may be answered from metadata, without touching the data files, so read real data:

for v in [95, 90, 80, 70, 60]:
    try:
        n = (spark.read.option("versionAsOf", v).table("main.sales.orders")
             .selectExpr("sum(hash(*)) AS h, count(*) AS n").first())
        print(v, "OK", n["n"])
    except Exception as err:
        print(v, "FAILED", str(err)[:120])

This does a full scan, so it can take a while on a large table.

#Step 2: what's recoverable

#Step 3: try storage-level recovery

If ADLS soft delete was on during the VACUUM, the Parquet files may still be recoverable.

az storage blob service-properties delete-policy show \
  --account-name stlake --auth-mode login

az storage blob list --account-name stlake --container-name lake \
  --prefix sales/orders/ --include d --auth-mode login \
  --query "[?deleted].{name:name,deleted:properties.deletedTime}" -o table

Support and commands differ for accounts with hierarchical namespace, so check the current documentation for your storage account type. You can also use the Azure portal to view and restore deleted blobs.

Caution

Restoring vacuumed files needs care. Stop writers first, restore a copy into a separate location where you can, and verify before touching production. Once restored, avoid the temptation to set spark.sql.files.ignoreMissingFiles as a fix: Spark then silently skips any file that's missing, and returns incomplete results that look valid.

For the wider recovery options, see backup and recovery for Delta.

If the files are truly gone, the most reliable path is to rebuild from an upstream source. That's why keeping bronze data matters.

#Step 4: set retention to match your recovery target

Decide how far back you need to go, and set both properties. To allow 30 days:

ALTER TABLE main.sales.orders SET TBLPROPERTIES (
  'delta.deletedFileRetentionDuration' = 'interval 30 days',
  'delta.logRetentionDuration' = 'interval 30 days'
);

Keep the log retention at least as long as the file retention. Otherwise, you pay to store files that no version can reach.

Consider the costs:

Estimate before you commit, especially for tables with frequent rewrites:

VACUUM main.sales.orders DRY RUN;

Warning

Never run VACUUM ... RETAIN 0 HOURS on a table in use, or turn off spark.databricks.delta.retentionDurationCheck.enabled to allow it. It removes files that running queries and streams need, destroys time travel and can't be undone. See FileNotFound after VACUUM.

#Step 5: use clones for long-term restore points

Keeping every version for a year would cost a lot. A deep clone copies only one chosen version's data, with its own retention, so it's a good fit for month-end or pre-release restore points:

CREATE OR REPLACE TABLE backup.sales.orders_2026_10_31
DEEP CLONE main.sales.orders
TIMESTAMP AS OF '2026-10-31T23:59:00';

Deep clones can refresh incrementally when you reuse the same target:

CREATE OR REPLACE TABLE backup.sales.orders_latest
DEEP CLONE main.sales.orders;

A shallow clone doesn't copy the data files. It refers to the source's files, so it isn't a backup, and vacuuming the source can break it. See deep and shallow clones.

#Step 6: test a restore

Restore into a clone rather than the real table during tests:

CREATE TABLE sandbox.tests.orders_restore_test
SHALLOW CLONE main.sales.orders VERSION AS OF 90;

SELECT count(*) FROM sandbox.tests.orders_restore_test;
RESTORE TABLE sandbox.tests.orders_restore_test TO VERSION AS OF 90;

Drop the test table afterwards, and don't vacuum the source while the test is underway.

#Prevent it next time

• Write down the recovery window for each important table, and set retention to match it.
• Alert if a table's retention falls below its target, or if VACUUM runs with an unusual retention value.
• Run a restore test each quarter, and record how long it took.
• Keep raw bronze data long enough to rebuild silver and gold.
• See the inspection cheat sheet for the commands used here.