Problem

You are using Databricks Delta archival support (AWS | Azure | GCP). You restored files from the archive storage, but when you run a query, the files are not recognized by Delta. You get a “delta archived files in limit” error message.

[DELTA_ARCHIVED_FILES_IN_LIMIT] Table delta.`<path>` does not contain enough records in non-archived files to satisfy the specified LIMIT of 10001 records.

This happens even after you have restored all files identified using the SHOW ARCHIVED FILES command.

Cause

The error DELTA_ARCHIVED_FILES_IN_LIMIT typically occurs when a query attempts to access data files older than the configured archival threshold in delta.timeUntilArchived. This is expected behaviour.

However, even after correctly identifying and restoring all the necessary files using the SHOW ARCHIVED FILES command, Delta still fails to recognize these restored files. This is a known limitation in the current version of the archival feature, where restored files from the archive are not properly detected by Delta.

Solution

You can work around this by temporarily disabling archival support and then accessing the restored files.

ALTER TABLE <table-name> UNSET TBLPROPERTIES ('delta.timeUntilArchived');

Alternatively, you can increase the value for delta.timeUntilArchived so the time period includes all of the data files that were restored. For example, if the current value is 10 days, but the files you restored are 30 days old, you can update to delta.timeUntilArchived 31 days. After the time period has been updated, Delta should recognize the restored files.