Problem You had a network issue (or similar) while a write operation was in progress. You are rerunning the job, but partially uncommitted files during the failed run are causing unwanted data duplication. Cause How Databricks commit protocol works: The DBIO commit protocol (AWS | Azure | GCP) is transactional. Files are only committed after a trans...

Problem While running the Databricks Delta Live Tables (DLT) pipeline, you encounter a WARN message in DLT event logs. Could not turn on CDF for table <table-name>. The table contains reserved columns  [_change_type, _commit_version, _commit_timestamp] that will be used internally as metadata for the table's Change Data Feed. Change Data Feed ...

Problem  When using a streaming application to stream data from a Delta table and write to a given destination, you notice data loss.  Cause In trying to separately address a failed streaming job by using  startingVersion=latest , the tradeoff is possible data loss. The restarted query will read only from the latest available Delta version of the so...

Problem You are running a series of structured streaming jobs and writing to a file sink. Every 10th run appears to run slower than the previous jobs. Cause The file sink creates a _spark_metadata folder in the target path. This metadata folder stores information about each batch, including which files are part of the batch. This is required to prov...

Problem You are attempting to query a Delta table when you get an IllegalStateException error saying that the metadata could not be recovered. Error in SQL statement: IllegalStateException:  The metadata of your Delta table couldn't be recovered while Reconstructing version: 691193. Did you manually delete files in the _delta_log directory? Set spar...