Problem

Your Delta Live Table (DLT) pipeline fails with a `SparkOutOfMemoryError`. You read in the error message the pipeline is unable to acquire enough memory. 

Error Message:
org.apache.spark.memory.SparkOutOfMemoryError:[UNABLE_TO_ACQUIRE_MEMORY] Unable to acquire 104857600 bytes of memory, got 38125049.SQLSTATE: 53200

Cause

The various Apache Spark components, including Photon engines, do not have sufficient resources for memory acquisition.

Several factors contribute to insufficient resources.

1. The DLT default configurations for memory allocation to your work instance type may not be sufficient for the pipeline’s requirements. 
2. The cluster mode and instance type may not be optimized for the pipeline's workload.
3. The pipeline may be running on a Preview channel, which is not recommended for production workloads.

Solution

Depending on your situation, either upgrade your worker instance type or switch your channel from Preview to Current.

Upgrade worker instance type

Change your worker instance type to one with higher memory capacity than the default. You can check the current instance types assigned in DLT by navigating to Pipeline settings > Advanced > Instance Types.

For more information, refer to the “Compute configuration options” section of the Configure a DLT pipeline (AWS | Azure | GCP) documentation and the “Select instance types to run a pipeline” section of the Configure compute for a DLT pipeline (AWS | Azure | GCP) documentation.

Switch the channel to Current 

To switch the pipeline from the Preview channel to the Current channel, in the UI navigate to Pipeline settings > Advanced > Channel and choose ‘Current’ from the dropdown menu. Save the pipeline settings to update.