Photon memory issue while querying a table with a large number of columns

Use the variant datatype in your CREATE TABLE AS SELECT query, or disable Photon.

Written by Raghavan Vaidhyaraman

Last published at: June 18th, 2025

Problem

While using a CREATE TABLE AS SELECT (CTAS) SQL statement on a Delta table, you receive the following error. 

Photon out of memory error query id: <query-id>

 

Cause

The table has more columns than what Photon’s architecture is designed to optimally handle.

 

Solution

In Databricks 15.3 and above, you can modify your CTAS SQL statement to create a table with one variant column (of type VARIANT) containing all the data.

CREATE TABLE <your-table> AS SELECT PARSE_JSON(<your-json-str>) <your-variant-column> FROM <your-data>

 

For more information, please refer to the VARIANT type (AWS | Azure | GCP) documentation and the Query variant data (AWS | Azure | GCP) documentation.

 

Alternatively, you can disable Photon. Run the following command in a notebook. Be aware that disabling may impact performance.

set enable_photon = false;