Find your workspace ID

Learn how to find your Databricks workspace ID in the web UI as well as via a notebook command....

Cannot start Databricks workspace from Google Cloud console

Check to make sure your Databricks subscription is active and that you are using the correct Google account....

Failed to add user error due to email or username already existing with a different case

You should ensure casing for usernames is consistent across all accounts and providers in your system....

Cannot access Databricks secrets when using a "No isolation shared" cluster

You cannot use dbutils.secrets.get() when admin protection for No isolation shared clusters is enabled in your account....

DNS resolution fails for a newly created Databricks workspace

Users should wait for 10 minutes before re-creating a workspace with the same name....

Reactivate a user that has been disabled with AAD at the Account and Workspace level

Using a default account API call, activate the user first at the account level, then the workspace level....

Slack alarm notifications test fails with “invalid_token”

Verify that the notification destination URL is still active, check the expiration status of the destination token, or create a new webhook in the destination. ...

Unable to manage pipeline permissions despite being pipeline owner or workspace admin

Enable the Clusters, Pools, Jobs Access Control workspace configuration....

Querying system.access.audit table not returning expected records

Query the system.access.audit table for the "filesystem" service and the "volumeDelete" action to track when entire volumes are deleted instead, or use external tools to track at file level....

Reactivate inactive users at the account level using OAuth token for non-SCIM scenarios

Create a service principal and use an OAuth token to activate users via an API call without having to use a SCIM token....

DIRECTORY_PROTECTED error when deleting a user’s home folder

Remove the user from the workspace, then you can delete the home folder....

Activate or deactivate a user in the account console or workspace using the API

Create a service principal and use the Update user details account API endpoint....

Cannot upload CSV or SH file types in workspace UI

Set enableWorkspaceFilesystem and enableFilesInWorkspace to true....

Getting “Remote repo not found” error while trying to create or sync Git repo

Add the outbound IP addresses the Databricks control plane uses for your workspace's region to your Git server's IP allowlist....

Can’t add members to external groups using the UI

Disable the immutable external groups preview if you’re an account admin, or use the API to update group details....

How to make service principals and groups workspace admins

Instructions for setting in the UI. ...

How to create alerts for expiring personal access tokens (PATs)

Use Databricks system tables and SQL alerts. ...

Cannot delete a user from a Databricks account

Contact your account team to change the account owner and then delete the previous account owner....

Setup cross account bucket access in Google Cloud

Use service accounts to grant access to storage buckets in another account....

Job fails with input/output error when displaying a data frame

Check your NSG and port settings. ...

Connection refused error when trying to connect to an external service from Databricks

Check your DNS resolution and port connectivity, report to your internal networking team, and then contact Databricks if the issue still persists....

Apache Spark driver failing with unexpected stop and restart message

Reduce the size of the objects being collected or processed in parallel. ...

Clusters are getting dynamic public IPs even after attaching NAT to the VPC

Check if the workspace has a public GKE; if yes, migrate to GCE....

Databricks support for IPv6

IPv6 is not supported. ...

Secret key rate drops after migrating to GCE compute cluster

Have your GCP administrator increase the minimum ports per VM instance....

Configure Simba ODBC driver with a proxy in Windows

How to configure the Simba ODBC driver to connect through a proxy server when using Windows....

“SSL_connect: certificate verify failed” error when trying to connect to Databricks from Tableau

Manually add intermediate certificates to the root certificates file in order to be recognized by the ODBC driver....

Error embedding AI/BI dashboard in Google Sites

You may need to whitelist additional Google domains based on your embed type....

“Failed to create space” message when attempting to create new Genie space

Disable “Enforce data processing within workspace Geography for Designated Services” for the workspace from the account console....

Install a private PyPI repo

How to install libraries from private PyPI repositories....

Cannot apply updated cluster policy

When performing an update to an existing cluster policy, the update does not apply unless you remove and re-add the policy....

Cluster Apache Spark configuration not applied

Values set in your cluster's Spark configuration are not applying correctly....

Cannot restart cluster

Ask your workspace administrator to update cluster access rights....

Cluster fails to start with dummy does not exist error

Cluster is not starting due to a `dummy does not exist` Apache Spark error message....

Cluster slowdown due to Ganglia metrics filling root partition

Resolve cluster slowdowns due to a Ganglia metric data explosion filling the root partition....

Failed to create cluster with invalid tag value

Cluster creation fails if optional tag values do not conform to cloud vendor requirements....

Set executor log level

Learn how to set the log levels on Databricks executors....

Auto termination is disabled when starting a job cluster

Auto termination policies are not supported on job clusters....

How to overwrite log4j configurations on Databricks clusters

Learn how to overwrite log4j configurations on Databricks clusters....

Apache Spark executor memory allocation

Understand how Spark executor memory allocation works in a Databricks cluster....

Configure a cluster to use a custom NTP server

Configure your clusters to use a custom NTP server (public or private) instead of using the default server....

Enable GCM cipher suites

Enable AES-GCM encryption (GCM cipher suites) for use with SSL connections to other clusters. Resolve javax.net.ssl.SSLHandshakeException error....

Enable retries in init script

Add a retry function to your init script....

Cannot set a custom PYTHONPATH

Setting a custom PYTHONPATH in an init script or in DCS is not supported....

Run a custom Databricks Runtime on your cluster

Configure your cluster to run a custom Databricks Runtime image via the UI or API....

Cluster init script fails with mirror sync in progress error

If the mirror you are using is not in sync with the main repository, apt-get update returns a Mirror sync in progress error....

Pin cluster configurations using the API

Pin up to 100 compute cluster configurations using the API....

Unpin cluster configurations using the API

Unpin compute cluster configurations using the API....

Apache Spark UI task logs intermittently return HTTP 500 error

If the Spark property spark.databricks.ui.logViewingEnabled is set to false, you cannot view task logs in the Spark UI....

Disable cluster-scoped init scripts on DBFS

Set a cluster policy to prevent users from creating clusters that load cluster-scoped init scripts from DBFS....

Cluster-named and cluster-scoped init script migration notebook

Easily migrate your cluster-named and cluster-scoped init scripts to cluster-scoped init scripts stored as workspace files....

Cluster fails with Fatal uncaught exception error. Failed to bind.

If other software uses port 6062, it can conflict with the IPython kernel REPL and prevent the driver node from starting....

Log delivery feature not generating log4j logs for executor folders

Log delivery only generates a log file for the driver folder. This is by design....

Use a cluster policy to disable Photon

You can use cluster policies to prevent users from creating clusters with Photon enabled....

DBFS init script detection notebook

Scan your workspace for init scripts on DBFS....

Workspace is not UC enabled

Troubleshooting errors related to workspace not being UC enabled...

Migration guidance for init scripts on DBFS

Init scripts on DBFS are end-of-life. You should migrate them to cloud storage, Unity Catalog volumes, or workspace files....

Databricks spark-submit jobs appear to “hang” and clusters do not auto-terminate

Embed system.exit code in your application to shutdown the Java virtual machine with exit code 0....

Apache Spark is configured to suppress INFO statements but they overwhelm logs anyway

Modify your log4j2 configuration file directly within the Databricks environment. ...

Jobs fail with error: There are already 1000 active runs (limit: 1000).

Identify and cancel job runs causing the issue, then schedule future job runs farther apart than a few minutes. ...

Cluster fails with a DRIVER_EVICTION error

Ensure that the driver instances are not running on preemptive nodes....

Init script stored on a volume fails to execute on cluster start

Init scripts created on Windows systems and uploaded to Unity Catalog volumes have CRLF as a newline which needs to be converted to LF before the cluster can process it. ...

Databricks API last_activity_time attribute shows incorrect timestamp

Use the cluster auto-termination feature to manage cluster termination based on inactivity....

BROADCAST_VARIABLE_NOT_LOADED or JVM_ATTRIBUTE_NOT_SUPPORTED errors when using broadcast variables in a shared access mode cluster

Use a single-user cluster or pass a variable into a function as a state instead. ...

404 error when installing krb5-user module

Manually remove the var directory path to refresh the cached data. ...

Cluster startup failure while running proxy-configured init script with other init scripts

Modify the proxy init script to bypass local addresses....

Init scripts failing with unexpected end of file error

Remove the special characters from the init script....

Unable to access the hive_metastore schema

Ensure all clusters use the same Hive metastore version and Apache Spark configurations are set. ...

Job fails while installing ODBC Driver 18 for SQL Server using an init script

Add msodbcsql18 to the LD_LIBRARY_PATH then append LD_LIBRARY_PATH path to /etc/environment....

Error when trying to use Apache Spark’s Pyspark offset method on DataFrames with serverless compute

Use the limit method or the monotonically_increasing_id() function instead....

Jobs failing with schema conversion error: cannot convert Parquet type INT32 to Photon type long

Set spark.databricks.photon.scan.enabled to false....

Cannot access Apache SparkContext object using addPyFile

Leverage the addArtifact API instead....

Cluster fails to launch with a Bootstrap Timeout error

Verify allowlisting of necessary services and correct configuration of your VPC/VNet. ...

Cluster fails to launch with error, “user specified an invalid argument”

Change the cluster owner to an active user....

Job executions failing on clusters using Docker Container Services with MalformedInputException error

Specify the correct character encoding when reading the file and change the LANG settings. ...

Enabling Dynamic Allocation leads to NODES_LOST scenario

Enable Autoscaling when you create a Databricks cluster. ...

Jobs failing with BindException error after upgrading to Databricks Runtime 11.3 LTS or above

Change the ipywidgets default port to another available port....

Cluster fails to initialize after a Databricks Runtime upgrade

Check your init scripts and then your Apache Spark configurations....

Receiving “no space left on device error” message when attempting to use Apache Spark

Switch to a VM instance with attached local storage or increase the size of the VM boot disk....

Missing the audit log event of a cluster deletion

Pin your cluster and monitor usage....

Increased job execution time after migrating from all-purpose to job cluster

Increase the Hive client pool size in the job cluster configuration to match the previous all-purpose compute setting....

NODES_LOST error during cluster upsizing when Apache Spark dynamic allocation is enabled

Remove the spark.dynamicAllocation.enabled Spark config from the compute configuration....

Listing Hive metastore tables in Catalog Explorer failing with error getting schemas

Remove the Apache Spark configuration "spark.databricks.session.share true". ...

Change the minor version of Python in a cluster

Use an init script to install the desired versions of Python and pyenv....

Row value assignments not reflecting expected output in code that loops through temporary views

Avoid using temporary views with the same name when using loops in Spark Connect. ...

Unresolved column error when using Apache Spark Connect to run a query to create a temporary view

Use unique names for each temporary view. ...

scala.collection.immutable.HashMap$HashMap1 class leading to OOM error in driver

Change the webUI events location to RockDB....

Unable to attach init scripts to an interactive cluster

Ensure the cluster creator exists in the workspace and has the necessary group memberships....

How to restrict selection of specific Databricks runtimes in the compute creation UI

Implement a regular expression (regex) validation to ensure only certain selections are available....

Cannot select a compute policy for a DLT Pipeline

Ensure that you have permission to use the policy and are using the correct compute type....

How to restrict cluster creation to single-node only

Add a JSON configuration to your compute policy....

Workflows are failing with a 'Could not reach driver of the cluster' error

Use a larger driver instance or increase the REPL timeout....

Cannot create cluster: spark conf: 'spark.databricks.cluster.profile' is not allowed when choosing an access mode

Use the flag is_single_node to create single node compute....

FileNotFoundError when trying to use Android Development Bridge (ADB) command line tool on a cluster

Explicitly install the ADB tool on your cluster and set in the correct system path....

Spark UI is empty for the job clusters after termination

For non-Spark tasks the Spark UI should be empty....

502 error when trying to access the Spark UI

Enable Spark UI to store its data on disk instead of in memory....

Pulumi fails to deploy workflows in serverless mode

Update the Pulumi provider, check for configuration errors, and remove the job cluster definition from the payload....

Unable to run interactive workloads using a dedicated (formerly single user) compute assigned to a service principal where you have the Service Principal User role

Use a dedicated compute linked to a user or group instead, or switch to standard compute to run interactive workloads....

Init script stored in volume fails with permission denied error

Ensure that the cluster owner has access to the volume and the init script....

Using Vertica Spark Connector to write a DataFrame to external database fails with [TABLE_OR_VIEW_NOT_FOUND] error

Switch to a non-Unity Catalog cluster....

XML file read executes slowly

Define the schema explicitly. ...

Performing count on Delta table using dedicated vs standard compute

Using a dedicated compute, you will be able to get the expected output by performing the count on the version of the Delta table from which the associated files are removed....

How to gather total cluster duration

Run a SQL query....

UnsupportedClassVersionError when running a Databricks job on Graviton machines

Specify an appropriate ARM architecture instead of AMD....

Cannot deploy the agent model with a compute resource assigned to a group

Use Dedicated access mode compute assigned to a user or service principal....

Cluster’s Apache Spark UI not appearing

Append the default Databricks daemon listener to your custom listener....

Unable to connect endpoints via non http/https port number when using standard (formerly shared) access mode

Use Databricks Runtime 12.2 LTS or above, or use an init script to open a required port....

How to install Python packages from a private PyPI repository on Databricks

Configure a cluster-wide index URL or install the package using a notebook cell....

Connection to external sources after configuring proxy settings is failing

Allowlist the required FQDNs, remove unnecessary Apache Spark configuration properties, check your proxy settings, and verify connectivity....

Job run fails with error message “Could not reach driver of cluster”

Increase the REPL launch timeout....

Standard (formerly shared) cluster not allowing use of Machine Learning Runtime

Use a dedicated compute with a group including the list of users....

Error: PERMISSION_DENIED when running notebooks with a group-assigned dedicated compute

Grant workspace access entitlement to the group....

How to disable JDBC on all-purpose compute

Run a cluster-scoped init script....

Lost workers due to autoscaling events

Benign message for termination events....

How to configure a compute policy to include Databricks Runtime versions using ML compute

Identify the corresponding ML runtime versions corresponding to your current Apache Spark versions and update your spark_version configuration. ...

Getting node-specific instead of cluster-wide memory usage data from system.compute.node_timeline

Join the table node_timeline with the table node_types in your query....

Too many execution contexts are open right now

Reduce the number of notebooks used to limit the number of execution contexts required for your job....

Job fails with InvalidParameterValue error and cluster not appearing in the job UI

Set workloadType.clients.jobs to 'true' using the API or the Databricks SDK for Python....

Error “[UNSUPPORTED_OPERATION] data type is not supported” when trying to query a view or table in serverless

Switch to serverless environment version 2....

Job fails with ArrayIndexOutOfBoundsException when migrating from classic to serverless compute

Rewrite the code using the get() function to tolerate invalid index access or disable ANSI SQL. ...

Init script taking longer than expected during package install using apt or pip

Add timestamps and descriptive logs around every major command for easier debugging....

Vertica write operation fails with String.isBlank() error

Either upgrade your Java version to 11 or above, or upgrade Databricks Runtime to 16.4 LTS or above....

Excel rounding off the numbers when downloading the query results

Cast numeric columns to STRING in the query then download the results in Excel....

Notebook run failure with module 'numpy._core' has no attribute 'multiarray' error in Databricks Runtime 16.4 LTS

Pin the compatible NumPy version in the cluster UI or during pip install....

Append to a DataFrame

Learn how to append to a DataFrame in Databricks....

How to improve performance with bucketing

Learn how to improve Databricks performance by using bucketing....

How to handle blob data contained in an XML file

Learn how to handle blob data contained in an XML file....

Simplify chained transformations

Learn how to simplify chained transformations on your DataFrame in Databricks....

Hive UDFs

Learn how to create and use a Hive UDF for Databricks....

Prevent duplicated columns when joining two DataFrames

Learn how to prevent duplicated columns when joining two DataFrames in Databricks....

Revoke all user privileges

Use a regex and a series of for loops to revoke all privileges for a single user....

How to handle corrupted Parquet files with different schema

Learn how to read Parquet files with a specific schema using Databricks....

No USAGE permission on database

User does not have USAGE permission on the database....

Nulls and empty strings in a partitioned column save as nulls

Learn why nulls and empty strings in a partitioned column save as nulls in Databricks....

Behavior of the randomSplit method

Learn about inconsistent behaviors when using the randomSplit method in Databricks....

Generate schema from case class

Learn how to generate a schema from a Scala case class....

How to specify skew hints in dataset and DataFrame-based join commands

Learn how to specify skew hints in Dataset and DataFrame-based join commands in Databricks....

How to update nested columns

Learn how to update nested columns in Databricks....

Incompatible schema in some files

Learn how to resolve incompatible schema in Parquet files with Databricks....

Unable to infer schema for ORC error

Apache Spark returns an error for ORC files if no schema is defined when reading from an empty directory or a base path with multiple subfolders....

User does not have permission SELECT on ANY File

Regular users cannot create tables without permission when access control is enabled....

Sync fails with [UPGRADE_NOT_SUPPORTED.HIVE_SERDE] Table is not eligible for upgrade from Hive Metastore to Unity Catalog

Convert your Hive SerDE tables to Delta format. ...

Parquet table counts not being reflected based on concurrent updates

Manually refresh the table in the notebook where the count was initially taken....

Empty string values convert to NULL values when saving a table as CSV or text-based file format

Use Delta as the target format for CSV files or other text-based data formats....

'CREATE OR REPLACE' SQL error in a Delta table

Correct the job schedule to ensure that only one query is executed at a time for a specific table....

Increased wait times between micro-batches in Auto Loader

Use file notification mode instead of the directory listing method....

Addressing performance issues with over-partitioned Delta tables

Implement liquid clustering for improved performance....

Count of corrupt_records returns zero in serverless

Collect records in an array and get the count of the array instead....

Error INVALID_TEMP_OBJ_REFERENCE when trying to create a view

Persist the temporary object to a location, then create your view....

Error when trying to parse XML in a shared mode cluster using the from_xml()function

Define an XML schema in a Data Definition Language (DDL) string first. ...

Error when trying to create a distributed Ray dataset using from_spark() function

Set spark.databricks.pyspark.dataFrameChunk.enabled to true....

INVALID_PARAMETER_VALUE error when trying to access a table or view with fine-grained access control

Upgrade the cluster's Databricks Runtime version to a version newer than 15.4 and use a single user access mode cluster....

Other users can see root (main) folder despite not having access permissions

Permanently delete items in the Trash and create separate folders for shared files or objects. ...

Execution error when trying to mount a storage account

Unmount the nested path and mount each storage account separately at distinct mount points....

Resolve Spark directory structure conflicts

Fixing java.lang.AssertionError: assertion failed: Conflicting directory structures detected...

Column values assigning in the order they are passed into Row() as arguments, not to the column name indicated

Create the DataFrame from a list of dictionaries or use the row.toDict() method....

Insufficient privileges error when querying views in a Unity Catalog metastore

Conduct a traceback of the permission tree for the view table and grant access where needed....

Getting error com.univocity.parsers.common.TextParsingException when parsing data

Make sure the delimiter used in your read operation matches the format of your input files....

ALTER TABLE command only reordering on metadata level

Use CREATE OR REPLACE TABLE or INSERT OVERWRITE TABLE instead....

Error when creating a DataFrame with values of different data types

Set spark.sql.pyspark.inferNestedDictAsStruct.enabled in the same notebook in a preceding cell....

How to create shallow clones of Delta tables without deletion vectors (DVs) enabled

In the cloned table, disable DVs then if necessary, remove references to DV files....

Queries on Delta Sharing fail with error "Error Message/Issue: Error while reading file delta-sharing"

Enable history on all tables you share using Delta Sharing....

Incorrect aggregation results when working with high decimal precision numbers in Databricks SQL

Use RDDs instead of SUM....

Dataskipping not working on timestamp column and files not getting pruned

Upgrade to Databricks Runtime 16.4 LTS or above, or enable an Apache Spark configuration to allow metadata query optimizations for timestamp data types....

Create tables on JSON datasets

Create tables on JSON datasets; requires SerDe JAR....

Optimize read performance from JDBC data sources

Learn how to optimize performance when reading from JDBC data sources in Databricks....

Failure to detect encoding in JSON

Learn how to resolve a failure to detect encoding of input JSON files when using BOM with Databricks....

Inconsistent timestamp results with JDBC applications

Timestamp records are inconsistent with JDBC applications when daylight saving time adjustments are made....

Kafka client terminated with OffsetOutOfRangeException

Kafka client is terminated with `OffsetOutOfRangeException` when trying to fetch messages...

Recursive references in Avro schema are not allowed

Apache Avro data sources cannot have recursive references in the schema when used with Spark....

SQL access control error when using Snowflake as a data source

Snowflake does not officially support schema as an option; you must use sfschema....

Column drift when reading multiple delimited files

Ensure that all files being processed together have the same schema....

NullPointerException when reading shapefiles from cloud storage on a Mosaic and GDAL enabled cluster

Zip the entire shapefile and upload to a Unity Catalog volume or DBFS storage. ...

MULTIPLE_XML_DATA_SOURCE error while working with XML data

Remove the external XML library from the cluster. ...

ALTER TABLE (drop partition) error in Unity Catalog external tables

For CSV, JSON, ORC, or data formats, use partition metadata logging. ...

“java.lang.IllegalStateException: Unexpected type: JSON” error when creating an external table from BigQuery

Upgrade your cluster to Databricks Runtime 14.0 or above. ...

Schema mismatch issue while reading parquet files

Fix the file schema or read the files separately. ...

Reading a CSV file in DROPMALFORMED still includes malformed rows in the result

...

Cannot see ingested data loaded from an external ORC table

Use the same Hive interface to ingest and read your Delta table. ...

Security Bulletin: Databricks JDBC Driver Vulnerability Advisory - [CVE-2024-49194]

Restart any long running clusters and update your JDBC driver to the latest version....

Error when creating a Delta table using the UI and external data in Delta format

Create the table using a notebook instead. ...

KeyProviderException error when trying to create an external table on an external schema with authentication at the notebook level

Set up authorization at the cluster configuration level instead....

Oracle Federation failing to find a data source

Upgrade compute runtime to 16.1 or use Pro SQL warehouse 2024.50...

Using LIKE statement causing slower performance in Lakehouse Federation query

Replace the LIKE statement in your query with filter options that can be passed as pushdown filters....

Total size of serialized results of tasks is larger than spark.driver.maxResultSize when using ODBC connection

Enable Cloud Fetch to retrieve larger datasets....

Multiple identical files being written to badRecordsPath instead of just one file when writing code to read a CSV file as a DataFrame

Use .option(“mode”, “PERMISSIVE”) instead....

Getting error when trying to connect to SFTP server from Databricks using passwordless authentication

Create an RSA authentication key to access a remote site from your Databricks account and preserve the private key....

CharConversionException when importing non UDF data from IBM Db2 to Databricks

Set driver charset configs....

SQL query on BigQuery table fails with ClassCastException error

Rename the column with an alias or fully specify the table and column names....

Error when trying to read data from MongoDB

Use Lakehouse Federation to access external data sources....

Oracle JDBC connection fails when using keystore and truststore wallets in a standard compute

Use an init script to make the files available to the compute....

Working with spatial data and performing a left-anti join leading to a performance bottleneck

Instead perform a left-outer join then filter on the join key....

Increase in processing time for CSV file with multiline option set to true

Split the CSV file at the source....

FileReadException on DBFS mounted filesystem

Use dbutils.fs.refreshMounts() to refresh mount points before referencing a DBFS path in your Spark job....

Directory view in the workspace UI does not match the result obtained using the dbutils command

Use the file:/ prefix when accessing workspace files via dbutils....

Error while calling o430.mount and the executor has failed to refresh mounts

Use the refreshMounts command....

Download files from DBFS with the web browser

Make sure the DBFS File Browser is enabled if you want to download files from DBFS via the web....

Shell command %sh ls does not work on DBFS files or directories when using a shared cluster

Use a single access mode cluster, dbutils.fs.ls, or migrate to volumes....

UnsupportedOperationException error when trying to run queries interacting with event logs and multiple tables

Split queries into multiple notebook cells or use serverless compute....

Null column values display as NaN

Null column values correctly display as NaN in Databricks SQL....

Retrieve queries owned by a disabled user

How to retrieve queries owned by a disabled user in Databricks SQL....

Job timeout when connecting to a SQL endpoint over JDBC

Increase the SocketTimeout value in the JDBC connection URL to prevent thread requests from timing out....

Slowness when fetching results in Databricks SQL

Ensure that cloud fetch is enabled for best performance when using ODBC/JDBC to fetch results....

ZORDER results in "Hilbert indexing can only be used on 9 or fewer columns" error

OPTIMIZE ZORDER BY command has a hard limit of nine columns....

Cannot customize Apache Spark config in Databricks SQL warehouse

You can only configure a limited set of global Spark properties when using a SQL warehouse....

SQL warehouse launch fails to start with "PERMISSION_DENIED"

Provide the owner with unrestricted cluster creation permissions or workspace admin permissions....

Update the Databricks SQL warehouse owner

Learn how to use the API to transfer ownership of a SQL warehouse to a new owner...

Unable to use dynamic variable passing to create a function in Databricks SQL Warehouse

Use a Python UDF in a notebook to dynamically pass the table name as a variable, then access the function in a notebook or DBSQL. ...

Error running parameterized SQL queries in Databricks Connect with VS Code

Pass the SQL parameter in Databricks Connect using string interpolation. ...

spark.rpc.message.maxSize error after running DESCRIBE HISTORY in SQL warehouse

You must limit or prune the query when using DESCRIBE HISTORY on large history logs....

DeltaInvariantViolationException: Exceeds char/varchar type length limitation error when writing a Delta table

...

PySparkAssertionError: Received incorrect server side session identifier for request

You must detach and reattach to reset the state....

Error 'View/Table Does Not Exist' during model or query runtime in SQL Analytics

Use the command ALTER VIEW to update the view definition....

Unable to read external Hive Metastore tables on SQL warehouses

Manually set the appropriate Apache Spark configurations in your admin settings for SQL warehouses....

Using percentile() to work with large datasets causing memory issues or OOM errors

Use approx_percentile() instead. ...

Databricks SQL Warehouse query finish with cache is not populating ‘open in Spark UI’ option

When a query uses 100% of the cache, there are no corresponding Spark jobs or stages to display in Spark UI....

Jobs in SQL Warehouse returning 403 error

Modify the service account permissions, regenerate the token, and verify permissions are set....

Photon ran out of memory while executing query

Ensure up-to-date statistics on all tables, simplify complex queries, or use Databricks Runtime 13.3 LTS or above. ...

‘Query timeout due to inactivity’ error when using the SQL execution API

Obtain a statement ID from the pending POST API call and make a GET API call instead. ...

ORC tables not recognized when processed in serverless warehouses

Recreate the ORC table as an Apache Spark format ORC table with "USING ORC" instead....

Filtering data with char(255) datatype column does not retrieve a result

Remove the blank space from the right side of the column, cast the column name as a string type, or upgrade to Databricks Runtime 15.4 LTS or above. ...

COPY INTO not loading new data to destination table

Use the ‘force’ = ‘true’ option in COPY_OPTIONS....

Changing ANSI mode inline doesn’t work when querying a view

Recreate the view with the ANSI mode you need, or use try_cast, try_add, or try_divide in query operations that are not ANSI compliant. ...

Using double quotes in a query causes a syntax error

Enable the doubleQuotedIdentifiers setting in your Spark configuration and set spark.sql.ansi.enabled to true....

New columns added to a table are not reflecting in a view

Rebuild the existing view or recreate the view using WITH SCHEMA EVOLUTION....

Not receiving emails from DBSQL alerts

Unblock noreply@databricks.com email from block list....

Error [DELTA_MERGE_INCOMPATIBLE_DECIMAL_TYPE] when performing a merge operation in Databricks Runtime 14.3 LTS or above

Set spark.sql.legacy.decimal.retainFractionDigitsOnTruncate to true or use the cast function to explicitly cast the columns to a specific decimal type....

Find long running queries in your SQL warehouse

Use the query history feature....

Creating a temporary view inside a function returns an [INVALID_TEMP_OBJ_REFERENCE] error

Make the function temporary so both the function and the temporary view have the same lifespan and session scope....

You want to declare temporary variables inside a function in Databricks SQL

Use CTEs to declare temporary variables....

Query using COPY INTO using a direct file directory pattern fails with “ERROR: Job aborted due to stage failure” OOM error

Use INSERT INTO with read_files() instead....

ETL jobs failing in Databricks SQL when casting Oracle NUMBER columns to Delta tables

Store numeric values with precision greater than DECIMAL(38,8) as strings and use Python’s decimal module via UDFs for high-precision arithmetic....

DOUBLE data type-defined columns lose precision for values exceeding 15 significant digits

Convert the column from DOUBLE to DECIMAL(p,s) using ALTER TABLE to preserve full numeric precision....

Using parse_json and explode to flatten data returns [DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE] error

Use the variant_get or variant_explode function to cast the payload to ARRAY first....

SQL warehouse will not launch after upgrading to serverless

Remove the overriding runtime version from the SQL warehouse configuration....

PERMISSION_DENIED error while trying to start a SQL warehouse

Make sure the SQL warehouse owner is an active member of the workspace....

How to pass parameters from workflow to a SQL notebook

Streamline dynamic SQL execution with parameterized queries and workflow configurations....

Getting “This feature is not supported” error when using dropDuplicates() operation on variant type fields

Use a different type or implement your own comparing method....

HTTP 500 error in the Apache Spark UI SQL/DataFrame

Increase the maxSize Spark configuration to process large SQL queries....

Getting REQUEST_LIMIT_EXCEEDED when using catalog information-schema

Reduce information schema query concurrency, add selective filters to the query, and avoid issuing information schema queries too frequently....

Query on Classic or Pro SQL warehouse failing with MetadataFetchFailedException

Change the spot instance policy to Reliability Optimized....

Korean characters breaking when downloading Genie results as CSV

Open the CSV file from within Excel instead to set the correct encoding....

Getting dashboard download successful message but not downloading

Validate the browser restrictions and work with your organization’s IT team to allow Databricks dashboard export....

Using Windows ODBC client to connect to cluster fails with “Bad Status” error

Remove any custom HTTP header entries referencing the HTTP path....

Job failing with NumberFormatException error

You can either temporarily disable CBO or refresh the column statistics to correct the metadata....

Casting numeric hash output to BINARY using CAST fails with [DATATYPE_MISMATCH.CAST_WITHOUT_SUGGESTION] error

Use TO_BINARY or TRY_TO_BINARY instead....

Scientific notation appears when casting DOUBLE to STRING in MERGE

Use the format_number() function or define the source and target columns using DECIMAL type....

Executing a federated query on a MySQL table to cast a TINYINT column failing with “Out of range” error

Set the property spark.databricks.sql.mysql.mapUnsignedTinyIntToShort to true....

SQL warehouse spins up despite no queries executed

Review your JDBC connections and adjust the connection settings....

SQL filters using NOT IN returning zero rows instead of expected number of rows

Use NULL-safe anti-join patterns or explicitly exclude NULLs when using NOT IN....

Smaller tables are not being broadcasted even after increasing ‘autoBroadcastJoinThreshold’ value

Set ‘nonEmptyPartitionRatioForBroadcastJoin’ to 0.0....

Casting DOUBLE values to STRING causes mismatched strings or unreadable outputs

CAST the column to BIGINT or DECIMAL, then CAST as STRING....

SKEW hint in Apache Spark leads to ClassCastException error

Ensure all skew hint values use the same numeric type....

MERGE returns "No rows returned" despite performing upserts

Upgrade Databricks Runtime or disable eager clustering....

Using reserved words as column names showing unexpected behavior

Enable spark.sql.ansi.enforceReservedKeywords and spark.sql.ansi.enabled Apache Spark configurations....

Apache Spark session is null in DBConnect

A `sparkSession is null while trying to executeCollectResult` error message occurs when using DBConnect....

Failed to create process error with Databricks CLI in Windows

Databricks CLI may not work correctly in Windows if your Python path has a space in it....

GeoSpark undefined function error with DBConnect

Use GeoSpark code with a DBConnect session....

Get Apache Spark config in DBConnect

Use a REST API call and DBConnect to get the Apache Spark configuration for your cluster....

ProtoSerializer stack overflow error in DBConnect

A stack overflow error in DBConnect indicates that you need to allocate more memory on the local PC....

Terraform registry does not have a provider error

You cannot install the Databricks Terraform provider if the required_providers block is not defined in your modules....

Databricks Connect job fails after a Databricks Runtime update

Use the most recent version of Databricks Connect that matches your Databricks Runtime version to avoid an error....

Databricks-sql-python package fails with self-signed certificate errors and code _ssl.c:1006

Make sure your certificate is valid and updated....

Paths behave differently on Git folders and workspace folders

Git folders can reference the project root, while workspace folders reference the current working directory....

Content size error when trying to import or export a notebook

Reduce the size of your notebook so it is under 10MB....

Missing TRACE level Terraform log files to troubleshoot template issue

Learn how to enable Terraform log files....

Deploying Databricks Asset Bundles through a CICD pipeline fails with 403 error

Configure the .gitignore file in your GitLab repository to exclude unnecessary hidden directories from the Asset Bundle deployment....

Jobs deployed using Databricks Asset Bundles show RUNNING in the terminal but then seem to hang indefinitely

Use the --no-wait flag with the databricks bundle run command....

Databricks API call to GitHub failing with 403 Forbidden error

Disable the IP allow list in GitHub....

Jobs using Apache Spark 3.5.1 and the Elasticsearch Hadoop connector failing with MicroBatchExecution error

Contact the Elastic team for assistance, or use Databricks Runtime 13.3 LTS and below....

Connection timeout error when trying to connect to Snowflake from Databricks environments

Set up and run SnowCD to diagnose and troubleshoot....

Can’t find where to set email notifications for DAB jobs in the UI

Use YAML syntax to set the notifications....

“Cannot create grants” Terraform error when using multiple databricks_grants blocks for the same catalog

Use dynamic blocks within a single databricks_grants resource or use databricks_grant for ad-hoc permission assignment from Terraform. ...

Databricks Apps - Spark Context initialization fails with Java gateway error

Use the Databricks SQL Connector for Python to query tables in Databricks Apps....

Special characters appearing as junk characters when using the Simba Spark ODBC driver to access Databricks tables from external applications

Set the SIMBA_APP_ANSI_ENCODING environment variable to ISO-8859-15....

Experiencing an exception indicating “ Yaml file exists as” referring to meta.yaml file when creating an MLflow experiment

Add mlflow.set_tracking_uri("databricks") to your code or remove the meta.yaml file referenced in the exception....

How to create an init script to collect tcp_dumps

Edit the cluster configuration to attach the tcp dump init script and collect network packet traces....

Unable to push GitHub Actions workflows from a Databricks repository

Use a Github PAT token with workflow scope....

Python SDK endpoint not found error when trying to use AccountClient

Use OAuth with a service principal to authenticate at the account level....

Volume log cluster configuration detail not displayed using DBCLI or SDK

Update the DBCLI or SDK library....

Databricks dashboards deployed with Databricks Asset Bundles (DAB) are duplicated on deployment

Do not sync source .lvdash files to DABs....

Compare two versions of a Delta table

Use time travel to compare two versions of a Delta table....

Delta Merge cannot resolve nested field

Delta Merge fails with a `Delta Merge cannot resolve 'field' due to data type mismatch` error message....

How Delta cache behaves on an autoscaling cluster

Learn how Delta cache behaves on an autoscaling cluster....

How to improve performance of Delta Lake MERGE INTO queries using partition pruning

Learn how to use partition pruning to improve the performance of Delta Lake MERGE INTO queries....

Best practices for dropping a managed Delta Lake table

Learn the best practices for dropping a managed Delta Lake table....

How to populate or update columns in an existing Delta table

Learn how to populate or update columns in an existing Delta table....

Identify duplicate data on append operations

...

Optimize a Delta sink in a structured streaming application

Optimize your Delta sink by using a mod value on the batchId to optimize when foreachBatch runs....

Unable to cast string to varchar

Use varchar type in Databricks Runtime 8.0 and above. It can only be used in table schema. It cannot be used in functions or operators....

Vaccuming with zero retention results in data loss

Do not disable spark.databricks.delta.retentionDurationCheck.enabled and run vacuum with retention zero to avoid data loss....

Z-Ordering will be ineffective, not collecting stats

Z-Ordering is ineffective, error about not collecting stats. Reorder table so the columns you want to optimize on are within the first 32 columns....

Change cluster config for Delta Live Table pipeline

Customize the cluster configuration when using a Delta Live Table pipeline....

Different tables with same data generate different plans when used in same query

Ensure that tables with the same data generate the same physical plans with Spark SQL....

Allow spaces and special characters in nested column names with Delta tables

Upgrade to Databricks Runtime 10.2 or later and use column mapping mode to allow spaces and special characters in column names....

Delta writing empty files when source is empty

Delta can write empty files under Databricks Runtime 7.3 LTS. You should upgrade to Databricks Runtime 9.1 LTS or above to resolve the issue....

Delta Live Tables pipelines are not running VACUUM automatically

You must have a maintenance cluster defined for VACUUM to run automatically....

VACUUM best practices on Delta Lake

Learn best practices for using, and troubleshooting, VACUUM on Delta Lake....

OPTIMIZE is only supported for Delta tables error on Delta Lake

Use CREATE OR REPLACE TABLE when moving Delta tables from one storage location to another....

Programmatically determine if a table is a Delta table or not

Use Python code in a Databricks notebook to determine if a table is a Delta table or not....

RESOURCE_LIMIT_EXCEEDED error when querying a Delta Sharing table

Delta Sharing has limits on the metadata size of a shared table. If you exceed these limits it generates an error....

Found duplicate columns error blocks creation of a Delta table

Duplicate column names are not allowed in Delta tables....

Hive-style partitions not found on Delta table after enabling column mapping mode

Delta Lake column mapping does not support Hive-style partitions....

Dropping and recreating Delta tables results in a DeltaVersionsNotContiguousException error

Instead of dropping and recreating Delta tables, use the CREATE OR REPLACE command....

"AnalysisException: Incompatible Format Detected" error when writing to OpenSearch

Make sure there is no _delta_log folder in your root directory....

AnalysisException error due to a schema mismatch

Modify the write command and set the mergeSchema property to true....

Timestamp change to underlying Apache Parquet/change data files while using Change Data Capture (CDC)

For timestamp-based queries, ensure that the original file timestamps are preserved during the migration process....

Unable to read Delta table with deletion vectors

Use the cluster with Databricks Runtime 12.2 LTS - 15.3 to query all deletion-vector-enabled Delta tables....

Unknown Apache Spark internal error when running Delta table queries

Reorganize the folder structure for the specific partition causing the problem. ...

Running OPTIMIZE on Delta tables causing ConcurrentDeleteDeleteException error

Disable AUTO OPTIMIZE....

DELTA_CLUSTERING_COLUMN_MISSING_STATS error when attempting to define liquid clustering for a delta table

Ensure that you have generated Delta statistics for the columns used as clustering keys....

INSERT operation fails while trying to execute multiple concurrent INSERT or MERGE operations to append data

Make sure the isolation levels are correctly set or refactor to remove conflicts....

You do not use deletion vectors, but see a file named deletion vector in your data path

It is an artifact of low shuffle merge and is removed on the next VACUUM run....

Databricks Runtime is not able to read data in a format other than Delta

Delete the transactional log folder or move it to a different location. ...

Running a Python UDF fails with permission error

Ask function owner to grant EXECUTE permission....

COUNT operation on a DataFrame returning zero or incorrect number of records

Schedule operations to run sequentially, save the DataFrame to a checkpoint, and/or use snapshot isolation....

Error [DELTA_CLUSTERING_SHOW_CREATE_TABLE_WITHOUT_CLUSTERING_COLUMNS] when running SHOW CREATE TABLE command

Upgrade to Databricks Runtime 15 or above or disable the liquid clustering table feature....

FileReadException error when trying to run streaming job reading from system tables

Increase job frequency or the maxVersionsPerRpc....

DeltaFileNotFoundException when reading a table

Ensure that Delta log files are not getting deleted prematurely....

Error FileNotFoundException while streaming job or reading Delta table even with ignoremissingfiles set

Use the FSCK repair command to synchronize the metadata with the actual data files....

Syntax error when running vacuum with USING INVENTORY command

Upgrade your Databricks Runtime to version 15.2 or above. ...

How to get the full size of a Delta table or partition

Use a Scala command and pass in either the root path or the path to the partition....

Error when trying to copy datasets to different regions using Delta Sharing

Split your intervals into separate columns for day, hour, minute and second; convert the interval to a string, or convert the entire interval to seconds and store as an integer. ...

ETL process fails to process a column and throws error Row group size has overflowed

Reduce the default row group size and increase the frequency of size checks....

InvalidSchemaException error when trying to insert data into a Delta table

Define a field type for any fields that use a StructType within a StructField....

Job fails with ExecutorLostFailure error due to excessive garbage collection (GC)

Broadcast the smaller table instead of the larger one....

Parquet table last modification retrieval returns NULL

List the files within the parquet table path and sort them by the modificationTime column. ...

Time travel SELECT query works on older dates even after VACUUM

This is expected behavior but you can also test that your VACUUM command ran successfully. ...

Unable to exclude columns from a table based on specific strings in the comments

Leverage DESCRIBE TABLE to retrieve and filter on the metadata which includes comments....

Delta table as a streaming source returns error DELTA_FILE_NOT_FOUND_DETAILED even though no user or lifecycle rule has deleted files

Consume the source table from scratch, consume from a specific version, or load data using a specific timestamp as filter....

Symlink format manifest fails when trying to enable liquid clustering on a table

Disable deletion vectors or avoid using Symlink format manifest, as appropriate for your context. ...

DataFrame to Ray Dataset conversions taking a long time to execute

Set spark.task.resource.gpu.amount to 0, modify num_cpus_worker_node, or enable Spark cluster auto-scaling....

Files restored from a Delta table archive are not recognized by Delta with archival support enabled

Temporarily disable archival support or increase the value for the table property delta.timeUntilArchived....

Job failing with DELTA_CHANGE_DATA_FILE_NOT_FOUND error

Use ignoreMissingFile config or a new checkpoint....

Column statistics missing when running ANALYZE TABLE COMPUTE STATISTICS after ANALYZE TABLE COMPUTE STATISTICS FOR ALL COLUMNS

Only run ANALYZE TABLE COMPUTE STATISTICS FOR ALL COLUMNS to ensure column statistics remain available. ...

Error “Job aborted due to stage failure” when trying to shallow clone a large table with significant DeltaLog entries

Increase ​​spark.driver.maxResultSize to 6 GB or greater....

Iceberg metadata not reflecting latest changes made to table, appearing out of sync

Add `MSCK REPAIR TABLE SYNC METADATA` to the end of the ingestion job, or run manually after each ingestion....

Recover a dropped table when a new table is created with the same name

Rename the existing table and then UNDROP the deleted table....

Find the number of files per partition in a Delta table

Retrieve the partition structure of a Delta table and display the number of data files per partition value....

Intermittent long-running OPTIMIZE command for liquid clustered table

Use Databricks Runtime 16.2 or above to run the OPTIMIZE command....

JOIN operation on masked column using masked values instead of unmasked ones

Create a materialized view or hash sensitive columns....

Automate VACUUM metrics logging for Delta table cleanup audits

Parse the operationMetrics from the Delta transaction log using DESCRIBE HISTORY, and store this information in a dedicated Delta audit table....

Track deleted files from VACUUM in Delta table history

Use DESCRIBE HISTORY to find VACUUM operations and check operationMetrics for deleted file count and size....

Casting BigNumeric values to Decimal(38, 38) fails with NUMERIC_VALUE_OUT_OF_RANGE error

Modify the data type to NUMERIC in BigQuery first....

Delta Sharing shared table appearing empty in the target Databricks workspace

Use the correct representation of the data type when adding partition filters....

PySpark merge operation with 'withSchemaEvolution' fails on serverless compute

Use a job or all-purpose compute in Databricks Runtime 15.4 LTS, or use SQL....

Photon memory issue while querying a table with a large number of columns

Use the variant datatype in your CREATE TABLE AS SELECT query, or disable Photon....

Error [DELTA_UNSUPPORTED_FEATURES_FOR_READ] when accessing a Delta table

Use Databricks Runtime 15.3 or above....

How to optimize SQL commands on the JOIN clause

Use UNION in your SQL code. ...

SELECT on VIEW not showing any data in the table after unsetting timezone

Use the Apache Spark configuration spark.conf.set("spark.sql.legacy.useCurrentConfigsForView", "true")....

Cannot alter existing column to Delta Lake generated column

Recreate the table with the desired generated expression. ...

S3 path data size for a Delta table is more than the table size seen from the describe detail output

Execute VACUUM to remove the stale files....

MAX() query on Delta tables performs slowly on string or timestamp columns

Enable metadata query optimization for clustered Delta tables using an Apache Spark configuration....

New timestamps appearing in Delta table history after copying the same data

New commit timestamps in DESCRIBE HISTORY are expected and cannot be preserved when recreating a table....

Running parallel ALTER TABLE ADD partition command fails with ConcurrentAppendException

Add all required partitions in a single ALTER TABLE ADD PARTITION command instead....

Query failing with UC_DEPENDENCY_DOES_NOT_EXIST in Unity Catalog

Ensure all masking functions referenced by the table are present and not deleted from Unity Catalog....

Unauthorized access exception when trying to access a Unity Catalog table with row filters or column masks

Use name-based SQL instead of file-based access....

Distinguish active and dead jobs

Learn how to distinguish between active and dead Databricks jobs....

Spark job fails with Driver is temporarily unavailable

Upgrade the driver node type, avoid memory-heavy operations like collect(), and isolate heavy workloads to reduce queuing and improve cluster reliability....

How to delete all jobs using the REST API

Learn how to delete all Databricks jobs using the REST API....

Job cluster limits on notebook output

Job clusters have a maximum notebook output size of 20 MB. If the output is larger, it results in an error....

Job fails, but Apache Spark tasks finish

Your job fails, but all of the Apache Spark tasks have completed successfully. You are using spark.stop() or System.exit(0) in your code....

Job fails due to job rate limit

Learn how to resolve Databricks job failures due to job rate limits....

Job fails with invalid access token

Jobs that run more than 48 hours fail with invalid access token error when the dbutils token expires....

Task deserialization time is high

Configure cluster-installed libraries to install on executors at cluster launch vs executor launch to speed up your job task runs....

Pass arguments to a notebook as a list

Use a JSON file to temporarily store arguments that you want to use in your notebook....

Uncommitted files causing data duplication

Partially uncommitted files from a failed write can result in apparent data duplication. Adjust VACUUM settings to resolve the issue....

Multi-task workflows using incorrect parameter values

If parallel tasks running on the same cluster use Scala companion objects the wrong values can be used due to sharing a single class in the JVM....

Job fails with Spark Shuffle FetchFailedException error

Disable the default Spark Shuffle service to work around a FetchFailedException error....

Users unable to view job results when using remote Git source

Databricks does not manage permission for remote repos, so you must sync changes with a local notebook so non-admin users can view results....

Single scheduled job tries to run multiple times

Ensure your cron syntax is correct when scheduling jobs. A wildcard in the wrong space can produce unexpected results....

Add custom tags to a Delta Live Tables pipeline

Manually edit the JSON configuration file to add custom tags....

Update notification settings for jobs with the Jobs API

You can use the Jobs API to add email notifications to some, or all, of the jobs in your workspace....

Stop all scheduled jobs

Use the included sample code to stop all of your scheduled jobs in the workspace....

Notebook stopping with file read error even if operation or command is still executing

Consider using Databricks Jobs for long-running operations. ...

Permissions error when trying to run job clusters

Ensure that the service principal has the 'Service Principal User' role....

Idle clusters causing inefficient resource use and increased costs

Set file arrival triggers....

Error when trying to create more new jobs than the limit quota

Confirm the amount of jobs you have in your workspace, then identify and delete the jobs you do not need....

Databricks cannot access a notebook in GitHub

Check your file type or GitHub credentials and permissions....

Unable to pass a param string value of more than 65,535 characters in a workflow using a JAR in a job

Pass the param using a text file in Workspace FileSystem (WSFS). ...

Trigger a job as a specific user with "Run As"

Use the UI or API to run a job as a specific user....

Need to see job creator when investigating a job but can only see service principal

Leverage the API to retrieve job creator details....

'Cluster does not support jobs workload' error during notebook or job run

Use a cluster policy that allows the dbutils.notebooks.run API, or run the code directly within a notebook to avoid the API. ...

Getting NullPointerException when using dbutils.secrets.get in jar jobs

Include the necessary dependencies for dbutils....

Long running jobs are terminated with a WORKSPACE_UPDATE error

Enable continuous jobs for long running workloads....

Filter condition in the for each task type not filtering correctly

Use param = :Param instead. ...

Jobs running longer than expected with 'Metastore_Down' events in event log

Run the VACUUM command to remove stale files, adjust the catalog update thread pool size in Databricks Runtime 14.3 LTS and above, or for read-only metastore databases, disable Delta catalog updates. ...

Previously working jobs now failing to execute with METASTORE_DOES_NOT_EXIST error

Only use the Databricks update job API to update the job cluster....

“No module named” error for dependent libraries within a job task

Set the libraries at the compute level for all-purpose compute, share libraries between tasks in non-serverless job clusters, or select from other options provided....

Job to insert Parquet file to table fails with error (FAILED_READ_FILE.PARQUET_COLUMN_DATA_TYPE_MISMATCH)

Fix the Parquet file’s schema to match the table’s schema....

String aggregation queries failing with data type mismatch error on serverless compute

Use explicit casting or disable ANSI_MODE....

Job with string data failing with [CAST_OVERFLOW_IN_TABLE_INSERT] overflow error

Change the data type of the source to an equivalent type as the target....

How to fetch the CREATE job JSON using an API call instead of the UI

Use the ‘settings’ key in the ‘/api/2.1/jobs/get’ API endpoint....

JDBC write operation fails with HiveSQLException error: The background threadpool cannot accept new task for execution

Change the spark.hive.server2.async.exec.threads, spark.hive.server2.async.exec.wait.queue.size, and spark.hive.server2.async.exec.keepalive.time configs to handle more concurrent asynchronous queries. ...

Error “number of currently active jobs exceeds hard limit of spark.databricks.maxActiveJobs” when trying to run an API request

Optimize job design first to the extent possible, then change the spark.databricks.maxActiveJobs setting to N, depending on your needs. ...

Can’t edit jobs created using Databricks Asset Bundles (DABs) using the UI

Disconnect the job from the source to make the job editable in the UI, or programmatically change the edit_mode field to “EDITABLE”....

Invalid cron syntax error when scheduling multiple values in a job’s day-of-week field

Use job linkage to manage each day-of-week requirement individually....

Apache Airflow-triggered jobs terminating before completing

Increase the job timeout threshold on the Airflow side....

Unable to select a single node cluster when using the default job compute policy

Create a custom job policy....

The dbt option is missing from menu options during job creation

The dbt task option appears in the jobs UI only if your workspace is on the premium or enterprise tier with Databricks Repos (Git integration) enabled....

How to prevent a job hanging at an external API call step

Add timeouts and error handling. ...

Apache Spark job fails with maxResultSize exception

Refactor your code, increase the driver maxResultSize, or enable Cloud Fetch....

Apache Spark job failing with SparkException: Job aborted due to stage failure error on dedicated compute

Execute queries involving fine-grained access control (FGAC) on standard compute....

Concurrent execution of CREATE OR REPLACE FUNCTION statements leads to intermittent ROUTINE_NOT_FOUND errors

Avoid recreating UDFs frequently, implement retry logic, or use explicit DROP and CREATE for UDFs....

How to configure a service principal with Git credentials for jobs with Git source

Add Git credentials for a service principal using the API....

Apache Spark job fails with EncryptionOperationNotPossibleException in Jasypt UDF

Enhance the UDF with input validation and exception handling....

How to remove untitled or stale jobs taking up count in a workspace job limit

Run the provided code blocks to identify untitled and stale jobs for deletion....

Members of a Gmail group email not receiving notifications

Allow external entities to email the group inbox. ...

Apache Spark jobs failing due to stage failure when using spot instances in a cluster

Use on-demand nodes instead of spot instances. ...

Broadcast join hash not being used despite hints

Refresh table statistics or use supported joins that allow for broadcast join....

Error compressed buffer size exceeds 2 GB when saving data

Set the Apache Spark configs to increase the frequency of row group size check....

Error when trying to use RDD code in shared clusters

Use a single-user cluster, which supports RDD functionality....

File corruption error on Apache Spark Streaming jobs during file processing in DBFS

Replace dbutils.fs operations with Hadoop filesystem methods. ...

Jobs failing at data shuffle stage with error org.apache.spark.shuffle.FetchFailedException

Analyze the shuffle data distribution across executors and join query strategies....

Apache Spark job output only giving the first JSON object instead of all records

Add appropriate line breaks between each JSON object or use Photon....

DAB job parameters not passing in correctly on the task level

Correct the syntax....

Recurring Apache Spark jobs with same data set size and cluster configuration vary in duration

Build your cluster with sufficient SSD memory, monitor your cluster’s disk usage, and optimize data storage....

Resolve invalid cast input error on serverless compute

Set spark.sql.ansi.enabled to false to resolve casting errors on serverless compute....

Seeing slow-running jobs while adaptive parallelism enabled

Disable adaptive parallelism....

Apache Spark jobs fail or stall when importing .whl from /Workspace on multi-node clusters

Allow all traffic on ports 1017,1015 and 1021....

Delta table deletion operations intermittently return empty results when data exists

Update your cluster configuration to exclude the optimizer rule....

Task count performance while using serverless compute not showing consistently

Absence of serverless task count performance information at times is expected....

FileNotFound Exception when using Apache Spark addFile on multi-node compute

Read directly from Databricks File System (DBFS) for Spark parallelization....

Cannot import TabularPrediction from AutoGluon

Cannot import TabularPrediction from AutoGluon v0.0.14 due to a namespace collision. Upgrade to AutoGluon v0.0.15....

How to correctly update a Maven library in Databricks

Learn how to correctly update a Maven library in Databricks....

Init script fails to download Maven JAR

Cluster init script fails to download a Maven JAR when trying to install a library....

Install package using previous CRAN snapshot

Avoid a package install error by installing from an earlier CRAN snapshot....

Install PyGraphViz

Install PyGraphViz with all required dependencies....

Install Turbodbc via init script

Install Turbodbc and its dependencies, libboost-all-dev, unixodbc-dev, and python-dev, with an init script....

Cannot uninstall library from UI

Learn what to do when you can't uninstall a library using the Databricks user interface....

Error when installing Cartopy on a cluster

Cartopy installation fails if libgeos and libproj are not installed....

Error when installing pyodbc on a cluster

Learn how to troubleshoot an error when installing pyodbc on a Databricks cluster....

Libraries fail with dependency exception

Learn why notebook-scoped libraries trigger an Apache Spark dependency exception; return a requirement cannot be satisfied error....

Reading .xlsx files with xlrd fails

xlrd no longer supports .xlsx files. Use openpyxl to read .xlsx files....

Remove Log4j 1.x JMSAppender and SocketServer classes from classpath

Remove Log4j 1.x JMSAppender and SocketServer classes from classpath....

Python command fails with AssertionError: wrong color format

Resolve a wrong color format AssertionError caused by nbconvert when a Python command fails....

PyPMML fails with Could not find py4j jar error

...

TensorFlow fails to import

TensorFlow fails to import if you have an incompatible version of protobuf installed on your cluster....

Verify the version of Log4j on your cluster

Verify the version of Log4j installed on your cluster and upgrade if required....

Apache Spark jobs fail with Environment directory not found error

Spark jobs appear to time out after you install a library because security rules are preventing workers from resolving the Python executable path....

Copy installed libraries from one cluster to another

Copy libraries from a source cluster to a target cluster with a custom Python script....

Failed to install Elasticsearch via Maven

If library dependencies are already installed, it can result in a library installation failure....

Cluster fails to start with InvalidGroup.NotFound error

If the network security group policy is not correctly configured your clusters will fail to start....

Add libraries to a job cluster to reduce idle time

How to add libraries to a job cluster and reduce idle time in Databricks...

PyArrow hotfix breaking change

PyArrow versions 0.14 - 14.0.0 contain a security vulnerability....

OpenSSL SSL_connect: SSL_ERROR_SYSCALL error

Use a cluster-scoped init script to install necessary SSL certificates to resolve a SSL_ERROR_SYSCALL error....

Notebook cells fail to run with "Failure Starting repl." and Pandas "check_dependencies" errors

Ensure you do not have a dependency mismatch with the NumPy and/or Pandas versions installed on your cluster....

Virtualenv creation failure due to setuptools >= 71.0.0

Pin setuptools version 70.3.0....

Libraries failing with owner or network errors on Databricks Runtime 13.3 LTS - current (15.3)

Manually adjust your custom index URL. ...

Maven Libraries Start Failing with Timed-Out Errors When Updating to Databricks Runtime 11.3 LTS - 15.3 (current)

Whitelist Maven Central and the new Maven repo....

Paths behave differently on Git folders and workspace folders

Git folders can reference the project root, while workspace folders reference the current working directory....

GDAL library installation

Troubleshooting GDAL init script issues...

Installing lme4 fails with a Matrix version error

Upgrade the version of Matrix to 1.6.2 or above before installing lme4....

User not found error while trying to install a library on a shared cluster

Change cluster ownership and reconfigure all libraries under the new owner....

Init script to set up Dask library fails and cluster won’t start

Modify the initialization script to include a validation check for the required environment variables first....

Office365 library installation causes numpy.dtype size change error while executing notebook commands

Pin the Moviepy library version that uses the NumPy version compatible with your Databricks Runtime version....

Fixture not found error when using pytest on a cluster

Downgrade pytest to version 8.3.2 or upgrade Databricks Runtime to 16.1 or above....

Error when attempting to use 'stream_read_table' function from Sparklyr in 14.3 LTS ML

Install Sparklyr version 1.8.6 in the cluster's libraries, restart the cluster, and rerun the function to ensure the error is fixed....

Library installation attempted on the driver node of the cluster failed

Uninstall and reinstall the library. ...

Py4JJavaError when trying to install libraries on SSL-encrypted cluster

Set the Apache Spark configuration to disable SSL....

Unable to install R package 'Survminer' on cluster

Use an init script to install a compatible version of R and the necessary dependencies....

DLTImportException error when importing the DLT module

Use a cluster-scoped init script targeting the job or cell commands in a notebook. ...

ClassNotFoundException error when executing a job or notebook with a custom Kryo serializer

Use an init script or use the spark.jars property in your Apache spark configuration....

Error during Maven library installation: ERROR_MAVEN_LIBRARY_RESOLUTION

Install libraries separately, host a private Maven mirror, or if necessary use another Maven mirror....

Error when attempting to install torch for R package

Install the required dependencies before installing torch for R....

Getting ValueError when trying to import PMML files using PyPMML

Remove -XX:+PrintFlagsFinal flag from Java options....

Getting timeout error during Maven library installation on Databricks cluster

Configure your cluster to use private repos as the default repository, and disable the default Maven Central resolver and Apache Spark packages resolver....

Notebook or workflow fails with “Error : Py4JError: Could not find py4j jar at” error after trying to install PyPMML on a cluster

Install your Py4J library into the expected location....

Error while using the Unstructured library

Use apt-get to install poppler-utils....

Uploaded artifacts to volume using Databricks Asset Bundles (DAB) not appearing

In your DAB YAML file, set the ‘workspace.artifact_path’ property to the desired volume path....

ArcGIS library installation fails with subprocess-exited-with-error

Install the missing dependency using an init script....

Saving an Excel file to Unity Catalog location returning a "RETRIES_EXCEEDED" exception

Use a Databricks Runtime version with a compatible, tested Apache Spark version. ...

Error when trying to install cx_Oracle library on a standard access mode compute

Create an init script to install the library in a custom location outside of '/databricks/*`....

Upgrading to the latest mssql-jdbc driver in Databricks Runtime manually

Use an init script....

xlwings Python library initialization failing

Upgrade to xlwings PRO....

Job deployed with DAB to install packages from private PyPI-compliant repository failing

Configure the PIP_EXTRA_INDEX_URL environment variable as part of the cluster specification in the databricks.yml file....

ModuleNotFound Error on serverless compute after package install and Python restart

Persist src/ in your workspace or repo and re-add it to sys.path after restart....

Conda fails to download packages from Anaconda

Conda fails to download packages with PackagesNotFoundError when you try to install packages from Anaconda....

Download artifacts from MLflow

How to download artifacts from MLflow to local storage....

How to extract feature information for tree-based Apache SparkML pipeline models

Learn how to extract feature information for tree-based ML pipeline models in Databricks....

Fitting an Apache SparkML model throws error

Learn how to resolve errors thrown by Databricks when fitting a SparkML model or pipeline....

H2O.ai Sparkling Water cluster not reachable

H2O.ai Sparkling Water cluster not reachable if the version of the Sparkling Water package does not match the version of Spark used on your cluster....

How to perform group K-fold cross validation with Apache Spark

Learn how to perform group K-fold cross validation with Apache Spark on Databricks....

MLflow project fails to access an Apache Hive table

Resolve "Table or view not found" error when an MLflow project fails to access an Apache Hive table....

How to speed up cross-validation

Learn how to improve cross-validation performance in SparkML with Databricks....

Hyperopt fails with maxNumConcurrentTasks error

Do NOT install Hyperopt on a Databricks Runtime for Machine Learning cluster....

Incorrect results when using documents as inputs

Your model does not return expected results when documents are input using TfidfVectorizer. JSON array...

Experiment warning when custom artifact storage location is used

Resolve experiment warnings when a custom artifact storage location is used instead of the MLflow managed location....

Experiment warning when legacy artifact storage location is used

Resolve experiment warnings when a legacy artifact storage location is used instead of the MLflow managed location....

KNN model using pyfunc returns ModuleNotFoundError or FileNotFoundError

Predictions using pyfunc on a KNN model returns a ModuleNotFoundError or FileNotFoundError....

OSError when accessing MLflow experiment artifacts

Resolve an `OSError` when trying to access, download, or log MLflow experiment artifacts....

PERMISSION_DENIED error when accessing MLflow experiment artifact

Resolve a PERMISSION_DENIED error when trying to access MLflow experiment artifacts....

Runs are not nested when SparkTrials is enabled in Hyperopt

When SparkTrials is enabled in Hyperopt, MLflow runs are not nested under the parent run....

Unable to update a serving endpoint even with necessary permissions

Find the creating user in the workspace and re-add the permissions....

Status FAILURE when attempting to update a Direct Vector Access Index

Ensure that the embedding_dimension parameter matches the length of the embedding field. ...

Table not available while creating AutoML experiment model

Migrate to Unified Compute (UC), with the option to use AutoML with the Python API in the meantime....

Failing API calls in MLflow because of Float64 column values

Explicitly log the input schema while creating the model....

Trying to load an MLflow model using a Python script returns Py4JJavaError

Upgrade MLflow to version 2.15.0 or above. ...

TypeError with an unexpected keyword argument 'query_type' when attempting to perform hybrid similarity search using the databricks-vectorsearch package

Ensure you have the latest version of the databricks-vectorsearch package installed in your environment....

Error message when trying to send a JSON input data request to a model endpoint

Pass one more input parameter from the list of input columns in the endpoint’s model signature....

CUDA out of memory error message in GPU clusters

Change the GPU device used by your driver and/or worker nodes....

Slow model fitting when implementing Alternating Least Squares using Apache Spark PySpark

Override the block default to match the total cores available and consider using compute-optimized instances. ...

Vector search index queries with the `%` character not performing partial-string matches

When using the `LIKE` operator in vector search filters, specify the exact string you want to match...

Column name error when using Apache Spark Mlib feature transformers

When flattening the DataFrame, rename nested columns using an underscore instead of a dot....

Logging a model with MLflow in a PySpark pipeline throws a TempDir class assertion error

Upgrade your MLflow version to 2.16.0 or higher....

MLflow API 429 errors when transitioning models

Add retry logic with exponential backoff to avoid hitting the rate limit....

Model serving endpoint creation succeeds but deployment fails and error stack trace has message _ARRAY_API not found

Include NumPy as a pip dependency and specify the version range to be installed. ...

Slowness when using the foundational model API with pay-per-token mode

Switch to provisioned throughput mode for high throughput and performance guarantee requirements....

Getting ValueError: ndarray is not supported by dataframe_to_mds when converting an Apache Spark DataFrame to MDS format using Mosaic Streaming

Properly pass the data type of the elements of the array column in the mds_kwargs....

Spark ML to ONNX Model Conversion does not produce the same model - predictions differ

Define the TARGET_OPSET and then pass it as the target_opset parameter of the convert_sparkml function. ...

MLflow exception error when trying to migrate models from Workspace Model Registry

Set the registry URI to the Workspace Model Registry before running the MLflow operations. ...

Loading models using MLflow causes TypeError around unexpected number of arguments

Run your cluster using a Databricks Runtime version that has the same Python version as the one used to log and register your model....

ExecutionException error when trying to use Conda as an environment manager in MLflow

Update the MLflow environment manager from Conda to virtualenv. ...

MLflow error "INVALID_PARAMETER_VALUE" during model training and logging process

Ensure each MLflow run maintains a unique set of parameters or use nested runs to log each parameter distinctly within a session. ...

SparkException error when trying to use an Apache Spark UDF to create and dynamically pass a prompt to the ai_query() function

Use Unity Catalog (UC) UDFs instead of Spark UDFs. ...

“Connection pool is full” error when pulling models from S3 with MLflow

Increase the maximum size of the MLflow connection pool....

Using MLflow API call to load a model taking the same amount of time every call and artifacts downloading from scratch

Save the model artifacts locally and then load the model from a local path....

java.lang.OutOfMemoryError error when using collect() from sparklyr

Use arrow_collect() in a custom function to avoid Spark’s 2GB limit when collecting large datasets in R....

Parameter workload_size always executing SMALL when using the databricks-agents library to update existing model serving endpoints

Update the databricks-agents library to version 0.17.0 or later....

'INVALID_PARAMETER_VALUE' error when creating a Google Vertex AI serving endpoint

Ensure the entire private key is used....

Model serving endpoint creation fails with BadRequest error

Shorten the catalog name or specify the endpoint_name explicitly when using agents.deploy....

Not enough disk space error when downloading a model from Hugging Face

Change the default download directory from the root partition to a location with available space....

SHAP figure not appearing in the artifacts after running the mflow.evaluate() call, despite setting log_model_explainability = True

Ensure that all features are in the expected format and re-run the call. ...

Error when trying to use VectorAssembler while on a standard access mode cluster with a non-ML Databricks Runtime

Use a Dedicated (formerly single user) access mode cluster, assigned to a group of users....

Getting NoModuleFound or attribute error when using the Flash Attention model in MLflow

Use mlflow.transformers.log_model with a custom wheel version of flash-attn....

Lineage for the output table with predictions is not tracked in MLflow when training a model

Save the output table as a CSV file and log it as an artifact. ...

GPU metrics indicate that the GPU is not being used during model inference

Ensure that you are sending the model to the GPU in your code....

AttributeError: 'ExportMetricsResponse' when retrieving serving endpoint metrics

Use a custom Python script to download and save the serving endpoint metrics to a file in the Prometheus format....

MlflowClient().search_runs returns only a subset of runs

Use mlflow.search_runs() instead of MlflowClient.search_runs()....

Function ai_similarity failing with “Unexpected server response" error

Remove NULL values from the data before passing it to the function. ...

Tackling schema issues that arise for ML models trained outside of Databricks

Use the code from the external environment to retrain the model within Databricks before fine-tuning....

Serving an AutoML model failing when deployed to an endpoint with "Failed to deploy modelName: served entity creation aborted" error

Ensure the model environment includes an explicit version pin for NumPy....

SSL error when invoking Databricks model serving endpoint

Reduce or split the payload size....

PERMISSION_DENIED error while running AutoML experiment with group-assigned cluster

Create a /Workspace/Groups/ folder for each group you plan to use with a group cluster to ensure correct permissions....

CUDA OutOfMemoryError tried to allocate MiB while performing model training on the GPU compute

Use higher GPU nodes or reduce precision from 32 bit to 16 bit....

Creation failure error when trying to create a vector search index

Use a text embedding model....

Receiving a CuDNN version mismatch error when running TensorFlow within a 16.3 ML runtime environment

Downgrade TensorFlow to a version that is compatible with CuDNN 9.1 (the version bundled with PyTorch in the environment)....

Google AI Studio key fails with Mosaic AI Model Serving through Vertex AI provider

Use Mosaic AI Model Serving with custom provider and OpenAI-compatible endpoint....

Streamlit app deployed as Databricks App failing with JAVA_GATEWAY_EXITED error

Replace the direct SparkSession instantiation with a supported remote-connection SDK or driver. ...

API permission error when using a Vertex AI model in your notebook

Explicitly set the project and credentials in your notebook before making Vertex AI API calls....

Tag update failure on serving endpoint

Upgrade databricks-sdk to0.53.0 or greater, then use EndpointTag objects with add_tags....

Time zone conversion is not visibly applied when using display() on timezone-aware pandas datetime columns

Format timezone-aware datetime columns as strings with offsets before displaying, and optionally configure the Apache Spark session’s time zone....

No space left on device while downloading Hugging Face model

Use a scalable storage to download the model....

Error "You cannot use dbutils within a Spark job" when registering a model in Databricks

Re-instantiate WorkspaceClient when needed, rather than storing it as an attribute of an object....

Autoscaling is slow with an external metastore

Improve autoscaling performance by only installing metastore jars to the driver....

Data too long for column error

If a column exceeds 4000 characters it is too big for the default datatype and returns an error....

Drop database without deletion

Use Hive commands to drop a database without deleting the underlying storage folder....

Error in CREATE TABLE with external Hive metastore

CREATE TABLE error with MySQL 8.0 in external Hive metastore due to charset....

Japanese character support in external metastore

Use Japanese characters in tables in an external metastore....

Parquet timestamp requires Hive metastore 1.2 or above

Update the Hive metastore to version 1.2 or above to use TIMESTAMP with a Parquet table....

Error while establishing user sessions to Oracle database through an external Hive metastore

Increase the session limit on your Oracle database or increase your pool size using an Apache Spark configuration....

Learn about Apache Hive metastore costs

The Hive metastore does not have direct costs, but it does have associated use costs....

Seeing unexpected system generated queries in the query history

These system generated queries are part of normal Databricks behavior and are safe....

Fetching the last access date/time of tables across all workspaces runs slowly and inefficiently

Databricks recommends using audit log system tables. ...

Getting a 404 error when creating a serverless budget policy with the workspace-level API

Use the account-level API instead....

JSON reader parses values as null

When you read a JSON file, the Spark JSON reader returns null values instead of the actual data....

display() does not show microseconds correctly

Use show() to display timestamp values with microsecond precision. display() is limited to millisecond precision....

Error: Received command c on object id p0

You see the error message `INFO:py4j.java_gateway:Received command c on object id p0` after running Python code with imported libraries....

Failure when accessing or mounting storage

Do not mount storage to the root mount path....

Update job permissions for multiple users

Use the job permissions API to update permissions for multiple users....

Generate browser HAR files

Learn how to record HAR files in your web browser. These are very useful when troubleshooting UI issues....

Get workspace configuration details

Display the complete configuration details for your Databricks workspace....

Iterate through all jobs in the workspace using Jobs API 2.1

Use the Jobs API 2.1 to iterate through and display a list of jobs in your workspace....

Generate a list of all workspace admins

Use the included sample code to generate a list of all workspace admins....

%run magic command not working as expected with %python listed first

Ensure that the %run command is the first line in the command cell....

Permissions error when accessing Unity Catalog tables

Use a shared cluster or warehouse, use serverless, or upgrade to Databricks Runtime 15.4 LTS or above. ...

Error when trying to read a notebook from another notebook in the same workspace

Use the Databricks SDK to read a notebook’s contents. ...

Git-integrated workloads fail in Databricks with “PERMISSION_DENIED: Invalid Git provider credentials” error

Check Git provider credentials and token settings....

Unable to import from DBC file: INVALID_PARAMETER_VALUE error

You must set "direct_download": "true" when exporting via the API....

404 Client Error: Not Found for url while running Delta Sharing client load_as_spark command

Check your endpoint URL and use the correct external location URI....