Updated September 12th, 2024 by raahat.varma
Cluster terminates automatically even if operation or command is still executing
Problem While executing a long-running operation or command within a notebook on an interactive cluster, you notice the cluster terminates automatically. This disrupts your workflows, results in incomplete processes, and requires the need to restart the operation. Cause The Workspace File System (WSFS) token for interactive sessions has a 36-hour ti...
0 min reading timeUpdated September 12th, 2024 by raahat.varma
Apache Spark is configured to suppress INFO statements but they overwhelm logs anyway
Problem You receive INFO statements despite configuring the Apache Spark settings to suppress INFO and give specific WARN statements. This issue is observed even after setting the ' py4j' logger to WARN and configuring the logging in the Spark config in the Databricks UI to WARN . The problem persists, leading to an overflow of INFO logs, which can...
0 min reading timeLoad More