Updated September 12th, 2024 by shubham.chhabra
Databricks spark-submit jobs appear to “hang” and clusters do not auto-terminate
Problem Databricks spark-submit jobs appear to “hang,” either after the user class’s main method completes (regardless of success or failure, for Java / Scala jobs), or upon a Python script exit (for Python jobs). Clusters do not auto-terminate in this case. Cause For Java, Scala, and Python programs, Spark does not automatically call System....
0 min reading timeLoad More