Unable to infer schema for ORC error

Problem You are trying to read ORC files from a directory when you get an error message: org.apache.spark.sql.AnalysisException: Unable to infer schema for ORC. It must be specified manually. Cause An Unable to infer the schema for ORC error occurs when the schema is not defined and Apache Spark cannot infer the schema due to: An empty directory. Us...

1 min reading time

High-cached memory in clusters but no active applications submitted

Problem You notice an unexpectedly high-cached memory in your clusters but no active applications submitted. This issue becomes apparent when you observe significant cached data in your  Metrics tab. Cause Running multiple Apache Spark streaming applications or notebooks on the cluster can lead to memory overload on an interactive cluster in limited...

0 min reading time