Auto Loader does not pick up files for processing when uploading via an Azure function
Problem When you upload files to a source location using an Azure function in Auto Loader, Auto Loader does not pick up the files for processing. The files are also not available in the queue, which Auto Loader sets automatically. The process does work, however, with manual intervention. Cause Auto Loader listens for the ‘ FlushWithClose ’ event to...
0 min reading timeIncorrect input record count in Apache Spark streaming application logs/micro-batch metrics
Problem When observing the logs of Spark Streaming applications, you notice the metric numInputRows in the micro batch metrics or number of input records, event.progress.numInputRows , logged using the StreamingQueryListener does not match the expected count. This leads to confusion about the actual number of records being processed. Cause A di...
0 min reading timeUnclear how to control micro-batch size on a streaming table in Delta Live Tables (DLT)
Problem You want to c ontrol the micro-batch size on a streaming table, which is created in the same Delta Live Tables (DLT) pipeline using rate limiters, but it is not clear how to achieve this in DLT. Cause The dlt.readStream() function in Delta Live Tables (DLT) does not directly support the rate limit configuration maxBytesPerTrigger option. ...
0 min reading time