Loading…
Topic
Structured Streaming
3 posts about Structured Streaming. Every summary links to the original.
- 3 posts
- 1 company
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
What is Real-Time Mode in Apache Spark?
Real-Time Mode is a trigger for Apache Spark Structured Streaming that processes each record as soon as it arrives, reducing end-to-end latency from seconds to milliseconds. It replaces micro-batch scheduling with long-running tasks, pipeline scheduling, and a streaming shuffle to remove batch-boundary waits and blocking shuffles. The trigger interval controls checkpoint frequency, not record wait time, and records flow continuously through long-running tasks. Real-Time Mode supports Kafka-based sources and sinks, row-based foreach sinks, and most streaming operators including stateful ones like aggregations and transformWithState. In published benchmarks, it achieves single-digit millisecond latency for stateless queries and roughly 140 ms p99 for stateful queries.
Read, don't copy: replacing file feeds with OpenSharing - Part 2 (How to Implement)
OpenSharing replaces file feeds by letting recipients read Delta tables incrementally without copying data; the data stays in the provider’s storage, retaining lineage and access control. The provider creates a share WITH HISTORY and, optionally, enables Change Data Feed, which becomes automatic in Databricks Runtime 19 or above for tables with row tracking enabled. The recipient mounts the share as a catalog and uses spark.readStream with Trigger.AvailableNow or a table update trigger to process only newly available data. A demonstration using commit history confirms the checkpoint tracks the last version read: a second run writes only 200 new rows instead of reprocessing the full 1200 rows. Operational advantages include a single governed source table instead of separate exports, backfills using historical table versions, and clear boundaries: the provider owns the shared data product, and the recipient owns downstream processing.
Introducing Apache Spark® 4.3 - Now available in Databricks Runtime 19
Apache Spark 4.3 introduces stateful processing in Real-Time Mode, achieving sub-150 ms P99 end-to-end latency for streaming aggregation, deduplication, and custom stateful processing using transformWithState. This eliminates the need for a separate streaming engine for millisecond-scale latency in fraud detection, user activity monitoring, and device monitoring, enabled by native concurrent stage scheduling and pipelined shuffles. The release also includes Project Feather for faster iteration on small data by reducing execution overhead, and SCD Type 2 Auto CDC for tracking record changes over time. SQL enhancements include JSON_TABLE, VARIANT transformations, ASOF JOIN for temporal matching, and a nanosecond timestamp preview. Spark 4.3 marks a shift to quarterly feature releases, with over 200 contributors addressing more than 1,100 Jira tickets, and is now available in Databricks Runtime 19.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.