DatabricksDatabricks StaffExplainer
What is Real-Time Mode in Apache Spark?
Summary
Real-Time Mode is a trigger for Apache Spark Structured Streaming that processes each record as soon as it arrives, reducing end-to-end latency from seconds to milliseconds. It replaces micro-batch scheduling with long-running tasks, pipeline scheduling, and a streaming shuffle to remove batch-boundary waits and blocking shuffles. The trigger interval controls checkpoint frequency, not record wait time, and records flow continuously through long-running tasks. Real-Time Mode supports Kafka-based sources and sinks, row-based foreach sinks, and most streaming operators including stateful ones like aggregations and transformWithState. In published benchmarks, it achieves single-digit millisecond latency for stateless queries and roughly 140 ms p99 for stateful queries.
Context
Before Real-Time Mode, teams that needed sub-second responses often added a second, stream-only engine. Real-Time Mode removes that requirement for most workloads by bringing Spark streaming from seconds down to milliseconds of end-to-end latency on the same engine and APIs used for batch and micro-batch streaming.
Approach / What changed
Real-Time Mode replaces micro-batch scheduling with long-running tasks that keep reading and processing data, removing batch-boundary waits and blocking shuffles. It uses continuous data flow, pipeline scheduling, and a streaming shuffle so records move from source to sink without waiting for batch boundaries. The trigger interval controls how often Spark commits a checkpoint and reports progress, not how long a record waits.
Takeaways
Real-Time Mode replaces micro-batch scheduling with long-running tasks, pipeline scheduling, and a streaming shuffle to eliminate batch-boundary waits and blocking shuffles.
The trigger interval controls how often Spark commits a checkpoint and reports progress, not how long a record waits; records flow continuously through long-running tasks.
Real-Time Mode supports stateful operators including aggregations, deduplication, and transformWithState, but cannot write directly to Delta Lake; results must be written to Kafka and then to Delta via a separate micro-batch query.
FAQ
What is Real-Time Mode in Spark?
Real-Time Mode is a Structured Streaming trigger that processes records continuously as they arrive instead of in micro-batches. It brings Spark streaming down to millisecond, end-to-end latency without a code rewrite.
Is Real-Time Mode the same as continuous processing?
No. Continuous processing was an earlier experimental mode with limited operator support. Real-Time Mode is a new design with long-running tasks, pipeline scheduling and a streaming shuffle, and it supports stateful operators.
Can Real-Time Mode write to Delta Lake?
Not directly. Real-Time Mode writes to Kafka or a foreach sink. To land results in a Delta table, run a separate micro-batch query from Kafka to Delta.
Source and freshness
- Source
- databricks.com
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.