Loading…
Topic
Real-Time Mode
2 posts about Real-Time Mode. Every summary links to the original.
- 2 posts
- 1 company
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
What is Real-Time Mode in Apache Spark?
Real-Time Mode is a trigger for Apache Spark Structured Streaming that processes each record as soon as it arrives, reducing end-to-end latency from seconds to milliseconds. It replaces micro-batch scheduling with long-running tasks, pipeline scheduling, and a streaming shuffle to remove batch-boundary waits and blocking shuffles. The trigger interval controls checkpoint frequency, not record wait time, and records flow continuously through long-running tasks. Real-Time Mode supports Kafka-based sources and sinks, row-based foreach sinks, and most streaming operators including stateful ones like aggregations and transformWithState. In published benchmarks, it achieves single-digit millisecond latency for stateless queries and roughly 140 ms p99 for stateful queries.
Apache Spark vs Apache Flink: which should you use?
Apache Spark is a unified engine for batch, streaming, SQL, and AI, while Apache Flink is a stream-native processor designed for specialized event-processing workloads. Spark’s Real-Time Mode now delivers millisecond end-to-end latency, matching Flink on stateless transformations and achieving lower p99 latency on stateful aggregation and stream-static joins in Databricks’ published benchmark. Production users like MakeMyTrip chose Spark Real-Time Mode to avoid operating a second engine, cutting P50 latency to 44 milliseconds and P99 from over a minute to about 500 milliseconds. Flink remains the better fit for specialized workloads such as very large keyed state, complex event processing with CEP or SQL MATCH_RECOGNIZE, or teams that already run a Flink platform. Choosing Spark avoids duplicating code, skills, and governance across two engines, as the same DataFrame and SQL logic runs as batch, micro-batch, or continuous stream.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.