DatabricksDatabricks StaffBenchmark
Apache Spark vs Apache Flink: which should you use?
Summary
Apache Spark is a unified engine for batch, streaming, SQL, and AI, while Apache Flink is a stream-native processor designed for specialized event-processing workloads. Spark’s Real-Time Mode now delivers millisecond end-to-end latency, matching Flink on stateless transformations and achieving lower p99 latency on stateful aggregation and stream-static joins in Databricks’ published benchmark. Production users like MakeMyTrip chose Spark Real-Time Mode to avoid operating a second engine, cutting P50 latency to 44 milliseconds and P99 from over a minute to about 500 milliseconds. Flink remains the better fit for specialized workloads such as very large keyed state, complex event processing with CEP or SQL MATCH_RECOGNIZE, or teams that already run a Flink platform. Choosing Spark avoids duplicating code, skills, and governance across two engines, as the same DataFrame and SQL logic runs as batch, micro-batch, or continuous stream.
Context
Apache Spark is a unified engine for batch, streaming, SQL, and AI; Apache Flink is a stream-native processor. The latency gap that once justified running both engines has narrowed with Spark’s Real-Time Mode, which delivers millisecond end-to-end latency. Teams must decide whether to operate one engine (Spark) or two (Spark plus Flink).
Approach / What changed
Use Real-Time Mode, a trigger on an existing Structured Streaming query, to achieve millisecond latency without rewriting code. The same DataFrame and SQL logic runs as batch, micro-batch, or continuous stream.
Takeaways
Spark Real-Time Mode processes records continuously with millisecond latency, matching Flink on stateless work and outperforming it on stateful aggregation and stream-static joins in Databricks’ benchmark.
Real-Time Mode is a trigger on an existing Structured Streaming query, so the same DataFrame and SQL code moves between latency tiers without a rewrite.
Flink is the better fit for very large keyed state, complex event processing with CEP or SQL MATCH_RECOGNIZE, or teams that already run a Flink platform for standalone streams.
FAQ
Is Apache Spark faster than Apache Flink?
In Databricks' published benchmark, Spark Real-Time Mode matched Flink on stateless workloads and had lower p99 latency on stateful aggregation and stream-static joins. Results depend on the workload, so benchmark your own pipeline.
Is Spark streaming still micro-batch?
Not only. Spark Structured Streaming defaults to micro-batch, and Real-Time Mode processes records continuously with millisecond latency. You choose per query by setting the trigger.
When is Flink the better choice?
Flink can fit specialized workloads such as very large keyed state, complex event processing, or teams that already run a Flink platform for standalone streams.
Source and freshness
- Source
- databricks.com
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.