---
title: "What is Real-Time Mode in Apache Spark?"
description: "Real-Time Mode is a trigger for Apache Spark Structured Streaming that processes each record as soon as it arrives, reducing end-to-end latency from seconds to milliseconds. It replaces micro-batch scheduling with long-running tasks, pipeline scheduling, and a streaming shuffle to remove batch-boundary waits and blocking shuffles. The trigger interval controls checkpoint frequency, not record wait time, and records flow continuously through long-running tasks. Real-Time Mode supports Kafka-based sources and sinks, row-based foreach sinks, and most streaming operators including stateful ones like aggregations and transformWithState. In published benchmarks, it achieves single-digit millisecond latency for stateless queries and roughly 140 ms p99 for stateful queries."
---

# What is Real-Time Mode in Apache Spark?

[Databricks](https://yomu.fyi/company/databricks) · Databricks Staff · Nov 16, 2026

**Type:** Explainer

## Summary

Real-Time Mode is a trigger for Apache Spark Structured Streaming that processes each record as soon as it arrives, reducing end-to-end latency from seconds to milliseconds. It replaces micro-batch scheduling with long-running tasks, pipeline scheduling, and a streaming shuffle to remove batch-boundary waits and blocking shuffles. The trigger interval controls checkpoint frequency, not record wait time, and records flow continuously through long-running tasks. Real-Time Mode supports Kafka-based sources and sinks, row-based foreach sinks, and most streaming operators including stateful ones like aggregations and transformWithState. In published benchmarks, it achieves single-digit millisecond latency for stateless queries and roughly 140 ms p99 for stateful queries.

## Context

Before Real-Time Mode, teams that needed sub-second responses often added a second, stream-only engine. Real-Time Mode removes that requirement for most workloads by bringing Spark streaming from seconds down to milliseconds of end-to-end latency on the same engine and APIs used for batch and micro-batch streaming.

## Approach / What changed

Real-Time Mode replaces micro-batch scheduling with long-running tasks that keep reading and processing data, removing batch-boundary waits and blocking shuffles. It uses continuous data flow, pipeline scheduling, and a streaming shuffle so records move from source to sink without waiting for batch boundaries. The trigger interval controls how often Spark commits a checkpoint and reports progress, not how long a record waits.

## Takeaways

- Real-Time Mode replaces micro-batch scheduling with long-running tasks, pipeline scheduling, and a streaming shuffle to eliminate batch-boundary waits and blocking shuffles.
- The trigger interval controls how often Spark commits a checkpoint and reports progress, not how long a record waits; records flow continuously through long-running tasks.
- Real-Time Mode supports stateful operators including aggregations, deduplication, and transformWithState, but cannot write directly to Delta Lake; results must be written to Kafka and then to Delta via a separate micro-batch query.

## FAQ

### What is Real-Time Mode in Spark?

Real-Time Mode is a Structured Streaming trigger that processes records continuously as they arrive instead of in micro-batches. It brings Spark streaming down to millisecond, end-to-end latency without a code rewrite.

### Is Real-Time Mode the same as continuous processing?

No. Continuous processing was an earlier experimental mode with limited operator support. Real-Time Mode is a new design with long-running tasks, pipeline scheduling and a streaming shuffle, and it supports stateful operators.

### Can Real-Time Mode write to Delta Lake?

Not directly. Real-Time Mode writes to Kafka or a foreach sink. To land results in a Delta table, run a separate micro-batch query from Kafka to Delta.

**Tags:** [Apache Spark](https://yomu.fyi/topic/apache-spark), [Databricks](https://yomu.fyi/topic/databricks), [Real-Time Mode](https://yomu.fyi/topic/real-time-mode), [Structured Streaming](https://yomu.fyi/topic/structured-streaming)

- Source: [Databricks](https://www.databricks.com/blog/what-is-real-time-mode)
- Source URL: https://www.databricks.com/blog/what-is-real-time-mode
- Ingested by Yomu: 2026-11-16T18:00:15.915Z

[Read original post](https://www.databricks.com/blog/what-is-real-time-mode)
