---
title: "Introducing Apache Spark® 4.3 - Now available in Databricks Runtime 19"
description: "Apache Spark 4.3 introduces stateful processing in Real-Time Mode, achieving sub-150 ms P99 end-to-end latency for streaming aggregation, deduplication, and custom stateful processing using transformWithState. This eliminates the need for a separate streaming engine for millisecond-scale latency in fraud detection, user activity monitoring, and device monitoring, enabled by native concurrent stage scheduling and pipelined shuffles. The release also includes Project Feather for faster iteration on small data by reducing execution overhead, and SCD Type 2 Auto CDC for tracking record changes over time. SQL enhancements include JSON_TABLE, VARIANT transformations, ASOF JOIN for temporal matching, and a nanosecond timestamp preview. Spark 4.3 marks a shift to quarterly feature releases, with over 200 contributors addressing more than 1,100 Jira tickets, and is now available in Databricks Runtime 19."
---

# Introducing Apache Spark® 4.3 - Now available in Databricks Runtime 19

[Databricks](https://yomu.fyi/company/databricks) · Jungtaek Lim, Wenchen Fan, Andreas Neumann, Serge Rielau, Hyukjin Kwon, Jerry Peng, DB Tsai, Daniel Tenedorio, Xiao Li, Reynold Xin · Nov 5, 2026

**Type:** Announcement

## Summary

Apache Spark 4.3 introduces stateful processing in Real-Time Mode, achieving sub-150 ms P99 end-to-end latency for streaming aggregation, deduplication, and custom stateful processing using transformWithState. This eliminates the need for a separate streaming engine for millisecond-scale latency in fraud detection, user activity monitoring, and device monitoring, enabled by native concurrent stage scheduling and pipelined shuffles. The release also includes Project Feather for faster iteration on small data by reducing execution overhead, and SCD Type 2 Auto CDC for tracking record changes over time. SQL enhancements include JSON\_TABLE, VARIANT transformations, ASOF JOIN for temporal matching, and a nanosecond timestamp preview. Spark 4.3 marks a shift to quarterly feature releases, with over 200 contributors addressing more than 1,100 Jira tickets, and is now available in Databricks Runtime 19.

## Context

Apache Spark 4.3 extends Real-Time Mode to stateful processing, enabling millisecond-scale latency for streaming aggregations, deduplication, and custom stateful operations without a separate streaming engine.

## Approach / What changed

Native concurrent stage scheduling and pipelined shuffles (PipelinedShuffleDependency) allow producer and consumer stages to run simultaneously, with StreamingShuffleManager transferring records incrementally. Project Feather introduces opt-in shuffle-free single-task execution for small, single-partition query fragments.

## Takeaways

- Spark 4.3’s Real-Time Mode now supports stateful operations like streaming aggregation, dropDuplicates, and transformWithState, achieving sub-150 ms P99 latency in benchmarks.
- The release introduces PipelinedShuffleDependency and StreamingShuffleManager, enabling concurrent stage execution and incremental data transfer for continuous streaming.
- Project Feather reduces execution overhead for small data by enabling opt-in shuffle-free single-task execution, making Spark more responsive for iterative development.

## FAQ

### What latency does Spark 4.3 achieve for stateful streaming?

Spark 4.3 achieves sub-150 ms P99 end-to-end latency for stateful workloads including dropDuplicates (136 ms), window aggregation (139 ms), and transformWithState (141 ms) in benchmarks.

### How does Spark 4.3 enable low-latency stateful processing?

It introduces native concurrent stage scheduling and pipelined shuffles via PipelinedShuffleDependency, allowing producer and consumer stages to run concurrently and transfer records incrementally.

### What is Project Feather?

Project Feather targets unnecessary execution overhead for small data by introducing opt-in shuffle-free single-task execution for eligible small, single-partition query fragments, making Spark more responsive for iterative development.

**Tags:** [Apache Spark](https://yomu.fyi/topic/apache-spark), [Performance](https://yomu.fyi/topic/performance), [SQL](https://yomu.fyi/topic/sql), [Structured Streaming](https://yomu.fyi/topic/structured-streaming)

- Source: [Databricks](https://www.databricks.com/blog/introducing-apache-spark-43-now-available-databricks-runtime-19)
- Source URL: https://www.databricks.com/blog/introducing-apache-spark-43-now-available-databricks-runtime-19
- Ingested by Yomu: 2026-11-05T00:00:15.676Z

[Read original post](https://www.databricks.com/blog/introducing-apache-spark-43-now-available-databricks-runtime-19)
