---
title: "Announcing On-Demand State Repartitioning for Apache Spark™ Structured Streaming on Databricks"
description: "Stateful Apache Spark Structured Streaming queries historically locked their state-store partition count into the checkpoint, so changing spark.sql.shuffle.partitions after a workload grew had no effect without rebuilding and losing accumulated state. Databricks Runtime 18 and later introduces on-demand state repartitioning, a Public Preview feature that preserves the checkpoint while moving state to a new partition layout. Operators set spark.sql.streaming.stateStore.partitions, stop the query, and restart it; the query finishes any pending microbatch, physically rehashes and redistributes state, then resumes with the requested count. The feature applies to aggregations, stream-stream joins, deduplication, sessionization, and transformWithState, provided the RocksDB state store is used. Repartitioning pauses processing and its duration is reported in controlBatch.REPARTITION; Coveo reports a 40% reduction in related Amazon S3 API costs."
---

# Announcing On-Demand State Repartitioning for Apache Spark™ Structured Streaming on Databricks

[Databricks](https://yomu.fyi/company/databricks) · Thangam Vaiyapuri, Jay Palaniappan, B. Micheal Okutubo, Zifei Feng · Sep 14, 2026

**Type:** Announcement

## Summary

Stateful Apache Spark Structured Streaming queries historically locked their state-store partition count into the checkpoint, so changing spark.sql.shuffle.partitions after a workload grew had no effect without rebuilding and losing accumulated state. Databricks Runtime 18 and later introduces on-demand state repartitioning, a Public Preview feature that preserves the checkpoint while moving state to a new partition layout. Operators set spark.sql.streaming.stateStore.partitions, stop the query, and restart it; the query finishes any pending microbatch, physically rehashes and redistributes state, then resumes with the requested count. The feature applies to aggregations, stream-stream joins, deduplication, sessionization, and transformWithState, provided the RocksDB state store is used. Repartitioning pauses processing and its duration is reported in controlBatch.REPARTITION; Coveo reports a 40% reduction in related Amazon S3 API costs.

## Context

Stateful streaming state is stored in partition-specific RocksDB instances within the checkpoint. Because changing the partition count could make keys map to different partitions than their stored state, Structured Streaming previously fixed the count at checkpoint creation.

## Approach / What changed

Use Databricks Runtime 18 or later with the RocksDB state store, set spark.sql.streaming.stateStore.partitions to the desired value, and restart the query with the same checkpoint. The restart completes any pending microbatch, performs a one-time physical repartition that re-hashes and redistributes state, and then resumes processing. The dedicated setting takes precedence over spark.sql.shuffle.partitions.

## Takeaways

- spark.sql.streaming.stateStore.partitions takes precedence over spark.sql.shuffle.partitions for stateful queries, allowing a changed partition count to persist across a restart.
- Repartitioning is a one-time operation that physically redistributes checkpointed state and requires a processing pause; its duration appears as controlBatch.REPARTITION in durationMs metrics.
- The feature supports stateful aggregations, stream-stream joins, deduplication, sessionization, and transformWithState when the RocksDB state store is used.

## FAQ

### Does changing spark.sql.shuffle.partitions still resize an existing stateful query?

No. The original partition count is stored in the checkpoint, so changing spark.sql.shuffle.partitions alone is ignored. Use spark.sql.streaming.stateStore.partitions instead; it takes precedence for stateful queries.

### What happens when a query restarts with a new state partition count?

The query first finishes the last planned microbatch if one is pending, then physically re-hashes and redistributes state across the requested number of partitions. Processing resumes after that one-time repartition, with the existing state preserved.

### How can operators measure the pause caused by repartitioning?

After the next microbatch completes, StreamingQueryProgress includes the repartition duration in durationMs under controlBatch.REPARTITION. The runtime depends on the amount of state, and the post says most workloads are expected to take only a few seconds.

**Tags:** [Databricks](https://yomu.fyi/topic/databricks), [Spark](https://yomu.fyi/topic/spark)

- Source: [Databricks](https://www.databricks.com/blog/announcing-demand-state-repartitioning-apache-sparktm-structured-streaming-databricks)
- Source URL: https://www.databricks.com/blog/announcing-demand-state-repartitioning-apache-sparktm-structured-streaming-databricks
- Ingested by Yomu: 2026-09-15T00:02:56.802Z

[Read original post](https://www.databricks.com/blog/announcing-demand-state-repartitioning-apache-sparktm-structured-streaming-databricks)