How Databricks Feature Store serves features with sub-second freshness

Summary

Databricks Feature Store addresses the lag between batch-computed machine-learning features and decisions that depend on events from seconds earlier. It lets a feature definition drive both offline batch processing and online streaming pipelines, orchestrating Kafka ingestion, Spark Real-Time Mode (RTM), Lakebase, and Model Serving. For a rolling 10-minute transaction sum, RTM updates per entity state in local RocksDB, expires contributions per event, and writes revised values to Lakebase through a streaming JDBC sink. The described path reports end-to-end p99 200ms latency from Kafka arrival to online feature availability, while Model Serving retrieves declared dependencies automatically at inference. Tumbling and sliding windows remain options when fewer, cheaper updates are acceptable.

Context

Scheduled Spark batch jobs provide historic baseline features but introduce minutes to hours of lag. The source says existing feature store platforms cannot provide seconds- or milliseconds-level freshness, forcing data scientists to build complex streaming-specific aggregation logic and custom hosted infrastructure for fresh signals.

Approach / What changed

A single infrastructure-agnostic feature definition is used for offline and online computation. Kafka events flow through serverless Spark Real-Time Mode pipelines, which perform concurrent stateful processing with local RocksDB, write updated aggregates to Lakebase through a streaming JDBC sink, and expose them through Model Serving, which automatically retrieves recorded feature dependencies at inference time.

Takeaways

• Spark Real-Time Mode processes streaming stages concurrently and updates rolling aggregates as rows arrive, rather than waiting for sequential microbatches to complete.
• Rolling windows follow each event timestamp with millisecond resolution; unlike tumbling and sliding windows, they keep an aggregate current as new events arrive and expire.
• The Feature Store retains an offline copy of ingested Kafka data, calculates historical feature values, and supports point-in-time accurate joins for training and online-feature backfills.

Related reading