DatabricksTian Tan, Sixuan He, Nick Joung, Julia PowellTutorial
Build a real-time next-best-offer recommender using Databricks Feature Views
Summary
Real-time personalization in retail banking requires aggregating clickstream data into time-windowed features for both training and serving, but separate pipelines create duplicate definitions that can drift. Databricks Feature Views solve this by letting you define features once for point-in-time-correct training and low-latency online serving. The open-source Solution Accelerator builds a next-best-offer recommender that ranks offers like credit card upgrades and personal loans using a LightGBM ranker. Streaming features from Kafka are continuously aggregated into the online store using RollingWindow and SawtoothWindow declarations, eliminating separate hand-written streaming jobs. In benchmarks, the ranking endpoint achieved 34 ms p95 latency and sustained approximately 56 requests per second on the smallest configuration.
Context
Real-time personalization in retail banking requires aggregating raw clickstream data into time-windowed features for both training and serving. Maintaining separate batch and streaming pipelines creates duplicate feature definitions that can drift and degrade model accuracy. Databricks Feature Views solve this by letting you define features once for both purposes.
Approach / What changed
The accelerator uses Databricks Feature Views to define batch and streaming features once, then reuses them for point-in-time-correct training and low-latency online serving. A LightGBM ranker is trained on historical features and served on a route-optimized Model Serving endpoint. Streaming features from Kafka are continuously aggregated into the online store using RollingWindow and SawtoothWindow declarations, eliminating separate hand-written streaming jobs.
Takeaways
Feature Views let you define batch and streaming features once, then reuse them for both point-in-time-correct training and low-latency online serving, preventing train/serve skew.
The ranking endpoint achieved 34 ms p95 latency including online feature retrieval and inference, and sustained approximately 56 requests per second on the smallest configuration.
Streaming features from Kafka become model-visible in approximately a tenth of a second using RollingWindow and SawtoothWindow declarations, without a separate hand-written streaming job.
FAQ
What are Feature Views?
Feature Views let you define feature transformations once and reuse them across model training and online serving. They support batch and streaming features, with point-in-time-correct historical values for training and low-latency access to current values for inference.
How do Feature Views prevent train/serve skew?
Feature Views use shared feature definitions across training and serving, avoiding inconsistencies caused by maintaining separate transformation logic. The create_training_set API also uses point-in-time joins to reconstruct feature values as they existed at each historical label, preventing future data leakage.
What performance does the accelerator achieve?
In the benchmark configuration, the ranking endpoint achieved 34 ms p95 latency, including online feature retrieval and inference. The ranking endpoint sustained approximately 56 requests per second, while the Feature Serving endpoint sustained approximately 390 requests per second before exceeding the 50 ms p95 latency budget.
Source and freshness
- Source
- databricks.com
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.