# Predicting Rider Conversion in Sparse Data Environments with Bayesian Trees

[Lyft](https://yomu.fyi/company/lyft) · Zammit Alban · Mar 30, 2026

**Type:** Problem & solution

## Summary

Lyft requires real-time predictions of whether a rider will request a ride after viewing price and ETA details to balance supply and demand and personalize user experiences. Traditional machine learning models overfit on high-cardinality, sparse context intersections, whereas deep neural networks introduce unacceptable inference latency during live user sessions. To address this, engineers created a hierarchical Bayesian tree framework that organizes session data into increasingly granular context partitions. Each node houses a parametric model trained top-down, applying Gaussian Bayesian priors and L2 penalties relative to parent parameters to smooth predictions in data-rare child segments. Using simple parametric models at each node also permits monotonic constraints, guaranteeing logically consistent predictions aligned with domain expectations.

## Context

Lyft needs to predict rider session conversion in real-time to manage marketplace supply-demand balance and user experiences. Slicing data across high-cardinality categorical features causes severe data sparsity where standard models like XGBoost overfit, while complex deep neural networks or LLMs add prohibitive inference latency.

## Approach / What changed

The team built a hierarchical Bayesian tree that splits data across partition keys such as spatial, temporal, and congestion contexts. Nodes run parametric models trained top-down with an L2 penalty on parameter divergence from parent nodes, functioning as a Gaussian prior that balances parent averages with child-specific data while supporting monotonic constraints.

## Takeaways

- Standard gradient boosted trees overfit on thinly sliced categorical context intersections with few historical samples, while deep neural networks increase inference latency beyond real-time requirements.
- Bayesian trees structure session data hierarchically across partition keys such as city region, temporal context, and supply-demand congestion levels.
- Node models use top-down training with Gaussian Bayesian priors and L2 regularization against parent parameters, smoothly defaulting sparse child leaves to parent averages while allowing monotonic constraints.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Performance](https://yomu.fyi/topic/performance)

[Read original post](https://eng.lyft.com/predicting-rider-conversion-in-sparse-data-environments-with-bayesian-trees-07227ff92789)