Predicting Rider Conversion in Sparse Data Environments with Bayesian Trees

Summary

Lyft requires real-time predictions of whether a rider will request a ride after viewing price and ETA details to balance supply and demand and personalize user experiences. Traditional machine learning models overfit on high-cardinality, sparse context intersections, whereas deep neural networks introduce unacceptable inference latency during live user sessions. To address this, engineers created a hierarchical Bayesian tree framework that organizes session data into increasingly granular context partitions. Each node houses a parametric model trained top-down, applying Gaussian Bayesian priors and L2 penalties relative to parent parameters to smooth predictions in data-rare child segments. Using simple parametric models at each node also permits monotonic constraints, guaranteeing logically consistent predictions aligned with domain expectations.

Context

Lyft needs to predict rider session conversion in real-time to manage marketplace supply-demand balance and user experiences. Slicing data across high-cardinality categorical features causes severe data sparsity where standard models like XGBoost overfit, while complex deep neural networks or LLMs add prohibitive inference latency.

Approach / What changed

The team built a hierarchical Bayesian tree that splits data across partition keys such as spatial, temporal, and congestion contexts. Nodes run parametric models trained top-down with an L2 penalty on parameter divergence from parent nodes, functioning as a Gaussian prior that balances parent averages with child-specific data while supporting monotonic constraints.

Takeaways

• Standard gradient boosted trees overfit on thinly sliced categorical context intersections with few historical samples, while deep neural networks increase inference latency beyond real-time requirements.
• Bayesian trees structure session data hierarchically across partition keys such as city region, temporal context, and supply-demand congestion levels.
• Node models use top-down training with Gaussian Bayesian priors and L2 regularization against parent parameters, smoothly defaulting sparse child leaves to parent averages while allowing monotonic constraints.

Related reading

• Pinterest · Apr 27, 2026

  From Clicks to Conversions: Architecting Shopping Conversion Candidate Generation at Pinterest

  Pinterest built a dedicated shopping candidate generation model to optimize for lower-funnel purchase conversions rather than relying strictly on engagement retrieval models. Because offsite conversion events are sparse and delayed, engineers trained a single multi-surface two-tower model using conversion data supplemented by duration-weighted click engagement and unengaged ad impressions as hard negatives. The two-tower retrieval architecture incorporates a parallel DCN v2 and MLP layer structure that allows simultaneous learning from the same inputs without information bottlenecks. Later revisions transitioned the system from a multi-head loss structure to a unified single-head multi-task architecture paired with an advertiser-level objective. Deployed across more than 600 million monthly active users, the system increased shopping conversion volume by 2.3% and boosted return on ad spend by 3.1% for United States shopping campaigns.

  Pinterest Engineering

• Grab · Sep 26, 2025