Predicting Rider Conversion in Sparse Data Environments with Bayesian Trees

Summary

Lyft requires real-time predictions of whether a rider will request a ride after viewing price and ETA details to balance supply and demand and personalize user experiences. Traditional machine learning models overfit on high-cardinality, sparse context intersections, whereas deep neural networks introduce unacceptable inference latency during live user sessions. To address this, engineers created a hierarchical Bayesian tree framework that organizes session data into increasingly granular context partitions. Each node houses a parametric model trained top-down, applying Gaussian Bayesian priors and L2 penalties relative to parent parameters to smooth predictions in data-rare child segments. Using simple parametric models at each node also permits monotonic constraints, guaranteeing logically consistent predictions aligned with domain expectations.

Takeaways

1. 1

   Standard gradient boosted trees overfit on thinly sliced categorical context intersections with few historical samples, while deep neural networks increase inference latency beyond real-time requirements.

2. 2

   Bayesian trees structure session data hierarchically across partition keys such as city region, temporal context, and supply-demand congestion levels.

3. 3

   Node models use top-down training with Gaussian Bayesian priors and L2 regularization against parent parameters, smoothly defaulting sparse child leaves to parent averages while allowing monotonic constraints.