Making User-Sequence Data More Cost-Efficient, Faster, and Easier to Use

Summary

User sequences serve as core primitives powering machine learning models across ranking, retrieval, and recommendation surfaces at Pinterest. Managing these sequences previously caused high infrastructure costs, operational fragility, and data drift between offline training pipelines and online serving systems. In response, engineering teams redesigned the platform around a single definition executed across multiple runtimes, including real-time streaming, batch backfills, and online inference. The updated architecture pairs a unified enrichment engine with columnar time-partitioned storage and standardized validation tiers. Consequently, the redesign achieved lower storage and network costs, faster signal onboarding via configuration, and improved engagement metrics across major recommendation surfaces.

Takeaways

1. 1

   Defining signals and events once across batch, real-time, and serving runtimes eliminates data drift between offline training and online inference.

2. 2

   Validating legacy-to-new pipeline migrations requires a two-tiered comparison strategy spanning event-level field matching and end-to-end sequence output comparisons alongside A/B testing.

3. 3

   Adopting columnar, time-partitioned storage and reducing network transfer per request lowered infrastructure costs while boosting engagement metrics on major surfaces.