Making User-Sequence Data More Cost-Efficient, Faster, and Easier to Use

Summary

User sequences serve as core primitives powering machine learning models across ranking, retrieval, and recommendation surfaces at Pinterest. Managing these sequences previously caused high infrastructure costs, operational fragility, and data drift between offline training pipelines and online serving systems. In response, engineering teams redesigned the platform around a single definition executed across multiple runtimes, including real-time streaming, batch backfills, and online inference. The updated architecture pairs a unified enrichment engine with columnar time-partitioned storage and standardized validation tiers. Consequently, the redesign achieved lower storage and network costs, faster signal onboarding via configuration, and improved engagement metrics across major recommendation surfaces.

Context

At Pinterest, user sequences power offline training, offline analysis, and online inference across Home feed, Related Pins, and Search Results. Managing user sequences across multiple teams and models had become expensive, fragile, and prone to split-brain data drift between batch training datasets and real-time serving systems.

Approach / What changed

Pinterest redesigned its user-sequence infrastructure around a 'one definition, many runtimes' architecture using a single configuration surface and shared execution engine across streaming, batch, and online serving. The system incorporates columnar time-partitioned storage, event-level and sequence-level shadow validation, and configuration-driven signal onboarding.

Takeaways

• Defining signals and events once across batch, real-time, and serving runtimes eliminates data drift between offline training and online inference.
• Validating legacy-to-new pipeline migrations requires a two-tiered comparison strategy spanning event-level field matching and end-to-end sequence output comparisons alongside A/B testing.
• Adopting columnar, time-partitioned storage and reducing network transfer per request lowered infrastructure costs while boosting engagement metrics on major surfaces.

Related reading

• Grab · Dec 5, 2024

  How we seamlessly migrated high volume real-time streaming traffic from one service to another with zero data loss and duplication

  Grab split a backend service's read and write functionalities into separate services to allow independent scaling. Migrating the write path required transferring processing from 16 source Kafka streams—averaging 20,000 reads per second into DynamoDB tables and output streams—with zero data loss or duplication. Standard feature flags were ruled out because rollout propagation delays could introduce minutes of duplicate or missing data during flag toggling. Instead, engineers extracted processing logic into a shared monorepo commons package that used coordinated timestamps to trigger simultaneous cutovers across both services. Temporary validation sinks verified processing accuracy in production prior to the cutover, completing the stream-by-stream migration across three weeks without downtime.

  Md Riyadh

• Meta · Aug 5, 2026