Automated Schema Evolution in Pinterest’s Next-Generation DB Ingestion Framework

Summary

Pinterest's CDC ingestion platform relies on a multi-stage pipeline built with Kafka, Flink, Spark, and Iceberg, where upstream schema updates risk disrupting data transformations and offline storage. To prevent pipeline failures and schema drift, Pinterest introduced an automated schema evolution framework based on a staged convergence model. Upstream changes are detected via push-based DDL messages and daily pull-based validation checks. The workflow automatically restricts automated updates to additive changes and numeric precision widening, updating Iceberg table definitions and regenerating Flink and Spark code through auditable pull requests. By decoupling schema divergence, code deployment, and data convergence, the system maintains continuous ingestion availability while working toward eventual schema consistency across all storage layers.

Context

Pinterest's CDC ingestion platform spans multiple stages across Kafka, Flink, Spark, and Iceberg. Upstream database schema changes acted as cross-system contracts across ingestion, transformation, storage, and backfills. Without unified automation, manual updates risked schema drift, broken Flink jobs, blocked Spark upserts, and inconsistencies between online and offline data representations.

Approach / What changed

Pinterest built an automated schema evolution framework using a staged convergence model. Changes are detected via push-based DDL CDC messages and daily pull-based schema comparisons. The framework restricts automated updates to additive changes and numeric precision widening, updating Iceberg metadata and regenerating Flink and Spark code via an auditable pull-request workflow.

Takeaways

• Automated schema evolution is restricted to additive changes and numeric precision widening to preserve backward compatibility and avoid complex historical backfills.
• The framework employs a multi-stage convergence model where Iceberg schemas update first, followed by sequential rollouts of generated Spark and Flink code to maintain pipeline availability.
• Upstream schema changes are detected through low-latency push-based DDL CDC messages paired with daily pull-based comparison jobs acting as a safety net.

Related reading

• Pinterest · Apr 27, 2026

  From Clicks to Conversions: Architecting Shopping Conversion Candidate Generation at Pinterest

  Pinterest built a dedicated shopping candidate generation model to optimize for lower-funnel purchase conversions rather than relying strictly on engagement retrieval models. Because offsite conversion events are sparse and delayed, engineers trained a single multi-surface two-tower model using conversion data supplemented by duration-weighted click engagement and unengaged ad impressions as hard negatives. The two-tower retrieval architecture incorporates a parallel DCN v2 and MLP layer structure that allows simultaneous learning from the same inputs without information bottlenecks. Later revisions transitioned the system from a multi-head loss structure to a unified single-head multi-task architecture paired with an advertiser-level objective. Deployed across more than 600 million monthly active users, the system increased shopping conversion volume by 2.3% and boosted return on ad spend by 3.1% for United States shopping campaigns.

  Pinterest Engineering

• Pinterest