Automated Schema Evolution in Pinterest’s Next-Generation DB Ingestion Framework

Summary

Pinterest's CDC ingestion platform relies on a multi-stage pipeline built with Kafka, Flink, Spark, and Iceberg, where upstream schema updates risk disrupting data transformations and offline storage. To prevent pipeline failures and schema drift, Pinterest introduced an automated schema evolution framework based on a staged convergence model. Upstream changes are detected via push-based DDL messages and daily pull-based validation checks. The workflow automatically restricts automated updates to additive changes and numeric precision widening, updating Iceberg table definitions and regenerating Flink and Spark code through auditable pull requests. By decoupling schema divergence, code deployment, and data convergence, the system maintains continuous ingestion availability while working toward eventual schema consistency across all storage layers.

Takeaways

1. 1

   Automated schema evolution is restricted to additive changes and numeric precision widening to preserve backward compatibility and avoid complex historical backfills.

2. 2

   The framework employs a multi-stage convergence model where Iceberg schemas update first, followed by sequential rollouts of generated Spark and Flink code to maintain pipeline availability.

3. 3

   Upstream schema changes are detected through low-latency push-based DDL CDC messages paired with daily pull-based comparison jobs acting as a safety net.