# Automated Schema Evolution in Pinterest’s Next-Generation DB Ingestion Framework

[Pinterest](https://yomu.fyi/company/pinterest) · Pinterest Engineering · Jun 24, 2026

**Type:** Problem & solution

## Summary

Pinterest's CDC ingestion platform relies on a multi-stage pipeline built with Kafka, Flink, Spark, and Iceberg, where upstream schema updates risk disrupting data transformations and offline storage. To prevent pipeline failures and schema drift, Pinterest introduced an automated schema evolution framework based on a staged convergence model. Upstream changes are detected via push-based DDL messages and daily pull-based validation checks. The workflow automatically restricts automated updates to additive changes and numeric precision widening, updating Iceberg table definitions and regenerating Flink and Spark code through auditable pull requests. By decoupling schema divergence, code deployment, and data convergence, the system maintains continuous ingestion availability while working toward eventual schema consistency across all storage layers.

## Context

Pinterest's CDC ingestion platform spans multiple stages across Kafka, Flink, Spark, and Iceberg. Upstream database schema changes acted as cross-system contracts across ingestion, transformation, storage, and backfills. Without unified automation, manual updates risked schema drift, broken Flink jobs, blocked Spark upserts, and inconsistencies between online and offline data representations.

## Approach / What changed

Pinterest built an automated schema evolution framework using a staged convergence model. Changes are detected via push-based DDL CDC messages and daily pull-based schema comparisons. The framework restricts automated updates to additive changes and numeric precision widening, updating Iceberg metadata and regenerating Flink and Spark code via an auditable pull-request workflow.

## Takeaways

- Automated schema evolution is restricted to additive changes and numeric precision widening to preserve backward compatibility and avoid complex historical backfills.
- The framework employs a multi-stage convergence model where Iceberg schemas update first, followed by sequential rollouts of generated Spark and Flink code to maintain pipeline availability.
- Upstream schema changes are detected through low-latency push-based DDL CDC messages paired with daily pull-based comparison jobs acting as a safety net.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [Data Pipelines](https://yomu.fyi/topic/data-pipelines), [Kafka](https://yomu.fyi/topic/kafka), [Reliability](https://yomu.fyi/topic/reliability), [Streaming](https://yomu.fyi/topic/streaming)

[Read original post](https://medium.com/pinterest-engineering/automated-schema-evolution-in-pinterests-next-generation-db-ingestion-framework-36c5c07070de)