Enhancing Flink deployment with shadow testing

Summary

Grab’s Coban data streaming team addresses production-only Apache Flink deployment failures, which can leave an application unavailable for about 10 minutes while the system observes it and rolls it back. Its Shadow Testing stage runs a new version in parallel with the current Main version, replaying production data without affecting Main outputs and allowing behavior and regressions to be assessed before promotion. The Shadow deployment uses a shadow Kubernetes namespace, a shadow-prefixed application name, isShadow=true, a distinct Job ID, isolated sinks, and connector-specific separation such as suffixed Kafka consumer groups and shifted Debezium Server ID ranges. By default, Shadow runs for one hour while its job-manager status is monitored; users then continue to Main deployment, which receives a 10-minute health check and automatic rollback if unhealthy. The framework isolates Kafka and S3 outputs, prefixes metrics and logs for observability, and plans more connector support.

Takeaways

1. 1

   Shadow Testing runs a new Flink version beside Main and compares behavior using replicated production data without allowing Shadow results to interfere with Main outputs.

2. 2

   Kafka Shadow sources use consumer groups suffixed with -shadow, while Debezium Change Data Capture sources shift Server ID ranges so both applications receive independent full data streams.

3. 3

   Shadow runs for one hour by default; after users continue, Main receives a 10-minute health check and is automatically rolled back if its job manager is unhealthy.