# Enhancing Flink deployment with shadow testing

[Grab](https://yomu.fyi/company/grab) · Tee Long Lang · May 7, 2026

## Summary

Production deployments of Apache Flink applications at Grab previously risked up to ten minutes of downtime when new versions failed under live traffic or could not resume from existing checkpoints. To eliminate these deployment failures, Grab's Coban data streaming team introduced an optional shadow testing stage directly into their production Kubernetes deployment pipeline. The shadow pipeline runs new application releases in parallel with the active application for a default observation period of one hour before prompting promotion to main. Dynamic runtime configuration based on an injected environment variable redirects source consumer groups and isolates output to dedicated shadow sinks, metrics, and logs.

## Takeaways

- Shadow testing deploys alongside the active application in a dedicated shadow Kubernetes namespace and executes for a default one-hour evaluation period to verify job manager stability.
- Source connectors avoid interfering with production streams by appending a suffix to Kafka consumer groups or shifting Debezium change data capture Server ID ranges.
- Sink connectors isolate test output by routing data to a dedicated shadow Kafka cluster with an eight-hour retention policy and a separate shadow S3 bucket with a seven-day lifecycle rule.

**Tags:** [Deployment](https://yomu.fyi/topic/deployment), [Kafka](https://yomu.fyi/topic/kafka), [Kubernetes](https://yomu.fyi/topic/kubernetes), [Reliability](https://yomu.fyi/topic/reliability), [Streaming](https://yomu.fyi/topic/streaming)

[Read original post](https://engineering.grab.com/enchancing-flink-shadow-testing)