# Safer deployment of streaming applications

[Grab](https://yomu.fyi/company/grab) · Shi Kai Ng · May 2, 2023

**Type:** Problem & solution

## Summary

Stateful stream processing frameworks like Apache Flink present unique deployment challenges because conventional canary and blue-green strategies can cause data inaccuracies or state divergence. Grab's real-time data platform team encountered risks of state loss, manual rollback overhead, and absent health checks in their Kubernetes and Spinnaker deployment pipeline. To resolve these operational issues, the team redesigned the deployment workflow around automated Flink savepointing and programmatic health monitoring. The new pipeline halts existing applications after capturing state snapshots and Kafka offsets, monitors target deployments via API health probes, and executes automated rollbacks using versioned ConfigMaps and replica metadata annotations. This automated process ensures state consistency during upgrades and eliminates manual intervention during deployment failures.

## Context

Deploying stateful Flink streaming applications on Kubernetes at Grab was error-prone because existing CI/CD processes lacked automated health probes, risked losing application state, and required slow, manual rollbacks across interconnected streaming pipelines.

## Approach / What changed

Grab built an automated Spinnaker deployment workflow that captures Flink savepoints via API before shutdown, deploys new versions with stored replica count annotations, conducts automated API health probes after a stabilisation period, and rolls back configuration and infrastructure upon failure.

## Takeaways

- Canary deployments are unsuitable for stateful Flink stream joins because splitting traffic across versions leads to incomplete data inputs and inaccurate processing.
- Capturing a Flink savepoint before stopping an application preserves Kafka consumer offsets, ensuring consistent data replay and zero message loss if a rollback occurs.
- Dynamic TaskManager replica counts from autoscaling are not tracked in standard Kubernetes deployment specs, necessitating custom metadata annotations for accurate cluster rollbacks.

**Tags:** [CI/CD](https://yomu.fyi/topic/ci-cd), [Deployment](https://yomu.fyi/topic/deployment), [Kafka](https://yomu.fyi/topic/kafka), [Kubernetes](https://yomu.fyi/topic/kubernetes), [Streaming](https://yomu.fyi/topic/streaming)

[Read original post](https://engineering.grab.com/safer-flink-deployments)