# How we seamlessly migrated high volume real-time streaming traffic from one service to another with zero data loss and duplication

[Grab](https://yomu.fyi/company/grab) · Md Riyadh · Dec 5, 2024

## Summary

Grab split a backend service's read and write functionalities into separate services to allow independent scaling. Migrating the write path required transferring processing from 16 source Kafka streams—averaging 20,000 reads per second into DynamoDB tables and output streams—with zero data loss or duplication. Standard feature flags were ruled out because rollout propagation delays could introduce minutes of duplicate or missing data during flag toggling. Instead, engineers extracted processing logic into a shared monorepo commons package that used coordinated timestamps to trigger simultaneous cutovers across both services. Temporary validation sinks verified processing accuracy in production prior to the cutover, completing the stream-by-stream migration across three weeks without downtime.

## Takeaways

- Feature flags were rejected for the switchover because multi-minute rollout propagation windows risked data loss and duplicate writes across services.
- Engineers validated the new service in production prior to switchover by routing its processed stream data into dedicated temporary DynamoDB tables and sink streams.
- Coordinating cutovers using shared timestamp maps presents trade-offs, as unexpected rollbacks in either service can lead to unavoidable data inconsistencies.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [Kafka](https://yomu.fyi/topic/kafka), [Migrations](https://yomu.fyi/topic/migration), [Streaming](https://yomu.fyi/topic/streaming)

[Read original post](https://engineering.grab.com/seamless-migration)