How we seamlessly migrated high volume real-time streaming traffic from one service to another with zero data loss and duplication

Summary

Grab migrated its stream-processing (write) functionality from an existing service into a separate service while handling 20,000 reads per second from 16 Kafka streams and writing to output streams and DynamoDB tables. The migration required gradual stream-by-stream rollout, production validation, and a no-downtime switchover; feature flags were rejected because deployment timing could cause loss, duplication, or inconsistent values. The team moved processing into shared /commons code, created temporary validation sinks, and scheduled each switchover with a shared map[string]time.Time, allowing both services to change writers at the same time. During each cutover, both services consumed streams, the old service stopped writing to resources, and monitoring continued before old consumption was removed; the result was zero data loss or duplication, no downtime, and completion in about three weeks. It was temporary and tested, but rolling back either service could create inconsistency if systems fell out of sync.

Takeaways

1. 1

   Both services consumed incoming streams during validation, but only the old service wrote to the actual sink resources while the new service used temporary validation sinks.

2. 2

   A shared per-stream map[string]time.Time provided identical switchover timing in both services, avoiding overlap or gaps when changing the active writer.

3. 3

   The migration completed in about three weeks with zero reported data loss or duplication and no downtime, but rollback could cause inconsistency if the services fell out of sync.