Migrating Counter Service storage: Design choices and learnings

Summary

Grab’s Counter Service supports time-windowed fraud queries at tens of thousands of QPS and about a billion requests per day, but an infrastructure review prompted migration from a wide-column database to Aerospike. The team first separated Rust business logic from storage, then added an enum-dispatched facade with Single, WithShadow, and WithSplit modes so traffic could be compared and shifted without downtime. They also redesigned the writer path around Aerospike, including a map-based schema and a single atomic increment, while testing index placement and reverting from an NVMe-backed primary index after hot-key p99 spikes. Production p99 read latency improved by roughly 50%, disk usage fell from about 3 TB to 1 TB, and per-node cost dropped 45–50%; the rollout reported zero downtime and no data-integrity issues.

Takeaways

1. 1

   The reader facade avoided trait-object boxed-future allocations by using enum dispatch, while keeping backend-specific driver access inside storage modules.

2. 2

   Aerospike’s map-based schema and single atomic increment replaced the original three-table read-modify-write path, contributing to lower disk usage and simpler writes.

3. 3

   The migration reported roughly 50% better production p99 read latency, 45–50% lower per-node cost, zero downtime, and no data-integrity issues.