# How We Scaled Our Cache and Got a Good Night's Sleep

[Grab](https://yomu.fyi/company/grab) · Gao Chao · Jun 19, 2017

**Type:** Problem & solution

## Summary

Growing business load on the Common Data Service (CDS) created potential bottlenecks for its single-threaded Redis cache on ElastiCache, necessitating horizontal scaling for greater capacity and throughput. After ruling out master-slave replication and intermediate Twemproxy setups due to memory constraints and proxy I/O bottlenecks, the team implemented client-side sharding. Using an internal Go package for consistent hashing, CDS instances hash cache keys locally to determine the target shard. The implementation encapsulates hashing inside a thin \`ShardedCache\` wrapper sharing the original cache interface while supporting Ketama and custom hash functions. Deploying via double-writing cron jobs during off-peak hours reduced database read pressure and improved P99 latency.

## Context

As business growth increased cache traffic, the Common Data Service's single-threaded Redis setup on ElastiCache risked becoming a bottleneck and lacked sufficient memory space for future caching needs.

## Approach / What changed

Implemented client-side custom sharding using an internal Go consistent hashing package, wrapping the original cache interface in a \`ShardedCache\` layer with Ketama hashing, and deployed using double-writing cron jobs during off-peak hours.

## Takeaways

- Master-slave Redis replication failed to solve memory space limitations, while Twemproxy was rejected because it introduces an I/O bottleneck under heavy load.
- The \`ShardedCache\` was implemented in Go as a thin wrapper embedding the standard \`Cache\` interface to enable seamless implementation swapping and custom hashing injection.
- A two-day migration strategy utilized off-peak deployment alongside double-writing cron jobs to warm up the new shards and prevent database query spikes.

**Tags:** [Caching](https://yomu.fyi/topic/caching), [Go](https://yomu.fyi/topic/go), [Performance](https://yomu.fyi/topic/performance), [Redis](https://yomu.fyi/topic/redis), [Scalability](https://yomu.fyi/topic/scalability)

[Read original post](https://engineering.grab.com/how-we-scaled-our-cache-and-got-a-good-nights-sleep)