Evaluating performance impact of removing Redis-cache from a Scylla-backed service

Summary

A Rust service at Grab serves tens of thousands of peak QPS for batched count requests, using Scylla, Redis, and MySQL counter configurations. Its 15-minute cache-key truncation and five-minute TTL caused cache-hit declines and recurring Scylla QPS spikes at the end of each quarter-hour, when recent-data requests generated new keys. The experiment progressively disabled Redis, while a daily major compaction job was added to Scylla nodes; the service continued combining minutely, hourly, and daily tables for range calculations. With Redis fully decommissioned, Scylla cache hits rose from about 30% to 70% of reads, cache misses stayed about the same despite doubled traffic, peak CPU moved from roughly 50% to 52%, and overall service performance was maintained while eliminating a component responsible for about 25% of service cost.

Takeaways

1. 1

   The 15-minute cache-key truncation created new keys for recent-data requests at 00, 15, 30, and 45 minutes past each hour, producing recurring cache-hit drops and Scylla QPS spikes.

2. 2

   After Redis was disabled, Scylla’s own cache served about 70% of reads versus about 30% before the experiment; Scylla cache misses remained about the same despite doubled Scylla traffic.

3. 3

   Peak Scylla CPU usage increased only from about 50% to 52%, while the removed Redis component had represented about 25% of the overall service cost.