Loading…
Topic
Redis
50 posts about Redis. Every summary links to the original.
- 50 posts
- 14 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
How We Designed the Quotas Microservice to Prevent Resource Abuse
As Grab migrated from a monolith to hundreds of microservices, managing global rate limiting became essential to prevent cascading failures and resource exhaustion. To avoid putting a rate limiting service on the critical path of every API call, Grab built Quotas, an asynchronous rate limiting system. Client services use a lightweight SDK and middleware to read rate limiting decisions from local in-memory caches and stream usage metrics asynchronously via Apache Kafka. The Quotas service aggregates usage data locally, flushes stats to Redis periodically, and publishes updated rate limiting decisions back over Kafka topics. In production, Quotas successfully handles 200k peak transactions per second with decision enforcement delays capped at 200 milliseconds.
Rolling the Heroku Redis Fleet
Heroku proactively rolled its Redis fleet to a version not vulnerable to CVE-2018-11218 during an embargo, aiming to protect customers while limiting disruption and downtime. The operation relied on immutable infrastructure: engineers built, tested, reviewed, and released a new server image, then replaced High Availability standbys before scheduling customer maintenance. During each update, the team verified replication, broke the link so the standby could accept writes, redirected application config vars to the new primary, de-provisioned the old primary, and created a replacement standby. Automation controlled concurrency and limited replacement work to the on-call operator’s business hours, while customers received advance and completion emails. The fleet roll covered all Heroku Redis customers within five weeks of notification and finished one week before the public announcement; the process also applies to Heroku Postgres and Apache Kafka on Heroku.
Dealing with the Meltdown Patch at Grab
AWS infrastructure maintenance related to Meltdown patches led to severe CPU utilization spikes across Grab's ElastiCache Redis instances. Because Redis is single-threaded, spikes past 50% CPU on two-vCPU instances threatened service capacity, and initial Multi-AZ failovers only provided temporary relief until the new master nodes received rolling patches. To handle the increased overhead before their peak traffic window, the engineering team horizontally scaled both clustered and non-clustered Redis fleets. For Redis 3.2.4 clusters lacking live re-sharding support, they provisioned larger clusters, warmed caches, and redirected traffic. Non-clustered workloads were resolved by provisioning extra nodes, migrating compatible services to Redis Cluster, or updating application code to shard data across multiple instances.
Migrating Existing Datastores
Grab's Identity team faced imminent memory exhaustion on a single Redis node used to cache mobile authentication tokens under rapid user growth. Because read traffic outweighed write traffic by roughly 200 times, the team opted for an AWS ElastiCache cluster with three shards and two read replicas per shard. They executed a zero-downtime, six-phase migration plan while handling a peak load of 20,000 queries per second. The migration transitioned through initial one-time data replication, asynchronous shadow writes, synchronous dual writes, asynchronous read validation, switching primary reads, and final write cleanup. Controlled by feature flags and monitored with metrics at every stage, the migration completed without invalidating tokens or causing service disruptions.
How We Scaled Our Cache and Got a Good Night's Sleep
Growing business load on the Common Data Service (CDS) created potential bottlenecks for its single-threaded Redis cache on ElastiCache, necessitating horizontal scaling for greater capacity and throughput. After ruling out master-slave replication and intermediate Twemproxy setups due to memory constraints and proxy I/O bottlenecks, the team implemented client-side sharding. Using an internal Go package for consistent hashing, CDS instances hash cache keys locally to determine the target shard. The implementation encapsulates hashing inside a thin `ShardedCache` wrapper sharing the original cache interface while supporting Ketama and custom hash functions. Deploying via double-writing cron jobs during off-peak hours reduced database read pressure and improved P99 latency.
Hello RedBeat: A New Celery Beat Scheduler
Heroku Connect relied on Celery Beat to trigger frequent periodic tasks while synchronizing Salesforce and Heroku Postgres data, but the default file-based scheduler developed intermittent pauses and hangs that required manual intervention. Existing SQL-backed scheduling introduced blocked transactions and pauses from periodically saving and reloading the full schedule, while an existing Redis-backed package retained similar reload behavior. RedBeat moves the active schedule and runtime metadata into Redis, allowing updates through Redis keys that Beat reads on its next tick without pausing or reloading. It also uses a lock to prevent concurrent Beat daemons, supports schedule updates from other languages, and has run in production for more than a year without recurring scheduler-related incidents.
A Key Expired in Redis, You Won't Believe What Happened Next
Grab experienced an issue where its Unicorn API served stale data for up to 45 to 60 minutes despite expected cache invalidation times totaling around 11 minutes. The setup utilized ElastiCache Redis 2.x configured with a single master node for writes and two read-only slaves handling reads. Investigation revealed that in Redis 2.x, slave nodes do not expire keys on their own and only delete them upon receiving an explicit DEL command from the master. Because the master only actively checks and deletes 200 random keys per second, clearing expired keys across roughly 5.6 million cached items mathematically required over 110 hours, resulting in slaves serving expired data.
The Curious Case of the Phantom Instance
Datadog dashboards for the grab_attention cluster displayed periodic 1.5X step increases in Elastic Load Balancer (ELB) health check requests and ElastiCache Redis connections, creating the illusion of an untracked instance running outside Auto Scaling Group records. Inspecting instance hostname tags revealed that the existing two instances were simply receiving elevated ping counts from the load balancers. AWS Support clarified that ELB scaling events provision new nodes while keeping old nodes running for roughly 90 minutes to handle cached DNS clients. Investigation also revealed that two separate ELBs were attached to the cluster, altering expected request baselines. Furthermore, the application's health check endpoint initiated a non-pooled Redis connection on every request, directly translating load balancer pings into database connection spikes.
Improved production stability with circuit breakers
Heroku’s API consumes more service endpoints than it serves, making its availability dependent on public, internal, and third-party services that can produce tail latency, 500 errors, broken sockets, or bad deploys. To contain these failures, the team wrapped external calls with Redis-backed circuit breakers that count errors over a time window, trip at an error-rate threshold, and support fallback responses or user-friendly exceptions. CB2 provides this implementation, including individually identified breakers, while percentage-based thresholds avoid the scaling problem of fixed error counts. Deployment starts with logging-only breakers so parameters and unexpected internal-API behavior can be reviewed before traffic is affected, followed by monitoring state changes with safeguards against duplicate notifications caused by race conditions. The post also recommends explicit, low network-level timeouts, arguing that slow services can damage stability more than fast failures.
How to Secure the Celery Task Queue with Fernet
Celery jobs sent through a Redis broker can expose their contents because Redis does not currently support SSL and Heroku add-ons communicate over the public web. The proposed solution encrypts jobs before they travel between dynos and Redis by wrapping Kombu’s default serializers with Fernet symmetric encryption from Cryptography. The open-source kombu-fernet-serializers library supplies these encrypted serializers through Setuptools entry points; setup requires adding the package, storing a Fernet key in KOMBU_FERNET_KEY, and configuring Celery to use fernet_json with MIMETYPE in CELERY_ACCEPT_CONTENT. The example creates a Celery app with REDIS_URL as its broker and uses the serializer settings, retaining Redis’s straightforward configuration while addressing sensitive information in transit.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.