Zero downtime Upgrade: Yelp’s Cassandra 4.x Upgrade Story

Summary

Yelp's Database Reliability Engineering team upgraded over a thousand Cassandra nodes from version 3.11 to 4.1 without downtime. The upgrade delivered reliability, observability, and performance improvements, including Java 11 support, configurable guardrails, and fixed incremental repairs. To avoid the high costs, consistency compromises, and data streaming delays of standing up separate datacenters, the team selected an in-place rolling upgrade strategy. The rollout orchestrated staged node updates alongside compatible versions of the Stargate proxy and custom data pipeline connectors without requiring client code modifications. Although Stargate 2.x introduced regressions on range and multi-partition queries that required downgrading to 1.x, the finished migration achieved up to a 58% reduction in p99 latency across key clusters.

Context

Yelp operated over a thousand Cassandra nodes on version 3.11 orchestrated via Kubernetes operators. Upgrading to 4.1 was needed because version 3.11 reached end-of-life, and 4.1 unblocked future Cassandra 5 upgrades while providing Java 11 support, guardrails, and performance improvements.

Approach / What changed

Yelp executed an in-place rolling upgrade orchestrated by automated scripts with checkpointing. The rollout staged node and Stargate proxy updates, managed version-specific images via dedicated branches and environment variables, paused anti-entropy repairs during migration, and disabled schema changes.

Takeaways

• Running an in-place rolling upgrade avoided the large hardware costs and weeks of data streaming required by building a parallel datacenter.
• Performance regressions on range and multi-partition queries caused by Stargate 2.x were resolved by downgrading the proxy service back to version 1.x.
• Post-upgrade schema disagreements on CDC-enabled clusters were remediated by executing dummy schema changes across multiple nodes to force convergence.

Related reading

• Yelp · May 27, 2026

  Beyond the Menu Tree: How Yelp Built a Smarter Customer Success Chatbot with AI

  Yelp transitioned from a legacy customer support chatbot that relied on rigid phrase matching and static menus to a retrieval-augmented generation system. The updated architecture classifies incoming queries using a large language model to route users into one of five specialized workflows, including QA, Billing, Refund, Cancel, and Review. For general QA, the system uses FAISS to search an in-memory vectorstore built exclusively from article metadata such as titles, summaries, and headers rather than raw article text. Retrieved articles are supplied to the language model as prompt context, followed by automated validations for safety, character limits, and hallucinated hyperlinks. In production A/B testing, this metadata-driven design doubled the chatbot resolution rate while achieving approximately 94% recall@5.

  Lina Lee, Machine Learning Engineer; Nelson Lee, Engineering Manager

• Yelp · Mar 27, 2026