---
title: "Upgrading M3DB from v1.1 to v1.5"
description: "Uber upgraded M3DB from v1.1 to v1.5 after three years on its previous open-source deployment, seeking a repeatable rollout process for a roughly 100-cluster, 5,500-node environment. The effort addressed scale-specific validation: offline benchmarks measured correctness, latency, success rates, bootstrap and tick durations, and persistence across restarts and replacements, while online shadow testing compared sampled production traffic between old and new versions. Within each cluster, upgrades were limited to one node at a time because writes used a two-thirds majority quorum. A rollout-related read-quorum incident exposed missing replica data; the cause was bootstrap misconfiguration, corrected by using “exclude_commitlog,” since production workloads did not employ WAL. Production then showed significantly lower memory usage and improved node bootstrap times, supporting continued, data-driven upgrades and contributions to M3DB's benchmarking and documentation."
---

# Upgrading M3DB from v1.1 to v1.5

[Uber](https://yomu.fyi/company/uber) · May 9, 2026

**Type:** Problem & solution

## Summary

Uber upgraded M3DB from v1.1 to v1.5 after three years on its previous open-source deployment, seeking a repeatable rollout process for a roughly 100-cluster, 5,500-node environment. The effort addressed scale-specific validation: offline benchmarks measured correctness, latency, success rates, bootstrap and tick durations, and persistence across restarts and replacements, while online shadow testing compared sampled production traffic between old and new versions. Within each cluster, upgrades were limited to one node at a time because writes used a two-thirds majority quorum. A rollout-related read-quorum incident exposed missing replica data; the cause was bootstrap misconfiguration, corrected by using “exclude\_commitlog,” since production workloads did not employ WAL. Production then showed significantly lower memory usage and improved node bootstrap times, supporting continued, data-driven upgrades and contributions to M3DB's benchmarking and documentation.

## Context

The upgrade targeted Uber’s M3DB deployment, which handled approximately 1 billion write RPS, 80 million read RPS, about 6,000 servers, and roughly 100 clusters with 5,500 nodes. The source emphasizes that small regressions could threaten the observability platform and that testing needed to account for frequent node additions, removals, and replacements.

## Approach / What changed

The team combined offline and online testing. Offline testing used a distributed benchmark service against a 10-node cluster to compare versions, exercise topology changes and rollouts, and verify data after upgrades. Online testing duplicated configurable samples of production reads and writes into shadow clusters, establishing an older-version baseline before upgrading and comparing performance and correctness indicators. Production rollouts upgraded only one node per cluster at a time.

## Takeaways

- Offline validation covered correctness, latency, success rates, bootstrap duration, tick duration, and persistence through restarts and node replacements.
- A read-quorum issue after rollout was traced to bootstrap configuration; using “exclude\_commitlog” made nodes recover lost data from peers, matching workloads that did not use WAL.
- Production results showed a significant reduction in memory usage and improved node bootstrap times, while CPU usage did not drop because the servers were memory-bound rather than CPU-bound.

## FAQ

### Why could only one node be upgraded at a time within a cluster?

Uber’s writes used a two-thirds majority quorum, so upgrading multiple nodes simultaneously could risk data loss. Clusters could be upgraded in parallel, but each individual cluster followed a one-node-at-a-time constraint.

### What caused the missing replica data after the rollout?

The upgraded nodes used a default bootstrap configuration that did not appropriately recover data from peers. The team changed the mode to “exclude\_commitlog,” which was suitable because production workloads did not employ WAL.

### How were the new and old M3DB versions compared?

Offline benchmarks ran equivalent tests on both versions, including topology operations and correctness checks. Online shadow clusters first established a baseline on the old version, then received sampled production traffic after upgrading to the new version for comparison.

**Tags:** [Distributed Systems](https://yomu.fyi/topic/distributed-systems), [M3DB](https://yomu.fyi/topic/m3db), [Testing](https://yomu.fyi/topic/testing), [Time Series](https://yomu.fyi/topic/time-series)

- Source: [Uber](https://www.uber.com/us/en/blog/upgrading-m3db)
- Source URL: https://www.uber.com/us/en/blog/upgrading-m3db
- Ingested by Yomu: 2026-09-14T12:01:26.113Z

[Read original post](https://www.uber.com/us/en/blog/upgrading-m3db)