Topic

M3DB

2 posts about M3DB. Every summary links to the original.

Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.

Newest first

Posts

Uber
From Chaos to Control: Addressing Shard Distribution Challenges in M3DB with Subclusters
M3DB’s sharded placement algorithm lets replicas spread across nodes subject primarily to isolation-group constraints, but large clusters experienced broad failure and maintenance impacts. The subclustered placement algorithm partitions nodes into fixed-size, self-contained groups, each owning a distinct shard slice while preserving replica separation across isolation groups. It requires equal instance weights, scaling in multiples of the subcluster size, a replica factor that cannot change through AddReplica, and instancesPerSubcluster divisible by RF. During scaling, a greedy donation strategy simulates removing each candidate shard, ranks the resulting per-node skew, and chooses the shard that leaves the donor most balanced, avoiding a second rebalancing move. The resulting model contains recovery within subclusters, enables parallel maintenance, and reduces arbitrary shard sharing from O(cluster) to O(subcluster), while the original algorithm remains suitable for smaller clusters or impractical scaling steps.

Uber
Upgrading M3DB from v1.1 to v1.5
Uber upgraded M3DB from v1.1 to v1.5 after three years on its previous open-source deployment, seeking a repeatable rollout process for a roughly 100-cluster, 5,500-node environment. The effort addressed scale-specific validation: offline benchmarks measured correctness, latency, success rates, bootstrap and tick durations, and persistence across restarts and replacements, while online shadow testing compared sampled production traffic between old and new versions. Within each cluster, upgrades were limited to one node at a time because writes used a two-thirds majority quorum. A rollout-related read-quorum incident exposed missing replica data; the cause was bootstrap misconfiguration, corrected by using “exclude_commitlog,” since production workloads did not employ WAL. Production then showed significantly lower memory usage and improved node bootstrap times, supporting continued, data-driven upgrades and contributions to M3DB's benchmarking and documentation.

The best engineering reads, once a week.

Topic

2 posts about M3DB. Every summary links to the original.

Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.

Newest first

Uber
From Chaos to Control: Addressing Shard Distribution Challenges in M3DB with Subclusters
M3DB’s sharded placement algorithm lets replicas spread across nodes subject primarily to isolation-group constraints, but large clusters experienced broad failure and maintenance impacts. The subclustered placement algorithm partitions nodes into fixed-size, self-contained groups, each owning a distinct shard slice while preserving replica separation across isolation groups. It requires equal instance weights, scaling in multiples of the subcluster size, a replica factor that cannot change through AddReplica, and instancesPerSubcluster divisible by RF. During scaling, a greedy donation strategy simulates removing each candidate shard, ranks the resulting per-node skew, and chooses the shard that leaves the donor most balanced, avoiding a second rebalancing move. The resulting model contains recovery within subclusters, enables parallel maintenance, and reduces arbitrary shard sharing from O(cluster) to O(subcluster), while the original algorithm remains suitable for smaller clusters or impractical scaling steps.

Uber
Upgrading M3DB from v1.1 to v1.5
Uber upgraded M3DB from v1.1 to v1.5 after three years on its previous open-source deployment, seeking a repeatable rollout process for a roughly 100-cluster, 5,500-node environment. The effort addressed scale-specific validation: offline benchmarks measured correctness, latency, success rates, bootstrap and tick durations, and persistence across restarts and replacements, while online shadow testing compared sampled production traffic between old and new versions. Within each cluster, upgrades were limited to one node at a time because writes used a two-thirds majority quorum. A rollout-related read-quorum incident exposed missing replica data; the cause was bootstrap misconfiguration, corrected by using “exclude_commitlog,” since production workloads did not employ WAL. Production then showed significantly lower memory usage and improved node bootstrap times, supporting continued, data-driven upgrades and contributions to M3DB's benchmarking and documentation.

The best engineering reads, once a week.