How Stripe uses graph search and state machines to auto-remediate a global database fleet

Summary

Stripe describes how its Document Database fleet previously handled degraded MongoDB shards with single-step, hard-coded plugins that struggled with combined failures, intermediate states, and more than 40 shard layouts. The replacement models shard topology and health as a directed graph, with infrastructure operations as edges and safety invariants defining valid states. A simulator uses breadth-first search to find minimum-operation recovery paths, while an abstraction over Temporal contexts lets the same remediation logic run in memory without real infrastructure calls; later, the system also uses Dijkstra's algorithm. The system reduced pager volume by 30%—about 200 pages annually—reclaimed 12 days of fleet health each year, and handled a new layout with zero code changes, according to the post.

Takeaways

1. 1

   Breadth-first search prunes states that violate invariants such as unsafe vote counts, insufficient healthy voting nodes, or invalid priority and vote combinations.

2. 2

   A CommonContext abstraction allows remediation workflows to use either Temporal's production context or an in-memory simulation with simulated Source of Truth and Ground Reality providers.

3. 3

   Reported results include 30% fewer pages, approximately 200 fewer pages per year, 12 fewer days of degraded shard health annually, and zero code changes for a newly onboarded layout.