---
title: "Resilience"
description: "7 posts about Resilience, summarised, each linking to the original."
---

# Resilience
> 7 posts about Resilience, summarised, each linking to the original.

## Articles

### [How we raised Databricks cluster rate limits by 7.5×](https://yomu.fyi/post/how-we-raised-databricks-cluster-rate-limits-by-7-5x.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Heqing Ya, Tan Zhang, Ankit Batra, Abhishek Rai
- Published: Nov 18, 2026
- Type: Problem & solution

Databricks’ Compute Platform team replaced fixed request-rate limits for cluster creation with adaptive concurrency limits to better handle long-running, variable-cost operations. Fixed limits turned away requests during healthy periods yet accepted too many when dependencies slowed, requiring manual tuning. Adaptive concurrency tracks active provisioning workflows with slots and adjusts the slot count based on congestion signals such as provisioning latency and thread pool saturation. The rollout analysis reported a 7.5× increase in default cluster-creation request-rate limits, more than 99.99% fewer rate-limit errors, and removal of over a hundred customer-specific overrides. The fixed API request-rate limit remains as a final safety guardrail against exceptionally high incoming traffic.


### [How to run a DR readiness engagement](https://yomu.fyi/post/how-to-run-a-dr-readiness-engagement.md)
- Company: [AWS](https://yomu.fyi/company/aws.md)
- Author: Peter Garnepudi
- Published: Nov 2, 2026
- Type: Tutorial

The article describes a structured disaster recovery readiness engagement that compresses the three business decisions and progressive validation framework into a weeks-long process. The kickoff brings together application owner, business owner, and platform team to produce a boundary map listing every service and account in the recovery path with assigned owners and timelines. Assessment uses Well-Architected and Operational Readiness Reviews to surface dependency gaps, which are then addressed with owners and fix dates before testing proceeds. Validation progresses through tabletop exercises, planned failovers, and surprise drills, with each test’s findings fed back into the three business decisions to update the recovery commitment. An enterprise example shows the engagement exposed a missed 5-hour RTO, missing authentication platform, and manual DNS failover, then over four months automated failover with ARC routing controls and pre-provisioned resources to pass the second exercise.


### [Validating your DR commitment: From controlled test to real-world readiness](https://yomu.fyi/post/validating-your-dr-commitment-from-controlled-test-to-real-world-readi.md)
- Company: [AWS](https://yomu.fyi/company/aws.md)
- Author: Peter Garnepudi
- Published: Nov 2, 2026
- Type: Explainer

Disaster recovery commitments defined on paper need validation under conditions that expose coordination, dependency, and provisioning gaps. A scheduled failover with advance notice confirms traffic shifts, data consistency in the recovery Region, and application function afterward, but it does not test who declares failover or whether teams coordinate under pressure. The post describes four progressive validation levels: tabletop exercise, planned failover, surprise drill, and game day with fault injection using AWS Fault Injection Service. In one engagement, stopped DR instances depended on launching capacity and database replicas during the exercise without confirming Regional capacity or account quotas; a planned failover found Route 53 records still resolving to the primary Region 45 minutes in, and recovery took more than twice the 5-hour target. Findings map back to the recovery boundary, business-anchored target, and dependency-reconciled commitment, and a later exercise passed.


### [Three business decisions that define your DR success](https://yomu.fyi/post/three-business-decisions-that-define-your-dr-success.md)
- Company: [AWS](https://yomu.fyi/company/aws.md)
- Author: Peter Garnepudi
- Published: Nov 2, 2026
- Type: Explainer

A disaster recovery strategy executes against targets the business defines, and three decisions made before architecture selection often determine whether teams meet them. The first is drawing the recovery boundary around a business function rather than individual applications, so shared services in other accounts fall inside scope. The second is anchoring recovery time and recovery point objectives to business consequences, with stakeholders confirming the consequence of downtime and the data that must survive. The third is accounting for the full dependency chain, because an application recovers only as fast as its slowest dependency, including services owned by other teams. In one engagement, a shared authentication platform stayed in the primary Region while applications failed over, breaking user login; redefining scope around business functions exposed the dependency and supported a successful test four months later.


### [What took the Internet offline in Q3 2026](https://yomu.fyi/post/what-took-the-internet-offline-in-q3-2026.md)
- Company: [Cloudflare](https://yomu.fyi/company/cloudflare.md)
- Author: Lai Yi Ohlsen
- Published: Oct 27, 2026
- Type: Explainer

Cloudflare Radar recorded 49 Internet disruptions in Q3 2026, with government-directed shutdowns leading at 19 incidents, mostly exam-related in Iraq. Power outages caused 14 disruptions, including repeated grid failures in Cuba (traffic dropped 18-59%), a national outage in Honduras from an interconnection facility failure (63% drop), and two outages in Ukraine from power grid strain and drone strikes (95% drop). Natural disasters included Super Typhoon Bavi in Guam, a 7.4 earthquake in Colombia damaging fiber, a 7.1 earthquake in Japan, and floods in Nepal. A cooling system failure in a Phoenix data center caused near-total traffic loss from Namecheap. Despite these, many outages were brief: 14 recovered in under an hour and 16 in under three hours.


### [Recommended resilience testing in the next generation of AWS Resilience Hub](https://yomu.fyi/post/recommended-resilience-testing-in-the-next-generation-of-aws-resilienc.md)
- Company: [AWS](https://yomu.fyi/company/aws.md)
- Author: Jennifer Moran
- Published: Oct 26, 2026
- Type: Announcement

AWS Resilience Hub introduced recommended resilience testing, which uses AWS Fault Injection Service to automatically create fault injection experiments from existing policies and resource knowledge. The feature offers four pre-configured tests: Availability Zone recovery, dependency validation, Multi-Region isolation, and Multi-Region recovery. Dependency validation blocks connectivity to selected dependencies using DNS query log analysis with a 35-day look-back window, and validates whether the application remains healthy when they are unavailable, surfacing misclassified dependencies. Each test run produces a pass or fail outcome evaluated against success criteria and the recovery time objective defined in your resilience policy, with an auto-generated report capturing a point-in-time snapshot of resources. This capability removes manual experiment design, helps organizations prove their architecture holds under failure, and feeds results into compliance and reporting workflows.


### [Building Kubernetes resilience with Krkn: A 4-phase chaos engineering blueprint](https://yomu.fyi/post/building-kubernetes-resilience-with-krkn-a-4-phase-chaos-engineering-b.md)
- Company: [Red Hat](https://yomu.fyi/company/red-hat.md)
- Author: Naga Ravi Chaitanya Elluri
- Published: Oct 22, 2026
- Type: Tutorial

The post presents a four-phase chaos engineering blueprint for Kubernetes using Krkn. Phase 1, Strategize, defines a resilience baseline by selecting a critical user journey, mapping dependencies, and writing a hypothesis linking failure to expected behavior with SLO-based acceptance criteria. Phase 2, Execute and harden, validates recovery through a consistent sequence of prepare, disrupt and observe, and restore and verify, using Prometheus and application checks. Phase 3, Explore, uses Krkn-AI’s evolutionary algorithms to uncover hidden failure paths, including latency, packet loss, and resource pressure combinations, with customizable configurations. Phase 4, Future-proof and CI, builds reusable workflows using Krkn Operator and Chaos Studio’s visual editor, integrates them into CI/CD pipelines, tracks resilience scores with weighted SLO checks (default 3 for critical, 1 for warnings), and emphasizes repeatability, version-controlled test plans, and pre-flight validation before infrastructure changes, such as verifying that CPU tuning does not increase recovery time beyond a 60-second requirement.
