---
title: "How Intelligent Load Shedding Prevents Cascading Failures in Tier-0 Systems"
description: "Salesforce’s Cloud Atlas team operates a globally distributed identity data store that must maintain five nines of availability, roughly five minutes of downtime per year. The team redesigned service protection after per-instance rate limiting proved insufficient under autoscaling and increasingly bursty, unpredictable agent-driven traffic. The new approach combines intelligent load shedding, which monitors queue time as an early indicator of system pressure before error rates spike, with coordinator-free global quota management. Servers independently measure and share customer usage to throttle noisy tenants without a centralized coordinator, preventing one customer’s surge from affecting others. This design prevents cascading failures by shedding lower-priority work before overload spreads across dependent services, ensuring graceful degradation rather than widespread outage."
---

# How Intelligent Load Shedding Prevents Cascading Failures in Tier-0 Systems

[Salesforce](https://yomu.fyi/company/salesforce) · Scott Nyberg · Sep 24, 2026

**Type:** Problem & solution

## Summary

Salesforce’s Cloud Atlas team operates a globally distributed identity data store that must maintain five nines of availability, roughly five minutes of downtime per year. The team redesigned service protection after per-instance rate limiting proved insufficient under autoscaling and increasingly bursty, unpredictable agent-driven traffic. The new approach combines intelligent load shedding, which monitors queue time as an early indicator of system pressure before error rates spike, with coordinator-free global quota management. Servers independently measure and share customer usage to throttle noisy tenants without a centralized coordinator, preventing one customer’s surge from affecting others. This design prevents cascading failures by shedding lower-priority work before overload spreads across dependent services, ensuring graceful degradation rather than widespread outage.

## Context

Cloud Atlas is a globally distributed identity data store that powers a large portion of Salesforce authentication, requiring five nines availability. The team faced challenges as per-instance rate limiting became outdated with autoscaling, and bursty agent-driven traffic created noisy-neighbor problems. Localized overload could cascade through retries, affecting dependent services.

## Approach / What changed

The team implemented intelligent load shedding that monitors queue time to estimate system pressure and sheds lower-priority work before error rates spike. They also deployed coordinator-free global quota management where servers independently measure and share customer usage to throttle noisy tenants without a centralized coordinator, allowing the system to adapt as servers scale.

## Takeaways

- Per-instance rate limiting becomes ineffective under autoscaling because each server’s limits become outdated and busy servers reject requests while capacity remains elsewhere.
- Monitoring queue time as an early indicator of system pressure allows load shedding to react before error rates spike, preventing cascading failures from retries.
- Coordinator-free global quota management prevents noisy tenants from consuming shared capacity by having servers independently measure and share customer usage to reach throttling decisions.

## FAQ

### Why did per-instance rate limiting fail under autoscaling?

Per-instance rate limiting became outdated as instances were added or removed, and each server only understood its own workload. Busy servers could reject requests even while capacity remained available elsewhere, and it could not prevent noisy tenants from consuming shared resources.

### How does intelligent load shedding detect overload early?

The system monitors queue time as an early indicator of system pressure. Latency rises before requests begin failing, so load shedding can react while the system is slowing down, shedding lower-priority work before cascading failures begin.

### What design principle prevents the protection system from becoming a single point of failure?

The team avoided centralized rate limiting because it would introduce another critical dependency. Instead, they used coordinator-free global quota management where servers independently reach throttling decisions by sharing customer usage information, ensuring no single coordinator becomes a bottleneck.

**Tags:** [Autoscaling](https://yomu.fyi/topic/autoscaling), [Distributed Systems](https://yomu.fyi/topic/distributed-systems), [Load Shedding](https://yomu.fyi/topic/load-shedding), [Multi-tenant](https://yomu.fyi/topic/multi-tenant), [Rate Limiting](https://yomu.fyi/topic/rate-limiting)

- Source: [Salesforce](https://engineering.salesforce.com/how-intelligent-load-shedding-prevents-cascading-failures-in-tier-0-systems)
- Source URL: https://engineering.salesforce.com/how-intelligent-load-shedding-prevents-cascading-failures-in-tier-0-systems
- Ingested by Yomu: 2026-09-24T18:00:12.695Z

[Read original post](https://engineering.salesforce.com/how-intelligent-load-shedding-prevents-cascading-failures-in-tier-0-systems)