Topic

Backpressure

1 post about Backpressure. Every summary links to the original.

See the idea

Backpressure explainers

Loading When the Pipeline Pushes Back image
Trade-offs Backpressure Distributed Systems Performance Rate Limiting Streaming
When the Pipeline Pushes Back
Backpressure is a feedback loop that slows work when downstream capacity or health deteriorates. Compare its gains, costs, and signals for choosing a control point.

Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.

Newest first

Canva
Worker Backpressure (Part 1)
Canva’s queue workers were designed to consume messages greedily, but that behavior could amplify dependency failures, waste scarce capacity, fill dead-letter queues, and create on-call work. Worker Backpressure adds a local feedback loop to the shared queue library: each processed message reports success or failure, a pluggable controller maintains a 0.0-to-1.0 backoff factor against a configured failure-rate set point, and permits scale concurrency at each poll. The mechanism requires no external coordinator or network calls and adds two arithmetic operations to the runtime path, while trading some throughput for failure containment. In two production incidents, it reduced worker pressure during errors and recovered automatically; one sustained-overload incident recorded a 2.76% average failure rate, 22 DLQ messages over 32.5 hours, and no paging, while rollout to all queue workers remained the immediate goal.

The best engineering reads, once a week.

Topic

1 post about Backpressure. Every summary links to the original.

See the idea

Loading When the Pipeline Pushes Back image
Trade-offs Backpressure Distributed Systems Performance Rate Limiting Streaming
When the Pipeline Pushes Back
Backpressure is a feedback loop that slows work when downstream capacity or health deteriorates. Compare its gains, costs, and signals for choosing a control point.

Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.

Newest first

Canva
Worker Backpressure (Part 1)
Canva’s queue workers were designed to consume messages greedily, but that behavior could amplify dependency failures, waste scarce capacity, fill dead-letter queues, and create on-call work. Worker Backpressure adds a local feedback loop to the shared queue library: each processed message reports success or failure, a pluggable controller maintains a 0.0-to-1.0 backoff factor against a configured failure-rate set point, and permits scale concurrency at each poll. The mechanism requires no external coordinator or network calls and adds two arithmetic operations to the runtime path, while trading some throughput for failure containment. In two production incidents, it reduced worker pressure during errors and recovered automatically; one sustained-overload incident recorded a 2.76% average failure rate, 22 DLQ messages over 32.5 hours, and no paging, while rollout to all queue workers remained the immediate goal.

The best engineering reads, once a week.