When the Pipeline Pushes Back
Backpressure is a feedback loop that slows work when downstream capacity or health deteriorates. Compare its gains, costs, and signals for choosing a control point.
Loading post…
CanvaMikalai BarysauExplainer
Worker Backpressure (Part 1)
Summary
Canva’s queue workers were designed to consume messages greedily, but that behavior could amplify dependency failures, waste scarce capacity, fill dead-letter queues, and create on-call work. Worker Backpressure adds a local feedback loop to the shared queue library: each processed message reports success or failure, a pluggable controller maintains a 0.0-to-1.0 backoff factor against a configured failure-rate set point, and permits scale concurrency at each poll. The mechanism requires no external coordinator or network calls and adds two arithmetic operations to the runtime path, while trading some throughput for failure containment. In two production incidents, it reduced worker pressure during errors and recovered automatically; one sustained-overload incident recorded a 2.76% average failure rate, 22 DLQ messages over 32.5 hours, and no paging, while rollout to all queue workers remained the immediate goal.
Context
Canva runs asynchronous work through thousands of queues and diverse workers. When a dependency slows or fails, an unprotected worker can continue pulling messages, increasing load, consuming scarce resources, and sending retries to a dead-letter queue.
Approach / What changed
Worker Backpressure is built into Canva’s queue library as a local feedback loop. Workers record each message outcome, a pluggable controller converts those outcomes into a backoff factor based on a configured failure-rate set point, and permits reduce concurrent processing as the factor rises; concurrency increases again as outcomes improve.
Takeaways
The controller treats success or failure after each message as a dependency-health signal and maintains a backoff factor from 0.0, full speed, to 1.0, fully backed off.
Backpressure regulates worker concurrency locally, without an external coordinator or added network calls; its runtime cost is two arithmetic operations.
The mechanism reduced pressure during two production incidents, but the first iteration accepts lower throughput and uses only success-versus-failure as its control signal.
FAQ
When does Worker Backpressure begin throttling a worker?
The controller compares observed failures with a configured set point. It does nothing while the failure rate remains below that threshold, then increases backoff as the rate rises and scales down permitted concurrency.
Does Worker Backpressure require a coordinator or extra network requests?
No. The mechanism runs locally inside the queue worker, using one arithmetic operation after each outcome and another to scale requested concurrency at each poll.
What trade-offs does the first implementation make?
Backpressure can reduce throughput while containing a dependency failure, although the incidents had enough headroom to absorb the slowdown. It also uses success versus failure as a single proxy for dependency health; latency and messages in flight may be added as signals later.
Source and freshness
- Source
- canva.dev
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
See the idea
Visual explainers
Apache Camel: From Many App Signals to Governed Actions
Apache Camel is shown as the integration path that brings application actions into one request workflow. The diagram clarifies real-time versus scheduled ingestion, mapping, merging, authorization, and audit.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.