---
title: "Worker Backpressure (Part 1)"
description: "Canva’s queue workers were designed to consume messages greedily, but that behavior could amplify dependency failures, waste scarce capacity, fill dead-letter queues, and create on-call work. Worker Backpressure adds a local feedback loop to the shared queue library: each processed message reports success or failure, a pluggable controller maintains a 0.0-to-1.0 backoff factor against a configured failure-rate set point, and permits scale concurrency at each poll. The mechanism requires no external coordinator or network calls and adds two arithmetic operations to the runtime path, while trading some throughput for failure containment. In two production incidents, it reduced worker pressure during errors and recovered automatically; one sustained-overload incident recorded a 2.76% average failure rate, 22 DLQ messages over 32.5 hours, and no paging, while rollout to all queue workers remained the immediate goal."
---

# Worker Backpressure (Part 1)

[Canva](https://yomu.fyi/company/canva) · Mikalai Barysau · Sep 17, 2026

**Type:** Explainer

## Summary

Canva’s queue workers were designed to consume messages greedily, but that behavior could amplify dependency failures, waste scarce capacity, fill dead-letter queues, and create on-call work. Worker Backpressure adds a local feedback loop to the shared queue library: each processed message reports success or failure, a pluggable controller maintains a 0.0-to-1.0 backoff factor against a configured failure-rate set point, and permits scale concurrency at each poll. The mechanism requires no external coordinator or network calls and adds two arithmetic operations to the runtime path, while trading some throughput for failure containment. In two production incidents, it reduced worker pressure during errors and recovered automatically; one sustained-overload incident recorded a 2.76% average failure rate, 22 DLQ messages over 32.5 hours, and no paging, while rollout to all queue workers remained the immediate goal.

## Context

Canva runs asynchronous work through thousands of queues and diverse workers. When a dependency slows or fails, an unprotected worker can continue pulling messages, increasing load, consuming scarce resources, and sending retries to a dead-letter queue.

## Approach / What changed

Worker Backpressure is built into Canva’s queue library as a local feedback loop. Workers record each message outcome, a pluggable controller converts those outcomes into a backoff factor based on a configured failure-rate set point, and permits reduce concurrent processing as the factor rises; concurrency increases again as outcomes improve.

## Takeaways

- The controller treats success or failure after each message as a dependency-health signal and maintains a backoff factor from 0.0, full speed, to 1.0, fully backed off.
- Backpressure regulates worker concurrency locally, without an external coordinator or added network calls; its runtime cost is two arithmetic operations.
- The mechanism reduced pressure during two production incidents, but the first iteration accepts lower throughput and uses only success-versus-failure as its control signal.

## FAQ

### When does Worker Backpressure begin throttling a worker?

The controller compares observed failures with a configured set point. It does nothing while the failure rate remains below that threshold, then increases backoff as the rate rises and scales down permitted concurrency.

### Does Worker Backpressure require a coordinator or extra network requests?

No. The mechanism runs locally inside the queue worker, using one arithmetic operation after each outcome and another to scale requested concurrency at each poll.

### What trade-offs does the first implementation make?

Backpressure can reduce throughput while containing a dependency failure, although the incidents had enough headroom to absorb the slowdown. It also uses success versus failure as a single proxy for dependency health; latency and messages in flight may be added as signals later.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [Backpressure](https://yomu.fyi/topic/backpressure), [Distributed Systems](https://yomu.fyi/topic/distributed-systems), [Reliability](https://yomu.fyi/topic/reliability)

- Source: [Canva](https://www.canva.dev/blog/engineering/worker-backpressure-part-1-how-we-taught-our-queue-workers-to-slow-down)
- Source URL: https://www.canva.dev/blog/engineering/worker-backpressure-part-1-how-we-taught-our-queue-workers-to-slow-down
- Ingested by Yomu: 2026-09-17T08:00:51.961Z

[Read original post](https://www.canva.dev/blog/engineering/worker-backpressure-part-1-how-we-taught-our-queue-workers-to-slow-down)