---
title: "When the Pipeline Pushes Back"
description: "Backpressure is a feedback loop that slows work when downstream capacity or health deteriorates. Compare its gains, costs, and signals for choosing a control point."
---

# When the Pipeline Pushes Back

**Type:** [Trade-offs](https://yomu.fyi/visual-explainers/category/trade-offs.md)

![A left-to-right pipeline shows producers feeding a queue and workers, with feedback from failures and latency reducing intake. Three panels compare benefits, costs, and conditions for using backpressure.](https://yomufyi.s3.us-east-1.amazonaws.com/visual-explainers/b458429c-459b-4c60-8c5e-c061d71862cd/image-2839a90f-dc1d-4216-aeee-77d7d0e34385.webp)

> Backpressure is a feedback loop that slows work when downstream capacity or health deteriorates. Compare its gains, costs, and signals for choosing a control point.

## Explanation

Backpressure keeps a fast producer from overwhelming workers, dependencies, or downstream storage by feeding system conditions back into concurrency or intake decisions. The diagram contrasts local worker feedback, tuned rate limits, and stream-processing controls, then weighs their benefits against reduced peak throughput and added operating logic. Use the decision cue to place control where failure containment, latency, or capacity protection matters most.

## FAQ

### Why can backpressure improve reliability even when it reduces throughput?

It can reduce worker pressure during dependency failures, helping contain failures, preserve scarce capacity, and avoid filling dead-letter queues. The trade-off is lower peak throughput while the system slows intake or concurrency.

### Where should the feedback loop live: at the queue, workers, or stream processor?

Place it at the nearest control point that can observe the limiting condition and safely reduce work. Canva’s example uses local worker feedback, while the MakeMyTrip example combines tuned backpressure with rate limits in a streaming pipeline.

### What signals should trigger a backpressure response?

Useful signals include processed-message success or failure and rising downstream pressure; the supplied examples also discuss failure-rate targets, latency, and rate limits. The controller should connect those signals to an explicit intake or concurrency adjustment.

**Tags:** [Backpressure](https://yomu.fyi/topic/backpressure.md), [Distributed Systems](https://yomu.fyi/topic/distributed-systems.md), [Performance](https://yomu.fyi/topic/performance.md), [Rate Limiting](https://yomu.fyi/topic/rate-limiting.md), [Streaming](https://yomu.fyi/topic/streaming.md)

## Related reading

### [Google Cloud Use Cases: How Enterprises Run Data & AI on GCP with Databricks](https://yomu.fyi/post/google-cloud-use-cases-how-enterprises-run-data-ai-on-gcp-with-databri.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Sep 18, 2026
- Type: Explainer

The guide presents Google Cloud as a unified platform for enterprise data engineering, analytics, machine learning, and generative AI, with managed services intended to reduce infrastructure and integration work. It focuses on practical architectures, including medallion ETL pipelines that store Bronze, Silver, and Gold Delta Lake tables in Google Cloud Storage and expose Gold data to BigQuery through an optimized connector. For streaming workloads, Cloud Pub/Sub supplies ingestion while Apache Spark Structured Streaming processes continuous micro-batches; together with Delta Lake ACID transactions, this supports exactly-once results despite duplicates or out-of-order messages. The text also describes Cloud Run and Cloud Functions for containerized or event-driven workloads, and identifies retrieval-augmented generation applications on Vertex AI as a common starting point for generative AI. It concludes that integrated service combinations, rather than isolated tools, are the central strategic advantage for organizations building evolving data and AI platforms on Google Cloud.


### [How MakeMyTrip achieved millisecond personalization at scale with Databricks](https://yomu.fyi/post/how-makemytrip-achieved-millisecond-personalization-at-scale-with-data.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Sitesh Sharma, Aditya Kumar, Navneeth Nair
- Published: Sep 18, 2026
- Type: Problem & solution

MakeMyTrip needed sub-second personalization for “last-searched” hotels across its consumer and corporate travel businesses, serving millions of daily users. Spark Structured Streaming’s micro-batch mode consistently delivered one to two seconds, while adopting Flink would have added a second engine, duplicated logic and increased operational overhead. The team instead adopted Databricks Real-Time Mode (RTM), using continuous data flow, concurrent pipeline scheduling and streaming shuffle to process events with millisecond-level latency on Spark. Its unified pipeline merges B2C and B2B Kafka streams, enriches events with each user’s “Last N” hotels from Aerospike, and writes results to Redis for serving in under 50ms. After tuning partitions, backpressure and rate limits, the production deployment reduced P50 latency from approximately 1.23 seconds to 44ms, P99 latency from over one minute to about 500ms, and increased click-through rate by 7%.


### [The official ClickHouse provider for Apache Airflow is now available](https://yomu.fyi/post/the-official-clickhouse-provider-for-apache-airflow-is-now-available.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: Sep 17, 2026
- Type: Announcement

Apache Airflow users can now use an upstream ClickHouse provider, offering an officially maintained alternative to community plugins or custom integration code. The provider, apache-airflow-providers-clickhousedb, uses ClickHouse Connect over HTTP(S), integrates with Airflow’s common SQL framework, and supplies ClickHouseHook for bulk inserts, streaming, and client-specific operations. The post documents installation, connection setup, standard SQLExecuteQueryOperator DAGs, bulk insertion, and local or managed deployment through Astronomer, including TLS settings for ClickHouse Cloud. Production examples include Astronomer’s Astro Observe, Chartmetric’s 15-million-row playlist pipeline every five minutes, and ClickHouse’s warehouse, which schedules 76 DAGs across more than 40 sources and moves about 6 billion rows daily. The provider is available now, with future capabilities guided by community requests.


### [Worker Backpressure (Part 1)](https://yomu.fyi/post/worker-backpressure-part-1.md)
- Company: [Canva](https://yomu.fyi/company/canva.md)
- Author: Mikalai Barysau
- Published: Sep 17, 2026
- Type: Explainer

Canva’s queue workers were designed to consume messages greedily, but that behavior could amplify dependency failures, waste scarce capacity, fill dead-letter queues, and create on-call work. Worker Backpressure adds a local feedback loop to the shared queue library: each processed message reports success or failure, a pluggable controller maintains a 0.0-to-1.0 backoff factor against a configured failure-rate set point, and permits scale concurrency at each poll. The mechanism requires no external coordinator or network calls and adds two arithmetic operations to the runtime path, while trading some throughput for failure containment. In two production incidents, it reduced worker pressure during errors and recovered automatically; one sustained-overload incident recorded a 2.76% average failure rate, 22 DLQ messages over 32.5 hours, and no paging, while rollout to all queue workers remained the immediate goal.