Enabling Seamless Kafka Async Queuing with Consumer Proxy

Summary

Uber’s Kafka deployment grew from 1 million to 12 million messages per second, but using Kafka’s ordered, partition-bound consumer model for asynchronous queueing created partition-scalability limits and head-of-line blocking. A one-second RPC example would require 1,000 partitions to reach 1,000 events per second, while non-uniform latency and poison-pill messages could stall unrelated work. Consumer Proxy addresses the mismatch by fetching messages through Kafka’s binary protocol, pushing them individually over gRPC, collecting processing statuses, and committing offsets only when safe. It also provides features including out-of-order commits, dead-letter-queue operations, flow control, and circuit breaking while hiding Kafka consumer-group management from services. At Uber, Consumer Proxy became the primary async-queueing solution, with hundreds of services onboarded; adaptive flow control and Kafka cluster federation are planned next steps.

Takeaways

1. 1

   Kafka’s one-message-at-a-time partition processing can let elevated latency or a poison-pill message block later messages in the same partition, even when those messages target unaffected work.

2. 2

   Consumer Proxy presents queueing consumers with a gRPC interface while handling Kafka fetching, status aggregation, and safe offset commits centrally.

3. 3

   Consumer Proxy uses processing timeouts, offset-commit tracking, gRPC response codes, and a circuit breaker for flow control; adaptive flow control is planned to reduce configuration tuning.