Introducing uFowarder: The Consumer Proxy for Kafka Async Queuing

Summary

Uber introduces uForwarder, an open-source Consumer Proxy for Kafka Async Queue workloads, after deploying a push-based system across more than 1,000 consumer services. The proxy fetches messages through Kafka’s binary protocol, sends them individually to consumer services over gRPC, collects status codes and processing results, and commits offsets when safe. Production challenges included head-of-line blocking from messages that fail before reaching handlers, growing hardware costs, external data sources that are not ready when events arrive, and requirements for production/non-production and zone isolation. The implemented solutions include context-aware routing through message and gRPC headers, active blocking detection and mitigation, hardware-efficiency improvements, and DelayProcessManager, which pauses only delayed partitions and guarantees at-least delay rather than exact timing. The project is open source, with planned work on consumer-lag handling and direct Protobuf support.

Takeaways

1. 1

   The Consumer Proxy hides Kafka consumer-management complexity behind a gRPC interface and commits offsets after aggregating consumer-service processing results.

2. 2

   Context-aware routing carries subsetting context from Kafka message headers into gRPC request headers, allowing load balancers to isolate production/non-production and zone traffic without splitting topics.

3. 3

   DelayProcessManager pauses only partitions whose first unprocessed message has not met its delay, buffers polled messages, and provides an at-least delay rather than an exact delay.