Loading post…
Presto® on Apache Kafka® At Uber Scale
Summary
Uber connected Presto and Apache Kafka to provide lightweight, interactive SQL queries over Kafka topics for ad-hoc analysis of real-time data. The work addresses operational requests such as checking whether an order UUID appears in a service’s input or output streams, where streaming processors are poorly suited to historical point lookups and real-time OLAP requires onboarding and dedicated resources. The enhanced connector discovers Kafka clusters, topics, and schemas on demand, supports multiple Kafka clusters, caches metadata, enforces _timestamp or _partition_offset filters, and uses a shared Kafka client ID with broker quotas to control consumption. After rollout, example lookups returned within a few seconds, daily usage reached 6,000 queries, and the teams reported improved productivity, while concurrent queries may take longer under quota control.
Context
Uber data teams needed timely, ad-hoc analysis of Kafka streams, including checking whether specific UUIDs appeared in input or output topics. Existing stream-processing systems were not suited to point lookups over past events, while real-time OLAP stores required onboarding, tuning, and dedicated storage and compute resources.
Approach / What changed
The Kafka connector for Presto was enhanced with runtime discovery of Kafka clusters, topics, and schemas, support for multiple clusters, and metadata caching. Queries must include _timestamp or _partition_offset filters, and Kafka broker quotas are applied through a shared client ID across Presto workers to limit consumption and protect cluster stability.
Takeaways
Runtime metadata discovery lets newly onboarded Kafka topics and schemas become queryable without restarting the connector.
Kafka queries without a _timestamp or _partition_offset filter are rejected to prevent excessive data consumption.
A shared Kafka client ID places Presto workers under one broker quota pool; this protects Kafka clusters but can make concurrent queries take longer.
Source and freshness
- Source
- uber.com
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.