---
title: "Presto® on Apache Kafka® At Uber Scale"
description: "Uber connected Presto and Apache Kafka to provide lightweight, interactive SQL queries over Kafka topics for ad-hoc analysis of real-time data. The work addresses operational requests such as checking whether an order UUID appears in a service’s input or output streams, where streaming processors are poorly suited to historical point lookups and real-time OLAP requires onboarding and dedicated resources. The enhanced connector discovers Kafka clusters, topics, and schemas on demand, supports multiple Kafka clusters, caches metadata, enforces _timestamp or _partition_offset filters, and uses a shared Kafka client ID with broker quotas to control consumption. After rollout, example lookups returned within a few seconds, daily usage reached 6,000 queries, and the teams reported improved productivity, while concurrent queries may take longer under quota control."
---

# Presto® on Apache Kafka® At Uber Scale

[Uber](https://yomu.fyi/company/uber) · Jul 2, 2026

**Type:** Problem & solution

## Summary

Uber connected Presto and Apache Kafka to provide lightweight, interactive SQL queries over Kafka topics for ad-hoc analysis of real-time data. The work addresses operational requests such as checking whether an order UUID appears in a service’s input or output streams, where streaming processors are poorly suited to historical point lookups and real-time OLAP requires onboarding and dedicated resources. The enhanced connector discovers Kafka clusters, topics, and schemas on demand, supports multiple Kafka clusters, caches metadata, enforces \_timestamp or \_partition\_offset filters, and uses a shared Kafka client ID with broker quotas to control consumption. After rollout, example lookups returned within a few seconds, daily usage reached 6,000 queries, and the teams reported improved productivity, while concurrent queries may take longer under quota control.

## Context

Uber data teams needed timely, ad-hoc analysis of Kafka streams, including checking whether specific UUIDs appeared in input or output topics. Existing stream-processing systems were not suited to point lookups over past events, while real-time OLAP stores required onboarding, tuning, and dedicated storage and compute resources.

## Approach / What changed

The Kafka connector for Presto was enhanced with runtime discovery of Kafka clusters, topics, and schemas, support for multiple clusters, and metadata caching. Queries must include \_timestamp or \_partition\_offset filters, and Kafka broker quotas are applied through a shared client ID across Presto workers to limit consumption and protect cluster stability.

## Takeaways

- Runtime metadata discovery lets newly onboarded Kafka topics and schemas become queryable without restarting the connector.
- Kafka queries without a \_timestamp or \_partition\_offset filter are rejected to prevent excessive data consumption.
- A shared Kafka client ID places Presto workers under one broker quota pool; this protects Kafka clusters but can make concurrent queries take longer.

**Tags:** [Kafka](https://yomu.fyi/topic/kafka), [Presto](https://yomu.fyi/topic/presto), [SQL](https://yomu.fyi/topic/sql)

- Source: [Uber](https://www.uber.com/us/en/blog/presto-on-apache-kafka-at-uber-scale)
- Source URL: https://www.uber.com/us/en/blog/presto-on-apache-kafka-at-uber-scale
- Ingested by Yomu: 2026-09-11T00:02:07.812Z

[Read original post](https://www.uber.com/us/en/blog/presto-on-apache-kafka-at-uber-scale)