Loading…
Ingesting the Milky Way: Petabyte-Scale with Zerobus Ingest
Aleksandar Tomić, Victoria Bukta, Nikola Obradović, Danilo Najkov, Branko Grbić, Milos Milovanovic
- Source
- Databricks
- Published
- Added to Yomu
Summary
The post benchmarks Zerobus Ingest, a managed, serverless, push-based service that writes producer data directly to Delta tables governed by Unity Catalog, against a petabyte-scale telemetry workload. Using NASA’s NEOWISE dataset and Locust, the test modeled fan-in from 2,048 concurrent streams, using Protocol Buffer 2 data over approximately 24 hours. The design replaces static partition-based ordering with stream-connection ordering, allowing heuristic routing across pods, dynamic partitioning, and autoscaling while existing streams drain. It also uses zeroparser, a zero-copy protobuf decoder whose design relies on Rust’s lifetime system and supports dynamic descriptors at about 1 GB/s per CPU core. The test sustained 12 GB/s to one table, ingested 1.04 trillion rows, and reached 1 petabyte within 24 hours; Zerobus Ingest is generally available, with additional APIs on its roadmap.
Context
Continuous telemetry from IoT sensors, satellite arrays, and autonomous vehicles produces high-volume time-series streams that must land in a fast, reliable, queryable destination. The post identifies the operational burden of typical Kafka-based workloads, including infrastructure tuning, partition and broker decisions, and connector maintenance.
Approach / What changed
Zerobus Ingest uses a push-based API to write directly to Delta tables governed by Unity Catalog. Its design uses stream-connection ordering instead of partition-level ordering, heuristic routing across pods for dynamic autoscaling, the zero-copy zeroparser protobuf decoder, and a low-latency write-ahead log with asynchronous acknowledgements. The benchmark used Locust on Kubernetes to generate fan-in from 2,048 concurrent streams.
Takeaways
- Ordering is attached to a producer’s stream connection rather than a partition, allowing pods to be added or removed while preserving ordering for each active stream.
- Zeroparser supports dynamically supplied protobuf descriptors through single-pass parsing without memory allocations; the post reports about 1 GB/s of protobuf parsing per CPU core.
- The benchmark sustained 12 GB/s to a single table for 24 hours, ingesting 1.04 trillion records from 2,048 concurrent streams using Protocol Buffer 2.