---
title: "Ingesting the Milky Way: Petabyte-Scale with Zerobus Ingest"
description: "The post benchmarks Zerobus Ingest, a managed, serverless, push-based service that writes producer data directly to Delta tables governed by Unity Catalog, against a petabyte-scale telemetry workload. Using NASA’s NEOWISE dataset and Locust, the test modeled fan-in from 2,048 concurrent streams, using Protocol Buffer 2 data over approximately 24 hours. The design replaces static partition-based ordering with stream-connection ordering, allowing heuristic routing across pods, dynamic partitioning, and autoscaling while existing streams drain. It also uses zeroparser, a zero-copy protobuf decoder whose design relies on Rust’s lifetime system and supports dynamic descriptors at about 1 GB/s per CPU core. The test sustained 12 GB/s to one table, ingested 1.04 trillion rows, and reached 1 petabyte within 24 hours; Zerobus Ingest is generally available, with additional APIs on its roadmap."
---

# Ingesting the Milky Way: Petabyte-Scale with Zerobus Ingest

[Databricks](https://yomu.fyi/company/databricks) · Aleksandar Tomić, Victoria Bukta, Nikola Obradović, Danilo Najkov, Branko Grbić, Milos Milovanovic · Jun 11, 2026

**Type:** Benchmark

## Summary

The post benchmarks Zerobus Ingest, a managed, serverless, push-based service that writes producer data directly to Delta tables governed by Unity Catalog, against a petabyte-scale telemetry workload. Using NASA’s NEOWISE dataset and Locust, the test modeled fan-in from 2,048 concurrent streams, using Protocol Buffer 2 data over approximately 24 hours. The design replaces static partition-based ordering with stream-connection ordering, allowing heuristic routing across pods, dynamic partitioning, and autoscaling while existing streams drain. It also uses zeroparser, a zero-copy protobuf decoder whose design relies on Rust’s lifetime system and supports dynamic descriptors at about 1 GB/s per CPU core. The test sustained 12 GB/s to one table, ingested 1.04 trillion rows, and reached 1 petabyte within 24 hours; Zerobus Ingest is generally available, with additional APIs on its roadmap.

## Context

Continuous telemetry from IoT sensors, satellite arrays, and autonomous vehicles produces high-volume time-series streams that must land in a fast, reliable, queryable destination. The post identifies the operational burden of typical Kafka-based workloads, including infrastructure tuning, partition and broker decisions, and connector maintenance.

## Approach / What changed

Zerobus Ingest uses a push-based API to write directly to Delta tables governed by Unity Catalog. Its design uses stream-connection ordering instead of partition-level ordering, heuristic routing across pods for dynamic autoscaling, the zero-copy zeroparser protobuf decoder, and a low-latency write-ahead log with asynchronous acknowledgements. The benchmark used Locust on Kubernetes to generate fan-in from 2,048 concurrent streams.

## Takeaways

- Ordering is attached to a producer’s stream connection rather than a partition, allowing pods to be added or removed while preserving ordering for each active stream.
- Zeroparser supports dynamically supplied protobuf descriptors through single-pass parsing without memory allocations; the post reports about 1 GB/s of protobuf parsing per CPU core.
- The benchmark sustained 12 GB/s to a single table for 24 hours, ingesting 1.04 trillion records from 2,048 concurrent streams using Protocol Buffer 2.

**Tags:** [Delta Lake](https://yomu.fyi/topic/delta-lake), [gRPC](https://yomu.fyi/topic/grpc), [Kubernetes](https://yomu.fyi/topic/kubernetes), [Rust](https://yomu.fyi/topic/rust), [Unity Catalog](https://yomu.fyi/topic/unity-catalog)

- Source: [Databricks](https://www.databricks.com/blog/ingesting-milky-way-petabyte-scale-zerobus-ingest)
- Source URL: https://www.databricks.com/blog/ingesting-milky-way-petabyte-scale-zerobus-ingest
- Ingested by Yomu: 2026-08-30T17:03:09.653Z

[Read original post](https://www.databricks.com/blog/ingesting-milky-way-petabyte-scale-zerobus-ingest)