Ingesting the Milky Way: Petabyte-Scale with Zerobus Ingest

Summary

The post benchmarks Zerobus Ingest, a managed, serverless, push-based service that writes producer data directly to Delta tables governed by Unity Catalog, against a petabyte-scale telemetry workload. Using NASA’s NEOWISE dataset and Locust, the test modeled fan-in from 2,048 concurrent streams, using Protocol Buffer 2 data over approximately 24 hours. The design replaces static partition-based ordering with stream-connection ordering, allowing heuristic routing across pods, dynamic partitioning, and autoscaling while existing streams drain. It also uses zeroparser, a zero-copy protobuf decoder whose design relies on Rust’s lifetime system and supports dynamic descriptors at about 1 GB/s per CPU core. The test sustained 12 GB/s to one table, ingested 1.04 trillion rows, and reached 1 petabyte within 24 hours; Zerobus Ingest is generally available, with additional APIs on its roadmap.

Takeaways

1. 1

   Ordering is attached to a producer’s stream connection rather than a partition, allowing pods to be added or removed while preserving ordering for each active stream.

2. 2

   Zeroparser supports dynamically supplied protobuf descriptors through single-pass parsing without memory allocations; the post reports about 1 GB/s of protobuf parsing per CPU core.

3. 3

   The benchmark sustained 12 GB/s to a single table for 24 hours, ingesting 1.04 trillion records from 2,048 concurrent streams using Protocol Buffer 2.