Loading…
Powering self-driving vehicle analytics at Avride with ClickHouse Cloud
ClickhouseClickHouse
- Source
- Clickhouse
- Published
- Added to Yomu
Summary
Avride’s expanding autonomous passenger vehicle and delivery robot fleet generated thousands of data points per minute, overwhelming its legacy Apache Iceberg and Parquet metadata indexing architecture. Iceberg’s optimistic concurrency model struggled with parallel writes, pushing ingestion latencies to hours or days, inflating query times to 10–20 seconds, and requiring raw data duplication across storage locations. To resolve these bottlenecks, Avride built a custom indexing and metrics warehouse powered by ClickHouse Cloud on AWS backed by Amazon S3 object storage. The new architecture dropped ingestion delays to seconds, reduced worst-case lookup latencies to under one second, eliminated the need for bespoke caching layers, and removed payload duplication. Beyond operational analytics, the unified ClickHouse platform now supports internal ride-replay debugging tools, simulation workflows, and high-cardinality vehicle C++ performance trace profiling.
Context
Avride relied on Apache Iceberg as a metadata and indexing layer over Parquet files, but its optimistic concurrency model could not scale with parallel writes from a growing autonomous vehicle fleet. Data ingestion took hours or days, scene lookups required 10 to 20 seconds, teams were forced to build uncoordinated caching layers, and Iceberg's data custody model caused petabyte-scale raw data duplication.
Approach / What changed
Avride replaced Iceberg by building a custom indexing solution and data warehouse on ClickHouse Cloud backed by Amazon S3 object storage. Adopting ClickHouse Cloud enabled separation of storage and compute, reduced cluster maintenance overhead, and unified multi-location indexing without duplicating underlying payload files.
Takeaways
- Ingestion latency dropped from hours or days under Iceberg to seconds using ClickHouse, resolving bottlenecks caused by optimistic concurrency during parallel vehicle data writes.
- Index lookup latencies dropped from 10–20 seconds to under one second in the worst case and under 100 milliseconds on warm connections, rendering bespoke caching layers unnecessary.
- ClickHouse allowed Avride to index metadata without taking custody of underlying payloads, eliminating duplicate raw storage at petabyte scale and enabling unified multi-location queries.