Ceilometer: Uber’s Adaptive Benchmarking Framework

Summary

Ceilometer is Uber’s adaptive benchmarking framework for qualifying new server types, cloud SKUs, kernel upgrades, configuration changes, and machine-learning frameworks before production rollout. It replaces manual, fragmented testing with standardized configurations, automated cluster bring-up and service deployment, portable containerized harnesses, and reporting that includes server specifications, reference SLAs, normalization, resource utilization, cost efficiency, and reliability signals. Its distributed architecture runs synthetic and production-like workloads across dedicated machines, processes and enriches raw results, stores them in blob storage and a centralized data warehouse, and exposes them through an analytics service. The framework covers CPU and memory benchmarks, tools such as SpecCPU2017, SPECjbb2015, NetPerf, and FIO, plus stateful, stateless, batch, and ML workloads; Uber reports uses including regression detection, database validation, and canary testing of external upgrades.

Takeaways

1. 1

   Ceilometer’s reporting combines server specifications, expected SLAs from standard reference data, and normalization to support contextual A/B comparisons.

2. 2

   The platform processes benchmark results through validation, normalization, and enrichment before storing them in blob storage and Uber’s centralized Data Warehouse for analytics.

3. 3

   A containerized harness lets external vendors run Uber’s standardized synthetic benchmarks in their own environments and return performance data as JSON files.