Loading post…
ClickhouseProblem & solution
Beyond the warehouse: How METRO Markets built a do-it-all data platform on ClickHouse Cloud
Summary
METRO Markets outgrew its self-hosted Impala and Kudu-based Hadoop stack as the marketplace expanded, with parallel stores, inconsistent reporting, and too much maintenance burdening a lean data team. They evaluated BigQuery, Snowflake, and ClickHouse against production queries, finding faster ClickHouse performance, including one response falling from four seconds to under 200 milliseconds, while its pricing supported more predictable costs. Airbyte moves CDC data from MySQL and Postgres into ClickHouse; ReplacingMergeTree handles deduplication during background merges, and BigQuery-hosted Google Analytics data enters through cloud storage and external tables. The resulting warehouse consolidates company data and supports seller analytics, credit risk features through Query API endpoints, observability, and emerging AI use cases, while teams query a shared dataset through compatible tools; planned Kafka and Flink integration would extend it to streaming workloads.
Context
METRO Markets’ self-hosted Impala and Kudu stack became difficult to scale and maintain as the business grew. Engineering teams created parallel data stores, business reporting relied on an increasingly unreliable system, and different teams worked from different numbers. The lean data team also spent substantial time checking failures instead of advancing new work. The company needed faster queries, predictable costs, and a shared, reliable view of its data.
Approach / What changed
METRO Markets selected ClickHouse Cloud after testing BigQuery, Snowflake, and ClickHouse against production queries. Airbyte loads CDC data from MySQL and Postgres, while ReplacingMergeTree supports deduplication during background merges. Google Analytics data is exported from BigQuery to cloud storage and loaded through external tables. ClickHouse’s MySQL-protocol endpoint enables access from existing services and analytics tools, and Query API endpoints support real-time applications such as seller analytics and credit-risk feature retrieval.
Takeaways
ClickHouse’s MySQL-protocol endpoint let PHP services, Grafana, Looker Studio, developers, and business analysts access the same data through their preferred tools.
Query API endpoints provide user characteristics and account flags to downstream systems and support a lightweight credit-risk feature store without intermediary systems.
METRO Markets chose managed ClickHouse Cloud after experiencing the operational burden of self-hosting; Kafka and Flink integration is planned for future streaming use cases.
Source and freshness
- Source
- clickhouse.com
- Published
- Added to Yomu
- Last updated
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.