Loading…
How Uken Games reduces observability costs by 87% with ClickHouse
- Source
- Clickhouse
- Published
- Added to Yomu
Summary
Uken Games replaced Datadog with an open-source observability stack built around ClickHouse to monitor backend services supporting mobile games with millions of players. The migration addressed rising Datadog costs and vendor lock-in while targeting pragmatic feature parity across metrics, traces, and logs. OpenTelemetry collectors run in two layers: lightweight agents on ECS instances pre-filter traces, while an autoscaling gateway batches and buffers writes to SigNoz on ClickHouse; managed Prometheus handles metrics, CloudWatch retains logs, and Grafana provides dashboards and alerts. Uken samples ordinary traces, retains errors and high-latency requests, computes alert metrics without sampling, and limits retention to two weeks. The resulting system stores all traces on one ClickHouse node using about 170 GB, operates without migration downtime, and costs about $12,000 annually—an 87% reduction.
Context
Uken monitored 300 containers, more than 50 EC2 machines, over 100 disks, dozens of databases, and more than 100 service deployments with Datadog. Although the system worked, costs repeatedly climbed to unsustainable levels, and Datadog’s proprietary agent and pricing model influenced architectural decisions and created vendor lock-in.
Approach / What changed
Uken replaced Datadog with an open-source-leaning stack using OpenTelemetry collectors, SigNoz on ClickHouse for traces, AWS-managed Prometheus for metrics, CloudWatch for logs, and Grafana for dashboards and alerting. Collectors use lightweight edge agents for pre-filtering and a gateway layer for autoscaling, batching, and buffering. The team also defined Grafana alerts in YAML and deployed them through CloudFormation and Jenkins.
Takeaways
- A single ClickHouse node stores all of Uken’s traces using about 170 GB of disk, while Grafana runs roughly 600 queries per minute across ClickHouse, CloudWatch, and Prometheus.
- Uken keeps errors and high-latency traces, samples other traces at a representative rate such as 5–10%, and computes alerting metrics without sampling to avoid losing operational signals.
- The new open-source observability system costs about $12,000 per year, reduces observability costs by 87%, and was rolled out with zero downtime.