Clever ingests 200x more logs at the same cost with ClickHouse Cloud

Summary

Clever, an identity platform for K-12 education, faced a fragmented logging setup as its 400 services and 200 AWS Lambdas generated about 150 TB of uncompressed logs monthly. Datadog indexed only about 10% and retained it for three days, while the remainder was in Athena and required SQL. Clever migrated to ClickHouse Cloud on AWS, indexing all logs with 60-day retention and reporting 200× more searchable log volume at no additional cost, with 10× compression. Its design extracts frequently queried JSON fields into columns, stores remaining attributes in a map alongside raw JSON, uses skipping indexes, date partitions, a 60-day TTL, and reverse Unix timestamp ordering; this ordering produced an 8–10× query speedup. A custom query layer translates LogQL to SQL, while separate development and production services and multi-region deployment address isolation and availability.

Takeaways

1. 1

   Clever increased searchable log volume from 10% of its monthly logs in Datadog to 100% in ClickHouse Cloud, a reported 200× increase, while retaining logs for 60 days at a 0% cost increase.

2. 2

   Reverse chronological ordering by negative Unix timestamp lets ClickHouse find recent records at the front of the data; Clever reports an 8–10× query-time speedup from this design.

3. 3

   A custom query layer translates LogQL to SQL, allowing engineers to use a familiar log exploration syntax while retaining SQL for more complex queries.