How Respan is scaling LLM observability with ClickHouse Cloud

Summary

Respan scaled its LLM gateway and observability platform from hundreds of daily requests to 50 million daily events, causing its original Django and Postgres architecture to stall under transaction contention at 50 to 100 requests per second. To overcome Postgres write-ahead logging bottlenecks, the team migrated the ingestion and analytics pipelines to ClickHouse Cloud. Engineers minimized row sizes by truncating large prompt text and writing structured metrics to MergeTree tables, then built incremental minute- and hour-level materialized views alongside scheduled refreshable views for user metadata. Distributed tracing spans are ingested asynchronously with analytical parent-child resolution, utilizing functions like argMax rather than the FINAL modifier to prevent costly table merges. This architectural shift allows Respan to achieve sub-second search across tens of millions of records and support customer dashboards with predictable query performance.

Context

Respan (formerly Keywords AI) builds an AI gateway with built-in observability for production LLM applications. As traffic grew to roughly 30 million requests and 50 million events per day, the original backend—a Django application logging each event directly to Postgres—began experiencing transaction contention and write-ahead log bottlenecks under steady write pressure of 50 to 100 requests per second.

Approach / What changed

Respan migrated to ClickHouse Cloud and structured its data model for memory efficiency and high write throughput. Large prompt inputs and model outputs are truncated, while metrics like latency, throughput, and cost are stored in typed MergeTree columns. Dashboards query incremental materialized views partitioned by organization and cascaded into hourly rollups, as well as 10-minute refreshable views for user metadata. Distributed tracing spans are ingested asynchronously without write-time coordination, and trace metrics are calculated using aggregate functions like argMax instead of the FINAL modifier.

Takeaways

• Respan migrated from Postgres to ClickHouse Cloud after Postgres transaction serialization and write-ahead logging created write bottlenecks at 50 to 100 requests per second.
• The team keeps row sizes small and ingestion rates high by storing typed analytical metrics while deliberately truncating large prompt inputs and model outputs before ingestion.
• To maintain fast dashboard performance and avoid full-table scans, Respan cascades minute-level materialized views into hourly rollups and avoids using the FINAL modifier by computing trace metrics with functions like argMax.

Related reading