ClickhouseProblem & solution
How OpenRouter observes, understands, and acts on 740 million LLM requests a day with ClickHouse Cloud
Summary
OpenRouter, a gateway between developers and LLM inference providers, uses ClickHouse Cloud as its single analytics store to handle over 740 million requests daily across 500+ models. Since adopting ClickHouse in November 2024, monthly requests grew 33x to 19.7 billion, with 110+ trillion tokens served per week. The platform writes request telemetry once and uses materialized views to produce rollups for customer dashboards, public model rankings, and endpoint health, powering observability, AI agents, Slack bots, and routing decisions. Internal AI agents, accessible via Slack, query the same ClickHouse data to answer support questions and trace failures without manual digging. The newest routing component, the Jev Router from TypeSafe AI, classifies prompts into task types, ranks models by real usage from ClickHouse, and feeds outcomes back into the store for continuous improvement.
Context
OpenRouter is a gateway that routes LLM requests to over 500 models across 80+ providers. It needed a single store to ingest large amounts of request telemetry and provide near-real-time rollups for customers, public rankings, and internal routing decisions. ClickHouse Cloud was adopted in November 2024.
Approach / What changed
OpenRouter adopted ClickHouse Cloud as its single analytics store, ingesting request telemetry via synchronous and asynchronous inserts and ClickPipes. Materialized views produce rollups for customer dashboards, public rankings, and endpoint health. A dbt and Airflow warehouse sits on top, and internal AI agents query the data through a bounded support API. The Jev Router from TypeSafe AI classifies prompts into ~30 task types, ranks models by real usage from ClickHouse, filters by user constraints, and writes routing outcomes back to ClickHouse for continuous feedback.
Takeaways
OpenRouter writes request telemetry once and uses ClickHouse materialized views to produce multiple rollups for different audiences (customers, providers, public).
Internal AI agents, accessible via Slack, query ClickHouse through a bounded support API to answer support questions and trace failures without manual data digging.
The Jev Router classifies prompts into ~30 task types, ranks models by real usage from ClickHouse, and writes routing outcomes back to ClickHouse for continuous feedback.
FAQ
How does OpenRouter handle 740 million requests per day?
OpenRouter uses ClickHouse Cloud with synchronous and asynchronous inserts, materialized views for rollups, and a dbt/Airflow warehouse. The raw generation log holds roughly 134 billion rows, and monthly requests grew from 600 million to 19.7 billion (33x) since adoption.
What is the Jev Router and how does it work?
The Jev Router from TypeSafe AI is a 'dealer's choice' mechanism. It classifies prompts into ~30 task types, ranks models by real usage from ClickHouse, filters by user constraints (e.g., cost tier, guardrails), and picks an endpoint with fallbacks. Outcomes are written back to ClickHouse.
How do internal teams access ClickHouse data?
Internal teams use AI agents via Slack that read dbt documentation and have skills for querying ClickHouse. The agents make ClickHouse reads through a bounded support API, allowing staff to trace failures and answer questions without manual digging.
Source and freshness
- Source
- clickhouse.com
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.