Loading…
Building a Unified Pipeline for AI Token Spend
Kedar Thakkar, Richard Wang, Veeral Patel
- Source
- Ramp
- Published
- Added to Yomu
Summary
AI token spend is volatile, consumption-based, and distributed across teams, making usage and cost difficult to monitor. Ramp's pipeline ingests LiteLLM and OpenRouter events through authenticated, tenant-scoped webhooks, Kafka, and ClickHouse, then aggregates them for REST dashboards and AI-generated forecasts. LiteLLM uses callbacks with token counts and metadata, while OpenRouter sends OTLP traces; ReplacingMergeTree deduplicates replayed events using business_id, source, and event_id. Customers can connect gateways with an API key, configure LiteLLM callbacks or OpenRouter Broadcast, and pass attribution metadata. The resulting visibility supports breakdowns by provider, model, department, user, project, and use case, and exposed phantom Gemini reasoning tokens after a LiteLLM upgrade, geo pricing premiums, runaway loops, and prompt bloat.
Context
AI spending is consumption-based, variable, and increasingly distributed across teams. The post describes how provider and gateway sprawl makes usage difficult to attribute and how sudden token increases can create latency and cost problems, including a Gemini incident caused by phantom reasoning tokens after a LiteLLM package upgrade.
Approach / What changed
Usage events from LiteLLM and OpenRouter are received through OAuth2-protected, tenant-scoped webhook endpoints, streamed through Kafka, stored in ClickHouse, and aggregated for dashboards, REST APIs, forecasts, and anomaly monitoring. Customers connect gateways with an API key and attach team, project, environment, or other metadata for attribution.
Takeaways
- Kafka absorbs bursty AI usage, while ClickHouse ReplacingMergeTree deduplicates replayed events using business_id, source, event_id, and created_at as the version column.
- LiteLLM supplies completion callbacks with token counts, cost, cache status, and user identity; OpenRouter exports OTLP traces through its Broadcast feature.
- Token-level visibility revealed cost and latency risks including phantom reasoning tokens, geo-pricing premiums, runaway agent loops, abandoned experiments, missing caching, and prompt bloat.