How we built it: Real-time analytics for Stripe Billing

Summary

Stripe built a real-time streaming analytics system for Billing after customers said they needed to detect emerging behavior and adapt pricing quickly, replacing a batch pipeline whose subscription updates lagged by 24 hours on average. The architecture uses Apache Flink to maintain compressed subscription history and incrementally process analytics events, while a custom Apache Spark job generates initial Flink state from billions of historical events and supports validation and exports. A new Apache Pinot v2 query engine performs windowed aggregations and joins without offline preaggregation; most updates are processed in under one minute, nearly all reach users within 15 minutes, and production query latency is under 300 milliseconds. Metric-definition changes trigger historical recalculation while new events are buffered and later replayed, preserving a consistent Dashboard view during the transition.

Context

Customers needed timely, high-quality billing data to identify emerging customer behavior and adapt pricing. The previous subscription analytics system relied on batch processing, resulting in an average 24-hour lag for updates, while customers also needed flexible metric definitions and responsive Dashboard queries.

Approach / What changed

The system transforms subscription and invoice updates into analytics events processed by Apache Flink, with a custom Apache Spark job generating initial Flink state from historical events. Apache Pinot's v2 engine handles windowed aggregations and joins at query time. Metric-definition changes use concurrent historical recalculation, temporary buffering of incoming events, state patching, and replay before the Dashboard switches to the updated data.

Takeaways

• Apache Flink incrementally updates a compressed subscription-history state, avoiding re-analysis of every payment whenever new subscription data arrives.
• Apache Pinot v2 windowed aggregation and join support removed offline preaggregation while preserving Dashboard query latency below 300 milliseconds in production.
• During metric-definition changes, new events continue processing under the old definition while buffered events wait for historical recalculation and are then replayed over the updated state.

Related reading