How we built it: Real-time analytics for Stripe Billing

Summary

Stripe built a real-time streaming analytics system for Billing after customers said they needed to detect emerging behavior and adapt pricing quickly, replacing a batch pipeline whose subscription updates lagged by 24 hours on average. The architecture uses Apache Flink to maintain compressed subscription history and incrementally process analytics events, while a custom Apache Spark job generates initial Flink state from billions of historical events and supports validation and exports. A new Apache Pinot v2 query engine performs windowed aggregations and joins without offline preaggregation; most updates are processed in under one minute, nearly all reach users within 15 minutes, and production query latency is under 300 milliseconds. Metric-definition changes trigger historical recalculation while new events are buffered and later replayed, preserving a consistent Dashboard view during the transition.

Takeaways

1. 1

   Apache Flink incrementally updates a compressed subscription-history state, avoiding re-analysis of every payment whenever new subscription data arrives.

2. 2

   Apache Pinot v2 windowed aggregation and join support removed offline preaggregation while preserving Dashboard query latency below 300 milliseconds in production.

3. 3

   During metric-definition changes, new events continue processing under the old definition while buffered events wait for historical recalculation and are then replayed over the updated state.