---
title: "Real-time payment analytics: Building a data pipeline from Stripe to AWS"
description: "Payment processors such as Stripe emit numerous transaction events, while dashboard and API-based or batch approaches can limit historical analysis, real-time visibility, and correlation with business metrics as volumes grow. The proposed pipeline sends signed Stripe webhooks through API Gateway to Lambda for validation and enrichment, buffers them in Kinesis Data Streams, and uses a second Lambda to transform and index records in OpenSearch. OpenSearch Dashboards provides visualizations, while CloudWatch tracks pipeline health; Kinesis retention and replay support recovery and reprocessing. The post also describes a Kinesis Data Firehose route that can remove the consumer Lambda and reduce costs but may add batching latency. The resulting design is presented as scalable to millions of transactions per day, with sub-minute payment metrics visibility and flexible historical querying, while security guidance includes encryption, IAM configuration, and regular audits."
---

# Real-time payment analytics: Building a data pipeline from Stripe to AWS

[Stripe](https://yomu.fyi/company/stripe) · James Beswick · Mar 14, 2025

**Type:** Tutorial

## Summary

Payment processors such as Stripe emit numerous transaction events, while dashboard and API-based or batch approaches can limit historical analysis, real-time visibility, and correlation with business metrics as volumes grow. The proposed pipeline sends signed Stripe webhooks through API Gateway to Lambda for validation and enrichment, buffers them in Kinesis Data Streams, and uses a second Lambda to transform and index records in OpenSearch. OpenSearch Dashboards provides visualizations, while CloudWatch tracks pipeline health; Kinesis retention and replay support recovery and reprocessing. The post also describes a Kinesis Data Firehose route that can remove the consumer Lambda and reduce costs but may add batching latency. The resulting design is presented as scalable to millions of transactions per day, with sub-minute payment metrics visibility and flexible historical querying, while security guidance includes encryption, IAM configuration, and regular audits.

## Context

Organizations need deeper payment analytics than Stripe's dashboard provides, including historical trend analysis, real-time visibility, and correlation with other business metrics. Batch processing and direct payment-provider API queries encounter rate limits, high latency for complex aggregations, limited historical trending, and schema evolution challenges as transaction volumes grow.

## Approach / What changed

The architecture receives signed Stripe webhooks through API Gateway, validates and enriches them with Lambda, publishes them to Kinesis Data Streams, and processes them with a second Lambda before indexing them in OpenSearch. OpenSearch Dashboards supports visualization and CloudWatch monitors pipeline health. Kinesis Data Firehose can replace the consumer Lambda, trading potentially higher batching latency for lower cost.

## Takeaways

- Each Kinesis shard supports up to 1,000 records per second and 1 MB per second of writes; the post provides a shard-count formula based on peak records per second and peak megabytes per second.
- A hybrid partition-key strategy preserves ordering for payments that require it, including subscriptions, while randomly distributing independent payments to reduce hot-spotting.
- Routing Kinesis events through Kinesis Data Firehose can reduce costs by removing a Lambda consumer, but batching may increase latency.

**Tags:** [AWS](https://yomu.fyi/topic/aws), [Scalability](https://yomu.fyi/topic/scalability), [Search](https://yomu.fyi/topic/search), [Serverless](https://yomu.fyi/topic/serverless), [Streaming](https://yomu.fyi/topic/streaming)

- Source: [Stripe](https://stripe.dev/blog/real-time-payment-analytics-stripe-to-aws-data-pipeline)
- Source URL: https://stripe.dev/blog/real-time-payment-analytics-stripe-to-aws-data-pipeline
- Ingested by Yomu: 2026-08-28T08:58:13.769Z

[Read original post](https://stripe.dev/blog/real-time-payment-analytics-stripe-to-aws-data-pipeline)