Unlocking powerful analytics with Stripe Data Pipeline and Google BigQuery

Sushant JainMar 21, 2025

Source: Stripe
Published: Mar 21, 2025
Added to Yomu: Aug 28, 2026

Summary

As businesses scale, the post addresses the need to move Stripe revenue data into BigQuery for recurring reporting and analysis. It describes configuring Stripe Data Pipeline to export transactions, invoices, subscriptions, and payments to Google Cloud Storage, then using a Python-based Google Cloud Run Job to discover the latest snapshot through data_load_times.json and enumerate each table’s Parquet files. The loader sends those files to BigQuery with the Google Cloud clients, creates missing tables, uses WRITE_TRUNCATE to retain the latest snapshot, and parallelizes table loads with ThreadPoolExecutor while collecting errors. After deployment and a test execution, the workflow is verified in BigQuery and scheduled to run every six hours, providing an automated path to refreshed Stripe data for analytics.

Context

Businesses need a reliable way to centralize Stripe revenue data for accounting, product, growth, reporting, and analysis as they scale. The post also identifies manual data handling, pipeline maintenance, outages, and delays as problems that automated delivery can reduce.

Approach / What changed

Configure Stripe Data Pipeline in the Stripe Dashboard to export modeled data to Google Cloud Storage, then deploy a Python script as a Google Cloud Run Job. The script reads data_load_times.json to find the latest table paths, lists Parquet files for LIVEMODE tables, loads them into BigQuery in parallel with the Google Cloud Storage and BigQuery clients, and records loading errors. The job is tested, verified in BigQuery, and scheduled to run every six hours.

Takeaways

data_load_times.json provides the latest folder path for each Stripe Data Pipeline table; the script filters entries by the configured LIVEMODE mode before listing Parquet files.
The BigQuery load configuration uses WRITE_TRUNCATE, so each run replaces the destination table and leaves the latest snapshot available rather than accumulating snapshots.
ThreadPoolExecutor runs table loads in parallel, while the deployment uses a Google Cloud Run Job with up to five retries and can be scheduled every six hours.

Data Pipelines Google Cloud Python

Read original post

Unlocking powerful analytics with Stripe Data Pipeline and Google BigQuery

Sushant JainMar 21, 2025

Source: Stripe
Published: Mar 21, 2025
Added to Yomu: Aug 28, 2026

Summary

Context

Approach / What changed

Takeaways

data_load_times.json provides the latest folder path for each Stripe Data Pipeline table; the script filters entries by the configured LIVEMODE mode before listing Parquet files.
The BigQuery load configuration uses WRITE_TRUNCATE, so each run replaces the destination table and leaves the latest snapshot available rather than accumulating snapshots.
ThreadPoolExecutor runs table loads in parallel, while the deployment uses a Google Cloud Run Job with up to five retries and can be scheduled every six hours.

Data Pipelines Google Cloud Python

Read original post

Related reading

Stripe · Mar 14, 2025
Real-time payment analytics: Building a data pipeline from Stripe to AWS
Payment processors such as Stripe emit numerous transaction events, while dashboard and API-based or batch approaches can limit historical analysis, real-time visibility, and correlation with business metrics as volumes grow. The proposed pipeline sends signed Stripe webhooks through API Gateway to Lambda for validation and enrichment, buffers them in Kinesis Data Streams, and uses a second Lambda to transform and index records in OpenSearch. OpenSearch Dashboards provides visualizations, while CloudWatch tracks pipeline health; Kinesis retention and replay support recovery and reprocessing. The post also describes a Kinesis Data Firehose route that can remove the consumer Lambda and reduce costs but may add batching latency. The resulting design is presented as scalable to millions of transactions per day, with sub-minute payment metrics visibility and flexible historical querying, while security guidance includes encryption, IAM configuration, and regular audits.
James Beswick
Stripe · Mar 4, 2025
Importing sales data from Stripe into AWS
The article presents a low-code method for analyzing Stripe business and customer data and moving scheduled report results into AWS. Stripe Sigma lets users query payment, subscription, and plan-related data with SQL, while Sigma Assistant can generate and execute queries from natural-language prompts such as cancellation analysis. For delivery, the article uses the sigma.scheduled_query_run.created event and Stripe's Event Destination to Amazon EventBridge, where a partner event bus and event rule trigger AWS services without a separate public webhook API. Its Lambda example downloads the report CSV with the Stripe API key, parses it into JavaScript objects, and publishes the result through Amazon SNS; the same event-driven pattern can support Lambda, Step Functions, or Glue workflows.
Hidetaka Okamoto
Stripe · Nov 27, 2024
Data access patterns for simple Stripe integrations
The article examines how applications should store and access product data in simple Stripe integrations as requirements for security, performance, and scalability evolve. It compares using Stripe’s built-in product fields with a separate database, while explaining that publishable API keys cannot retrieve product details and that secret or restricted keys must remain server-side. A web backend or serverless function can proxy requests securely, while CloudFront caching reduces repeated calls and latency but requires an appropriate refresh cadence. For richer metadata, inventory, variants, or custom attributes, the article describes combining DynamoDB with Lambda and Stripe, using Event Destinations or webhooks to synchronize changes and weighing that flexibility against database overhead.
Ben Smith
Stripe · Dec 9, 2024
Enhance your monitoring by integrating Stripe events with AWS CloudWatch Log Groups
Stripe events notify account owners about changes such as successful charges, failed invoice payments, and available reconciliation reports. Because Stripe retains events for 13 months but exposes older events than 30 days only as summaries, Amazon EventBridge can route them into an AWS account and Amazon CloudWatch Log Groups can provide longer-term monitoring and analysis. The setup uses CloudWatch metric filters to match event types such as invoice.payment_failed, convert matches into custom metrics, and support alarms, Amazon SNS notifications, dashboards, and Logs Insights queries. For example, a failed-invoice alarm can use a five-minute observation period, Sum statistics, and a threshold based on expected failure volume or anomaly detection. The resulting setup supports near-real-time visibility into successful charges and failed payments, historical trend analysis, troubleshooting, and responses to unusual activity.
Andrew Robinson