---
title: "Data Pipelines"
description: "248 posts about Data Pipelines, summarised, each linking to the original."
---

# Data Pipelines
> 248 posts about Data Pipelines, summarised, each linking to the original.

## Articles

### [Platform Thinking: Three Views from Two Sigma Leaders](https://yomu.fyi/post/platform-thinking-three-views-from-two-sigma-leaders.md)
- Company: [Two Sigma](https://yomu.fyi/company/two-sigma.md)
- Author: Emily Majewski
- Published: Oct 23, 2025
- Type: Explainer

Two Sigma structures its quantitative investment operations around foundational platforms that balance operational speed with scientific rigor. In data engineering, foundational teams provide both raw and curated datasets using BigQuery, CI/CD-managed transformation pipelines, and reusable data contracts termed ice cubes to serve ninety percent of use cases. Quantitative modeling incorporates open-source large language models trained strictly on point-in-time data to rapidly generate features while preventing temporal leakage. Long-term platform innovation follows an epsilon and omega strategy that couples overarching vision with iterative, learning-focused steps. Platform leadership categorizes artificial intelligence applications into advisory, oracle, operational, and agentic functions designed to enhance human productivity rather than replace researchers.


### [Unlock the power of images with AI Sheets](https://yomu.fyi/post/unlock-the-power-of-images-with-ai-sheets.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Ame Vi, Daniel Vila, Francisco Aranda, Damián Pumar, Leandro von Werra, Thomas Wolf
- Published: Oct 21, 2025
- Type: Announcement

Hugging Face released an update to AI Sheets, an open-source tool for transforming and enriching datasets without code. The update adds vision support powered by Inference Providers, enabling users to upload images, extract structured data, generate visuals, and perform image edits in a spreadsheet interface. Each column can execute AI actions configured with custom prompts and selected models. For text extraction and reasoning tasks, AI Sheets supports vision language models including Qwen/Qwen2.5-VL-7B-Instruct and Qwen/Qwen3-VL-235B-A22B-Reasoning, while image transformations use models like Qwen-Image-Edit. Users can iterate on prompts, provide feedback using thumbs-up ratings to create few-shot examples, and export completed datasets to the Hugging Face Hub, CSV, or Parquet.


### [Nemotron-Personas-India: Synthesized Data for Sovereign AI](https://yomu.fyi/post/nemotron-personas-india-synthesized-data-for-sovereign-ai.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Kiran Praveen, Utkarsh Vaidya, Evan A, Lipika Ramaswamy, Dhruv Nathawani, Dane Corneil, Yev Meyer
- Published: Oct 13, 2025
- Type: Announcement

Most open artificial intelligence datasets reflect Western norms and English-only contexts, limiting model effectiveness across India's multilingual and multi-script landscape. To address this shortage of culturally grounded training data, NVIDIA released Nemotron-Personas-India under a CC BY 4.0 license. The synthetic dataset contains 21 million personas across 3 million records spanning 7.7 billion tokens in English, Devanagari Hindi, and Latin-script Hindi. The generation pipeline utilized NeMo Data Designer, an Apache-2.0 licensed Probabilistic Graphical Model for demographic grounding against 2011 Census distributions, and GPT-OSS-120B for narrative text. All records are entirely synthetic to eliminate re-identification risks while enabling fine-tuning for region-aware AI agents and domain-specific copilots.


### [Unlocking Faster Insights with Experimenter-Defined Segmentations](https://yomu.fyi/post/unlocking-faster-insights-with-experimenter-defined-segmentations.md)
- Company: [Etsy](https://yomu.fyi/company/etsy.md)
- Author: Athena Chang
- Published: Oct 8, 2025
- Type: Problem & solution

At Etsy, Catapult supported only 13 hard-coded segmentations, forcing experimenters who needed other cuts to rely on slow, ad-hoc SQL analysis outside the platform. They replaced that fixed model with a Segmentation Library and creation tool where experimenters define reusable segmentations as SQL queries plus join columns connecting segment data to experimental units. Each definition is validated before production, then processed through a configuration-driven pipeline that detects new segmentations and generates independent tasks, improving failure isolation and allowing downstream work to wait only for required results. In the first three months, teams added 34 segmentations, increasing support from 13 to 47, while 78% of experimenters reported saving 2–5 hours monthly through custom analysis in Catapult.


### [Endian Communication Systems and Information Exchange in Bytes](https://yomu.fyi/post/endian-communication-systems-and-information-exchange-in-bytes.md)
- Company: [MongoDB](https://yomu.fyi/company/mongodb.md)
- Author: Nitish Joshi
- Published: Sep 25, 2025
- Type: Explainer

Endianness dictates the order in which multi-byte numbers are stored and transmitted across systems. While big-endian formats store the most significant byte first at the lowest address, little-endian systems place the least significant byte first. Communication across diverse architectures relies on consistent byte ordering to prevent critical interpretation errors in fields such as telecommunications, IoT, automotive, finance, and aerospace. To safely process disparate payloads, modern ingestion pipelines deploy an endian converter to normalize raw binary data into a unified format. These normalized data streams flow through Apache Kafka, undergo real-time processing and enrichment via Atlas Stream Processing, and persist in MongoDB Atlas for downstream analytics and visualization.


### [MongoDB SQL Interface: Now Available for Enterprise Advanced](https://yomu.fyi/post/mongodb-sql-interface-now-available-for-enterprise-advanced.md)
- Company: [MongoDB](https://yomu.fyi/company/mongodb.md)
- Author: Jourdain Patrick, Alexi Antonino
- Published: Sep 25, 2025
- Type: Announcement

MongoDB has released the general availability of the MongoDB SQL Interface for MongoDB Enterprise Advanced on self-managed deployments running MongoDB 6.0 or higher. Previously, teams relying on SQL-based business intelligence tools faced maintenance bottlenecks when learning MongoDB Query Language or creating custom extract, transform, and load pipelines. The new interface translates standard SQL queries into native MongoDB operations in real time using MongoSQL, a SQL-92 compatible dialect. Users utilize a Schema Builder CLI to generate JSON schemas that map nested document structures and store them alongside data for query validation. This native query translation replaces the MongoDB BI Connector, which will remain supported until September 2026, while offering improved performance and direct connectivity via JDBC and ODBC drivers.


### [SyGra: The One-Stop Framework for Building Data for LLMs and SLMs](https://yomu.fyi/post/sygra-the-one-stop-framework-for-building-data-for-llms-and-slms.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Bidyapati Pradhan, Vipul Mittal, Amit Kumar Saha, Surajit Dasgupta
- Published: Sep 22, 2025
- Type: Problem & solution

Training and aligning large language models and small language models requires datasets tailored for complex reasoning, preference tuning, multi-turn questioning, and format conversion. To eliminate the need for bespoke data preparation scripts, SyGra provides a low-code and no-code Python framework for automated dataset creation, transformation, and alignment. The library integrates with diverse inference backends, including vLLM, Hugging Face TGI, Triton, and Ollama, allowing teams to focus on prompt engineering rather than pipeline infrastructure. It supports data workflows ranging from converting PDFs and knowledge bases into question-answering pairs to filtering low-quality samples and generating Direct Preference Optimization datasets. These plug-and-play workflows reduce manual curation effort while accelerating model fine-tuning and retrieval-augmented generation pipelines.


### [How we built it: Real-time analytics for Stripe Billing](https://yomu.fyi/post/how-we-built-it-real-time-analytics-for-stripe-billing.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: Reed Trevelyan
- Published: Sep 16, 2025
- Type: Problem & solution

Stripe built a real-time streaming analytics system for Billing after customers said they needed to detect emerging behavior and adapt pricing quickly, replacing a batch pipeline whose subscription updates lagged by 24 hours on average. The architecture uses Apache Flink to maintain compressed subscription history and incrementally process analytics events, while a custom Apache Spark job generates initial Flink state from billions of historical events and supports validation and exports. A new Apache Pinot v2 query engine performs windowed aggregations and joins without offline preaggregation; most updates are processed in under one minute, nearly all reach users within 15 minutes, and production query latency is under 300 milliseconds. Metric-definition changes trigger historical recalculation while new events are buffered and later replayed, preserving a consistent Dashboard view during the transition.


### [\`LeRobotDataset:v3.0\`: Bringing large-scale datasets to \`lerobot\`](https://yomu.fyi/post/lerobotdataset-v3-0-bringing-large-scale-datasets-to-lerobot.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Francesco Capuano, Michel Aractingi, Quentin Lhoest, Caroline Pascal, Pepijn Kooijmans, Jade Choghari, Remi Cadene, Simon  Alibert, Adil Zouitine, Martino Russi, Steven Palma
- Published: Sep 16, 2025
- Type: Announcement

LeRobotDataset:v3.0 is a new release for lerobot, Hugging Face’s robotics library, addressing file-system limitations encountered when LeRobotDataset:v2 stored one episode per file at million-episode scale. Version 3 packs multiple episodes into shared Parquet and MP4 files, while relational JSON and chunked episode metadata identify episode boundaries and locate the corresponding tabular and visual data. The format separates tabular data, visual data, and metadata; it stores low-dimensional readings in Apache Parquet, concatenates camera frames into MP4 videos, and supports native temporal windowing through delta\_timestamps. It also introduces StreamingLeRobotDataset for processing batches from the Hugging Face Hub without downloading the full dataset locally, and provides a one-line conversion utility for datasets in the LeRobotDataset format. The release is planned for lerobot-v0.4.0 and is presented as a step toward datasets with millions of episodes.


### [Real-time vs batch reconciliation: Practical patterns for keeping data in sync](https://yomu.fyi/post/real-time-vs-batch-reconciliation-practical-patterns-for-keeping-data.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: James Beswick
- Published: Aug 23, 2025
- Type: Explainer

Product data reconciliation across internal databases, ERPs, and platforms such as Stripe becomes harder to maintain as catalogs and system relationships grow, affecting financial reporting, customer experience, and operations. The post compares real-time reconciliation, using tools such as Kafka, Flink, Delta Live Tables, and Debezium for more immediate consistency, with batch approaches built around Spark, AWS Glue, Snowflake, and dbt for periodic, large-scale checks at lower cost and complexity. It also presents validation options including Great Expectations, Apache NiFi, Talend, and Airbyte, and recommends defining data ownership, applying field-specific conflict rules, and routing critical mismatches to manual review. Soft deletes and tombstones preserve deletion history, while observability through synchronization latency, conflict rates, resolution patterns, and regular audits helps assess reconciliation health as product catalogs scale.


### [Keeping product data aligned with Stripe as your systems scale](https://yomu.fyi/post/keeping-product-data-aligned-with-stripe-as-your-systems-scale.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: James Beswick
- Published: Aug 21, 2025
- Type: Explainer

Keeping internal product data aligned with Stripe becomes an architectural challenge as catalogs grow from hundreds to millions of items. The problem spans independently operated systems with different data models, API constraints, rate limits, availability guarantees, and consistency models, while products may change faster than periodic reconciliation can complete. Stripe's payment-focused model covers identifiers, descriptions, prices, currencies, billing terms, and subscription behavior, whereas internal systems may contain categories, inventory, rich content, technical specifications, media, and sensitive cost data. The series examines manual verification, simple scripts, and periodic jobs for smaller catalogs, then points to streaming, change data capture, intelligent partitioning, and conflict resolution for high-scale reconciliation. It concludes that cross-system synchronization requires deliberate mapping and automated architecture because perfect synchronization is difficult to guarantee in eventually consistent distributed systems.


### [Unlocking powerful analytics with Stripe Data Pipeline and Google BigQuery](https://yomu.fyi/post/unlocking-powerful-analytics-with-stripe-data-pipeline-and-google-bigq.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: Sushant Jain
- Published: Mar 21, 2025
- Type: Tutorial

As businesses scale, the post addresses the need to move Stripe revenue data into BigQuery for recurring reporting and analysis. It describes configuring Stripe Data Pipeline to export transactions, invoices, subscriptions, and payments to Google Cloud Storage, then using a Python-based Google Cloud Run Job to discover the latest snapshot through data\_load\_times.json and enumerate each table’s Parquet files. The loader sends those files to BigQuery with the Google Cloud clients, creates missing tables, uses WRITE\_TRUNCATE to retain the latest snapshot, and parallelizes table loads with ThreadPoolExecutor while collecting errors. After deployment and a test execution, the workflow is verified in BigQuery and scheduled to run every six hours, providing an automated path to refreshed Stripe data for analytics.


### [Improving Hugo stability and addressing oncall challenges through automation](https://yomu.fyi/post/improving-hugo-stability-and-addressing-oncall-challenges-through-auto.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Shuguang Xiang
- Published: Mar 20, 2025
- Type: Problem & solution

Hugo, Grab’s data-ingestion platform for a data lake with more than 4,000 user-onboarded pipelines, faces failures that can leave users with limited visibility and on-call teams handling ad-hoc diagnosis, unclear resolution steps, and DPIs without clear RCA. The proposed automation architecture combines Signal, Diagnosis, RCA Table, Auto-resolution, Data Health API, and Data Health Workbench modules to monitor table health, identify root causes and assignees, execute fixes, and expose status to users and other platforms. Signals come from Airflow failure callbacks and Genchi’s SLA and data-completeness checks; diagnosis avoids parsing extensive logs, runs analysers in parallel according to signal type, and supports asynchronous custom handlers such as retry with backoff for transient errors. Reported outcomes include improved data visibility, faster issue resolution, reduced Hugo on-call workload, scalable handling of transient failures, and more precise data-contract incident management, while next steps include runtime optimisation, richer workbench details, and Flink diagnostics.


### [Building a Spark observability product with StarRocks: Real-time and historical performance analysis](https://yomu.fyi/post/building-a-spark-observability-product-with-starrocks-real-time-and-hi.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Huong Vuong
- Published: Mar 6, 2025
- Type: Problem & solution

Grab’s Iris project replaces a fragmented Spark observability stack with a centralised platform for real-time and historical analysis. The previous setup split data between Grafana, Superset, InfluxDB, and a data lake, creating usability, access-control, pipeline, metadata, and high-cardinality query challenges. The new architecture uses StarRocks as a unified store, ingesting Kafka directly, serving a custom Iris web application and Superset, and backing up to S3. Its relational data model links worker metrics through worker\_uuid and Spark event metrics through app\_id, while materialised views and dynamic partitioning support responsive queries and data retention as volumes grow. The resulting system consolidates metrics and metadata, simplifies ingestion and offline processing, and supports job-level showback/chargeback; the roadmap includes recommendations, advanced analytics, machine learning, and scalability improvements.


### [Importing sales data from Stripe into AWS](https://yomu.fyi/post/importing-sales-data-from-stripe-into-aws.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: Hidetaka Okamoto
- Published: Mar 4, 2025
- Type: Tutorial

The article presents a low-code method for analyzing Stripe business and customer data and moving scheduled report results into AWS. Stripe Sigma lets users query payment, subscription, and plan-related data with SQL, while Sigma Assistant can generate and execute queries from natural-language prompts such as cancellation analysis. For delivery, the article uses the sigma.scheduled\_query\_run.created event and Stripe's Event Destination to Amazon EventBridge, where a partner event bus and event rule trigger AWS services without a separate public webhook API. Its Lambda example downloads the report CSV with the Stripe API key, parses it into JavaScript objects, and publishes the result through Amazon SNS; the same event-driven pattern can support Lambda, Step Functions, or Glue workflows.


### [Tracking customer spend in an omnichannel or multiprocessor environment](https://yomu.fyi/post/tracking-customer-spend-in-an-omnichannel-or-multiprocessor-environmen.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: Andrew Robinson
- Published: Feb 19, 2025
- Type: Explainer

Merchants operating across websites, mobile apps, physical stores, and multiple payment processors struggle to connect spending because customers use physical cards, digital wallets, and tokenized methods. Payment Account Reference (PAR), a 29-character alphanumeric identifier introduced by EMVCo, provides a non-financial, non-reversible reference that maps one PAN to multiple tokens without exposing sensitive payment data. Because the PAR is independent of payment method and processor, merchants can link transactions across channels and preserve continuity when a card is replaced with a new PAN. The post presents an omnichannel retail example and describes potential benefits including unified customer profiles, personalization, cross-channel reconciliation, reduced PAN-related PCI burden, and processor flexibility.


### [The foundations of Canva’s continuous data platform with Snowpipe Streaming](https://yomu.fyi/post/the-foundations-of-canva-s-continuous-data-platform-with-snowpipe-stre.md)
- Company: [Canva](https://yomu.fyi/company/canva.md)
- Author: Jack Caperon
- Published: Jan 6, 2025
- Type: Problem & solution

As Canva expanded to over 200 million monthly active users, ingestion throughput reached 25 billion records daily, causing AWS Data Firehose costs to consume nearly half of the product analytics platform budget. To reduce these expenses and eliminate intermediate file staging, the engineering team integrated Snowflake's Snowpipe Streaming directly with their Java-based Kinesis Data Streams pipeline. The architecture streams records directly into Snowflake tables using logical channels with offset checkpoints while configuring client buffering up to a five-minute maximum lag. In handling Kinesis Client Library edge cases, record processors coordinate with the LeaseCoordinator to drop leases when channels become unhealthy. Operating in production for over six months, the system ingested more than 20.35 petabytes of data, decreased query latency to roughly ten minutes, and reduced overall cloud spend by 45 percent.


### [Developing a modern architecture for energy utilities with embedded finance](https://yomu.fyi/post/developing-a-modern-architecture-for-energy-utilities-with-embedded-fi.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: Rajan Patel
- Published: Dec 17, 2024
- Type: Explainer

Energy utilities can use embedded finance to create marketplace-like experiences for tariffs, solar installations, smart-home devices, storage, and related services while capturing more energy-related revenue. The proposed foundation combines Stripe Online Payment Flows, Billing, Payments, Connect, and Data Pipeline with AWS serverless services, including Lambda, EventBridge, DynamoDB, Redshift, S3, and QuickSight. A tariff enrollment or payment can trigger webhooks and real-time event processing, while transaction and customer data support dashboards, usage insights, personalized recommendations, subscriptions, and broader partner offerings. The architecture also describes PCI Level 1 security, encryption, Radar, Strong Customer Authentication, AWS IAM, KMS, Shield, WAF, Config, Security Hub, and CloudWatch for compliance and monitoring. It concludes that this integrated payment, analytics, and security foundation can help utilities improve customer engagement, support sustainable programs, and develop new revenue streams.


### [Unveiling the process: The creation of our powerful campaign builder](https://yomu.fyi/post/unveiling-the-process-the-creation-of-our-powerful-campaign-builder.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Jie Zhang
- Published: Sep 10, 2024
- Type: Explainer

Grab details the event processing architecture behind Trident, its internal marketing campaign platform that evaluates If This, Then That (IFTTT) logic over Kafka streams. The core processing unit is a treatment consisting of an event, optional conditions, and actions. Complex campaign capabilities—such as counters, limits, and delays across multiple hours via recursive SQS message scheduling—are assembled from multiple coordinated treatments. To simplify campaign creation, Grab introduced a flowchart-like visual builder represented as a JSON node tree that compiles recursively into treatments while persisting node-to-treatment mappings to reconcile edits over time.


### [LLM-powered data classification for data entities at scale](https://yomu.fyi/post/llm-powered-data-classification-for-data-entities-at-scale.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Hualin Liu
- Published: Jul 15, 2024
- Type: Problem & solution

Grab needed to classify sensitive data at the table and column level across petabytes of database tables and streaming schemas. Manual schema-tiering campaigns had resulted in half of all schemas receiving overly strict Tier 1 access controls, while an initial automated service using regex patterns and third-party machine learning produced high false-positive rates and lacked customizability. To address this, the Caspian data engineering and governance teams enhanced their internal orchestration service, Gemini, by integrating GPT-3.5 via Azure OpenAI. Gemini aggregates classification requests into mini-batches, handles API rate limits, and uses prompt engineering—including few-shot examples, curated tag libraries, and explicit JSON DTO schemas—to reliably tag columns for data owner verification.


[Newer posts](https://yomu.fyi/topic/data-pipelines/page/8.md) · [Older posts](https://yomu.fyi/topic/data-pipelines/page/10.md)