The GPU-Resident Astronomy Pipeline
A GPU-resident astronomy pipeline keeps every stage of image analysis — load, align, subtract, detect, review — inside GPU memory instead of round-tripping through the CPU.
Loading…
Topic
Data Pipelines
247 posts about Data Pipelines. Every summary links to the original.
- 247 posts
- 25 companies
- 9 explainers
See the idea
Data Pipelines explainers
BigQuery in the Right Lane: Five Checkpoints for AI Data Pipelines
A five-checkpoint workflow for keeping BigQuery in the right lane: land and shape data once, tune partitions and history windows, and federate instead of copying.
How to Make Apache Hudi Copy-On-Write Upserts Less Wasteful
Learn how Apache Hudi copy-on-write upserts can use row-level file and page locations to update changed data while avoiding unnecessary decompression and rewriting.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Unlock the power of images with AI Sheets
Hugging Face released an update to AI Sheets, an open-source tool for transforming and enriching datasets without code. The update adds vision support powered by Inference Providers, enabling users to upload images, extract structured data, generate visuals, and perform image edits in a spreadsheet interface. Each column can execute AI actions configured with custom prompts and selected models. For text extraction and reasoning tasks, AI Sheets supports vision language models including Qwen/Qwen2.5-VL-7B-Instruct and Qwen/Qwen3-VL-235B-A22B-Reasoning, while image transformations use models like Qwen-Image-Edit. Users can iterate on prompts, provide feedback using thumbs-up ratings to create few-shot examples, and export completed datasets to the Hugging Face Hub, CSV, or Parquet.
Nemotron-Personas-India: Synthesized Data for Sovereign AI
Most open artificial intelligence datasets reflect Western norms and English-only contexts, limiting model effectiveness across India's multilingual and multi-script landscape. To address this shortage of culturally grounded training data, NVIDIA released Nemotron-Personas-India under a CC BY 4.0 license. The synthetic dataset contains 21 million personas across 3 million records spanning 7.7 billion tokens in English, Devanagari Hindi, and Latin-script Hindi. The generation pipeline utilized NeMo Data Designer, an Apache-2.0 licensed Probabilistic Graphical Model for demographic grounding against 2011 Census distributions, and GPT-OSS-120B for narrative text. All records are entirely synthetic to eliminate re-identification risks while enabling fine-tuning for region-aware AI agents and domain-specific copilots.
Unlocking Faster Insights with Experimenter-Defined Segmentations
At Etsy, Catapult supported only 13 hard-coded segmentations, forcing experimenters who needed other cuts to rely on slow, ad-hoc SQL analysis outside the platform. They replaced that fixed model with a Segmentation Library and creation tool where experimenters define reusable segmentations as SQL queries plus join columns connecting segment data to experimental units. Each definition is validated before production, then processed through a configuration-driven pipeline that detects new segmentations and generates independent tasks, improving failure isolation and allowing downstream work to wait only for required results. In the first three months, teams added 34 segmentations, increasing support from 13 to 47, while 78% of experimenters reported saving 2–5 hours monthly through custom analysis in Catapult.
Endian Communication Systems and Information Exchange in Bytes
Endianness dictates the order in which multi-byte numbers are stored and transmitted across systems. While big-endian formats store the most significant byte first at the lowest address, little-endian systems place the least significant byte first. Communication across diverse architectures relies on consistent byte ordering to prevent critical interpretation errors in fields such as telecommunications, IoT, automotive, finance, and aerospace. To safely process disparate payloads, modern ingestion pipelines deploy an endian converter to normalize raw binary data into a unified format. These normalized data streams flow through Apache Kafka, undergo real-time processing and enrichment via Atlas Stream Processing, and persist in MongoDB Atlas for downstream analytics and visualization.
MongoDB SQL Interface: Now Available for Enterprise Advanced
MongoDB has released the general availability of the MongoDB SQL Interface for MongoDB Enterprise Advanced on self-managed deployments running MongoDB 6.0 or higher. Previously, teams relying on SQL-based business intelligence tools faced maintenance bottlenecks when learning MongoDB Query Language or creating custom extract, transform, and load pipelines. The new interface translates standard SQL queries into native MongoDB operations in real time using MongoSQL, a SQL-92 compatible dialect. Users utilize a Schema Builder CLI to generate JSON schemas that map nested document structures and store them alongside data for query validation. This native query translation replaces the MongoDB BI Connector, which will remain supported until September 2026, while offering improved performance and direct connectivity via JDBC and ODBC drivers.
SyGra: The One-Stop Framework for Building Data for LLMs and SLMs
Training and aligning large language models and small language models requires datasets tailored for complex reasoning, preference tuning, multi-turn questioning, and format conversion. To eliminate the need for bespoke data preparation scripts, SyGra provides a low-code and no-code Python framework for automated dataset creation, transformation, and alignment. The library integrates with diverse inference backends, including vLLM, Hugging Face TGI, Triton, and Ollama, allowing teams to focus on prompt engineering rather than pipeline infrastructure. It supports data workflows ranging from converting PDFs and knowledge bases into question-answering pairs to filtering low-quality samples and generating Direct Preference Optimization datasets. These plug-and-play workflows reduce manual curation effort while accelerating model fine-tuning and retrieval-augmented generation pipelines.
How we built it: Real-time analytics for Stripe Billing
Stripe built a real-time streaming analytics system for Billing after customers said they needed to detect emerging behavior and adapt pricing quickly, replacing a batch pipeline whose subscription updates lagged by 24 hours on average. The architecture uses Apache Flink to maintain compressed subscription history and incrementally process analytics events, while a custom Apache Spark job generates initial Flink state from billions of historical events and supports validation and exports. A new Apache Pinot v2 query engine performs windowed aggregations and joins without offline preaggregation; most updates are processed in under one minute, nearly all reach users within 15 minutes, and production query latency is under 300 milliseconds. Metric-definition changes trigger historical recalculation while new events are buffered and later replayed, preserving a consistent Dashboard view during the transition.
`LeRobotDataset:v3.0`: Bringing large-scale datasets to `lerobot`
LeRobotDataset:v3.0 is a new release for lerobot, Hugging Face’s robotics library, addressing file-system limitations encountered when LeRobotDataset:v2 stored one episode per file at million-episode scale. Version 3 packs multiple episodes into shared Parquet and MP4 files, while relational JSON and chunked episode metadata identify episode boundaries and locate the corresponding tabular and visual data. The format separates tabular data, visual data, and metadata; it stores low-dimensional readings in Apache Parquet, concatenates camera frames into MP4 videos, and supports native temporal windowing through delta_timestamps. It also introduces StreamingLeRobotDataset for processing batches from the Hugging Face Hub without downloading the full dataset locally, and provides a one-line conversion utility for datasets in the LeRobotDataset format. The release is planned for lerobot-v0.4.0 and is presented as a step toward datasets with millions of episodes.
Real-time vs batch reconciliation: Practical patterns for keeping data in sync
Product data reconciliation across internal databases, ERPs, and platforms such as Stripe becomes harder to maintain as catalogs and system relationships grow, affecting financial reporting, customer experience, and operations. The post compares real-time reconciliation, using tools such as Kafka, Flink, Delta Live Tables, and Debezium for more immediate consistency, with batch approaches built around Spark, AWS Glue, Snowflake, and dbt for periodic, large-scale checks at lower cost and complexity. It also presents validation options including Great Expectations, Apache NiFi, Talend, and Airbyte, and recommends defining data ownership, applying field-specific conflict rules, and routing critical mismatches to manual review. Soft deletes and tombstones preserve deletion history, while observability through synchronization latency, conflict rates, resolution patterns, and regular audits helps assess reconciliation health as product catalogs scale.
Keeping product data aligned with Stripe as your systems scale
Keeping internal product data aligned with Stripe becomes an architectural challenge as catalogs grow from hundreds to millions of items. The problem spans independently operated systems with different data models, API constraints, rate limits, availability guarantees, and consistency models, while products may change faster than periodic reconciliation can complete. Stripe's payment-focused model covers identifiers, descriptions, prices, currencies, billing terms, and subscription behavior, whereas internal systems may contain categories, inventory, rich content, technical specifications, media, and sensitive cost data. The series examines manual verification, simple scripts, and periodic jobs for smaller catalogs, then points to streaming, change data capture, intelligent partitioning, and conflict resolution for high-scale reconciliation. It concludes that cross-system synchronization requires deliberate mapping and automated architecture because perfect synchronization is difficult to guarantee in eventually consistent distributed systems.
Unlocking powerful analytics with Stripe Data Pipeline and Google BigQuery
As businesses scale, the post addresses the need to move Stripe revenue data into BigQuery for recurring reporting and analysis. It describes configuring Stripe Data Pipeline to export transactions, invoices, subscriptions, and payments to Google Cloud Storage, then using a Python-based Google Cloud Run Job to discover the latest snapshot through data_load_times.json and enumerate each table’s Parquet files. The loader sends those files to BigQuery with the Google Cloud clients, creates missing tables, uses WRITE_TRUNCATE to retain the latest snapshot, and parallelizes table loads with ThreadPoolExecutor while collecting errors. After deployment and a test execution, the workflow is verified in BigQuery and scheduled to run every six hours, providing an automated path to refreshed Stripe data for analytics.
Improving Hugo stability and addressing oncall challenges through automation
Hugo, Grab’s data-ingestion platform for a data lake with more than 4,000 user-onboarded pipelines, faces failures that can leave users with limited visibility and on-call teams handling ad-hoc diagnosis, unclear resolution steps, and DPIs without clear RCA. The proposed automation architecture combines Signal, Diagnosis, RCA Table, Auto-resolution, Data Health API, and Data Health Workbench modules to monitor table health, identify root causes and assignees, execute fixes, and expose status to users and other platforms. Signals come from Airflow failure callbacks and Genchi’s SLA and data-completeness checks; diagnosis avoids parsing extensive logs, runs analysers in parallel according to signal type, and supports asynchronous custom handlers such as retry with backoff for transient errors. Reported outcomes include improved data visibility, faster issue resolution, reduced Hugo on-call workload, scalable handling of transient failures, and more precise data-contract incident management, while next steps include runtime optimisation, richer workbench details, and Flink diagnostics.
Building a Spark observability product with StarRocks: Real-time and historical performance analysis
Grab’s Iris project replaces a fragmented Spark observability stack with a centralised platform for real-time and historical analysis. The previous setup split data between Grafana, Superset, InfluxDB, and a data lake, creating usability, access-control, pipeline, metadata, and high-cardinality query challenges. The new architecture uses StarRocks as a unified store, ingesting Kafka directly, serving a custom Iris web application and Superset, and backing up to S3. Its relational data model links worker metrics through worker_uuid and Spark event metrics through app_id, while materialised views and dynamic partitioning support responsive queries and data retention as volumes grow. The resulting system consolidates metrics and metadata, simplifies ingestion and offline processing, and supports job-level showback/chargeback; the roadmap includes recommendations, advanced analytics, machine learning, and scalability improvements.
Importing sales data from Stripe into AWS
The article presents a low-code method for analyzing Stripe business and customer data and moving scheduled report results into AWS. Stripe Sigma lets users query payment, subscription, and plan-related data with SQL, while Sigma Assistant can generate and execute queries from natural-language prompts such as cancellation analysis. For delivery, the article uses the sigma.scheduled_query_run.created event and Stripe's Event Destination to Amazon EventBridge, where a partner event bus and event rule trigger AWS services without a separate public webhook API. Its Lambda example downloads the report CSV with the Stripe API key, parses it into JavaScript objects, and publishes the result through Amazon SNS; the same event-driven pattern can support Lambda, Step Functions, or Glue workflows.
Tracking customer spend in an omnichannel or multiprocessor environment
Merchants operating across websites, mobile apps, physical stores, and multiple payment processors struggle to connect spending because customers use physical cards, digital wallets, and tokenized methods. Payment Account Reference (PAR), a 29-character alphanumeric identifier introduced by EMVCo, provides a non-financial, non-reversible reference that maps one PAN to multiple tokens without exposing sensitive payment data. Because the PAR is independent of payment method and processor, merchants can link transactions across channels and preserve continuity when a card is replaced with a new PAN. The post presents an omnichannel retail example and describes potential benefits including unified customer profiles, personalization, cross-channel reconciliation, reduced PAN-related PCI burden, and processor flexibility.
The foundations of Canva’s continuous data platform with Snowpipe Streaming
As Canva expanded to over 200 million monthly active users, ingestion throughput reached 25 billion records daily, causing AWS Data Firehose costs to consume nearly half of the product analytics platform budget. To reduce these expenses and eliminate intermediate file staging, the engineering team integrated Snowflake's Snowpipe Streaming directly with their Java-based Kinesis Data Streams pipeline. The architecture streams records directly into Snowflake tables using logical channels with offset checkpoints while configuring client buffering up to a five-minute maximum lag. In handling Kinesis Client Library edge cases, record processors coordinate with the LeaseCoordinator to drop leases when channels become unhealthy. Operating in production for over six months, the system ingested more than 20.35 petabytes of data, decreased query latency to roughly ten minutes, and reduced overall cloud spend by 45 percent.
Developing a modern architecture for energy utilities with embedded finance
Energy utilities can use embedded finance to create marketplace-like experiences for tariffs, solar installations, smart-home devices, storage, and related services while capturing more energy-related revenue. The proposed foundation combines Stripe Online Payment Flows, Billing, Payments, Connect, and Data Pipeline with AWS serverless services, including Lambda, EventBridge, DynamoDB, Redshift, S3, and QuickSight. A tariff enrollment or payment can trigger webhooks and real-time event processing, while transaction and customer data support dashboards, usage insights, personalized recommendations, subscriptions, and broader partner offerings. The architecture also describes PCI Level 1 security, encryption, Radar, Strong Customer Authentication, AWS IAM, KMS, Shield, WAF, Config, Security Hub, and CloudWatch for compliance and monitoring. It concludes that this integrated payment, analytics, and security foundation can help utilities improve customer engagement, support sustainable programs, and develop new revenue streams.
Unveiling the process: The creation of our powerful campaign builder
Grab details the event processing architecture behind Trident, its internal marketing campaign platform that evaluates If This, Then That (IFTTT) logic over Kafka streams. The core processing unit is a treatment consisting of an event, optional conditions, and actions. Complex campaign capabilities—such as counters, limits, and delays across multiple hours via recursive SQS message scheduling—are assembled from multiple coordinated treatments. To simplify campaign creation, Grab introduced a flowchart-like visual builder represented as a JSON node tree that compiles recursively into treatments while persisting node-to-treatment mappings to reconcile edits over time.
LLM-powered data classification for data entities at scale
Grab needed to classify sensitive data at the table and column level across petabytes of database tables and streaming schemas. Manual schema-tiering campaigns had resulted in half of all schemas receiving overly strict Tier 1 access controls, while an initial automated service using regex patterns and third-party machine learning produced high false-positive rates and lacked customizability. To address this, the Caspian data engineering and governance teams enhanced their internal orchestration service, Gemini, by integrating GPT-3.5 via Azure OpenAI. Gemini aggregates classification requests into mini-batches, handles API rate limits, and uses prompt engineering—including few-shot examples, curated tag libraries, and explicit JSON DTO schemas—to reliably tag columns for data owner verification.
Office Hours with Engineering Managing Director Mae Santos
Mae Santos, Head of Critical Applications and Data Reliability Engineering at Two Sigma, leads a global team across New York, Houston, London, and Tokyo to support critical applications and data pipelines. While software teams frequently prioritize functional features over operational stability, designing systems with reliability in mind from the beginning allows for progressive enhancements without complete rewrites. System architecture should be guided by concrete goals across four key dimensions: availability, observability, scalability, and supportability. In addition, mapping out interconnections among systems, infrastructure, data, and human processes helps teams manage dependencies effectively. Applying automation across support and delivery workflows further reinforces reliability by minimizing human error and accelerating execution.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.