Loading…
Topic
248 posts about Data Pipelines. Every summary links to the original.
See the idea
AWS Glue runs serverless Spark jobs that transform data in batches; engine-native pipelines transform during ingestion. This comparison shows how to pick between them by freshness, cost, and data fidelity.
A GPU-resident astronomy pipeline keeps every stage of image analysis — load, align, subtract, detect, review — inside GPU memory instead of round-tripping through the CPU.
A five-checkpoint workflow for keeping BigQuery in the right lane: land and shape data once, tune partitions and history windows, and federate instead of copying.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Stripe’s Data Pipeline now adds Postgres as a destination, enabling developers to sync Stripe data directly into their Supabase project without maintaining webhooks or working around API rate limits. Setup occurs entirely in the Stripe Dashboard: under Reporting, Data management, users select Data Pipeline, choose Postgres and Supabase, and authorize the connection. Stripe backfills historical records and continuously syncs charges, refunds, disputes, customers, subscriptions, and invoices into a dedicated stripe_sdp schema in the user’s Postgres database. A dedicated stripe_sdp role with scoped permissions, JWT direct connections using short-lived tokens, and enforced SSL ensure the database password is never exposed to Stripe. Developers can then query the synced tables with standard SQL, join them with application data, and implement features like MRR calculations, churn analysis, or subscription-based feature gating without extra API calls.
Databricks is introducing a Lakeflow Connect Free Tier to help teams ingest SaaS application and operational database data into the Databricks Data + AI Platform, addressing silos that can leave AI agents without complete enterprise context. Each workspace automatically receives 100 free DBUs per day for managed SaaS and database connectors only, with capacity advertised as up to 100 million records per workspace per day across eligible supported sources. Lakeflow Connect offers 30+ connectors, including SaaS services and databases, through a point-and-click UI and API, with Unity Catalog providing unified governance and end-to-end lineage. The offer uses standard pricing without volume-based premiums or row-based pricing, but actual capacity varies with workload characteristics, record sizes, and source types.
Data transformation turns raw data extracted from source systems into usable, trusted datasets for operational needs, analytics and decision-making. The process addresses inconsistencies, missing values and duplicates through techniques such as cleansing, de-duplication, integration, aggregation, validation and format revision, often mapping data to a target schema during ETL. Performance considerations include compaction, partitioning, file-size tuning, data skipping and pruning, and caching to reduce storage, scanning, processing time and latency. The article also compares on-premises, cloud, open-source, proprietary, no-code, low-code and codable ETL tools, then describes Spark Declarative Pipelines and Lakeflow Jobs on Databricks for quality checks, pipeline development, monitoring and orchestration. A Navy Federal Credit Union proof of concept illustrates a rapid deployment, with a proof of concept completed in a week and a dashboard released days after production deployment.
This user story describes three teams that moved production Postgres workloads from Neon to ClickHouse Managed Postgres after encountering reliability, performance, cost, and network-transfer problems as usage grew. ClickPipes handled schema migration, a parallel initial load, and change data capture, allowing teams to test synchronized databases before cutover. Reported outcomes included zero downtime or connection restarts, faster and more predictable queries, lower CPU or transfer costs, and p99 latency improving from seconds to single-digit milliseconds in one case. The post attributes these gains to local NVMe storage and eliminating transfer fees when analytics already runs on ClickHouse, while noting differences from Neon such as slower branch creation and manual application of several DDL changes.
Construction data is fragmented across systems, organizations, equipment types, and formats, limiting the operational context available for decision-making. Trackunit addresses this data-intelligence problem with IrisX, an operating data platform built on Databricks that connects equipment, machine, operator, site, and operational data. Its three-part model—connect, distill, and amplify—combines data engineering, analytics, and AI on an open platform, then delivers governed, construction-specific intelligence through applications, workflows, and AI interfaces. The platform supports natural-language analysis of equipment signals and contextual factors such as maintenance history, operating conditions, location, contracts, project needs, and asset availability. Examples described include battery-management guidance for OEMs, missed-invoice detection for rental companies, and asset redeployment for contractors, with reported value of approximately $3 million and $2 million in two examples.
ClickHouse Managed Postgres introduces a four-step onboarding experience designed to move users from provisioning a service to working with operational data and ClickHouse analytics. Existing onboarding placed new users on an empty Overview page, while the documentation quickstart required users to track progress outside the product. The new flow guides users through creating a service, migrating or ingesting data, running a query in the SQL Console, and configuring Postgres CDC to ClickHouse. Data can arrive through a managed Postgres-to-Postgres ClickPipe, a language client, a sample dataset, or a table created from scratch; pg_clickhouse provides a PostgreSQL query layer over ClickHouse foreign tables. The article reports less than a minute of replication lag and benchmarks showing five aggregations falling from 555 ms to 164 ms and a JOIN with aggregations from 1,246 ms to 170 ms on one million-row data set.
At scale, training jobs face frequent GPU interruptions, while remote-storage input pipelines can leave accelerators idle. Distributed Checkpoint (DCP) has each rank write a shard plus a .metadata file, allowing parallel saves and reloads onto a different number of GPUs; DCP is presented as useful for DDP as well as sharded training. Asynchronous saves stage data locally and upload in the background; cited measurements report 1.8x savings for a 2.8B-parameter DDP model and 58x for a 20B-parameter FSPD model on 32 H100s, though the comparison excludes torch.save network storage time. UCVolumeDataset and the accompanying DataLoader cache files on local NVMe and prefetch disjoint rank/worker slices while computation runs. Correct recovery also requires checkpointing data position plus seeds and RNG state so resumed sampling remains consistent.
Grab’s data mesh uses the Signals Marketplace to publish, discover, and reuse trusted data products, while certification establishes ownership, documentation, lineage, and quality controls but does not prevent runtime failures. Data Production Issues (DPIs) turn data-quality signals into an operational workflow that begins when Kinabalu detects unhealthy contract tests and closes when the underlying condition recovers. Triage promotes meaningful contract breaches, deduplicates repeated signals, groups related issues, and records affected assets, intervals, contracts, tests, and dependencies. For diagnosis, the Data Health API exposes a portable taxonomy of UPSTREAM_ERROR, PLATFORM_ERROR, JOB_ERROR, and DATA_ERROR, allowing ownership to follow the likely root cause while platforms retain detailed internal diagnostics. Combined with Hugo’s diagnosis and remediation patterns, the approach is presented as reducing resolution time and feeding operational reliability back into certification.
Traditional QSR reporting can show that a limited-time offer (LTO) missed systemwide expectations, but not whether the cause was weak demand, lower franchise participation, ingredient shortages, inconsistent execution, slower service, or menu cannibalization. The post presents Lovelytics’ QSR Executive Performance Control Tower, an AI blueprint/solution built on Databricks that connects corporate performance, supply chain and operations, customer and digital, and financial signals. It is designed to distinguish demand from operational and participation issues, link changes to likely drivers and financial impact, and give corporate and franchise leaders a daily briefing on what changed, why it matters, and where to focus. The stated advantage is faster, more actionable decision-making based on a governed, reusable intelligence layer rather than disconnected reports.
QuintoAndar’s customer data platform was split between a Postgres-based hot layer for low-latency lookups and a Databricks-based cold layer for analytical processing, creating duplicate code and expensive MERGE INTO operations. The team consolidated the pipeline by having its event gateway write directly to ClickHouse Cloud, which also serves API requests and analytical queries while remaining accessible to Trino, Spark, and Superset. Hightouch’s Customer Studio did not support ClickHouse, so QuintoAndar used ClickHouse Managed Postgres and the pg_clickhouse extension to expose ClickHouse tables as Postgres tables. Queries are pushed down to ClickHouse, while Hightouch’s planner and audit tables remain in NVMe-backed Postgres, allowing the existing activation workflow to continue with negligible round-trip impact for that use case.
The FISC Security Guidelines define control objectives for Japanese financial institutions, while the post addresses the challenge of translating them into auditable controls on Databricks data and AI workloads. The Databricks Data + AI Platform, Unity Catalog, and optional Enhanced Security Monitoring and Compliance Security Profile features are presented for identity and network controls, encryption, fine-grained access, audit logs, lineage, and hardened runtimes. For AI workloads, Unity Catalog’s Model Registry links training data, code versions, and evaluation metrics to model versions, while a separate framework maps 97 AI-specific risks to 73 controls. The FISC Customer Capabilities Mapping Matrix connects guideline units to platform capabilities and customer responsibilities, including configuration, SIEM integration, backup, recovery, and evidence; the Security Analysis Tool is suggested for finding workspace gaps, and the material is identified as educational guidance requiring expert review.
Capital markets finance must protect returns on a balance sheet exposed continuously to market, funding, and liquidity risk, while funding costs, capital consumption, and liquidity needs can change intraday. Fragmented desk P&L, treasury, and risk data require shared business context, with ontology and governance linking numbers to desks, products, positions, funding, capital, and permissions. Databricks’ Genie One is presented as a data-smart AI coworker that lets authorized finance users investigate governed data, trace emerging funding or profitability issues, and surface the definitions and source evidence behind capital and valuation figures. One top global bank reduced regulatory data processing for liquidity reporting from 10 hours to 8 minutes, while pricing, funding, risk, and capital decisions remain with accountable people.
Suprema Gaming migrated its analytics platform from Snowflake to ClickHouse Cloud on AWS as part of a company-wide move toward agentic operations. The previous stack delivered data about four hours late, required heavy dbt rebuilds, and often took minutes to answer queries, limiting real-time investigations and interactive analysis. Suprema rebuilt ingestion with ClickPipes for supported sources and Debezium plus Kafka for others, while dbt and GitHub Actions deploy the warehouse-layer structure without external cron jobs, Airflow, or a separate orchestrator; governed access uses role-based controls and query-time PII masking. Before cutover, every metric was validated against the reference source, and the resulting platform reduced warehouse spend by roughly 62%, improved direct-read performance 27x, and maintains about one-minute p95 freshness, supporting an early production deployment of ClickHouse Agents.
Airbnb has open-sourced project-lighthouse-anonymize, a Python library powering their privacy-preserving anonymization used to measure user experience disparities. The release is accompanied by technical papers detailing Core Mondrian, an extended partition-based anonymization algorithm, alongside a framework for measuring anonymized data quality. The library enforces k-anonymity and p-sensitive k-anonymity while offering built-in data quality metrics.
Enterprise data governance tools address fragmented data estates in which teams struggle to locate assets, identify ownership, protect sensitive information, and maintain consistent quality across warehouses, lakes, SaaS applications, and spreadsheets. Positioned above storage and compute, these tools read metadata, schemas, and query logs to provide a shared governance layer for cataloging, lineage, classification, policy enforcement, quality monitoring, and audit reporting. The guide distinguishes standalone catalogs, point solutions, enterprise suites, platform-native tools, and open-source options while treating AI and agent governance as an increasingly important capability for models, prompts, and autonomous agents. It recommends evaluating candidates against seven criteria, comparing total cost of ownership, and planning a phased rollout that starts with highest-risk domains; a lakehouse-native approach can reduce synchronization between separate governance systems.
Data mesh and data fabric address different constraints in distributed data architecture: mesh changes ownership and accountability, while fabric automates integration and governance. A data fabric uses active metadata, machine learning, centralized policy enforcement, and virtualized access to discover, govern, and connect data across heterogeneous environments with less data movement. A data mesh assigns domain teams responsibility for data products, quality, reliability, and lifecycle management, supported by self-serve infrastructure and federated computational governance. The article presents a lakehouse as a substrate where domains publish products while capabilities such as Unity Catalog, Delta Sharing, and Lakehouse Federation provide metadata, governance, and sharing. Its conclusion is that the approaches are complementary: organizations should identify whether organizational bottlenecks or technical fragmentation is the dominant constraint, then consider a hybrid.
Databricks is extending declarative ETL patterns into the Lakehouse SQL Editor so SQL practitioners can define recurring transformations without a dedicated pipelines-oriented environment. The initial operations cover append-only updates, change data capture through AUTO CDC, and targeted batch overwrites through REPLACE WHERE; flows can be scheduled, triggered by upstream updates, run on demand, or orchestrated through SQL tasks in Jobs. APPEND tracks processed source data, AUTO CDC handles keys, sequencing, deletes, and SCD Type 1 or Type 2 results, while Enzyme can incrementally process changed data for REPLACE WHERE. In Lakehouse benchmark testing, Enzyme-powered REPLACE WHERE ran 3.4x faster and 2.5x cheaper than traditional REPLACE WHERE, while users can retain procedural SQL and later move to the Lakeflow Pipelines Editor for multi-file development, dependencies, validation, and deployment.
Open table formats add metadata layers over Parquet or ORC files in object storage, enabling ACID transactions, schema evolution, time travel, and database-like consistency for data lakes. Apache Iceberg uses manifests and manifest lists in a metadata tree, supports partition evolution without rewriting existing files, and creates immutable snapshots for auditing, rollback, and reproducible queries. Delta records atomic writes in an ordered transaction log with checkpoints and is closely associated with Spark, while Hudi uses record-level indexes for frequent updates, change-data-capture, and near-real-time ingestion. All three share core capabilities but differ in partition evolution, update and delete strategies, and multi-engine support; Parquet and ORC remain file formats rather than table formats. The overview recommends scheduled compaction, metadata maintenance, health monitoring, and a production-volume proof of concept when selecting a format.
Transactional and analytical databases serve different workloads: OLTP handles high-volume, real-time reads and writes with ACID guarantees, while OLAP supports complex queries over large historical datasets for reporting and business intelligence. The comparison focuses on trade-offs involving latency, concurrency, data freshness, schema design, transaction size, and storage format. Row-oriented storage and normalized schemas support precise operational updates, whereas column-oriented storage and denormalized designs improve compression, scans, and aggregations across millions of rows. The source describes transactional systems as foundational for applications such as banking, e-commerce, healthcare, and reservations, while analytical systems support dashboards, forecasting, and ad-hoc analysis. Its recommended architecture is to run both systems independently and connect them with Change Data Capture, balancing operational reliability with analytical throughput.
Retail and consumer goods teams often split performance review, demand planning, marketing activation, store execution, and measurement across disconnected systems, making it difficult to connect analysis with results. The Databricks demo presents a unified application built on data unification, governance, and intelligence, combining point-of-sale, loyalty, supply chain, media spend, inventory, and store-operations signals with role-specific access. Users can move from KPI and forecast-versus-actual analysis to SKU or store what-if planning, audience selection, campaign activation, replenishment, price-tag updates, assigned tasks, and post-campaign measurement. Genie supports natural-language querying, while specialized agents and a coordinating supervisor connect sales insights, demand planning, audience building, in-store operations, and measurement. The stated design goal is a governed, shared workflow that lets teams test decisions, execute them across stores, and evaluate outcomes without losing operational context.
The best engineering reads, once a week.
