---
title: "Data Pipelines"
description: "248 posts about Data Pipelines, summarised, each linking to the original."
---

# Data Pipelines
> 248 posts about Data Pipelines, summarised, each linking to the original.

## Articles

### [Beyond parsing X12: Closing the gap for revenue cycle workflows in healthcare](https://yomu.fyi/post/beyond-parsing-x12-closing-the-gap-for-revenue-cycle-workflows-in-heal.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Aaron Zavora, Neel Shapur
- Published: Jun 2, 2026
- Type: Problem & solution

Healthcare X12 pipelines can normalize remittance and enrollment data without giving medical billers a place to act on denials, leaving short-pays in SQL queries and spreadsheets while filing deadlines approach. Genpact and Databricks address that workflow gap with a two-layer operational workbench inside Databricks: production extensions to the open-source x12-edi-parser and a secure web application over Databricks SQL. The engine adds contextual 834 enrollment tracking and decoded 835 CAS adjustments, while the interface provides claim and service-line drill-downs, denial queues, dashboards, quality gates, and appeal or correction drafting. Live Unity Catalog gold-view queries avoid ETL shadow copies and synchronized caches; PHI remains inside the secure perimeter, with row-level security and audit logging for reveals and workflow changes. The proposed next step is Claude-based appeal drafting through Databricks Foundation Model APIs, and a two-week configuration-based trial uses an organization's schema and twenty existing denials.


### [When history fails you, borrow from geography](https://yomu.fyi/post/when-history-fails-you-borrow-from-geography.md)
- Company: [Airbnb](https://yomu.fyi/company/airbnb.md)
- Author: Harrison Katz
- Published: Jun 2, 2026

When historical travel data became uninformative during the asynchronous post-COVID recovery, Airbnb redesigned its demand forecasting to borrow signals across geographies rather than waiting for local data. Using a hierarchical Bayesian framework, the team propagated updated posterior estimates from early-recovering corridors as informative priors for structurally similar, later-recovering corridors. This allowed accurate corridor-level demand forecasting in near real time during periods of high disruption and uneven market reopening.


### [Scaling up your microservice testing with Apache Spark](https://yomu.fyi/post/scaling-up-your-microservice-testing-with-apache-spark.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: Vivek Yadav
- Published: Jun 1, 2026
- Type: Explainer

Microservices whose behavior depends on long-tail production inputs—such as payment cost estimation—are difficult to validate with unit and integration tests alone. The described pattern separates deterministic service logic from transport and side effects, then invokes that logic in Apache Spark across historical records enriched with the required dependency state. The resulting datasets can compare current and candidate implementations, quantify changed outputs, model future rule changes, and identify affected transaction or merchant segments. Replay inputs and artifacts are minimized and protected through tokenized identifiers, redaction, access controls, and retention policies, while smaller privacy-safe golden datasets run in pull-request CI. The approach complements, rather than replaces, unit and integration tests: golden datasets cover known service behavior, and full replays expose changes across the real production distribution before deployment.


### [Open House 2026 Day 2: 10 years of open source and what the ecosystem is building next](https://yomu.fyi/post/open-house-2026-day-2-10-years-of-open-source-and-what-the-ecosystem-i.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: May 28, 2026
- Type: Announcement

Open House 2026 Day 2 recaps ClickHouse’s tenth year as an open-source project and the ecosystem forming around its data platform. The event reported more than 2,600 contributors, nearly 48,000 GitHub stars, over 200 integrations, and the launch of the House Mates partner program. Announcements included Fivetran’s generally available ClickHouse Cloud destination, a Google Cloud Dataflow Pub/Sub template, Sigma’s public-beta connector and ETL Cache Layer, and a ClickHouse dbt Fusion adapter available in alpha through the CLI. The recap also describes MCP and agent-oriented observability work, including a ClickStack MCP server that translates investigations into optimized ClickHouse queries. Internal testing reported up to 25% fewer tool calls, 2.5 times greater consistency, and nearly 20% better evaluation results.


### [Introducing House Mates: the ClickHouse partner community and program](https://yomu.fyi/post/introducing-house-mates-the-clickhouse-partner-community-and-program.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: May 27, 2026
- Type: Announcement

ClickHouse introduces House Mates, a partner community and program bringing together more than 60 ISV, consulting, services, and channel partners to help customers build, scale, and procure ClickHouse solutions. The inaugural cohort includes over 25 ISVs developing validated, production-ready integrations—among them dbt Labs, Fivetran, Temporal, Grafana Labs, Confluent, and Sigma—and over 35 consulting, services, and reseller partners trained and certified through ClickHouse Academy. The program addresses an ecosystem with more than 200 integrations in the wild, demand for trusted regional expertise and preferred procurement channels, and the limits of managing partnerships through one-off arrangements. Its Ignite, Accelerate, and Prime tiers define progression from enablement and first projects or integrations to deeper go-to-market, incentives, co-innovation, and strategic collaboration at scale, while customers gain validated tools, regional experts, and preferred purchasing routes.


### [5 countries, 12 days, one ClickHouse: inside Alexey's APJ AI Tour](https://yomu.fyi/post/5-countries-12-days-one-clickhouse-inside-alexey-s-apj-ai-tour.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: May 27, 2026
- Type: Announcement

ClickHouse CTO and creator Alexey Milovidov completed a 12-day APJ tour spanning India, Singapore, Indonesia, South Korea, and Japan, with customer meetings, industry events, executive sessions, press interviews, and community meetups. The itinerary covered six cities and included three Confluent Data Streaming World Tour events, four ClickHouse meetups, and more than 10 enterprise customer meetings across varied industries and workloads. Discussions centered on real-time analytics, connected products, and analytics at industrial scale, while in-person conversations supplied feedback that the post says helps improve ClickHouse. The trip concluded with the ClickHouse Japan Launch Event on April 27, marking the company’s official entry into that market. The account presents physical presence as a way to extend ClickHouse’s open-building culture beyond GitHub and strengthen its feedback loop with APJ customers and communities.


### [Postgres managed by ClickHouse is now in beta](https://yomu.fyi/post/postgres-managed-by-clickhouse-is-now-in-beta.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: Sai Srirampur
- Published: May 27, 2026
- Type: Announcement

ClickHouse Cloud has launched a fully managed Postgres service in beta to combine transactional and analytical workloads within a single platform. Backed by local NVMe storage, the service provides up to ten times faster transactional performance alongside native change data capture into ClickHouse. Developers can query across both transactional and analytical layers using the pg\_clickhouse extension without managing custom data pipelines. The platform also offers managed migration workflows via ClickPipes, database branching, point-in-time recovery, high availability with up to two standbys, and support for over ninety PostgreSQL extensions. More than fifty virtual machine configurations are available, ranging from single vCPU instances to clusters with ninety-six vCPUs and sixty terabytes of NVMe storage.


### [Integrating the Rust Delta Kernel into ClickHouse](https://yomu.fyi/post/integrating-the-rust-delta-kernel-into-clickhouse.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: Melvyn Peignon, Kseniia Sumarokova, Raúl Marín
- Published: May 22, 2026
- Type: Problem & solution

ClickHouse initially built native protocol support to read Delta Lake tables, but maintaining pace with the format's evolving specification created high complexity and fragmented features. To resolve this, engineers integrated the Rust Delta Kernel to offload protocol handling while retaining ClickHouse's optimized Parquet data reading. The Delta Kernel parses JSON transaction logs, resolves snapshots, manages transactional write metadata, and exposes schema evolution mapping alongside Change Data Feed streams. ClickHouse also contributed upstream enhancements to the Rust kernel, adding dynamic runtime logging and asynchronous metadata processing via FFI handles. While ClickHouse can attach to existing Delta tables, the integration cannot yet initialize empty Delta Lake tables.


### [The Hugo evolution: Engineering Grab's unified, one-click data ingestion platform with Apache Flink](https://yomu.fyi/post/the-hugo-evolution-engineering-grab-s-unified-one-click-data-ingestion.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Shuguang Xiang
- Published: May 22, 2026
- Type: Problem & solution

Hugo evolved from a batch-oriented Spark platform into Grab’s unified, self-service data ingestion system for near real-time pipelines. The earlier expansion across Kafka Connect, Sprinkler, and Hugo fragmented onboarding, forcing users to translate configurations and manage manual schema changes across systems. A custom automation layer and Apache Flink now orchestrate one-click MySQL CDC and Kafka ingestion, retiring Kafka Connect and Sprinkler for the data lake. For CDC, Flink reads the binlog directly, uses automated schema detection and dynamic validation, and writes to queryable Hive tables through Spark compaction; for Kafka, it fetches Protobuf schemas from Confluent Schema Registry at runtime and resumes from the latest checkpoint after restart. Onboarding now takes approximately three minutes for MySQL CDC and six for Kafka, although schema updates still require manual restarts; new pipeline adoption in the last year also exceeded the previous five years combined.


### [Making User-Sequence Data More Cost-Efficient, Faster, and Easier to Use](https://yomu.fyi/post/making-user-sequence-data-more-cost-efficient-faster-and-easier-to-use.md)
- Company: [Pinterest](https://yomu.fyi/company/pinterest.md)
- Author: Pinterest Engineering
- Published: May 21, 2026
- Type: Problem & solution

User sequences serve as core primitives powering machine learning models across ranking, retrieval, and recommendation surfaces at Pinterest. Managing these sequences previously caused high infrastructure costs, operational fragility, and data drift between offline training pipelines and online serving systems. In response, engineering teams redesigned the platform around a single definition executed across multiple runtimes, including real-time streaming, batch backfills, and online inference. The updated architecture pairs a unified enrichment engine with columnar time-partitioned storage and standardized validation tiers. Consequently, the redesign achieved lower storage and network costs, faster signal onboarding via configuration, and improved engagement metrics across major recommendation surfaces.


### [How Partition Access Visualizations Reduced our Data Lake S3 Cost by 33%](https://yomu.fyi/post/how-partition-access-visualizations-reduced-our-data-lake-s3-cost-by-3.md)
- Company: [Yelp](https://yomu.fyi/company/yelp.md)
- Author: Nick Del Nano, Data Streaming
- Published: May 21, 2026
- Type: Problem & solution

Data teams at Yelp struggled to identify table consumers and track how datasets were accessed across their petabyte-scale data lake. To solve this, the team built a usage attribution system using Amazon S3 server access logs to visualize time-based partition keys against access timestamps and IAM roles. These visualizations revealed distinctive patterns for daily batch jobs, backfills, and ad hoc queries, guiding migrations of thousands of tables to Apache Iceberg format. Furthermore, Yelp implemented a Default Access Retention strategy gated by IAM bucket policies to prevent accidental queries from resetting S3 Intelligent Tiering transitions. Combining these granular usage insights with targeted S3 storage classes reduced Yelp's S3 storage costs by 33 percent.


### [Unlock seamless and cost-effective marketing campaigns with Lakebase](https://yomu.fyi/post/unlock-seamless-and-cost-effective-marketing-campaigns-with-lakebase.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Thomas Nguyen
- Published: May 20, 2026
- Type: Problem & solution

Retail marketing teams often store customer segments in OLTP databases, creating underused capacity between campaigns and synchronization work whenever new segments are requested. Lakebase, Databricks’ implementation of a lakebase architecture, separates storage from compute and uses elastic, serverless Postgres that can scale to zero when idle and up for bursty campaign traffic. The integration with the Lakehouse uses managed Synced Tables for customer segments and Lakehouse Sync for continuous CDC-based replication of operational data into Unity Catalog Delta tables. The SAP Engagement Cloud example configures a Lakebase Autoscaling project, native Postgres credentials, an ISRG Root X1 certificate, and snapshot synchronization for datasets with more than 10% updated. The post concludes that this design lowers idle and sizing costs, reduces pipeline maintenance, and supports low-latency point lookups, while warning that Lakebase is not optimized for large scans or classic OLAP.


### [How the D. E. Shaw group powers high-cardinality observability at scale with ClickHouse](https://yomu.fyi/post/how-the-d-e-shaw-group-powers-high-cardinality-observability-at-scale.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: ClickHouse
- Published: May 15, 2026
- Type: Explainer

Inside D. E. Shaw group, a global investment and technology development firm, millions of compute workloads run monthly across its high-performance computing infrastructure. To support capacity planning and resource allocation, it collects task-level metrics for requested and utilized compute, producing millions of distinct time series that its previous observability platform could not accommodate. Engineers benchmarked databases on identical hardware and datasets; ClickHouse ingested approximately 3.5 million samples per second on one node versus roughly 480,000 for a competing solution, while similar queries completed in seconds instead of timing out. They retained InfluxDB line protocol, batched messages into ClickHouse, and built disk-based tooling to transform and stream historical data; the resulting four-server NVMe cluster stores about 68 TB compressed and sustains 530k records per second on average. Materialized views support capacity-planning dashboards, while OpenTelemetry tracing extends the platform to other observability workloads, with SQL migration and data modeling requiring engineering effort.


### [Powering self-driving vehicle analytics at Avride with ClickHouse Cloud](https://yomu.fyi/post/powering-self-driving-vehicle-analytics-at-avride-with-clickhouse-clou.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: ClickHouse
- Published: May 11, 2026
- Type: Problem & solution

Avride’s expanding autonomous passenger vehicle and delivery robot fleet generated thousands of data points per minute, overwhelming its legacy Apache Iceberg and Parquet metadata indexing architecture. Iceberg’s optimistic concurrency model struggled with parallel writes, pushing ingestion latencies to hours or days, inflating query times to 10–20 seconds, and requiring raw data duplication across storage locations. To resolve these bottlenecks, Avride built a custom indexing and metrics warehouse powered by ClickHouse Cloud on AWS backed by Amazon S3 object storage. The new architecture dropped ingestion delays to seconds, reduced worst-case lookup latencies to under one second, eliminated the need for bespoke caching layers, and removed payload duplication. Beyond operational analytics, the unified ClickHouse platform now supports internal ride-replay debugging tools, simulation workflows, and high-cardinality vehicle C++ performance trace profiling.


### [Agentic analytics starts with query-ready data: the write-side cost of Snowflake vs. ClickHouse](https://yomu.fyi/post/agentic-analytics-starts-with-query-ready-data-the-write-side-cost-of.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: Tom Schreiber, Lionel Palacin
- Published: May 6, 2026
- Type: Benchmark

Agentic analytics workflows introduce bursts of concurrent, low-latency analytical queries over continuously ingested data. Analytical query efficiency relies on columnar storage and chunk-level pruning, which require contiguous, sorted data to avoid full-table scans. Rather than evaluating warehouses solely at query runtime, this benchmark compares the write-side cost of maintaining query-ready data in ClickHouse and Snowflake during continuous ingestion at approximately one million rows per second. Snowflake relies on post-write clustering to organize data, whereas ClickHouse constructs sorted layouts on the write path and refines them via background merges. Across 100B, 200B, and 300B row checkpoints on the ClickBench dataset, ClickHouse achieved a query-ready layout at 22 times lower cost and delivered 28 times better write-side cost-performance.


### [How nOps rebuilt their cloud optimization platform on Databricks Lakebase, and why other ISVs should too](https://yomu.fyi/post/how-nops-rebuilt-their-cloud-optimization-platform-on-databricks-lakeb.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Bryan Smith
- Published: May 5, 2026
- Type: Explainer

nOps migrated its production cloud optimization platform to Databricks Lakebase, a fully managed PostgreSQL database integrated with the Databricks Lakehouse. Its previous setup separated customer-facing operational data from analytics, requiring scheduled jobs, cron-based change-detection logic, and synchronization work that introduced lag and operational overhead, especially as workloads expanded to GCP and Azure. In the new architecture, Lakebase is the central Postgres database and single source of truth for the front end and AI infrastructure, while the Lakehouse continuously consumes its data for analytics and metric computation. Databricks Metric Views are automatically surfaced in the application, and data flows one way without direct write-back. The reported results are faster data pipelines, lower operational overhead, and a better customer experience, with auto-scaling, auto-stop, restore, OAuth, and compatibility features supporting adoption.


### [From SSH to REST: A Security-Driven Modernization of Slack’s EMR Data Pipelines](https://yomu.fyi/post/from-ssh-to-rest-a-security-driven-modernization-of-slack-s-emr-data-p.md)
- Company: [Slack](https://yomu.fyi/company/slack.md)
- Author: Mahendran Vasagam
- Published: May 5, 2026
- Type: Problem & solution

Slack's data platform accumulated over 700 SSH-based Airflow operators executing jobs directly on AWS Elastic MapReduce clusters, creating broad security surfaces and operational instability. Stateful SSH connections frequently failed when Kubernetes pods restarted, leaving orphaned zombie processes and preventing infrastructure modernization such as migrating EMR clusters to child accounts. To eliminate SSH access, Slack adopted a REST-based architecture using Quarry, a gateway providing unified REST APIs and token authentication across compute engines. For non-Hadoop CLI workloads, Slack leveraged YARN Distributed Shell to execute arbitrary shell scripts stored in Amazon S3 within managed, isolated YARN containers. The phased migration achieved zero downtime across eight data regions, unlocking reliable monitoring, simplified onboarding, and future initiatives like Spark on Kubernetes.


### [Realtime or Pipelines? How to choose the right tool](https://yomu.fyi/post/realtime-or-pipelines-how-to-choose-the-right-tool.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: eduardo\_gurgel, riccardo\_busetti
- Published: May 5, 2026
- Type: Explainer

Supabase Realtime and Supabase Pipelines both read PostgreSQL changes through logical replication, but they target different jobs. Realtime delivers live updates and communication to browsers and apps through Broadcast, Presence, and Postgres Changes; its Postgres Changes path checks Row Level Security for subscribed users, uses temporary replication slots, and offers best-effort delivery without replay after disconnects. Pipelines is a managed change-data-capture system that performs an initial sync, then replicates selected publication data in batches to supported analytical destinations through a persistent replication slot, retaining progress and retrying transient failures. The choice depends on the destination and delivery needs: use Realtime for low-latency user experiences that can tolerate gaps, Pipelines for durable analytical copies with at-least-once processing and idempotent destination writes, and both when an application needs both capabilities.


### [Gala supercharges analytics performance with ClickHouse on AWS](https://yomu.fyi/post/gala-supercharges-analytics-performance-with-clickhouse-on-aws.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: ClickHouse
- Published: May 4, 2026
- Type: Problem & solution

Blockchain gaming platform Gala struggled to ingest and query expanding volumes of telemetry and blockchain data on its Databricks-based data infrastructure, resulting in multi-minute query times and heavy engineering maintenance overhead. To solve these bottlenecks, the company migrated its data platform to ClickHouse on AWS, ingesting data through Kafka, Airbyte, Amazon S3, and Fivetran. Gala leveraged ClickHouse's indexing capabilities alongside saved queries and API endpoints to power company-wide Metabase business intelligence dashboards for non-technical teams. Following the migration completed in December 2024, data capacity expanded from 3 TB to 9 TB while query times dropped from minutes to sub-second on previously unoptimized tables. The switch also lowered initial infrastructure costs by 30 percent, with plans underway to implement ClickHouse ClickPipes for improved data processing pipelines.


### [Data Science vs Data Engineering: Choosing Analysis or Infrastructure](https://yomu.fyi/post/data-science-vs-data-engineering-choosing-analysis-or-infrastructure.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: May 1, 2026
- Type: Explainer

This guide distinguishes data engineering from data science by their roles in the data value chain. Data engineers build and maintain ingestion pipelines, warehouses, lakes, access controls, and production systems, using tools such as SQL, Python, Scala, Java, Apache Spark, Kafka, orchestration platforms, and cloud infrastructure. Data scientists work with clean, accessible data to perform exploratory analysis, engineer features, train and validate statistical or machine-learning models, and communicate findings through visualization and storytelling. The roles collaborate through feedback on data quality, shared documentation, model deployment infrastructure, and versioned schemas, feature pipelines, and experiments. For choosing a path, the guide recommends trying a two-week pipeline project and a two-week ML project; it also outlines portfolio deliverables for each and says both roles require strong analytical and computer-science fundamentals.


[Newer posts](https://yomu.fyi/topic/data-pipelines/page/5.md) · [Older posts](https://yomu.fyi/topic/data-pipelines/page/7.md)