---
title: "Data Pipelines"
description: "248 posts about Data Pipelines, summarised, each linking to the original."
---

# Data Pipelines
> 248 posts about Data Pipelines, summarised, each linking to the original.

## Articles

### [June 2026 newsletter](https://yomu.fyi/post/june-2026-newsletter.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: Jun 18, 2026
- Type: Announcement

ClickHouse’s June 2026 newsletter marks the project’s ten years as open source, noting more than 2,000 contributors and community projects alongside engineering updates. It profiles Vasily Chekalkin’s 26.5 improvements to WebAssembly user-defined functions, including the DETERMINISTIC keyword and visibility in system.functions, and covers native Jaeger storage, the ClickCannon benchmark tool, and Open House announcements. The 26.5 release reports Top-N optimizations that make ORDER BY … LIMIT through JOIN 20.4× faster with 175× less memory on TPC-H SF100, while GROUP BY … LIMIT is 11.9× faster with 185× less peak memory. Other featured systems include a 15-million-rows-per-hour KPI pipeline, join-engineering work yielding a 26× improvement on join-heavy TPC-H workloads, and community guidance on inserts, columnar storage, ArrowStream, and RAG ingestion.


### [GCP Pub/Sub connector for ClickPipes is now in Private Preview](https://yomu.fyi/post/gcp-pub-sub-connector-for-clickpipes-is-now-in-private-preview.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: Jun 17, 2026
- Type: Announcement

ClickHouse has announced a GCP Pub/Sub connector for ClickPipes, now in Private Preview, to stream GCP infrastructure logs, application events, and operational signals into ClickHouse Cloud in real time. Previously, routing data from Pub/Sub into ClickHouse required GCS or a Dataflow pipeline; the connector instead manages subscription lifecycle, schema inference, and message delivery after a topic and target table are configured. It supports JSON, Avro, and Protobuf, Pub/Sub Schema Registry integration, automatic decompression, attribute-based filtering, and start positions of latest, earliest, or a specific timestamp. ClickPipes supports horizontal and vertical scaling, configurable replicas, automatic retries, built-in metrics and logs, and programmatic management through OpenAPI and the ClickHouse Terraform Provider. The connector is available upon request and free during Private Preview, with access offered through a waitlist or a ClickHouse Cloud account manager.


### [Data Warehouse Modernization: Roadmap, Architecture, and Services](https://yomu.fyi/post/data-warehouse-modernization-roadmap-architecture-and-services.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Jun 17, 2026
- Type: Explainer

Data warehouse modernization addresses legacy systems that cannot scale efficiently with growing data volumes, real-time analytics, machine learning, and self-service access. The proposed roadmap spans two to four years for large estates, moving from assessment and architecture design through high-impact workload migration, governance embedding, and optimization rather than relying on a risky big-bang cutover. Its target architecture favors a lakehouse or enhanced cloud data warehouse, with open formats such as Apache Iceberg or Delta Lake, separate compute and storage, and Bronze, Silver, and Gold layers supporting incremental ELT and lineage. The source says modernization can reduce infrastructure maintenance costs by 30–50%, compress query latency from hours to seconds, and halve redundant ETL pipelines, while also improving governance for sensitive data and enabling BI, machine learning, and generative AI workloads on a shared foundation.


### [What’s coming next to Free Edition](https://yomu.fyi/post/what-s-coming-next-to-free-edition.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Will Valori, Trang Le
- Published: Jun 17, 2026
- Type: Announcement

Databricks is expanding Free Edition, launched twelve months ago to help people learn modern data and AI, after providing more than $10M in credits to over 500,000 learners. The five additions are Genie Code, serverless GPUs subject to availability, Lakebase, Agent Bricks, and Lakeflow Designer. Genie Code can generate, run, interpret, and refine code autonomously for tasks such as dataset analysis, pipeline cleaning, and visualization, while GPUs support deep-learning workloads including training, fine-tuning, inference, and large-dataset processing. Lakebase adds a fully managed Postgres-compatible database for data apps and AI agents; Agent Bricks supplies composable tools, memory, orchestration, and evaluation; and Lakeflow Designer enables visual pipeline construction. Together, these additions are presented as a complete free toolkit covering data engineering, analytics, machine learning, app development, and AI agents, allowing users to build end-to-end data and AI projects in Free Edition.


### [92x faster queries: How Open Electricity uses ClickHouse to track Australia’s energy transition in real time](https://yomu.fyi/post/92x-faster-queries-how-open-electricity-uses-clickhouse-to-track-austr.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: Jun 17, 2026
- Type: Problem & solution

Open Electricity uses ClickHouse to serve live Australian energy data through a public platform and API spanning roughly one billion records. Migrating from Postgres and TimescaleDB addressed queries that could take more than five minutes, reducing a benchmark to just over three seconds and to 60 milliseconds with a materialized view. The new setup keeps heavy relational workloads in Postgres while ClickHouse handles analytics, using ReplacingMergeTree base tables and more than 20 materialized views for different aggregation patterns. It also uses substantially less hardware than the previous configuration: four cores, four GB RAM, and 200 GB disk versus 16 cores, 32 GB RAM, and 1.6 TB, while serving data added daily and API users.


### [What is data pipeline architecture?](https://yomu.fyi/post/what-is-data-pipeline-architecture.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Jun 16, 2026
- Type: Explainer

Data pipeline architecture is the end-to-end blueprint for collecting, processing, storing and delivering data from source systems to people, applications and models. It distinguishes logical design, which defines stages and responsibilities, from physical design, which assigns tools and infrastructure to those stages. The common four layers are ingestion, processing and transformation, storage, and serving and consumption, with orchestration and observability spanning the pipeline. The brief compares batch, streaming, Lambda, Kappa and medallion patterns, explaining trade-offs involving freshness, cost, complexity and operational burden. It also contrasts ETL with ELT and presents governance, monitoring and right-sized processing as reliability practices, concluding that architecture should match the use case and balance freshness, cost and reliability.


### [Build Your Own Transaction Foundation Model for Financial Intelligence](https://yomu.fyi/post/build-your-own-transaction-foundation-model-for-financial-intelligence.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Benjamin Wu
- Published: Jun 16, 2026
- Type: Tutorial

Production financial intelligence systems frequently rely on hand-engineered tabular features and rule sets that are brittle, expensive to maintain, and blind to historical sequential patterns. NVIDIA demonstrates an accelerated reference pipeline that replaces standard BPE tokenization with a GPU-based domain tokenizer, converting raw transactions into semantic tokens with an 8,192-token context window. Using the NeMo AutoModel library, a compact 29-million parameter decoder-only transformer is pretrained from scratch on unlabeled transaction sequences using causal language modeling. Learned sequence embeddings are extracted, compressed using PCA, and concatenated with raw tabular features to train a downstream GPU-accelerated XGBoost fraud detection model. On the IBM TabFormer benchmark dataset, this combined approach achieves a 41.76% lift in Average Precision over the baseline model relying solely on raw tabular features.


### [Teaching Sidekick to say no: automated data curation with LLM judge consensus](https://yomu.fyi/post/teaching-sidekick-to-say-no-automated-data-curation-with-llm-judge-con.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jun 15, 2026
- Type: Problem & solution

Sidekick’s customer-segmentation skill was fine-tuned on tens of thousands of successful, de-identified production queries, so it generated zero-result queries instead of refusing impossible requests such as finding customers by occupation. A small Toloka dataset of roughly 600 standard queries and 602 refusal annotations became a seed for four frontier LLM judges, which were calibrated against human-labeled examples and required to agree on both verdict and reasoning. The pipeline used mutually exclusive categories to resolve conflicting labels, filtered disagreements for human review, and fed accepted examples from later production traffic back into fine-tuning. With automated curation, the segmentation pass rate rose from 0.762 to 0.798 versus naive merging, while manual validation measured 86.3% refusal accuracy and a 4.6% false-positive rate. The resulting flywheel is being applied to other Sidekick skill models and data-quality problems.


### [What is enterprise intelligence?](https://yomu.fyi/post/what-is-enterprise-intelligence.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Jun 12, 2026
- Type: Explainer

Enterprise intelligence (EI) is presented as an organization-wide capability combining business intelligence, knowledge management, enterprise search and AI to turn structured and unstructured data into decisions and actions. Unlike traditional BI, which centers on dashboards, reports and structured data, EI connects these capabilities through a shared architecture and governed business context. The described stack includes a lakehouse-based data foundation, batch and streaming pipelines, governance, semantics, analytics, search, machine learning, generative AI and a decision layer. Shared definitions such as “active customer” and “monthly revenue” are intended to keep dashboards, queries and AI agents aligned, while maintained context addresses knowledge that becomes stale as the business changes. The result described is a common trusted source from which people, applications and agents can produce insights and initiate actions.


### [Forward Deployed Engineering: Delivering Business Outcomes with AI](https://yomu.fyi/post/forward-deployed-engineering-delivering-business-outcomes-with-ai.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Jason Martin
- Published: Jun 11, 2026
- Type: Announcement

Databricks is formalizing its Forward Deployed Engineering (FDE) organization to address customers’ shift from migration and data-pipeline requests toward business outcomes with AI. FDE brings Professional Services together around an engineering-led model that embeds engineers with customers, supports modernization and production AI, and works with partners and Databricks R&D. In the cited examples, teams migrated five-plus petabytes of JPMC Consumer and Community Banking Risk data and more than 500 notebooks in four months, while Fox used Lakebase, AI Search, Databricks Apps, and Model Serving to redesign fan experiences. The organization says its engagements use shared OKRs, rapid prototype-to-production delivery, embedded engineering, outcome-aligned commercial options, and global partner coverage, with Fox reporting that Sports AI users spend approximately twice as long in the app.


### [Metric Semantic Layer: How Lyft Governs and Scales Key Data Definitions](https://yomu.fyi/post/metric-semantic-layer-how-lyft-governs-and-scales-key-data-definitions.md)
- Company: [Lyft](https://yomu.fyi/company/lyft.md)
- Author: Iraklikhorguani
- Published: Jun 10, 2026
- Type: Problem & solution

As Lyft scaled, different teams developed conflicting definitions for key business metrics due to the lack of centralized version control and shared standards. To resolve this, Lyft implemented an internal Metric Semantic Layer as a Python package that stores authoritative metric definitions in YAML files with Jinja SQL templates. The system restricts onboarding to Golden Metrics used across multiple applications, requiring team-based approval from both Business and Operational Owners for any definition changes. Standardized definitions are exposed through Python APIs, integrated into the Amundsen data catalog and self-service user interfaces, and surfaced to AI tools via a Model Context Protocol.


### [Datadog and ClickHouse partner to bring full-fidelity data to modern observability](https://yomu.fyi/post/datadog-and-clickhouse-partner-to-bring-full-fidelity-data-to-modern-o.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: Jun 10, 2026
- Type: Announcement

ClickHouse and Datadog announce a partnership aimed at reducing the tradeoff between retaining complete telemetry and keeping it accessible through familiar observability workflows. New Preview capabilities let organizations route logs directly to ClickHouse through Datadog Observability Pipelines and search those logs from the Datadog Log Explorer. Pipelines can parse, enrich, filter, transform, and redact data before storage, while ClickHouse Cloud’s separation of storage and compute supports long-term retention, independently scaled query resources, and workload isolation. Federated log search queries data directly in ClickHouse, avoiding duplicate copies or re-ingestion while allowing high-volume logs to remain searchable and historical investigations to use full-fidelity telemetry. The partnership combines Datadog’s collection and investigation experience with ClickHouse’s storage and analytics foundation; the native integration and federated search are available in Preview.


### [Encoding Your Domain Expert: The Context Layer Behind Spotify's Data Assistant](https://yomu.fyi/post/encoding-your-domain-expert-the-context-layer-behind-spotify-s-data-as.md)
- Company: [Spotify](https://yomu.fyi/company/spotify.md)
- Author: Spotify Engineering
- Published: Jun 10, 2026
- Type: Problem & solution

Spotify developed an AI data assistant, Vedder, to scale access to over 70,000 internal datasets without overwhelming domain experts. Because raw warehouse schemas omit critical semantic nuances and overwhelm LLM context windows, engineers implemented a domain-owned context layer organized into clusters. Each cluster contains profiled datasets, expert-curated business documentation, and vetted question-and-SQL pairs that guide a ReAct-based agent. Automated query history extraction proved insufficient on its own, as domain curators accepted only 12.5% of inferred query pairs due to noise. Deployed across Slack, IDEs, and a dedicated web interface, the system maintains reliability through continuous cluster health monitoring that flags schema drift and degraded examples.


### [Scaling beyond one: How Airbnb evolved its data architecture for a multi-product world](https://yomu.fyi/post/scaling-beyond-one-how-airbnb-evolved-its-data-architecture-for-a-mult.md)
- Company: [Airbnb](https://yomu.fyi/company/airbnb.md)
- Author: Patrick Lam
- Published: Jun 9, 2026

To support its expansion from Homes into Experiences and Services, Airbnb evolved its offline data warehouse architecture to balance domain-specific needs with organizational consistency. Rather than enforcing a single global pattern, data teams adopted a framework of foundational principles that paired separate data models for unique product features with monolithic models for cross-cutting domains like payments and messaging.


### [How QRT powers real-time research and risk management at petabyte scale](https://yomu.fyi/post/how-qrt-powers-real-time-research-and-risk-management-at-petabyte-scal.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: Jun 9, 2026
- Type: Explainer

Qube Research & Technologies (QRT), a global quantitative investment manager headquartered in London, uses ClickHouse Cloud for a data platform that serves researchers and for near-real-time risk monitoring, management, and P&L. Founded in 2018 as a cloud-native firm after the spinout of Credit Suisse’s proprietary trading arm, QRT chose a cloud-based approach rather than inheriting decades of on-premise infrastructure. The platform supports two major systems, while ClickStack consolidates logs, metrics, and traces for observability; alerts, log queries, and metric aggregations run against a single ClickHouse instance using standard SQL. QRT says its previous database was constrained by writer and reader counts and total cluster load, whereas ClickHouse provides greater speed and scalability, and that a unified monitoring backend simplifies maintenance while remaining reliably fast.


### [Transforming solar and wind maintenance reports with Genie and AI agents](https://yomu.fyi/post/transforming-solar-and-wind-maintenance-reports-with-genie-and-ai-agen.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Maria Vallarelli
- Published: Jun 8, 2026
- Type: Problem & solution

Plenitude and Databricks built an agent-based system that turns solar and wind plant maintenance PDFs into structured data for cross-plant analysis. Event-driven ingestion uses Databricks Jobs and the ai\_parse\_document AI Function to extract text, tables, figures, and metadata, then stores page- and object-level JSON records in Delta Lake with coordinates, version history, and links to source reports. A Genie space uses Unity Catalog metadata, knowledge-store instructions, and SQL generation to answer natural-language questions, produce visualizations, and export results, while Agent Bricks can orchestrate multi-step workflows and downstream actions. The design also applies automatic liquid clustering to dynamic queries and row-level security to restrict results by country. The resulting data layer supports historical trends, plant comparisons, recurring-fault analysis, and a foundation for predictive maintenance, although the source frames predictive use as a future improvement.


### [Enterprise Data Strategy Roadmap for Business Outcomes](https://yomu.fyi/post/enterprise-data-strategy-roadmap-for-business-outcomes.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Jun 8, 2026
- Type: Explainer

An enterprise data strategy connects organizational data assets to measurable business outcomes, while fragmented architectures can leave data investments uncoordinated and limit real-time analysis and action. The roadmap starts with purpose, scope, executive sponsorship, measurable objectives, KPI mapping, and use-case prioritization based on business impact, feasibility, time to value, and organizational readiness. It then organizes governance, lifecycle management, data quality, target-state architecture, integration, analytics, team structure, compliance, and measurement as interdependent capabilities, emphasizing owners, stewards, decision rights, executable quality rules, and automated cleansing. Implementation proceeds through a time-boxed cross-functional pilot, documented learnings, and incremental scaling, with steering-committee oversight and governance that evolves through feedback. The text gives indicative timelines of 60 to 90 days for a focused pilot, 12 to 18 months for a foundational platform across multiple business units, and multiple years for a mature data-driven culture.


### [Scaling up your microservice testing with Apache Spark—Part 2](https://yomu.fyi/post/scaling-up-your-microservice-testing-with-apache-spark-part-2.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: Vivek Yadav
- Published: Jun 8, 2026
- Type: Tutorial

Part 2 describes how to build a repeatable Apache Spark replay harness for testing high-impact microservices against historical, production-shaped inputs without reimplementing the service. The design turns dependency state—such as database snapshots, versioned rules, merchant configuration, eligibility state, and logged service responses—into datasets, while representing side effects as output rows and matching fidelity to the decision under test. A thin Spark wrapper reconstructs ReplayInput objects, invokes the same deterministic decision engine used online, compares current and candidate results, and writes traceable diffs with intermediate values and rule explanations. The source cautions that incomplete context, distributed state, asynchronous workflows, and external actors limit replay’s predictive power, so it should model explicit assumptions rather than claim full production simulation. It presents migrations as a practical point to establish these boundaries and use replay to reduce risk.


### [The AI and Data Transformation Playbook for Enterprise Teams](https://yomu.fyi/post/the-ai-and-data-transformation-playbook-for-enterprise-teams.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Jun 5, 2026
- Type: Explainer

The playbook presents AI and data transformation as an enterprise capability built on trustworthy, well-governed data, rather than a model-only project. It explains how governance, scalable pipelines, and continuous quality monitoring address incompatible formats, missing values, duplicate records, and schema inconsistencies from systems such as CRM platforms, operational databases, IoT sensors, and cloud applications. Operational guidance covers KPI design, named pipeline ownership, review checkpoints, data mapping and lineage, automated de-duplication, deterministic enrichment, regulatory classification, and role-based access. It states that deterministic tests and human review remain necessary for AI-assisted code, while telemetry and quarterly audits help detect reliability, governance, and data quality degradation. The concluding position is that operational discipline—not model sophistication—supports reliable machine learning, predictive analytics, and generative AI outcomes.


### [Your guide to the Telecommunications Industry Experience at Data and AI Summit 2026](https://yomu.fyi/post/your-guide-to-the-telecommunications-industry-experience-at-data-and-a.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Elena Tesser, Nevash Pillay
- Published: Jun 4, 2026
- Type: Announcement

Data + AI Summit 2026 presents a Telecommunications Industry Experience for operators responding to surging network traffic, regulatory pressure, cybersecurity threats, competition, and customer churn. The event, scheduled for June 15–18 in San Francisco, positions unified data and AI, governed workflows, and production use cases as the basis for operationalized, AI-native telecom models. Its June 17 Telecommunications Industry Forum features keynotes, presentations, and executive panels on customer experience, autonomous network operations, fraud prevention, secure agent deployment, and the return from modernizing legacy data warehouses. Breakout sessions cover automated metadata generation for Genie, conversational AI/BI, Lakeflow pipelines with Agent Bricks, and data exfiltration protection with egress monitoring. The industry lounge will demonstrate Model as a Service and agentic real-time decisioning, while the agenda emphasizes peer examples and architectural blueprints for scaling AI under telecom governance and compliance.


[Newer posts](https://yomu.fyi/topic/data-pipelines/page/4.md) · [Older posts](https://yomu.fyi/topic/data-pipelines/page/6.md)