---
title: "What is data pipeline architecture?"
description: "Data pipeline architecture is the end-to-end blueprint for collecting, processing, storing and delivering data from source systems to people, applications and models. It distinguishes logical design, which defines stages and responsibilities, from physical design, which assigns tools and infrastructure to those stages. The common four layers are ingestion, processing and transformation, storage, and serving and consumption, with orchestration and observability spanning the pipeline. The brief compares batch, streaming, Lambda, Kappa and medallion patterns, explaining trade-offs involving freshness, cost, complexity and operational burden. It also contrasts ETL with ELT and presents governance, monitoring and right-sized processing as reliability practices, concluding that architecture should match the use case and balance freshness, cost and reliability."
---

# What is data pipeline architecture?

[Databricks](https://yomu.fyi/company/databricks) · Databricks Staff · Jun 16, 2026

**Type:** Explainer

## Summary

Data pipeline architecture is the end-to-end blueprint for collecting, processing, storing and delivering data from source systems to people, applications and models. It distinguishes logical design, which defines stages and responsibilities, from physical design, which assigns tools and infrastructure to those stages. The common four layers are ingestion, processing and transformation, storage, and serving and consumption, with orchestration and observability spanning the pipeline. The brief compares batch, streaming, Lambda, Kappa and medallion patterns, explaining trade-offs involving freshness, cost, complexity and operational burden. It also contrasts ETL with ELT and presents governance, monitoring and right-sized processing as reliability practices, concluding that architecture should match the use case and balance freshness, cost and reliability.

## Context

Data pipeline architecture determines how data moves from where it is created to where it becomes useful. The appropriate design varies by use case, such as nightly reporting versus real-time fraud detection, and must account for latency requirements, data volume, downstream use, freshness, cost and reliability.

## Approach / What changed

The text explains pipeline architecture through logical and physical design, four core layers, alternative stage models, common architectural patterns, and the ETL-versus-ELT transformation order. It also covers cross-cutting orchestration, observability and governance practices, including the use of batch or streaming according to freshness needs.

## Takeaways

- The four common layers are ingestion, processing and transformation, storage, and serving and consumption; orchestration and observability operate across all four.
- Batch pipelines are simpler, cheaper and easier to debug, while streaming supports sub-minute use cases such as fraud detection, real-time personalization and IoT monitoring at higher operational cost.
- Lambda maintains parallel batch and streaming paths, whereas Kappa uses one streaming pipeline and replays it for historical analysis, reducing duplicate logic and operational burden.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [Data Pipelines](https://yomu.fyi/topic/data-pipelines), [Lakehouse](https://yomu.fyi/topic/lakehouse), [Streaming](https://yomu.fyi/topic/streaming)

- Source: [Databricks](https://www.databricks.com/blog/what-is-data-pipeline-architecture)
- Source URL: https://www.databricks.com/blog/what-is-data-pipeline-architecture
- Ingested by Yomu: 2026-08-30T17:01:43.490Z

[Read original post](https://www.databricks.com/blog/what-is-data-pipeline-architecture)