What is data pipeline architecture?

Summary

Data pipeline architecture is the end-to-end blueprint for collecting, processing, storing and delivering data from source systems to people, applications and models. It distinguishes logical design, which defines stages and responsibilities, from physical design, which assigns tools and infrastructure to those stages. The common four layers are ingestion, processing and transformation, storage, and serving and consumption, with orchestration and observability spanning the pipeline. The brief compares batch, streaming, Lambda, Kappa and medallion patterns, explaining trade-offs involving freshness, cost, complexity and operational burden. It also contrasts ETL with ELT and presents governance, monitoring and right-sized processing as reliability practices, concluding that architecture should match the use case and balance freshness, cost and reliability.

Takeaways

1. 1

   The four common layers are ingestion, processing and transformation, storage, and serving and consumption; orchestration and observability operate across all four.

2. 2

   Batch pipelines are simpler, cheaper and easier to debug, while streaming supports sub-minute use cases such as fraud detection, real-time personalization and IoT monitoring at higher operational cost.

3. 3

   Lambda maintains parallel batch and streaming paths, whereas Kappa uses one streaming pipeline and replays it for historical analysis, reducing duplicate logic and operational burden.