---
title: "Apache Beam: Unified Batch and Stream Processing"
description: "Apache Beam is a unified programming model for batch and stream data processing. Learn how it handles event time, watermarks, and windowing to process data reliably at scale."
---

# Apache Beam: Unified Batch and Stream Processing

**Type:** [How it works](https://yomu.fyi/visual-explainers/category/how-it-works.md)

![Five-stage pipeline diagram showing Apache Beam's processing flow: data ingestion with timestamp extraction, windowing and grouping by time, watermark advancement to signal completeness, stateful aggregation and joins, and output to sinks.](https://yomufyi.s3.us-east-1.amazonaws.com/visual-explainers/19afff24-b5bc-483f-b5f9-776e9e392ef6/image-29c2aa48-6db8-40c6-9807-cf56edcd31aa.webp)

> Apache Beam is a unified programming model for batch and stream data processing. Learn how it handles event time, watermarks, and windowing to process data reliably at scale.

## Explanation

Apache Beam abstracts batch and stream processing into a single API, letting engineers write pipelines once and run them on multiple backends like Spark, Flink, or Google Cloud Dataflow. The core challenge in streaming is handling out-of-order events, late arrivals, and the gap between event time (when data was created) and processing time (when the system sees it). Beam solves this through windowing, which groups events by time, and watermarks, which signal when a window is complete. Understanding these concepts is essential for building reliable real-time systems that correctly aggregate clickstream signals, market data, or any time-sensitive workflow.

**Tags:** [Apache Beam](https://yomu.fyi/topic/apache-beam.md), [Data Engineering](https://yomu.fyi/topic/data-engineering.md), [Data Pipelines](https://yomu.fyi/topic/data-pipelines.md), [Distributed Systems](https://yomu.fyi/topic/distributed-systems.md), [Streaming](https://yomu.fyi/topic/streaming.md)

## Related reading

### [Pipelined SQL in ClickHouse 26.8](https://yomu.fyi/post/pipelined-sql-in-clickhouse-26-8.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: Sep 4, 2026
- Type: Explainer

ClickHouse 26.8 introduces pipelined SQL, extending the earlier ability to place FROM before SELECT with a |> operator for sequencing query transformations. Using the UK property prices dataset, the post compares conventional, FROM-first, and pipelined forms for filtering London sales, aggregating median prices by district, ordering results, and limiting output. Each pipeline stage acts as a complete-query checkpoint, while EXPLAIN SYNTAX shows that ClickHouse translates the syntax into nested standard SQL and optimizes the complete query without materializing every intermediate result. The syntax also supports EXTEND, reuse of aggregation aliases in later stages, views, and INSERT ... SELECT statements, though stage order affects which rows feed subsequent aggregations in these examples; the conclusion is that pipelines complement rather than replace conventional SQL.


### [Build a real-time market data app with ClickHouse and Massive](https://yomu.fyi/post/build-a-real-time-market-data-app-with-clickhouse-and-massive.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: Sep 4, 2026
- Type: Tutorial

The post presents a real-time market-data application that ingests trade and quote ticks from Massive and stores them in ClickHouse for low-latency querying and visualization. It distinguishes quotes, which contain bid and ask prices and sizes, from trades, which record executed price, size, timestamps, and related fields. The implementation uses an authenticated Massive WebSocket subscription, Node.js ingestion, separate ClickHouse MergeTree tables ordered by symbol and one-minute timestamp buckets, and React charts. It also describes materialized-view aggregation for visualization and supplies a runnable repository with setup, build, start, admin, and synthetic-seed paths; the conclusion recommends durable delivery, replay, authentication, and workload-specific tuning before production use.


### [What is Data Transformation?](https://yomu.fyi/post/what-is-data-transformation.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Sep 3, 2026
- Type: Explainer

Data transformation turns raw data extracted from source systems into usable, trusted datasets for operational needs, analytics and decision-making. The process addresses inconsistencies, missing values and duplicates through techniques such as cleansing, de-duplication, integration, aggregation, validation and format revision, often mapping data to a target schema during ETL. Performance considerations include compaction, partitioning, file-size tuning, data skipping and pruning, and caching to reduce storage, scanning, processing time and latency. The article also compares on-premises, cloud, open-source, proprietary, no-code, low-code and codable ETL tools, then describes Spark Declarative Pipelines and Lakeflow Jobs on Databricks for quality checks, pipeline development, monitoring and orchestration. A Navy Federal Credit Union proof of concept illustrates a rapid deployment, with a proof of concept completed in a week and a dashboard released days after production deployment.


### [Apache Beam for Search: Getting Started by Hacking Time](https://yomu.fyi/post/apache-beam-for-search-getting-started-by-hacking-time.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jan 8, 2021
- Type: Tutorial

Search relevance can use clickstream signals such as clicks and purchases to promote popular results and demote results users do not favor, but batch processing can delay those adjustments by days. The Discovery team at Shopify uses Apache Beam, a unified batch and stream processing system, to support real-time and historical search-behavior workflows, including boosting, popularity aggregates, offline evaluation sets, and reinforcement learning tasks. The article’s central lesson is that Beam requires careful handling of event time, processing time, delayed events, out-of-order data, and watermarks, especially when combining streams. A Kafka source can use Kafka processing time, client create time with a five-minute out-of-order allowance, or a custom timestamp policy that reads the SearchQueryEvent.searchTimestamp field; Beam windows add further buffering and lateness controls. Accurate timestamping is presented as the first milestone for replaying clickstream data reliably before tackling more complex operations such as streaming joins.