Apache Beam: Unified Batch and Stream Processing
Apache Beam is a unified programming model for batch and stream data processing. Learn how it handles event time, watermarks, and windowing to process data reliably at scale.
Loading post…
ClickhouseTutorial
Build a real-time market data app with ClickHouse and Massive
Summary
The post presents a real-time market-data application that ingests trade and quote ticks from Massive and stores them in ClickHouse for low-latency querying and visualization. It distinguishes quotes, which contain bid and ask prices and sizes, from trades, which record executed price, size, timestamps, and related fields. The implementation uses an authenticated Massive WebSocket subscription, Node.js ingestion, separate ClickHouse MergeTree tables ordered by symbol and one-minute timestamp buckets, and React charts. It also describes materialized-view aggregation for visualization and supplies a runnable repository with setup, build, start, admin, and synthetic-seed paths; the conclusion recommends durable delivery, replay, authentication, and workload-specific tuning before production use.
Context
Financial tick applications must ingest high-frequency trade and quote streams, potentially thousands of events per second across multiple symbols, while supporting low-latency queries. The post notes that even a few seconds of delay can affect trading outcomes and that WebSockets avoid the latency and overhead of repeatedly polling a REST API.
Approach / What changed
The example authenticates to Massive's stocks WebSocket endpoint, subscribes to trade and quote channels, and processes events with Node.js. ClickHouse stores trades and quotes in separate MergeTree tables ordered by symbol and one-minute timestamp buckets, while materialized views support aggregation for React visualizations. The repository includes setup, build, startup, administration, and synthetic-data paths.
Takeaways
Quotes and trades are modeled separately: quotes retain bid and ask prices and sizes, while trades retain executed price, size, timestamps, condition data, and trade-related fields.
The ClickHouse schemas use LowCardinality symbols, numeric timestamp and sequence fields, UInt32 arrays for conditions or indicators, and one-minute bucketing in the ORDER BY key.
The example warns that the local service has unauthenticated APIs and recommends adding access controls, durable delivery, replay, and workload-specific tuning before production use.
Source and freshness
- Source
- clickhouse.com
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
See the idea
Visual explainers
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.