---
title: "Streaming"
description: "123 posts about Streaming, summarised, each linking to the original."
---

# Streaming
> 123 posts about Streaming, summarised, each linking to the original.

## Articles

### [ClickHouse Release 26.2](https://yomu.fyi/post/clickhouse-release-26-2.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: ClickHouse
- Published: Mar 16, 2026
- Type: Announcement

ClickHouse version 26.2 introduces twenty-five new features, forty-three performance optimizations, and one hundred eighty-three bug fixes across the database engine. This release graduates both the text-index and the QBit vector embedding data type to production-ready status for search and retrieval workloads. For streaming ingestion pipelines where row arrival rates are low, the engine adds time-based block flushing configurations to write data parts before hitting default row or byte thresholds. The update also embeds the ClickStack observability user interface directly into the standard binary distribution, accessible through the local HTTP server port. Further enhancements include automatic min-max indexing for temporal columns in MergeTree tables, interactive time-based one-time password authentication in the client, and a high-performance prime number generator function.


### [Unordered mode for GCS ClickPipes is now available](https://yomu.fyi/post/unordered-mode-for-gcs-clickpipes-is-now-available.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: Marta Paes
- Published: Mar 16, 2026
- Type: Announcement

ClickHouse Cloud has added unordered mode to its Google Cloud Storage (GCS) ClickPipes connector, enabling continuous data ingestion regardless of file naming or arrival sequence. Standard GCS ClickPipes rely on polling every 30 seconds and require files to arrive in strict lexicographical order, which causes out-of-order data and backfills to be ignored. Unordered mode replaces polling by consuming OBJECT\_FINALIZE notifications from Google Cloud Pub/Sub whenever objects land in a bucket. State tracking is managed through a metadata store, and the connector automatically reprocesses batches upon failure while maintaining exactly-once insertion guarantees. Setup requires configuring a Pub/Sub topic and bucket notifications via tools like gcloud, followed by configuring the ClickPipe via the console or Terraform provider.


### [How Critical Manufacturing uses ClickHouse to bring real-time intelligence to the factory floor](https://yomu.fyi/post/how-critical-manufacturing-uses-clickhouse-to-bring-real-time-intellig.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: ClickHouse
- Published: Mar 10, 2026
- Type: Problem & solution

Critical Manufacturing faced performance and scaling bottlenecks while processing millions of factory floor events using a legacy Microsoft SQL Server analytics stack. Increasing data volumes caused historical queries to take hours, straining real-time dashboards and escalating operational maintenance. The engineering team replaced this architecture by migrating analytics workloads to ClickHouse connected directly to streaming Kafka topics. They implemented an ELT pattern alongside ReplacingMergeTree engines to deduplicate late-arriving sensor records and utilized TTL policies for automated lifecycle management. The resulting platform delivers sub-second analytical queries across historical and real-time operational data, expands into OpenTelemetry observability storage, and maintains embeddings for retrieval-augmented generation workloads.


### [What Replo learned optimizing 100+ billion events in ClickHouse](https://yomu.fyi/post/what-replo-learned-optimizing-100-billion-events-in-clickhouse.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: ClickHouse
- Published: Mar 9, 2026
- Type: Problem & solution

Replo built an analytics pipeline on ClickHouse to serve more than 4,000 Shopify merchants processing 3,000 to 5,000 events per second. The initial single-table architecture recalculated session metrics on the fly, leading to poor query efficiency as volume scaled. To speed up dashboard queries, the team implemented a mark-and-unmark pattern with SummingMergeTree and refreshable materialized views, but real-time deduplication and fractional attribution caused runaway query backlog loops. Replo resolved the issue by limiting real-time recomputations to a specialized table holding only the last 40 minutes of purchase-related events. This architecture allowed processing over 100 billion records while keeping query times fast and ingestion lag to approximately one minute.


### [Introducing Waypoint-1: Real-time interactive video diffusion from Overworld](https://yomu.fyi/post/introducing-waypoint-1-real-time-interactive-video-diffusion-from-over.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Andrew Lapp, Louis Castricato, Scott Fox, Shahbuland Matiana, David Rossi
- Published: Jan 20, 2026
- Type: Announcement

Overworld introduced Waypoint-1, a real-time interactive video diffusion model controllable through text prompts, mouse movements, and keyboard inputs. The architecture uses a frame-causal rectified flow transformer trained on compressed latent representations across 10,000 hours of video game footage paired with control signals. Pre-training uses diffusion forcing with causal attention, while post-training incorporates self forcing via DMD to mitigate autoregressive rollout error accumulation, enable one-pass classifier-free guidance, and support few-step denoising. To achieve low latency on consumer hardware, Overworld developed WorldEngine, a pure Python streaming inference library. Utilizing AdaLN feature caching, static rolling key-value caching with Flex Attention, fused matrix multiplications, and PyTorch compilation, the 2.3B parameter Waypoint-1-Small model reaches up to 60 frames per second on an RTX 5090 GPU.


### [Open Responses: What you need to know](https://yomu.fyi/post/open-responses-what-you-need-to-know.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: shaun smith, ben burtenshaw, merve, Pedro Cuenca
- Published: Jan 15, 2026
- Type: Announcement

Open Responses is a new open inference standard initiated by OpenAI, developed by the open source AI community, and backed by Hugging Face. While agentic workloads require multi-step planning and tool execution, the ecosystem has remained constrained by the legacy Chat Completion format and proprietary Responses implementations. Open Responses addresses this mismatch by standardizing semantic event streaming, provider-specific configuration options, and distinct roles for model providers and intermediary routers. The specification formalizes reasoning visibility across raw traces, protected encrypted content, and sanitized summaries while supporting provider-managed sub-agent execution loops. Developers can now orchestrate multi-step tool calls and stream reasoning deltas across heterogeneous providers through a single consistent API schema.


### [Behind the build: 2025's BFCM globe in a pinball machine 🌎🕹️](https://yomu.fyi/post/behind-the-build-2025-s-bfcm-globe-in-a-pinball-machine.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Dec 3, 2025
- Type: Explainer

The 2025 BFCM live globe was implemented inside a browser-based pinball machine to represent real-time merchant sales and celebrate milestones. Built with three.js and react-three-fiber, the experience runs at 120fps, uses Server-Sent Events for order arcs and live metrics, includes a 128x32 shader-driven dot-matrix display, and supports AR through react-three/xr. For performance, the team simulated pinball physics in 2D with Rapier2D and used raycasts for ramps and elevation, reducing the physics step from 2ms to 0.2ms. The scene also includes interactive mini games, infrastructure statistics, Easter eggs, and milestone-triggered reactions, while Apache FlinkSQL pipelines process event streams behind the commerce visualizations. The globe was also displayed on the Exosphere in Las Vegas for the third consecutive year.


### [Streaming datasets: 100x More Efficient](https://yomu.fyi/post/streaming-datasets-100x-more-efficient.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Andres Marafioti, Quentin Lhoest, ben burtenshaw, Pedro Cuenca, merve
- Published: Oct 27, 2025
- Type: Problem & solution

Training machine learning models on multi-terabyte datasets often requires long download times or suffers from network request storms when multiple DataLoader workers initialize independently. Hugging Face revamped the backend of its datasets library to make dataset streaming viable for large-scale distributed training without modifying the standard API. Startup latency was reduced by caching resolved data file lists across workers and bundling API calls, cutting initial request volume up to one hundred times. The team also implemented background prefetching for Parquet files and added configurable buffering parameters to keep GPUs fully saturated during training runs. Benchmark tests demonstrated up to twice the sample processing throughput and zero worker crashes across 256 concurrent workers on 64 H100 GPUs.


### [Endian Communication Systems and Information Exchange in Bytes](https://yomu.fyi/post/endian-communication-systems-and-information-exchange-in-bytes.md)
- Company: [MongoDB](https://yomu.fyi/company/mongodb.md)
- Author: Nitish Joshi
- Published: Sep 25, 2025
- Type: Explainer

Endianness dictates the order in which multi-byte numbers are stored and transmitted across systems. While big-endian formats store the most significant byte first at the lowest address, little-endian systems place the least significant byte first. Communication across diverse architectures relies on consistent byte ordering to prevent critical interpretation errors in fields such as telecommunications, IoT, automotive, finance, and aerospace. To safely process disparate payloads, modern ingestion pipelines deploy an endian converter to normalize raw binary data into a unified format. These normalized data streams flow through Apache Kafka, undergo real-time processing and enrichment via Atlas Stream Processing, and persist in MongoDB Atlas for downstream analytics and visualization.


### [Celebrating Excellence: MongoDB Global Partner Awards 2025](https://yomu.fyi/post/celebrating-excellence-mongodb-global-partner-awards-2025.md)
- Company: [MongoDB](https://yomu.fyi/company/mongodb.md)
- Author: Olivier Zieleniecki
- Published: Sep 18, 2025
- Type: Announcement

MongoDB announced the recipients of its 2025 Global Partner Awards, recognizing cloud providers, systems integrators, and technology vendors for driving enterprise modernization and artificial intelligence adoption. Microsoft earned Global Cloud Partner of the Year for joint integrations linking MongoDB Atlas on Azure with native Microsoft services across healthcare, telecommunications, and financial services. Amazon Web Services received the Global AI Cloud Partner award, highlighted by a joint deployment with Novo Nordisk that reduced a key workflow from twelve weeks to ten minutes using Amazon Bedrock and Atlas. Confluent was named Global Tech Partner of the Year with over 550 joint customer deployments focused on event-driven streaming and multi-agent systems alongside LangChain. Additional honorees included Google Cloud, Accenture, BigID, Pureinsights, gravity9, IBM, and Alibaba Cloud for their contributions across public sector solutions, database-as-a-service offerings, and generative AI frameworks.


### [庆祝卓越：MongoDB 全球合作伙伴奖 2025](https://yomu.fyi/post/mongodb-2025.md)
- Company: [MongoDB](https://yomu.fyi/company/mongodb.md)
- Author: Olivier Zieleniecki
- Published: Sep 18, 2025
- Type: Announcement

MongoDB announced its 2025 Global Partner Award recipients to recognize organizations driving AI adoption, legacy system modernization, and collaborative market expansion. Microsoft earned Global Cloud Partner for Azure integrations, while Amazon Web Services took Global AI Cloud Partner after assisting Novo Nordisk in cutting a key workflow from twelve weeks to ten minutes with Amazon Bedrock. Google Cloud received the Global Cloud GTM Partner award, and Confluent was named Global Technology Partner with over 550 joint deployments. LangChain and Pureinsights received honors for their integrations supporting retrieval-augmented generation and search solutions. Additional awards celebrated Accenture, BigID, gravity9, IBM, and Alibaba Cloud for their enterprise and public-sector impact.


### [우수성을 기념하기: 2025년 MongoDB 글로벌 파트너 어워드](https://yomu.fyi/post/2025-mongodb.md)
- Company: [MongoDB](https://yomu.fyi/company/mongodb.md)
- Author: Olivier Zieleniecki
- Published: Sep 18, 2025
- Type: Announcement

MongoDB announced the recipients of its 2025 Global Partner Awards to recognize partner contributions across cloud modernization, artificial intelligence, and enterprise integration. Microsoft received the Global Cloud Partner award for Azure integrations, while Amazon Web Services secured the Global AI Cloud Partner award following joint generative AI implementations such as reducing workflow durations for Novo Nordisk. Google Cloud earned the Global Cloud GTM Partner distinction through shared sales development programs, and Accenture took Global SI Partner honors after establishing a dedicated engineering Center of Excellence. Confluent was named Global Tech Partner with more than 550 joint streaming deployments, and LangChain was recognized as Global AI Tech Partner for vector search and agentic frameworks. The awards demonstrate partner-led technical solutions spanning distributed databases, data streaming pipelines, and scalable enterprise cloud architectures.


### [Celebrando la Excelencia: Premios Globales de Emparejar de MongoDB 2025](https://yomu.fyi/post/celebrando-la-excelencia-premios-globales-de-emparejar-de-mongodb-2025.md)
- Company: [MongoDB](https://yomu.fyi/company/mongodb.md)
- Author: Olivier Zieleniecki
- Published: Sep 18, 2025
- Type: Announcement

MongoDB presented its 2025 Global Partner Awards to honor partner organizations across cloud infrastructure, systems integration, generative AI, and enterprise data management. Microsoft received the Global Cloud Partner award for joint go-to-market solutions integrating MongoDB Atlas with native Azure services across healthcare, telecommunications, and financial services. Amazon Web Services earned the Global AI Cloud Partner award after delivering an Amazon Bedrock and MongoDB Atlas workflow implementation for Novo Nordisk that reduced processing time from twelve weeks to ten minutes. Confluent was recognized as Global Technology Partner for exceeding 550 joint deployments leveraging Apache Kafka for real-time data streaming and event-driven AI architectures. Other honored partners include Google Cloud for joint sales development programs, Accenture for software engineering centers of excellence, LangChain for retrieval-augmented generation tooling, and IBM for Watsonx.ai integrations.


### [Hommage à l’excellence : MongoDB Global Partner Awards 2025](https://yomu.fyi/post/hommage-a-l-excellence-mongodb-global-partner-awards-2025.md)
- Company: [MongoDB](https://yomu.fyi/company/mongodb.md)
- Author: Olivier Zieleniecki
- Published: Sep 18, 2025
- Type: Announcement

MongoDB announced the recipients of its 2025 Global Partner Awards, recognizing enterprise collaboration across cloud infrastructure, artificial intelligence, and systems modernization. Microsoft received the Global Cloud Partner award for joint integrations with Azure, while Amazon Web Services earned the Global AI Cloud Partner award following deployments combining Amazon Bedrock and MongoDB Atlas. Google Cloud was recognized for joint go-to-market initiatives, and Accenture received honours for establishing a dedicated MongoDB Center of Excellence within its software engineering division. Additional recognitions included Confluent for event-driven streaming deployments exceeding 550 customer implementations, BigID for data governance, gravity9 for modernization services, and Alibaba Cloud for managed database services. IBM and Accenture Federal Services were also acknowledged for strategic enterprise impact across mainframe architectures and public sector programs.


### [Seamlessly connect Stripe events to your frontend](https://yomu.fyi/post/seamlessly-connect-stripe-events-to-your-frontend.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: Josh Kahn
- Published: Sep 11, 2025
- Type: Explainer

Stripe events can connect backend state changes to responsive frontend interfaces for payments, subscriptions, connected accounts, and Stripe Terminal workflows. The post contrasts polling, which can increase traffic, latency, and delta-computation complexity, with push subscriptions, while noting that a hybrid of initial API loading and incremental updates is common. It describes webhook endpoints and Amazon EventBridge as ingestion choices, then emphasizes idempotency, ordering, duplicate detection, and failure handling. Frontend delivery options include WebSockets, MQTT, gRPC, Server-Sent Events, polling, AWS AppSync, and Momento, selected according to messaging pattern, direction, latency, scale, and operational trade-offs. An abbreviated lounge example uses a PaymentIntent and Terminal reader, routes payment\_intent.requires\_payment\_method through EventBridge to AWS AppSync Events, and lets a React client mark the reader ready to collect payment.


### [Real-time vs batch reconciliation: Practical patterns for keeping data in sync](https://yomu.fyi/post/real-time-vs-batch-reconciliation-practical-patterns-for-keeping-data.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: James Beswick
- Published: Aug 23, 2025
- Type: Explainer

Product data reconciliation across internal databases, ERPs, and platforms such as Stripe becomes harder to maintain as catalogs and system relationships grow, affecting financial reporting, customer experience, and operations. The post compares real-time reconciliation, using tools such as Kafka, Flink, Delta Live Tables, and Debezium for more immediate consistency, with batch approaches built around Spark, AWS Glue, Snowflake, and dbt for periodic, large-scale checks at lower cost and complexity. It also presents validation options including Great Expectations, Apache NiFi, Talend, and Airbyte, and recommends defining data ownership, applying field-specific conflict rules, and routing critical mismatches to manual review. Soft deletes and tombstones preserve deletion history, while observability through synchronization latency, conflict rates, resolution patterns, and regular audits helps assess reconciliation health as product catalogs scale.


### [Leveraging multimodal LLMs for Shopify’s global catalogue: Recap of expo talk at ICLR 2025](https://yomu.fyi/post/leveraging-multimodal-llms-for-shopify-s-global-catalogue-recap-of-exp.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jul 16, 2025
- Type: Explainer

Shopify’s Global Catalogue addresses fragmented, multilingual, multimodal product data that makes semantic search, faceting, matching, and AI-driven discovery difficult across millions of merchants. The system combines product data, understanding, matching, and reconciliation layers to standardize metadata, identify duplicate listings, and create canonical records with Universal Product IDs. Shopify fine-tunes vision large language models for multiple tasks per catalogue entity, while streaming infrastructure processes more than 10 million product updates daily and serves 40 million multimodal LLM-powered inferences. The resulting data supports merchant-admin suggestions, search, recommendations, embeddings, and conversational commerce, although scalability, accuracy, latency, model consolidation, graph reasoning, and continuous retraining remain active challenges.


### [The complete stream processing journey on FlinkSQL](https://yomu.fyi/post/the-complete-stream-processing-journey-on-flinksql.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Calvin Tran
- Published: Jun 12, 2025
- Type: Problem & solution

The post describes replacing per-user Zeppelin notebooks with a shared FlinkSQL gateway to improve interactive stream processing over Kafka data and support production pipelines. Zeppelin notebooks were limited by support for Flink 1.17, roughly five-minute cold starts, and isolation from internal platforms, while current Flink was 1.20. The replacement uses compute, integration, and query layers: a shared FlinkSQL gateway submits SQL through its REST API to session-cluster jobs, while a custom control plane handles authentication, authorization, session management, orchestration, and paginated results for UI and headless API clients. An interactive UI uses a Hive Metastore catalog to expose Kafka topics as tables, and the platform returns results in one minute rather than five; a configuration-based deployment tool provisions resources and deploys SQL-defined Flink pipelines within ten minutes. After release, the post reports an uptick in created pipelines and interactive queries, but it does not quantify that increase.


### [Real-time payment analytics: Building a data pipeline from Stripe to AWS](https://yomu.fyi/post/real-time-payment-analytics-building-a-data-pipeline-from-stripe-to-aw.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: James Beswick
- Published: Mar 14, 2025
- Type: Tutorial

Payment processors such as Stripe emit numerous transaction events, while dashboard and API-based or batch approaches can limit historical analysis, real-time visibility, and correlation with business metrics as volumes grow. The proposed pipeline sends signed Stripe webhooks through API Gateway to Lambda for validation and enrichment, buffers them in Kinesis Data Streams, and uses a second Lambda to transform and index records in OpenSearch. OpenSearch Dashboards provides visualizations, while CloudWatch tracks pipeline health; Kinesis retention and replay support recovery and reprocessing. The post also describes a Kinesis Data Firehose route that can remove the consumer Lambda and reduce costs but may add batching latency. The resulting design is presented as scalable to millions of transactions per day, with sub-minute payment metrics visibility and flexible historical querying, while security guidance includes encryption, IAM configuration, and regular audits.


### [The foundations of Canva’s continuous data platform with Snowpipe Streaming](https://yomu.fyi/post/the-foundations-of-canva-s-continuous-data-platform-with-snowpipe-stre.md)
- Company: [Canva](https://yomu.fyi/company/canva.md)
- Author: Jack Caperon
- Published: Jan 6, 2025
- Type: Problem & solution

As Canva expanded to over 200 million monthly active users, ingestion throughput reached 25 billion records daily, causing AWS Data Firehose costs to consume nearly half of the product analytics platform budget. To reduce these expenses and eliminate intermediate file staging, the engineering team integrated Snowflake's Snowpipe Streaming directly with their Java-based Kinesis Data Streams pipeline. The architecture streams records directly into Snowflake tables using logical channels with offset checkpoints while configuring client buffering up to a five-minute maximum lag. In handling Kinesis Client Library edge cases, record processors coordinate with the LeaseCoordinator to drop leases when channels become unhealthy. Operating in production for over six months, the system ingested more than 20.35 petabytes of data, decreased query latency to roughly ten minutes, and reduced overall cloud spend by 45 percent.


[Newer posts](https://yomu.fyi/topic/streaming/page/3.md) · [Older posts](https://yomu.fyi/topic/streaming/page/5.md)