---
title: "Streaming"
description: "123 posts about Streaming, summarised, each linking to the original."
---

# Streaming
> 123 posts about Streaming, summarised, each linking to the original.

## Articles

### [What to look for when selecting a real-time analytical database](https://yomu.fyi/post/what-to-look-for-when-selecting-a-real-time-analytical-database.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: Jun 4, 2026
- Type: Explainer

Real-time analytical databases must serve low-latency queries over data that is continuously inserted at high volume, rather than merely query a static dataset quickly. The article frames time-to-insight as the combination of time to ingest, time to transform and prepare data, and time to query it. It argues that batch-oriented ingestion, delayed visibility, scheduled materialized-view refreshes, and cache invalidation during writes can make production latency far worse than headline benchmarks. It recommends evaluating freshness under concurrent ingest and queries, whether one engine supports structured analytics, text search, vector similarity, and JSON, and whether index maintenance can avoid blocking ingestion. As an example, Bullet reportedly moved to ClickHouse Cloud, reducing data latency by 99.4% to under five seconds, achieving millisecond queries, and handling 1,000× more data at comparable cost.


### [Apache Spark Real-Time Mode for Gaming: A Better Way to Do Real-Time Sessionization](https://yomu.fyi/post/apache-spark-real-time-mode-for-gaming-a-better-way-to-do-real-time-se.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Neha Prabhu, Murali Talluri
- Published: Jun 3, 2026
- Type: Tutorial

Gaming platforms need sub-second session data for personalization, recommendations, content scheduling, and operational controls across millions of devices. The described pipeline uses Apache Spark Real-Time Mode with transformWithState to process Kafka events, maintain per-device session state, and emit scheduled heartbeats and timeout events without a separate streaming engine. Events are grouped by deviceId; a StatefulProcessor uses MapState keyed by session ID, while handleInputRows() handles starts and ends and handleExpiredTimer() registers and fires 30-second timers. The implementation was tested at about 500K input events per minute, 4M active sessions, and 8M heartbeat records, producing 16x output amplification. End-to-end latency reached 432 ms at p99, reported as 20x faster than micro-batch mode, and the source states that Real-Time Mode is generally available.


### [Executable UDFs are now in public beta on ClickHouse Cloud](https://yomu.fyi/post/executable-udfs-are-now-in-public-beta-on-clickhouse-cloud.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: May 30, 2026
- Type: Announcement

Executable UDFs are available in public beta on ClickHouse Cloud, allowing Python functions packaged as zip files to be called from SQL like built-in functions. ClickHouse runs them in a pool of long-lived sandboxed processes, keeping model code near the data and supporting use in ad-hoc queries, joins, and materialized views. The demonstration wraps a roughly 270K-parameter PyTorch autoencoder in an embed\_trade UDF that consumes 14 features, produces a 32-dimensional embedding and reconstruction error, and scores trades as they enter through a materialized view. Training uses streamed Parquet chunks, incremental StandardScaler fitting, early stopping, and saved model artifacts; the deployed pool keeps the 2MB model hot, so inference occurs once per trade and downstream queries read the resulting table.


### [Announcing Lakebase Change Data Feed (CDF)](https://yomu.fyi/post/announcing-lakebase-change-data-feed-cdf.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Pranav Aurora, Cheng Chen, Hristo Stoyanov
- Published: May 27, 2026
- Type: Announcement

Lakebase Change Data Feed (CDF) is available in Public Preview to reduce the manual effort of moving data from operational databases into downstream systems. The feed is enabled once for all tables in a project, stored and governed in Unity Catalog Managed Tables, and readable by engines, models, and agents without separate extraction pipelines. From one shared feed, teams can build streaming pipelines with SDP, create materialized views with DBSQL, or compute and store embeddings with Agent Bricks, while consumers remain isolated from the primary operational workload. The announcement positions Lakebase as the native Bronze layer in a medallion architecture, complementing Synced Tables and providing governance and lineage across the data lifecycle.


### [You’ve built the media products, now make them personalized](https://yomu.fyi/post/you-ve-built-the-media-products-now-make-them-personalized.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Elena Tesser
- Published: May 21, 2026
- Type: Explainer

Media companies may have launched streaming services, digital editions, and mobile apps, yet still face a digital product intelligence gap: product teams must answer behavioral data questions quickly enough to personalize experiences and improve engagement. Databricks Genie gives nontechnical product leaders a conversational interface that translates natural-language questions into SQL queries, visualizations, and actionable insights over governed enterprise data, without requiring code or analyst handoffs. The agent queries governed Delta Lake tables fed by streaming clickstreams, views, and session signals, combining event-level behavior, A/B tests, audience segments, and cross-platform data in conversational answers. The source says internal benchmark accuracy improved from 32% to over 90% through multi-LLM orchestration, specialized knowledge search, and parallel reasoning, and presents Genie as reliable enough for production personalization decisions.


### [clickhousectl v0.2.0: Postgres, ClickPipes and more](https://yomu.fyi/post/clickhousectl-v0-2-0-postgres-clickpipes-and-more.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: Al Brown
- Published: May 20, 2026
- Type: Announcement

clickhousectl v0.2.0 expands the beta CLI with local and ClickHouse Cloud managed Postgres, ClickPipes, SQL over HTTP for Cloud services, agent-oriented output changes, and a standalone Rust client for the ClickHouse Cloud API. Local Postgres runs in Docker, is keyed by name and major version, stores isolated data under .clickhouse/servers/-pg/data/, and supports versions 16, 17, and 18. ClickPipes covers batch, streaming, CDC, and snapshot sources including object storage, Kafka-compatible systems, Kinesis, Postgres, MySQL, MongoDB, and BigQuery. Cloud queries use a provisioned or lazily enabled Query Endpoint and service-scoped API key, while the endpoint uses sql\_console\_admin and the key has no organization-level roles. The release also adds Markdown-style tables and credential debugging, with more than 5,000 users reported five weeks into the project.


### [Agentic analytics starts with query-ready data: the write-side cost of Snowflake vs. ClickHouse](https://yomu.fyi/post/agentic-analytics-starts-with-query-ready-data-the-write-side-cost-of.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: Tom Schreiber, Lionel Palacin
- Published: May 6, 2026
- Type: Benchmark

Agentic analytics workflows introduce bursts of concurrent, low-latency analytical queries over continuously ingested data. Analytical query efficiency relies on columnar storage and chunk-level pruning, which require contiguous, sorted data to avoid full-table scans. Rather than evaluating warehouses solely at query runtime, this benchmark compares the write-side cost of maintaining query-ready data in ClickHouse and Snowflake during continuous ingestion at approximately one million rows per second. Snowflake relies on post-write clustering to organize data, whereas ClickHouse constructs sorted layouts on the write path and refines them via background merges. Across 100B, 200B, and 300B row checkpoints on the ClickBench dataset, ClickHouse achieved a query-ready layout at 22 times lower cost and delivered 28 times better write-side cost-performance.


### [Realtime or Pipelines? How to choose the right tool](https://yomu.fyi/post/realtime-or-pipelines-how-to-choose-the-right-tool.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: eduardo\_gurgel, riccardo\_busetti
- Published: May 5, 2026
- Type: Explainer

Supabase Realtime and Supabase Pipelines both read PostgreSQL changes through logical replication, but they target different jobs. Realtime delivers live updates and communication to browsers and apps through Broadcast, Presence, and Postgres Changes; its Postgres Changes path checks Row Level Security for subscribed users, uses temporary replication slots, and offers best-effort delivery without replay after disconnects. Pipelines is a managed change-data-capture system that performs an initial sync, then replicates selected publication data in batches to supported analytical destinations through a persistent replication slot, retaining progress and retrying transient failures. The choice depends on the destination and delivery needs: use Realtime for low-latency user experiences that can tolerate gaps, Pipelines for durable analytical copies with at-least-once processing and idempotent destination writes, and both when an application needs both capabilities.


### [Approximate answers, exact decisions: New sketch functions for analytics](https://yomu.fyi/post/approximate-answers-exact-decisions-new-sketch-functions-for-analytics.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Daniel Tenedorio, Kent Marten, Gengliang Wang, Chenhao Li
- Published: Apr 29, 2026
- Type: Announcement

Databricks now supports four sketch function families, built on Apache DataSketches, for replacing expensive exact percentiles, distinct counts, set operations, top-K rankings, and metric aggregations with bounded-memory approximations. KLL sketches answer quantiles, Theta sketches support unions, intersections, and differences, approximate top-K sketches track frequent items, and Tuple sketches combine distinct counting with aggregated metrics such as customer revenue. The sketches can be built during ETL, stored as columns in Delta tables, and merged on read, allowing dashboards and streaming pipelines to avoid rescanning raw data; SQL, DataFrame, and Structured Streaming pipelines are supported. The stated trade-off is configurable 1–2% relative error, with the source citing a 1000x speedup for applicable workloads. Exact methods remain appropriate for financial auditing, compliance reporting, and other precision-required uses.


### [Companies winning with AI built the data layer first](https://yomu.fyi/post/companies-winning-with-ai-built-the-data-layer-first.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Aly McGue
- Published: Apr 29, 2026
- Type: Explainer

Trinity Industries’ experience is presented as a case for treating the data layer, rather than models or dashboards, as the foundation of enterprise AI. The railcar manufacturer migrated 95% of its enterprise data to a single Databricks lakehouse, adopted Medallion architecture, moved transformations upstream, and retired legacy dashboards containing nearly 600 measures. That consolidation supports streaming ETA predictions, procurement agents, and Genie conversational analytics: the ETA model is reported as 50% more accurate than industry ETAs, agents helped increase on-time material delivery by 15%, and Genie handles more than 1,000 questions monthly. The migration took close to a year, followed by six to eight months of additional work, but the account argues that trusted, standardized data enables experimentation, automation, and broader employee access to analysis.


### [From months to minutes: Building real-time clinical data pipelines with natural language](https://yomu.fyi/post/from-months-to-minutes-building-real-time-clinical-data-pipelines-with.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Matthew Giglia, Tim Kessler, Assunta Carey-Saylor
- Published: Apr 28, 2026
- Type: Explainer

Healthcare data teams often spend months integrating EHR systems, normalizing HL7, CCD, and X12, and routing data through intermediary storage before analytics, creating latency and maintenance burden. Databricks and Redox describe a pipeline model that combines Redox MCP Server, natural-language prompts, and Databricks Zerobus Ingest to build integrations inside Databricks and stream clinical data directly into Unity Catalog managed tables. The post says Zerobus provides subsecond latency, while the MCP Server identifies environments, suggests workflow steps, executes integration tasks, and surfaces validation signals such as logs and performance summaries. A demonstrated workflow retrieved a recent patient admission as structured data with a plain-language summary, and Redox writeback can return AI outputs to EHRs for point-of-care action. The same foundation is presented as enabling real-time use cases and Redox Agents built with Databricks Genie Spaces.


### [Operational databases: How they work and when to use them](https://yomu.fyi/post/operational-databases-how-they-work-and-when-to-use-them.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Apr 24, 2026
- Type: Explainer

Operational databases, or online transaction processing (OLTP) databases, support day-to-day operations by processing real-time CRUD transactions with low latency, high concurrency, and ACID guarantees. The guide contrasts them with data warehouses and OLAP systems, which favor historical analysis, complex aggregations, and read-heavy workloads, while describing ETL, CDC, and operational data stores in the OLTP-to-OLAP pipeline. It argues that traditional OLTP systems can struggle with real-time analytics and AI because of siloed data, structured-data constraints, rigid schemas, scaling limits, and missing governance features. Databricks Lakebase is presented as a hybrid solution using separate storage and compute, serverless Postgres, elastic scaling, branching, shared lakehouse storage, and open formats. The proposed workflow connects existing OLTP systems through CDC or streaming into Delta Lake for fresh SQL, BI, ML, and AI access.


### [Beyond the spreadsheet: How Databricks is delivering the modern CFO in financial services](https://yomu.fyi/post/beyond-the-spreadsheet-how-databricks-is-delivering-the-modern-cfo-in.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Jennifer Miller, Marcela Granados, Andrea DeSosa, Alex Oberlander, Kim Hatton, Pavithra Rao, Naeem Rehman, Pravin Varma, Olga Deriy, Prasanna Selvaraj
- Published: Apr 22, 2026
- Type: Explainer

Financial-services CFOs are being asked to act as Strategists and Catalysts, but fragmented legacy systems, T+1 batch processing, opaque reporting logic, and mismatched business semantics keep them focused on stewardship and retrospective operations. The proposed answer is Databricks as a unified, governed platform combining real-time streaming, centralized lineage, data and AI, with Unity Catalog, Lakeflow, Genie, and Agent Bricks addressing trust, latency, access, and model reproducibility. Unity Catalog can connect semantic definitions and end-to-end lineage from transactions through regulatory reports and models, while Lakeflow supports continuous ledger and liquidity processing. The post cites a global bank reducing liquidity-reporting processing from 10 hours to 8 minutes and Nationwide Insurance reporting a 5-point combined-ratio improvement and 3-point expense-ratio improvement. It presents a Lakehouse-based CFO stack as a shift from reporting historical results toward real-time capital management and previews AI-driven deposit and PPNR modeling.


### [Multimodal Data Integration: Production Architectures for Healthcare AI](https://yomu.fyi/post/multimodal-data-integration-production-architectures-for-healthcare-ai.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Maks Khomutskyi
- Published: Apr 22, 2026
- Type: Problem & solution

Production healthcare AI often depends on combining genomics, imaging, clinical notes, and wearables, but separate modality stacks can leave initiatives stalled before deployment. A lakehouse pattern addresses this by landing each modality in governed Delta tables under Unity Catalog controls while supporting cross-modal features, lineage, time travel, and reproducible training sets. Genomics processing can use Glow on Spark, imaging embeddings can feed governed tables and vector search, notes can become temporally aware entities, and wearables can flow through Lakeflow Spark Declarative Pipelines for streaming feature windows. The post compares early, intermediate, late, and attention-based fusion, matching them to dimensionality, temporal, and availability constraints, with late fusion presented as a practical baseline when modalities are missing. It argues that modality masking, sparse attention, and transfer learning are recommended responses because incomplete data is the default and connects the architecture to governed tumor board review, faster cohort assembly, and more clinical workflows.


### [Personalization Strategies for Media Companies](https://yomu.fyi/post/personalization-strategies-for-media-companies.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Apr 22, 2026
- Type: Explainer

Media companies are expected to tailor content and messaging across channels, with personalization programs typically targeting lower churn, deeper sessions, and higher subscription revenue. The proposed foundation starts with an audit of CRM, email, warehouse, and customer data platform sources, followed by required-field definitions, ownership mapping, consent tagging, and ingestion-quality checks. Teams then map segment behaviors and conversion or retention paths, including negative signals, before selecting a CDP with real-time ingestion, identity resolution, unified profiles, segmentation, activation, and governance. Identity rules should connect anonymous and authenticated activity across web, mobile, and connected TV, while event-level streaming delivers messages through email, push, in-app, and advertising channels with minimal latency. The roadmap recommends an MVP comprising unified ingestion, a homepage recommendation surface, and triggered re-engagement email, with performance, churn, revenue lift, and model drift monitored through iteration.


### [From Static Policies to Autonomous Insurance: How AI enables Real-Time Coverage](https://yomu.fyi/post/from-static-policies-to-autonomous-insurance-how-ai-enables-real-time.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Amee Vora, Anindita Mahapatra, Marcela Granados
- Published: Apr 20, 2026
- Type: Explainer

Insurance products often rely on annual pricing, periodic risk assessment, and coverage changes tied to major life events, even though driving behavior, digital fraud exposure, and travel risk can change in seconds. The post presents embedded insurance and usage-based insurance as a shift toward contextual, just-in-time coverage delivered through apps, checkout flows, vehicle purchases, and partner platforms. Its automotive example combines telematics, geospatial data, and real-time weather signals to update risk scores and premiums from trip behavior such as braking, following distance, mileage, location, and environmental conditions. A Databricks Lakehouse architecture registers devices, ingests telemetry through Zerobus, curates data in a governed Medallion architecture, and exposes features through Lakebase and Databricks One, with Unity Catalog supporting access controls, lineage, explainability, and audit trails. The proposed trajectory leads from embedded coverage toward autonomous insurance whose policies continuously adapt to risk.


### [“A generational leap”: How Trio unified payment analytics and cut storage by 88% with ClickHouse Cloud](https://yomu.fyi/post/a-generational-leap-how-trio-unified-payment-analytics-and-cut-storage.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: ClickHouse
- Published: Apr 17, 2026
- Type: Problem & solution

Brazilian payment processor Trio faced severe data challenges after scaling and acquiring PayBrokers, processing over 243 million payments in early 2025 across disparate legacy databases. Because financial reconciliation and compliance demand absolute precision, the engineering team unified their analytics layer by migrating to ClickHouse Cloud. To handle out-of-order and duplicate events from their Redpanda streaming pipelines, Trio implemented sliding-window refreshable materialized views alongside ReplacingMergeTree engines rather than recomputing entire historical datasets. A custom ETL service manages schema drift before inserting clean records into ClickHouse, while batch ingestion is coordinated through Airflow. Consequently, Trio reduced storage consumption by approximately 88% through columnar compression, ingested a 5-billion-row migration without issue, and maintained real-time query performance across more than one billion daily ledger data points.


### [What is operational analytics?](https://yomu.fyi/post/what-is-operational-analytics.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Apr 15, 2026
- Type: Explainer

Operational analytics uses real-time or near-real-time data to monitor day-to-day operations and support immediate decisions, unlike traditional analytics, which generally relies on batch data to explain past performance. Its workflow collects logs, events, clickstreams, telemetry, transactions, and other signals; streams them into a unified platform for cleaning, transformation, and enrichment; analyzes incoming data with rules, models, or anomaly detection; and sends results to dashboards, alerts, or operational applications. This can help teams detect issues earlier, reduce mean time to detect (MTTD) and mean time to respond (MTTR), improve forecasting, coordinate across departments, and act on inventory, customer, or system changes. The approach also requires integrating heterogeneous systems, maintaining data quality and schemas, embedding insights into existing workflows, and building reliable low-latency pipelines, with tools such as Lakeflow, Databricks SQL, and AI and machine learning capabilities presented as examples.


### [Building Biz Ask Anything: From Prototype to Product](https://yomu.fyi/post/building-biz-ask-anything-from-prototype-to-product.md)
- Company: [Yelp](https://yomu.fyi/company/yelp.md)
- Author: Maria Christoforaki, Group Tech Lead; Shree Shalini Pusapati, Software Engineer
- Published: Mar 27, 2026
- Type: Problem & solution

Yelp expanded Yelp Assistant to business pages by developing a question-answering architecture that extracts concise, evidence-backed answers from reviews, photos, and structured facts. The production system replaces static snapshots with near-real-time streaming pipelines for reviews and structured properties alongside weekly batch ingestion for menus and website data. For each incoming query, the system evaluates intent, retrieves relevant context through keyword and embedding searches, and constructs targeted prompts with dynamically selected few-shot examples. Synthesized answers are augmented with supporting visuals and citations before being streamed token-by-token to users. Generating suggested questions directly from specific business content rather than generic categories improved user engagement by roughly fifty percent and reduced unanswerable queries by twenty-six percent.


### [March 2026 newsletter](https://yomu.fyi/post/march-2026-newsletter.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: Mark Needham
- Published: Mar 19, 2026
- Type: Announcement

The March 2026 ClickHouse newsletter details major platform updates, community deployments, and architectural practices for high-throughput analytics. Recent product announcements include ClickHouse 26.2 featuring time-based block flushing for streaming feeds, general availability of native inverted indexes for full-text search, and the release of chDB 4 with a Pandas-compatible DataStore API. Community implementations show Cloudflare managing 1.61 quadrillion daily events, while Trigger.dev isolated multi-tenant SQL execution using a domain-specific query language that compiles safe ClickHouse queries. In data optimization techniques, Hookdeck accelerated webhook payload search from over 30 seconds to under 400 milliseconds by partitioning payload hashes into typed bucket columns and scanning iterative time windows. Additional operational guidance emphasizes setting the ttl\_only\_drop\_parts parameter and aligning table partitioning with time units so background merges drop entire data parts cleanly without triggering mutation rewrites.


[Newer posts](https://yomu.fyi/topic/streaming/page/2.md) · [Older posts](https://yomu.fyi/topic/streaming/page/4.md)