Loading…
Clickhouse
High-performance open-source columnar database management system designed for real-time analytics and data warehousing at scale
- 235 posts summarised
- latest
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
From their blog
Latest articles
Shopify powers observability for global-scale commerce with ClickHouse
Shopify rebuilt its observability infrastructure after metrics, logs, and traces became fragmented across vendors, with unpredictable costs and limited cross-signal analysis. It created Observe, a unified platform that runs metrics, logs, traces, profiles, and exceptions on self-hosted ClickHouse, treating them as structured, time-ordered, high-dimensional events and serving roughly 50 million events per second normally and 100 million at BFCM peak, or about 110 GB/s. Kafka buffers telemetry for large synchronous durable writes, while materialized views support flexible schemas, autocomplete, typed hot keys, and identifier-based correlation; the latter reportedly makes targeted searches 10x faster. Shopify reports a 16x query improvement out of the box, exceeding 30x at peak performance, with data queryable in under a minute, and says it would likely choose ClickHouse Cloud if starting again because managed operations could reduce the self-hosting burden.
How Sony LIV uses ClickHouse Cloud to deliver live streaming analytics at billion-row scale
Sony LIV’s legacy analytics environment combined batch pipelines, Elasticsearch, BigQuery, and other telemetry stores, creating slow, costly, fragmented analysis as streaming activity reached billions of events per day. During live sports, the platform needed sub-second queries, high ingestion throughput, cost-efficient long-term storage, and horizontal scaling for sudden traffic spikes. Sony LIV consolidated these workloads on ClickHouse Cloud: client, streaming, CDN, and advertising systems send events through Amazon Kinesis and ClickPipes into a ClickHouse cluster serving dashboards, product analytics, QoE, and SRE workflows. The source reports that queries previously taking tens of seconds to minutes now run in sub-second to low-single-digit seconds, improving operational responsiveness, engineering productivity, and the handling of major live-event traffic.
What's New with Monitoring in PostgreSQL 19
PostgreSQL 19, still in beta when this was written, adds monitoring and observability changes spanning lock contention, logging, WAL, wait events, remote messages, and multixacts. The default for log_lock_waits changes to on, while log_min_messages gains per-process-type levels and autoanalyze receives its own duration setting. A new wal_fpi_bytes counter exposes full-page-image volume cluster-wide, per session, per query, and in VACUUM and ANALYZE logs; new wait events expose WAL write, flush, and replay stages plus COPY file, pipe, and program I/O. Remote server messages now use normal PostgreSQL logging with origin prefixes, and pg_get_multixact_stats() surfaces multixact usage that previously required filesystem inspection and educated guessing. The post also lists upgrade work for parsers, metric collectors, wait-event dictionaries, configuration UIs, and alert thresholds, while noting that final release notes may differ.
Announcing support for ClickStack in the ClickHouse Terraform provider
The ClickHouse Terraform provider now supports managing ClickStack resources for both self-hosted deployments and ClickHouse Cloud. Dashboards, alerts, sources, saved searches, connections, and webhooks can be kept in version control, reviewed, and applied with the standard terraform plan and terraform apply workflow, helping keep configurations aligned across environments. ClickStack is implemented as a dedicated service module in the existing ClickHouse/clickhouse provider, using Cloud organization credentials plus a service ID for Managed ClickStack, or an endpoint and personal API access key for self-hosted deployments. The work also extends the API contract with named schemas, consistent integer fields, structured validation errors, and dashboard JSON validation during planning when the validation endpoint is available. Support is beta from provider version 3.25, with import and generated-configuration workflows for existing resources; if validation is unavailable, planning warns and validation is deferred to apply.
What’s new in the ClickHouse Grafana plugin
ClickHouse’s Grafana plugin has shipped a set of changes aimed at making common observability tasks—finding log lines, narrowing to services, and detecting deployments—faster without requiring SQL. Version 4.20 adds single-table configuration and compact query mode, where users search logs, apply type-aware filters, review query history, and see or edit generated ClickHouse SQL. Investigation can proceed by clicking through collapsible OpenTelemetry Resource, Scope, and Log attributes, filtering values or selected message text, while dashboards can discover Map keys and values across tables. The release also includes interactive Logs Explorer, Traces Explorer, and per-service OpenTelemetry dashboards, plus guided variable and annotation editors that generate editable SQL for distinct values and change markers such as deployments and rollbacks. Additional changes cover schema-introspection caching, trace-ID lookup, Grafana 13 compatibility, SQL validation, backend OpenTelemetry instrumentation, and forwarding Grafana identity to ClickHouse; compact mode initially remains limited to single-table data sources.
ClickStack and Hud bring runtime intelligence to AI-powered development
ClickHouse and Hud announce an integration that combines ClickStack’s service-level observability with Hud’s function-level runtime forensics for AI-assisted software development. As coding agents generate more code, the post identifies the central challenge as assessing production risk, detecting regressions, and giving engineers enough context to fix them. Teams connect the ClickStack and Hud MCP servers to a single coding agent, while shared trace IDs correlate OpenTelemetry traces and logs with Hud’s function-level metrics and forensic data; Hud’s Runtime Code Sensor requires no manual instrumentation and runs with low overhead in production. The resulting workflows support pre-deployment risk assessment against production behavior, release monitoring, auto-triage and auto-fix, and rollback or code-level remediation when issues arise.
How iFood built its agentic security platform on ClickHouse Cloud
iFood’s security incident response team needed high-volume log ingestion, long-term retention, and rapid access to months of history, but its Databricks-based platform became costly to scale beyond 130 TB and imposed query timeouts. The team ran a production-data proof of concept and rebuilt its tooling around ClickHouse Cloud on AWS: logs land in S3 and flow through ClickPipes into ClickHouse, with Querybook preserving notebook-based investigations and ClickStack planned for operational metrics. With ClickHouse Cloud, iFood reports 9–16x faster queries, expected cost reductions of 40–50%, and data freshness improving from hourly batches to 2–10 minutes; alerts that previously ran hourly can now run about every 10 minutes. That foundation also supports agentic threat hunting: multiple sub-agents query more than 30 TB in parallel, reducing a hunt from about a week to roughly two hours, while Langfuse is used to inspect and tune agent analyses.
What's new in pg_clickhouse v0.10.0: Subqueries, TPC-H Speedups, C Driver, and Aggregates
pg_clickhouse v0.10.0 expands pushdown for analytic workloads, with full pushdown across the TPC-H suite remaining the stated target. The release moves the scoreboard from 12 of 22 fully pushed-down queries to 16, including Q17, which falls from 32.7 seconds to 37 milliseconds when its correlated aggregate subquery is shipped to ClickHouse. SubPlan handling now turns PostgreSQL subqueries into ClickHouse subqueries, while NOT IN and related operators receive guards that preserve PostgreSQL’s three-valued NULL semantics; compatibility requires ClickHouse 25.8 or later for the correlated-subquery shape. The binary driver was rebuilt on a plain-C client library and hardened against concurrency bugs. The release also broadens pushed-down functions and aggregates, adds clickhouse_perform() and clickhouse_server_version(), and leaves six TPC-H queries plus other roadmap items unresolved.
bitdrift brings ClickHouse-powered mobile observability to ClickStack
bitdrift has joined ClickHouse’s House Mates program as a mobile observability partner, bringing mobile-native telemetry, tracing, and debugging to ClickStack. ClickStack provides an open-source or managed observability stack built on ClickHouse, with HyperDX and an OpenTelemetry Collector for logs, metrics, traces, and OTLP ingestion. bitdrift’s Capture SDK supports iOS, Android, and React Native, stores unsampled telemetry in a fixed-size on-device ring buffer, and uses control-plane configuration to decide what is collected or uploaded. Shared OpenTelemetry trace context lets engineers follow a mobile request’s trace ID from a session timeline into ClickStack, where backend traces, dependencies, logs, metrics, database operations, and infrastructure behavior are available, creating a continuous debugging path across the device and distributed systems.
Whats new in ClickStack - June + July
ClickStack’s June–July releases add a broader set of observability features, with work centered on trace navigation, metrics sources, dashboards, and filter performance. The trace waterfall now assigns stable colors by service, keeps span durations outside bars, preserves clickable proportions for very short spans, and adds a minimap plus OpenTelemetry span-link navigation. Metrics support includes quantiles for exponential histograms, a leaner default schema, and an experimental UI connection to external Prometheus-compatible endpoints, while PromQL is proxied to the configured endpoint. Dashboard filters can narrow one another, event patterns can appear as tiles, kiosk mode provides a read-only display, and filter-value discovery prefers ClickHouse text indexes when available. Additional changes cover alert annotations, browser RUM dashboard views, external links from table tiles, grid snapping, and service-map coloring by latency, error rate, or throughput.
How Musinsa scaled its audience engine with ClickHouse Cloud and reduced TCO by 71.4%
Musinsa’s Audience Engine powers a customer data platform that targets audiences across 16.4 million members. It stores each user-to-audience relationship as a separate row, producing roughly 1.1 billion cohort-to-user mappings across 2,500 audiences and potentially 16 million new rows when one audience is added. Its self-hosted ClickHouse deployment coupled storage and compute, forced workloads to share a cluster, and created operational burdens as data and query demands grew. Musinsa migrated to ClickHouse Cloud on AWS, using separated S3-backed storage, workload-specific compute, automatic idling, vertical autoscaling, and ClickPipes for direct real-time ingestion. The company reports an 86.5% reduction in storage costs and a total cost of ownership reduction of up to 71.4%, while shifting engineering effort toward CDP business logic.
What's new in ClickHouse Managed Postgres: Customer notifications, better observability, faster backups, extensions, and more
ClickHouse reports a set of updates to its Managed Postgres beta, aimed at simplifying onboarding and advancing its unified Postgres-and-ClickHouse data stack. The redesigned flow supports sample-data loading or PostgreSQL migration through ClickPipes, built-in SQL exploration, replication to ClickHouse, and cross-system queries through pg_clickhouse; services also notify users when disk usage remains above 85%, with console, email, and optional Slack channels. Observability now includes searchable PostgreSQL logs, CPU, memory, disk, IOPS, connection and per-table metrics, query insights, a Prometheus-compatible /metrics endpoint, and an OpenAPI slow-query API. Backups use more CPU and disk throughput on larger servers, making backup and restore times predictable even on 10+ TB servers, while extensions add pg_re2 and improvements to pg_clickhouse and pg_stat_ch, including regex pushdown, Arrow batching, sampling, and safer worker exits; Google Cloud Platform availability and further maintenance, configuration, and integration work are planned.
ClickHouse Release 26.7
ClickHouse 26.7 is a monthly release with 61 new features, 112 performance optimizations, and 329 bug fixes, including improvements to aggregation, joins, vector search, phrase search, query analysis, and URL access. Its combination of optimize_aggregation_in_order and the new optimize_aggregation_in_order_limit turns matching GROUP BY, ORDER BY, and LIMIT queries into low-memory streaming pipelines that can stop reading after N groups. In a TPC-H scale-factor-100 test, the optimized query ran 313× faster, processed 339× fewer rows, and used 592× less peak memory. The release also adds build-side JOIN pruning, LLVM JIT compilation for supported regular expressions, and macOS improvements including faster allocation-heavy workloads, concurrent remote-replica reads, SSD cache dictionaries, FileLog, and STREAM support.
How Physical Intelligence unified its robotics data stack with Postgres managed by ClickHouse
Physical Intelligence, which builds robotics foundation models, needed a data stack that could support both transactional operations and high-cardinality analytical workloads. Its previous single RDS Postgres instance could not handle those demands, so the team migrated to ClickHouse Cloud and adopted Postgres managed by ClickHouse for OLTP alongside ClickHouse for OLAP. Postgres is automatically replicated into ClickHouse through ClickPipes, while materialized views, full-text indices, and a query agent help researchers search across the company’s data. The resulting stack supports faster, easier exploration as data volume grows from the 10 billion to 100 billion row range, and also handles telemetry through ClickHouse with ClickStack and Grafana.
What is WAL backpressure, and why does ClickHouse Managed Postgres need it?
Postgres writes every change to the write-ahead log (WAL), and Managed Postgres uploads completed segments to object storage for point-in-time recovery; if uploads lag, retained WAL can fill the disk and cause a PANIC shutdown. ClickHouse Managed Postgres counters this with a data-plane systemd timer that checks pending segments every 15 seconds and uses cgroup v2 I/O controls to throttle client write bandwidth at 80%, 50%, or 20% of provisioned throughput as backlog thresholds rise. The drain path—including the archiver, checkpointer, and related processes—is placed in an immune cgroup, while client backends are throttled and reads remain uncapped. In a test with a 500 MB/s gp3 volume and an archive limited to 4 MB/s, the backlog reached 3,433 segments and 53 GiB before recovery. Once the limit was removed, the archiver cleared the queue, the throttle released automatically, and the disk stayed below 33% full.
How Mercado Libre rebuilt its observability platform on ClickHouse Cloud with 50x faster trace queries
Mercado Libre rebuilt its O11y events observability platform to answer granular business-level questions such as why a payment failed across a large microservice environment. The previous storage-layer design performed transformation, filtering, and aggregation at query time, causing queries to run longer than five minutes and making high-cardinality analysis expensive. The team moved storage to ClickHouse Cloud on AWS while retaining its existing ingestion path, and shifted heavy work into the database with materialized views, layered retention, and a dedicated lookup table for high-cardinality filtering. O11y events uses OpenTelemetry instrumentation with business-flow attributes and samples all traffic for critical applications. The resulting system reduced trace-query time to about four seconds, a 50x speedup, achieved up to 89% compression, and scaled from 7 million to 400 million spans per minute in ClickHouse.
Fixed cadence to seconds: making ClickHouse Cloud autoscaling more reactive
ClickHouse Cloud’s vertical autoscaling recommender originally ran on a fixed timer, creating latency when workloads suddenly increased and services needed resources before the next scheduled pass. The redesign keeps the periodic sweep as a safety net while adding a reactive fast path, using controller-runtime as an event-processing layer with sources feeding a shared rate-limited work queue and reconciler. The queue provides deduplication, exponential backoff, bounded concurrency, leader election, and metrics, while ClickHouse stores near-real-time signals that a polling source reads; the excerpt notes polling sets a floor on reaction speed. The reactive path is rolling out cautiously in observe-only mode, with periodic reconciliation remaining the main driver, and the stated goal is to trigger scale-ups for critical events such as out-of-memory conditions in seconds rather than waiting for the next scheduled run.
ClickHouse launches ClickHouse Labs with Andy Pavlo as VP of Database Research
ClickHouse announced the launch of ClickHouse Labs, a new research group led by Andy Pavlo, who joins as Vice President of Database Research. The group is intended to advance foundational work in database systems and share it openly with the broader community. Its research spans database and data architectures, query processing, systems performance, and infrastructure for both ClickHouse and PostgreSQL, with applications including real-time analytics, data warehousing, observability, and AI. Researchers will build and test ideas against demanding real-world workloads, collaborate with ClickHouse engineers, and publish findings in leading research venues so others can evaluate, extend, and build on them.
Andy Pavlo joins ClickHouse to establish ClickHouse Labs
Andy Pavlo announces that he is joining ClickHouse to establish and lead ClickHouse Labs, an industry research team focused on database systems. He describes ClickHouse’s early use of C++ and vectorized query execution with SIMD in 2016, and says its development has remained relevant to his academic research. Labs will work directly with ClickHouse engineers, customers, collaborators, and industry partners rather than operating separately, while also collaborating with the PostgreSQL team on performance and reliability for its managed service. An immediate priority is validating and pushing an existing backlog of ideas and optimizations into production, followed by research into hardware, algorithms, data structures, execution strategies, and DBMS development for AI and agentic workloads. The stated aim is to combine scientific research with technology that matters to users.
July 2026 newsletter
The July 2026 newsletter rounds up ClickHouse releases, community projects, benchmarks, engineering work, and upcoming events in real-time data warehousing. Version 26.6 adds hypothetical skip indexes, cascading refreshable materialized-view dependencies, experimental continuous queries, AI embedding functions, PNG output, CLI documentation, and geospatial improvements. A CostBench comparison running the full analytics path over 28 hours found better cost-performance for ClickHouse Cloud than Snowflake Interactive Tables, which required a continuously running refresh warehouse for pre-aggregated data. Other features cover conversational analytics with ClickHouse Cloud, LibreChat, and MCP; Postgres change-data capture through Debezium and Kafka; local agent queries with chDB; high-concurrency tuning; and Silk, whose fiber runtime targets lower I/O tail latency. Quick reads report ingestion, caching, security, and multi-tenant SQL developments.
Topics they write about
All topics →Weekly digest
The best engineering reads, once a week.