---
title: "Scalability"
description: "379 posts about Scalability, summarised, each linking to the original."
---

# Scalability
> 379 posts about Scalability, summarised, each linking to the original.

## Articles

### [How we’re building a data platform for a new user: agents](https://yomu.fyi/post/how-we-re-building-a-data-platform-for-a-new-user-agents.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: Al Brown
- Published: Mar 27, 2026
- Type: Explainer

AI agents are emerging as autonomous database users that generate SQL, execute queries, and iterate in seconds. Because traditional analytics workflows built for human analysts cannot keep up with these rapid loops, ClickHouse is adapting its platform for agentic workloads. The proposed Agentic Data Stack combines LibreChat for conversational interfaces, ClickHouse with Model Context Protocol servers for the data layer, and Langfuse for observability. Supporting developer tooling includes chDB 4 with a Pandas-like API, IDE integrations, and ClickHouse Agent Skills to enforce database design best practices. Platforms like Anthropic and Character.AI already leverage ClickHouse's high-concurrency aggregation and real-time ingestion capabilities to handle massive data scales required by modern AI applications.


### [Top 10 best practices tips for ClickHouse](https://yomu.fyi/post/top-10-best-practices-tips-for-clickhouse.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: Yonatan Dolan
- Published: Mar 26, 2026
- Type: Explainer

ClickHouse delivers rapid analytical queries across massive datasets, but suboptimal schemas and queries can significantly reduce performance and storage efficiency. Aligning table definitions with ClickHouse storage and execution mechanics resolves these bottlenecks. Key practices include placing low-cardinality columns first in the ORDER BY clause to enable granule pruning, and selecting optimal data types while avoiding Nullable wrappers to cut memory overhead. Developers should treat partitioning as a data management tool for TTL drops or merge engines rather than a generic speed booster, preventing fragmented parts from slowing query execution. Recent improvements like the Analyzer query planner have also made joins viable, though dictionaries remain faster for static lookups.


### [More Visibility, Less Guesswork: ClickHouse Cloud's New Monitoring Capabilities](https://yomu.fyi/post/more-visibility-less-guesswork-clickhouse-cloud-s-new-monitoring-capab.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: Mihir Gokhale
- Published: Mar 25, 2026
- Type: Announcement

ClickHouse Cloud released new monitoring capabilities and dashboards to give platform administrators deeper visibility into deployment health and server behavior. A newly introduced Overview page consolidates critical signals into a unified view for rapid health checks. Additionally, an updated Infrastructure page tracks service scaling over time, presenting CPU and memory utilization with new aggregation options alongside modals explaining automatic scaling decisions. The platform now sends automated email and Slack notifications when deployments encounter risks such as excessive parts, failed mutations, or query concurrency saturation. These additions allow administrators to proactively identify operational bottlenecks and configure scaling limits without spending excessive time searching for telemetry data.


### [Intelligent security at ClickHouse speed: How Cogent Security built an AI-native vulnerability management platform](https://yomu.fyi/post/intelligent-security-at-clickhouse-speed-how-cogent-security-built-an.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: ClickHouse
- Published: Mar 24, 2026
- Type: Problem & solution

Enterprise security defenders face shrinking exploitation windows alongside massive volumes of vulnerability findings across disconnected tools. To overcome query bottlenecks from traditional relational databases, Cogent Security designed a lakehouse serving architecture using Apache Iceberg and per-customer ClickHouse instances. The architecture relies on query-driven denormalized tables, custom compression codecs, and pre-computed projections to serve hundreds of millions of rows with sub-second latency. On top of this data layer, Cogent implemented an agentic text-to-SQL system that iteratively queries live data, references an in-house Ontology Service for semantic context, and applies majority-voted evaluations. This combination improved text-to-SQL accuracy from 40% to 94% while providing sub-three-second response times across 98.7% of queries at 100 million rows.


### [Smarter Auto-Scaling for ClickHouse: The Two-Window Approach](https://yomu.fyi/post/smarter-auto-scaling-for-clickhouse-the-two-window-approach.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: Ashwath Singh, Manas Alekar
- Published: Mar 24, 2026
- Type: Problem & solution

ClickHouse Cloud required a vertical auto-scaling mechanism that preserves rapid scale-ups while avoiding prolonged over-provisioning after traffic drops. Operating on a single 30-hour lookback window forced scale-downs to take up to 30 hours, whereas an earlier fixed-factor CPU algorithm caused severe oscillations. The engineering team deployed a dual-window recommender pairing a 3-hour window for responsive scale-downs with a 30-hour window for single-step scale-ups. Alongside a trend-based hunting check to resolve conflicting window recommendations, the team introduced target-tracking CPU scaling using geometric mean watermarks and 10-minute median smoothing. This dual-window architecture reduced scale-down latency to 3 hours, prevented scaling oscillations, and significantly reduced infrastructure costs for variable database workloads.


### [Structured Logging in .NET with Serilog and ClickHouse](https://yomu.fyi/post/structured-logging-in-net-with-serilog-and-clickhouse.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: Alex Soffronow Pagonidis
- Published: Mar 23, 2026
- Type: Tutorial

Traditional logging solutions like Seq, ELK, and cloud platforms often face scaling bottlenecks, including slower queries, higher operational costs, and complex pipeline management. To address this, an ASP.NET service can write structured logs directly to ClickHouse using Serilog and the Serilog.Sinks.ClickHouse library. The sink transmits data over HTTP and provides a fluent C# builder to configure table schemas, typed JSON columns, full-text indexes, and MergeTree engine parameters. ClickHouse stores log data in a columnar format that delivers 10 to 20 times compression while enabling direct SQL queries, JSON property filtering, and real-time aggregations through materialized views. Furthermore, the sink includes an in-memory buffer to handle transient network drops with configurable batching, retry mechanisms, and callback hooks for failure alerting.


### [March 2026 newsletter](https://yomu.fyi/post/march-2026-newsletter.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: Mark Needham
- Published: Mar 19, 2026
- Type: Announcement

The March 2026 ClickHouse newsletter details major platform updates, community deployments, and architectural practices for high-throughput analytics. Recent product announcements include ClickHouse 26.2 featuring time-based block flushing for streaming feeds, general availability of native inverted indexes for full-text search, and the release of chDB 4 with a Pandas-compatible DataStore API. Community implementations show Cloudflare managing 1.61 quadrillion daily events, while Trigger.dev isolated multi-tenant SQL execution using a domain-specific query language that compiles safe ClickHouse queries. In data optimization techniques, Hookdeck accelerated webhook payload search from over 30 seconds to under 400 milliseconds by partitioning payload hashes into typed bucket columns and scanning iterative time windows. Additional operational guidance emphasizes setting the ttl\_only\_drop\_parts parameter and aligning table partitioning with time units so background merges drop entire data parts cleanly without triggering mutation rewrites.


### [How Socialpruf built a faster, more reliable data stack by replacing Neon with Postgres managed by ClickHouse](https://yomu.fyi/post/how-socialpruf-built-a-faster-more-reliable-data-stack-by-replacing-ne.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: ClickHouse
- Published: Mar 17, 2026
- Type: Problem & solution

Socialpruf, a social media performance tracking platform ingesting hundreds of posts per second, experienced analytics latency, connection dropouts, and high network transfer costs while running Postgres on Neon alongside ClickHouse Cloud. To resolve these issues, the team migrated their transactional system of record to NVMe-backed Postgres managed by ClickHouse, colocating transactional and analytical workloads. Although standard Postgres logical replication failed on their 0.5 TB dataset, they achieved continuous synchronization using PeerDB before conducting a production cutover. The migration eliminated connection dropouts, reduced network transfer fees, and improved overall Postgres query performance by approximately 30 percent. To manage thousands of concurrent connections generated by ingestion workers, Socialpruf implemented ClickHouse's bundled, horizontally scalable PgBouncer instances.


### [AI is redrawing the database market](https://yomu.fyi/post/ai-is-redrawing-the-database-market.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: Tanya Bragin
- Published: Mar 17, 2026
- Type: Explainer

Emerging artificial intelligence workloads are fundamentally transforming data platform requirements across real-time analytics, data warehousing, and observability. Conversational analytics and autonomous agents replace periodic batch queries with rapid bursts of concurrent, low-latency requests, exposing the architectural limits of traditional cloud data warehouses and standalone transactional databases. To resolve these constraints, architectures are converging on engine-level integrations between Postgres for transactions and ClickHouse for analytical scale, while unifying operational observability and business event storage. ClickHouse also incorporates turnkey agent interfaces through its acquisition of LibreChat alongside dedicated execution tracing from tools like Langfuse. These combined capabilities ensure organizations can reliably support interactive agentic workflows without incurring severe latency penalties or compounding infrastructure costs.


### [Designing the new async-native ClickHouse Python client](https://yomu.fyi/post/designing-the-new-async-native-clickhouse-python-client.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: Joe Spadola
- Published: Mar 16, 2026
- Type: Problem & solution

The official ClickHouse Python client, clickhouse-connect, originally addressed asynchronous execution by wrapping its synchronous HTTP client in a thread pool executor. However, this executor model experienced thread exhaustion, GIL contention, and severe tail latency variance under heavy concurrent workloads. To resolve these scalability bottlenecks, the developers redesigned the client around a half-sync/half-async pattern utilizing aiohttp for async network I/O. A custom bounded queue bridges streaming socket reads on the event loop with synchronous, CPU-intensive binary parsing running inside background threads. Benchmarks demonstrate that this architecture stabilizes tail latencies, delivering an average P95 latency of 556 milliseconds compared to 869 milliseconds for the legacy executor client.


### [AI doesn’t always generate perfect ClickHouse schemas (yet)](https://yomu.fyi/post/ai-doesn-t-always-generate-perfect-clickhouse-schemas-yet.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: Al Brown
- Published: Mar 13, 2026
- Type: Explainer

LLMs can generate syntactically valid ClickHouse schemas whose advanced choices are poorly matched to a workload. The post examines recurring pitfalls including query-driven partitioning, per-column codecs, premature projections, materialized-view sprawl, JSON configurations, and forced deduplication with OPTIMIZE TABLE ... FINAL. It explains that partitions primarily support data management, incremental materialized views add work to every insert, and ReplacingMergeTree deduplicates during background merges rather than at insertion. The recommended method is to begin with a simple MergeTree or ReplacingMergeTree, a well-chosen ORDER BY, default compression, and no optional structures, then measure production behavior before adding complexity or seeking human expertise for high-scale, interconnected designs.


### [Shopify’s journey to faster breadth-first GraphQL execution](https://yomu.fyi/post/shopify-s-journey-to-faster-breadth-first-graphql-execution.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Mar 12, 2026
- Type: Explainer

Shopify describes how high-cardinality, deeply nested GraphQL queries exposed execution overhead beyond data loading, with field resolvers assembling responses consuming much of request time. Its GraphQL Cardinal engine replaces conventional depth-first traversal with breadth-first execution, resolving each selected field once across an aggregated set of objects and reducing repeated field-level work. For large list queries, Shopify reports up to 15x faster execution, 90% less memory use, and reductions of many seconds in P50 latency. The migration included a tuned interpreter, cheaper tracer behavior, shadow verification, benchmarks, and ongoing conversion of tens of thousands of legacy resolvers, while future work includes asynchronous patterns and lower-level C bindings.


### [Announcing General Availability of ClickHouse Full-text Search](https://yomu.fyi/post/announcing-general-availability-of-clickhouse-full-text-search.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: Melvyn Peignon
- Published: Mar 10, 2026
- Type: Announcement

ClickHouse has reached general availability for its native Full-text Search feature, designed to accelerate token-based text filtering across large datasets. The capability relies on deterministic inverted indexes rather than probabilistic Bloom filter skip indexes, mapping individual tokens directly to row numbers. It supports plain strings, string arrays, and map keys or values while allowing configurable pre-processing and tokenization using standard SQL expressions. The system does not implement relevance scoring or store positional data for ranking, focusing instead on rapid filtering paired with large-scale analytical aggregations. Testing shows significant reductions in scanned granules alongside substantial latency improvements, though text indexes require more storage than Bloom filters.


### [How Critical Manufacturing uses ClickHouse to bring real-time intelligence to the factory floor](https://yomu.fyi/post/how-critical-manufacturing-uses-clickhouse-to-bring-real-time-intellig.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: ClickHouse
- Published: Mar 10, 2026
- Type: Problem & solution

Critical Manufacturing faced performance and scaling bottlenecks while processing millions of factory floor events using a legacy Microsoft SQL Server analytics stack. Increasing data volumes caused historical queries to take hours, straining real-time dashboards and escalating operational maintenance. The engineering team replaced this architecture by migrating analytics workloads to ClickHouse connected directly to streaming Kafka topics. They implemented an ELT pattern alongside ReplacingMergeTree engines to deduplicate late-arriving sensor records and utilized TTL policies for automated lifecycle management. The resulting platform delivers sub-second analytical queries across historical and real-time operational data, expands into OpenTelemetry observability storage, and maintains embeddings for retrieval-augmented generation workloads.


### [What Replo learned optimizing 100+ billion events in ClickHouse](https://yomu.fyi/post/what-replo-learned-optimizing-100-billion-events-in-clickhouse.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: ClickHouse
- Published: Mar 9, 2026
- Type: Problem & solution

Replo built an analytics pipeline on ClickHouse to serve more than 4,000 Shopify merchants processing 3,000 to 5,000 events per second. The initial single-table architecture recalculated session metrics on the fly, leading to poor query efficiency as volume scaled. To speed up dashboard queries, the team implemented a mark-and-unmark pattern with SummingMergeTree and refreshable materialized views, but real-time deduplication and fractional attribution caused runaway query backlog loops. Replo resolved the issue by limiting real-time recomputations to a specialized table holding only the last 40 minutes of purchase-related events. This architecture allowed processing over 100 billion records while keeping query times fast and ingestion lag to approximately one minute.


### [Ulysses Sequence Parallelism: Training with Million-Token Contexts](https://yomu.fyi/post/ulysses-sequence-parallelism-training-with-million-token-contexts.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Kashif Rasul, Stas Bekman
- Published: Mar 9, 2026
- Type: Explainer

Ulysses Sequence Parallelism addresses the memory and compute pressure of training language models on sequences extending beyond tens of thousands of tokens, where attention remains quadratic in sequence length even with FlashAttention. It shards tokens across GPUs, projects local queries, keys, and values, then uses two all-to-all operations to exchange data so each GPU computes a subset of attention heads before restoring sequence-sharded outputs. The post describes integrations with Accelerate, the Transformers Trainer, and TRL's SFTTrainer, including variable-length handling and weighted loss aggregation for uneven valid-token counts. In reported four-GPU tests, SP=4 reduced per-GPU memory sufficiently for 96K-token training on H100 80GB GPUs, reached 13,396 tokens per second at 64K, and matched non-SP canonical token-normalized loss within 0.000005 maximum absolute difference.


### [Designing flexible payment flows with Checkout Session](https://yomu.fyi/post/designing-flexible-payment-flows-with-checkout-session.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: Allison Farris
- Published: Mar 6, 2026
- Type: Tutorial

The post addresses the architectural challenge of taking a digital product business global while supporting localized tax, currencies, one-time purchases, subscriptions, and a branded embedded checkout. It recommends Checkout Sessions as the orchestration layer, with Stripe creating and managing the underlying Payment Intent while declarative parameters handle features such as Stripe Tax, Adaptive Pricing, subscriptions, and promotional codes. The implementation creates a session with ui\_mode, mode, automatic\_tax, adaptive\_pricing, line items, and optional payment\_intent\_data, then passes the client secret to Stripe Elements for the Payment Element and Currency Selector. This approach supports localized payment methods and currency conversion without a backend rewrite, while direct Payment Intents remain appropriate when an application needs full control over carts, discounts, taxes, or its own checkout abstraction.


### [PRX Part 3 — Training a Text-to-Image Model in 24h!](https://yomu.fyi/post/prx-part-3-training-a-text-to-image-model-in-24h.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: David Bertoin, Roman Frigg, Jon Almazán
- Published: Mar 3, 2026
- Type: Benchmark

PRX Part 3 reports a 24-hour speedrun for training a text-to-image diffusion model, combining techniques that performed well in earlier experiments under a budget of about $1,500 on 32 H200 GPUs. The recipe uses x-prediction in pixel space with 32-pixel patches and a 256-dimensional initial bottleneck, followed by LPIPS and DINOv2 perceptual losses, TREAD token routing, REPA alignment with DINOv3, and Muon for matrix parameters while Adam handles the rest. Training ran for 100,000 steps at 512px with batch size 1,024, then 20,000 steps at 1024px with batch size 512 and no REPA, across three synthetic datasets. The resulting model is described as usable, with strong prompt following and consistent aesthetics, though texture glitches, anatomy issues, and difficulty with hard prompts remain. The post attributes these limitations mainly to undertraining and limited data diversity and presents the recipe as a foundation for further scaling.


### [The generative recommender behind Shopify's commerce engine](https://yomu.fyi/post/the-generative-recommender-behind-shopify-s-commerce-engine.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Feb 25, 2026
- Type: Explainer

Shopify describes a foundational generative recommender that predicts next products, and in some contexts ads, from buyer event sequences spanning searches, views, add-to-carts, favorites, and purchases. The system uses an autoregressive model with causal masking, time-aware attention combining RoPE-inspired rotary encoding with relative attention bias, and sampled softmax with shared and positive-aware hard negatives. Built on HSTU, it combines richer activity data, ensemble-focused training for incremental recall, and optimized CUDA kernels; the full training pipeline reached up to 7.3x the speed of a baseline implementation. In an online test of an August model version, relative shop orders rose 0.94%, high-quality click-through rate rose 5%, conversion rate rose 0.71%, and final served product recall at 2 rose 4.8%.


### [Scaling Localization with AI at Lyft](https://yomu.fyi/post/scaling-localization-with-ai-at-lyft.md)
- Company: [Lyft](https://yomu.fyi/company/lyft.md)
- Author: Stefan Zier
- Published: Feb 19, 2026
- Type: Problem & solution

Lyft re-architected its batch localization pipeline to combine artificial intelligence with professional linguist oversight. The previous workflow relied exclusively on human translation, which resulted in multi-day turnarounds and linear cost scaling that hindered rapid market expansions. Under the new architecture, a dual-path system generates three candidate translations using a fast generation model, evaluates them against defined rubrics using a reasoning-focused model, and immediately releases approved drafts. Linguists review the drafts asynchronously in the Smartling translation management system to establish the final system of record. This pipeline reduced delivery latency from days to minutes while achieving a state where 95% of machine-generated translations require no significant changes during human review.


[Newer posts](https://yomu.fyi/topic/scalability/page/9.md) · [Older posts](https://yomu.fyi/topic/scalability/page/11.md)