---
title: "Scalability"
description: "379 posts about Scalability, summarised, each linking to the original."
---

# Scalability
> 379 posts about Scalability, summarised, each linking to the original.

## Articles

### [Native random sampling in ClickHouse](https://yomu.fyi/post/native-random-sampling-in-clickhouse.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: Mark Needham
- Published: May 22, 2026
- Type: Tutorial

ClickHouse native random sampling enables aggregate queries to execute against a representative fraction of data when querying full datasets is too slow. To maintain sampling accuracy, tables require a sample key derived from high-cardinality columns using hash functions like sipHash64 to ensure an even distribution. The sample key must be included in the primary key ORDER BY expression, preferably at the front to let ClickHouse quickly skip index granules. Queries can sample data deterministically by specifying a fractional ratio or a minimum row threshold, scaling aggregate counts using the virtual column \_sample\_factor. In benchmark tests on a 30-million-row dataset, sampling 10% of records reduced query latency by approximately 80% while retaining approximate metrics suitable for exploratory analysis.


### [Making User-Sequence Data More Cost-Efficient, Faster, and Easier to Use](https://yomu.fyi/post/making-user-sequence-data-more-cost-efficient-faster-and-easier-to-use.md)
- Company: [Pinterest](https://yomu.fyi/company/pinterest.md)
- Author: Pinterest Engineering
- Published: May 21, 2026
- Type: Problem & solution

User sequences serve as core primitives powering machine learning models across ranking, retrieval, and recommendation surfaces at Pinterest. Managing these sequences previously caused high infrastructure costs, operational fragility, and data drift between offline training pipelines and online serving systems. In response, engineering teams redesigned the platform around a single definition executed across multiple runtimes, including real-time streaming, batch backfills, and online inference. The updated architecture pairs a unified enrichment engine with columnar time-partitioned storage and standardized validation tiers. Consequently, the redesign achieved lower storage and network costs, faster signal onboarding via configuration, and improved engagement metrics across major recommendation surfaces.


### [May 2026 newsletter](https://yomu.fyi/post/may-2026-newsletter.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: Mark Needham
- Published: May 21, 2026
- Type: Announcement

The May 2026 ClickHouse newsletter details enterprise observability migrations, ecosystem integrations, and core database enhancements. Digital banking platform Qonto migrated from Grafana Tempo to ClickHouse Cloud, compressing 231 TB of raw trace metadata to 376 GB while expanding query windows to two weeks and introducing an AI incident companion. Similarly, LINE MAN Wongnai consolidated multiple business clusters onto self-hosted ClickHouse, achieving 1.5 million rows per second peak ingest, 100% trace retention, and a 53% reduction in observability costs. Comparative log analytics benchmarks demonstrated that ClickHouse utilized 5x less disk space and achieved 4 to 6x faster cold query execution than Elasticsearch across 50 billion rows. Additional updates include the ClickHouse 26.4 release featuring improved SQL compatibility, new agent telemetry adapters in Mastra, and SQL-based alerting capabilities in ClickStack.


### [How Partition Access Visualizations Reduced our Data Lake S3 Cost by 33%](https://yomu.fyi/post/how-partition-access-visualizations-reduced-our-data-lake-s3-cost-by-3.md)
- Company: [Yelp](https://yomu.fyi/company/yelp.md)
- Author: Nick Del Nano, Data Streaming
- Published: May 21, 2026
- Type: Problem & solution

Data teams at Yelp struggled to identify table consumers and track how datasets were accessed across their petabyte-scale data lake. To solve this, the team built a usage attribution system using Amazon S3 server access logs to visualize time-based partition keys against access timestamps and IAM roles. These visualizations revealed distinctive patterns for daily batch jobs, backfills, and ad hoc queries, guiding migrations of thousands of tables to Apache Iceberg format. Furthermore, Yelp implemented a Default Access Retention strategy gated by IAM bucket policies to prevent accidental queries from resetting S3 Intelligent Tiering transitions. Combining these granular usage insights with targeted S3 storage classes reduced Yelp's S3 storage costs by 33 percent.


### [Scaling Airbnb’s identity graph with a unified knowledge graph infrastructure](https://yomu.fyi/post/scaling-airbnb-s-identity-graph-with-a-unified-knowledge-graph-infrast.md)
- Company: [Airbnb](https://yomu.fyi/company/airbnb.md)
- Author: Lucen Zhao
- Published: May 19, 2026

Airbnb migrated its massive identity graph from a third-party graph database vendor to a unified, in-house knowledge graph platform to solve long-tail latency and scaling bottlenecks. Built on JanusGraph with AWS DynamoDB for persistence and OpenSearch for indexing, the new architecture decouples storage operations from graph traversal logic. The migration improved P99 query latency, eliminated routine instance reboots, and supported a tenfold increase in write throughput.


### [Ready for your busiest day: How we scale](https://yomu.fyi/post/ready-for-your-busiest-day-how-we-scale.md)
- Company: [Intercom](https://yomu.fyi/company/intercom.md)
- Author: Ryan Sherlock
- Published: May 19, 2026
- Type: Explainer

Fin describes how its customer service platform is designed to absorb high-volume events while preserving customer isolation, availability, and operational control. At daily peaks, it handles over 150,000 customer requests per second and more than 70,000 asynchronous requests per second; its source-of-truth data spans 128 Vitess shards, while Elasticsearch supports over 650TB, 1.7 trillion documents, and peaks above 40,000 requests per second. The architecture favors AWS primitives, Vitess managed by PlanetScale, online and incremental migrations, fair queues, feature flags, heartbeat metrics, and automatic rollback, with roughly 250 production ships per workday and merge-to-production time under 10 minutes. The stated conclusion is that scale depends less on headline capacity than on clear scaling levers, tenant fairness, customer-outcome monitoring, reversible change, and routine infrastructure improvement without planned customer-impacting downtime.


### [How ChatFeatured migrated from PlanetScale Postgres to Postgres Managed by ClickHouse to power AI brand discovery](https://yomu.fyi/post/how-chatfeatured-migrated-from-planetscale-postgres-to-postgres-manage.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: ClickHouse
- Published: May 18, 2026
- Type: Problem & solution

ChatFeatured, a Toronto-based startup, helps brands track and improve how AI models discover and recommend them, including by analyzing cited sources and guiding content production. As prompt ingestion grew to about 1,000 prompts daily across six AI models, managed Postgres instances became slow, CPU reached 90% during ingestion windows, and PlanetScale hit an IOPS limit; analytics queries remained the deeper constraint. The team migrated from PlanetScale to Postgres managed by ClickHouse using the Postgres-to-Postgres ClickPipes connector, which handled initial load, schema transfer, continuous sync, and a five-minute maintenance window; the migration took 30 minutes overall. This setup retains Postgres for transactional workloads while providing a built-in path to ClickHouse for analytics, and a 30-day LifestyleRx query fell from about two and a half minutes to less than a second, with 20x compression reported for CDN log data.


### [Making LLMs faster without sacrificing accuracy](https://yomu.fyi/post/making-llms-faster-without-sacrificing-accuracy.md)
- Company: amazon.science
- Author: Tao Yu, Youngsuk Park
- Published: May 15, 2026
- Type: Problem & solution

Standard scaling laws optimize parameter count and training data budgets to minimize loss but ignore internal Transformer architecture configurations. To address throughput disparities among equally sized models, researchers developed a scaling framework incorporating hidden size, the multilayer perceptron to attention parameter ratio, and grouped-query attention. The method calibrates a correction surface over standard Chinchilla scaling laws to independently optimize separable architectural parameters for target accuracy. Evaluated across more than 200 models up to three billion parameters, the framework produced the Panda model family for maximum accuracy and the Surefire family for Pareto efficiency. Surefire models matched or exceeded LLaMA-3.2 accuracy while increasing inference throughput by 12% to 47% across vLLM and SGLang serving systems on modern GPUs.


### [How the D. E. Shaw group powers high-cardinality observability at scale with ClickHouse](https://yomu.fyi/post/how-the-d-e-shaw-group-powers-high-cardinality-observability-at-scale.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: ClickHouse
- Published: May 15, 2026
- Type: Explainer

Inside D. E. Shaw group, a global investment and technology development firm, millions of compute workloads run monthly across its high-performance computing infrastructure. To support capacity planning and resource allocation, it collects task-level metrics for requested and utilized compute, producing millions of distinct time series that its previous observability platform could not accommodate. Engineers benchmarked databases on identical hardware and datasets; ClickHouse ingested approximately 3.5 million samples per second on one node versus roughly 480,000 for a competing solution, while similar queries completed in seconds instead of timing out. They retained InfluxDB line protocol, batched messages into ClickHouse, and built disk-based tooling to transform and stream historical data; the resulting four-server NVMe cluster stores about 68 TB compressed and sustains 530k records per second on average. Materialized views support capacity-planning dashboards, while OpenTelemetry tracing extends the platform to other observability workloads, with SQL migration and data modeling requiring engineering effort.


### [ClickHouse vs Prometheus for High Cardinality, Part 2: Cardinality in ClickHouse](https://yomu.fyi/post/clickhouse-vs-prometheus-for-high-cardinality-part-2-cardinality-in-cl.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: Rory Crispin, Dale McDiarmid
- Published: May 15, 2026
- Type: Explainer

Traditional time-series databases like Prometheus experience write amplification, memory overhead, and operational challenges when high-cardinality label combinations churn rapidly. ClickHouse alters these economics by abandoning dedicated series objects in favor of a wide events model, storing telemetry as rows in columnar tables with dynamic Map attributes and metric measurements. Dynamic attributes can be handled using sharded Map types alongside materialized columns and text indexes for frequently queried labels. This design trades ingestion overhead for query-time processing, relying on columnar compression, partition pruning, and parallel scans to compute aggregations on demand. ClickHouse avoids the ingest-side bottlenecks of high cardinality, though Prometheus remains preferable for moderate cardinality, native PromQL semantics, and single-series lookups.


### [ClickHouse vs Prometheus for High Cardinality, Part 1: Understanding the Problem](https://yomu.fyi/post/clickhouse-vs-prometheus-for-high-cardinality-part-1-understanding-the.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: Rory Crispin, Dale McDiarmid
- Published: May 14, 2026
- Type: Explainer

Observability systems frequently encounter high cardinality when metrics contain numerous unique label combinations across multidimensional cloud infrastructure. In Prometheus, every unique label combination instantiates an independent time series that incurs metadata overhead, requires in-memory memSeries registration, and continually expands inverted index posting lists. The ingestion engine stores samples in two-hour Head blocks using compressed 120-sample chunks with XOR-based encodings, but rapid series creation and elevated scrape frequencies multiply chunk allocations and memory consumption. When executing read operations, broad aggregations and regular expressions require intersecting expansive posting lists and decoding entire chunks sequentially because the engine cannot push value predicates directly into compressed storage blocks. Consequently, managing high-cardinality workloads in series-oriented architectures forces operational trade-offs between system stability and granular visibility into ephemeral components.


### [ClickHouse Cloud: Fast, Updatable Lookups with the Join Table Engine](https://yomu.fyi/post/clickhouse-cloud-fast-updatable-lookups-with-the-join-table-engine.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: Hellmar Becker
- Published: May 12, 2026
- Type: Explainer

ClickHouse Cloud provides an in-memory Join table engine for precomputed hash-map lookups and joins, aimed at enriching append-only fact data with dimensions that change occasionally. With Join(ANY, LEFT, key), inserting another row for a key acts as an upsert, while joinGet and matching JOIN clauses use the in-memory structure. Open-source Join tables are not distributed and persist each INSERT batch as separate .bin files without automatic compaction, but ClickHouse Cloud transparently uses SharedJoin with MergeTree for ALL or ReplacingMergeTree for ANY. In the demonstrated enrichment query, duplicate inserts for id 1 leave value 42 visible, while the underlying table retains duplicates until merging; the in-memory load applies FINAL and always selects the latest entry, making ANY LEFT suitable for frequently updated dimensions.


### [We replaced Redis with MySQL for inventory reservations—and it scaled](https://yomu.fyi/post/we-replaced-redis-with-mysql-for-inventory-reservations-and-it-scaled.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: May 12, 2026
- Type: Problem & solution

Shopify replaced its Redis-based inventory reservation system with MySQL to unify reservations with the inventory ledger and provide ACID coordination between payment and inventory updates. The new design stores one row per sellable unit in a bounded pool of up to 1,000 rows per item/location combination, uses MySQL 8's SKIP LOCKED to avoid waiting on locked units, and replenishes depleted pools. Composite primary keys reduced row locks, READ COMMITTED avoided gap locks that interfered with replenishment, and consistent lock ordering prevented deadlocks. The investigation found connection-pool usage elsewhere in checkout, rather than reservation query speed alone, was the immediate ceiling; cleanup removed 50% of primary-database reads and 33% of transactions, while configuration changes added headroom. Shopify validated the design in shadow mode, then rolled it out gradually with a Redis rollback path, meeting high-throughput targets during peak 2025 traffic.


### [Powering self-driving vehicle analytics at Avride with ClickHouse Cloud](https://yomu.fyi/post/powering-self-driving-vehicle-analytics-at-avride-with-clickhouse-clou.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: ClickHouse
- Published: May 11, 2026
- Type: Problem & solution

Avride’s expanding autonomous passenger vehicle and delivery robot fleet generated thousands of data points per minute, overwhelming its legacy Apache Iceberg and Parquet metadata indexing architecture. Iceberg’s optimistic concurrency model struggled with parallel writes, pushing ingestion latencies to hours or days, inflating query times to 10–20 seconds, and requiring raw data duplication across storage locations. To resolve these bottlenecks, Avride built a custom indexing and metrics warehouse powered by ClickHouse Cloud on AWS backed by Amazon S3 object storage. The new architecture dropped ingestion delays to seconds, reduced worst-case lookup latencies to under one second, eliminated the need for bespoke caching layers, and removed payload duplication. Beyond operational analytics, the unified ClickHouse platform now supports internal ride-replay debugging tools, simulation workflows, and high-cardinality vehicle C++ performance trace profiling.


### [AI Is Changing What Customers Need From a Database. MongoDB 8.3 Is Built for It](https://yomu.fyi/post/ai-is-changing-what-customers-need-from-a-database-mongodb-8-3-is-buil.md)
- Company: [MongoDB](https://yomu.fyi/company/mongodb.md)
- Author: Ben Cefalo
- Published: May 7, 2026
- Type: Announcement

MongoDB announced the general availability of MongoDB 8.3, designed to support high-throughput AI workloads without requiring application code changes. Compared to version 8.0, the release provides a 35 percent boost in write throughput, a 45 percent increase in read speeds, and 15 percent faster ACID transactions. Additionally, cross-region connectivity for AWS PrivateLink is now generally available, enabling private traffic routing between Atlas clusters across distinct AWS regions without public internet exposure. MongoDB Atlas supports deployments across 130 regions spanning AWS, Google Cloud, and Microsoft Azure, including multi-cloud cluster topologies. These infrastructure updates target enterprise demands for sub-100ms retrieval, sub-second context updates, zero downtime, and strict data residency compliance.


### [Navigating uncertainty in Amazon's middle-mile network](https://yomu.fyi/post/navigating-uncertainty-in-amazon-apos-s-middle-mile-network.md)
- Company: amazon.science
- Author: Ruth Misener, Hana Ku, Georgios Paschos
- Published: May 6, 2026
- Type: Problem & solution

Amazon's middle-mile logistics network coordinates tens of millions of packages across hundreds of facilities while managing subtle demand variations and major operational disruptions. To solve this mixed-integer optimization challenge without combinatorial explosion, Amazon models routes at coarse time resolutions augmented by precomputed 15-minute timing bounds. The system uses Monte Carlo methods alongside a graph attention network featuring interconnected site graphs and origin-destination graphs to capture complex spatial correlations and flow interdependencies. By generating synthetic flow permutations, planners stress-test candidate network designs against hundreds of plausible scenarios to prioritize routing optionality. This risk-aware modeling approach preserves network flexibility, absorbs severe shocks, and yields potential cost savings while safeguarding delivery promises.


### [Agentic analytics starts with query-ready data: the write-side cost of Snowflake vs. ClickHouse](https://yomu.fyi/post/agentic-analytics-starts-with-query-ready-data-the-write-side-cost-of.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: Tom Schreiber, Lionel Palacin
- Published: May 6, 2026
- Type: Benchmark

Agentic analytics workflows introduce bursts of concurrent, low-latency analytical queries over continuously ingested data. Analytical query efficiency relies on columnar storage and chunk-level pruning, which require contiguous, sorted data to avoid full-table scans. Rather than evaluating warehouses solely at query runtime, this benchmark compares the write-side cost of maintaining query-ready data in ClickHouse and Snowflake during continuous ingestion at approximately one million rows per second. Snowflake relies on post-write clustering to organize data, whereas ClickHouse constructs sorted layouts on the write path and refines them via background merges. Across 100B, 200B, and 300B row checkpoints on the ClickBench dataset, ClickHouse achieved a query-ready layout at 22 times lower cost and delivered 28 times better write-side cost-performance.


### [How mechanism design theory helps optimize Amazon-vendor collaboration](https://yomu.fyi/post/how-mechanism-design-theory-helps-optimize-amazon-vendor-collaboration.md)
- Company: amazon.science
- Author: Dirk Bergemann
- Published: May 5, 2026
- Type: Problem & solution

Amazon and its vendors face an asymmetric information problem when coordinating purchase orders across fulfillment centers because neither party shares proprietary cost, capacity, or demand data. To solve this coordination gap without exposing sensitive information, Amazon's Supply Chain Optimization Technologies organization designed Flo Pro by integrating the Vickrey-Clarke-Groves mechanism with the Consensus Planning Protocol. The distributed optimization framework uses the alternating-direction method of multipliers to coordinate iterative agent responses, calculating cost-benefit transfers across rolling six-week planning horizons. In lower-dimensional settings, Amazon also evaluated a transparent menu-of-contracts approach where vendors select optimal plans priced at Amazon's incurred costs. A nine-week pilot with a consumer-product manufacturer confirmed that Flo Pro successfully aligns incentives and achieves measurable cost savings.


### [Gala supercharges analytics performance with ClickHouse on AWS](https://yomu.fyi/post/gala-supercharges-analytics-performance-with-clickhouse-on-aws.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: ClickHouse
- Published: May 4, 2026
- Type: Problem & solution

Blockchain gaming platform Gala struggled to ingest and query expanding volumes of telemetry and blockchain data on its Databricks-based data infrastructure, resulting in multi-minute query times and heavy engineering maintenance overhead. To solve these bottlenecks, the company migrated its data platform to ClickHouse on AWS, ingesting data through Kafka, Airbyte, Amazon S3, and Fivetran. Gala leveraged ClickHouse's indexing capabilities alongside saved queries and API endpoints to power company-wide Metabase business intelligence dashboards for non-technical teams. Following the migration completed in December 2024, data capacity expanded from 3 TB to 9 TB while query times dropped from minutes to sub-second on previously unoptimized tables. The switch also lowered initial infrastructure costs by 30 percent, with plans underway to implement ClickHouse ClickPipes for improved data processing pipelines.


### [Optimizing ML Workload Network Efficiency (Part I): Feature Trimmer](https://yomu.fyi/post/optimizing-ml-workload-network-efficiency-part-i-feature-trimmer.md)
- Company: [Pinterest](https://yomu.fyi/company/pinterest.md)
- Author: Pinterest Engineering
- Published: May 1, 2026
- Type: Problem & solution

Pinterest deployed a root-leaf architecture for online machine learning serving to decouple CPU-based feature preprocessing from GPU-based inference. Because the root cluster retrieved a union of features across models and fanned out the full set to every leaf, network bandwidth became a severe bottleneck. After initial tests with fbthrift LZ4 compression yielded a 20% bandwidth reduction at the cost of CPU and latency overhead, the team implemented Feature Trimmer. This mechanism uses exported model signatures as the source of truth to prune unused features on the root before transmission. By dispatching only required features per model, the system reduced network bandwidth, decreased latency, and enabled cluster rightsizing that saved over $4M annually.


[Newer posts](https://yomu.fyi/topic/scalability/page/7.md) · [Older posts](https://yomu.fyi/topic/scalability/page/9.md)