---
title: "Scalability"
description: "379 posts about Scalability, summarised, each linking to the original."
---

# Scalability
> 379 posts about Scalability, summarised, each linking to the original.

## Articles

### [Goodbye limitations, hello data: How Qonto is rethinking observability with ClickHouse Cloud](https://yomu.fyi/post/goodbye-limitations-hello-data-how-qonto-is-rethinking-observability-w.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: ClickHouse Team
- Published: May 1, 2026
- Type: Problem & solution

Qonto struggled with slow queries, system instability, and aggressive data sampling when handling distributed traces in Grafana Tempo. To enable wide-event querying without strict cardinality limits, the team migrated their telemetry storage to ClickHouse Cloud. Telemetry from applications, Kubernetes, and GitHub routes through OpenTelemetry collectors over AWS PrivateLink into ClickHouse, while Grafana remains the primary dashboard interface. By compressing 231 TB of uncompressed trace attributes down to 376 GB, Qonto achieved substantial storage savings and integrated an AI incident companion using the ClickHouse Model Context Protocol. The new architecture eliminated cardinality policing, allowed product and support teams to self-serve incident investigations, and enabled real-time streaming pre-aggregations using Apache Flink.


### [ClickHouse at Google Cloud Next '26](https://yomu.fyi/post/clickhouse-at-google-cloud-next-26.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: ClickHouse
- Published: Apr 27, 2026
- Type: Announcement

ClickHouse announced several product updates and integrations designed to deepen native compatibility with Google Cloud. Migration of ClickHouse Cloud to Google Axion ARM-based processors demonstrated a 30 to 55 percent query speed increase and reduced data load times from 64 to 26 seconds on ClickBench benchmarks. An integration with Google Lakehouse Runtime Catalog enables querying Iceberg tables on Google Cloud Storage via the DataLakeCatalog engine without data movement. Furthermore, Bring Your Own Cloud reached general availability on Google Cloud Marketplace, and ClickPipes added an unordered mode using Pub/Sub notifications for GCS ingestion. Google Antigravity integrated ClickHouse through an MCP server to translate natural language into SQL and generate interactive React charts.


### [ClickHouse Cloud on Google Cloud Now Powered by Google Axion Processors: 30–55% Faster Queries, ~15% Fewer Compute Credits](https://yomu.fyi/post/clickhouse-cloud-on-google-cloud-now-powered-by-google-axion-processor.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: ClickHouse
- Published: Apr 22, 2026
- Type: Announcement

ClickHouse Cloud transitioned its Google Cloud infrastructure from x86 N2D instances to Arm-based Google Axion C4A instances to resolve CPU throttling and performance ceilings under peak query loads. The migration leverages Axion's dedicated physical cores per vCPU, higher memory bandwidth, local SSDs, and improved thermal and clock frequency stability. In automated ClickBench testing across 100 million rows, every query in the 43-query benchmark improved by 30% to 55%, while data load times decreased by approximately half. A production analysis of 38 migrated customer services demonstrated an average 15% reduction in compute credits consumed per query across diverse production workloads. ClickHouse Cloud executed this rollout using automated validation checks, scheduled maintenance windows, and capacity coordination with Google Cloud.


### [Index sharding in ClickHouse Cloud: Petabyte-scale data needs petabyte-scale indexing](https://yomu.fyi/post/index-sharding-in-clickhouse-cloud-petabyte-scale-data-needs-petabyte.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: James Cunningham
- Published: Apr 21, 2026
- Type: Problem & solution

In large-scale ClickHouse Cloud deployments, replicas traditionally load entire primary and secondary indexes from object storage into working memory, creating massive redundant memory overhead as fleets expand. Index sharding resolves this bottleneck by distributing the index analysis phase across available replicas using consistent hashing over data parts. Instead of every node loading full index structures, each replica loads and analyzes only its assigned fraction of primary keys, bloom filters, text indexes, or vector indexes. The query initiator merges the resulting granule match ranges, allowing parallel replicas to stream data without any single machine retaining the whole index. This approach freed working memory for query execution and accelerated index analysis by up to 7.7x in benchmark tests on a 50 billion row table.


### [ClickHouse expands strategic collaboration with Google Cloud, bringing deeper integration, cloud flexibility, and next-generation performance](https://yomu.fyi/post/clickhouse-expands-strategic-collaboration-with-google-cloud-bringing.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: Tanya Bragin
- Published: Apr 21, 2026
- Type: Announcement

ClickHouse announced an expanded strategic collaboration with Google Cloud introducing four core architectural milestones. First, native integration with Google Cloud Lakehouse allows direct sub-second queries across data lakes and ClickHouse deployments without ETL pipelines or data duplication. Second, the Bring Your Own Cloud deployment model lets organizations run managed ClickHouse Cloud inside their own Google Cloud Virtual Private Cloud to retain data sovereignty and network controls. Third, ClickHouse Cloud is migrating automatically to Google's custom Arm-based Axion processors, boosting query throughput and cost efficiency for analytical workloads without requiring application changes. Finally, a new ClickHouse MCP server integration connects ClickHouse with Google Antigravity to enhance developer and analyst workflows.


### [What Is a Transactional Database?](https://yomu.fyi/post/what-is-a-transactional-database.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Apr 20, 2026
- Type: Explainer

Transactional databases handle large volumes of short, real-time read and write operations for systems such as orders, payments, account balances, and customer records. They support online transaction processing (OLTP) by treating related steps as one logical unit: successful work commits, while failures roll back the full transaction. ACID—atomicity, consistency, isolation, and durability—protects integrity under concurrency and failures, with row-oriented storage optimized for individual-record access, locking and isolation levels coordinating concurrent work, and write-ahead logging (WAL) supporting crash recovery. The source positions these databases as foundations for banking, e-commerce, healthcare, reservations, and telecommunications, while noting that row-oriented execution is less suitable for large analytical workloads and that horizontal scaling is more complex than in distributed NoSQL systems.


### [How we built Stripe Credits: A programmable, auditable way to pay your Stripe fees](https://yomu.fyi/post/how-we-built-stripe-credits-a-programmable-auditable-way-to-pay-your-s.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: Pratik Gupta
- Published: Apr 16, 2026
- Type: Explainer

Stripe Fee Credits replaces ad-hoc discounts and manual cash adjustments with a programmable, auditable virtual balance dedicated to eligible Stripe fees. The architecture has three layers: a standalone gRPC credits engine for metadata, policies, lifecycle, and audit history; logical financial accounts built on Money Movement and Storage; and fee-settlement orchestration that applies eligible credits before the primary balance. Logical accounts carry ledger balances without representing real cash, while platform checks restrict Originated Money Transfers to logical-to-logical movements and exclude these accounts from cash sweeps. Prepaid and Stripe-funded credits share the user experience but use separate funds flows: prepayments create deferred revenue and collect tax at drawdown, whereas Stripe-funded credits are booked as Stripe costs with taxes written off. The resulting infrastructure supports incentives, prepaid annual fees, and other programs across 26+ countries, with Dashboard balances and real-time auditability, while avoiding bespoke engineering for each commercial model.


### [New Research Reveals Overcoming Legacy Tech Issues Key to AI Success](https://yomu.fyi/post/new-research-reveals-overcoming-legacy-tech-issues-key-to-ai-success.md)
- Company: [MongoDB](https://yomu.fyi/company/mongodb.md)
- Author: Dr. William Lee
- Published: Apr 14, 2026
- Type: Benchmark

An IDC survey of 1,400 organizations across eight Asia/Pacific markets reveals that legacy architectures and accumulated data debt severely hinder enterprise artificial intelligence initiatives. While 43% of surveyed organizations cite legacy systems as a major obstacle to application development, 95% report project delays and 90% have experienced failed modernization attempts. In contrast, high-performing organizations treat modernization as a continuous discipline across people, processes, and cloud-centric data platforms capable of processing both structured and unstructured workloads. These leading enterprises achieve nearly three times more digital revenue growth than peers still constrained by technical debt. IDC notes that addressing technical and data debt proactively avoids escalating AI failure rates and delivers superior return on investment compared to superficial lift-and-shift migrations.


### [Selective Test Execution at Stripe: Fast CI for a 50M-line Ruby monorepo](https://yomu.fyi/post/selective-test-execution-at-stripe-fast-ci-for-a-50m-line-ruby-monorep.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: Aditya Anchuri
- Published: Apr 9, 2026
- Type: Problem & solution

Stripe’s Ruby monorepo contains about 50 million lines, 100,000 test files, and roughly 1.2 million test units, making sequential execution impractical at an estimated four months. To keep CI within minutes, Selective Test Execution (STE) runs tests whose recorded file accesses intersect files changed since a suitable baseline, averaging about 5% of the suite and under 10% of always-run compute. A dynamically linked C++ library loaded with LD\_PRELOAD intercepts file opens, associates them with hierarchical test scopes, and propagates tracing to child processes; logs are aggregated into a selection index outside the syscall hot path. Guardrails cover tests affected by directory discovery, previously failing tests, selective linting, generated files, and reproducible database-backed baselines ordered by Monotonic Revision IDs. The system is presented as a way to preserve safety and affordability while scaling CI for a highly dynamic language.


### [The three villains to agentic observability: retention, sampling and rollups](https://yomu.fyi/post/the-three-villains-to-agentic-observability-retention-sampling-and-rol.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: Mike Shi
- Published: Apr 8, 2026
- Type: Problem & solution

Observability practices increasingly depend on data sampling, short retention windows, and rollups to control storage costs. While human operators historically compensated for these compromises with intuition, AI agents lack implicit operational context and require full-fidelity telemetry to diagnose issues and justify conclusions. Head sampling drops unobserved spans early, tail sampling risks missing implicit non-error anomalies, and aggressive log expiration prevents long-term cyclical trend analysis. Addressing these constraints requires decoupling storage and compute via high-compression object storage and dedicated compute pools. Deploying real-time analytical databases such as ClickHouse enables high-concurrency, low-latency SQL querying, log clustering, and structured in-database summarization necessary for automated reasoning loops.


### [How Eisan made POS analytics faster, cheaper, and more reliable with ClickHouse Cloud](https://yomu.fyi/post/how-eisan-made-pos-analytics-faster-cheaper-and-more-reliable-with-cli.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: ClickHouse
- Published: Apr 8, 2026
- Type: Problem & solution

Eisan System Development moved a retail analytics workload from a QlikView-based platform to ClickHouse Cloud after rising ID-POS data volumes caused slower processing, failed analyses, and increasing licensing and server costs. Client records from supermarkets and drugstores support basket analysis, cross analysis, and real-time personalization, with some datasets nearing 200 million records annually. The team compared open-source options including PostgreSQL and YugabyteDB, then tested ClickHouse configurations, including a 30-core, 120-GB setup, while accounting for cache memory usage through load testing. In comparison, ClickHouse ran on slightly lower-spec hardware under high-load conditions without observed performance deterioration, and Eisan reports no cases so far where it failed to return a result; the team is now targeting 100 billion records and exploring off-hours processing and scheduled scaling.


### [MongoDB Predictive Auto-Scaling: An Experiment](https://yomu.fyi/post/mongodb-predictive-auto-scaling-an-experiment.md)
- Company: [MongoDB](https://yomu.fyi/company/mongodb.md)
- Author: A. Jesse Jiryu Davis, Matthieu Humeau
- Published: Apr 7, 2026
- Type: Problem & solution

MongoDB Atlas initially relied on a reactive auto-scaler that took minutes to adjust instance sizes and only transitioned between adjacent tiers. Because these delays caused transient server over-utilization or unnecessary customer costs, researchers prototyped predictive auto-scaling to forecast demand. The experimental architecture used short-term and long-term forecasters for customer-driven metrics, an estimator using boosted decision trees to project CPU utilization, and a planner to pick the most cost-effective tier. In testing, the prototype kept replica sets closer to target CPU ranges while saving an average of nine cents per replica set hourly. MongoDB subsequently implemented a production version that scales replica sets up proactively before expected spikes while retaining reactive downscaling.


### [Zero downtime Upgrade: Yelp’s Cassandra 4.x Upgrade Story](https://yomu.fyi/post/zero-downtime-upgrade-yelp-s-cassandra-4-x-upgrade-story.md)
- Company: [Yelp](https://yomu.fyi/company/yelp.md)
- Author: Mark Surnin and Muhammad Junaid Muzammil, Software Engineer
- Published: Apr 7, 2026
- Type: Problem & solution

Yelp's Database Reliability Engineering team upgraded over a thousand Cassandra nodes from version 3.11 to 4.1 without downtime. The upgrade delivered reliability, observability, and performance improvements, including Java 11 support, configurable guardrails, and fixed incremental repairs. To avoid the high costs, consistency compromises, and data streaming delays of standing up separate datacenters, the team selected an in-place rolling upgrade strategy. The rollout orchestrated staged node updates alongside compatible versions of the Stargate proxy and custom data pipeline connectors without requiring client code modifications. Although Stargate 2.x introduced regressions on range and multi-partition queries that required downgrading to 1.x, the finished migration achieved up to a 58% reduction in p99 latency across key clusters.


### [How Respan is scaling LLM observability with ClickHouse Cloud](https://yomu.fyi/post/how-respan-is-scaling-llm-observability-with-clickhouse-cloud.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: ClickHouse
- Published: Apr 6, 2026
- Type: Problem & solution

Respan scaled its LLM gateway and observability platform from hundreds of daily requests to 50 million daily events, causing its original Django and Postgres architecture to stall under transaction contention at 50 to 100 requests per second. To overcome Postgres write-ahead logging bottlenecks, the team migrated the ingestion and analytics pipelines to ClickHouse Cloud. Engineers minimized row sizes by truncating large prompt text and writing structured metrics to MergeTree tables, then built incremental minute- and hour-level materialized views alongside scheduled refreshable views for user metadata. Distributed tracing spans are ingested asynchronously with analytical parent-child resolution, utilizing functions like argMax rather than the FINAL modifier to prevent costly table merges. This architectural shift allows Respan to achieve sub-second search across tens of millions of records and support customer dashboards with predictable query performance.


### [Improving storage efficiency in Magic Pocket, our immutable blob store](https://yomu.fyi/post/improving-storage-efficiency-in-magic-pocket-our-immutable-blob-store.md)
- Company: [Dropbox](https://yomu.fyi/company/dropbox.md)
- Author: Ilya Yakovlev,Andrew Cheung,Binoy Dash,Simran Jumani,Dmitriy Meyerzon,Mark Breitenbach,Ishan Mishra,Kazuaki Okumura,Mike White,Kevin Altschuler,Facundo Agriel
- Published: Apr 2, 2026

After introducing a direct erasure-coding service, Dropbox encountered severe storage fragmentation in its immutable blob store, Magic Pocket, due to a large number of under-filled volumes. Because their legacy compaction strategy was only designed to top off near-full volumes, the team created new multi-strategy compaction methods to aggressively consolidate sparse volumes. These approaches leverage dynamic programming and streaming pipelines to rapidly reclaim unused disk capacity at exabyte scale.


### [100,000 GitHub stars](https://yomu.fyi/post/100-000-github-stars.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: paul\_copplestone
- Published: Apr 2, 2026
- Type: Announcement

Supabase reports reaching 100,000 GitHub stars and says eight million developers are building with the platform. Founded in 2020 by two people around a Postgres database, the project says it optimized for developers having a good time rather than for star counts, treating the milestone as evidence of a community that contributed, stayed involved, and built unexpected projects. Its open-source approach centers on supporting existing tools and communities: projects run unmodified Postgres, PostgREST handles billions of API requests daily, pgvector stores embeddings, Deno underpins Edge Functions, imgproxy handles image transformations, and Phoenix supports Realtime at almost 1.5 billion messages per day. The team says it aims to help make Postgres the default database for new internet projects, including by pursuing petabyte scale and pluggable storage.


### [PostgresBench: A Reproducible Benchmark for Postgres Services](https://yomu.fyi/post/postgresbench-a-reproducible-benchmark-for-postgres-services.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: Lionel Palacin
- Published: Apr 2, 2026
- Type: Benchmark

PostgresBench is a public, reproducible benchmarking suite created to compare transactional performance across managed PostgreSQL providers. The benchmark uses the standard pgbench tool with a TPC-B-like write-intensive workload executing across 256 clients and 16 threads for ten-minute durations. Evaluators tested five services—including Postgres managed by ClickHouse, AWS Aurora, AWS RDS, Neon, and Crunchy Bridge—under 100 GB and 500 GB scale factors using default configurations in the us-east-2 region. Across tested configurations, Postgres managed by ClickHouse delivered higher transactions per second and lower P99 latencies due to physically co-located NVMe primary storage. These results demonstrate that remote network storage layers, such as EBS volumes, introduce significant latency overhead during sustained write and fsync operations.


### [Training mRNA Language Models Across 25 Species for $165](https://yomu.fyi/post/training-mrna-language-models-across-25-species-for-165.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Maziyar Panahi
- Published: Mar 31, 2026
- Type: Benchmark

OpenMed describes an end-to-end protein engineering pipeline that connects ESMFold structure prediction, ProteinMPNN sequence design, and custom mRNA codon optimization. The codon-language-model study compares CodonBERT, ModernBERT, and several CodonRoBERTa variants using 250,000 E. coli coding sequences, a 69-token codon vocabulary, masked-language modeling, and training on four 80GB A100 GPUs. CodonRoBERTa-large-v2 achieved perplexity 4.10, CAI Spearman correlation 0.404, and 7.7% synonymous recovery, while ModernBERT-base reached perplexity 26.24 and correlation 0.070. The work then trained a multi-species suite on 381,283 sequences spanning 25 species, producing four production models in 55 GPU-hours and documenting planned extensions such as larger training data and additional prediction heads.


### [TRL v1.0: Post-Training Library Built to Move with the Field](https://yomu.fyi/post/trl-v1-0-post-training-library-built-to-move-with-the-field.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Quentin Gallouédec, Steven Liu, Pedro Cuenca, Sergio Paniego
- Published: Mar 31, 2026
- Type: Announcement

TRL v1.0 marks a transition from research codebase to a library intended for production systems, with clearer stability expectations. It implements more than 75 post-training methods, but its design prioritizes trying, comparing, and using them over coverage alone. The library responds to a field in which PPO, preference optimization methods such as DPO, ORPO, and KTO, and RLVR-style approaches such as GRPO repeatedly changed which components of the training stack appear essential. TRL addresses that volatility with a stable core under semantic versioning, an experimental surface for faster-moving methods, and deliberately limited abstractions that favor explicit implementations over generic hierarchies. v1.0 also follows distributed breaking changes across the 0.x releases, so migration from the last 0.x version is described as minimal, while future work targets asynchronous GRPO, scaling, and actionable training diagnostics.


### [How Padlet uses ClickHouse Cloud to power real-time classroom analytics](https://yomu.fyi/post/how-padlet-uses-clickhouse-cloud-to-power-real-time-classroom-analytic.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: ClickHouse
- Published: Mar 30, 2026
- Type: Problem & solution

Padlet needed a real-time analytics system to provide teachers with immediate insights into student engagement across 40 million monthly unique users. To avoid the operational overhead of managing database clusters with a small engineering team, the company adopted ClickHouse Cloud and ClickPipes for managed data ingestion. Active user sessions emit 30-second heartbeat events that stream alongside page views into ClickHouse. Materialized views pre-aggregate engagement duration and view counts by date, while native HyperLogLog sketches process unique visitor calculations efficiently across arbitrary time ranges. In production, this architecture ingested roughly eight billion events in a single month and delivered analytical query results with a 45-millisecond median latency.


[Newer posts](https://yomu.fyi/topic/scalability/page/8.md) · [Older posts](https://yomu.fyi/topic/scalability/page/10.md)