---
title: "Scalability"
description: "379 posts about Scalability, summarised, each linking to the original."
---

# Scalability
> 379 posts about Scalability, summarised, each linking to the original.

## Articles

### [Datadog and ClickHouse partner to bring full-fidelity data to modern observability](https://yomu.fyi/post/datadog-and-clickhouse-partner-to-bring-full-fidelity-data-to-modern-o.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: Jun 10, 2026
- Type: Announcement

ClickHouse and Datadog announce a partnership aimed at reducing the tradeoff between retaining complete telemetry and keeping it accessible through familiar observability workflows. New Preview capabilities let organizations route logs directly to ClickHouse through Datadog Observability Pipelines and search those logs from the Datadog Log Explorer. Pipelines can parse, enrich, filter, transform, and redact data before storage, while ClickHouse Cloud’s separation of storage and compute supports long-term retention, independently scaled query resources, and workload isolation. Federated log search queries data directly in ClickHouse, avoiding duplicate copies or re-ingestion while allowing high-volume logs to remain searchable and historical investigations to use full-fidelity telemetry. The partnership combines Datadog’s collection and investigation experience with ClickHouse’s storage and analytics foundation; the native integration and federated search are available in Preview.


### [Quick: An internal hosting platform for the AI era](https://yomu.fyi/post/quick-an-internal-hosting-platform-for-the-ai-era.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jun 10, 2026
- Type: Explainer

Shopify built Quick to make internally created prototypes, dashboards, tools, and games easier to share than deploy, giving employees a secure URL for a folder of HTML and assets. Launched internally in July 2025, it places sites behind Identity-Aware Proxy, serves bucket content through NGINX and gcsfuse, and deploys with a wrapper around gcloud rsync. A service in front of CloudSQL provides database and realtime APIs, while server-held credentials support calls to Shopify’s AI proxy; file uploads, BigQuery access, WebSockets, and employee identity are also available. Quick now hosts more than 50,000 sites, has been used by over half of Shopify employees, and runs on one $200-per-month VM, with constraints, rate limiting, and a migration from Node to Go supporting its operation.


### [Scaling beyond one: How Airbnb evolved its data architecture for a multi-product world](https://yomu.fyi/post/scaling-beyond-one-how-airbnb-evolved-its-data-architecture-for-a-mult.md)
- Company: [Airbnb](https://yomu.fyi/company/airbnb.md)
- Author: Patrick Lam
- Published: Jun 9, 2026

To support its expansion from Homes into Experiences and Services, Airbnb evolved its offline data warehouse architecture to balance domain-specific needs with organizational consistency. Rather than enforcing a single global pattern, data teams adopted a framework of foundational principles that paired separate data models for unique product features with monolithic models for cross-cutting domains like payments and messaging.


### [ClickHouse Agents: Claude-powered agentic analytics, now in public beta](https://yomu.fyi/post/clickhouse-agents-claude-powered-agentic-analytics-now-in-public-beta.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: Jun 9, 2026
- Type: Announcement

ClickHouse announced the public beta of ClickHouse Agents, a fully managed agentic analytics service in ClickHouse Cloud powered by Claude. It provides a no-code builder for agents grounded in live ClickHouse data, plus a managed chat experience in the Cloud console, so users need no SQL or separate hosting setup. Built on LibreChat, the service supports natural-language questions, SQL and visualizations, sandboxed Bash, Python, and JavaScript execution, shareable artifacts, skills, memories, MCP-compatible connections, and multi-agent workflows. Claude Sonnet and Haiku are pre-configured, while enterprise controls include SSO, encrypted message storage, and governed data access. ClickHouse says its internal deployment handles roughly 70% of data warehouse queries and processes more than 45 million tokens daily, confirming its view that real-time analytics is a natural context for high-volume agent workloads.


### [How QRT powers real-time research and risk management at petabyte scale](https://yomu.fyi/post/how-qrt-powers-real-time-research-and-risk-management-at-petabyte-scal.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: Jun 9, 2026
- Type: Explainer

Qube Research & Technologies (QRT), a global quantitative investment manager headquartered in London, uses ClickHouse Cloud for a data platform that serves researchers and for near-real-time risk monitoring, management, and P&L. Founded in 2018 as a cloud-native firm after the spinout of Credit Suisse’s proprietary trading arm, QRT chose a cloud-based approach rather than inheriting decades of on-premise infrastructure. The platform supports two major systems, while ClickStack consolidates logs, metrics, and traces for observability; alerts, log queries, and metric aggregations run against a single ClickHouse instance using standard SQL. QRT says its previous database was constrained by writer and reader counts and total cluster load, whereas ClickHouse provides greater speed and scalability, and that a unified monitoring backend simplifies maintenance while remaining reliably fast.


### [Managing Elasticsearch Reindex at Scale: Performance, Reliability, and Observability](https://yomu.fyi/post/managing-elasticsearch-reindex-at-scale-performance-reliability-and-ob.md)
- Company: [Palantir](https://yomu.fyi/company/palantir.md)
- Author: Palantir
- Published: Jun 8, 2026
- Type: Explainer

Palantir’s document store needs to repair or rebuild Elasticsearch indices when mappings, settings, cluster versions, or index state change, while continuing to serve live search and writes. Its reindex machinery reconstructs schema-aware indices from the strongly consistent database, rather than relying on Elasticsearch’s \_reindex API, and is designed to limit manual work and resource consumption across many deployments and schemas. For an online reindex, it creates a physical shadow index with the intended mappings and settings, backfills it from the database, dual-writes current API changes to both indices, and promotes the shadow index after completion; searches remain on the primary during the rebuild. Cross-cluster migrations use a separate machine with cluster-specific metadata, configuration, index and alias evolution, and preservation of old-cluster indices to support rollback.


### [ClickHouse appoints new leader for Asia Pacific and expands global go-to-market leadership team](https://yomu.fyi/post/clickhouse-appoints-new-leader-for-asia-pacific-and-expands-global-go.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: Jun 8, 2026
- Type: Announcement

ClickHouse announced an expansion of its global go-to-market leadership team, appointing Ed Lenta as Vice President, Asia Pacific and Japan (APJ). Based in Singapore, Lenta will lead regional go-to-market efforts and accelerate adoption of ClickHouse Cloud and the open-source database, following leadership roles at Databricks, AWS, and VMware. The company also appointed Takeshi Kaneko as Country Manager of Japan and added executives for financial services, public sector, global solutions architecture, and revenue strategy and operations. ClickHouse said the appointments support its growth after surpassing $250 million in annual run-rate revenue, more than triple the prior year, and exceeding 4,000 customers, including more than 1,000 net new customers in one quarter.


### [Monitor SLAs and scale ClickHouse Cloud with clickhousectl and agents](https://yomu.fyi/post/monitor-slas-and-scale-clickhouse-cloud-with-clickhousectl-and-agents.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: Jun 5, 2026
- Type: Tutorial

ClickHouse Cloud can automatically scale infrastructure up and down, but teams may need to monitor latency SLAs for specific workloads such as a frontend dashboard requiring p99 under 200 ms. The guide tags queries with log\_comment, reads system.query\_log through clickhousectl, and aggregates query\_duration\_ms over a defined window to measure the SLA. When it is breached, operators compare query volume and latency with Prometheus-compatible CPU, memory, and concurrency metrics to distinguish increased concurrency from resource contention, then choose horizontal replica scaling or vertical replica sizing. A cron-based script can scale out when p99 exceeds a threshold, while an agent can inspect evidence and apply one scaling action using clickhousectl; actions are recorded in the ClickHouse Cloud activity log.


### [Sitar-agent: Building a reliable dynamic configuration sidecar at scale](https://yomu.fyi/post/sitar-agent-building-a-reliable-dynamic-configuration-sidecar-at-scale.md)
- Company: [Airbnb](https://yomu.fyi/company/airbnb.md)
- Author: Bo Teng
- Published: Jun 4, 2026

Airbnb modernized sitar-agent, a Kubernetes sidecar that delivers dynamic configuration updates to thousands of polyglot service instances without requiring redeployments. The architecture uses periodic AWS S3 snapshots to enable fast, decoupled pod startups and maintains local file-based storage for the main container to read configs with in-memory caching. Key design decisions include retaining an isolated sidecar over an in-process library and optimizing pull-based polling with server-side caching.


### [What to look for when selecting a real-time analytical database](https://yomu.fyi/post/what-to-look-for-when-selecting-a-real-time-analytical-database.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: Jun 4, 2026
- Type: Explainer

Real-time analytical databases must serve low-latency queries over data that is continuously inserted at high volume, rather than merely query a static dataset quickly. The article frames time-to-insight as the combination of time to ingest, time to transform and prepare data, and time to query it. It argues that batch-oriented ingestion, delayed visibility, scheduled materialized-view refreshes, and cache invalidation during writes can make production latency far worse than headline benchmarks. It recommends evaluating freshness under concurrent ingest and queries, whether one engine supports structured analytics, text search, vector similarity, and JSON, and whether index maintenance can avoid blocking ingestion. As an example, Bullet reportedly moved to ClickHouse Cloud, reducing data latency by 99.4% to under five seconds, achieving millisecond queries, and handling 1,000× more data at comparable cost.


### [Multigres v0.1 Alpha: an operating system for Postgres](https://yomu.fyi/post/multigres-v0-1-alpha-an-operating-system-for-postgres.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: sugu\_sougoumarane
- Published: Jun 4, 2026
- Type: Announcement

Multigres v0.1 alpha is an open-source release of a system intended to provide Vitess-grade horizontal scaling, high availability, and operational simplicity for Postgres. It manages Postgres instances with sharding, connection pooling, automatic failover, and backup orchestration, while this release adds advanced pooling, automatic failovers, and a Kubernetes operator. Its high-availability protocol builds on unmodified Postgres replication, supports user-defined durability policies, resolves split-brain scenarios without losing succeeded commits, and permits replica changes while running. The release is suitable for experimentation but not production workloads: v0.1 is single-shard, sharding is absent, the custom-resource API may change, backward compatibility is not guaranteed, and performance benchmarks are still in progress.


### [Supabase Series F](https://yomu.fyi/post/supabase-series-f.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: paul\_copplestone
- Published: Jun 4, 2026
- Type: Announcement

Supabase has raised a $500M Series F at a $10B pre-money valuation, led by GIC. Existing investors participated, Stripe invested again, and Georgian and Salesforce Ventures joined as new investors; the funding will accelerate open-source and Postgres tooling, support growth, and provide employee liquidity. Supabase is also releasing Multigres v0.1 alpha, an open-source, self-hostable scalable operating system for Postgres designed to provide high availability and operational simplicity; a future release is intended to add Vitess-grade horizontal scaling. The alpha is ready to try but not production-ready, with production-readiness expected in a few months. Over the past year, database launches grew 600%, more than 60% of new databases came from AI tools, and nearly 10 million developers built on Supabase.


### [Slack AI: The Path to Multi-Cloud](https://yomu.fyi/post/slack-ai-the-path-to-multi-cloud.md)
- Company: [Slack](https://yomu.fyi/company/slack.md)
- Author: Shaurya Kethireddy
- Published: May 28, 2026
- Type: Problem & solution

Slack evolved its Slack AI serving infrastructure across multiple phases to handle enterprise LLM workloads reliably and securely. The initial deployment on AWS SageMaker provided zero-knowledge escrow VPC isolation and FedRAMP compliance, but engineers faced scaling latency, GPU scarcity, and significant operational overhead. Slack then migrated live traffic to Amazon Bedrock to leverage managed Model Units and eliminate model release lag without customer-facing incidents. However, fixed Provisioned Throughput commitments and regional peak traffic variations created persistent underutilization challenges. Consequently, Slack expanded into a multi-cloud orchestration architecture that normalizes disparate provider APIs, integrates unified cross-cloud telemetry, and routes traffic dynamically around latency spikes and outages.


### [How flat is replacing fat in AWS data center networks](https://yomu.fyi/post/how-flat-is-replacing-fat-in-aws-data-center-networks.md)
- Company: amazon.science
- Author: Giacomo Bernardi, Ratul Mahajan, Seshadhri Comandur
- Published: May 28, 2026
- Type: Problem & solution

Traditional data center networks rely on hierarchical fat-tree topologies, which introduce overhead, upper-layer congestion, and vulnerability to router failures. Although flat random networks theoretically maximize route diversity and resilience, routing computation and cabling complexity historically made them impractical for physical data centers. AWS addressed these constraints by introducing Resilient Network Graphs, a quasi-random architecture powered by a passive optical device called ShuffleBox and a routing algorithm named Spraypoint. Spraypoint sprays packets across random neighbors toward designated waypoints, unlocking twice as many independent paths while operating within commodity router memory limits. Adopted as the default architecture across new AWS facilities, this design reduces router counts by 69%, lowers projected network power consumption by 40%, and increases throughput by up to 33%.


### [Shipping a Trillion Parameters With a Hub Bucket: Delta Weight Sync in TRL](https://yomu.fyi/post/shipping-a-trillion-parameters-with-a-hub-bucket-delta-weight-sync-in.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Amine Dirhoussi, Quentin Gallouédec, Kashif Rasul, Lewis Tunstall, Edward Beeching, Albert Villanova del Moral, Leandro von Werra, Sergio Paniego
- Published: May 27, 2026
- Type: Problem & solution

Async RL weight synchronization currently requires the trainer to send a complete model to the inference engine after each optimizer step, creating a critical-path transfer that can reach roughly a terabyte for a frontier 1T-parameter checkpoint. The article attributes the opportunity for reduction to bf16 arithmetic: at RL learning rates, many updates fall below the bf16 visibility threshold, leaving about 99% of weights bit-identical between steps and more than 98% even in the reported worst case. TRL's delta-weight-sync implementation records changed elements around the optimizer step, encodes them in sparse safetensors, uploads each delta to a Hugging Face Bucket, and lets vLLM fetch and apply it. On Qwen3-0.6B, this reduces each per-step payload from 1.2 GB to 20–35 MB, while a demonstrated setup connected separate trainer, vLLM, and Wordle Spaces through one bucket. Open issues include extra CPU bf16 snapshots, fixed anchor cadence, and unmeasured multi-node FSDP2 support.


### [Modern Java at Stripe: Language upgrades as a service](https://yomu.fyi/post/modern-java-at-stripe-language-upgrades-as-a-service.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: Mahib Hosain
- Published: May 27, 2026
- Type: Problem & solution

Large JVM codebases often stall Java upgrades because build-graph constraints require coordinated changes across many owners, while manual scripts and stale inventories repeatedly rediscover blockers. Stripe built AutoJDK to continuously analyze the live build graph, compute eligibility using reverse-dependency, runtime, framework, and dependency-ordering constraints, and generate bounded, idempotent pull requests rather than opaque rewrites. It also tracks adoption and blocker categories, uses AI assistance for selected remediation tasks, and can operate continuously or on specified packages and services. Centralized JDK and runtime policy, CI checks, multi-JDK support, static analysis, and packaging applications with platform-specific JDKs support gradual transitions. After three months, the upgrader had merged more than 160 PRs, made JDK 17 the most common target, eliminated Java 8 services, and reduced the JDK 8 and 11 footprint by roughly 25K targets.


### [Introducing House Mates: the ClickHouse partner community and program](https://yomu.fyi/post/introducing-house-mates-the-clickhouse-partner-community-and-program.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: May 27, 2026
- Type: Announcement

ClickHouse introduces House Mates, a partner community and program bringing together more than 60 ISV, consulting, services, and channel partners to help customers build, scale, and procure ClickHouse solutions. The inaugural cohort includes over 25 ISVs developing validated, production-ready integrations—among them dbt Labs, Fivetran, Temporal, Grafana Labs, Confluent, and Sigma—and over 35 consulting, services, and reseller partners trained and certified through ClickHouse Academy. The program addresses an ecosystem with more than 200 integrations in the wild, demand for trusted regional expertise and preferred procurement channels, and the limits of managing partnerships through one-off arrangements. Its Ignite, Accelerate, and Prime tiers define progression from enablement and first projects or integrations to deeper go-to-market, incentives, co-innovation, and strategic collaboration at scale, while customers gain validated tools, regional experts, and preferred purchasing routes.


### [Introducing ClickStack Cloud: Serverless observability powered by ClickHouse](https://yomu.fyi/post/introducing-clickstack-cloud-serverless-observability-powered-by-click.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: May 27, 2026
- Type: Announcement

ClickStack Cloud is entering private preview as a fully managed, serverless observability service built on ClickHouse for teams that want ClickHouse-powered logs, metrics, and traces without operating the underlying infrastructure. Users send OpenTelemetry data to a managed OTLP endpoint, while the service handles ingestion, durable buffering through an object-storage-backed event queue, scaling, storage, and query serving, then exposes telemetry in the ClickStack UI with integrated authentication and RBAC. The preview separates ingestion and query resources and is developing automatic tuning based on query patterns, including materialized fields, primary-key adjustments, materialized views, and indexes. Dedicated query compute for agentic and high-volume analytical workloads is planned after the preview, while pricing remains unfinished and the existing Managed ClickStack offering remains intended for teams requiring direct infrastructure and schema control.


### [Introducing CostBench: an open benchmark for data warehouse cost-performance](https://yomu.fyi/post/introducing-costbench-an-open-benchmark-for-data-warehouse-cost-perfor.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: Tom Schreiber, Lionel Palacin
- Published: May 27, 2026
- Type: Benchmark

Traditional data warehouse benchmarks measure query execution speed without accounting for the varying compute costs across platforms. CostBench introduces an open benchmarking framework designed to evaluate data warehouses on a performance-per-dollar metric across both read and write workloads. Its initial release evaluates ClickHouse Cloud, Snowflake, Databricks, BigQuery, and Redshift using 43 production-derived analytical queries on an anonymized dataset combined with each vendor's compute billing model. In these read-side tests, ClickHouse Cloud maintained high speed at lower costs as data scaled, outperforming the nearest competitor by 23 times in cost-performance. The benchmark repository openly publishes its scripts, configurations, pricing assumptions, and raw JSON results to ensure all comparative metrics remain fully inspectable and reproducible.


### [Introducing multi-stage distributed query execution in ClickHouse Cloud](https://yomu.fyi/post/introducing-multi-stage-distributed-query-execution-in-clickhouse-clou.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: Alexander Gololobov
- Published: May 27, 2026
- Type: Announcement

ClickHouse Cloud introduced multi-stage distributed query execution to improve scaling for large joins and high-cardinality aggregations across multi-node clusters. Prior execution models relied on physical sharding or parallel replicas, which lacked the ability to freely repartition intermediate data between query stages. By splitting query plans into stages connected by exchange operators, worker nodes can shuffle, broadcast, scatter, or gather intermediate data during execution. In TPC-H benchmarks at scale factor 100 on eight nodes, total query time dropped from 117.6 seconds to 54.7 seconds, with scan-dominated queries showing near-linear 7.4x speedups. While currently in private preview with a rule-based planner, future work includes developing a cost-based optimizer to select optimal aggregation and join strategies automatically.


[Newer posts](https://yomu.fyi/topic/scalability/page/6.md) · [Older posts](https://yomu.fyi/topic/scalability/page/8.md)