Loading…
Topic
375 posts about Scalability. Every summary links to the original.
See the idea
Consolidating agent memory, retrieval, and governance onto MongoDB Atlas removes sync and vendor layers, but ties scaling and control to one platform. This trade-off view shows the payoff, the cost, and the condition that decides it.
Autoscaling trades responsiveness, stability, and cost: this explainer shows how signal choice, scaling direction, and capacity limits shape the right control loop for a workload.
Apache Pinot is a distributed real-time OLAP datastore for low-latency analytics. This architecture view shows how streams, processing, partitioning, indexed storage, and replica-aware serving fit together.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Qube Research & Technologies (QRT), a global quantitative investment manager headquartered in London, uses ClickHouse Cloud for a data platform that serves researchers and for near-real-time risk monitoring, management, and P&L. Founded in 2018 as a cloud-native firm after the spinout of Credit Suisse’s proprietary trading arm, QRT chose a cloud-based approach rather than inheriting decades of on-premise infrastructure. The platform supports two major systems, while ClickStack consolidates logs, metrics, and traces for observability; alerts, log queries, and metric aggregations run against a single ClickHouse instance using standard SQL. QRT says its previous database was constrained by writer and reader counts and total cluster load, whereas ClickHouse provides greater speed and scalability, and that a unified monitoring backend simplifies maintenance while remaining reliably fast.
Palantir’s document store needs to repair or rebuild Elasticsearch indices when mappings, settings, cluster versions, or index state change, while continuing to serve live search and writes. Its reindex machinery reconstructs schema-aware indices from the strongly consistent database, rather than relying on Elasticsearch’s _reindex API, and is designed to limit manual work and resource consumption across many deployments and schemas. For an online reindex, it creates a physical shadow index with the intended mappings and settings, backfills it from the database, dual-writes current API changes to both indices, and promotes the shadow index after completion; searches remain on the primary during the rebuild. Cross-cluster migrations use a separate machine with cluster-specific metadata, configuration, index and alias evolution, and preservation of old-cluster indices to support rollback.
ClickHouse announced an expansion of its global go-to-market leadership team, appointing Ed Lenta as Vice President, Asia Pacific and Japan (APJ). Based in Singapore, Lenta will lead regional go-to-market efforts and accelerate adoption of ClickHouse Cloud and the open-source database, following leadership roles at Databricks, AWS, and VMware. The company also appointed Takeshi Kaneko as Country Manager of Japan and added executives for financial services, public sector, global solutions architecture, and revenue strategy and operations. ClickHouse said the appointments support its growth after surpassing $250 million in annual run-rate revenue, more than triple the prior year, and exceeding 4,000 customers, including more than 1,000 net new customers in one quarter.
ClickHouse Cloud can automatically scale infrastructure up and down, but teams may need to monitor latency SLAs for specific workloads such as a frontend dashboard requiring p99 under 200 ms. The guide tags queries with log_comment, reads system.query_log through clickhousectl, and aggregates query_duration_ms over a defined window to measure the SLA. When it is breached, operators compare query volume and latency with Prometheus-compatible CPU, memory, and concurrency metrics to distinguish increased concurrency from resource contention, then choose horizontal replica scaling or vertical replica sizing. A cron-based script can scale out when p99 exceeds a threshold, while an agent can inspect evidence and apply one scaling action using clickhousectl; actions are recorded in the ClickHouse Cloud activity log.
Airbnb modernized sitar-agent, a Kubernetes sidecar that delivers dynamic configuration updates to thousands of polyglot service instances without requiring redeployments. The architecture uses periodic AWS S3 snapshots to enable fast, decoupled pod startups and maintains local file-based storage for the main container to read configs with in-memory caching. Key design decisions include retaining an isolated sidecar over an in-process library and optimizing pull-based polling with server-side caching.
Real-time analytical databases must serve low-latency queries over data that is continuously inserted at high volume, rather than merely query a static dataset quickly. The article frames time-to-insight as the combination of time to ingest, time to transform and prepare data, and time to query it. It argues that batch-oriented ingestion, delayed visibility, scheduled materialized-view refreshes, and cache invalidation during writes can make production latency far worse than headline benchmarks. It recommends evaluating freshness under concurrent ingest and queries, whether one engine supports structured analytics, text search, vector similarity, and JSON, and whether index maintenance can avoid blocking ingestion. As an example, Bullet reportedly moved to ClickHouse Cloud, reducing data latency by 99.4% to under five seconds, achieving millisecond queries, and handling 1,000× more data at comparable cost.
Multigres v0.1 alpha is an open-source release of a system intended to provide Vitess-grade horizontal scaling, high availability, and operational simplicity for Postgres. It manages Postgres instances with sharding, connection pooling, automatic failover, and backup orchestration, while this release adds advanced pooling, automatic failovers, and a Kubernetes operator. Its high-availability protocol builds on unmodified Postgres replication, supports user-defined durability policies, resolves split-brain scenarios without losing succeeded commits, and permits replica changes while running. The release is suitable for experimentation but not production workloads: v0.1 is single-shard, sharding is absent, the custom-resource API may change, backward compatibility is not guaranteed, and performance benchmarks are still in progress.
Supabase has raised a $500M Series F at a $10B pre-money valuation, led by GIC. Existing investors participated, Stripe invested again, and Georgian and Salesforce Ventures joined as new investors; the funding will accelerate open-source and Postgres tooling, support growth, and provide employee liquidity. Supabase is also releasing Multigres v0.1 alpha, an open-source, self-hostable scalable operating system for Postgres designed to provide high availability and operational simplicity; a future release is intended to add Vitess-grade horizontal scaling. The alpha is ready to try but not production-ready, with production-readiness expected in a few months. Over the past year, database launches grew 600%, more than 60% of new databases came from AI tools, and nearly 10 million developers built on Supabase.
Slack evolved its Slack AI serving infrastructure across multiple phases to handle enterprise LLM workloads reliably and securely. The initial deployment on AWS SageMaker provided zero-knowledge escrow VPC isolation and FedRAMP compliance, but engineers faced scaling latency, GPU scarcity, and significant operational overhead. Slack then migrated live traffic to Amazon Bedrock to leverage managed Model Units and eliminate model release lag without customer-facing incidents. However, fixed Provisioned Throughput commitments and regional peak traffic variations created persistent underutilization challenges. Consequently, Slack expanded into a multi-cloud orchestration architecture that normalizes disparate provider APIs, integrates unified cross-cloud telemetry, and routes traffic dynamically around latency spikes and outages.
amazon.science
Traditional data center networks rely on hierarchical fat-tree topologies, which introduce overhead, upper-layer congestion, and vulnerability to router failures. Although flat random networks theoretically maximize route diversity and resilience, routing computation and cabling complexity historically made them impractical for physical data centers. AWS addressed these constraints by introducing Resilient Network Graphs, a quasi-random architecture powered by a passive optical device called ShuffleBox and a routing algorithm named Spraypoint. Spraypoint sprays packets across random neighbors toward designated waypoints, unlocking twice as many independent paths while operating within commodity router memory limits. Adopted as the default architecture across new AWS facilities, this design reduces router counts by 69%, lowers projected network power consumption by 40%, and increases throughput by up to 33%.
Async RL weight synchronization currently requires the trainer to send a complete model to the inference engine after each optimizer step, creating a critical-path transfer that can reach roughly a terabyte for a frontier 1T-parameter checkpoint. The article attributes the opportunity for reduction to bf16 arithmetic: at RL learning rates, many updates fall below the bf16 visibility threshold, leaving about 99% of weights bit-identical between steps and more than 98% even in the reported worst case. TRL's delta-weight-sync implementation records changed elements around the optimizer step, encodes them in sparse safetensors, uploads each delta to a Hugging Face Bucket, and lets vLLM fetch and apply it. On Qwen3-0.6B, this reduces each per-step payload from 1.2 GB to 20–35 MB, while a demonstrated setup connected separate trainer, vLLM, and Wordle Spaces through one bucket. Open issues include extra CPU bf16 snapshots, fixed anchor cadence, and unmeasured multi-node FSDP2 support.
Large JVM codebases often stall Java upgrades because build-graph constraints require coordinated changes across many owners, while manual scripts and stale inventories repeatedly rediscover blockers. Stripe built AutoJDK to continuously analyze the live build graph, compute eligibility using reverse-dependency, runtime, framework, and dependency-ordering constraints, and generate bounded, idempotent pull requests rather than opaque rewrites. It also tracks adoption and blocker categories, uses AI assistance for selected remediation tasks, and can operate continuously or on specified packages and services. Centralized JDK and runtime policy, CI checks, multi-JDK support, static analysis, and packaging applications with platform-specific JDKs support gradual transitions. After three months, the upgrader had merged more than 160 PRs, made JDK 17 the most common target, eliminated Java 8 services, and reduced the JDK 8 and 11 footprint by roughly 25K targets.
ClickHouse introduces House Mates, a partner community and program bringing together more than 60 ISV, consulting, services, and channel partners to help customers build, scale, and procure ClickHouse solutions. The inaugural cohort includes over 25 ISVs developing validated, production-ready integrations—among them dbt Labs, Fivetran, Temporal, Grafana Labs, Confluent, and Sigma—and over 35 consulting, services, and reseller partners trained and certified through ClickHouse Academy. The program addresses an ecosystem with more than 200 integrations in the wild, demand for trusted regional expertise and preferred procurement channels, and the limits of managing partnerships through one-off arrangements. Its Ignite, Accelerate, and Prime tiers define progression from enablement and first projects or integrations to deeper go-to-market, incentives, co-innovation, and strategic collaboration at scale, while customers gain validated tools, regional experts, and preferred purchasing routes.
ClickStack Cloud is entering private preview as a fully managed, serverless observability service built on ClickHouse for teams that want ClickHouse-powered logs, metrics, and traces without operating the underlying infrastructure. Users send OpenTelemetry data to a managed OTLP endpoint, while the service handles ingestion, durable buffering through an object-storage-backed event queue, scaling, storage, and query serving, then exposes telemetry in the ClickStack UI with integrated authentication and RBAC. The preview separates ingestion and query resources and is developing automatic tuning based on query patterns, including materialized fields, primary-key adjustments, materialized views, and indexes. Dedicated query compute for agentic and high-volume analytical workloads is planned after the preview, while pricing remains unfinished and the existing Managed ClickStack offering remains intended for teams requiring direct infrastructure and schema control.
Traditional data warehouse benchmarks measure query execution speed without accounting for the varying compute costs across platforms. CostBench introduces an open benchmarking framework designed to evaluate data warehouses on a performance-per-dollar metric across both read and write workloads. Its initial release evaluates ClickHouse Cloud, Snowflake, Databricks, BigQuery, and Redshift using 43 production-derived analytical queries on an anonymized dataset combined with each vendor's compute billing model. In these read-side tests, ClickHouse Cloud maintained high speed at lower costs as data scaled, outperforming the nearest competitor by 23 times in cost-performance. The benchmark repository openly publishes its scripts, configurations, pricing assumptions, and raw JSON results to ensure all comparative metrics remain fully inspectable and reproducible.
ClickHouse Cloud introduced multi-stage distributed query execution to improve scaling for large joins and high-cardinality aggregations across multi-node clusters. Prior execution models relied on physical sharding or parallel replicas, which lacked the ability to freely repartition intermediate data between query stages. By splitting query plans into stages connected by exchange operators, worker nodes can shuffle, broadcast, scatter, or gather intermediate data during execution. In TPC-H benchmarks at scale factor 100 on eight nodes, total query time dropped from 117.6 seconds to 54.7 seconds, with scan-dominated queries showing near-linear 7.4x speedups. While currently in private preview with a rule-based planner, future work includes developing a cost-based optimizer to select optimal aggregation and join strategies automatically.
ClickHouse native random sampling enables aggregate queries to execute against a representative fraction of data when querying full datasets is too slow. To maintain sampling accuracy, tables require a sample key derived from high-cardinality columns using hash functions like sipHash64 to ensure an even distribution. The sample key must be included in the primary key ORDER BY expression, preferably at the front to let ClickHouse quickly skip index granules. Queries can sample data deterministically by specifying a fractional ratio or a minimum row threshold, scaling aggregate counts using the virtual column _sample_factor. In benchmark tests on a 30-million-row dataset, sampling 10% of records reduced query latency by approximately 80% while retaining approximate metrics suitable for exploratory analysis.
User sequences serve as core primitives powering machine learning models across ranking, retrieval, and recommendation surfaces at Pinterest. Managing these sequences previously caused high infrastructure costs, operational fragility, and data drift between offline training pipelines and online serving systems. In response, engineering teams redesigned the platform around a single definition executed across multiple runtimes, including real-time streaming, batch backfills, and online inference. The updated architecture pairs a unified enrichment engine with columnar time-partitioned storage and standardized validation tiers. Consequently, the redesign achieved lower storage and network costs, faster signal onboarding via configuration, and improved engagement metrics across major recommendation surfaces.
The May 2026 ClickHouse newsletter details enterprise observability migrations, ecosystem integrations, and core database enhancements. Digital banking platform Qonto migrated from Grafana Tempo to ClickHouse Cloud, compressing 231 TB of raw trace metadata to 376 GB while expanding query windows to two weeks and introducing an AI incident companion. Similarly, LINE MAN Wongnai consolidated multiple business clusters onto self-hosted ClickHouse, achieving 1.5 million rows per second peak ingest, 100% trace retention, and a 53% reduction in observability costs. Comparative log analytics benchmarks demonstrated that ClickHouse utilized 5x less disk space and achieved 4 to 6x faster cold query execution than Elasticsearch across 50 billion rows. Additional updates include the ClickHouse 26.4 release featuring improved SQL compatibility, new agent telemetry adapters in Mastra, and SQL-based alerting capabilities in ClickStack.
Data teams at Yelp struggled to identify table consumers and track how datasets were accessed across their petabyte-scale data lake. To solve this, the team built a usage attribution system using Amazon S3 server access logs to visualize time-based partition keys against access timestamps and IAM roles. These visualizations revealed distinctive patterns for daily batch jobs, backfills, and ad hoc queries, guiding migrations of thousands of tables to Apache Iceberg format. Furthermore, Yelp implemented a Default Access Retention strategy gated by IAM bucket policies to prevent accidental queries from resetting S3 Intelligent Tiering transitions. Combining these granular usage insights with targeted S3 storage classes reduced Yelp's S3 storage costs by 33 percent.
The best engineering reads, once a week.
