---
title: "Performance"
description: "817 posts about Performance, summarised, each linking to the original."
---

# Performance
> 817 posts about Performance, summarised, each linking to the original.

## Articles

### [Shipping a Trillion Parameters With a Hub Bucket: Delta Weight Sync in TRL](https://yomu.fyi/post/shipping-a-trillion-parameters-with-a-hub-bucket-delta-weight-sync-in.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Amine Dirhoussi, Quentin Gallouédec, Kashif Rasul, Lewis Tunstall, Edward Beeching, Albert Villanova del Moral, Leandro von Werra, Sergio Paniego
- Published: May 27, 2026
- Type: Problem & solution

Async RL weight synchronization currently requires the trainer to send a complete model to the inference engine after each optimizer step, creating a critical-path transfer that can reach roughly a terabyte for a frontier 1T-parameter checkpoint. The article attributes the opportunity for reduction to bf16 arithmetic: at RL learning rates, many updates fall below the bf16 visibility threshold, leaving about 99% of weights bit-identical between steps and more than 98% even in the reported worst case. TRL's delta-weight-sync implementation records changed elements around the optimizer step, encodes them in sparse safetensors, uploads each delta to a Hugging Face Bucket, and lets vLLM fetch and apply it. On Qwen3-0.6B, this reduces each per-step payload from 1.2 GB to 20–35 MB, while a demonstrated setup connected separate trainer, vLLM, and Wordle Spaces through one bucket. Open issues include extra CPU bf16 snapshots, fixed anchor cadence, and unmeasured multi-node FSDP2 support.


### [5 countries, 12 days, one ClickHouse: inside Alexey's APJ AI Tour](https://yomu.fyi/post/5-countries-12-days-one-clickhouse-inside-alexey-s-apj-ai-tour.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: May 27, 2026
- Type: Announcement

ClickHouse CTO and creator Alexey Milovidov completed a 12-day APJ tour spanning India, Singapore, Indonesia, South Korea, and Japan, with customer meetings, industry events, executive sessions, press interviews, and community meetups. The itinerary covered six cities and included three Confluent Data Streaming World Tour events, four ClickHouse meetups, and more than 10 enterprise customer meetings across varied industries and workloads. Discussions centered on real-time analytics, connected products, and analytics at industrial scale, while in-person conversations supplied feedback that the post says helps improve ClickHouse. The trip concluded with the ClickHouse Japan Launch Event on April 27, marking the company’s official entry into that market. The account presents physical presence as a way to extend ClickHouse’s open-building culture beyond GitHub and strengthen its feedback loop with APJ customers and communities.


### [Open House 2026 Day 1: Real-time data without lock-in and what teams can build next](https://yomu.fyi/post/open-house-2026-day-1-real-time-data-without-lock-in-and-what-teams-ca.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: May 27, 2026
- Type: Announcement

ClickHouse’s Open House 2026 Day 1 announced updates spanning ClickHouse Cloud, Postgres, distributed query execution, observability, AI agents, and lakehouse interoperability. ClickHouse Postgres entered public beta with local NVMe storage, high availability, S3-backed point-in-time recovery, and a native CDC path into ClickHouse; early benchmarks reported more than 5x the transactions per second of AWS RDS. Multi-stage distributed queries repartition intermediate data between stages to scale joins and high-cardinality aggregations, reducing TPC-H SF100 runtime from 117.6 seconds on one node to 54.7 seconds on eight nodes. Additional launches included a Claude-powered ClickHouse Agents public beta, an AI-enhanced query builder, Monitoring v2, and Langfuse V4’s reported 200x query performance improvement, while the wrap-up framed the releases around real-time systems without lock-in.


### [Introducing ClickStack Cloud: Serverless observability powered by ClickHouse](https://yomu.fyi/post/introducing-clickstack-cloud-serverless-observability-powered-by-click.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: May 27, 2026
- Type: Announcement

ClickStack Cloud is entering private preview as a fully managed, serverless observability service built on ClickHouse for teams that want ClickHouse-powered logs, metrics, and traces without operating the underlying infrastructure. Users send OpenTelemetry data to a managed OTLP endpoint, while the service handles ingestion, durable buffering through an object-storage-backed event queue, scaling, storage, and query serving, then exposes telemetry in the ClickStack UI with integrated authentication and RBAC. The preview separates ingestion and query resources and is developing automatic tuning based on query patterns, including materialized fields, primary-key adjustments, materialized views, and indexes. Dedicated query compute for agentic and high-volume analytical workloads is planned after the preview, while pricing remains unfinished and the existing Managed ClickStack offering remains intended for teams requiring direct infrastructure and schema control.


### [Introducing CostBench: an open benchmark for data warehouse cost-performance](https://yomu.fyi/post/introducing-costbench-an-open-benchmark-for-data-warehouse-cost-perfor.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: Tom Schreiber, Lionel Palacin
- Published: May 27, 2026
- Type: Benchmark

Traditional data warehouse benchmarks measure query execution speed without accounting for the varying compute costs across platforms. CostBench introduces an open benchmarking framework designed to evaluate data warehouses on a performance-per-dollar metric across both read and write workloads. Its initial release evaluates ClickHouse Cloud, Snowflake, Databricks, BigQuery, and Redshift using 43 production-derived analytical queries on an anonymized dataset combined with each vendor's compute billing model. In these read-side tests, ClickHouse Cloud maintained high speed at lower costs as data scaled, outperforming the nearest competitor by 23 times in cost-performance. The benchmark repository openly publishes its scripts, configurations, pricing assumptions, and raw JSON results to ensure all comparative metrics remain fully inspectable and reproducible.


### [Postgres managed by ClickHouse is now in beta](https://yomu.fyi/post/postgres-managed-by-clickhouse-is-now-in-beta.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: Sai Srirampur
- Published: May 27, 2026
- Type: Announcement

ClickHouse Cloud has launched a fully managed Postgres service in beta to combine transactional and analytical workloads within a single platform. Backed by local NVMe storage, the service provides up to ten times faster transactional performance alongside native change data capture into ClickHouse. Developers can query across both transactional and analytical layers using the pg\_clickhouse extension without managing custom data pipelines. The platform also offers managed migration workflows via ClickPipes, database branching, point-in-time recovery, high availability with up to two standbys, and support for over ninety PostgreSQL extensions. More than fifty virtual machine configurations are available, ranging from single vCPU instances to clusters with ninety-six vCPUs and sixty terabytes of NVMe storage.


### [Introducing multi-stage distributed query execution in ClickHouse Cloud](https://yomu.fyi/post/introducing-multi-stage-distributed-query-execution-in-clickhouse-clou.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: Alexander Gololobov
- Published: May 27, 2026
- Type: Announcement

ClickHouse Cloud introduced multi-stage distributed query execution to improve scaling for large joins and high-cardinality aggregations across multi-node clusters. Prior execution models relied on physical sharding or parallel replicas, which lacked the ability to freely repartition intermediate data between query stages. By splitting query plans into stages connected by exchange operators, worker nodes can shuffle, broadcast, scatter, or gather intermediate data during execution. In TPC-H benchmarks at scale factor 100 on eight nodes, total query time dropped from 117.6 seconds to 54.7 seconds, with scan-dominated queries showing near-linear 7.4x speedups. While currently in private preview with a rule-based planner, future work includes developing a cost-based optimizer to select optimal aggregation and join strategies automatically.


### [Native random sampling in ClickHouse](https://yomu.fyi/post/native-random-sampling-in-clickhouse.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: Mark Needham
- Published: May 22, 2026
- Type: Tutorial

ClickHouse native random sampling enables aggregate queries to execute against a representative fraction of data when querying full datasets is too slow. To maintain sampling accuracy, tables require a sample key derived from high-cardinality columns using hash functions like sipHash64 to ensure an even distribution. The sample key must be included in the primary key ORDER BY expression, preferably at the front to let ClickHouse quickly skip index granules. Queries can sample data deterministically by specifying a fractional ratio or a minimum row threshold, scaling aggregate counts using the virtual column \_sample\_factor. In benchmark tests on a 30-million-row dataset, sampling 10% of records reduced query latency by approximately 80% while retaining approximate metrics suitable for exploratory analysis.


### [Integrating the Rust Delta Kernel into ClickHouse](https://yomu.fyi/post/integrating-the-rust-delta-kernel-into-clickhouse.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: Melvyn Peignon, Kseniia Sumarokova, Raúl Marín
- Published: May 22, 2026
- Type: Problem & solution

ClickHouse initially built native protocol support to read Delta Lake tables, but maintaining pace with the format's evolving specification created high complexity and fragmented features. To resolve this, engineers integrated the Rust Delta Kernel to offload protocol handling while retaining ClickHouse's optimized Parquet data reading. The Delta Kernel parses JSON transaction logs, resolves snapshots, manages transactional write metadata, and exposes schema evolution mapping alongside Change Data Feed streams. ClickHouse also contributed upstream enhancements to the Rust kernel, adding dynamic runtime logging and asynchronous metadata processing via FFI handles. While ClickHouse can attach to existing Delta tables, the integration cannot yet initialize empty Delta Lake tables.


### [May 2026 newsletter](https://yomu.fyi/post/may-2026-newsletter.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: Mark Needham
- Published: May 21, 2026
- Type: Announcement

The May 2026 ClickHouse newsletter details enterprise observability migrations, ecosystem integrations, and core database enhancements. Digital banking platform Qonto migrated from Grafana Tempo to ClickHouse Cloud, compressing 231 TB of raw trace metadata to 376 GB while expanding query windows to two weeks and introducing an AI incident companion. Similarly, LINE MAN Wongnai consolidated multiple business clusters onto self-hosted ClickHouse, achieving 1.5 million rows per second peak ingest, 100% trace retention, and a 53% reduction in observability costs. Comparative log analytics benchmarks demonstrated that ClickHouse utilized 5x less disk space and achieved 4 to 6x faster cold query execution than Elasticsearch across 50 billion rows. Additional updates include the ClickHouse 26.4 release featuring improved SQL compatibility, new agent telemetry adapters in Mastra, and SQL-based alerting capabilities in ClickStack.


### [How Partition Access Visualizations Reduced our Data Lake S3 Cost by 33%](https://yomu.fyi/post/how-partition-access-visualizations-reduced-our-data-lake-s3-cost-by-3.md)
- Company: [Yelp](https://yomu.fyi/company/yelp.md)
- Author: Nick Del Nano, Data Streaming
- Published: May 21, 2026
- Type: Problem & solution

Data teams at Yelp struggled to identify table consumers and track how datasets were accessed across their petabyte-scale data lake. To solve this, the team built a usage attribution system using Amazon S3 server access logs to visualize time-based partition keys against access timestamps and IAM roles. These visualizations revealed distinctive patterns for daily batch jobs, backfills, and ad hoc queries, guiding migrations of thousands of tables to Apache Iceberg format. Furthermore, Yelp implemented a Default Access Retention strategy gated by IAM bucket policies to prevent accidental queries from resetting S3 Intelligent Tiering transitions. Combining these granular usage insights with targeted S3 storage classes reduced Yelp's S3 storage costs by 33 percent.


### [Optimizing Our Build Times by Migrating from Webpack to Rspack](https://yomu.fyi/post/optimizing-our-build-times-by-migrating-from-webpack-to-rspack.md)
- Company: [Yelp](https://yomu.fyi/company/yelp.md)
- Author: Benson Pan, Software Engineer
- Published: May 20, 2026
- Type: Problem & solution

As Yelp's JavaScript monorepo expanded over time, Webpack build speeds became a significant engineering bottleneck. To resolve this without rewriting complex configurations, the team selected Rspack for its direct compatibility with existing Webpack setups and plugins. Yelp implemented an adapter pattern to generate Rspack configurations dynamically from original Webpack files, enabling a staged opt-in rollout across development tools and CI environments. Following the initial migration, engineers addressed performance issues in barrel files using automated codemods to replace star re-exports and import-then-export patterns with named re-exports. Combined with Rspack's portable persistent caching, the migration reduced integration build times by approximately 52 percent and warm cache builds by up to 80 percent.


### [OlmoEarth v1.1: A more efficient family of Earth observation models](https://yomu.fyi/post/olmoearth-v1-1-a-more-efficient-family-of-earth-observation-models.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Kyle Wiggers
- Published: May 19, 2026
- Type: Announcement

OlmoEarth v1.1 is a new family of Earth observation models designed to reduce the compute required for large-scale satellite imagery predictions. Compute dominates the lifecycle from data export through preprocessing, inference, and post-processing, motivating a focus on transformer token sequence length. For Sentinel-2, the models collapse three resolution-specific tokens into one, reducing token counts threefold; this initially caused a 10 ppt drop on the m-eurosat kNN benchmark, so the pre-training regimen was modified. OlmoEarth v1.1 runs up to three times cheaper than v1 while maintaining similar performance across a mix of research benchmarks and partner-built tasks, although some regressions remain, and the same dataset is used to isolate methodological effects.


### [Scaling Airbnb’s identity graph with a unified knowledge graph infrastructure](https://yomu.fyi/post/scaling-airbnb-s-identity-graph-with-a-unified-knowledge-graph-infrast.md)
- Company: [Airbnb](https://yomu.fyi/company/airbnb.md)
- Author: Lucen Zhao
- Published: May 19, 2026

Airbnb migrated its massive identity graph from a third-party graph database vendor to a unified, in-house knowledge graph platform to solve long-tail latency and scaling bottlenecks. Built on JanusGraph with AWS DynamoDB for persistence and OpenSearch for indexing, the new architecture decouples storage operations from graph traversal logic. The migration improved P99 query latency, eliminated routine instance reboots, and supported a tenfold increase in write throughput.


### [What's new in ClickStack - April 2026](https://yomu.fyi/post/what-s-new-in-clickstack-april-2026.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: The ClickStack Team
- Published: May 19, 2026
- Type: Announcement

ClickStack's April release introduces several core enhancements across querying, alerting, and dashboard visualization capabilities. The platform added SQL-powered alerting, allowing teams to express complex statistical rules like rolling baselines and anomaly detection using native ClickHouse SQL and window functions without external pipelines. To improve search and filtering performance across common observability workloads, the default otel\_logs schema was redesigned with ClickHouse text indexes and adjusted ordering keys. Furthermore, heatmaps transitioned into a first-class chart type inside the shared visualization system for latency distribution tracking. The update also delivers persistent alert execution error reporting, expanded threshold comparison operators, and per-series number formatting across dashboards.


### [Introducing the Ettin Reranker Family](https://yomu.fyi/post/introducing-the-ettin-reranker-family.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Tom Aarsen
- Published: May 19, 2026
- Type: Announcement

Tom Aarsen introduces six Sentence Transformers CrossEncoder rerankers, ranging from 17M to 1B parameters, alongside their training data and full recipe. Each model uses an Ettin ModernBERT encoder with unpadded attention, RoPE, GeGLU, up to 8,192 tokens of context, and a modular classification head whose ablations favored CLS pooling over mean pooling. Training applies pointwise MSE distillation to scores from mixedbread-ai/mxbai-rerank-large-v2 using a broad and retrieval-focused dataset mixture. The conclusion reports state-of-the-art results at every released size, with the 400M model within 0.0024 and the 1B model within 0.0001 of the 1.54B teacher on MTEB, while all models outperform the ms-marco-MiniLM-L\*-v2 family on MTEB and NanoBEIR.


### [How we improved image download sizes on Medium with just four characters](https://yomu.fyi/post/how-we-improved-image-download-sizes-on-medium-with-just-four-characte.md)
- Company: [Medium](https://yomu.fyi/company/medium.md)
- Author: Scott Batson
- Published: May 18, 2026
- Type: Explainer

Image-heavy websites need browsers to choose efficient image variants, but srcset and sizes can require complex breakpoint rules for responsive layouts. The post distinguishes picture, which lets authors prescribe sources based on conditions, from srcset, which gives browsers candidate dimensions and lets them choose using context such as connectivity and cache state. It explains that browsers begin fetching images before layout is rendered, while lazy-loaded images can defer fetching until their dimensions are known. With WebKit's March 24, 2026 support for auto in sizes on lazy-loaded images, the recommendation is to prepend auto to those sizes attributes; unsupported browsers ignore the unparseable rule. On Medium, the change made the home feed's image-related download chunk about 7% smaller on a small, but not mobile-small, display, and the text says WordPress updated every instance broadly.


### [PaddleOCR 3.5: Running OCR and Document Parsing Tasks with a Transformers Backend](https://yomu.fyi/post/paddleocr-3-5-running-ocr-and-document-parsing-tasks-with-a-transforme.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: AlexZhang, cuicheng, Jun Zhang, Manhui Lin, Yue Zhang
- Published: May 18, 2026
- Type: Announcement

PaddleOCR 3.5 adds a more flexible inference-engine interface for OCR and document parsing, allowing supported models to run with a Hugging Face Transformers backend selected through engine="transformers". It keeps pipelines managed by PaddleOCR while exposing backend-specific settings such as dtype, device placement, and attention implementation through engine\_config. The release supports command-line and Python usage, with examples configuring GPU execution and float32, bfloat16, or SDPA options according to hardware and deployment environment. This provides a more natural path for RAG, Document AI, search, analytics, and agent stacks centered on PyTorch and Transformers, while the default paddle\_static backend is usually recommended when maximizing OCR or document parsing throughput is the priority.


### [How ChatFeatured migrated from PlanetScale Postgres to Postgres Managed by ClickHouse to power AI brand discovery](https://yomu.fyi/post/how-chatfeatured-migrated-from-planetscale-postgres-to-postgres-manage.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: ClickHouse
- Published: May 18, 2026
- Type: Problem & solution

ChatFeatured, a Toronto-based startup, helps brands track and improve how AI models discover and recommend them, including by analyzing cited sources and guiding content production. As prompt ingestion grew to about 1,000 prompts daily across six AI models, managed Postgres instances became slow, CPU reached 90% during ingestion windows, and PlanetScale hit an IOPS limit; analytics queries remained the deeper constraint. The team migrated from PlanetScale to Postgres managed by ClickHouse using the Postgres-to-Postgres ClickPipes connector, which handled initial load, schema transfer, continuous sync, and a five-minute maintenance window; the migration took 30 minutes overall. This setup retains Postgres for transactional workloads while providing a built-in path to ClickHouse for analytics, and a 30-day LifestyleRx query fell from about two and a half minutes to less than a second, with 20x compression reported for CDN log data.


### [Making LLMs faster without sacrificing accuracy](https://yomu.fyi/post/making-llms-faster-without-sacrificing-accuracy.md)
- Company: amazon.science
- Author: Tao Yu, Youngsuk Park
- Published: May 15, 2026
- Type: Problem & solution

Standard scaling laws optimize parameter count and training data budgets to minimize loss but ignore internal Transformer architecture configurations. To address throughput disparities among equally sized models, researchers developed a scaling framework incorporating hidden size, the multilayer perceptron to attention parameter ratio, and grouped-query attention. The method calibrates a correction surface over standard Chinchilla scaling laws to independently optimize separable architectural parameters for target accuracy. Evaluated across more than 200 models up to three billion parameters, the framework produced the Panda model family for maximum accuracy and the Surefire family for Pareto efficiency. Surefire models matched or exceeded LLaMA-3.2 accuracy while increasing inference throughput by 12% to 47% across vLLM and SGLang serving systems on modern GPUs.


[Newer posts](https://yomu.fyi/topic/performance/page/18.md) · [Older posts](https://yomu.fyi/topic/performance/page/20.md)