---
title: "Performance"
description: "817 posts about Performance, summarised, each linking to the original."
---

# Performance
> 817 posts about Performance, summarised, each linking to the original.

## Articles

### [How Much Memory Does Your Agent Actually Need?](https://yomu.fyi/post/how-much-memory-does-your-agent-actually-need.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Vatche Isahagian, Gaodan Fang, Jayaram Radhakrishnan, Punleuk Oum, Ashwath Vaithinathan Aravindan, Evelyn Duesterwald, G Thomas, Vinod Muthusamy, Merve Unuvar, Ayhan Sebin
- Published: Aug 18, 2026
- Type: Benchmark

The post examines how much agentic memory different models can use effectively, evaluating ALTK-Evolve's self-distilled guidelines across eight models on AppWorld's 585 multi-step tasks. Memory is produced by extracting and consolidating lessons from successful and unsuccessful trajectories, then delivering either the full guideline set on every ReAct step or a fixed core plus task-relevant retrieval, without updating model weights. Results identify three patterns: strong models with headroom benefited from the full set, gpt-oss-120b gained 16.1 percentage points in task completion from curated retrieval, and GLM-5 showed no measurable gain; stricter scenario completion sometimes improved more. Curated retrieval gave gpt-oss-120b that gain with only 5% more tokens versus 51% for its full set, while prompt caching is presented as a way to reduce recurring input cost, supporting calibration to model capability rather than simply accumulating memory.


### [How AI Coding Agents Can Unlock Materials Simulation with NVIDIA ALCHEMI Toolkit](https://yomu.fyi/post/how-ai-coding-agents-can-unlock-materials-simulation-with-nvidia-alche.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Elizabeth Goodman
- Published: Aug 18, 2026
- Type: Benchmark

Atomistic simulation combines scientific judgment, compute-efficient implementation, and an accessible interface to the simulation stack, but the MLIP ecosystem lacks the mature interfaces available for classical force fields. NVIDIA ALCHEMI Toolkit addresses implementation with composable, PyTorch-native, GPU-accelerated building blocks and in-flight batching, while its agent skills and reference files supply API patterns to coding agents. The post benchmarks 45 generated pipelines across silicon equation of state, copper adsorption, and lithium self-diffusion, checking code features and running representative workloads on NVIDIA H200 GPUs. All 45 scripts used batched GPU execution, 38 completed screening, and all 15 production representatives completed; however, agents did not assess physical validity, and thermostat choices could suppress measured lithium diffusion by 3–5×, so independent validation against DFT, experiment, or another reference remains necessary.


### [Improving infrastructure efficiency for growing demand in the age of AI](https://yomu.fyi/post/improving-infrastructure-efficiency-for-growing-demand-in-the-age-of-a.md)
- Company: [Dropbox](https://yomu.fyi/company/dropbox.md)
- Author: Ilya Yakovlev
- Published: Aug 18, 2026

Dropbox manages growing infrastructure demands from AI and standard workloads by maximizing the efficiency of its existing data centers rather than just building more capacity. Through a system-level approach spanning software, hardware, and physical facility design, the company dynamically manages power and balances workloads. These optimizations, combined with higher storage density and data-driven hardware lifecycle management, have significantly reduced energy use per unit of storage.


### [Run Massive-Scale UMAP in Minutes Using Multiple GPUs—Without Losing Accuracy](https://yomu.fyi/post/run-massive-scale-umap-in-minutes-using-multiple-gpus-without-losing-a.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Tanya Lenz
- Published: Aug 18, 2026
- Type: Explainer

UMAP workflows on tens to hundreds of millions of vectors are constrained by the cost of constructing an all-neighbors k-nearest-neighbor graph, especially when repeated runs support exploration and parameter tuning. The NVIDIA cuML and cuVS 25.06 feature distributes this graph-construction stage across multiple GPUs while retaining the out-of-core design that partitions data into balanced clusters, overlaps vectors across nearby clusters, computes local graphs, and merges them into a global graph. knn\_n\_clusters and knn\_overlap\_factor expose the memory, runtime, and graph-quality trade-off: more clusters reduce per-GPU data, while more overlap can improve nearest-neighbor recall at higher computational and memory cost. On eight NVIDIA H100 GPUs, the reported evaluation found up to 74x end-to-end speedup over a projected CPU runtime for the 106M-vector MIRACL dataset, completing it in 8 minutes. The post states that performance continues to improve from one to eight GPUs while preserving comparable embedding quality across configurations.


### [How Sony LIV uses ClickHouse Cloud to deliver live streaming analytics at billion-row scale](https://yomu.fyi/post/how-sony-liv-uses-clickhouse-cloud-to-deliver-live-streaming-analytics.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: Aug 18, 2026
- Type: Problem & solution

Sony LIV’s legacy analytics environment combined batch pipelines, Elasticsearch, BigQuery, and other telemetry stores, creating slow, costly, fragmented analysis as streaming activity reached billions of events per day. During live sports, the platform needed sub-second queries, high ingestion throughput, cost-efficient long-term storage, and horizontal scaling for sudden traffic spikes. Sony LIV consolidated these workloads on ClickHouse Cloud: client, streaming, CDN, and advertising systems send events through Amazon Kinesis and ClickPipes into a ClickHouse cluster serving dashboards, product analytics, QoE, and SRE workflows. The source reports that queries previously taking tens of seconds to minutes now run in sub-second to low-single-digit seconds, improving operational responsiveness, engineering productivity, and the handling of major live-event traffic.


### [Evaluating AI Agents Live at the Grounded Reasoning Cup](https://yomu.fyi/post/evaluating-ai-agents-live-at-the-grounded-reasoning-cup.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks AI Research Team
- Published: Aug 18, 2026
- Type: Benchmark

The Grounded Reasoning Cup evaluated whether AI agents could generalize grounded reasoning improvements from OfficeQA to a newly released, enterprise-style document benchmark. Eleven academic teams had about two months to build agents with a partner lab’s model family, then applied them live to OfficeQA Pro V2, based on U.S. Treasury Accounts of Receipts and Expenditures, after a 36-hour release window. Stanford won with 63.3% accuracy, while the top systems combined document preprocessing, targeted or hybrid retrieval, specialized tools, parallelism, and answer verification; 18.8% of questions remained unsolved by every team. The results show that benchmark gains did not reliably transfer, and that parsing, retrieval, tool use, verification, and operational infrastructure can matter as much as model choice in end-to-end enterprise reasoning.


### [What's New with Monitoring in PostgreSQL 19](https://yomu.fyi/post/what-s-new-with-monitoring-in-postgresql-19.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: Aug 18, 2026
- Type: Explainer

PostgreSQL 19, still in beta when this was written, adds monitoring and observability changes spanning lock contention, logging, WAL, wait events, remote messages, and multixacts. The default for log\_lock\_waits changes to on, while log\_min\_messages gains per-process-type levels and autoanalyze receives its own duration setting. A new wal\_fpi\_bytes counter exposes full-page-image volume cluster-wide, per session, per query, and in VACUUM and ANALYZE logs; new wait events expose WAL write, flush, and replay stages plus COPY file, pipe, and program I/O. Remote server messages now use normal PostgreSQL logging with origin prefixes, and pg\_get\_multixact\_stats() surfaces multixact usage that previously required filesystem inspection and educated guessing. The post also lists upgrade work for parsers, metric collectors, wait-event dictionaries, configuration UIs, and alert thresholds, while noting that final release notes may differ.


### [Multi-Vector (Late Interaction) Embedding Models with Sentence Transformers](https://yomu.fyi/post/multi-vector-late-interaction-embedding-models-with-sentence-transform.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Tom Aarsen, Antoine Chaffin, Raphael Sourty
- Published: Aug 18, 2026
- Type: Tutorial

Sentence Transformers v6.0 adds MultiVectorEncoder, bringing ColBERT-style late-interaction retrieval into the library’s existing API for dense, sparse, and reranker models. The encoder retains one projected, typically 128-dimensional vector per token instead of pooling a document into one vector, allowing documents to be encoded independently and indexed offline. At scoring time, MaxSim finds each query token’s highest dot-product match among document tokens and sums those maxima; contextual embeddings support both paraphrase matching and exact-token evidence. The guide covers checkpoint loading, encoding, scoring, semantic search, retrieval and reranking, indexing, token pooling, inference speed, evaluation, and retrieval from page images, audio, and video. It also quantifies the storage trade-off: 4,874 passages produced 608,414 token vectors and a 311.5 MB float32 index, while fast-plaid compression reduced that example to 92 MB.


### [How Databricks Feature Store serves features with sub-second freshness](https://yomu.fyi/post/how-databricks-feature-store-serves-features-with-sub-second-freshness.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Ian Ackerman, Nick Joung, Abhay Bothra
- Published: Aug 17, 2026
- Type: Explainer

Databricks Feature Store addresses the lag between batch-computed machine-learning features and decisions that depend on events from seconds earlier. It lets a feature definition drive both offline batch processing and online streaming pipelines, orchestrating Kafka ingestion, Spark Real-Time Mode (RTM), Lakebase, and Model Serving. For a rolling 10-minute transaction sum, RTM updates per entity state in local RocksDB, expires contributions per event, and writes revised values to Lakebase through a streaming JDBC sink. The described path reports end-to-end p99 200ms latency from Kafka arrival to online feature availability, while Model Serving retrieves declared dependencies automatically at inference. Tumbling and sliding windows remain options when fewer, cheaper updates are acceptable.


### [Same Cluster, 33 Points More Utilization: What Changed Was the Order](https://yomu.fyi/post/same-cluster-33-points-more-utilization-what-changed-was-the-order.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Gabriel Pimenta de Freitas Cardoso, Breno de Almeida Beleza, Francisco de Almeida Rocha Alves, Bruno Duarte
- Published: Aug 17, 2026
- Type: Benchmark

The post presents a constraint-aware GPU allocator as a response to utilization losses caused by fixed real-time reservations and FIFO ordering under contention. It compares the allocator with FIFO on identical hardware and workloads across seven benchmark scenarios, modeling batch-like jobs as contiguous GPU blocks and real-time inference as a timestep-by-timestep demand curve with bounded GPU swaps. Across five contention scenarios, utilization improved from 52–85% to 72–88%, while priority-weighted value increased 24.6–105.1%; the strongest case gained 33 percentage points of utilization and 105% value, and the scale test gained 15.9% value at identical utilization and throughput. The scheduler optimizes a 24-hour horizon but commits only the current timestep, re-running every 30–60 minutes so fresh observations absorb forecast error while preserving running work.


### [The prototyping tax is killing your AI roadmap](https://yomu.fyi/post/the-prototyping-tax-is-killing-your-ai-roadmap.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Navdeep Alam, Zhe Sun
- Published: Aug 17, 2026
- Type: Problem & solution

The post argues that the “prototyping tax”—delay between an AI-powered idea and a working MVP—causes initiatives to lose momentum, often because organizational context is fragmented across teams, tools, and codebases. Traditional encapsulation and siloed domain knowledge can leave agents with syntax and contracts but not business intent, making production codebases harder to navigate than personal projects. The proposed response is agentic development grounded in business semantics and governed data, with alignment occurring through working builds while CI/CD, code review, and production rigor remain unchanged. The post recommends tracking time-to-prototype, first-pass acceptance rate, and PoC-to-production rate to distinguish faster demos from durable delivery. It cites a benchmark of 401 data tasks and Abacus Insights results: 77% accuracy at roughly half the cost for a platform-native agent, plus about 40% less manual effort and onboarding to first value in roughly half the time.


### [Developing Nemotron 3.5 Lightning NVFP4 with QAD Using NVIDIA Model Optimizer](https://yomu.fyi/post/developing-nemotron-3-5-lightning-nvfp4-with-qad-using-nvidia-model-op.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Tanya Lenz
- Published: Aug 17, 2026
- Type: Tutorial

NVIDIA’s post presents a workflow for developing an NVFP4 checkpoint of the Nemotron 3.5 Lightning 30B-A3B model, targeting lower memory use and higher inference throughput without sacrificing accuracy. The method starts with post-training quantization (PTQ), producing a W4A16 student from the BF16 teacher, then applies quantization-aware distillation (QAD), where simulated quantization and KL-divergence loss align student logits with the frozen teacher. For the selected four\_over\_six recipe, 32K calibration performed best, and the recipe using W4A16 Mamba linears offered the best reported trade-off between accuracy degradation and inference performance; the checkpoint is 22 GB versus 66 GB and delivers up to 4x faster throughput. The workflow includes Model Optimizer examples and launchers for Megatron-LM and Megatron-Bridge, covering calibration, distillation, evaluation, and export.


### [No Dumb Questions: What is AI context architecture? Why not just build your own?](https://yomu.fyi/post/no-dumb-questions-what-is-ai-context-architecture-why-not-just-build-y.md)
- Company: [Stack Overflow](https://yomu.fyi/company/stack-overflow.md)
- Author: Phoebe Sajor
- Published: Aug 14, 2026
- Type: Explainer

AI context architecture defines the boundaries, data access, memory, and fallback behavior that constrain AI agents and make their outputs more predictable. The discussion distinguishes architecture as system design, infrastructure as the storage and delivery of context, and engineering as implementation; MCP is treated as an architectural protocol, while RAG spans all three layers. Examples include restricting retrieval to car or sports-car tires, preserving agent history, and routing incomplete or low-trust knowledge to subject-matter experts through a human-in-the-loop flow. The buy-versus-build discussion notes that coding a custom architecture is possible, but buying can provide accumulated experience with data treatment, edge cases, and recurring operational problems, while good architecture also reduces token costs and aims for consistent, predictable answers.


### [Total eclipse of the Internet: traffic impacts in Iceland, Spain, and Portugal](https://yomu.fyi/post/total-eclipse-of-the-internet-traffic-impacts-in-iceland-spain-and-por.md)
- Company: [Cloudflare](https://yomu.fyi/company/cloudflare.md)
- Author: Sabina Zejnilovic
- Published: Aug 13, 2026
- Type: Explainer

On August 12, a total solar eclipse crossed the North Atlantic and Europe, passing directly over Iceland, northern Spain, and Portugal. Cloudflare Radar analyzed HTTP request volumes in five-minute intervals across affected countries, comparing the data against a baseline derived from the previous three Wednesdays. Traffic reductions aligned closely with peak solar obscuration, with totality regions recording drops between 15% and 30% and specific areas shifting by up to -46.7%. Once the sun reappeared, request volumes rebounded to normal levels within minutes as users resumed device activity. The observed traffic declines were driven by aggregate human behavior rather than network failures.


### [Smart Routing in Unity AI Gateway: Match frontier quality with 30%+ lower cost per task](https://yomu.fyi/post/smart-routing-in-unity-ai-gateway-match-frontier-quality-with-30-lower.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Ankit Mathur, Ivan Zhou, Bryan Qiu, Rohit Agrawal, Elise Gonzales, Kelly Albano
- Published: Aug 13, 2026
- Type: Announcement

Unity AI Gateway’s Smart Routing, now in Beta, addresses the cost and choice overload of assigning coding tasks to increasingly diverse models and harnesses. It uses task-aware routing: a small, low-latency model classifies the initial task description and metadata, then a policy selects a cheaper, medium, or frontier model; Omnigent can also choose the coding harness and route sub-agents. The design keeps consecutive turns on one model to preserve cache-hit rates, while escalating complex work and delegating simpler tasks to lower-cost options. On internal coding workloads, Smart Routing achieved 35% savings and outperformed any single model at 65% of Opus 5’s cost; public benchmarks reported matched Opus 5 performance with 56% savings. The team identifies mid-session reassessment, real-session evaluation, and affordable switching as continuing challenges.


### [Record, train, and deploy from one place with Strands Agents, LeRobot, and Hugging Face Storage Buckets](https://yomu.fyi/post/record-train-and-deploy-from-one-place-with-strands-agents-lerobot-and.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Sundar Raghavan, Steven Palma, Cagatay Cali, Arron Bailiss, Yin Song
- Published: Aug 13, 2026
- Type: Tutorial

This walkthrough presents a streaming data loop for Strands Robots that records robot demonstrations, stores them in Hugging Face Storage Buckets, trains from the growing dataset, and deploys a resulting policy back to hardware. It uses the LeRobot on-disk format throughout: Robot("so100") records through DatasetRecorder, sync\_dataset\_to\_bucket writes changed bytes to a mutable bucket, and stream\_dataset reads the dataset from the Hub without a full local download while decoding camera video on the fly. The agent can decide which episodes to keep and when to retrain, while the same Robot object records and streams data; deployment changes the robot mode keyword from simulation to real. The default simulation path runs with a mock policy that produces a valid dataset but not a useful one, while hardware, bucket, and training paths require additional dependencies.


### [What’s new in the ClickHouse Grafana plugin](https://yomu.fyi/post/what-s-new-in-the-clickhouse-grafana-plugin.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: Aug 13, 2026
- Type: Announcement

ClickHouse’s Grafana plugin has shipped a set of changes aimed at making common observability tasks—finding log lines, narrowing to services, and detecting deployments—faster without requiring SQL. Version 4.20 adds single-table configuration and compact query mode, where users search logs, apply type-aware filters, review query history, and see or edit generated ClickHouse SQL. Investigation can proceed by clicking through collapsible OpenTelemetry Resource, Scope, and Log attributes, filtering values or selected message text, while dashboards can discover Map keys and values across tables. The release also includes interactive Logs Explorer, Traces Explorer, and per-service OpenTelemetry dashboards, plus guided variable and annotation editors that generate editable SQL for distinct values and change markers such as deployments and rollbacks. Additional changes cover schema-introspection caching, trace-ID lookup, Grafana 13 compatibility, SQL validation, backend OpenTelemetry instrumentation, and forwarding Grafana identity to ClickHouse; compact mode initially remains limited to single-table data sources.


### [Reducing Text2SQL latency with parameterized query templates](https://yomu.fyi/post/reducing-text2sql-latency-with-parameterized-query-templates.md)
- Company: [AWS](https://yomu.fyi/company/aws.md)
- Author: Yury Brukau
- Published: Aug 13, 2026
- Type: Problem & solution

In Text2SQL systems, generating SQL on the fly with frontier foundation models can cause response times of 25 to 30 seconds due to 60,000-token prompts and model latency. To solve this without degrading accuracy, an intelligent caching layer was built using parameterized SQL query templates paired with question embeddings. When an incoming question matches a cached template via semantic search above a confidence threshold, named entity recognition extracts values to populate template placeholders, bypassing the initial SQL generation step. In production, this approach reached a 60% cache hit rate after two weeks, cutting per-hit token consumption by roughly 90% and reducing overall latency by 80% to under 5 seconds.


### [HeyGen x Google Cloud: Bringing Avatar IV to TPUs](https://yomu.fyi/post/heygen-x-google-cloud-bringing-avatar-iv-to-tpus.md)
- Company: [Google](https://yomu.fyi/company/google.md)
- Published: Aug 13, 2026
- Type: Benchmark

HeyGen and Google Cloud ported Avatar IV, an 18B-plus-parameter diffusion stack for streamed talking-head video, to an eight-chip Trillium (v6e) host. The pipeline uses two diffusion transformers and a VAE decoder, with FSDP weight sharding and Ulysses sequence parallelism across the mesh, and runs through torchax, JAX, XLA, and shape-specific Pallas kernels. Optimization targeted exposed all-to-all collectives, frame-misaligned sparse-attention blocks, serial softmax work, layout copies, scheduler choices, and custom-kernel cost estimates; changes were measured per video chunk against output-quality gates. The resulting pipeline was 1.86× faster than the first working TPU version, comparable in performance to the 8×H100 production setup and up to 25% more cost efficient per generated-video minute, while shipped changes preserved byte-identical output or passed defined similarity review.


### [Databricks Network Configuration delivery to Tens of Millions of Serverless VMs](https://yomu.fyi/post/databricks-network-configuration-delivery-to-tens-of-millions-of-serve.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Manish Bansal, Yankai Zhang, Chen He
- Published: Aug 12, 2026
- Type: Problem & solution

Databricks’ serverless platform launches tens of millions of VMs daily, and each VM needs network configuration before serving customer workloads. The original design synchronously aggregated data from multiple upstream services on the cluster-creation critical path, producing 5,000-ms p99 RPC latency and availability bottlenecks as request volume reached billions per day. The replacement uses change events through a message queue, asynchronous per-workspace recomputation, partition-local snapshots, and periodic reconciliation to recover from missed events. The serving path now performs a single snapshot-store read without upstream calls, while events carry workspace and resource identifiers rather than customer data. After rollout, p99 RPC latency fell to 125 ms, server success rate rose to 99.99%, upstream call volume dropped 86%, and the legacy synchronous framework was deprecated.


[Newer posts](https://yomu.fyi/topic/performance/page/6.md) · [Older posts](https://yomu.fyi/topic/performance/page/8.md)