Loading…
Browse by type
Benchmark reads
Benchmark posts on Yomu, summarised newest first.
- 175 posts
- 20 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Benchmark posts
How Netflix Made AI Coding Agents More Efficient with JetBrains IDE Intelligence
Netflix integrated JetBrains’ headless code-intelligence engine into coding agents Claude Code and Codex to address agents’ struggles navigating Java/Kotlin code across compiled JAR boundaries. The engine provides deterministic indexing of project source files and libraries, and Netflix built a daemon and CLI to make the LSP callable by agents, keeping the index warm across repeated calls. In benchmarks on two real codebases—an internal Spring Boot service and the open-source DGS Framework—agents using JetBrains intelligence were typically 20-30% more efficient in tokens and time, with up to 65% improvement for stronger models like Opus 5 at high effort. Correctness held steady or improved across all configurations, and the gains were consistent across every model and agent tested. Netflix plans to extend the same benchmark methodology to Python, TypeScript, and Go, as coding agents for those languages face similar problems.
Plan or Retrieve First? A Scale Study of Tool Discovery Architectures
Tool-augmented LLM agents must discover relevant APIs from large registries before planning, but finite context windows prevent injecting all tool definitions into a single prompt. Two competing architectures are Retrieve-then-Plan (RtP), which embeds the query and retrieves similar tools via nearest-neighbor search, and Plan-then-Retrieve (PtR), which first generates abstract tool stubs as retrieval queries. Evaluating on ToolBench with 46,984 API endpoints across 50 categories reveals a clear complexity-dependent crossover: RtP outperforms PtR on single-tool queries while PtR outperforms RtP on multi-tool queries. PtR’s stub-hallucination failure mode acts as a double-edged sword—on simple queries it abstracts away proper-noun signals, yet on complex queries it decomposes multi-faceted intent into functional sub-needs. PtR roughly doubles inference cost, which pays off on multi-tool queries but hurts on single-tool ones, motivating complexity-adaptive routing driven by RtP’s top-1 retrieval score.
Local and Distributed Large LLM Inference with MLX and RDMA on Mac Studio
MLX enables running large language models locally on Apple silicon by exploiting the unified memory architecture of Mac Studio. The latest MLX release supports distributed inference across multiple Macs using RDMA over Thunderbolt, with pipeline or tensor parallelization schemes, and uses the jaccl communication collective. Benchmarks compare M3 Ultra and M5 Ultra across three models: Qwen 3.8 (27B) and Mistral Medium 3.5 (128B) fit on a single 512GB Mac, while the newly released Muse Spark Open (816B) requires parallelization across up to four machines at its 4-bit NVFP4 quantization. The M5 Ultra’s GPU Neural Accelerator delivers up to 4.6 times faster time-to-first-token than the M3 Ultra, while its 1.2 TB/s memory bandwidth improves generation throughput by 1.6 times for dense models. On four M5 Ultra Mac Studios with tensor parallelism, Muse Spark Open achieves over 36 tokens per second.
Sizing Lakebase Search: lessons from a real app and 4 million passages
Slide Hub, an internal Databricks app for slide search, uses Lakebase Postgres with Lakebase Search extensions (lakebase_vector and lakebase_text) to run hybrid vector and BM25 search on the same database as application data. The app stores 23k slides and indexes up to 4M passages from the MIRACL English corpus. Benchmarking shows that below 8 Compute Units (CU), Lakebase Postgres is the bottleneck; above 8 CU, the app’s worker count limits throughput. Candidate pool size significantly affects cold-start latency: 200 candidates preserves 90.5% of top-10 results while touching a third of the pages of 1,000 candidates. A 14-minute load test cost 0.13, with Lakebase compute billing only while active. For production, a 0.5 CU minimum costs ~ 31-40/month and avoids cold-start delays.
Why Redshift Serverless can’t match ClickHouse Cloud for real-time analytics
CostBench streamed 113.2 billion stock-market quotes into both systems while four aggregate queries ran every ten minutes and two drill-down queries ran every hour. ClickHouse asynchronous inserts order raw data and build pre-aggregations during ingestion, while Redshift relies on separate background sorting and materialized view refreshes. ClickHouse Cloud led in preparation cost ($28.69 versus 522.03), normalized query cost ( 0.05 versus $24.97), and accumulated query runtime (58.12 seconds versus 1.69 hours). The combined score shows ClickHouse Cloud delivered 1,996× better end-to-end performance per dollar in this continuous-ingestion comparison. The difference originates in the preparation path: ClickHouse makes data query-ready during inserts, whereas Redshift’s streaming, sorting, and refresh steps advance independently.
Sharding etcd on hosted control planes and what the numbers actually say
Sharding etcd on hosted control planes splits the key-value store into multiple instances by resource kind, a tech preview feature tested with a main shard for cluster state and a dedicated events shard on Red Hat OpenShift 4.21 with HyperShift. Under event pressure from synthetic storms and realistic crashlooping pods, the sharded configuration showed up to 96% fewer slow applies, faster compaction and raft snapshot durations, and 60% more WAL fsyncs and 90% more backend commits combined. The sharded arm never failed the event storm workload, while the standard arm occasionally did. Pod-ready latencies and workload finish times remained comparable across both configurations. The trade-offs are roughly 1.8 to 2.3 times the etcd container memory and a compaction regression under large custom resource quota pressure that is still under investigation.
Mix, Don't Tune: Bilingual Pre-Training Outperforms Hyperparameter Search in Data-Constrained Settings
In data-constrained language model pre-training, two remedies exist: aggressive hyperparameter tuning such as high weight decay, and mixing in data from a high-resource auxiliary language to expand the training distribution. This study systematically compares these levers across four model scales from 150M to 1.43B parameters, using Arabic as the low-resource target and English as the auxiliary, over approximately 1000 pre-training runs. Mixing yields larger improvements than hyperparameter tuning on both validation loss and downstream task accuracy, and the gap grows with model size. Mixing boosts performance by an amount equivalent to 2–3× the unique target data on validation loss and 2–13× on downstream task accuracy, with the gain scaling steeply with model size. The authors recommend mixing in a high-resource language, prioritizing the mixing ratio over hyperparameter tuning, and transferring hyperparameters from a small proxy model via μP, noting that target-language validation loss systematically underestimates mixing’s value.
Large Language Models for Machine Translation Quality Annotation: Humans and Models Are Both Challenged
The paper evaluates LLMs as alternatives to human judgment for machine translation quality annotation, comparing agreement with human annotators on two schemes: Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA). Tests cover a long-context set of 70 language pairs plus WMT23 and WMT25 data, examining both score and error span annotation agreement across language pairs and domains. Results show LLM-human agreement sometimes exceeds human-human agreement, but both vary substantially across annotation schemes, language pairs, and domains and remain unreliable for most settings. Humans are particularly challenged by fine-grained MQM annotations and low-resource language pairs, while LLMs struggle with minor errors, wrong language variants, and error span annotation. The authors suggest human-LLM collaborative annotation pipelines as a way to address the reliability issues identified in this work.
Why BigQuery can’t match ClickHouse Cloud for real-time analytics
ClickHouse Cloud delivered 438× better end-to-end performance per dollar than BigQuery under Capacity pricing, and 512× under On-demand pricing, while continuously ingesting 113.2 billion stock-market quotes. The CostBench methodology measures preparation cost, query cost, and query runtime as data arrives. ClickHouse Cloud led in all three components: preparation cost was 8.5× lower than BigQuery Capacity, normalized query cost was 94× lower, and accumulated query runtime was 50.8× lower. The difference originates in how each system handles incoming data: ClickHouse orders raw data and builds current summaries during inserts, while BigQuery makes new rows queryable before background clustering and materialized view refresh complete. This leaves BigQuery aggregate queries to include rows that refresh has not yet summarized, increasing cost and runtime.
The lakehouse is a better data warehouse: 2026 benchmarks and proof
The data warehouse served structured SQL workloads for two decades, but the lakehouse now supports BI, data engineering, ML, streaming, and AI agents from a single governed platform. Databricks reports a 77% improvement in its production SQL workload mix since 2022, with customer migrations showing 90% faster queries (Lumen) and 80-90% faster analytics (Trek Bicycle). Lakehouse//RT delivers sub-second analytical queries at high concurrency directly on governed lakehouse tables, with Databricks reporting sub-100-ms latency at 12,000 queries per second, collapsing the traditional architecture of separate serving databases. The lakehouse eliminates data copying between systems, handles unstructured data, and runs on open formats like Delta Lake and Apache Iceberg, governed by Unity Catalog. Customer cost reductions range from 25% to 75%, with AXA Japan reducing warehouse costs by 60% and ETL costs by 70%, driven by eliminating redundant systems and data copies.
Why Databricks can’t match ClickHouse Cloud for real-time analytics
The CostBench benchmark streamed 113.2 billion stock-market quotes at one million rows per second while running aggregate and drill-down queries to compare ClickHouse Cloud and Databricks for real-time analytics under continuous ingestion. ClickHouse Cloud delivered 752× better end-to-end performance per dollar than the Databricks stack using Serverless SQL, Zerobus Ingest, liquid clustering, and incremental materialized views. ClickHouse Cloud led in all three components: preparation cost ($28.69 versus 695.60), normalized query cost ( 0.05 versus $2.65), and accumulated query runtime (56.39 seconds versus 29.10 minutes). The gap stems from ClickHouse preparing ordered raw data and current summaries on the insert path, while Databricks requires separate clustering and refresh services that cost more and can leave queries with unfinished layout work. Databricks’ new Lakehouse//RT, still in beta, was not included in this comparison.
The Agent Said It Was Done. The Database Disagreed.
ThinkingBox, a benchmark from Microsoft and Hugging Face, grades AI agents on the database state and side effects they leave behind rather than on tool calls or final responses. In a common-set ablation covering 121,680 valid trials across 12 LLM models, 79,853 attempts failed executable checks, and 67.24% of those failures still terminated cleanly and invoked a state-changing tool without error, with wrong field values in 77.61% of them. The benchmark runs every task 20 independent times from identical clean backends, reporting pass@1, pass@20, and observed 20/20 consistency to separate capability from reliability. For example, Kimi-K3 solved 93.89% of tasks at least once but only 13.41% consistently, while Claude Opus 5 completed 47.53% of tasks on every attempt, and Claude Opus 5.5 gained no additional dependability over its predecessor despite higher pass@1. ThinkingBox is available through Hugging Face’s OpenEnv with MIT-licensed code and CDLA-Permissive-2.0 benchmark data.
Benchmarking AI decision models against traditional guardrails
Enterprise generative AI applications require guardrails that balance flexibility with reliability. Decision models such as TypeSafe AI’s Jev produce fixed decisions with guaranteed schema and type safety, operating faster and cheaper than LLM-as-a-judge approaches. The Red Hat AI Safety team benchmarked nine guardrails across four methodologies for prompt injection and content safety tasks: pre-trained CPU-scale classifiers like Red Hat’s DeBERTa-v3 and Granite Guardian HAP-125m, older zero-shot models like BART-large-mnli, modern LLM-as-a-judge safety models including Shieldstral and Nemotron-3.5, and Jev-style decision models including Laya and DiffusionGemma. Each guardrail was evaluated using stock and custom risk policies, with Jev and DiffusionGemma using zero-shot questions. The results highlight trade-offs: Jev provides zero-shot flexibility and low latency, but purpose-built classifiers often match its speed and cost while delivering superior task-specific performance, and LLM-as-a-judge models can be tuned for higher accuracy.
GPU virtualization at scale: AMD MI300X SR-IOV on Red Hat OpenStack Services on OpenShift
Enterprise AI environments are moving from dedicated GPU servers to shared accelerator infrastructure, requiring higher GPU utilization with strong isolation. AMD Instinct MI300X GPUs support hardware-assisted SR-IOV in CPX mode, each exposing eight virtual functions with separate PCI and VRAM allocation. When integrated with Red Hat OpenStack Services on OpenShift, the control plane runs on OpenShift while RHEL compute nodes use EDPM, with Nova scheduling VFs via VFIO PCI passthrough. Validation on an 8-GPU system created 64 VFs, demonstrated 64 VMs with one VF each, and tested up to 32 VFs per VM. BabelStream benchmarks showed approximately 96-99% of bare-metal memory bandwidth, while the testing also identified limitations in GPU peer-to-peer DMA and runtime VF lifecycle.
On the Effectiveness-Fluency Trade-Off in LLM Conditioning: A Systematic Study
Controlling the output of Large Language Models (LLMs) is a central challenge for their reliable deployment, yet current conditioning methods are often evaluated narrowly on effectiveness without considering generation quality. This study systematically investigates a range of conditioning methods in both concept injection and removal scenarios, including efficient steering, activation steering, prompting, and supervised fine-tuning. The authors find that efficient steering methods frequently achieve their conditioning goal at a steep cost to fluency, degrading generation quality. They identify a critical previously overlooked interaction: activation steering methods are far less effective on instruction-tuned models than on their base counterparts. Simple prompting and supervised fine-tuning are viable options for concept injection but are not as good at concept removal, while cheaply computed textual metrics highly correlate with costly LLM-as-judge scores, providing practical insights for evaluation.
We tested our own WAF with frontier AI models. Here’s what we found
Cloudflare built a WAF tester that uses frontier AI models to iteratively mutate attack payloads based on real-time response feedback. The tester, implemented in Python, orchestrates an LLM proposal and review loop against an authorized customer staging environment protected by Cloudflare’s WAF. It ran 45 scenarios across six attack categories—XSS, SQLi, CMDi, SSRF, path traversal/LFI, and Log4j—starting from known blocked exploits and varying encoding, request placement, or destination representation. The vast majority of 1,107 recorded attempts were blocked; the few non-blocked requests were triaged by humans and used to create new detections that harden security for all Cloudflare customers. The post stresses that a WAF bypass still needs an exploitable application to succeed, so patching software remains essential, and suggests combining Managed Rules, Attack Signature Detection, and positive security controls for defense in depth.
Can your Postgres survive a bad query?
Postgres memory tuning is difficult because work_mem applies per operation, not per query, and parallel workers multiply memory usage without a query-wide cap. Some executor structures, such as those in recursive queries, have no disk-backed fallback and can grow until the backend runs out of memory. A benchmark using a recursive query that exhausts memory tested ClickHouse Managed Postgres, Cloud SQL, PlanetScale, and Amazon RDS on 16 GiB instances. ClickHouse Managed Postgres enforces a memory ceiling at 57% of host RAM, rejecting queries that exceed it, while RDS allows backends to consume nearly all RAM until the host crashes. Cloud SQL and PlanetScale enforce similar caps, but ClickHouse Managed Postgres’s ceiling is lower, a deliberate tradeoff that returns clean query failures instead of forcing every client through crash recovery.
How NVIDIA DSX MaxLPS Maximizes AI Factory Throughput and Efficiency
AI factories often underutilize power capacity due to static provisioning for peak GPU draw. NVIDIA DSX MaxLPS uses policy-governed dynamic power allocation to reclaim stranded capacity. In a joint evaluation with Nscale using Kimi K2.5 workloads on GB300 NVL72 systems, DSX MaxLPS enabled 37% more GPUs (192 vs 140) within the same 264.4 kW budget. Aggregate throughput increased 49.2% to 1.62 million tokens/s, while per-instance throughput remained unchanged and median latency stayed within 5% of baseline. However, P99 time to first token increased 17%, highlighting the need to evaluate tail latency alongside capacity gains.
Scaling GitOps: The "pull" architecture for global fleets
Red Hat conducted a performance and scale test of the argocd-agent pull architecture for GitOps, using 42 physical machines, 500 spoke clusters, and software including OpenShift 4.21 and OpenShift GitOps 1.20. The pull model decentralizes reconciliation to lightweight agents on each managed cluster, which pull configurations from a central hub, improving security by eliminating inbound administrative ports and reducing cross-cluster network traffic by up to 80%. In the bootstrap scenario, generating and syncing 31,200 applications across 480 clusters completed in 13 minutes 3 seconds, while the sync-to-commit scenario, which includes a polling delay, took 13 minutes. The test revealed bottlenecks including Kubernetes API limits on the hub, informer processing slowdowns causing watch restarts, event writer retry issues under load, and ApplicationSet controller RAM consumption. The team also deleted all 31,200 applications in 4 minutes 53 seconds using cascading deletion, and the insights are informing future OpenShift GitOps releases.
Postgres on NVMe: performance and the convergence of transactions and analytics
The post examines how local NVMe storage changes Postgres performance, particularly when working sets exceed memory and disk latency becomes the bottleneck. It argues that while faster storage accelerates transactional workloads, it does not alter the row-store layout, so analytical queries still require different approaches. Benchmark results show 9.2× more throughput on NVMe, with a median of 16,030 TPS versus 1,734 on EBS. The post introduces WalShadow, an open-source replication engine that reads the physical Postgres WAL to enable sub-second replication to ClickHouse without logical slots. It concludes that failover is solvable with tools like Patroni, and that storage latency is only part of the performance problem.
Top topics
- Architecture834
- Performance816
- LLMs668
- Machine Learning622
- Open Source567
- AI Agents559
- AI476
- Developer Experience448
Weekly digest
The best engineering reads, once a week.