Loading…
Browse by type
Benchmark reads
Benchmark posts on Yomu, summarised newest first.
- 174 posts
- 20 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Benchmark posts
How Much Memory Does Your Agent Actually Need?
The post examines how much agentic memory different models can use effectively, evaluating ALTK-Evolve's self-distilled guidelines across eight models on AppWorld's 585 multi-step tasks. Memory is produced by extracting and consolidating lessons from successful and unsuccessful trajectories, then delivering either the full guideline set on every ReAct step or a fixed core plus task-relevant retrieval, without updating model weights. Results identify three patterns: strong models with headroom benefited from the full set, gpt-oss-120b gained 16.1 percentage points in task completion from curated retrieval, and GLM-5 showed no measurable gain; stricter scenario completion sometimes improved more. Curated retrieval gave gpt-oss-120b that gain with only 5% more tokens versus 51% for its full set, while prompt caching is presented as a way to reduce recurring input cost, supporting calibration to model capability rather than simply accumulating memory.
How AI Coding Agents Can Unlock Materials Simulation with NVIDIA ALCHEMI Toolkit
Atomistic simulation combines scientific judgment, compute-efficient implementation, and an accessible interface to the simulation stack, but the MLIP ecosystem lacks the mature interfaces available for classical force fields. NVIDIA ALCHEMI Toolkit addresses implementation with composable, PyTorch-native, GPU-accelerated building blocks and in-flight batching, while its agent skills and reference files supply API patterns to coding agents. The post benchmarks 45 generated pipelines across silicon equation of state, copper adsorption, and lithium self-diffusion, checking code features and running representative workloads on NVIDIA H200 GPUs. All 45 scripts used batched GPU execution, 38 completed screening, and all 15 production representatives completed; however, agents did not assess physical validity, and thermostat choices could suppress measured lithium diffusion by 3–5×, so independent validation against DFT, experiment, or another reference remains necessary.
Evaluating AI Agents Live at the Grounded Reasoning Cup
The Grounded Reasoning Cup evaluated whether AI agents could generalize grounded reasoning improvements from OfficeQA to a newly released, enterprise-style document benchmark. Eleven academic teams had about two months to build agents with a partner lab’s model family, then applied them live to OfficeQA Pro V2, based on U.S. Treasury Accounts of Receipts and Expenditures, after a 36-hour release window. Stanford won with 63.3% accuracy, while the top systems combined document preprocessing, targeted or hybrid retrieval, specialized tools, parallelism, and answer verification; 18.8% of questions remained unsolved by every team. The results show that benchmark gains did not reliably transfer, and that parsing, retrieval, tool use, verification, and operational infrastructure can matter as much as model choice in end-to-end enterprise reasoning.
DeepSeek V4 Pro 0813 vs GPT-5.6 Sol on DeepSWE: Cost, Coding, and Routing
DeepSeek V4 Pro 0813 costs $0.24 per rollout and achieves 62.8% pass@1 on DeepSWE, while GPT-5.6 Sol reaches 72.7% pass@1 at $8.37 per rollout—a 35x price gap. At four attempts, DeepSeek V4 Pro 0813 surpasses Sol, with 88.5% pass@4 versus 85.8%. Sol is faster and more reliable, completing a rollout in 17 minutes and 53 steps, and breaking tests in only 11% of its failures. A cascade strategy—running DeepSeek V4 Pro 0813 first and escalating to Sol on test failure—solves 83.0% of tasks at $3.35 each, outperforming either model alone. Per $100, DeepSeek V4 Pro 0813 solves 260 tasks versus Sol’s 9, making it the value leader for budget-constrained workloads.
Same Cluster, 33 Points More Utilization: What Changed Was the Order
The post presents a constraint-aware GPU allocator as a response to utilization losses caused by fixed real-time reservations and FIFO ordering under contention. It compares the allocator with FIFO on identical hardware and workloads across seven benchmark scenarios, modeling batch-like jobs as contiguous GPU blocks and real-time inference as a timestep-by-timestep demand curve with bounded GPU swaps. Across five contention scenarios, utilization improved from 52–85% to 72–88%, while priority-weighted value increased 24.6–105.1%; the strongest case gained 33 percentage points of utilization and 105% value, and the scale test gained 15.9% value at identical utilization and throughput. The scheduler optimizes a 24-hour horizon but commits only the current timestep, re-running every 30–60 minutes so fresh observations absorb forecast error while preserving running work.
DeepSeek V4 Pro 0813 vs Claude Fable 5 on DeepSWE: Cost, Coding, and Routing
The DeepSWE benchmark comparison between DeepSeek V4 Pro 0813 and Claude Fable 5 reveals a trade-off between cost and accuracy. Claude Fable 5 leads on the first attempt with 69.7% pass@1 versus DeepSeek V4 Pro’s 62.8%, but costs 90 times more per rollout ($21.63 vs $0.24). Under repeated attempts, DeepSeek V4 Pro surpasses Fable, achieving 88.5% pass@4 compared to Fable’s 84.1%. The two models show unusually low per-task correlation (0.39), meaning they fail on different tasks. A cascade strategy that runs DeepSeek V4 Pro first and escalates to Claude Fable 5 only on failures solves 82.7% of tasks at $8.28 per task, outperforming either model used alone.
State of Open Models: Summer 2026 Observations
The report analyzes Hugging Face Hub activity from January through August 2026 to assess how open-model releases, usage, licensing, and agent traffic are changing. Public repositories expanded from 2.43 to 2.96 million models, while 85.6% of models had fewer than 200 lifetime downloads and 1.5% of repositories captured 99.2% of downloads. Chinese labs reached monthly ceilings of 754B to 2.78T parameters, while broader portfolios such as Qwen’s generated far more downloads than frontier-only strategies; likes and downloads measured attention and infrastructure dependence differently. Large Chinese releases were often Apache 2.0 or MIT licensed, though newer trillion-parameter models introduced restrictions, while agents became the Hub’s leading user and autonomous-agent activity exposed the practical value of running quantized open models locally.
HeyGen x Google Cloud: Bringing Avatar IV to TPUs
HeyGen and Google Cloud ported Avatar IV, an 18B-plus-parameter diffusion stack for streamed talking-head video, to an eight-chip Trillium (v6e) host. The pipeline uses two diffusion transformers and a VAE decoder, with FSDP weight sharding and Ulysses sequence parallelism across the mesh, and runs through torchax, JAX, XLA, and shape-specific Pallas kernels. Optimization targeted exposed all-to-all collectives, frame-misaligned sparse-attention blocks, serial softmax work, layout copies, scheduler choices, and custom-kernel cost estimates; changes were measured per video chunk against output-quality gates. The resulting pipeline was 1.86× faster than the first working TPU version, comparable in performance to the 8×H100 production setup and up to 25% more cost efficient per generated-video minute, while shipped changes preserved byte-identical output or passed defined similarity review.
What We Learned by Reproducing 2,200 papers from ICML
An ICML 2026 reproduction hackathon examined whether coding agents could audit research at conference scale, amid 23,918 submissions and 6,352 accepted papers. From July 15 to August 2, 1,221 participants published 6,816 Trackio logbooks covering 2,226 papers and 35,908 claims; an open-weights GLM-5.2 judge assigned per-claim verdicts, while participants used cloud jobs and toy reproductions when full experiments were impossible. At least one claim was independently verified in 51% of examined papers, including 266 fully reproduced papers, while 23% had a falsified or contested claim and 242 papers produced opposing verdicts across teams. Adversarial re-checks confirmed issues including a paging-theorem error, a reverse-versus-forward-KL mismatch, and padded evaluation tokens, but also caught false falsifications, leading to the conclusion that agents expand auditing capacity while humans remain necessary to steer experiments and assess cases such as image quality.
Grab Bench: Evaluating AI on Grab-shaped production work
Grab Bench is a configurable evaluation harness for assessing AI systems on production-shaped work where plausible outputs can violate metric definitions, tool contracts, evidence boundaries, or hidden code invariants. It uses YAML configuration, task plugins, provider and modality checks, row-level records, model summaries, and deterministic scorers or LLM judges according to each task. Synthetic or redacted cases preserve production-relevant difficulty while protecting live data; profile reasoning requires ontology-valid JSON claims and faithful evidence IDs, while coding tasks exercise hidden tests and anti-gaming checks. The harness also records tokens, latency, skip reasons, configurations, and failure tags so teams can diagnose whether errors reflect unsupported claims, parameter drift, hidden invariants, cost, or capability gaps. It concludes that synthetic evaluations test contract adherence under controlled pressure but do not establish production uplift.
Thinking of ACE? We Can Do It with Fewer Tokens
ALTK-Evolve and ACE are agentic-memory systems that turn an LLM agent’s past trajectories into reusable lessons without weight updates or human labels. Both preserve individual lessons and attach evidence counts rather than compressing experience into a short summary, but ACE injects its full evolving playbook at every step while ALTK-Evolve consolidates typed, provenance-linked guidelines and retrieves a model-appropriate subset or the full set. On AppWorld’s 168-task test_normal split with the same ReAct setup, ALTK-Evolve reached 89.3 TGC and 80.4 SGC on DeepSeek-V3.2 at 263K tokens per task, versus ACE’s 80.4 and 73.2 at 634K. With gpt-oss-120b, it reached 56.0 TGC and 37.5 SGC at 116K tokens, narrowly exceeding ACE’s 54.8 and 35.7 at 777K. The comparison uses single pass@1 runs and in-house ACE results on matched models, while the authors attribute the cost difference to delivery rather than lesson quality.
Which brand won the football tournament footwear battle
An analysis of footwear conversations around the 2026 World Cup compares how sponsorship, search demand, social attention, and sentiment shaped the competition between Adidas, Nike, Puma, New Balance, and other brands. Consumer Research tracked tournament-related mentions from June 11 to July 19 and paired them with search queries across the US, UK, and Canada. Adidas led tournament-specific footwear conversation with 63% of mentions, while Nike led broader brand discussion with 995k mentions, brand search momentum, and 78% positive sentiment. Nike’s cultural activations, including LISA’s “Rip The Script” campaign and Ronaldo’s Mercurial moments, generated shareable attention, whereas Adidas benefited from official sponsorship, the Trionda match ball, and player storytelling. Pink cleats became a cross-brand liability, attracting more than 5.5k mentions with nearly 70% negative sentiment, showing that visibility did not automatically produce favorable attention.
Introducing OfficeQA Pro V2: A New Benchmark for Enterprise Grounded-Reasoning
OfficeQA Pro V2 benchmarks whether AI agents generalize grounded reasoning to unfamiliar enterprise document collections rather than specialize in one corpus or task distribution. It contains 90 questions grounded in roughly 120,000 pages of U.S. Treasury accounting records spanning 1793–2024, including dense tables, charts, revised values, and changing reporting conventions. Questions were generated with asynth from seeded inputs, corpus evidence searches, traceable reasoning, and quality gates, then evaluated with deterministic exact match at 0.0% tolerance. Out-of-the-box frontier agents averaged 37.5% in one matched comparison, while competition systems averaged 41.1% and the winning team reached 63.3%; Databricks Genie raised matched-model accuracy by 24.0 percentage points on average, although failures remained in parsing, temporal reconciliation, and entity or category interpretation.
DeepSeek-V4 Flash 0731 vs GPT-5.6 Luna on DeepSWE: Cost and Coding
On the DeepSWE coding benchmark, GPT-5.6 Luna achieves 67.2% pass@1 versus DeepSeek-V4 Flash’s 53.3%, a 14-point gap. DeepSeek costs $0.10 per task, roughly six times less than Luna’s $0.61, delivering 532 solves per $100 versus Luna’s 110. A cascade that runs DeepSeek first and escalates only on failure solves 78.9% of tasks at $0.385 each, beating Luna’s solo accuracy and cutting cost by 37%. DeepSeek fails more gracefully (9% broken tests vs 15%) but is slower and dramatically weaker on JavaScript (35% vs 60%). Luna dominates seven of eight task domains, while DeepSeek leads only on query and config languages, illustrating how inexpensive models can combine with flagships for superior price-performance.
NVIDIA Vera Storage Benchmarks: Faster Encryption, Compression, Integrity Checking, and Recovery for AI-Native Storage
Agentic AI architectures require storage systems to secure, compress, and validate massive streams of persistent data and key-value cache entries across concurrent reasoning loops. These tasks run primarily on CPUs and frequently create processing bottlenecks that prevent modern networks and flash storage from operating at full capacity. NVIDIA evaluated the Vera CPU within the BlueField-4 STX Storage Processor using a purpose-built in-memory microbenchmark suite running standard libraries like OpenSSL, Zstandard, and LZ4. The results demonstrated that Vera outperformed x86 processors across core storage primitives including encryption, decompression, Reed-Solomon recovery, and CRC32C integrity checking. For a sequential multi-stage pipeline executing compression followed by encryption, Vera achieved up to 3.21x higher throughput than the comparison x86 CPU.
amazon.science
A new benchmark for evaluating patient-facing health AI agents
Healthcare AI benchmarks often rely on static medical knowledge questions or clinician-facing tool evaluations, failing to test multiturn patient interactions and clinical workflow execution. To address this gap, PatientAgentBench introduces an automated evaluation pipeline using synthetic patient charts, dual-agent conversations, and an LLM-as-a-jury panel. The evaluation framework applies over one hundred clinician-vetted criteria across six dimensions, including clinical safety, triage quality, and workflow accuracy. Testing frontier models revealed a severity paradox where agents scored higher on obvious emergencies than on routine administrative requests harboring hidden risks. Common safety failures included omitting crisis hotline resources and fabricating clinical information, showing that raw foundation model capability alone cannot guarantee safe patient-facing execution.
NVIDIA Nemotron 3 Ultra Leads Open Models on Accuracy and Efficiency in Agentic RTL Coding
Register transfer level (RTL) development relies heavily on iterative verification and tool feedback, which poses challenges for standard code generation models. To address realistic hardware design tasks, the ACE-RTL agent pairs an iterative generate-test-reflect workflow with NVIDIA Nemotron 3 Ultra. The model utilizes a hybrid Mamba-Attention Mixture-of-Experts architecture trained on a synthetic dataset encompassing code generation, editing, and error debugging. Across the comprehensive Verilog design problems benchmark, ACE-RTL with Nemotron 3 Ultra attained a 97.1% average pass rate, outperforming GLM 5.2 and Kimi K2.6. Additionally, Nemotron 3 Ultra achieved its accuracy while consuming an average of 6,629 tokens per iteration, which is substantially lower than competing models.
Kimi K3 vs GPT-5.6 Sol on DeepSWE: Cost, Coding, and Routing
GPT-5.6 Sol and Kimi K3 were compared on the DeepSWE software engineering benchmark using 904 graded rollouts across 113 tasks. Sol wins single-shot pass@1 (72.7% vs 68.5%) and reliability (84.5% tasks solved four-for-four versus 76.6%), while Kimi K3 leads pass@4 (89.4% vs 85.8%) with wider coverage at a substantially lower cost—$4.65 per rollout versus Sol’s $8.37, delivering 14.7 solves per $100 against Sol’s 5.3. The two models have a low task correlation of 0.46 and disagree on 18 tasks, enabling a cascade routing strategy that runs Kimi first and escalates to Sol when the test suite rejects the result, achieving 85.6% accuracy, beating both models individually and a perfect one-shot oracle router. Kimi K3 is open-weight and available on Together AI.
Kimi K3 vs Claude Fable 5 on DeepSWE: Cost and Coding
On DeepSWE, Kimi K3 trails Claude Fable 5 by 1.4 points at pass@1 (68.5% vs 69.9%) but leads at pass@2 (82.0 vs 80.2) and pass@4 (89.4% vs 88.5%) across 452 rollouts. Kimi K3 costs $4.65 per rollout versus $13.41 for Fable, delivering 2.8x more solved tasks per dollar. It covers 89.4% of the benchmark, solving 101 of 113 tasks, while Fable solves 100, though Fable is more reliable with 58 tasks solved four-for-four versus Kimi’s 45. The models share the highest cross-vendor similarity (0.72) and fail on the same tasks, with Kimi leading in Go and Fable leading in Python, JavaScript, TypeScript, and Rust. As an open-weight model, Kimi K3 offers a cost-effective alternative for retry-tolerant workloads.
Why A Frontier Data Agent Outperforms General Coding Agents in Quality and Cost
An evaluation compared Genie Code with three widely used coding agents from major AI labs on 401 real internal data tasks, using a shared 20-minute wall-clock budget and independent grading. Genie Code achieved 76.6% accuracy at an estimated $0.55 per task, outperforming rivals at 55.9–72.1% accuracy and $0.91–$1.16 per task. Its advantage is attributed to semantic search across catalog and workspace assets, persistent memory of tables and business logic, and enterprise-context understanding, which reduce exploratory tool use. On discovery-heavy tasks, general-purpose agents often wandered through large workspaces or timed out on inefficient scans, while Genie Code averaged 8.3 tool calls per task. The benchmark found no accuracy-cost trade-off, although the authors note that Genie Ontology was disabled and expect it to strengthen the advantage.
Top topics
- Architecture834
- Performance816
- LLMs665
- Machine Learning622
- Open Source566
- AI Agents557
- AI475
- Developer Experience448
Weekly digest
The best engineering reads, once a week.