Loading…
Browse by type
Benchmark reads
Benchmark posts on Yomu, summarised newest first.
- 170 posts
- 20 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Benchmark posts
REVERSAL-BENCH: A Reversibility Axis and Reset Oracle for Measuring the Reset-Free RL Cliff
REVERSAL-BENCH addresses the challenge of evaluating reset-free reinforcement learning when environments contain irreversible transitions. It controls reversibility with continuous parameter ρ ∈ [0, 1] and supplies a reset oracle, a ground-truth mechanism for verifying state recoverability across eight manipulation settings and five physics engines. Evaluations spanning standard actor-critic, safe reinforcement learning, and specialized reset-free methods find a sharp reversibility cliff: reset-free agents become absorbed into irrecoverable states as ρ increases, while episodic agents maintain steady learning. Geometrically identical reversible counterparts indicate the breakdown is caused by irreversibility rather than obstacle complexity, and the effect persists under learned policies in full physics simulations. The released suite includes a multi-simulator dataset with recoverability labels and a reset oracle; a safety shield predicts recoverability accurately, but active recovery mainly works when agents can steer away from traps.
TensorRT Edge-LLM Completes the MLPerf Edge Agentic Benchmark 6.4x Faster on Jetson AGX Thor
Edge agentic workloads require repeated tool-calling turns, growing conversation histories, and valid function calls on constrained devices rather than single-prompt responses. On one NVIDIA Jetson AGX Thor Developer Kit, TensorRT Edge-LLM ran Qwen3.6-27B in SingleStream mode and completed the benchmark’s 1,007-turn performance workload in 24 minutes and 36 seconds, versus 2 hours and 37 minutes for the llama.cpp reference run. It reported 52.33 tokens per second, 247.12 ms median time to first token, and 87.94% BFCL overall accuracy. The implementation combines NVFP4 weights and activations, FP8 KV cache, reusable prompt prefixes with recurrent-state restoration, and tree-based multi-token prediction; roughly 96% of prompt tokens used hot cache, while tree-based MTP could achieve about 40% more decoding performance than linear MTP with three draft steps.
Shared Selective Persistent Memory for Agentic LLM Systems
Agentic LLM systems that generate code through multi-turn tool use lose configuration choices, domain constraints, schemas, and tool-use patterns when each session starts from zero. The proposed shared selective persistent memory retains task specifications, data schemas, tool configurations, and output constraints while discarding session-specific reasoning traces, and allows workspaces to be transferred across users with role-based access control. In a deployed collaborative workspace platform, agents create and maintain git-versioned dashboards, reports, and data-driven documents from CSV, SQL, REST API, and MCP sources; draft isolation supports risk-free changes and restoration without re-invoking the model. Results across three enterprise scenarios show 96% task completion, compared with 79% without memory and 71% with full history, while zero-token refresh cut recurring-update task time 14× and summary-driven generation reduced token cost 97× versus raw data injection.
Understanding W8A8 INT8 LLM quantization: Accuracy and performance results
The post evaluates whether W8A8 INT8 quantization can reduce the size of a Llama 3.1 8B Instruct model without sacrificing accuracy or inference performance. Using SmoothQuant to address activation outliers and GPTQ with error compensation for BF16 weights, the model was compressed from 14.9 GB to 8.0 GB after calibration on WikiText-2. Identical zero-shot lm-eval-harness tests across MMLU, ARC Easy, HellaSwag, and IFEval found MMLU, ARC Easy, and HellaSwag deltas smaller than their standard errors, while the IFEval change was described as likely within a similar noise range; the post concludes no measurable degradation. vLLM and GuideLLM tests on one NVIDIA L40S reported 29% more concurrent requests, 24% lower synchronous TTFT, and 44% faster token generation at maximum load, while ITL degraded more quickly under high concurrency because of dynamic activation quantization overhead.
Putting Captions to the Test: Evaluating Video Caption Quality through Multiple-Choice Question Answering
Evaluating video captions is difficult because reference-matching metrics can penalize valid descriptions that use different wording or focus on different visual content, while offering little fine-grained diagnosis. The paper reframes quality around information fidelity: captions should cover salient visual information and remain factually accurate. It introduces CapQuiz, a reference-free benchmark that tests caption utility through human-verified, fine-grained multiple-choice questions derived from videos, organized into 10 Descriptive and Inferential question types across 24 domains. It also proposes CapF1, which combines CapP for factuality with CapR for coverage; extensive experiments report significantly stronger correlation with human judgments than existing metrics and interpretable insight into model performance.
Autonomous LLM post-training with Tunix on TPUs
autofinetune applies autonomous research loops to LLM post-training, covering supervised fine-tuning (SFT) and reinforcement learning with GRPO through Tunix, Gemma, Cloud TPUs, Antigravity CLI, and Gemini Flash 3.7. A human defines the arena in program.md and supplies a self-contained run.py, while an agent edits hyperparameters, launches jobs, evaluates a target metric, keeps winning Git commits, reverts regressions, and logs results in results.tsv. On FunctionGemma 270M with google/mobile-actions, 20 Cloud TPU v5e-1 experiments tested LoRA settings, optimization choices, schedules, batch size, and seeds without changing the dataset, epochs, or architecture. A second study used Gemma 3 1B, GSM8K, and Cloud TPU v6e-1 for 40 GRPO experiments over two to three days, improving the combined numerical-and-format reward by about 10%.
Measuring political bias in Claude
The post sets out an effort to make Claude politically even-handed: balanced, accurate, comprehensive, respectful, and reluctant to offer unsolicited political opinions or persuasion. It describes system-prompt guidance and reinforcement-learning-based character training, including traits that encourage neutral terminology, representation of competing perspectives, and restraint on sensitive political topics. For measurement, the Paired Prompts method compares responses to the same contentious issue framed from opposing ideological positions, scoring dimensions such as equal engagement and acknowledgment of opposing views; an automated version applies it to thousands of prompts across hundreds of stances. In the reported comparison, Claude Sonnet 4.5 was more even-handed than GPT-5 and Llama 4, and similar to Grok 4 and Gemini 2.5 Pro. The evaluation is open-sourced, with limitations including a focus on current US discourse and single-turn interactions.
How Full-Stack NIM Optimizations Deliver 2.5x More Users on Nemotron 3 Ultra
Production serving for agentic AI must maximize concurrent users on fixed GPU infrastructure while preserving latency, especially with long prompts, reused context, and streamed responses. The article presents NVIDIA NIM as a deployable microservice that packages validated model- and GPU-aware choices, then measures Nemotron 3 Ultra NIM 2.0.12 against an open-source baseline on four B200 GPUs using a 64K/400 workload, 76% KV reuse, and 50 TPS per user (20 ms ITL). The optimized stack combines autotuned MoE and Mamba kernels, tensor and expert-aware parallelism, prefix and Mamba state-cache reuse, scheduling, batching, memory tuning, and MTP speculative decoding. At the stated target, throughput rises from 718 to 1,997 output tokens per second—more than 2.5x—while the article recommends replaying representative traffic with NVIDIA AIPerf to select a Pareto point meeting the application’s latency constraint.
ClickHouse Cloud vs. Snowflake: What drives the real-time performance-per-dollar gap
The benchmark compares ClickHouse Cloud with Snowflake under a continuous real-time workload, asking whether ingested data becomes query-ready quickly enough for interactive analytics. Both systems received the same 113.2-billion-row NBBO dataset at a target of 1 million rows per second, with matched sorting or clustering keys, pre-aggregations, query schedules, and approximately 16 CPUs of query-serving compute. ClickHouse used native asynchronous inserts and maintained ordering and pre-aggregation during ingestion, while Snowflake used Snowpipe Streaming and Interactive MV refresh as separately metered services. In the end-to-end CostBench comparison, ClickHouse Cloud delivered 412× better performance per dollar: Snowflake’s path cost 3.32× more and its matched queries took 124× longer in total. The comparison attributes the gap to query-readiness work, refresh lag, reconciliation, and growing drill-down latency.
Prompting Claude's long context window
An evaluation of Claude’s 100,000-token context window tests prompting techniques for recalling specific information from long documents. Researchers built randomized collages from sections of a July 13 government document, using Claude-generated multiple-choice questions and filtering out questions it could not answer from the source passage alone. They compared a baseline with unrelated examples, dynamically selected contextual examples, and two or five examples, each with or without a scratchpad that instructed Claude Instant 1.2 to extract relevant quotes across roughly 70K and 95K token contexts. Scratchpads and contextual examples generally improved performance, with more examples performing better, although recall could decline when the answer passage was at the end. For Claude 2, the best prompting strategy raised performance from 0.939 to 0.961, a reported 36% reduction in errors; reproducible code was released in the Anthropic Cookbook.
Adaptive Instructed-Retriever: Frontier-Quality Search at 2x Lower Latency
Adaptive Instructed-Retriever addresses the tension between retrieval quality and latency for enterprise data agents, especially on multi-hop questions that benefit from sequential search. It combines parallel single-step retrieval with bounded multi-step search, using a fixed upper limit and an adaptive stopping policy that spends extra steps only when they are likely to improve results. The model is trained with online reinforcement learning using CISPO, a reward that balances trajectory quality against search cost, and synthetic multi-hop data alongside Instructed-Retriever-1 data. Across seven held-out internal and external benchmarks, it matches leading third-party and open-source models while answering in 5.8 seconds, reported as more than twice as fast as the compared alternatives, and substantially improves over single-step search. Varying the step penalty produces checkpoints spanning different quality-latency tradeoffs for production workloads.
Benchmark Shows MongoDB Atlas Manages Thousands of Payments Per Second with Zero Data Loss
A benchmark of the Icon Payments Framework (IPF), trusted by Citi, NatWest, and BNP Paribas, on MongoDB Atlas demonstrated extreme performance and resilience for instant payment processing. The system sustained 6,000 end-to-end payments per second, requiring approximately 430,000 simultaneous database operations, with a mean end-to-end latency of 0.51 seconds and persistence latency below 30 milliseconds. Upgrading from MongoDB Atlas 8.0 to 8.3 reduced persistence latency by roughly 50%, from 40 to 20 milliseconds at 4,000 payments per second, and throughput scaled near-linearly across the 500 to 6,000 range while latency remained flat until peak and hot-shard CPU utilization sat at about half. During a primary failover, a new node was elected in five seconds with zero data loss and no application restart. An application node failure recovered to 3,500 transactions per second in 90 seconds with zero data loss.
What we learned mapping a year’s worth of AI-enabled cyber threats
An analysis of 832 accounts banned for malicious cyber activity between March 2025 and March 2026 maps their behavior to MITRE ATT&CK to assess how well security frameworks capture AI-enabled threats. It finds AI moving beyond preparation, including malware writing by 560 accounts (67.3%), toward post-compromise operations such as account discovery and lateral movement, while medium-or-higher-risk actors rose from 33% to 56% across the two six-month periods. Traditional signals, including technique counts and platforms, correlate poorly with risk; higher-risk actors instead build scaffolding that chains attack stages with minimal human input. The report argues ATT&CK lacks coverage for agentic orchestration, illustrated by a disrupted operation in which Claude Code executed commands, exploited vulnerabilities, stole credentials, and made tactical decisions, earning the maximum risk score of 100 despite mapping to 30 techniques across 13 tactics. The findings informed cyber safeguards for capable models and discussions with MITRE about extending the framework.
Measuring real-time performance per dollar under continuous load: CostBench’s first end-to-end results
CostBench measures real-time performance per dollar by evaluating the full path from continuously arriving rows to query-ready data and query results. Its end-to-end workload streams more than 100 billion stock-market quotes at a target rate of 1 million rows per second while aggregate and drill-down queries run, comparing ClickHouse Cloud, Snowflake, BigQuery, and Redshift Serverless through their recommended push-based ingestion paths. The benchmark accounts for fresh-data-path work—columnar ingestion, pruning-friendly layouts, and current pre-aggregations—alongside query-serving cost and accumulated runtime. ClickHouse Cloud recorded the lowest fresh-data-path cost, query-serving cost, and accumulated runtime, while the alternatives showed 412–1,996× worse end-to-end performance per dollar; the source says it was the only tested system keeping both data representations current during ingestion. The post introduces follow-up architecture and billing analyses.
Evaluating LLM guardrail configs locally with EvalHub
This installment evaluates local NeMo Guardrails configurations against large-scale risk datasets using EvalHub, following earlier manual testing. EvalHub provides a unified interface for AI evaluation, with provider integrations, reusable benchmarks, Hugging Face dataset inputs, and both local and Kubernetes deployment options. The benchmark configuration maps dataset prompt and label columns to block and pass labels, while balanced safe and risky examples prevent an always-block guardrail from appearing perfect. On the prompt-injection benchmark, the nuanced regex configuration achieved 51.1% accuracy with 13.7 ms median latency; DeBERTa reached 88.93% accuracy with 45.2 ms median latency. The conclusion favors layered guardrails: targeted regex checks can provide speed, with DeBERTa handling other cases, and EvalHub supplies reproducible measurements of the latency–efficacy trade-off.
REFACTOR-VLA: Unsupervised Library Learning of Typed Motor Programs
REFACTOR-VLA addresses the weakness of monolithic vision-language-action (VLA) models, which generate raw commands or short action sequences and struggle with long-horizon tasks and interpretability. Its wake/sleep architecture uses a Behavioral-Equivalence Kernel (BEK) to cluster motor-program segments by outcomes from a learned latent world model, then produces typed lambda terms for a library-conditioned rectified-flow action decoder. Skills are admitted only when they satisfy both Minimum Description Length (MDL) and return-preservation gates, while training proceeds through world-model warmup, wake-phase policy optimization, and sleep-phase skill discovery. On the full LIBERO benchmark, expanding the world model from 188 million to 430 million parameters worsened results on all four suites, showing that scale alone was not beneficial. Adding supervised contrastive InfoNCE loss during warmup substantially improved sleep-phase clustering, with reported NMI scores of 0.462, 0.867, 0.915, and 0.754 for the Object, Spatial, Goal, and LIBERO-10 suites, respectively, under n = 3 multi-seeding.
BenchMIRT: What are LLM benchmarks actually measuring?
BenchMIRT is a method for auditing LLM benchmarks at the individual-prompt level, where questions may measure capabilities beyond a benchmark’s stated goal. It applies multidimensional Item Response Theory (MIRT) to model and question results, estimating model abilities plus each item’s difficulty and discrimination across dimensions. Trained on 100 LLMs, 16 benchmarks, and more than 34K questions, it independently recovered safety and general reasoning as dominant dimensions, repeatedly producing the same result. The analysis found, for example, that BBQ and WMDP aligned more strongly with reasoning than safety, while HarmBench combined safety- and reasoning-associated question groups; retaining 10% of items generally preserved capability rankings, and held-out correctness prediction reached 79% versus 70% for a baseline. BenchMIRT’s dimensions depend on the benchmark set, and its item-level transparency can also enable evaluation trimming.
The Agentic Analytics Benchmark: Measuring model accuracy and efficiency in analytical agents
The post presents data-agent-mnist, an open harness for measuring analytical agents on correctness, cost, and speed against a warehouse-shaped workload rather than a single text-to-SQL query. The authors extracted 201 production analytics questions from DWAINE, curated and anonymized them, reconstructed a deterministic synthetic warehouse with 18 tables and 865 columns, and evaluated 28 proprietary and open-weight models. The benchmark preserves schema discovery, multi-turn querying, planning, and multi-hop joins across curated marts and a dimensional CRM layer, with provider-diverse ground truth and judging described as part of the workflow. On the authors’ warehouse, Claude Fable 5.1 ranked first for correctness at 76.6%, while the full run cost $1 with DeepSeek V4 Flash versus $52 with Fable 5.1, an 11-percentage-point correctness difference. The harness is intended to identify the best production fit for a specific warehouse, not a universal winner.
From Leaderboards to Model Profiles: A Deep Dive Evaluation of LLMs for Agentic Coding
Resolve rate alone can hide how coding agents reach the same outcome. This evaluation compares Claude Opus 4.7, Gemini 3.5 Flash, GPT-5.5, and Qwen 3.6 27B FP8 through a Junie pipeline across four benchmark datasets, combining test results with trajectory, cost, efficiency, patch quality, and process measures. In one private benchmark, Opus and Gemini solved the same number of tasks, but Opus averaged 184 steps at USD 2.79 per run versus Gemini's 271 steps at USD 1.24. Across 523 tasks, Opus resolved 267 (51.1%) and Gemini 254 (48.6%); shared outcomes masked differences in navigation, validation, redundancy, hallucination, and patch scope. The wider comparison found GPT-5.5 had the highest resolve rate and always ran an executable check, Opus led patch-quality metrics, and Qwen delivered lower cost with lower resolution, while the authors caution that profiles depend on the Junie scaffold and LLM judges are diagnostic rather than ground truth.
GLM-5.3 vs. GLM-5.3 Flash on DeepSWE: Cost, Coding, and Routing
The GLM-5.3 Flash, a distilled variant of the full GLM-5.3, achieves a pass@1 of 63.4% on the DeepSWE benchmark at $0.24 per rollout, 17 times cheaper than the full model’s 69.0% at $3.99. The pass@4 gap narrows to 2.6 points (87.6% versus 85.0%). Distillation removed consistency rather than capability: the Flash solves 93 of the 99 tasks the full model solves, but its ability to convert extra effort into a win is lost (46% win rate on flaky tasks versus 61%). A cascade that runs Flash first and escalates to the full model only when the answer is rejected solves 80.9% of tasks at $1.70 each, 57% lower cost than using the full model alone.
Top topics
- Architecture830
- Performance813
- LLMs656
- Machine Learning619
- Open Source563
- AI Agents546
- AI461
- Developer Experience447
Weekly digest
The best engineering reads, once a week.