Loading…
Browse by type
Benchmark reads
Benchmark posts on Yomu, summarised newest first.
- 170 posts
- 20 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Benchmark posts
Agent Seer: Synthesizing Scenarios from Specification Understanding
Agent Seer addresses the problem of creating realistic, scalable test scenarios for AI agents that use external tools, where hand-built cases require domain expertise and static benchmarks cannot track changing APIs. It synthesizes scenarios from a single Model Context Protocol (MCP) specification containing function names, natural-language descriptions, and typed parameter schemas, without examples, live tool access, or domain-specific tuning. The pipeline enriches raw schemas, generates graded scenarios with synthetic tool outputs, and expands them into mock-data-grounded, multi-turn dialogues, then evaluates tool-calling correctness and conversational coherence across seven MCP specifications spanning domains and tool-suite sizes. It achieves strong quality across all domains and complete tool coverage on small and medium specifications; parameter schema complexity correlates most strongly with quality variation, while tool-suite size has a smaller orthogonal role. Among imperfect scenarios, argument value accuracy is the dominant failure mode, which coarse-grained name-match metrics do not reveal.
Enhancing Agent Retrieval with Structured Chart Extraction
Enterprise retrieval agents often struggle with questions that require reading values, labels, or patterns in charts, because text-only indexes and captions omit fine-grained visual data. The post evaluates structured chart extraction as a way to improve retrieval and answering, comparing caption-only figures with figures enriched by chart JSON from ai_parse_document. Its pipeline uses ai_prep_search to create retrieval chunks, a 300-million-parameter text embedding model for indexing, and Genie for retrieval and answer generation. Evaluation covers 310 chart-heavy questions from ViDoRe V3 and 114 visually grounded questions in the synthetic Chart-RAG dataset, with answer correctness, Hit Rate@10, and nDCG@10 measured across three runs. Chart JSON improves retrieval and answer quality; adding the top three retrieved images raises correctness further, reaching 75.9% on ViDoRe V3 and 75.1% on Chart-RAG, exceeding four multimodal embedding baselines.
How Much Code Do Developers Really Let Agents Write?
JetBrains surveyed over 15,000 professional developers globally in 2026 to measure how much production code is generated by AI agents versus manual authoring. On average, respondents report that agents fully write approximately 47% of their code, while 38% involves AI assistance and 27% is written manually. Adoption varies across ecosystems, with Go, JavaScript, and TypeScript developers reporting the highest agentic code generation at 54% to 55%, while C and C++ developers retain the highest manual share. Senior developers lead adoption over juniors, with about a quarter delegating more than 80% of code generation to agents. Cluster analysis categorizes the developer population into agentic coders (31%), AI-assisted coders (47%), and manual coders (23%).
Enterprise-Grade Precision for Long-Context Multimodal Embedding Inference on Cloud TPU
Embedding models convert text, images, and audio into dense vectors used for semantic search, recommendation, intent classification, content discovery, and clustering. The post describes production support for long-context Qwen3 Embedding models on Cloud TPUs through native vLLM TPU integration and GKE Custom Compute Classes, allowing prioritized autoscaling across TPU and GPU capacity pools. Its implementation pads vocabulary tensors for hardware-safe sharding, hardens lazy loading, pre-warms sharding-aware JAX/XLA compilation caches, and uses a hybrid StepPool with CachedRequestState to preserve pooling state during chunked prefill and preemption. Parity evaluations target cosine similarity of at least 0.999 for text and 0.995 for multimodal inputs; on TPU Ironwood, Qwen3-Embedding-8B reached 83,996 total tokens per second and 5.13 requests per second at 16K+ sequence length with bf16 and TP=4.
Quantization-Aware Healing: a compressed, 4-bit model that outperforms its full-precision original
Structural compression and 4-bit quantization can reduce an LLM’s reasoning, mathematical, and coding capabilities, while conventional healing methods either require costly quantization-aware training or distill from an already degraded checkpoint. Quantization-Aware Healing (QAH) instead distills the original, full-size, full-precision model into a structurally compressed student running in MXFP4, using KL divergence over teacher logits. A chunked KL loss allows healing on documents up to 32k tokens within a fixed GPU memory budget. Applied to GPT-OSS 120B compressed to 60B parameters, QAH’s 4-bit model outperformed its 60B bfloat16 counterpart on 7 of 9 benchmarks, including gains of 7.4 points on AA-LCR and 5.6 on AIME 2025. In a separate comparison, QAH reached its peak about seven times faster than QAT and avoided QAT’s later 19-point collapse.
NVIDIA Vera Rubin and Blackwell Set a New Standard for Agentic AI Performance per Watt
Agentic AI inference has shifted from single-turn requests to long, stateful workflows that reason, invoke tools, coordinate subagents, and reuse growing context, increasing token and power demands. The post presents SemiAnalysis AgentX, an InferenceX benchmark that replays prerecorded Claude Code sessions turn by turn, preserving context, input/output lengths, reasoning time, tool-call latency, and KV-cache pressure while varying concurrency. It evaluates tokens per megawatt alongside E2E normalized interactivity, standard interactivity, E2E latency, and TTFT; NVIDIA-measured Vera Rubin NVL72 results, pending SemiAnalysis review, show up to 30x higher throughput per megawatt than GB300 NVL72 at 160 tokens per second per user. GB300 also delivers up to 15x H200 throughput per megawatt, up to 10x lower cost per million tokens, and roughly 80x higher throughput on Kimi K3 2.8T, supported by MoE runtimes, communication overlap, Dynamo, and NVLink.
Solving Agentic AI Fleet Challenges with NVIDIA Vera CPU
Agentic AI fleets must handle unpredictable trajectories in which long sequential reasoning chains are interrupted by short bursts of parallel tool calls, retrieval, sandbox, or sub-agent work. Telemetry from 163,594 sessions found over 97% had unique trajectory profiles, making fleet sizing around several specialized CPU designs impractical; the sequential path is latency-bound while fan-out requires concurrency. It presents NVIDIA Vera CPU as a single balanced design point, combining strong loaded per-core performance, concurrency, memory bandwidth, and a low-latency monolithic architecture, with Olympus cores using a wide front end, branch prediction, and deep out-of-order execution. Using estimated SPEC CPU® 2026 results across four agentic workloads, it reports up to 1.5x the agentic performance of AMD Venice, while noting that Venice figures were estimated from SPECrate® 2026 data and that results may vary.
GPU-Accelerated Clustering for Financial Instruments at Scale
Quantitative strategies need instrument groupings that capture both ordinary co-movement and dependence during market stress, while remaining stable and inexpensive to recompute. The workflow constructs rolling absolute Pearson-correlation and tail pairwise dependence matrices (TPDM), factorizes them with nonnegative SymNMF using AdaptGrow, and derives soft loadings, hard argmax labels, and structural-break diagnostics. A trace-based formulation reduces estimated peak storage from approximately 20n2 to 4n2 bytes, while row-sharded distributed execution communicates O(nk) data; the system uses eigenspectrum-based selection between full-batch and block-stochastic AdaGrad. Across 250 synthetic windows, the setup tests planted membership changes and a joint tail-stress episode: ARI is intended to identify reclassification, whereas TPDM reveals co-crash dependence that ordinary correlation can miss. Reported factorization timings include 13.0 and 12.4 seconds for 100,000 instruments across four GB200 GPUs, and approximately two and four minutes for one million instruments across 64 GPUs on 16 nodes, not end-to-end timings.
amazon.science
SOP-Bench: A new benchmark for evaluating AI agents on real business procedures
AI agents often struggle to execute standard operating procedures because real-world business workflows contain implicit domain knowledge and procedural ambiguity that standard benchmarks overlook. To evaluate agent reliability, researchers introduced SOP-Bench, an open framework featuring more than 2,000 runnable tasks across 12 industrial domains paired with functioning tool interfaces and ground-truth answers. Domain experts authored the workflows, while an Anthropic Claude 3.5 Sonnet v2 model generated mock APIs, tool code, and edge-case datasets before expert verification. Experiments evaluating function-calling and reasoning agents across 11 frontier models revealed that newer models did not consistently outperform older versions, and providing 20 extraneous tools nearly halved task success. The benchmark, baseline agents, and evaluation code are released on GitHub and HuggingFace for reproducible testing.
GLM-5.3 vs. GPT-5.6 Sol on DeepSWE: Cost, Coding, and Routing
GLM-5.3 and GPT-5.6 Sol were evaluated on 113 DeepSWE tasks with 904 rollouts, comparing accuracy, cost, and failure patterns. Sol edges ahead at pass@1 (72.7% vs 69.0%), but GLM-5.3 leads at pass@4 (87.6% vs 85.8%) at $3.99 per rollout, half Sol’s $8.37 cost and yielding 17 solves per $100 versus Sol’s 9. Their low per-task correlation of 0.43 makes a cascade effective: GLM-5.3 first, then Sol on test failure, achieving 85.9% solved at $6.61 per task. Sol runs faster (19 minutes, 61 steps vs 35 minutes, 124 steps) and achieves higher reliability (61 tasks solved all four tries vs 48), but it breaks passing tests in 20% of failures versus GLM-5.3’s 11%.
GLM-5.3 vs. Claude Fable 5 on DeepSWE: Cost, Coding, and Routing
GLM-5.3 and Claude Fable 5 were compared on the DeepSWE benchmark across 113 tasks with four trials each, totaling 904 rollouts. On first-attempt accuracy, Fable 5 scored 69.7% and GLM-5.3 scored 69.0%, a statistical tie inside error bars. GLM-5.3 led on pass@2 (81.1% vs. 77.1%) and pass@4 (87.6% vs. 84.1%), and cost $3.99 per rollout versus Fable’s $21.63, a 5.4x price gap. Per $100, GLM-5.3 solved 17 tasks while Fable solved 3. Both models regress test suites in only 11% of failures, and GLM-5.3 averages 35 minutes per rollout with 80k output tokens versus Fable’s 34 minutes and 114k tokens. The models are near-substitutes with 0.65 per-task correlation; GLM-5.3 won five of eight domains including JavaScript and concurrency, while Fable excelled only in Rust and serialization, making GLM-5.3 the practical default.
Measuring benchmark optimization in speech recognition
Public voice AI benchmark scores can overstate speech-recognition ability when models learn benchmark-specific patterns rather than improve underlying transcription. Traditional evaluations may miss reliability, naturalness, contextual appropriateness, and other real-world conditions, motivating held-out sets in Real World VoiceEQ, the Open-ASR Leaderboard, and the Far-field ASR Leaderboard. The study introduces three tests and evaluates 11 widely used open-source ASR models, finding that several reproduce VoxPopuli and LibriSpeech reference transcripts when audio contradicts them, words are silenced, or two written forms sound identical. Models also sometimes use acoustic cues to identify dataset membership and select its expected transcription policy, while fresh post-cutoff data and other interventions often restore more audio-faithful output; the findings support fully held-out evaluation and broader metrics than a single public WER score.
Up to 3.2x Faster Inference with LFM2.5-DSpark
Liquid AI released DSpark draft checkpoints for LFM2.5-1.2B-Instruct, LFM2.5-2.6B, and LFM2.5-8B-A1B, adding speculative decoding with a minimal memory increase. DSpark uses a DFlash-style parallel backbone conditioned on target context features, a lightweight sequential Markov-chain head, and a confidence-scheduled verifier that prunes low-confidence suffixes. The draft models are simplified attention-only networks with five layers and a block of nine, trained for 15 epochs on SFT, chat, code, and function-calling data; each contains roughly 300M parameters. Across five datasets, LFM2.5-2.6B averaged 2.67x speedup on an H100 and 2.27x on an M4 Max, while reducing multi-tool function-calling latency by 57% on average. Greedy decoding remains identical to baseline by construction, and integrations are available in llama.cpp and SGLang, with on-device gains for the 8B-A1B model limited by its current MoE Metal implementation.
POSETTE Talk Recap - Postgres Isn't Slow. Your Storage Is
At POSETTE 2026, Sai Srirampur’s talk examines how storage limits can produce slow ingestion, erratic P95 reads, lagging autovacuum, slower checkpoints, and logical replication lag in large PostgreSQL workloads. The benchmark ran the same 3.3-billion-row, random-UPDATE workload on four local-NVMe clusters and four baseline gp3 EBS clusters, using m6id.4xlarge instances, 64 clients, and 16 worker threads. NVMe reached median 16,030 TPS versus 1,734 TPS on EBS, reduced median latency from 36.9 ms to 4.0 ms, and completed the tested VACUUM and replication-backlog tasks faster. The analysis attributes most of the gap to storage waits rather than CPU work, while the proposed production model combines local NVMe with quorum-based replication, external base backups, and archived WAL for recovery.
Evaluating AI Agent Skill Performance with NVIDIA SkillEvaluator
NVIDIA SkillEvaluator measures whether packaged agent skills improve agent performance by comparing task runs with and without a skill installed. It combines three evaluation tiers: static safety and structure checks, embedding-based distinctiveness analysis, and live evaluations in isolated Harbor sandboxes. In the August 12, 2026 benchmark snapshot, more than 300 verified skills spanning over 30 NVIDIA products were tested across Claude Code and Codex, with macro-averaged Skill Lift of +31 points across all dimensions and +39 excluding Security. Correctness rose from 46 to 87 and Effectiveness from 39 to 78, while Security moved from 97 to 98; token and runtime effects varied by skill, with one reducing both and another increasing them.
Examining Human-Like Behaviors in LLMs: A Multi-Dimensional Analysis of Model Behaviors, User Factors, and System Prompts
Large language models frequently exhibit human-like behaviors such as expressing emotions, building relationships, and setting boundaries, yet empirical guidance on appropriate usage remains limited. To address this gap, researchers analyzed 21,000 multi-turn conversations across four models: gpt-4o, gpt-4.1-mini, claude-sonnet-4.6, and gemini-2.5-flash. The study combined LLM-as-a-judge techniques with human evaluation to measure behavior prevalence, perceived appropriateness, and prompt controllability across diverse user goals and profiles. Human evaluators judged LLM self-referential and relationship-building behaviors as less appropriate than identical human actions, whereas boundary-maintaining behaviors were rated as more appropriate for models. System prompting effectively steered these behaviors, though the authors emphasize that rigorous evaluation remains necessary to prevent unintended side effects.
How Much Memory Does Your Agent Actually Need?
The post examines how much agentic memory different models can use effectively, evaluating ALTK-Evolve's self-distilled guidelines across eight models on AppWorld's 585 multi-step tasks. Memory is produced by extracting and consolidating lessons from successful and unsuccessful trajectories, then delivering either the full guideline set on every ReAct step or a fixed core plus task-relevant retrieval, without updating model weights. Results identify three patterns: strong models with headroom benefited from the full set, gpt-oss-120b gained 16.1 percentage points in task completion from curated retrieval, and GLM-5 showed no measurable gain; stricter scenario completion sometimes improved more. Curated retrieval gave gpt-oss-120b that gain with only 5% more tokens versus 51% for its full set, while prompt caching is presented as a way to reduce recurring input cost, supporting calibration to model capability rather than simply accumulating memory.
How AI Coding Agents Can Unlock Materials Simulation with NVIDIA ALCHEMI Toolkit
Atomistic simulation combines scientific judgment, compute-efficient implementation, and an accessible interface to the simulation stack, but the MLIP ecosystem lacks the mature interfaces available for classical force fields. NVIDIA ALCHEMI Toolkit addresses implementation with composable, PyTorch-native, GPU-accelerated building blocks and in-flight batching, while its agent skills and reference files supply API patterns to coding agents. The post benchmarks 45 generated pipelines across silicon equation of state, copper adsorption, and lithium self-diffusion, checking code features and running representative workloads on NVIDIA H200 GPUs. All 45 scripts used batched GPU execution, 38 completed screening, and all 15 production representatives completed; however, agents did not assess physical validity, and thermostat choices could suppress measured lithium diffusion by 3–5×, so independent validation against DFT, experiment, or another reference remains necessary.
Evaluating AI Agents Live at the Grounded Reasoning Cup
The Grounded Reasoning Cup evaluated whether AI agents could generalize grounded reasoning improvements from OfficeQA to a newly released, enterprise-style document benchmark. Eleven academic teams had about two months to build agents with a partner lab’s model family, then applied them live to OfficeQA Pro V2, based on U.S. Treasury Accounts of Receipts and Expenditures, after a 36-hour release window. Stanford won with 63.3% accuracy, while the top systems combined document preprocessing, targeted or hybrid retrieval, specialized tools, parallelism, and answer verification; 18.8% of questions remained unsolved by every team. The results show that benchmark gains did not reliably transfer, and that parsing, retrieval, tool use, verification, and operational infrastructure can matter as much as model choice in end-to-end enterprise reasoning.
DeepSeek V4 Pro 0813 vs GPT-5.6 Sol on DeepSWE: Cost, Coding, and Routing
DeepSeek V4 Pro 0813 costs $0.24 per rollout and achieves 62.8% pass@1 on DeepSWE, while GPT-5.6 Sol reaches 72.7% pass@1 at $8.37 per rollout—a 35x price gap. At four attempts, DeepSeek V4 Pro 0813 surpasses Sol, with 88.5% pass@4 versus 85.8%. Sol is faster and more reliable, completing a rollout in 17 minutes and 53 steps, and breaking tests in only 11% of its failures. A cascade strategy—running DeepSeek V4 Pro 0813 first and escalating to Sol on test failure—solves 83.0% of tasks at $3.35 each, outperforming either model alone. Per $100, DeepSeek V4 Pro 0813 solves 260 tasks versus Sol’s 9, making it the value leader for budget-constrained workloads.
Top topics
- Architecture832
- Performance814
- LLMs658
- Machine Learning620
- Open Source563
- AI Agents549
- AI465
- Developer Experience448
Weekly digest
The best engineering reads, once a week.