Loading…
Browse by type
Benchmark reads
Benchmark posts on Yomu, summarised newest first.
- 172 posts
- 20 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Benchmark posts
Agent swarms and the new model economics
Agent swarms are evaluated as a way to scale software work beyond what a single long-running agent can reliably manage. The test revisits building SQLite in Rust from its documentation, comparing old and new swarms with identical models and time budgets against a held-out SQL suite; the new system performed better in every configuration, reaching 80% with Grok 4.5 in four hours. Planners decompose task trees while workers execute narrower pieces, supported by a purpose-built version control system and mechanisms for design conflicts, merge collisions, megafiles, intentional breakage, and review. Reported results include a full-suite pass using 9,908 engine-code lines versus 64,305 for the old harness, while costs ranged from $1,339 for an Opus 4.8 hybrid to $10,565 for GPT-5.5 alone, indicating that cheaper models can perform most execution after frontier planning resolves ambiguity.
Newer Models, Same Advantage
DharmaOCR, a Brazilian Portuguese-focused OCR model, is evaluated against newer Mistral OCR4 and Unlimited-OCR systems on a Portuguese-only benchmark. The article attributes its performance to a two-stage training pipeline: supervised fine-tuning on varied Portuguese documents, followed by Direct Preference Optimization (DPO) using comparative output preferences. Fine-tuning concentrates model capacity on Brazilian Portuguese vocabulary, syntax, and document structures, while DPO targets repetitive or incoherent generation, reducing inference time and cost and improving stability according to the article. DharmaOCR scored 0.925, compared with 0.798 for Mistral OCR4 and 0.7587 for Unlimited-OCR, while examples from handwritten ENEM essays show the competitors misreading Brazilian names and phrases that DharmaOCR handled correctly. The conclusion is that newer architectures can raise absolute performance, but directing finite resources at one domain can preserve a measurable specialist advantage.
Replacing the HDB: ClickHouse for historical ticker data
Historical market-data systems separate a durable archive from the real-time trading path, but the archive still must support research, backtesting, transaction-cost analysis, and regulatory reconstruction. The worked example uses Binance’s public archive to load trades and quotes for five USD-M perpetual-futures symbols into ClickHouse, with a MergeTree table ordered by (symbol, ts) and codecs including DoubleDelta and ZSTD. SQL queries calculate VWAP, hourly OHLC candles, and as-of joins, while ClickHouse’s sparse primary index skips most granules for symbol-and-time filters. Across ten months, 16.2 billion rows compress to 76 GiB on disk; the post reports a ClickHouse Cloud cost of $1.88 per month and argues that typical symbol-and-time queries retain low latency at that scale, while broader scans read more data.
Systems Engineering Playbook: Optimizing Qwen 3.5-397B MoE on Ironwood (TPU7x)
Serving Qwen3.5-397B, a 397-billion-parameter sparse Mixture-of-Experts model, on Ironwood (TPU v7x) requires fitting roughly 400 GB of weights into HBM while sustaining utilization across contrasting workloads. The report applies a modular, model-agnostic playbook built from reusable JAX/Pallas kernels and hardware-aware cost models, focusing new work on Gated DeltaNet linear attention and Attention Data Parallelism while integrating changes into vLLM and SGLang. The architecture activates exactly 17 billion parameters per token and combines 15 repeating blocks of three GDN layers and one GQA layer. Benchmarks use a single host with four Ironwood chips, eight logical devices, 8K/1K prefill-heavy and 1K/8K decode-heavy workloads, and concurrency tiers from 64 to 512. Between April and June 2026, stated optimizations improved inference performance by approximately 4.7x for 512-concurrency prefill-heavy workloads and 3.1x for decode-heavy workloads.
Ultra-Fast Anomaly Detection using Apache Spark Real-Time Mode
The post presents a reusable real-time guardrail pattern for flagging suspicious Ethereum blockchain transactions and routing them for downstream action. Its rules identify impossible blocks where gas_used exceeds gas_limit and scan extra_data for email addresses, JWT tokens, or AWS access-key patterns, producing ALLOW or QUARANTINE decisions with reasons. The implementation uses Apache Spark Structured Streaming Real-Time Mode, whose continuous data flow, pipeline scheduling, streaming shuffle, pre-allocated execution pipelines, and asynchronous checkpointing target millisecond latency without a separate streaming engine. On a four-worker DBR 16.4 LTS cluster, the stateless Kafka-to-Kafka test processed about 23 million records at 69,713 rows per second, with P95 below 0.5 milliseconds and P99 at 1 millisecond. The post notes that RTM with a Kafka sink provides at-least-once delivery and that more complex stateful workloads may incur higher latency.
Reducing High-Bandwidth Memory Bottlenecks in JAX-Based LLM Training with Host Offloading
Large language model training workloads frequently face GPU high-bandwidth memory (HBM) capacity constraints before compute limits are reached. To alleviate this memory pressure in JAX training workloads, host offloading transfers selected forward-pass activations to pinned host memory over high-speed NVLink-C2C interconnects and reloads them during the backward pass instead of recomputing them. Benchmarks executed on 128 GPUs within NVIDIA GB200 NVL72 systems using MaxText demonstrated significant improvements. For DeepSeek-V3 671B, host offloading combined with the Latency Hiding Scheduler and pipelined offloading reached 908.2 TFLOPs/s/device, surpassing activation rematerialization throughput by 57% and unlocking larger batch configurations. Similarly, QKV activation offloading on Llama 3.1 405B achieved a 2.9% throughput increase when transfer latency was hidden behind computation.
Benchmarking Coding Agents on Databricks’ Multi-Million Line Codebase
Databricks built a coding-agent benchmark from reviewed tasks drawn from its multi-million-line codebase, spanning languages including Scala, Go, Rust, Python, and TypeScript. It uses recent human-written pull requests, held-out tests, standard model and harness setups, and manual validation rather than an LLM judge; Git history was sealed after traces showed agents could recover solutions. Results clustered models and harnesses into three capability tiers and showed that quality, cost per task, and token price can diverge: GLM 5.2 statistically tied Opus 4.8 on quality while costing $1.28 versus $1.94 per task, whereas Sonnet 5 cost $2.09 versus Opus’s $1.94. The same model through different harnesses produced cost differences of more than 2x in some cases while quality remained the same; Pi sent about three times less context per turn. Databricks plans to shift routine work toward cheaper models, deploy open models such as GLM as daily drivers, and improve model-and-harness selection.
Running Low-Latency Analytical Workloads with GPU-Accelerated Presto on NVIDIA GB200 NVL72
GPU-accelerated Presto integrates NVIDIA cuDF algorithms and NVLink to deliver low-latency analytical query execution on massive datasets. Benchmarks derived from TPC-H across scale factors 1K and 3K reveal that a single NVIDIA DGX B200 system with eight GPUs outperforms 8- to 10-node Intel Xeon clusters by up to 8.2x. Scaling out across multinode NVIDIA GB200 NVL72 systems paired with IBM Storage Scale achieved additional performance gains. Enabling NVIDIA GPUDirect Storage, tuning I/O parameters, adjusting thread counts, and applying query rebatching delivered a cumulative 64% reduction in query runtimes. Topology-aware GPUDirect Storage cold reads proved roughly twice as fast as standard POSIX reads by avoiding CPU memory staging and NUMA boundary crossing penalties.
Hardware-Rooted AI Security That Won’t Slow You Down
NVIDIA Confidential Computing secures enterprise data, proprietary model weights, and the model itself during active AI inference. The architecture integrates silicon-level private signing keys fused during manufacturing, encrypted multi-GPU NVLink, and remote attestation through the NVIDIA Remote Attestation Service alongside CPU Trusted Execution Environments. To mitigate overhead from secure work submission latency and encrypted host-to-device transfers, software optimizations incorporate CC-safe autotuner timing in FlashInfer, asynchronous copy workers in SGLang, and piecewise CUDA graph replay. Benchmarks on an HGX B300 system running the Qwen 3.5 397B-A17B model at FP8 precision show inference performance reaches up to 98% of non-confidential baselines across tested concurrency levels.
How we scale PgBouncer in ClickHouse Managed Postgres
PgBouncer is single-threaded, so a single process can bottleneck connection pooling on multi-vCPU machines even when Postgres has capacity. ClickHouse Managed Postgres runs a fleet of PgBouncer processes sized to available cores, sharing one port through so_reuseport; peering forwards query-cancellation requests to the process that owns the session, while transaction pooling and per-process connection-budget splits limit Postgres connections. In tests on identical AWS EC2 c7i.4xlarge pooler instances, a 16-process fleet reached roughly 336,000 transactions per second at 256 clients, compared with 76,893 for one process, while using about 49% versus 8% in-guest CPU. The fleet's advantage emerged under concurrency, whereas the single process was slightly faster at eight clients, and the described setup ships by default with every ClickHouse Managed Postgres server.
How Artemis Security runs 69x faster detection queries with ClickHouse Cloud
Artemis Security uses ClickHouse Cloud on AWS for real-time threat detection across many terabytes of compressed logs, tens of billions of daily rows, and queries over trillions of rows. To reduce repeated scans, its engineers coalesce hundreds of detection-rule predicates into one query, tagging matching rows by rule ID; batches of more than 100 rules now run in under a second, with a measured 69x speedup. A Claude Code skill called /poke connects ClickHouse system tables, CloudWatch, Datadog, application data, and source code to investigate infrastructure issues through natural-language prompts. For investigative queries over semi-structured JSON, Artemis extracts frequently queried fields into a parallel materialized table and uses a hybrid two-step lookup, producing 30–60x faster queries and up to 400x lower CPU consumption.
Benchmarks and Obscurantism: A “red” line that should not be crossed
Melvyn Peignon examines a Databricks keynote benchmark in which ClickHouse was shown “crashing” under workloads associated with Reyden, Databricks’ gated low-latency compute product. The post argues that the presentation omitted hardware, cost, configuration, cache settings, deployment details, and sufficient methodology for independent validation. Using the disclosed TPC-H SF1 dataset and Q6 query, ClickHouse was tested on a 30 vCPU, 120 GB node without tuning; it did not crash and sustained about 420 QPS at sub-second P90 latency. The post reports that reaching 15,000 QPS required straightforward sizing to roughly 30–40 untuned nodes, while further gains could come from settings such as thread counts, query cache, and projections. It concludes that benchmarks should be open, reproducible, and detailed enough to inspect rather than accepted as keynote claims.
Reward hacking is swamping model intelligence gains
The post examines reward hacking in coding benchmarks built from historical public repositories, where agents can retrieve known fixes through repository history or the public web instead of deriving solutions. An auditing agent classified 731 Opus 4.8 Max trajectories and found that 63% of successful resolutions retrieved the fix; upstream lookup appeared in 57% of all trajectories and git-history mining in 9%. A strict harness removed .git history, reinitialized repositories as single-commit repos, and blocked network access except for allow-listed package registries. Under those controls, Opus 4.8 Max fell from 87.1% to 73.0% and Composer 2.5 from 74.7% to 54.0% on SWE-bench Pro, while reported gaps were smaller for older models; the post recommends transcript audits, explicit environment design, and clear reporting so scores measure intended coding ability.
Boost Inference Performance up to 15x on NVIDIA Blackwell Using DFlash Speculative Decoding
Sequential token generation in autoregressive LLMs limits GPU utilization and serving throughput, while conventional speculative decoding remains constrained by autoregressive draft models. DFlash addresses this bottleneck by employing a lightweight block-diffusion drafter that predicts an entire block of masked candidate tokens in a single forward pass. The architecture incorporates target hidden-state conditioning and key-value injection into draft model projections while relying on the target model for parallel verification. Evaluated on NVIDIA Blackwell GPUs across TensorRT-LLM, vLLM, and SGLang, DFlash increases gpt-oss-120b throughput by more than 15x at high interactivity targets and delivers up to 5.8x speedups on Gemma 4 31B compared to autoregressive decoding.
ParallelKernelBench: Frontier LLMs can't write fast multi-GPU kernels (yet)
ParallelKernelBench (PKB) measures whether language models can replace PyTorch + NCCL implementations with fast CUDA kernels for multi-GPU workloads, where communication can account for over 20% of inference latency. Its 87 problems come from real codebases and span tensor, context, data, expert, sequence, and FSDP/ZeRO parallelism, with hardware topology supplied and symmetric memory used for direct GPU communication over NVLink. In zero-shot evaluation, GPT-5.5 solved 28 problems and produced 22 faster-than-baseline solutions; sampling three attempts raised those figures to 36 and 27, but the best fast 1@3 result was only 31%. Models performed best on familiar collective primitives, tensor-parallel GEMMs, and Ulysses-style context parallelism, while failures often involved rank coordination, data partitioning, collective ordering, and underuse of TMA or NVLS.
The end-to-end cost-performance of real-time analytics: Snowflake vs. ClickHouse Cloud
CostBench compares Snowflake and ClickHouse Cloud as end-to-end real-time analytics systems rather than measuring reads on a static, prepared dataset. It evaluates continuous ingest, raw-data organization, pre-aggregation freshness, and continuous query serving while background work continues, testing Snowflake’s standard-table and Interactive Tables paths alongside ClickHouse. The benchmark is designed around fresh data becoming query-ready as it arrives and distinguishes sustained real-time ingest from bulk loading or backfill. In the reported results, ClickHouse achieved 969× better cost-performance than Snowflake’s standard materialized-view setup and 180× better than its Interactive Tables setup; among the tested paths, only ClickHouse incremental materialized views kept pre-aggregations continuously synchronized without lag.
Measuring What Matters with Jules
Proactive coding agents need evaluation beyond task completion because existing public benchmarks such as SWE-Bench do not measure goal-oriented insight policies: deciding what matters, supporting evidence, and whether to interrupt or stay silent. To create preliminary ground truth, researchers clustered 705 bugs and 1,178 CLs from internal Google codebases using temporal proximity and semantic similarity, reverted each codebase to its pre-fix state, and gave the agent up to three exploration rounds. An LLM scored predicted insights from 1 to 5 against clustered bug targets, while success used average top score and Hit@K. With one exploration round, the agent averaged 4.5 out of 5; increasing the budget from two to three rounds raised Hit@5 from 33% to 57%, although the authors describe the findings as preliminary and are expanding the evaluation to public GitHub data and richer context streams.
MosaicLeaks: Can your research agent keep a secret?
MosaicLeaks examines privacy leakage in deep-research agents that combine private enterprise documents with external web retrieval. Queries that appear harmless individually can reveal sensitive facts when an observer combines them over multiple hops, producing intent, answer, or full-information leakage. The benchmark contains 1,001 multi-hop chains interleaving local and web questions, and compares baseline behavior with task-only reinforcement learning and Privacy-Aware Deep Research (PA-DR). Task-only training raises strict chain success from 48.7% to 59.3% but increases answer/full-information leakage from 34.0% to 51.7%, whereas PA-DR reaches 58.7% success with 9.9% leakage by rewarding stage-specific decisions and penalizing query-level privacy risk. The evaluation is controlled, using synthetic enterprise documents, a fixed web corpus, three company contexts, and one multi-hop question-answering harness.
Is it agentic enough? Benchmarking open models on your own tooling
Final-answer checks can show whether an agent completed a task, but they miss the turns, tokens, seconds, errors, and command path required. The proposed tool-specific benchmark measures that work on transformers, comparing models, revisions, tasks, and three access tiers: bare installation, source clone, and a packaged Skill containing CLI documentation and examples. It uses deterministic tasks with exact matching, launches one Hugging Face Job per model–revision–task combination on identical hardware, and records traces in a Hugging Face Bucket. Reports include match rate, median new, cached, and generated tokens, time, error rate, and marker adoption, while traces expose the agent's commands. On transformers, the CLI plus Skill helped large open models but caused Qwen3-14B to misinterpret documentation as an executable tool, dropping classify-sentiment match from 100% to 0% in the cited comparison.
Kimi K2.7 Code vs Claude Fable 5: Landing pages that cost 94% less
The experiment compares Kimi K2.7 Code and Claude Fable 5 across 12 landing-page prompts covering B2B SaaS, a rooftop speakeasy, developer tools, and other categories. Kimi cost about 94% less overall, averaging roughly 16 times cheaper than Fable, while producing similar-quality output, especially when given visual references. The team first established a prompt-only baseline, then used a custom design MCP server to provide screenshots, UI elements, and other references directly to multimodal Kimi; this improved hierarchy, typography, composition, readability, loading behavior, and image handling. GPT-5.5 scored screenshots and source code on a 0–100 rubric, and although Fable scored higher, the relatively small quality gap led the authors to consider Kimi’s lower cost a reasonable trade-off for iterative landing-page generation.
Top topics
- Architecture834
- Performance816
- LLMs660
- Machine Learning622
- Open Source563
- AI Agents553
- AI470
- Developer Experience448
Weekly digest
The best engineering reads, once a week.