Loading…
Browse by type
Benchmark reads
Benchmark posts on Yomu, summarised newest first.
- 175 posts
- 20 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Benchmark posts
MosaicLeaks: Can your research agent keep a secret?
MosaicLeaks examines privacy leakage in deep-research agents that combine private enterprise documents with external web retrieval. Queries that appear harmless individually can reveal sensitive facts when an observer combines them over multiple hops, producing intent, answer, or full-information leakage. The benchmark contains 1,001 multi-hop chains interleaving local and web questions, and compares baseline behavior with task-only reinforcement learning and Privacy-Aware Deep Research (PA-DR). Task-only training raises strict chain success from 48.7% to 59.3% but increases answer/full-information leakage from 34.0% to 51.7%, whereas PA-DR reaches 58.7% success with 9.9% leakage by rewarding stage-specific decisions and penalizing query-level privacy risk. The evaluation is controlled, using synthetic enterprise documents, a fixed web corpus, three company contexts, and one multi-hop question-answering harness.
Is it agentic enough? Benchmarking open models on your own tooling
Final-answer checks can show whether an agent completed a task, but they miss the turns, tokens, seconds, errors, and command path required. The proposed tool-specific benchmark measures that work on transformers, comparing models, revisions, tasks, and three access tiers: bare installation, source clone, and a packaged Skill containing CLI documentation and examples. It uses deterministic tasks with exact matching, launches one Hugging Face Job per model–revision–task combination on identical hardware, and records traces in a Hugging Face Bucket. Reports include match rate, median new, cached, and generated tokens, time, error rate, and marker adoption, while traces expose the agent's commands. On transformers, the CLI plus Skill helped large open models but caused Qwen3-14B to misinterpret documentation as an executable tool, dropping classify-sentiment match from 100% to 0% in the cited comparison.
Kimi K2.7 Code vs Claude Fable 5: Landing pages that cost 94% less
The experiment compares Kimi K2.7 Code and Claude Fable 5 across 12 landing-page prompts covering B2B SaaS, a rooftop speakeasy, developer tools, and other categories. Kimi cost about 94% less overall, averaging roughly 16 times cheaper than Fable, while producing similar-quality output, especially when given visual references. The team first established a prompt-only baseline, then used a custom design MCP server to provide screenshots, UI elements, and other references directly to multimodal Kimi; this improved hierarchy, typography, composition, readability, loading behavior, and image handling. GPT-5.5 scored screenshots and source code on a 0–100 rubric, and although Fable scored higher, the relatively small quality gap led the authors to consider Kimi’s lower cost a reasonable trade-off for iterative landing-page generation.
NVIDIA Blackwell Tops MLPerf Training 6.0 with Industry-Leading Scale and Performance
NVIDIA submitted results across all MLPerf Training v6.0 benchmarks, setting performance records on workloads including the DeepSeek-V3 and GPT-OSS-20B Mixture of Experts models. To scale training up to 8,192 Blackwell GPUs, NVIDIA combined hardware cluster designs with scale-out networking via Spectrum-X Ethernet and Quantum InfiniBand. The engineering team eliminated CPU-GPU synchronization bottlenecks in token-dropless MoEs by implementing full-iteration CUDA graphs with synchronization-free operators and paged stashing. Further software optimizations included CuTe DSL kernel fusions, an MXFP8 attention block in the Transformer Engine, and transitioning router elementwise math to FP32. Across these benchmarks, the GB300 NVL72 platform achieved the fastest time to train at scale and the highest normalized per-accelerator performance.
How to Optimize Transformer-Based Models for Low-Precision Training
Accelerating transformer training requires understanding how low-precision formats such as FP8 and NVFP4 impact specific general matrix multiplication (GEMM) workloads. Transformer configurations do not explicitly reveal these shapes, making microbenchmarks necessary before committing to full training runs. NVIDIA Transformer Engine enables quantization and kernel dispatch across precisions, which can be evaluated in realistic autocast mode or kernel-only prequantized mode. Profiling the ESM2-15B model on NVIDIA B300 GPUs demonstrated that NVFP4 achieved a 1.79x blended forward propagation speedup over MXFP8 and up to 4.01x over BF16 in prequantized execution. Although large GEMM dimensions successfully overcome quantization overheads, dynamic scaling, Hadamard transforms, and kernel selection asymmetries moderate real-world gains.
Ingesting the Milky Way: Petabyte-Scale with Zerobus Ingest
The post benchmarks Zerobus Ingest, a managed, serverless, push-based service that writes producer data directly to Delta tables governed by Unity Catalog, against a petabyte-scale telemetry workload. Using NASA’s NEOWISE dataset and Locust, the test modeled fan-in from 2,048 concurrent streams, using Protocol Buffer 2 data over approximately 24 hours. The design replaces static partition-based ordering with stream-connection ordering, allowing heuristic routing across pods, dynamic partitioning, and autoscaling while existing streams drain. It also uses zeroparser, a zero-copy protobuf decoder whose design relies on Rust’s lifetime system and supports dynamic descriptors at about 1 GB/s per CPU core. The test sustained 12 GB/s to one table, ingested 1.04 trillion rows, and reached 1 petabyte within 24 hours; Zerobus Ingest is generally available, with additional APIs on its roadmap.
Agentic Testing: Where Agents Fit in the E2E Testing Stack
Traditional end-to-end tests validate rigid user journeys, whereas agentic tests verify whether broad goals can be achieved by adapting actions dynamically. To evaluate agentic testing tradeoffs, researchers executed over 200 runs across Playwright Model Context Protocol (MCP), Playwright CLI, and agent-generated Playwright tests using Claude models. Playwright MCP demonstrated high reliability with failure rates of 0% on simple thread replies and approximately 12% on complex search discovery flows. Playwright CLI and generated code struggled more on complex workflows, exhibiting failure rates of approximately 20% and 48% respectively. Although generated tests were faster with average runtimes of roughly three minutes, agentic testing provides a distinct exploratory layer atop deterministic CI test suites.
Building ClickCannon - a tool for benchmarking ClickHouse
ClickCannon was created to answer how much CPU, memory, and disk are needed for ClickStack observability workloads, while accounting for schema, ingestion rate, hardware, and concurrent queries. It replays about 2 TiB of real OpenTelemetry trace data, converted to ClickHouse Native format with ZSTD compression, rather than relying only on synthetic datasets. A custom multithreaded Go reader/inserter provides control over uncompressed throughput and exposes stream metrics such as compressed and uncompressed bytes, row counts, and block counts, while the framework simulates users and query workloads during ingestion. The project evolved into an open-source, generic ClickHouse benchmarking tool for custom insert and query workloads, with fine-grained control over throughput, concurrency, and user behavior; the post reports that it can push substantial throughput from a single machine.
Designing the hf CLI as an agent-optimized way to work with the Hub
The post presents a redesign of hf, the Hugging Face Hub’s official CLI, for both human users and coding agents that download, upload, manage repositories, run Jobs, and administer other Hub resources. The CLI detects agents through environment variables, switches to compact, complete TSV output, and supports explicit human, agent, JSON, and quiet formats; agent mode was introduced in hf v1.9.0. Next-command hints, stderr separation, non-interactive failure messages, retry-safe operations, confirmations, and dry-run previews support multi-step automation without removing human-friendly terminal behavior. Benchmarks report equal-or-better success with roughly 1.3× to 1.8× fewer tokens than curl or the SDK on comparable tasks, and up to 6× fewer on complex workflows; the generated hf-cli skill reduced tool calls by about 30% without reducing token use or improving reliability.
Stack Benchmarking
Ramp’s Stack is an AI-native accounting suite designed to automate repetitive book-closing work, including reconciliations, variance analysis, data entry, and schedules and accruals. To avoid overfitting to individual design partners, Ramp built a benchmark of synthetic business worlds, realistic accounting tasks, and accountant-written grading criteria, including standard and roll-forward worlds for testing memory transfer. The benchmark contains 237 tasks and 3,469 grading criteria in the analyzed slice, and supports repeated runs to compare models, prompts, tools, skills, harness changes, and memory behavior. Optimization included ablating skills, shrinking a spreadsheet skill from 14,000 to 5,000 characters, and tuning the system against end-to-end task performance rather than narrow evaluations. The resulting Stack system achieved the highest agent performance, with 4% higher accuracy and 3% better Pass@1, while remaining on the latency frontier with GPT 5.4; schedules and accruals were harder for raw models than variance analysis.
TPC-H for less than a cent: ClickHouse Cloud vs. Snowflake, Databricks, BigQuery, and Redshift
ClickHouse Cloud is benchmarked against Snowflake, Databricks, BigQuery, and Redshift on the join-heavy TPC-H workload, with the main comparison using SF100: 100 GB, 866M rows, and 22 queries. Using a single 59-core AWS compute node, the benchmark measures fastest hot runs from three executions with result caching disabled, applies public billing models and comparable Enterprise-tier regional pricing, and publishes scripts, queries, and result files for reproduction. ClickHouse completed SF100 in 19.8 seconds for $0.063 and ranked first on the cost-performance score, while Snowflake Large was faster at 15.9 seconds but cost $0.143; at SF10, ClickHouse ran the workload in 2.9 seconds for $0.009. The results are attributed to two years of join engineering, including faster hash joins, improved planning, correlated subquery support, lazy column replication, runtime filters, and statistics-based join reordering.
Introducing CostBench: an open benchmark for data warehouse cost-performance
Traditional data warehouse benchmarks measure query execution speed without accounting for the varying compute costs across platforms. CostBench introduces an open benchmarking framework designed to evaluate data warehouses on a performance-per-dollar metric across both read and write workloads. Its initial release evaluates ClickHouse Cloud, Snowflake, Databricks, BigQuery, and Redshift using 43 production-derived analytical queries on an anonymized dataset combined with each vendor's compute billing model. In these read-side tests, ClickHouse Cloud maintained high speed at lower costs as data scaled, outperforming the nearest competitor by 23 times in cost-performance. The benchmark repository openly publishes its scripts, configurations, pricing assumptions, and raw JSON results to ensure all comparative metrics remain fully inspectable and reproducible.
Fast Copy-On-Write within Apache Parquet for Data Lakehouse ACID Upserts
Large-scale lakehouse upserts can become slow and resource-intensive, especially in Apache Hudi copy-on-write mode, where affected files are read, modified, and rewritten in full. The approach adds a Parquet row-level secondary index mapping records to files and row IDs, allowing updates to locate the specific data pages containing changed records. During copy-on-write, related pages are updated while unrelated pages are copied as byte buffers, avoiding decompression, re-encoding, and record reassembly for skipped data. Benchmarks using TPC-DS sales data, Spark jobs with matching vCore and memory settings, and update rates from 5% to 50% found significantly faster speeds than traditional approaches, including Delta Lake’s default configuration. The authors plan to integrate the row-level index and fast copy-on-write into Apache Hudi for use cases such as incremental ingestion, data deletion, and CDC.
Using MemAlign to Improve Evaluation of Traditional Machine Learning in Genie Code
Genie Code’s traditional machine learning workflows are difficult to evaluate because notebook quality depends on code, best practices, and adaptation to data and task. The team built an evaluation pipeline with nine LLM judges, each scoring applicable dimensions—including data exploration, imputation, training, MLflow logging, and cell organization—from 1 to 3, using test cases across classification, regression, and forecasting. On 50 human-graded notebooks, initial judge alignment was weak in several areas, including model training (MAE 0.680) and model use (0.562), with positivity bias contributing to missed technical flaws. Applying MLflow’s MemAlign, which combines generalized semantic guidelines with retrieved episodic examples, reduced judge error by 74–89% on the most important dimensions; removing episodic memory weakened most gains, while data exploration showed a nonsignificant regression.
Pushing the Frontier for Data Agents with Genie
Genie is Databricks’ data agent for complex questions across structured enterprise assets—tables, dashboards, and notebooks—and unstructured sources including workspace files, Google Drive, and Sharepoint. Unlike coding agents operating in static environments, it must discover relevant assets at enterprise scale, determine authoritative knowledge from potentially contradictory sources, and handle questions without verifiable tests or guaranteed answers. It combines specialized knowledge search using semantic context and metadata, parallel thinking across sampled trajectories, and Multi-LLM orchestration with optimized prompts for distinct sub-agents. On an internal benchmark of real-world data-analysis tasks, these techniques raised accuracy from 32% to over 90% against a leading coding agent while reducing cost and latency; specialized search alone improved table-discovery performance by up to 40%, while parallel thinking added latency and token costs before further optimization.
Zero-Growth Stack, Real Gains: How Stack Allocation Can Save 10% CPU in Go
At Uber, repeated Go goroutine stack expansion consumed substantial CPU in a service whose stack-memory usage remained low, making memory a viable trade-off. Go starts goroutines with 2KB stacks and doubles them when needed, copying the previous stack; the post also notes that adaptive stack sizing and stack shrinking can add runtime cost. Uber patched the runtime, disabled adaptive sizing and shrinking, and used profile-guided analysis of pprof traces plus assembly-derived function stack usage to choose a static power-of-two starting size. For the profiled service, the analyzer identified about 19KB of required stack usage, leading to a 32KB setting instead of 2KB. After deployment, stack-growth CPU fell from close to 10% to below 1%, while most instances used about 50MB of stack and a few reached 200MB.
How lakebase architecture delivers 5x faster Postgres writes
Lakebase’s separation of Postgres compute and distributed storage removes the local-disk torn-page risk that necessitates full page writes (FPW) in traditional deployments. The change disables FPW on compute and moves full page image generation into the pageserver, which creates reset points after a configured number of WAL deltas while preserving bounded read reconstruction. In HammerDB TPROC-C tests, 32-vCPU throughput rose from 95,686 to 439,300 new orders per minute, while average WAL per transaction fell from 58KB to under 4KB, a 94% reduction. Production validation reported WAL reductions from 30MB/s to 1MB/s on a 56-vCPU project, p99 read latency improvements of 30% to 50%, and up to threefold regional storage p99 improvement. The rollout reached all Lakebase Serverless and Neon databases without restarts or customer interruption, using the existing XLOG_FPW_CHANGE mechanism.
Agentic analytics starts with query-ready data: the write-side cost of Snowflake vs. ClickHouse
Agentic analytics workflows introduce bursts of concurrent, low-latency analytical queries over continuously ingested data. Analytical query efficiency relies on columnar storage and chunk-level pruning, which require contiguous, sorted data to avoid full-table scans. Rather than evaluating warehouses solely at query runtime, this benchmark compares the write-side cost of maintaining query-ready data in ClickHouse and Snowflake during continuous ingestion at approximately one million rows per second. Snowflake relies on post-write clustering to organize data, whereas ClickHouse constructs sorted layouts on the write path and refines them via background merges. Across 100B, 200B, and 300B row checkpoints on the ClickBench dataset, ClickHouse achieved a query-ready layout at 22 times lower cost and delivered 28 times better write-side cost-performance.
We Tested Marketing Incentives to AI Agents. Here's What Happened.
Ramp tested whether AI agents would relay a tracked referral incentive to people researching spend-management software. Across roughly 50 marketing pages, Cloudflare Workers served bot-specific variants—Markdown, stripped HTML, or schema-heavy content—while unique URLs and logs attributed downstream activity. Markdown surfaced most reliably, and bot classification proved difficult: major assistants were labeled “AI Assistants,” OpenAI SearchBot cached aggressively, and DeepSeek required TLS fingerprinting plus ASN detection. Over 32 days, agent relays rose from roughly 40 at 2.5 weeks to nearly 10 times that total, with Claude responsible for most matches; Perplexity moved from vague references to one specific branded mention, while ChatGPT remained at zero. The experiment concludes that agents can relay offers, but model-specific content strategies, citation-aware page selection, and better attribution are still needed.
Do you still need Elasticsearch for log analytics? ClickHouse says no.
ClickHouse introduced a redesigned full-text search powered by inverted indexes that feeds matching documents directly into its vectorized analytical engine for observability and log analytics workloads. To evaluate this capability, the open-source versions of ClickHouse and Elasticsearch were benchmarked on single-node AWS EC2 instances across synthetic OpenTelemetry log datasets scaling up to 50 billion rows. The benchmark evaluated nine representative query patterns combining text search with filtering, grouping, counting, and time-series aggregations under both cold and hot cache conditions. ClickHouse achieved a roughly fivefold smaller storage footprint on disk while running cold queries four to six times faster and hot queries approximately 1.7 to 2.6 times faster than Elasticsearch. Furthermore, ClickHouse ingested the 50-billion-row dataset in under four hours out of the box, whereas the tuned Elasticsearch setup required approximately five days.
Top topics
- Architecture834
- Performance817
- LLMs673
- Machine Learning622
- Open Source568
- AI Agents564
- AI483
- Developer Experience448
Weekly digest
The best engineering reads, once a week.