Loading…
Topic
21 posts about Benchmark. Every summary links to the original.
See the idea
Arabic is not a single target for a language model. This explainer maps its four parts — dialect surface, data pipeline, model, and evaluation — so you can tell where an Arabic feature really succeeds or fails.
A comparison of Apache Lucene's BM25 scoring and StableTfl for paginated search: BM25 maximizes relevance, while StableTfl trades a small relevance drop for consistent ranking across replicas and pages.
BioNeMo Inference Runtime runs one pipeline per target: parse, tokenize, generate features, infer on a GPU replica, and write a structure. This diagram traces those stages and the throughput trade-off.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
The October 2026 ClickHouse newsletter announces CostBench results where ClickHouse Cloud delivered 752× better performance per dollar than Databricks and 438× to 512× better than BigQuery on real-time analytics workloads. The 26.9 release extends LIMIT with AFTER and UNTIL boundary conditions, adds incremental refresh for materialized views, and introduces CREATE TOKEN for time-limited credentials. Executable user-defined functions are now generally available on ClickHouse Cloud, supporting precompiled Rust, Go, C++, and JavaScript binaries with outbound network access and a deterministic flag. Featured community member Hoang Minh V has had over 100 pull requests merged since May, including a fast path for PromQL aggregations and fixes to TimeSeries reads. The newsletter also covers the official Apache Airflow provider for ClickHouse, the chDB durable layer for agent memory, and a Postgres round-up on memory safety and local NVMe storage.
Slide Hub, an internal Databricks app for slide search, uses Lakebase Postgres with Lakebase Search extensions (lakebase_vector and lakebase_text) to run hybrid vector and BM25 search on the same database as application data. The app stores 23k slides and indexes up to 4M passages from the MIRACL English corpus. Benchmarking shows that below 8 Compute Units (CU), Lakebase Postgres is the bottleneck; above 8 CU, the app’s worker count limits throughput. Candidate pool size significantly affects cold-start latency: 200 candidates preserves 90.5% of top-10 results while touching a third of the pages of 1,000 candidates. A 14-minute load test cost 0.13, with Lakebase compute billing only while active. For production, a 0.5 CU minimum costs ~ 31-40/month and avoids cold-start delays.
CostBench streamed 113.2 billion stock-market quotes into both systems while four aggregate queries ran every ten minutes and two drill-down queries ran every hour. ClickHouse asynchronous inserts order raw data and build pre-aggregations during ingestion, while Redshift relies on separate background sorting and materialized view refreshes. ClickHouse Cloud led in preparation cost ($28.69 versus 522.03), normalized query cost ( 0.05 versus $24.97), and accumulated query runtime (58.12 seconds versus 1.69 hours). The combined score shows ClickHouse Cloud delivered 1,996× better end-to-end performance per dollar in this continuous-ingestion comparison. The difference originates in the preparation path: ClickHouse makes data query-ready during inserts, whereas Redshift’s streaming, sorting, and refresh steps advance independently.
The paper evaluates LLMs as alternatives to human judgment for machine translation quality annotation, comparing agreement with human annotators on two schemes: Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA). Tests cover a long-context set of 70 language pairs plus WMT23 and WMT25 data, examining both score and error span annotation agreement across language pairs and domains. Results show LLM-human agreement sometimes exceeds human-human agreement, but both vary substantially across annotation schemes, language pairs, and domains and remain unreliable for most settings. Humans are particularly challenged by fine-grained MQM annotations and low-resource language pairs, while LLMs struggle with minor errors, wrong language variants, and error span annotation. The authors suggest human-LLM collaborative annotation pipelines as a way to address the reliability issues identified in this work.
Klaus 2022 Customer Service Quality Benchmark Report, based on a survey of 300 support leaders, identifies three key trends. First, support teams are measuring impact more than ever: average CSAT rose 14% to 89%, and average NPS increased from 45 to 51; 10% more teams track CSAT, and NPS tracking jumped from 28% to 48%. Second, maintaining quality at scale is the top challenge, with 40% citing reducing repetitive queries; teams use first-party data, chatbots, and automation to deliver personal, fast support while scaling. Third, proactive and self-serve support are growing priorities: 70% plan to invest more in proactive support, and 73% found self-service reduced workload; knowledge bases compound in value over time. The report recommends focusing on the right metrics, collaborating with sales and marketing, and continuously learning and optimizing to elevate support’s business role.
ClickHouse Cloud delivered 438× better end-to-end performance per dollar than BigQuery under Capacity pricing, and 512× under On-demand pricing, while continuously ingesting 113.2 billion stock-market quotes. The CostBench methodology measures preparation cost, query cost, and query runtime as data arrives. ClickHouse Cloud led in all three components: preparation cost was 8.5× lower than BigQuery Capacity, normalized query cost was 94× lower, and accumulated query runtime was 50.8× lower. The difference originates in how each system handles incoming data: ClickHouse orders raw data and builds current summaries during inserts, while BigQuery makes new rows queryable before background clustering and materialized view refresh complete. This leaves BigQuery aggregate queries to include rows that refresh has not yet summarized, increasing cost and runtime.
The CostBench benchmark streamed 113.2 billion stock-market quotes at one million rows per second while running aggregate and drill-down queries to compare ClickHouse Cloud and Databricks for real-time analytics under continuous ingestion. ClickHouse Cloud delivered 752× better end-to-end performance per dollar than the Databricks stack using Serverless SQL, Zerobus Ingest, liquid clustering, and incremental materialized views. ClickHouse Cloud led in all three components: preparation cost ($28.69 versus 695.60), normalized query cost ( 0.05 versus $2.65), and accumulated query runtime (56.39 seconds versus 29.10 minutes). The gap stems from ClickHouse preparing ordered raw data and current summaries on the insert path, while Databricks requires separate clustering and refresh services that cost more and can leave queries with unfinished layout work. Databricks’ new Lakehouse//RT, still in beta, was not included in this comparison.
Falcon-Emirati-7B is a dialect-specialized language model built on Falcon-H1-Arabic to understand and generate Emirati Arabic, a Gulf dialect with distinct vocabulary and cultural context that Modern Standard Arabic models miss. The model uses a hybrid architecture combining Mamba state-space models and Transformer attention running in parallel inside every block, with fused outputs for efficient long-context processing. Training required a dedicated data pipeline of authentic Emirati web content, MSA material about Emirati culture, and synthetic data guided by strict glossaries and style rules to cover gaps in available dialectal text. On the Alyah benchmark, Falcon-Emirati-7B scored 84.83%, outperforming larger multilingual models, and achieved 85.57% on the UAE cultural understanding task in ArabCulture-Dialogue. The 7B parameter size was chosen as the best balance of quality against training and inference cost for dialect adaptation.
ThinkingBox, a benchmark from Microsoft and Hugging Face, grades AI agents on the database state and side effects they leave behind rather than on tool calls or final responses. In a common-set ablation covering 121,680 valid trials across 12 LLM models, 79,853 attempts failed executable checks, and 67.24% of those failures still terminated cleanly and invoked a state-changing tool without error, with wrong field values in 77.61% of them. The benchmark runs every task 20 independent times from identical clean backends, reporting pass@1, pass@20, and observed 20/20 consistency to separate capability from reliability. For example, Kimi-K3 solved 93.89% of tasks at least once but only 13.41% consistently, while Claude Opus 5 completed 47.53% of tasks on every attempt, and Claude Opus 5.5 gained no additional dependability over its predecessor despite higher pass@1. ThinkingBox is available through Hugging Face’s OpenEnv with MIT-licensed code and CDLA-Permissive-2.0 benchmark data.
Supabase announced a code-first approach at its Select 2026 event, enabling developers to store schema and configuration in code so coding agents can set up and modify projects end to end. Local Supabase now runs as native processes without Docker, allowing multiple instances per directory and compatibility with sandboxes like Claude Code. A new MCP server lets application users connect their own agents, authenticating through existing Auth flows and enforcing row-level security on every tool call. Operational tools include Health Check Advisors for error rates, Database Connections for session management, Explorer notebooks for unified SQL and log queries, and query_logs for agent monitoring. Scaling features encompass Multigres for multi-node Postgres high availability, OrioleDB to avoid table bloat and VACUUM, Supabase Pipelines for near real-time data streaming to analytics platforms, and dbarena for transparent benchmarks.
Enterprise generative AI applications require guardrails that balance flexibility with reliability. Decision models such as TypeSafe AI’s Jev produce fixed decisions with guaranteed schema and type safety, operating faster and cheaper than LLM-as-a-judge approaches. The Red Hat AI Safety team benchmarked nine guardrails across four methodologies for prompt injection and content safety tasks: pre-trained CPU-scale classifiers like Red Hat’s DeBERTa-v3 and Granite Guardian HAP-125m, older zero-shot models like BART-large-mnli, modern LLM-as-a-judge safety models including Shieldstral and Nemotron-3.5, and Jev-style decision models including Laya and DiffusionGemma. Each guardrail was evaluated using stock and custom risk policies, with Jev and DiffusionGemma using zero-shot questions. The results highlight trade-offs: Jev provides zero-shot flexibility and low latency, but purpose-built classifiers often match its speed and cost while delivering superior task-specific performance, and LLM-as-a-judge models can be tuned for higher accuracy.
ShopGym converts live storefronts into self-contained, resettable sandbox shops for training and evaluating shopping agents, addressing challenges from changing products and bot detection. Its ShopArena layer separates exploration from generation using anonymized specifications and coding agents with execution-verification loops that repair failures without regenerating the whole shop. ShopGuru then synthesizes benchmark tasks across seven skill categories using rule-based and LLM generators grounded in each shop’s catalog, navigation, and policies. Validation over 224 tasks across six shops, three synthetic and three real, showed synthetic shops retain key structural properties and agent performance positively correlates with live storefronts when tested with BrowserGym and an internal system. The framework aims to provide reproducible environments for agents to fail and learn before interacting with live stores, with plans to expand to more retail domains and task types.
The open-source TTS model landscape has exploded, with over 8,000 models on Hugging Face Hub, yet evaluation methods remain unstandardized and cannot scale. The Open TTS Leaderboard addresses this by using objective metrics such as word error rate (WER), speaker similarity (SIM), and speed (RTFx and TTFA) to evaluate models in hours instead of weeks. It supports multilingual evaluation and voice cloning, providing separate tabs for listening to outputs and comparing streaming performance. Unlike arena-based leaderboards that significantly underrepresent open models due to high hosting costs, this leaderboard prioritizes open-source models. The project encourages community participation through feedback and voting; evaluation scripts will be open-sourced soon, and future updates will incorporate new datasets, languages, and metrics.
Postgres memory tuning is difficult because work_mem applies per operation, not per query, and parallel workers multiply memory usage without a query-wide cap. Some executor structures, such as those in recursive queries, have no disk-backed fallback and can grow until the backend runs out of memory. A benchmark using a recursive query that exhausts memory tested ClickHouse Managed Postgres, Cloud SQL, PlanetScale, and Amazon RDS on 16 GiB instances. ClickHouse Managed Postgres enforces a memory ceiling at 57% of host RAM, rejecting queries that exceed it, while RDS allows backends to consume nearly all RAM until the host crashes. Cloud SQL and PlanetScale enforce similar caps, but ClickHouse Managed Postgres’s ceiling is lower, a deliberate tradeoff that returns clean query failures instead of forcing every client through crash recovery.
The UK AI Security Institute (AISI) has begun using EvalEval’s Every Eval Ever (EEE) schema and Evaluation Cards platform to openly share LLM evaluation results. This collaboration, stemming from a joint workshop at NeurIPS 2025, aims to make benchmark results more reproducible by providing standardized configuration and context alongside verified scores. AISI released results for five benchmarks—HealthBench, FrontierMath, Humanity’s Last Exam, SWE-Bench Pro, and Terminal-Bench 2.0—across six frontier models, plus two cyber evaluations using a partially overlapping model set. The release accompanies a paper studying how inference-time compute and evaluation protocol affect performance, revealing that models continued solving tasks when given correctness feedback. By adopting a shared reporting schema, evaluators can compare findings across studies and understand how setup choices influence reported performance, while EvalEval invites model developers, evaluation creators, and researchers to contribute to the shared infrastructure.
Standard LLM benchmarks are insufficient for evaluating AI agents because they score static outputs rather than multi-step tool-call sequences against a live environment. The post explains that agent evaluation requires two scoring layers: step-level (process scoring) checks each call’s validity and usefulness, while end-to-end (E2E) scoring checks only the final environment state. A trace logs every step and environment state, and metrics roll up through a hierarchy of benchmark, trial, task, turn, and step. Key metrics include task success rate, consistency, tool-call precision, argument accuracy, steps per success, and cost per success. The post argues that benchmarks are converging on tool use because deployed agents always use tools, and enterprise benchmarks should measure task completion against real APIs and policies.
A benchmark of the Icon Payments Framework (IPF), trusted by Citi, NatWest, and BNP Paribas, on MongoDB Atlas demonstrated extreme performance and resilience for instant payment processing. The system sustained 6,000 end-to-end payments per second, requiring approximately 430,000 simultaneous database operations, with a mean end-to-end latency of 0.51 seconds and persistence latency below 30 milliseconds. Upgrading from MongoDB Atlas 8.0 to 8.3 reduced persistence latency by roughly 50%, from 40 to 20 milliseconds at 4,000 payments per second, and throughput scaled near-linearly across the 500 to 6,000 range while latency remained flat until peak and hot-shard CPU utilization sat at about half. During a primary failover, a new node was elected in five seconds with zero data loss and no application restart. An application node failure recovered to 3,500 transactions per second in 90 seconds with zero data loss.
GLM-5.3 and GPT-5.6 Sol were evaluated on 113 DeepSWE tasks with 904 rollouts, comparing accuracy, cost, and failure patterns. Sol edges ahead at pass@1 (72.7% vs 69.0%), but GLM-5.3 leads at pass@4 (87.6% vs 85.8%) at $3.99 per rollout, half Sol’s $8.37 cost and yielding 17 solves per $100 versus Sol’s 9. Their low per-task correlation of 0.43 makes a cascade effective: GLM-5.3 first, then Sol on test failure, achieving 85.9% solved at $6.61 per task. Sol runs faster (19 minutes, 61 steps vs 35 minutes, 124 steps) and achieves higher reliability (61 tasks solved all four tries vs 48), but it breaks passing tests in 20% of failures versus GLM-5.3’s 11%.
GLM-5.3 and Claude Fable 5 were compared on the DeepSWE benchmark across 113 tasks with four trials each, totaling 904 rollouts. On first-attempt accuracy, Fable 5 scored 69.7% and GLM-5.3 scored 69.0%, a statistical tie inside error bars. GLM-5.3 led on pass@2 (81.1% vs. 77.1%) and pass@4 (87.6% vs. 84.1%), and cost $3.99 per rollout versus Fable’s $21.63, a 5.4x price gap. Per $100, GLM-5.3 solved 17 tasks while Fable solved 3. Both models regress test suites in only 11% of failures, and GLM-5.3 averages 35 minutes per rollout with 80k output tokens versus Fable’s 34 minutes and 114k tokens. The models are near-substitutes with 0.65 per-task correlation; GLM-5.3 won five of eight domains including JavaScript and concurrency, while Fable excelled only in Rust and serialization, making GLM-5.3 the practical default.
DeepSeek V4 Pro 0813 costs $0.24 per rollout and achieves 62.8% pass@1 on DeepSWE, while GPT-5.6 Sol reaches 72.7% pass@1 at $8.37 per rollout—a 35x price gap. At four attempts, DeepSeek V4 Pro 0813 surpasses Sol, with 88.5% pass@4 versus 85.8%. Sol is faster and more reliable, completing a rollout in 17 minutes and 53 steps, and breaking tests in only 11% of its failures. A cascade strategy—running DeepSeek V4 Pro 0813 first and escalating to Sol on test failure—solves 83.0% of tasks at $3.35 each, outperforming either model alone. Per $100, DeepSeek V4 Pro 0813 solves 260 tasks versus Sol’s 9, making it the value leader for budget-constrained workloads.
The best engineering reads, once a week.
