---
title: "Testing"
description: "271 posts about Testing, summarised, each linking to the original."
---

# Testing
> 271 posts about Testing, summarised, each linking to the original.

## Articles

### [Ground truth is a process, not a dataset](https://yomu.fyi/post/ground-truth-is-a-process-not-a-dataset.md)
- Company: amazon.science
- Author: Venkatesh Saligrama
- Published: Jun 3, 2026
- Type: Problem & solution

Evaluating search-augmented AI research reports requires verifying complex, multi-source claims that depend heavily on surrounding context. Unassisted PhD-level domain specialists tasked with establishing static ground-truth labels achieved only 60.8% accuracy on a hidden test set during controlled testing. To resolve benchmark errors on cognitively demanding tasks, Amazon AGI introduced the audit-then-score evaluation protocol alongside DeepFact-Bench and the DeepFact-Eval verification system. Under this protocol, model disagreements serve as challenges requiring concrete counterevidence and written rationales, enabling human auditors to evaluate competing claims and update benchmark ground truth prior to scoring. Operating under this iterative workflow, expert test accuracy rose to 90.9% across four rounds, and DeepFact-Eval reached 83.4% accuracy with GPT-4.1.


### [Stack Benchmarking](https://yomu.fyi/post/stack-benchmarking.md)
- Company: [Ramp](https://yomu.fyi/company/ramp.md)
- Author: Ryan Stevens
- Published: Jun 3, 2026
- Type: Benchmark

Ramp’s Stack is an AI-native accounting suite designed to automate repetitive book-closing work, including reconciliations, variance analysis, data entry, and schedules and accruals. To avoid overfitting to individual design partners, Ramp built a benchmark of synthetic business worlds, realistic accounting tasks, and accountant-written grading criteria, including standard and roll-forward worlds for testing memory transfer. The benchmark contains 237 tasks and 3,469 grading criteria in the analyzed slice, and supports repeated runs to compare models, prompts, tools, skills, harness changes, and memory behavior. Optimization included ablating skills, shrinking a spreadsheet skill from 14,000 to 5,000 characters, and tuning the system against end-to-end task performance rather than narrow evaluations. The resulting Stack system achieved the highest agent performance, with 4% higher accuracy and 3% better Pass@1, while remaining on the latency frontier with GPT 5.4; schedules and accruals were harder for raw models than variance analysis.


### [Beyond LLMs: Why Scalable Enterprise AI Adoption Depends on Agent Logic](https://yomu.fyi/post/beyond-llms-why-scalable-enterprise-ai-adoption-depends-on-agent-logic.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Nicholas Fuller
- Published: Jun 1, 2026
- Type: Explainer

Enterprise workflows are dynamic, long-running, API- and data-rich, and constrained by policies or regulations, creating challenges that an LLM alone may not handle efficiently. The article defines agent logic as software primitives—including knowledge graphs, algorithms, and program-analysis libraries—within an agent harness that steer models toward relevant workflow context. Examples span legacy COBOL and PL/1 understanding, automated test generation, incident investigation and code remediation, and compliance modernization; reported results include ~30× lower token consumption for application understanding, 20–45% higher test coverage, and up to 4.0× better incident-investigation performance than a ReAct baseline. A Condition Insights example reports 57% fewer unsupported claims and 77% lower average token usage, supporting the conclusion that structured guidance can improve quality, cost, and rule adherence.


### [Scaling up your microservice testing with Apache Spark](https://yomu.fyi/post/scaling-up-your-microservice-testing-with-apache-spark.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: Vivek Yadav
- Published: Jun 1, 2026
- Type: Explainer

Microservices whose behavior depends on long-tail production inputs—such as payment cost estimation—are difficult to validate with unit and integration tests alone. The described pattern separates deterministic service logic from transport and side effects, then invokes that logic in Apache Spark across historical records enriched with the required dependency state. The resulting datasets can compare current and candidate implementations, quantify changed outputs, model future rule changes, and identify affected transaction or merchant segments. Replay inputs and artifacts are minimized and protected through tokenized identifiers, redaction, access controls, and retention policies, while smaller privacy-safe golden datasets run in pull-request CI. The approach complements, rather than replaces, unit and integration tests: golden datasets cover known service behavior, and full replays expose changes across the real production distribution before deployment.


### [Beyond code generation: rethinking engineering productivity in the age of AI agents](https://yomu.fyi/post/beyond-code-generation-rethinking-engineering-productivity-in-the-age.md)
- Company: [Dropbox](https://yomu.fyi/company/dropbox.md)
- Author: Ilya Yakovlev,Andrew Cheung,Binoy Dash,Simran Jumani,Dmitriy Meyerzon,Mark Breitenbach,Ishan Mishra,Kazuaki Okumura
- Published: May 28, 2026

Dropbox shares how widespread AI code generation shifts software development bottlenecks downstream into code review, CI infrastructure, and validation pipelines. To adapt, they built Nova, an internal coding agent platform that safely automates scoped tasks such as migrations and flaky test remediation. They also evolved their developer productivity framework to measure end-to-end customer impact and code quality rather than simple pull request throughput.


### [Introducing Nova, our internal platform for coding agents](https://yomu.fyi/post/introducing-nova-our-internal-platform-for-coding-agents.md)
- Company: [Dropbox](https://yomu.fyi/company/dropbox.md)
- Author: Ilya Yakovlev,Andrew Cheung,Binoy Dash,Simran Jumani,Dmitriy Meyerzon,Mark Breitenbach,Ishan Mishra,Kazuaki Okumura,Mike White,Kevin Altschuler
- Published: May 21, 2026

Dropbox developed Nova, an internal platform that runs AI coding agents in isolated cloud environments integrated with their Bazel monorepo. The platform supports both interactive developer workflows and autonomous background tasks, such as automated CI debugging, flaky test remediation, and codebase-wide migrations. By pairing code generation with automated validation and strict execution guardrails, Nova ensures generated fixes are tested and reproducible.


### [You can't whisper at an AI agent](https://yomu.fyi/post/you-can-t-whisper-at-an-ai-agent.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: James Beswick
- Published: May 14, 2026
- Type: Explainer

Stripe engineers ran roughly a dozen experiments to determine how to get AI agents to use current API versions, follow integration best practices, and load the Stripe skill. Passive hints, including SDK files, AGENTS.md guidance, and API warning responses, were largely ignored, while active prompts and distribution mechanisms performed better. A modular, progressively disclosed skill structure outperformed a monolithic file by roughly 10% and reduced token usage; a CLI login prompt led 30–35% of viewers to copy its install command, and explicit compatibility-mode errors prompted agents to correct version mismatches. The results suggest that agent-facing developer experience is primarily a distribution problem, with hard steering through errors and loaded instructions more reliable than soft hints agents may never encounter.


### [An Engineer’s Guide to Better AI Skills: Implementing a Testing Process to Optimize Agent…](https://yomu.fyi/post/an-engineer-s-guide-to-better-ai-skills-implementing-a-testing-process.md)
- Company: [Pinterest](https://yomu.fyi/company/pinterest.md)
- Author: Pinterest Engineering
- Published: May 12, 2026
- Type: Problem & solution

Software engineers deploying AI agents often encounter unreliable skill invocation for repository-specific architectures, such as Pinterest's iOS framework. To quantify and resolve this issue, a Bash test harness piped positive and negative prompts into Pin-agent and Claude Code while parsing streamed JSON logs for execution signatures. Initial vanilla benchmarks yielded baseline accuracies of 73% for GPT 5.2-codex and 62% for Opus 4.5 Claude. Optimizations such as expanding YAML frontmatter descriptions with architectural context, utilizing capitalized imperative directives, and adding reference tables in AGENTS.md raised invocation rates. Both agents reliably loaded skills when prompts contained explicit invocations, demonstrating that empirical testing paired with descriptive prompting stabilizes agent skill adoption.


### [Upgrading M3DB from v1.1 to v1.5](https://yomu.fyi/post/upgrading-m3db-from-v1-1-to-v1-5.md)
- Company: [Uber](https://yomu.fyi/company/uber.md)
- Published: May 9, 2026
- Type: Problem & solution

Uber upgraded M3DB from v1.1 to v1.5 after three years on its previous open-source deployment, seeking a repeatable rollout process for a roughly 100-cluster, 5,500-node environment. The effort addressed scale-specific validation: offline benchmarks measured correctness, latency, success rates, bootstrap and tick durations, and persistence across restarts and replacements, while online shadow testing compared sampled production traffic between old and new versions. Within each cluster, upgrades were limited to one node at a time because writes used a two-thirds majority quorum. A rollout-related read-quorum incident exposed missing replica data; the cause was bootstrap misconfiguration, corrected by using “exclude\_commitlog,” since production workloads did not employ WAL. Production then showed significantly lower memory usage and improved node bootstrap times, supporting continued, data-driven upgrades and contributions to M3DB's benchmarking and documentation.


### [Bootstrapping Composer with autoinstall](https://yomu.fyi/post/bootstrapping-composer-with-autoinstall.md)
- Company: [Cursor](https://yomu.fyi/company/cursor.md)
- Author: Shomil Jain, Joshua Warner & Andrew Zhai
- Published: May 6, 2026
- Type: Explainer

RL training depends on runnable environments, because broken setup can consume tokens and compute without producing a reward signal or can make a task unsolvable. Composer autoinstall addresses this by using earlier Composer models to turn unconfigured repository checkouts into working mock environments. A Cursor agent first proposes 10 commands and expected outputs, while a separate Composer agent configures the checkout, mocks dependencies, and is evaluated against three selected commands. The process can create placeholder files, images, database tables, MinIO configurations, Docker containers, and startup scripts, restarting up to five times before discarding an environment. In a Celo monorepo experiment, the agent installed dependencies including Foundry and eventually mocked a user to run a minimal application; Composer 2 later scored 61.7% on Terminal-Bench versus 47.9% for Composer 1.5.


### [Evaluating AI at Scale: How Thumbtack Approaches Reliability, Safety, and Quality in GenAI](https://yomu.fyi/post/evaluating-ai-at-scale-how-thumbtack-approaches-reliability-safety-and.md)
- Company: [Thumbtack](https://yomu.fyi/company/thumbtack.md)
- Author: Thumbtack Engineering
- Published: Apr 29, 2026
- Type: Problem & solution

Thumbtack evaluates generative AI features across customer and service professional experiences to prevent unsupported claims, tone inconsistencies, and safety issues. The engineering organization transitioned from decentralized evaluation efforts to a dedicated cross-functional Evals team that provides shared infrastructure while preserving product team autonomy. System architectures combine rule-based checks, AI-as-a-judge scorers powered by DeepEval, trust and safety assessments, and crowdsourced human reviews. Teams leverage three parallel orchestration paths, including an MLflow-tracked stack, nightly Databricks batch jobs integrated with data warehouses, and multi-layer human-in-the-loop pipelines. Future milestones target centralized LLM judge registries, full task journey evaluations, automated judge-writing workflows, and cost-aware evaluation orchestration.


### [Leveraging AI to build a faster iOS CI system](https://yomu.fyi/post/leveraging-ai-to-build-a-faster-ios-ci-system.md)
- Company: [Thumbtack](https://yomu.fyi/company/thumbtack.md)
- Author: Muhieddine EL Kaissi
- Published: Apr 21, 2026
- Type: Problem & solution

iOS continuous integration at Thumbtack previously required 30 to 40 minutes per pull request because every change triggered all 400 or more tests for an affected app. To solve node exhaustion and prepare for increased AI-generated code volume, the infrastructure team built Smart Test Selection. The system relies on a nightly pipeline that executes the full test suite with coverage instrumentation to generate a JSON map connecting source files to test classes. When a pull request opens, the pipeline looks up changed files in this map and runs only the relevant tests instead of the full suite. Built using Groovy, Ruby, and Bash, the tool reduces typical continuous integration time by approximately 50 percent while cutting documentation-only validation by about 90 percent.


### [The PR you would have opened yourself](https://yomu.fyi/post/the-pr-you-would-have-opened-yourself.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Pedro Cuenca, Awni Hannun
- Published: Apr 16, 2026
- Type: Announcement

mlx-lm contributors can use a Skill and a separate test harness to port language models from transformers, with the goal of making new architectures available shortly after their upstream implementations land. The work addresses rising agent-generated pull-request volume and the difficulty of preserving implicit codebase conventions, user contracts, and performance-sensitive behavior that agents may not understand. Given a conversion prompt, the Skill sets up environments, finds model variants and checkpoints, reads transformers modeling code, writes an MLX implementation, and iterates through tests including dtype checks, generation examples, numerical comparisons, and per-layer comparisons. The resulting pull requests disclose agent assistance and include detailed verification data, while contributors remain responsible for reviewing and accepting the results. Current limitations include shared-utility refactors, vision-language models, quantized uploads, and thinking-specific tests.


### [Getting started with clickhousectl: the ClickHouse CLI for local and cloud](https://yomu.fyi/post/getting-started-with-clickhousectl-the-clickhouse-cli-for-local-and-cl.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: Al Brown
- Published: Apr 16, 2026
- Type: Tutorial

ClickHouse developers often face friction managing local versions, isolated instances, project scaffolding, and Cloud operations without relying on complex shell scripts. The open-source command-line tool clickhousectl addresses this by unifying local environment management and ClickHouse Cloud workflows. Locally, it downloads shared binaries, initializes standardized directory structures, and spins up named, isolated servers with automatic port assignments to facilitate upgrade testing. For cloud workflows, the CLI enforces a split authentication model where browser OAuth provides read-only access and API keys permit state mutations like service creation and scaling. Furthermore, the tool includes Claude Code agent skills covering best practices and deployment workflows, alongside support for CI/CD pipeline automation via environment variables and JSON output.


### [Inside VAKRA: Reasoning, Tool Use, and Failure Modes of Agents](https://yomu.fyi/post/inside-vakra-reasoning-tool-use-and-failure-modes-of-agents.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Ankita Naik, danish, Ben, Anupama Murthi, Praveen Venkateswaran, Siyu, Ayhan Sebin
- Published: Apr 15, 2026
- Type: Benchmark

VAKRA is an executable, tool-grounded benchmark for evaluating whether AI agents can complete enterprise-like workflows that combine API calls, document retrieval, and natural-language constraints. It provides more than 8,000 locally hosted APIs backed by real databases across 62 domains, plus domain-aligned documents, and uses full execution traces to assess multi-step behavior. The four capability groups cover API chaining, dashboard-API tool selection, multi-hop dashboard reasoning, and multi-hop multi-source reasoning with multi-turn dialogue and optional tool-use policies; tasks can require three to seven reasoning steps, while some segments specify one to twelve tool calls or up to five logical hops. Reported analyses find that models perform best on single-hop questions, degrade as API hops increase, and often violate policies or fail to retrieve enough information when constraints restrict the relevant source, exposing a gap between isolated tool competence and end-to-end reliability.


### [Autoresearch isn’t just for training models](https://yomu.fyi/post/autoresearch-isn-t-just-for-training-models.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Apr 15, 2026
- Type: Problem & solution

Repeated visual-regression failures made Polaris CI feedback cycles take about 30 minutes, prompting an attempt to optimize build time rather than fix individual tasks. The approach uses Pi to create an Autoresearch extension that repeatedly forms a hypothesis, measures the build against a 19.1-second baseline, and keeps only changes that improve the metric, discarding crashes and slower runs. The loop identified wasted VRT work: the full component pipeline ran before Storybook recompilation, while TypeScript transformed all 580 component files although 105 needed it. Removing those inefficiencies made the build 65% faster, although the system also produced unacceptable hacks that were discarded. The extension later gained multi-metric support, consistent iteration scripts, auto commits, and open-source adoption, with reported internal improvements including unit tests running 300 times faster and Playwright tests becoming faster.


### [AI Agents Know About Supabase. They Don't Always Use It Right.](https://yomu.fyi/post/ai-agents-know-about-supabase-they-don-t-always-use-it-right.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: pedro\_rodrigues
- Published: Apr 9, 2026
- Type: Announcement

Supabase is releasing Supabase Agent Skills, an open-source instruction set for AI coding agents working with Supabase through Claude Code, Codex, GitHub Copilot, Cursor, and compatible Agent Skills implementations. The skill addresses recurring errors such as omitted RLS policies, hallucinated CLI commands, views missing security\_invoker = true, and reliance on outdated training data by requiring documentation checks and embedding critical security guidance in SKILL.md. It also guides CLI and MCP workflows and recommends direct schema changes on local or staging databases, followed by database advisors and a migration after the schema stabilizes. Early evaluations across six scenarios found MCP + Skill scores of 67% for Claude Opus 4.6, 71% for Claude Sonnet 4.6, 88% for Codex GPT-5.4, and 71% for GPT-5.4 Mini, though the sample was small.


### [Selective Test Execution at Stripe: Fast CI for a 50M-line Ruby monorepo](https://yomu.fyi/post/selective-test-execution-at-stripe-fast-ci-for-a-50m-line-ruby-monorep.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: Aditya Anchuri
- Published: Apr 9, 2026
- Type: Problem & solution

Stripe’s Ruby monorepo contains about 50 million lines, 100,000 test files, and roughly 1.2 million test units, making sequential execution impractical at an estimated four months. To keep CI within minutes, Selective Test Execution (STE) runs tests whose recorded file accesses intersect files changed since a suitable baseline, averaging about 5% of the suite and under 10% of always-run compute. A dynamically linked C++ library loaded with LD\_PRELOAD intercepts file opens, associates them with hierarchical test scopes, and propagates tracing to child processes; logs are aggregated into a selection index outside the syscall hot path. Guardrails cover tests affected by directory discovery, previously failing tests, selective linting, generated files, and reproducible database-backed baselines ordered by Monotonic Revision IDs. The system is presented as a way to preserve safety and affordability while scaling CI for a highly dynamic language.


### [Headless Screenshot Testing in Thumbtack’s Android Apps](https://yomu.fyi/post/headless-screenshot-testing-in-thumbtack-s-android-apps.md)
- Company: [Thumbtack](https://yomu.fyi/company/thumbtack.md)
- Author: Brian Terczynski
- Published: Mar 31, 2026
- Type: Problem & solution

Thumbtack required a faster, emulator-free method to execute over 1,500 Android screenshot tests during continuous integration. Previously, running UI tests on physical devices or emulators proved too slow for code review verification jobs, causing tests to run post-merge and allowing regressions to persist. After evaluating Paparazzi and Google's experimental Compose Preview Testing, the team selected Roborazzi, which runs on top of Robolectric's native graphics engine within a standard JVM. This headless setup accommodates custom Activities, Dagger field injection, and legacy View systems while executing alongside standard unit tests. Moving screenshot tests to the unit test level enabled gating merges on passing tests, colocating tests inside feature modules, and increasing total CI job runtime by only twenty percent.


### [A New Framework for Evaluating Voice Agents (EVA)](https://yomu.fyi/post/a-new-framework-for-evaluating-voice-agents-eva.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Tara Bogavelli, Gabrielle Gauthier Melancon, Katrina Stankiewicz, Nifemi Bamgbose, Hoang Nguyen, Raghav Mehndiratta, Hari Subramani, Fanny Riols
- Published: Mar 24, 2026
- Type: Benchmark

EVA is an end-to-end framework for evaluating conversational voice agents across task accuracy and spoken conversational experience. It simulates multi-turn live-audio conversations with a user simulator, the evaluated voice agent, deterministic tool execution, validation, and metrics derived from recordings, transcripts, and tool-call logs. The framework reports EVA-A for Accuracy and EVA-X for Experience, combines code-based and LLM- or LALM-as-Judge metrics, and uses pass@3 and pass^3 across three trials to capture peak performance and consistency. Its initial release contains 50 reproducible English-language airline scenarios and 15 tools covering rebooking, itinerary changes, cancellations, standby, and vouchers. Results for 20 cascade and audio-native systems show a consistent tradeoff: agents stronger at task completion tend to provide worse experiences, while a substantial gap between pass@3 and pass^3 indicates inconsistent success.


[Newer posts](https://yomu.fyi/topic/testing/page/5.md) · [Older posts](https://yomu.fyi/topic/testing/page/7.md)