---
title: "Testing"
description: "271 posts about Testing, summarised, each linking to the original."
---

# Testing
> 271 posts about Testing, summarised, each linking to the original.

## Articles

### [A new benchmark for evaluating patient-facing health AI agents](https://yomu.fyi/post/a-new-benchmark-for-evaluating-patient-facing-health-ai-agents.md)
- Company: amazon.science
- Author: Korosh Vatanparvar, Ashutosh Joshi
- Published: Jul 29, 2026
- Type: Benchmark

Healthcare AI benchmarks often rely on static medical knowledge questions or clinician-facing tool evaluations, failing to test multiturn patient interactions and clinical workflow execution. To address this gap, PatientAgentBench introduces an automated evaluation pipeline using synthetic patient charts, dual-agent conversations, and an LLM-as-a-jury panel. The evaluation framework applies over one hundred clinician-vetted criteria across six dimensions, including clinical safety, triage quality, and workflow accuracy. Testing frontier models revealed a severity paradox where agents scored higher on obvious emergencies than on routine administrative requests harboring hidden risks. Common safety failures included omitting crisis hotline resources and fabricating clinical information, showing that raw foundation model capability alone cannot guarantee safe patient-facing execution.


### [Building an agentic harness that outlasts the model](https://yomu.fyi/post/building-an-agentic-harness-that-outlasts-the-model.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jul 29, 2026
- Type: Explainer

Shopify’s Application Security team built an agentic harness that scans software for vulnerabilities, proves candidates with real tests, and opens Shopify-specific remediation pull requests. Its Dispatch orchestrator partitions repositories, runs category-specific Hunting agents in parallel, verifies findings sequentially with a different model, and persists architecture artifacts for later diff-based scans. Test oracles require realistic integration or functional coverage, including cross-tenant fixtures for IDOR checks, while unproven candidates are rejected or downgraded. Across more than 80 applications and thousands of scans, the workflow produced over 300 findings, including two rated Critical by Shopify’s severity calculator, with full scans costing $50–$300 and incremental scans $5–$50. The post concludes that durable value lies in the harness’s verification, partitioning, cross-model review, and deterministic infrastructure rather than in any single model.


### [Eval-driven development: Lessons from evaluating GenAI at scale](https://yomu.fyi/post/eval-driven-development-lessons-from-evaluating-genai-at-scale.md)
- Company: [Airbnb](https://yomu.fyi/company/airbnb.md)
- Author: Rohit Girme
- Published: Jul 28, 2026

Airbnb outlines its framework for eval-driven development (EDD), treating GenAI evaluation as a continuous engineering discipline rather than an afterthought. The strategy combines programmatic checks, calibrated LLM-as-a-judge evaluators, and human review to detect subtle quality regressions. By inspecting intermediate agentic execution traces and real-world failure modes, teams ensure generative models remain reliable and aligned with product requirements.


### [How Vercel used Cursor to build Queues](https://yomu.fyi/post/how-vercel-used-cursor-to-build-queues.md)
- Company: [Cursor](https://yomu.fyi/company/cursor.md)
- Author: Cursor
- Published: Jul 28, 2026
- Type: Explainer

Vercel built its durable event streaming system, Vercel Queues, entirely with Cursor, using agents across three rewrites and validating changes against real AWS environments. Engineers had agents work on the Queues API, connect to AWS, inspect DynamoDB and Kinesis side effects, inject faults, simulate bugs, and verify results, while Cursor retained context through long sessions and compaction events. The workflow also supported model switching, parallel planning agents, inline review, stacked PRs, and asynchronous Cloud Agents. Among Vercel’s Cursor power users, PR throughput increased 54%, PR cycle time decreased 89%, resources per feature fell 27%, and the company reports 104 hours saved per developer annually. Product managers also use Cursor for repository analysis, product analytics, and small fixes without drawing on engineering resources.


### [NVIDIA Cosmos-H-Dreams: Bringing Real-Time Generative Simulation to Surgical Robotics](https://yomu.fyi/post/nvidia-cosmos-h-dreams-bringing-real-time-generative-simulation-to-sur.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Lukas Zbinden, Javier Gamazo, Mostafa Toloui, Sean Huver
- Published: Jul 27, 2026
- Type: Announcement

Surgical robotics needs safer, faster ways to train and evaluate vision-language-action policies because physical experiments are costly, slow to reproduce, and vulnerable to damaging instruments or biological material. NVIDIA introduces Cosmos-H-Dreams, a real-time, action-conditioned generative simulator that specializes Cosmos-H-Surgical-Simulator for da Vinci Research Kit tabletop suturing and generates future frames from an initial RGB image and live robot kinematics. Its causal student is trained through teacher fine-tuning, causal warmup, and self-forcing distillation, including failure episodes and long autoregressive horizons, while few-step diffusion reduces generation cost. Served through FlashDreams with streaming KV caching, CUDA Graph capture, and model compilation, it reaches approximately 160 frames per second on one NVIDIA RTX PRO 6000, compared with roughly 10 frames per second for standard inference. The system supports human or learned-policy control, but is presented as a research and development platform rather than a diagnostic system or physical-robot controller.


### [NVIDIA Nemotron 3 Ultra Leads Open Models on Accuracy and Efficiency in Agentic RTL Coding](https://yomu.fyi/post/nvidia-nemotron-3-ultra-leads-open-models-on-accuracy-and-efficiency-i.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Nirmal Kumar Juluru
- Published: Jul 27, 2026
- Type: Benchmark

Register transfer level (RTL) development relies heavily on iterative verification and tool feedback, which poses challenges for standard code generation models. To address realistic hardware design tasks, the ACE-RTL agent pairs an iterative generate-test-reflect workflow with NVIDIA Nemotron 3 Ultra. The model utilizes a hybrid Mamba-Attention Mixture-of-Experts architecture trained on a synthetic dataset encompassing code generation, editing, and error debugging. Across the comprehensive Verilog design problems benchmark, ACE-RTL with Nemotron 3 Ultra attained a 97.1% average pass rate, outperforming GLM 5.2 and Kimi K2.6. Additionally, Nemotron 3 Ultra achieved its accuracy while consuming an average of 6,629 tokens per iteration, which is substantially lower than competing models.


### [Amazon is investing in the Lean Focused Research Organization](https://yomu.fyi/post/amazon-is-investing-in-the-lean-focused-research-organization.md)
- Company: amazon.science
- Author: Byron Cook, Shawn Bice
- Published: Jul 26, 2026
- Type: Announcement

As AI agents handle high-stakes operational and financial decisions, traditional software testing fails to provide exhaustive correctness guarantees across all possible inputs. Mathematical proof offers deterministic verification that systems cannot behave incorrectly, but requires accessible tooling to function at modern software scale. To address this, Amazon announced substantial long-term financial backing for the Lean Focused Research Organization, representing the largest donation in the organization's history. Internally, Amazon relies on Lean for verifying differential-privacy mechanisms in AWS Clean Rooms, hardware compilation in AWS Neuron, and policy constraints within Amazon Bedrock AgentCore. Developing Lean externally through an open-source research organization provides independent auditability for regulators and accelerates community-driven tooling for neurosymbolic AI safety.


### [Debugging Ray Tracing Applications Using NVIDIA OptiX Toolkit](https://yomu.fyi/post/debugging-ray-tracing-applications-using-nvidia-optix-toolkit.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Tanya Lenz
- Published: Jul 23, 2026
- Type: Problem & solution

NVIDIA OptiX ray tracing applications often present difficult debugging challenges, including invalid API arguments, black frames, and concurrency issues across thousands of GPU threads. The open-source NVIDIA OptiX Toolkit addresses these diagnostic hurdles by offering unified error checking and targeted device-side debug logging. Host-side macros like OTK\_ERROR\_CHECK wrap OptiX, CUDA runtime, and CUDA driver API calls to enforce uniform error validation while capturing file and line diagnostics. For device-side troubleshooting, the toolkit introduces a DebugLocation utility to isolate logging to specific launch indices instead of flooding the output stream with concurrent thread data. This device mechanism also supports interactive one-shot debug dumps and screen highlights around the inspected pixel.


### [Coding Agent Horror Stories: The Agent That Deleted Production](https://yomu.fyi/post/coding-agent-horror-stories-the-agent-that-deleted-production.md)
- Company: [Docker](https://yomu.fyi/company/docker.md)
- Author: Jennifer Kohl
- Published: Jul 20, 2026
- Type: Announcement

Recent industry updates focus on AI agent security, container maintenance, and developer workflows within Docker ecosystems. The OpenAI and Hugging Face incident involved 17,600 attacker actions, demonstrating that AI agent security requires automated system controls to constrain, observe, and govern agents rather than relying on human review. In software maintenance, Docker Extended Lifecycle Support (ELS) addresses escalating supply-chain risks and MinIO end of life by maintaining patches, building software from source, and enforcing local developer policies. For continuous integration, Docker Sandboxes enable isolated AI agents to run Testcontainers tests, implement code fixes, and submit draft pull requests within GitHub Actions. Additionally, Docker Verified Publisher applications have transitioned to a self-serve process accessible directly through Docker Hub.


### [DE Dost: Teaching Machines to Be a Friend on the Last Mile](https://yomu.fyi/post/de-dost-teaching-machines-to-be-a-friend-on-the-last-mile.md)
- Company: [Swiggy](https://yomu.fyi/company/swiggy.md)
- Author: Aishwary Yadav
- Published: Jul 17, 2026
- Type: Explainer

DE Dost is a GenAI onboarding companion embedded in Swiggy’s delivery-executive app, designed for new DEs who receive information but lack confidence-building, immediate guidance. It supports questions in nine languages across SOPs, earnings, payouts, app navigation, shifts, and support, adapting detail and tone to conversational context. Before the first LLM call, a graph state assembles user context, preprocessing resolves references, and embedding-based semantic routing separates general knowledge-base questions from personal diagnostics. Retrieval uses authored QnA pairs plus hybrid semantic and lexical scoring to handle paraphrases and terms such as “MG” and “COD,” while guardrails constrain follow-ups to available knowledge. Evaluation spans about 800 cases per language across fleet and payout combinations, with macro accuracy trending at 90%+ and a target P50 latency of 4.5 seconds after reducing an initial 15-second pilot.


### [Build a Secure C# MCP App with Cross App Access (XAA)](https://yomu.fyi/post/build-a-secure-c-mcp-app-with-cross-app-access-xaa.md)
- Company: [Okta](https://yomu.fyi/company/okta.md)
- Author: Aasawari Sahasrabuddhe
- Published: Jul 16, 2026
- Type: Tutorial

Cross App Access (XAA) addresses the gap between a user’s authenticated identity and a downstream MCP server, API, or agent tool that must trust that identity without another login. The described flow uses OIDC and PKCE for sign-in, then RFC 8693 token exchange to turn the ID token into a JWT Authorization Grant (JAG), followed by RFC 7523 exchange at the MCP authorization server for a scoped bearer token. A C# MCP SDK IdentityAssertionGrantProvider encapsulates both exchanges, while ASP.NET Core OIDC middleware persists the ID token and an HTTP transport presents the resulting token to the MCP server. The sample connects to a Todo resource through the xaa.dev playground, completes MCP initialization, lists resources, and reads todo0://todos; the article states that the end-to-end implementation fits in under 50 lines of C#.


### [Upgrading Checkout Blocks app to Polaris web components](https://yomu.fyi/post/upgrading-checkout-blocks-app-to-polaris-web-components.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jul 16, 2026
- Type: Problem & solution

Checkout Blocks upgraded all five UI extensions from React and Shopify’s legacy Remote UI bridge to remote-dom with Preact and Polaris web components, while moving from JavaScript to TypeScript and API version 2026-01. The migration addressed checkout’s performance sensitivity and Shopify’s requirement that deployments move off versions earlier than 2026-01. A parallel core-next library allowed extensions to transition incrementally, with shared providers, telemetry, hooks, helpers, and rules adapted before legacy core was removed. To meet the remote-dom CLI’s 64KB gzip limit, the team dropped react-reconciler, replaced liquidjs with a smaller parser validated against more than 42,000 lines of merchant configurations, replaced dayjs, and retained markdown-to-jsx through Preact compatibility. Transferred bundles fell 40% to 85%, while weighted ELT improved 8% at P50 and 7% at P90.


### [Introducing Real World VoiceEQ: Measuring the human quality of voice AI](https://yomu.fyi/post/introducing-real-world-voiceeq-measuring-the-human-quality-of-voice-ai.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: David Ayllon, Alice, Jeff Brooks, Franc Camps Febrer, Jakub Piotr Cłapa, Theo Lebryk, Jens Madsen, Olya Ossipova, Sharath Rao, Hoon Shin, Tigran, Rashish Tandon, Panagiotis Tzirakis
- Published: Jul 15, 2026
- Type: Announcement

Existing voice-AI benchmarks increasingly suggest near-human performance, while real-world conversations still expose failures involving speaker consistency, hesitation, accents, noise, emotion, and natural responses. Real World VoiceEQ measures human quality in voice interaction by testing whether systems recognize, produce, and respond to acoustic information omitted by transcripts, including tone, emotion, speaker identity, and background context. It evaluates more than 40 proprietary and open-source models across 15+ dimensions and 60+ metrics covering ASR, TTS, S2S, and Speech Understanding, using more than 1 million human ratings, including 785,000 TTS and 48,000 STS ratings. Findings indicate that capabilities are specialized, speech-to-speech systems often underuse paralinguistic cues, and traditional benchmarks can overestimate performance in noise and other conversational conditions. The research also reports that human agreement remains essential for subjective acoustic and social judgments because SLM evaluators were less reliable outside clear, verifiable tasks.


### [From weeks to a day: how we made LLM evaluation fast enough to iterate on](https://yomu.fyi/post/from-weeks-to-a-day-how-we-made-llm-evaluation-fast-enough-to-iterate.md)
- Company: [Airbnb](https://yomu.fyi/company/airbnb.md)
- Author: Baharak Saberidokht
- Published: Jul 14, 2026

Airbnb built a four-layer LLM infrastructure framework to reduce iteration and evaluation turnaround from weeks to a single day. By using per-sample caching for generated references and judge scores, they established a deterministic evaluation foundation that separates genuine model drift from measurement noise. This setup enables rapid, bounded model hotfixes using micro LoRA adapters and comprehensive end-to-end validation across component boundaries.


### [How to Evaluate General-Purpose Robot Policies for Real-World Deployment](https://yomu.fyi/post/how-to-evaluate-general-purpose-robot-policies-for-real-world-deployme.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Brad Nemire
- Published: Jul 12, 2026
- Type: Problem & solution

Evaluating robotics foundation models accurately in simulation remains challenging due to visual domain overlap, benchmark saturation, diagnostic gaps, and small sample sizes. RoboLab addresses these deficiencies by providing a robot-agnostic simulation platform that supports rapid task creation and diagnostic evaluation suites. Tasks in RoboLab can be compiled across varying robot embodiments and target visual, procedural, and relational competencies. Statistical analysis using the Clopper-Pearson method reveals that narrowing the confidence interval around a 90 percent success rate from ten to two percentage points demands increasing rollouts from 70 to 1,030. Developed by NVIDIA Research, RoboLab powers the open-source Isaac Lab-Arena simulation framework to establish scalable evaluation for real-world robotic policies.


### [Create a LangChain Deep Agents Harness Profile for NVIDIA Nemotron 3 Ultra to Improve Performance](https://yomu.fyi/post/create-a-langchain-deep-agents-harness-profile-for-nvidia-nemotron-3-u.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Sean Lopp
- Published: Jul 8, 2026
- Type: Tutorial

The tutorial shows how to create and validate a LangChain Deep Agents harness profile for NVIDIA Nemotron 3 Ultra using an evaluation benchmark rather than fine-tuning the model. It starts with a baseline, analyzes failed trajectories, and iterates on prompts, exclusions, or added middleware while rerunning tests to detect regressions. One failure occurred when read\_file returned the first page of a large file without signaling that more content existed, so the model answered before paginating to the line containing opal-fox-91. The proposed ReadFileContinuationNoticeMiddleware appends a continuation notice when a full page is returned, raising the reported benchmark score from 94/127 to 96/127 and resolving all three read\_file tests. The source also describes automating this propose, verify, and retain-or-reject loop, with safeguards against overfitting and regressions.


### [How do you turn AI coding chaos into a repeatable playbook?](https://yomu.fyi/post/how-do-you-turn-ai-coding-chaos-into-a-repeatable-playbook.md)
- Company: [Stack Overflow](https://yomu.fyi/company/stack-overflow.md)
- Author: Eira May
- Published: Jul 2, 2026
- Type: Explainer

Software development dynamics are shifting as code generation costs decrease, requiring engineering organizations to adapt their leadership and processes. Snowflake addressed this transformation across its software lifecycle by first encouraging widespread, unrestricted experimentation with coding agents before formalizing successful practices. The engineering organization codified these practices into fourteen distinct AI design patterns, including strategies such as planning in markdown and isolating parallel agents. This structured approach, combined with dedicated focus weeks and a progress scale, allowed engineering teams to accelerate both inner and outer loops of development. Consequently, a team of three engineers leveraged coding agents to deliver a 40x improvement to Snowflake's query compiler while significantly shortening release validation times.


### [Keeping funds where they belong: Implementing fund segregation in Stripe Connect](https://yomu.fyi/post/keeping-funds-where-they-belong-implementing-fund-segregation-in-strip.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: Andrew Robinson
- Published: Jul 2, 2026
- Type: Explainer

Marketplaces using Stripe Connect’s separate charges and transfers can pool seller funds with platform revenue, exposing them to payouts, unrelated disputes, fees, refunds, and operational mistakes. In response to anticipated PSD3 changes and tighter Commercial Agent Exemptions, Stripe introduced Segregated Separate Charges and Transfers, which places captured payment funds in an allocated balance that cannot fund payouts and can only be transferred to connected accounts. The integration enables allocated\_funds on PaymentIntents, transfer\_group tracking, source\_transaction-linked transfers, and application fees collected from the same allocation; balances are inspected through the API. The post also details split transfers, refund and reversal ordering, dispute handling, prerequisites, and limitations including card-only support, preview-header requirements, and sandbox-only testing, presenting segregation as a way to make fund flows traceable and operationally safer.


### [Mastering Agentic Techniques: AI Agent Reinforcement Learning](https://yomu.fyi/post/mastering-agentic-techniques-ai-agent-reinforcement-learning.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Elizabeth Goodman
- Published: Jul 1, 2026
- Type: Tutorial

Language model agents often fail across long-horizon workflows, repeat tool-call errors, and output invalid schemas when relying solely on prompting or retrieval. Reinforcement learning with verifiable rewards provides a practical training signal for domain-specific tasks by scoring model rollouts against deterministic verifiers such as schema checkers, unit tests, and execution sandboxes. Teams can implement group relative policy optimization to sample multiple trajectories per prompt and update policy weights based on relative performance within the group. Establishing environments with explicit harness, state, and verifier definitions allows agents to learn multi-step tool use, whereas static datasets fail to capture sequential dependencies. Inspecting checkpoint rollouts and testing against held-out tasks prevents reward hacking while continuously turning operational failures into verifiable benchmark environments.


### [Reading the agent traces is how you make the call your eval can't](https://yomu.fyi/post/reading-the-agent-traces-is-how-you-make-the-call-your-eval-can-t.md)
- Company: [Sentry](https://yomu.fyi/company/sentry.md)
- Author: Sergiy Dybskiy
- Published: Jul 1, 2026
- Type: Explainer

Automated evaluations often miss unknown failure modes in AI agents because developers only write tests for anticipated errors. While building an itinerary agent for AI Engineer World's Fair, an open-weight model routed a speaker query to a tool returning only tracks, leading the model to hallucinate famous speakers and falsely claim the tool provided that data. Because a tool was executed, the trace appeared grounded until inspectable tool outputs revealed the discrepancy. To resolve this, the author retained the inexpensive model, corrected query routing, tightened system prompts against ungrounded entities, and added a custom entity-diffing eval in CI. Ultimately, directly reading agent telemetry traces alongside errors and logs is essential for diagnosing root causes and deciding trade-offs between model costs, prompt adjustments, and routing fixes.


[Newer posts](https://yomu.fyi/topic/testing/page/3.md) · [Older posts](https://yomu.fyi/topic/testing/page/5.md)