Agentic Testing: Where Agents Fit in the E2E Testing Stack

Summary

Traditional end-to-end tests validate rigid user journeys, whereas agentic tests verify whether broad goals can be achieved by adapting actions dynamically. To evaluate agentic testing tradeoffs, researchers executed over 200 runs across Playwright Model Context Protocol (MCP), Playwright CLI, and agent-generated Playwright tests using Claude models. Playwright MCP demonstrated high reliability with failure rates of 0% on simple thread replies and approximately 12% on complex search discovery flows. Playwright CLI and generated code struggled more on complex workflows, exhibiting failure rates of approximately 20% and 48% respectively. Although generated tests were faster with average runtimes of roughly three minutes, agentic testing provides a distinct exploratory layer atop deterministic CI test suites.

Context

Traditional end-to-end testing enforces specific deterministic UI paths, making it unclear whether high-cost, multi-minute agentic tests that verify goals dynamically can practically fit into modern testing stacks.

Approach / What changed

Ran over 200 automated executions across simple and complex workflows to compare three execution models: an agent using Playwright MCP, an agent using Playwright CLI, and agent-generated Playwright code using Claude models.

Takeaways

• Playwright MCP achieved failure rates of 0% on simple workflows and roughly 12% on complex workflows, outperforming Playwright CLI which failed 12% to 20% of the time due to execution and session instability.
• Generated Playwright tests executed fastest with an average duration of approximately 3 minutes, but their failure rate spiked to around 48% on complex workflows due to UI state variability and abstraction mismatches.
• Agentic testing functions as an exploratory and debugging layer atop the testing pyramid rather than replacing fast, low-cost deterministic end-to-end tests in CI.

Related reading

• Grab · Dec 23, 2019

  Marionette - Enabling E2E User-scenario Simulation

  Conducting end-to-end testing across Grab's transport microservices became difficult due to service availability, environment construction, cross-service authentication, and complex data setups for real-world user accounts. To address these challenges without relying on physical mobile devices or emulators, Grab built Marionette, an internal simulation platform for passenger and driver interactions. The platform provisions required test data, coordinates booking lifecycles, and isolates test executions across distinct user groups using localized cohorts. Engineers can configure driver and passenger behaviors, execute workflows, and run load or integration tests through a dedicated user interface, a Go SDK, and RESTful APIs.

  Anish Jha

• Grab · Mar 31, 2023

  Evolution of quality at Grab

  As Grab expanded its superapp, manual QA sign-offs led to prolonged testing cycles and a peak in major production defects around 2019. To counter defect leakage and improve delivery pace, the engineering organization transitioned to a shift-left testing strategy across its software development lifecycle. Quality engineers began writing Given/When/Then acceptance tests prior to coding so that software engineers could execute them during development, accompanied by Definition of Ready and Definition of Done standards. The team also structured a multi-layer test strategy spanning unit tests, pipeline-executed UI component tests with mocked APIs, backend integration tests, and a custom internal end-to-end framework. Between 2019 and 2022, these changes reduced major and critical production issues by 60% and development-phase critical bugs by 40%.

  Abby Alcantara

• Dropbox · May 28, 2026

  Beyond code generation: rethinking engineering productivity in the age of AI agents

  Dropbox shares how widespread AI code generation shifts software development bottlenecks downstream into code review, CI infrastructure, and validation pipelines. To adapt, they built Nova, an internal coding agent platform that safely automates scoped tasks such as migrations and flaky test remediation. They also evolved their developer productivity framework to measure end-to-end customer impact and code quality rather than simple pull request throughput.

  Ilya Yakovlev,Andrew Cheung,Binoy Dash,Simran Jumani,Dmitriy Meyerzon,Mark Breitenbach,Ishan Mishra,Kazuaki Okumura

• Grab · Jun 22, 2026

  Scaling out Distroless adoption With AI

  Grab is transitioning its microservices to Distroless base images to eliminate unnecessary binaries and reduce vulnerability risks, but the migration risks runtime failures from missing shared objects and system utilities. To safely validate container execution in continuous integration without staging dependencies, the team relied on medium tests that run containerized services alongside internal dependencies managed by Testcontainers. Because hundreds of services lacked this test harness, Grab implemented an agentic workflow using Claude Code and Model Context Protocol integrations to inspect repositories, generate test boilerplate, and resolve configuration errors. Once test baselines are established, an automated patch-test-compare pipeline updates Dockerfiles, constructs multi-stage builds for necessary dynamic libraries, and creates draft merge requests for human approval.

  Jia Yee Chong