Agentic Testing: Where Agents Fit in the E2E Testing Stack

Summary

Traditional end-to-end tests validate rigid user journeys, whereas agentic tests verify whether broad goals can be achieved by adapting actions dynamically. To evaluate agentic testing tradeoffs, researchers executed over 200 runs across Playwright Model Context Protocol (MCP), Playwright CLI, and agent-generated Playwright tests using Claude models. Playwright MCP demonstrated high reliability with failure rates of 0% on simple thread replies and approximately 12% on complex search discovery flows. Playwright CLI and generated code struggled more on complex workflows, exhibiting failure rates of approximately 20% and 48% respectively. Although generated tests were faster with average runtimes of roughly three minutes, agentic testing provides a distinct exploratory layer atop deterministic CI test suites.

Takeaways

1. 1

   Playwright MCP achieved failure rates of 0% on simple workflows and roughly 12% on complex workflows, outperforming Playwright CLI which failed 12% to 20% of the time due to execution and session instability.

2. 2

   Generated Playwright tests executed fastest with an average duration of approximately 3 minutes, but their failure rate spiked to around 48% on complex workflows due to UI state variability and abstraction mismatches.

3. 3

   Agentic testing functions as an exploratory and debugging layer atop the testing pyramid rather than replacing fast, low-cost deterministic end-to-end tests in CI.