Agentic testing shifts E2E focus from rigid journeys to goal-based verification. While too slow and costly for every PR, it provides a powerful exploratory layer that adapts to UI changes and handles complex state transitions where traditional deterministic scripts often fail.
Agent-driven end-to-end (E2E) tests add a new exploratory layer to testing, but should they replace traditional deterministic tests? We ran more than 200 agentic E2E workflows using the Playwright MCP, Playwright CLI, and agent-generated Playwright tests in test workspaces using non-production data to find out how agentic testing could fit into both our and your testing stacks.
Traditional end-to-end tests validate a specific journey through the UI.
click → click → type → assert
Agent-driven tests instead validate whether a goal can be achieved, often expressed as an instruction (e.g. “send a thread message”):
goal → agent adapts → verify result
This difference can be summarized simply:
Tests enforce journeys. Agents verify goals.
Across our agentic test runs, the overall workflow remained consistent (e.g. login → search → result → clear), but the exact sequence of actions varied. In practice, agents took different paths to reach the same outcome:
Agents can still validate intermediate steps when needed, but this flexibility comes with tradeoffs in reliability, cost, and execution time, which we explore in the next sections.
Agent-driven E2E testing looks promising, but it raises a real question: can something that costs $15–30 per run and takes over 10 minutes actually fit into modern testing workflows?
At first glance, the answer seems like no. But in 200+ runs, we found they are fundamentally different from traditional tests. They can be highly reliable and have a clear place in the testing stack.
This is largely due to recent advances in large language models, which enable agents to write code, debug failures, and interact directly with user interfaces. These capabilities introduce a new execution model for testing, but where they fit in existing E2E workflows is not always clear.
To understand how agent-driven tests can fit into E2E workflows, we ran 200+ automated executions across multiple configurations to measure reliability, execution speed, and cost.
We evaluated three different approaches:
claude -p)We used two flows to cover different levels of complexity. These flows were kept consistent across all experiments to allow for direct comparison.
For agent-driven approaches, we evaluated two input types:
The difference is not the level of detail, but how that detail is represented: natural language requires the agent to interpret and map instructions to actions, while YAML defines that mapping more explicitly.
Each configuration was run 20 times. The experiment matrix below shows the full setup:
| Exp | Execution Model | Input Type | Tools | Thread Reply | Search Discovery |
| 1 | Agent (Playwright MCP) | NL | MCP | 20 | 20 |
| 2 | Agent (Playwright MCP) | YAML | MCP | 20 | 20 |
| 3 | Agent (Playwright CLI) | NL | CLI | 20 | 20 |
| 4 | Agent (Playwright CLI) | YAML | CLI | 20 | 20 |
| 5 | Agent (Generated Tests) | NL | Code | 20 | 20 |
Before diving into individual metrics, here’s a quick look at how the different approaches performed overall across both natural language and YAML-based executions.
| Approach |
Failure rate (thread reply) |
Failure rate (search discovery) |
Avg runtime |
| Agent (Playwright MCP) | 0% | ~12% | ~5–8 min |
| Agent (Playwright CLI) | ~12% | ~20% | ~9–11 min |
| Generated Playwright Tests | ~8% | ~48% | ~3 min |
The following sections break down these results by individual metrics.
One of the clearest patterns we saw was how reliability changed as flows became more complex.
Across the agentic Playwright flows, the Playwright MCP was the more reliable configuration, consistently achieving near‑zero failure rates on simple scenarios and remaining within 0–12% on more complex flows. In contrast, the Playwright CLI showed higher failure rates (roughly 12–20%), with many failures caused by execution issues such as authentication handling, navigation timing, and session instability rather than model reasoning.
Generated Playwright tests performed reasonably well on simple flows (~8% failure rate), but degraded significantly on more complex workflows (~48%). These tests were not entirely wrong, as they typically progressed through 70-80% of the flow before breaking on a final interaction or assertion. Failures were primarily caused by variability in UI state and abstraction mismatches. These tests were generated from loosely specified natural language flows and reused existing page object abstractions, which sometimes interfered with precise element targeting in more complex scenarios.
Overall, the reliability gap widened with increasing complexity, suggesting that the agent-native execution models like MCP provide more stable behavior as flows get harder. One likely reason is how each model handles state. MCP keeps a live, stable view of the app, while CLI rebuilds state from snapshots at each step. As flows get longer, small inconsistencies in how the UI is interpreted or timed can accumulate and lead to failures. Another likely factor is in-session context. In MCP-based runs, the agent appears to reuse successful interactions from earlier steps in the same flow, while CLI can feel more like starting from scratch at each step. We didn’t explicitly measure this, but it may also contribute to the gap.
When it came to speed, generated tests were consistently the fastest.
| Approach | Average Duration |
| Generated Playwright Tests | ~3 minutes |
| Agent (Playwright MCP) | ~5–8 minutes |
| Agent (Playwright CLI) | ~9–11 minutes |
For generated tests, the runtime includes both test generation and execution. Each test was generated once and executed five times, and the numbers above reflect the average duration per run. In practice, the raw execution was much faster: ~32 seconds for thread reply and ~45 seconds for search discovery. In CI environments where tests run repeatedly, the one-time generation cost becomes negligible, allowing deterministic tests to scale more efficiently.
Agent-driven workflows pay this cost on every run. Each step typically involves:
Another pattern we saw was how differently agents navigate the UI.
Only about 20% of runs followed the exact same sequence of actions. In most runs, the agent discovered different valid UI paths to reach the same goal.
For example, while still reaching the same final state, the agent might:
To measure this, we compared action signatures across runs. An action signature is the ordered list of tool calls and UI actions performed by the agent (e.g. API calls, browser clicks, form interactions). Action signatures were normalized before comparison: parameters, wait/snapshot actions, and equivalent tool variants (e.g. fill vs type) were collapsed so that only meaningful differences in the action sequence were counted.
Across runs, most action sequences differed even when the final outcome was correct. This highlights a key difference between approaches: traditional E2E tests enforce a single deterministic journey through the UI, while agents explore the interface and verify whether the goal state can still be reached.
Cost stood out in our experiments. Agent-driven runs were typically $15–30 per execution, compared to much cheaper traditional test runs.
To understand where this cost came from, we analyzed token usage across different execution models by running the same search discovery flow.
| Approach | Tokens |
| MCP (Opus 4.6) | ~3.8M |
| MCP (Sonnet 4.5) | ~3.5M |
| MCP (Haiku 4.5) | ~5.7M |
| CLI (Opus 4.6) | ~6M |
| Code Gen (Opus 4.6) | ~7M |
The first thing that stood out was that how the agent was executed mattered more than which model powered it. Haiku did use more tokens than Sonnet or Opus in our runs, but all of the MCP-based approaches still used fewer tokens overall than the CLI and Code Gen approaches for the same flow.
To understand why, we looked at how Claude Code executes agent sessions. The underlying API is stateless and every turn re-sends the full system prompt plus the entire conversation history. This means cost is not driven by model output, which is negligible, but by how quickly context accumulates and how many turns the agent takes to complete the flow.
| Approach | Turns |
| MCP (Opus 4.6) | ~40 |
| MCP (Sonnet 4.5) | ~40 |
| MCP (Haiku 4.5) | ~60 |
| CLI (Opus 4.6) | ~85 |
| Code Gen (Opus 4.6) | ~70 |
On average, CLI took 85 turns compared to MCP’s ~40-60 because each browser interaction was split across multiple commands, such as actions, waits, snapshots, reads, and element lookups. MCP combined interaction and state return into a single round trip. Each additional turn pays the full system prompt tax plus re-sends all prior conversation context.
What fills that context? For MCP and CLI approaches, browser snapshots are the primary payload. Playwright MCP returns accessibility tree snapshots as part of its browser interaction responses, and these accumulate in the conversation window across all subsequent turns. For Code Gen, the accumulated context comes from test runner output containing full error traces, assertion failures, and DOM state on each retry cycle.
In our analysis, the majority of the cost was retransmission of previously seen content. Only a small fraction of tokens represented new information per turn. The biggest factors affecting cost are turn count and context growth rate rather than model reasoning or output generation.
At this stage, we focused primarily on reliability and behavior, so token usage was not optimized. Opportunities to reduce cost include prompt caching, context compaction, and reducing snapshot frequency.
Due to the cost, agent-driven testing may currently be better suited for targeted debugging or exploratory testing than for high-frequency CI execution, although cost may improve with future models and tooling.
Another important takeaway was how much the execution environment affected reliability, not just the model itself.
| Approach | Failure rate |
| Agent (Playwright MCP) | 0–12% |
| Agent (Playwright CLI) | 12–20% |
Most failures in CLI-based runs came from authentication and navigation issues (sign-in errors, timeouts, and session instability), suggesting that many failures were caused by the execution layer rather than the agent’s reasoning.
The Playwright MCP provides structured browser primitives and tighter integration with the agent’s tool-calling workflow, while CLI-based execution introduces additional layers between the agent and the browser.
Parallelization also differed. MCP runs were easy to execute concurrently, while CLI-based runs were difficult to parallelize in our setup and were mostly executed sequentially.
These results suggest that reliability, speed, and cost depend not just on the model, but also on how stable and well-designed the execution environment is.
Our experiments focused on single-session UI workflows. More complex scenarios, such as cross-workspace flows or workflows that open multiple browser windows, introduce a different set of challenges where the choice of execution model may matter as much as the agent itself.
Both MCP and CLI-based approaches could support these workflows, but with different tradeoffs. MCP may run into cost issues as observation loops grow over longer flows, while CLI-based approaches may introduce additional coordination complexity when managing multiple browser sessions, on top of the higher token usage observed in our experiments. We did not explore these scenarios here, but they are an important consideration for teams evaluating agent-driven testing.
So where does agent-driven testing actually fit?
Rather than replacing existing approaches, it adds a new capability on top of them.
Best suited for fast, repeatable regression checks in CI.
Agent-driven workflows operate differently from deterministic tests. Instead of executing a predefined script, agents operate from a goal: they observe the UI, reason about the current state, and determine how to reach the desired outcome.
From a system perspective, agentic testing still operates at the same level as E2E tests, validating real user workflows through the UI. The difference is in how those workflows are executed.
For this reason, the most effective testing strategies of the future will combine both. Deterministic tests provide a stable foundation for CI, while agentic testing adds a distinct layer at the top of the testing pyramid for exploration, debugging, and validating complex behaviors.
Huge thanks to the DevXP AI team for building and supporting tools like Claude Code, as well as the metrics infrastructure that made these experiments possible. That foundation made it much easier to run, analyze, and iterate on hundreds of executions.
Special thanks to our managers, Dave Harrington and Vani Anantha, for supporting experiments at a scale that definitely kept the token counters busy, and briefly put us on our internal token usage leaderboard.
We also want to thank the Frontend Test Frameworks team for their help throughout the process, from early ideas to validation and feedback. Special thanks to Lucy Cheng, Natalie Stormann, Roopa Thanisraj, Ilaria Varriale, and Crescencio Zul for their thoughtful input and support along the way.
Interested in solving real problems, making developers’ lives easier, or just building some pretty cool tools? If this kind of work excites you, whether it’s pushing the boundaries of testing or building agent-driven systems and rethinking developer workflows, we’re hiring.
Apply now
Continue reading on the original blog to support the author
Read full articleThis article provides a blueprint for scaling enterprise LLM infrastructure. It details the transition from manual GPU management to managed services, highlighting how to balance security, cost-efficiency, and reliability through strategic multi-cloud orchestration and capacity forecasting.
Managing context in long-run agentic systems is critical as context windows fill and performance degrades. This architecture shows how to use structured memory and specialized agent roles to maintain coherence and accuracy across complex, multi-step workflows.
Scaling notification systems requires balancing high-volume delivery with user cognitive load. Slack's rebuild demonstrates how architectural simplification and cross-platform consistency reduce technical debt and improve UX by making complex systems predictable.
This article details how Slack built robust AI agent systems for security investigations by moving from single prompts to chained, structured model invocations, offering a blueprint for reliable AI application development.