Skip to content
Back to blog
Aug 17, 20267 min read

From locator failure to an evidence-backed repair suggestion

Lessons learned from a synthetic Playwright lab for test design, failure triage, and review-only self-healing.

AI QAPlaywrightTest AutomationHuman Review

A test failure is only a symptom at first

A failed end-to-end test initially tells us only that an expectation was not met. The locator may be outdated, the UI may have changed at the domain level, or a backend failure may have interrupted the same user flow. Looking only at the assertion message can therefore lead to a fast but wrong repair.

I built a small AI QA Automation Lab to explore this problem. It combines a synthetic React shop with Playwright, API tests, unit tests, and structured AI functions for test design and failure analysis.

Evidence before suggestion

In the demo scenario, a locator is deliberately invalidated. The framework then collects more than the error message: the affected locator, a DOM excerpt, visible page text, available locator candidates, browser and network events, a screenshot, and a Playwright trace.

This information is turned into a structured analysis. The result should show the likely failure category, the evidence behind it, and a sensible next action. A red test therefore becomes a traceable diagnosis before it becomes a repair.

Failure-to-repair instead of blind self-healing

The most important design decision is separating suggestion from application. The AI may identify a possible new locator and produce a diff, but it may not write that diff into the test code automatically. The next step remains human review.

That is more valuable for real QA work than a spectacular autopilot. A new locator may work technically while targeting the wrong component. The test can turn green and still stop covering the original risk.

Guardrails belong inside the workflow

That is why the lab uses a mock provider by default. Results remain reproducible and local runs create no API cost. The AI functions return structured suggestions instead of uncontrolled prose. Screenshots, DOM content, and logs are intended only for synthetic test data.

These rules are not security decoration added afterwards. They shape the feature itself: which data is collected, how uncertainty is represented, which changes may be suggested, and how a team verifies that a repaired test still represents the intended risk.

The valuable output is traceability

AI can accelerate QA work when it collects evidence, exposes uncertainty, and limits changes to reviewable suggestions. The most valuable output is not a magical repair, but the connection between failure, evidence, classification, and a verifiable next step.

For production systems, I would therefore not start with automatic self-healing. I would start with a review layer. This is where AI delivery and conventional quality engineering meet: AI accelerates analysis and suggestions, while domain judgement, responsibility, and approval remain part of engineering.

If the connection between AI, test strategy, and traceable delivery interests you, the experiment offers a practical starting point for a conversation about QA workflows.