Your Playwright test failed.
The build is red. A notification arrives. Someone opens the report.
And then the real work begins.
Was this a product bug? A flaky test? A selector that changed? Missing test data? An environment issue? Or simply a report with too little context?
For many teams, test execution is already automated. Understanding the result is not.
The 5 questions to answer after a failed run
1. Can we identify the exact run?
Capture the essentials before discussing the failure:
- Test name
- Browser and environment
- Commit or build reference
- Execution time
- The system or person that triggered the run
Without this context, a red result is difficult to reproduce and easy to misinterpret.
2. Is the failure reproducible?
One failed run is evidence, not yet a diagnosis.
Does it fail again locally? Only in CI? Only in one browser? Did it pass in the previous build?
This helps teams avoid treating every red test as a product defect.
3. Do we have the evidence we need?
A useful result review needs more than a pass/fail status:
- Playwright trace
- Screenshot or video
- Error message
- Relevant logs
- Test steps around the failure
The goal is not to collect every artifact. It is to make the next decision possible.
4. What kind of failure is this?
A simple classification reduces unnecessary back-and-forth:
- Product behaviour
- Flaky test
- Selector issue
- Test data
- Environment or timing
- Missing context
It is fine to mark a result as uncertain. Uncertainty is better than a confident but incorrect conclusion.
5. Who owns the next action?
Every result review should end with a concrete next step:
- Reproduce the failure
- Fix the product behaviour
- Improve the test
- Investigate the environment
- Collect missing evidence
And every action needs an owner.
The real bottleneck comes after execution
Playwright is excellent at running tests and producing valuable artifacts. But a team still needs to review those artifacts, align on what happened and decide what to do next.
The important question is not only:
Did the test pass?
It is:
Is the result clear enough for the team to act on?
Free Playwright Result Review Checklist
XYVA QA Studio is built for this moment: teams run Playwright locally and review results together, while code and credentials remain in their own environment.
Top comments (2)
One addition from running automation against real phones: the passing run needs the same five questions.
We had green runs where the action never happened. A system permission prompt was sitting on top of the app, the tap landed on a button behind it, and nothing failed, so nobody opened the report.
What fixed it was checking the outcome somewhere the test does not control (for us, the public page the action should have changed) and counting a pass only when that agrees. Your "uncertain" class helped too: we record "unknown" as its own result instead of rounding it to pass or fail, and it gets checked against the public page later.
I’ve put together the 5 questions teams should answer after a failed Playwright run — including a free checklist. Link in my profile.