A CI build goes red at 4:47 p.m.
The failing step is click Save. The screenshot shows a modal. The error message says the element wasn't clickable.
Was the button covered by a loading spinner? Did the API reject the request? Did the browser render something differently? Nobody knows yet.
The test took 40 seconds to run. Figuring out what happened might take the rest of the afternoon.
This is the part of test automation that gets surprisingly little attention. We spend weeks comparing frameworks and building test coverage, then give engineers almost nothing useful when a test fails.
I think that's backwards. The point isn't to have more automated tests. It's to make it easier to ship changes without guessing.
A green checkmark isn't the product
The conversation usually starts with a framework choice.
Should the team use Playwright or Puppeteer? Both can drive a browser. Both can click a button and check the result. But those capabilities don't tell you much about what happens on a bad Tuesday when a test fails only in CI.
A useful comparison of Puppeteer and Playwright makes the distinction clearer: Playwright offers more built-in help with traces, isolation, and multi-browser testing, while Puppeteer can be a sensible fit for narrower, Chromium-focused work.
The question I'd ask isn't, "Which framework writes a shorter test?"
It's, "Which one will help the next person understand a failure without calling the person who wrote it?"
That changes how you evaluate tools. A test that takes two minutes to write but requires 45 minutes of detective work every time it fails isn't cheap.
And the framework is only one piece of the problem.
Stop asserting things nobody promised
API tests can be just as noisy as browser tests, even without a browser involved.
Imagine an endpoint that returns a customer record. A developer adds an optional field, preferredLanguage. The API remains compatible with existing clients, but your test compares the entire response to a saved JSON object.
Suddenly the build is red. Nothing broke for the customer. The test was just too specific.
The same issue shows up with partial updates. If a PATCH request is supposed to change a customer's display name, the important questions are whether the name changed and whether unrelated fields were preserved. Insisting that every response contain an identical collection of fields can miss the point entirely.
This guide to stable API contract tests gets into the details of optional fields, partial updates, and version drift.
It's a useful reminder that a good test checks a promise the system actually makes. When we test incidental implementation details instead, normal product development starts looking like a series of regressions.
Visual testing has a noise budget
Screenshot comparisons are another place where teams accidentally manufacture work.
Let's say someone changes the spacing inside a navigation menu. Your visual testing tool flags 65 screenshots across several viewport sizes. Some of those changes matter. Many are duplicates of the same harmless adjustment.
Now someone has to click through all 65. After a few weeks, approving screenshots becomes a routine chore. That's when real regressions start slipping through.
For that reason, I like the approach in this visual regression benchmarking plan: evaluate the noise a tool creates, the baseline review process, and how it handles responsive breakpoints. Don't just count how many differences it detects.
You want a system that makes meaningful changes obvious. A tool that finds every single pixel difference but teaches your team to ignore its alerts has failed, even if its detection rate looks impressive on a sales slide.
A failure should arrive with evidence
Here's a reasonable rule for browser tests: a failed run should contain enough information for someone who wasn't watching it to start investigating.
A screenshot helps, but it doesn't tell the whole story. For a hard-to-reproduce failure, I'd want the failing step, browser and environment details, a timestamp, relevant console errors, and network activity. Video or a trace can be especially useful when the problem involves timing.
That's essentially the idea behind a reproducible browser failure bundle. The interesting part isn't collecting every possible artifact. It's collecting the right artifacts at the moment something goes wrong.
There is a security tradeoff here, too. Network captures and traces can contain tokens, cookies, or customer data, so teams need sensible redaction and access controls.
The difference in day-to-day work is significant. Instead of asking someone to rerun a test and record their screen, you can inspect the failed run as it happened.
Don't make Jira and Slack another debugging problem
Even when a test produces a useful failure report, the team still has to decide what to do with it.
This is where test management tools get interesting, and where feature checklists can be misleading.
If the team already works in Jira and Slack, another dashboard isn't automatically helpful. The important questions are more mundane. Can a failed test be connected to the right ticket? Can someone tell whether the issue is new or already known? Does the Slack notification identify a person who can take action? Is there a history of what changed?
The guide to choosing test management tools for Jira and Slack teams focuses on those workflow details rather than counting integrations.
A notification that says "17 tests failed" is rarely enough. It's just an invitation for five people to ask the same question.
And having two systems with conflicting status information can be worse than having one slightly limited system everyone trusts.
Measure the time after the failure
If I were evaluating a testing setup, I'd start with a small experiment.
Take a few ordinary changes: an optional API field, a slow-loading modal, a responsive layout adjustment, and a genuine broken checkout flow. Run the tests and hand the failures to an engineer who didn't write them.
Then watch what happens.
How long does it take to understand each failure? How many alerts turn out to be irrelevant? Can the engineer reproduce the problem from the artifacts? Does the result reach the right person without three Slack conversations?
Those answers tell you far more about the cost of your test suite than the number of tests it can execute per minute.
Fast tests are great. Broad coverage is great. But if every failure creates a small investigation project, your automation is saving machine time and spending human time.
That's not a particularly good trade.
Top comments (0)