Half my test suite asserts that nothing happened

Last month I shipped a bug where every test passed.

The extension I maintain fills web forms with test data. It has two modes: use a profile you saved yourself, or generate fresh data. A user picks "Generated test data", the extension fills the form — and quietly uses their saved profile instead.

Not a crash. Not an error in the console. The form filled. Every field had a value in it. The only thing wrong was that the values were the wrong ones, and nothing anywhere said so.

Every test in the suite was green.

Why it passed

The bug was in a fallback. The code resolved which profile to use, and when nothing matched it fell back to the first item in the list. The built-in "generated data" option isn't a saved profile, so it never matched, so it always fell through to the fallback. Once you saved a single profile, you could never choose generated data again.

Here is the part worth sitting with: the fallback did exactly what a fallback is supposed to do. It picked something reasonable instead of failing. And every test I had written asked the same kind of question — did the form get filled? Yes. Do the fields have values? Yes. Is the data valid? Yes.

Not one test asked is this the data the user actually asked for? And no test at all asked the harder version: when the user picks the built-in option, does the code correctly do nothing with the saved profiles?

That is the whole category. I had a suite full of "did it work" tests and nothing that said "did it correctly do nothing".

The tests nobody writes

Once you start looking for this, it is everywhere. Test suites are written as a description of what software should do, because that is how features are specified, planned and reviewed. "Should not" is a different question, and it does not turn up just because the "should" list got longer.

For a tool that writes into forms, the "should not" list is long and it matters more than the other one:

• A CAPTCHA field must never be filled. Filling it is worse than doing nothing, because it looks like an answer.
• A hidden CSRF token must never be touched.
• disabled and readonly fields stay as they are.
• File inputs stay empty.
• Anything invisible stays untouched.
• A form in a cross-origin iframe is not filled, and cannot be — reaching into another site's frame would need standing access to that site.

A filler that fails to fill a field is annoying. A filler that writes into a hidden token field and breaks the submit is worse, and the user will not know why. The failures on that list are all silent, which is exactly why they need explicit tests: nothing else will surface them.

So now, before a feature gets written, I write the refusal cases as fixtures. The question I ask about a plan is not only "is this right" but "what should this refuse to do, and what happens if it is asked to anyway". The second pass finds things the first one never surfaces. Every time.

Right now the suite grades 75 cases and 10 of them assert refusal rather than success. Those 10 are the ones I would keep if I had to throw the rest away.

Mocks were part of the problem

The other half of the fix was where the tests run.

My original suite tested the field classifier and the fill engine against mock DOM objects. That is fast and it is easy to write, and it is also how the entitlement bug survived: the mocks reflected what I believed the DOM did, and the belief was the thing that was wrong.

A browser extension has a particularly bad case of this. The engine is injected into a page it has never seen, in a browser build I do not control, against a framework that patches native DOM behaviour underneath it. React, to take the most common example, keeps its own record of what is in each input. If something writes to input.value directly, React compares the DOM against that record, sees no difference, and never fires onChange. The text is on screen. The application never received it. (I wrote about that specific mechanism in more detail here.)

None of that is visible to a mock. A mock will happily tell you the value was set.

So the suite got rebuilt as a real page. It loads the same four files the service worker injects — not a copy, not a test build, the actual engine — and runs them against a page full of deliberately awkward fields:

• React, Vue and Angular controlled inputs
• open shadow roots, including nested ones, and custom elements
• fields classifiable only by an autocomplete token
• fields with no useful signal at all — no label, no placeholder, no name
• field names containing dots and dashes that break naive selector matching
• same-origin iframes
• and the refusal cases above

56 fixtures, 75 graded cases. The last run: 65 passed, 0 failed, 10 correctly skipped.

Then I shipped the suite inside the product

This was the part I did not expect to be useful, and it turned out to be the most useful.

The self-test is not a CI artefact. It is a page inside the extension. Anyone who installs it can open it and run the real engine in their own browser, on their own build, and see the score.

Two reasons that matters.

The first is honest self-interest: browser updates break extensions. I test on my machine, on my Chrome version, in my OS. A user three versions ahead on a platform I have never touched will find failures I cannot reproduce and probably cannot even imagine. Now they can run the suite and send me a report instead of an uninstall.

The second reason is about trust, and it is the one I would repeat to anyone building a tool that touches other people's pages. My store listing makes a set of claims: it handles React, it handles shadow DOM, it leaves CAPTCHA fields alone, it makes no network requests. Anyone can write that. Screenshots prove nothing — I made the screenshot. A test suite the user runs themselves is the only version of that claim they do not have to take my word for.

"Know it works in your browser" turned out to be a better thing to offer than any feature on the list.

What it still does not catch

I want to be straight about the limits, because a scoreboard is easy to over-trust.

The bug that cost me the most money this year was not in the extension at all. A payment webhook fired on a completed transaction, and the payload did not always contain the buyer's email address — sometimes it carried only a customer id. The code was correct. The tests were correct. They were written against the payload shape in the documentation, and the documentation showed the field.

I found it on a real purchase, which is a bad way to find it.

No fixture would have caught that, because the fixture would have used the documented shape too. The only thing that catches it is logging raw payloads from live traffic and looking at what actually arrives. Fixtures test your code against your understanding. Where the understanding is wrong, they agree with you.

So: fixtures for what you control, real payloads for what you do not.

What I would tell myself a year ago

1. Write the refusal cases as fixtures, before the feature. Ask "what should this refuse to do" as a separate question from "is this plan right". It surfaces different answers.
2. Run the real code in the real runtime. Mocks encode your assumptions, and your assumptions are what is broken.
3. A green suite is evidence about the questions you asked, not about the software. Mine was fully green while a headline feature was silently broken.
4. Consider shipping the suite. If your tool runs inside somebody else's environment, letting them verify it there is worth more than another screenshot.

---

I build FormForge, an offline form filler for QA and development work. The self-test page described here ships with it — free tier included, no account needed. If you run it and something fails, I would genuinely like to see the report: [e-mail]