I almost killed a project because my test harness was measuring its own bug

I was one command away from spending real API money on an experiment whose answer would have been wrong — and wrong in the most dangerous direction: it would have told me the thing I was testing worked, for the wrong reason, or failed for a reason that had nothing to do with it.

What saved me was boring. Before running the paid version, I ran the free version and actually read the raw data with my own eyes. This post is about what I found there, why measurement bugs tend to lie in your favor, and the rule I now follow before any expensive evaluation.

The question I was trying to answer

The project was a small accessibility tool. The pipeline looked like this:

1. Load a real web page in a headless browser (Playwright).
2. Run axe-core on it to detect accessibility violations — missing button labels, images without alt text, invalid ARIA attributes, that kind of thing.
3. For each violation, hand the offending HTML element to an LLM and ask it for a fix in a strict format:

BEFORE:
```html
<the element as it is now>
```
AFTER:
```html
<the fixed element>
```
EXPLANATION: ...

(The code comments you'll see below are in Japanese — that's how they are in the actual repo, and I've kept them as-is rather than rewrite code for the post.)

1. Score the result.

The whole project hinged on one kill question: are the AI's fix suggestions actually usable? If not, there was no product. So I wrote a verification harness that would run the pipeline over a list of real sites and classify every response.

The classifier was strict on purpose. A response was only counted as a valid, applicable fix if it passed a chain of checks, in this order:

// 判定順序は仕様通り: api_error -> no_text -> truncated -> malformed -> unresolved -> noop -> before_mismatch -> source_truncated -> valid_applicable

The two checks that matter for this story are noop and before_mismatch:

if (normalizeHtml(parsed.after) === normalizeHtml(parsed.before)) return 'noop';
if (normalizeHtml(parsed.before) !== normalizeHtml(nodeHtml)) return 'before_mismatch';

In plain English: the model's BEFORE block has to match the element I gave it. That is how I make sure the fix can actually be applied to the page as a find-and-replace patch, rather than being a hallucinated rewrite of something that doesn't exist.

That check felt rigorous. It turned out to be the exact place where the harness lied to me.

The free run

The paid run needed an API key that I didn't have yet. While waiting, I added a --dry-run flag:

//   node k2-verify.js --dry-run  APIを一切呼ばずscanUrlの配線確認のみ

It runs the browser and axe-core against every site, builds every input the LLM would have received, and writes them all to disk — but never calls the API. Zero cost. The point was just to confirm the wiring.

I almost didn't read the output. The run completed, the summary table looked plausible, and the natural next step was "wait for the key." Instead I opened the raw JSON and read the actual HTML snippets I was about to send to the model.

Many of them were broken.

Bug 1: axe-core's node.html is not the element

I had assumed that violation.nodes[i].html — the HTML string axe-core attaches to every violation — was the element's markup. It's not. It's a summary of the element, built for humans reading a report.

I didn't trust my memory about how it was built, so I opened node_modules/axe-core/axe.js and read the function that produces it. In the version I had installed (4.12.1), _getElementSource takes a maxLength that defaults to 300 and an attrLimit that defaults to 20:

function _getElementSource(node) {
  var _ref24 = arguments.length > 1 && arguments[1] !== undefined ? arguments[1] : {}, _ref24$maxLength = _ref24.maxLength, maxLength = _ref24$maxLength === void 0 ? 300 : _ref24$maxLength, _ref24$attrLimit = _ref24.attrLimit, attrLimit = _ref24$attrLimit === void 0 ? 20 : _ref24$attrLimit;
  ...
  var deepStr = getOuterHtml(node);
  if (deepStr.length > maxLength) {
    return getTruncatedElementSource(node, {
      maxLength: maxLength,

and the attribute truncation is this:

function truncate(str, attrLimit) {
  return str.length <= attrLimit ? str : str.substring(0, attrLimit) + '...';
}

So when an element's outerHTML is longer than 300 characters, you don't get the element. You get its opening tag only — children gone, closing tag gone — with long attribute values cut to 20 characters plus ....

That's a completely reasonable design for a violations report. It's a terrible input for "please write a patch for this exact markup."

Why this bug points toward "pass"

Here's the part that still makes me uncomfortable.

When you hand an LLM a truncated, unclosed element and ask for BEFORE and AFTER, a well-behaved model copies what you gave it into BEFORE. Truncated attribute and all. Then my strict classifier compares BEFORE to the input I sent — the same truncated string — and they match.

before_mismatch passes. The response gets counted as an applicable fix.

But it isn't applicable. The real page doesn't contain <a href="https://example.co..."> with no closing tag. Nobody can find-and-replace that into a real codebase. The "patch" is a patch against a string that only exists inside my harness.

My most rigorous check was rigorous against the wrong ground truth. And because the error made the model look better, I would never have gone looking for it after a good-looking result. I would have celebrated.

The detection trap

My first instinct for detecting truncated inputs was to search for the ... marker. That turned out to be wrong in both directions:

• False positives: real page text contains .... A heading like <h3>Be Bold...</h3> or an attribute like title="..." isn't truncation, it's content.
• False negatives: if every attribute on an element is short, the opening-tag-only fallback leaves no marker at all. A <html lang="en"> with its entire document body silently dropped looks perfectly clean.

The reliable signal ended up being structural: a non-void element whose snippet has no closing tag has lost its children. A string search for a marker was never going to be enough, and I only found that out because I read the library source instead of guessing what it "probably" does.

The fix: re-fetch the real element

axe-core gives you a target array — a selector path to the element. So after axe runs, and before the browser context closes, I go back to the live page and ask for the real outerHTML:

async function refetchOuterHtml(page, target) {
  try {
    const locator = buildLocatorForTarget(page, target);
    const html = await locator.first().evaluate((el) => el.outerHTML);
    if (typeof html !== 'string') {
      throw new Error('evaluate did not return a string (outerHTML unavailable)');
    }
    return { ok: true, html, error: null };
  } catch (err) {
    return { ok: false, html: null, error: err.message };
  }
}

I still need some size cap for a prompt, so I wrote my own truncation — one that keeps the opening tag fully intact, says explicitly what was removed, and keeps the element closed:

const marker = `\n<!-- rammendo: children omitted (${omittedChars} chars) -->\n`;
const closing = tagName ? `</${tagName}>` : '';
return { html: `${openingTag}${marker}${closing}`, childrenOmitted: true };

The difference from the library's version isn't "truncate vs. don't truncate." It's that this truncation is declared. The model can see that children were omitted, the result record carries childrenOmitted: true, and the scorer can treat it accordingly instead of mistaking it for the full element.

And crucially: if the re-fetch fails, I don't quietly fall back to axe's string. I fall back loudly:

console.warn(
  `[rammendo] outerHTML再取得失敗 (rule=${violation.id}, url=${pageUrl}): ${refetch.error} — axeのnode.htmlにフォールバックします`
);
node.__rammendo = {
  html: node.html,
  sourceKind: 'axe_fallback',
  ...

Every record carries a sourceKind, and the summary report has a row that counts axe_fallback and prints a large warning block if it's anything other than zero. A silent fallback would have recreated the exact lie I had just removed, the first time a selector failed to resolve.

Bug 2: I was counting other people's bugs as my customer's

The second problem showed up in the same free run, in a different column.

One site had a lot of ARIA violations — far more than the rest. Reading the raw records, almost all of them were inside an embedded third-party booking widget. axe-core scans through iframes by default, which is the right default for an audit ("your users experience this page, including the widget"). It's the wrong default for my question, which was "can the AI fix the site owner's code?" The site owner can't edit markup that lives inside someone else's iframe.

So that site's numbers were inflated by violations nobody on that site could fix, and any fix the LLM wrote for them would be a fix to code nobody could apply.

The tempting fix is a heuristic: "if the class name contains the vendor's name, skip it." I explicitly didn't do that. Class names are guesses. Instead I used two mechanical signals:

1. axe's target array is longer than one entry when the element is inside a frame. The leading entries are the frame path.
2. For the outermost frame, read its actual src, resolve it against the page URL, and compare origins.

const thirdParty = iframeOrigin !== pageOrigin;
return { inIframe: true, framePath, iframeSrc, iframeOrigin, pageOrigin, thirdParty, originNote: null };

And when the signal is missing — an iframe with no src, or about:blank, or a srcdoc — I refuse to guess:

originNote = 'iframeにsrc属性が無いか、srcdoc/about:blank等の非URL値。推測でthird party判定はしない';

Third-party violations aren't deleted. They're reported in their own section, with the provider's origin, as "this is real, but it isn't in your code." Hiding them would be its own kind of dishonesty; mixing them into the fixable count was the other kind.

Why measurement bugs lean toward "pass"

Both bugs share a shape, and it's the reason I'm writing this up.

A bug in the thing you're testing usually makes results worse, and bad results get investigated. A bug in the instrument can make results better, and good results almost never get investigated. Nobody reruns an experiment that came out the way they hoped.

Bug 1 is the clean example. The strict "BEFORE must match the input" check is a self-consistency check. If the input is wrong, self-consistency still holds — the model is perfectly consistent with garbage. Every check that compares an output to the harness's own input inherits whatever is wrong with the input.

Bug 2 points the other way — inflated failure counts — but it has the same root: the instrument was answering a slightly different question from the one I asked. axe-core answers "what's wrong with this page as rendered?" I needed "what's wrong with this site owner's code?" The two differ by exactly the stuff inside third-party frames.

In both cases, if I had gone straight to the paid run, I would have gotten a number, and the number would have been about my harness rather than about the AI.

A quick checklist for "is my harness lying?"

After this, I wrote down the questions I now ask before I trust any number an eval harness produces. They're cheap, and most of them take less time than reading this section.

Where does each input come from, exactly? Not "from the scanner" — which field, produced by which function, with which defaults. If the answer involves a library, I want to have seen the line of code. In this case the answer was "a display string with a 300-character cap," and that one fact changed everything downstream.

What does a perfect score look like if the model does nothing clever at all? If a model that simply echoes its input back would pass a check, that check is measuring the harness, not the model. My BEFORE comparison failed this test: an echo passes it by construction. It's still a useful check — it rejects hallucinated rewrites — but it can't tell you whether the input was right.

What fraction of records took a fallback path? If the answer isn't printed in the summary, I don't know it, and "I don't know" tends to quietly become "zero" in my head. Now it's a row in the report with a warning attached.

Is my denominator the population I care about? The third-party iframe problem was a denominator problem. The scanner's population was "everything rendered on the page"; mine was "everything the site owner can edit." Counting the wrong population produces a precise answer to a question nobody asked.

Would I notice if this check broke silently? For each check I ask what the report would look like if it always returned "pass." If the honest answer is "about the same, maybe slightly better," that check needs a test of its own — a deliberately broken input that it has to reject.

None of this requires a framework. It's a short list in a markdown file next to the harness, and I read it before every paid run.

What I do now

These are the rules I took away. None of them are novel; I just hadn't been following them.

1. Before any paid evaluation, do a free run and read the raw inputs.
Not the summary table. The actual payloads that would have gone to the model. A dry-run mode costs almost nothing to add, and it's the only stage where the inputs exist without results to bias how you read them.

2. Ask which direction the instrument would lie.
For every check, I now ask: "if this check is broken, does it fail too much or pass too much?" Checks that would pass too much get extra scrutiny, because nothing downstream will catch them. Self-consistency checks — output compared to the harness's own input — are the first suspects.

3. Read the library source for any data transformation you depend on.
"axe gives you the element's HTML" was a belief, not a fact. Opening node_modules and finding maxLength and attrLimit took a few minutes. Anything a library does to your data — truncation, normalization, summarizing, deduplication — is worth confirming in code rather than in memory or docs.

4. Make every fallback visible and countable.
If the good path fails and you fall back to a worse source, tag the record and count it in the report. A fallback that can't be seen in the summary is just the original bug with extra steps.

5. Separate "what the tool reports" from "what you're actually asking."
A scanner's scope (the rendered page, including iframes) and your question's scope (the code a specific person can change) are often slightly different. Write the question down explicitly, then check whether the instrument's default matches it.

6. When a signal is missing, record that it's missing.
For the iframe check, "no src" doesn't mean first-party or third-party. It means unknown, and the record says so in a note. Guessing would have been quieter and worse.

The uncomfortable bit

The scariest part of this isn't that the harness had bugs. Every harness has bugs. It's that I was about to use the result to make a go/no-go decision about the whole project.

If the bugs had pushed the numbers down, I might have concluded "AI fixes aren't good enough" and dropped an idea because of a problem in my own wiring that took an afternoon to fix. If they had pushed the numbers up, I would have built on a foundation that wasn't there. Either way, the decision would have been made by my instrument, not by the thing I thought I was measuring.

The free run didn't make the project succeed. It made the eventual measurement mean what I wanted it to mean. That's a much smaller claim, and it's the one I actually care about now.

If you're building an eval harness for an LLM feature — especially one where the model's output gets compared to the input you constructed — I'd be curious how you guard against the same thing. Do you read raw payloads before every run, or do you have checks that catch a broken input automatically?