The test passed because it was reading our own input

We run browser agents that fill out job application forms on company hiring systems. One of them reported a form as fully completed. Seventeen fields written, seventeen confirmed, submit clicked.

Zero of those seventeen values reached the employer.

The agent was not lying. It was reading the wrong thing, and the wrong thing agreed with it every single time.

The widget

The field was a react-select combobox. You know the shape: a text input on top, a menu of options underneath, and a hidden value the form actually submits.

Our agent typed into the input, read the input back, saw its own text sitting there, and marked the field done.

Here is what we measured in a real browser on a real posting, driving the same widget three ways:

how the widget was driven	options that appeared
synthetic mousedown	0
typing, which is all our write path ever did	0
a real, trusted click	244

react-select opens its menu on a trusted mousedown. A synthetic event does not qualify. So on our agent's path, no menu ever opened, nothing was ever selected, and the hidden value stayed empty.

The visible input, meanwhile, was holding our text perfectly. Our readback matched. Success.

This is a feedback defect, not a fill defect

That distinction took us far too long, so I want to state it plainly.

The filling was never the broken part. The scoring was.

Our guard did this:

await fill(selector, value);
const readback = await getValue(selector);
if (loosely(readback, value)) return { success: true };  // <- reading ourselves

Every term in that condition is under our own control. We wrote the value, we read the field we wrote it into, we compared our string to our string. There is no state in that check that the other side produced. It cannot fail while the writing works, and it cannot succeed only when the submission works, which is the property you actually want from a check.

Because the field scored as done, the agent never escalated to the dedicated commit path that would have clicked the option properly. That path exists. It is model-elective, so nothing calls it for you. A false green is worse than a red, because a red at least routes you to the fallback.

The production shape, on real runs:

• one application: 0 of 17 combobox writes actually committed
• another, same form, same run: all 3 writes on the dedicated path landed, and 6 of 8 on the agent path went out empty
• another: 8 empty, one of them reading back the literal placeholder text "Select…" and calling it a value

The form then refused at submit with no error the agent could see, because from the browser's point of view nothing was wrong. Required fields were simply empty.

The rule we wrote down afterwards

Never score a write on a read-back of your own input.

Score it on something the other side produced. For a combobox, that means asking the widget what it committed:

• the rendered selection chip, not the filter text
• the hidden backing input the form validates against
• the library's own committed-state fingerprint

And then the part that makes it safe to ship:

Three states, not two. committed, uncommitted, and unknown. unknown changes nothing.

That third state is load-bearing. A plain ARIA combobox, where the typed text legitimately is the value, must not be failed by a checker built for react-select. If we cannot identify the widget, cannot resolve the scope, or anything throws, we return unknown and leave the previous verdict alone. A checker that is confidently wrong on an unfamiliar widget is a new bug wearing the old bug's clothes.

Two bugs in our own patch were caught by tests written for exactly that:

1. Concluding uncommitted at the first react-select-looking element answers before reading the hidden input that actually matters.
2. Treating a throw as a failure fails every widget we have never seen.

Where else this lives

Once you have the shape, you see it everywhere:

• an upload marked done because the file input's files.length is 1, which is a property of your own DOM write, not of any byte reaching a server
• a save confirmed by re-reading local state instead of the response
• a queue publish confirmed by the absence of a thrown error
• a metric that reads zero because the instrument was not armed, which is not the same as the event not happening

The generalisation we now apply to every check we write:

Would this check read differently on the success case and the failure case? If not, it is not evidence, it is a mirror.

Waiting longer does not help either. We tried increasing the delay before the readback, on the theory that the menu was slow. It was reverted. There was no menu to wait for.

---

I build AI Applyd. We apply on the company's own hiring system, and we only count an application as sent when their system confirms it, for exactly the reason above: our own click is not evidence of anything.