Ava Bagherzadeh

Posted on Aug 17

Your automation says "submitted". Here's how to verify a form was actually submitted

#webdev #ai #testing #automation

To verify a form was actually submitted in browser automation, you have to stop trusting the one signal
every framework hands you for free: your own success log.

TL;DR — a green submitted: true log is your own click printed back to you. Three signals actually establish delivery: a request on the wire, page state only the server could produce, and the counterparty answering unprompted.

I spent a long time debugging a pipeline that fills and submits real application forms on employer
hiring systems. Not a test fixture, not a mock. Real Greenhouse, Lever, Ashby, Workday, iCIMS forms,
each one a different shape, most of them behind at least one thing that does not want a script there.

The bug I could not find for weeks turned out not to be a bug in the filling. It was a bug in how I was
measuring the filling. This post is the part I wish someone had written for me.

The core mistake: scoring a run on a read-back of your own input

Here is the shape of almost every automation log I have written, and probably yours:

await page.setInputFiles('input[type=file]', resumePath)
log.info('resume staged', { staged: true, files: 1 })

await page.click('button[type=submit]')
log.info('submitted', { submitted: true })

Both of those lines are lies, in a specific and dangerous way. They are not false. The calls really
did return without throwing. They are lies because of what they claim to measure.

staged: true does not mean a file was staged. It means I asked for a file to be staged and got no
exception. submitted: true does not mean anything was submitted. It means I clicked at some
coordinates. Every value in those log lines originated on my side of the wire. Not one of them is the
other side answering.

Once you see it you cannot unsee it. A tool that types a value into a box and then prints the value it
typed has proved nothing. My dashboard was full of green.

The concrete failure that taught me this

The specific thing that was happening, and it is worth knowing if you run any browser automation
against a remote browser rather than a local one:

setInputFiles has more than one code path. Against a local browser it hands over a real filesystem
path. Against a remote browser it cannot do that, because the file lives on your machine and the
browser is somewhere else, so the library takes a different branch that ships the bytes over the
protocol. If that branch does not get what it expects, you can end up with a file input that is
technically populated and contains zero bytes. No exception. No warning. files: 1.

Twenty-three separate debugging sessions of reading my own logs found nothing, because my logs were
built entirely out of my own inputs and my own inputs were all fine. The thing that found it in about
ten minutes was opening a real browser next to the automated one, doing the same upload by hand, and
diffing the two network panels. One request carried a multipart body of a few hundred kilobytes. The
other carried nothing.

The divergence between a human doing it and your rig doing it IS the bug. If you are stuck, stop
reading logs and go get that diff.

Three signals that are not a read-back of your own input

Ranked by how much I trust them.

1. The request on the wire

Not "did I click submit" but "did a request leave, where did it go, what status came back, and how big
was the body". You can get this from CDP, from page.on('request') / page.on('response'), or from a
proxy. This is the cheapest real signal available and most people skip straight past it to assertions
about the DOM.

One trap that cost me real time: a %{http_code} of 000 is not a server verdict. It means the
request never completed. I read a wall of 000s as "the server is rejecting us" when the truth was
"nothing ever finished". A hang and a rejection look identical in a summary table and mean completely
opposite things.

2. Page state that only the server could have produced

A URL that changed to a path you did not navigate to. A confirmation string that your code never
authored. A form that is now gone from the DOM. The test is always the same question: could my own
code have produced this string? If yes, it is not evidence.

Be careful with the negative version of this. A selector that does not exist on this particular vendor's
page returns empty, and empty is indistinguishable from "the check ran and found nothing wrong". A
reader that is silently absent will report success forever.

3. The counterparty answering on its own initiative

The strongest signal, and the only one I would put in front of a user: something arrives from the other
side, through a channel you do not control, that you did not trigger. In my case that is an inbound
email from the employer's system. It cannot be faked by my own optimism because my code has no way to
write it.

This is worth the architecture cost. We have a confidence value that is reserved, unreachable from
our own telemetry by construction. Only an inbound message from the other side can set it. That
constraint is annoying exactly once, when you are tempted to set it from a place you should not, and it
pays for itself permanently after that.

Two instrument rules that generalise past automation

These are the two that changed how I debug everything, not just browsers.

An absence is not evidence until you have proved you were watching. Before you believe any zero,
empty list, or "nothing happened", prove three things: the instrument was armed at all, it was armed
before the event and still armed after, and the mechanism works when a human does it. One of my
instruments armed about eighteen seconds after the thing it was supposed to measure, so a working fix
read as a clean failure and I "reverted" it. If you cannot prove all three, write UNOBSERVED, never
"absent", and go fix the instrument before you touch the subject.

A wrapper's exit code is not the work's exit code. cmd | tail -30 exits with tail's status.
cmd; echo "done $?" exits with echo's status. cmd || true exits zero by definition. I have twice
been shown a green result by a chain whose middle had been killed by the OOM reaper. Write
cmd > file 2>&1; echo $? and then actually read the file.

The uncomfortable part

When you build verification this way your numbers get worse. Mine did. A dashboard that used to be a
column of green became three columns, and the third one says the thing did not land.

That third column is the only reason any of the other numbers mean anything. A smaller honest number is
worth more than a bigger invented one, and it turns out to be much easier to debug a product that tells
you the truth about itself.

Full disclosure, this came out of building AI Applyd, which fills and submits
job applications on the employer's own hiring system and then waits for that system to confirm the
application arrived before it counts as sent. The verification problem above is basically the whole
product. Happy to go deeper on any of the three signals in the comments. The network-level one in
particular is where most of the wins were.

DEV Community