DEV Community

John Slowik
John Slowik

Posted on AI-assisted

Trust, but Verify: How a Team of AI Bots Kept My Experiment Honest

I almost published the wrong number.

On a checkout form, the big screen looked cheaper than the small one: 0.82 times the tokens. That's a tidy story, and it was wrong. A Chrome payment-autofill popup, not the viewport, had added an extra step on the small screen. Once that popup was turned off, the big screen used about 1.16 times the input tokens, or about 1.12 times the cost. The interesting part isn't that I reran a test. It's that a skeptical reviewer bot refused to let the tidy number stand, and that the hypotheses were already written down before anyone spent another dollar.

If you're going to let AI agents run a measurement you care about, the process is the product. Let's walk through how this one was set up, and what it caught.

Why this post exists

I wrote a companion post about what viewport size actually costs an AI tester. The short version: on a page that needed no scrolling, a 2560x1440 screen used 2.19 times the input tokens of 1024x768, or about 2.1 times the cost. When the bigger screen removed scrolling, that premium shrank to about 1.16 times the tokens. The bill tracked the number of steps more than the pixels.

That post is about the numbers. This one is about how they stayed honest. The experiment ran in a single day, November 16, 2026, on two public demo pages, and came in at $21.70 of a $60 cap I set up front. Talk is cheap. The review is what I want to show you.

The team

I asked a small team of AI agents to split the work, and I kept the budget and the calls.

A writer/coordinator turned the idea into a brief, relayed my approvals in my own words, drafted the findings post, and later became the single inbox for the rest of the team.

A builder designed the harness, ran the calibration, the pilots, and the full sets of runs, and committed the plans before any real spend. It did not start until I approved the work in writing and handed over an API key.

An infrastructure bot stood up a local environment at zero cloud cost: TodoMVC behind a simple sign-in page, then Bootstrap's sample checkout form, served on localhost, with Chrome in kiosk mode on a virtual display at 1024x768, 1280x800, 1920x1080, and 2560x1440. Each run got a fresh profile. A teardown script wiped the environment when a run finished.

A skeptical reviewer treated the plan the way a careful colleague treats a paper. It bumped the design from three runs per screen size to ten, required a pilot and a locked analysis plan, made the team write the hypotheses and a stop rule before the full runs, and set cost caps under my $60 limit. Then it opened the raw result files itself and wrote the caveats the findings post had to carry.

An editor read the first draft against those results and the public docs, and sent it back with homework.

If that sounds like a lot of process for two boring demo pages, you're right. That was the idea.

flowchart TD
  John[John: budget, scope, and the final call]
  Coord[Writer / coordinator]
  Infra[Infrastructure bot]
  Builder[Builder]
  Review[Skeptical reviewer]
  Editor[Editor]

  John -->|approves spend and scope in writing| Coord
  Coord -->|relays his exact words| Builder
  Coord -->|asks for a local environment| Infra
  Infra -->|up and down scripts, zero cloud spend| Builder
  Coord -->|asks for a methodology review| Review
  Builder -->|plans, pilots, and results| Review
  Review -->|fixes, caveats, and a verdict| Coord
  Builder -->|results for the write-up| Coord
  Coord -->|draft of the findings post| Editor
  Editor -->|catches token vs cost, median vs average| Coord
  Coord -->|revised draft| John

What kept the numbers honest

None of this is exotic. It's the same hygiene you'd want from a human experiment, just applied to bots that will happily keep going if you let them.

Lock the plan before you look at the data. The first flow's analysis plan was committed before the full runs. The scrolling flow's plan, including three hypotheses and a stop rule, was locked before anyone made an API call on it. When a scoring rule had to change after the pilot, that change was written down and committed before the full runs, not edited quietly after the fact.

Say what would count as a result, ahead of time. One pre-registered claim was only that the scrolling page's big-to-small token ratio would come in below the 2.19x from the no-scroll page. A ratio below 1 did show up later. The reviewer required that I call that observed, not pre-registered.

Run enough times, and do a small pilot first. My original brief asked for at least three runs per size. The reviewer wanted ten, interleaved, with a median and a spread. That mattered. On the no-scrolling page, 1280x800 used 1.091 times the tokens of 1024x768, but the confidence interval crossed 1, because two of those ten runs finished in four API calls instead of five and saved more than the extra pixels cost. With three runs, it would have been easy to call that a real increase. It wasn't a clear one.

Measure, don't assume. The builder calibrated each screenshot size from the API's own usage fields, rather than trusting the napkin formula. A zero-cost check confirmed the new harness still sent the same request bodies, byte for byte, when it ran the original flow.

And I owned the money. Hard stops sat under the $60 I had approved. The builder would not start a run, or a rerun, without my go-ahead in my own words. The reviewer then checked the reported numbers against the raw files. Trust, but verify.

A couple of limits got written down instead of wished away. Thinking can't be turned off on this model, so it ran at "adaptive, low," not "off." The API also rejects sampling settings like temperature. Those went into the locked plan.

What the process actually caught

Alright. Here's the part that earned the process.

Chrome opened one pixel short. On this virtual display, kiosk Chrome came up a pixel narrower and shorter than asked, leaving a thin black strip on the right and bottom. The builder caught it in a dry run, resized the window after startup, and confirmed the screenshot matched the viewport at all four sizes.

A scoring rule punished formatting, not failure. The plan tagged a run as a "refusal" if the model's final message didn't start with DONE. In the pilot, one run finished the task, passed the page check, and still got tagged, because it wrote a preamble and put DONE on the next line. That's what a pilot is for. The reviewer required a logged amendment before the full runs: call it a refusal only if the page check fails and the model declines the work or asks for permission. Under the old rule, four successful runs in the full set would have been marked as refusals just because of how the final message was formatted. The new rule tagged none of them.

Then there was the popup that almost became a finding. A dry run of the checkout form already showed Chrome's payment-autofill bubble, the "this form does not use a secure connection" notice. I left it in, because it appeared at every size. After the full runs, 1024x768 took 3.9 calls on average and a median of 28,850 tokens, with nine Escape presses and re-scrolls. The 2560 screen never scrolled and used exactly 23,693 input tokens every time, so the ratio came out 0.82x. The reviewer would not let that stand as a general law. I approved a separate rerun at 1024 and 1280 with the popup off. To be clear, that was a follow-up after the main results, not part of the pre-registered plan. All twenty runs passed. Every 1024 run now took three calls and 20,395 tokens, and the ratio moved to 1.16 times the input tokens, about 1.12 times the cost. The 0.82x figure was an artifact of my test setup. It is not a finding.

When the first draft of the findings post treated those 2.19x and 1.16x figures as if they were dollar ratios, the editor did the arithmetic on the table. $0.382 divided by $0.184 is about 2.1x, not 2.19x. $0.115 divided by $0.103 is about 1.12x, not 1.16x. A median had also been called an average. Both got fixed before I saw a finished draft.

I had already approved the scrolling flow in writing, under the same $60 cap, and then got asked about it again from another direction. After that, I had every experiment message go through the coordinator, so I wasn't getting the same question from several bots at once. If you're going to run a team of agents, give them one inbox that talks to you.

What I'd do again

I'd split the roles again. I'd lock the plan, pre-register the claims, put a dollar cap on the table, and have a reviewer who is not the builder open the files. I would not claim this proves that multi-agent setups are reliable in general. This was one day, two simple public pages, one model, and one browser. The process caught errors in this experiment. That's all I'm willing to say, and it's enough.

So, trust, but verify. If you've used a team of agents on a measurement you care about, what did your reviewer catch that the builder missed?

Top comments (0)