Your Agent Marks the Task Done When the Unit Tests Pass — It Never Started the App

Last week I watched an agent close a ticket labeled "fix login timeout." It had rewritten the retry loop, added backoff, updated the unit tests, and green-lit the branch. The PR description was confident. The CI was green.

I clicked "run" on the local dev server, opened the login page, typed a fake email, and waited. The spinner went forever. Same bug. The unit tests were green because they only exercised the retry function in isolation — they never actually booted the HTTP server, never hit the real route, never rendered the page.

This is a structural failure mode, not a one-off.

Why the agent stops at the unit test

An agent works inside a feedback loop you set up. If the loop is "run npm test," it will optimize for npm test going green. It will not, unprompted, also:

• Boot the dev server
• Wait for the port to come up
• Open a headless browser
• Click the login button
• Wait for a response
• Tell you the actual status code

None of that is in the test command. So it isn't in the agent's world.

The unit tests are green because the agent wrote them against the unit it just refactored. The integration boundary — where the retry loop meets the real HTTP client meets the real auth endpoint meets the real frontend — is a no-man's-land the agent never has to cross.

The cheap gate

You don't need a full end-to-end suite to close this gap. You need one observable thing that proves the app actually started and the route actually responded.

The cheapest version I've landed on:

1. After the agent says "done," ask it to boot the app in the background.
2. Curl the route it just touched. Print the status code and the first line of the response body.
3. If the boot fails or the curl times out, the task is not done — no matter what the unit tests say.

That's it. Three lines of shell. It catches the class of bug where the unit test passes and the route 500s because of a missing env var, a bad import, or a port-binding typo the unit tests never see.

When the work touches API behavior specifically, the gate gets narrower: re-run the spec against the live endpoint and look for a mismatch between what the OpenAPI says and what the server returns. We built that loop directly into Powerduck because I kept doing it by hand — boot the server, hit each documented path, diff the response against the spec, and only call the PR done when the two agree.

The trap of trusting the green tick

The deeper mistake is reading "tests passed" as "the thing works." They are two different claims:

• Tests passed: the code the agent wrote behaves the way the tests expect.
• The thing works: a user can actually do the thing the ticket asked for.

The second claim requires the app to be running. The first doesn't.

Agents will keep delivering the first claim until you require the second. Until then, green CI is a report about the unit test file, not about the product.

What to change in the prompt

The single prompt edit that moved the needle for me: replace "make the tests pass" with "boot the app, exercise the route, paste the output." The first sentence optimizes for the test file. The second optimizes for the running system.

It's a small shift. It moves the agent's definition of "done" from the test runner to the thing you actually care about.

Comments

[Kent Bodrov]

For the login-timeout bug you opened with, the route check could still pass while the spinner stays forever. I’d keep one browser test that makes the auth request time out and asserts that the spinner clears, an error appears, and the user can retry. That checks the failure the ticket describes. Starting the server and getting a valid HTTP response is useful evidence, but it doesn’t exercise that recovery path.

[Preetest Team]

This matches what we see with agents in a different place: not "did the server start" but "did the page the agent never opened still work". The cheap gate you describe (boot it, curl the route, print the status) catches the dead server. The next cheap gate, same spirit, catches the live server that is quietly wrong:

1. curl -sI the route and look at the status AND the headers. An agent that "fixed login" by moving the token into the query string will return 200 with the secret in the URL. We had exactly that in one of our own apps (an SSO token in ?t=) and every unit test was green.
2. curl an unknown path. If it answers 200, the router the agent wrote swallows every error as the home page, and your "it works" check cannot distinguish a working route from a fallback.
3. Open the page at 390 px once. Agents design at the width of their own preview pane; the bottom nav on one of ours had its last item clipped off-screen and nothing in CI knew.

Disclosure: I'm on the team behind Preetest, an AI website-testing platform by Yamama Software, so I am biased toward "test the deployed URL, not the function". But the three lines above need no tool, and they fit in the same stop hook you are describing.