The AI says the failing tests are fixed. You run npm test and two are still red. So you paste the output back, it apologises, changes something, says "fixed" again, and the cycle repeats with you as the only person who ever actually runs the tests.
The fix is to take the word "fixed" out of the AI's hands. Let it work in rounds, run the test suite after each round, and let the exit code of npm test decide whether it is done. The osFoundry desktop app ships a ready-made loop for exactly this, and it takes three steps to start.
1. Pick the "Get the JS suite green" loop
Open Loops and click New session. The dialog lists the starter loops that come with the app. Type "JS suite" in the search box and select Get the JS suite green. There are siblings for other stacks too (Go, Java, Python and Rust), found by searching "suite green".
Before running anything, read the card on the right, because it says how the loop decides it is finished. Under "How it grades itself" it says: Runs npm test — a zero exit code means done.
That one line is the whole point. The loop isn't finished when the AI believes it is finished. Inside a run, each round is graded against the goal by a separate grading call, not by the agent doing the work, and when the run ends the app runs npm test itself. If the AI writes "all tests pass" and npm test exits with 1, the run is recorded as failed verification, however sure the AI sounded.
The goal text matters too: "For each failing test, find why it fails and fix the cause. Never edit an assertion to match wrong behaviour — if a test is genuinely wrong, say so and stop." The quickest way for any AI to turn a suite green is to change what the tests expect, so the loop's instructions forbid it.
2. Run it once, in your project folder
Under Start it, choose Run once now. "Just add it" creates the loop without running it, which is useful later when you want to attach a schedule. Pick a project to file the run under, then set Works in to the folder of the repository with the failing tests. This matters. If you leave it on "Each agent's own folder", the loop runs commands in whatever folder each agent happens to have, and npm test would test the wrong code.
Press Run it now. The new session shows up in the sessions list within a few seconds, running on the models this session uses.
It can't run forever. The card shows the two limits: at most 4 rounds, and it stops after 3 failures in a row. Both are set in the loop's definition. Four rounds is the budget for one run, and the failure cap stops the loop from re-arming itself when its verification keeps failing, so a problem that is really a design question, not a mechanical fix, doesn't keep spending tokens.
The header of the session keeps both counters in view: which round it is on, and how many verification failures in a row it has had.
3. Watch the rounds
Open the Rounds tab. Each round is a row, with how many of the goal's criteria were met and a +1 when a round made progress. Expand a round to see which tests it got passing.
In this example run, round 1 fixed one of three failing tests, so the loop went again. Round 2 got two, round 3 got all three, and the loop stopped on its own without using its fourth round. The session in the list now says Verified, meaning the check passed, not that the AI said so.
If the loop runs out of rounds or hits the failure cap instead, the session doesn't say Verified, and the rounds show where it got stuck. That is usually the moment to read the remaining failure yourself: either the test is wrong (and the goal tells the AI to say so rather than edit it) or the fix needs a decision a test runner can't make.
What the check actually does
The verification command runs through the same command tool the agents use, in the folder you chose, and the app reads the exit code it reports. Zero is a pass, anything else is a fail, and a command that times out counts as a fail even if it reports 0. A check that couldn't run at all, for example because the folder wasn't reachable, isn't treated as your code failing. For that run the app falls back to the goal grader's verdict, the one the Rounds tab shows, and runs that stopped for reasons that say nothing about the work, such as a cancel, don't count toward the three failures.
If the loop's tool list is narrowed, it has to include the command tool for the check to run. The starter loop already does. The tool scope page explains how an agent's tools are limited, and agent sessions covers following long runs like this one. The app is free from the osFoundry download page.
Top comments (0)