Every week I watch the same scene play out in a team chat or a review thread: an agent fails a task, someone rewrites the prompt, the agent fails again, and the verdict lands on the model rather than on the system around it. In the evaluation harnesses I have been building, that verdict is usually wrong, because the model is rarely the actual bottleneck. The bottleneck is the loop that connects the task, the patch, and the repository's own checks.
The loop, not the model
A benchmark measures a model in isolation, but your repository measures a model inside a system: task specification, patch generation, patch application, type checks, lint, tests, and the error text that flows back into the next attempt. Most teams tune the first and last steps of that system and treat the middle as fixed, which is exactly where agents actually die. A patch that does not apply, a typecheck that never runs, a test failure that is never fed back — each one is a broken link, and no model upgrade repairs a broken link.
There is a popular argument right now that constraints make better engineers, and another that writing less code is the whole point of adopting agents. Both conversations miss the mechanism that actually moves the needle, which is the speed and quality of feedback around each change. A mid-tier model inside a tight loop will outperform a frontier model inside a loop that drops its errors on the floor, and that is a claim you can verify in your own repository in an afternoon.
A loop you can run tonight
The artifact I keep coming back to is a small script that pushes one task through the repository's own checks and logs the outcome as a single row in a table. The agent command is deliberately a variable, because every setup differs; in my case, the slot is filled by MonkeyCode's free model access, which makes the iteration cost effectively zero.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
#!/usr/bin/env bash
# agent-loop.sh — run one agent task through the repo's own checks
set -euo pipefail
AGENT_CMD="${AGENT_CMD:-}" # command that writes a patch to stdout
TASK_FILE="${1:-task.md}"
BRANCH="agent-loop-$(date +%s)"
[ -n "$AGENT_CMD" ] || { echo "set AGENT_CMD to produce a patch"; exit 2; }
git checkout -b "$BRANCH"
# 1. The agent produces a patch
$AGENT_CMD "$(cat "$TASK_FILE")" > patch.diff
# 2. Apply only if it parses cleanly
git apply --check patch.diff || { echo "unparseable patch"; exit 1; }
git apply patch.diff
# 3. The repo's own checks, not a benchmark
# adjust these three lines to your stack (pytest, cargo test, go test, ...)
npm run typecheck > typecheck.log 2>&1 || true
npm run lint > lint.log 2>&1 || true
npm test > test.log 2>&1 || true
# 4. Log the prompt hash and failure count
PROMPT_HASH=$(sha256sum "$TASK_FILE" | cut -c1-12)
FAILS=$(grep -cE '^(FAIL|✗)' test.log || true)
printf '%s\t%s\t%s\n' "$(date +%F)" "$PROMPT_HASH" "$FAILS" >> loop-results.tsv
git checkout main
git branch -D "$BRANCH"
The key design choice is step three, because the loop does not run a benchmark suite; it runs the checks your team already relies on every day. That makes the result meaningful for your codebase and useless as a general claim about the model, which is exactly the trade you want from a diagnostic tool. The task file matters just as much: a good one states the expected behavior, the files you suspect are involved, and one acceptance criterion that the tests can confirm.
Schedule it on a free server
A loop that only runs when you remember to run it is not a loop; it is a chore, and it will die within two weeks. The second piece of the setup is a scheduled job on a free server, which turns the script into a nightly regression probe that runs without anyone's laptop:
0 2 * * * cd /srv/agent-loop && ./agent-loop.sh tasks/regression-01.md
After a week you have a small table with one row per run, and a quick aggregation shows whether the same task keeps failing across time:
awk -F'\t' '{print $2, $3}' loop-results.tsv | sort | uniq -c
A stable failure count across runs means the task specification is the problem, while a count that bounces around means model variance, and a count that trends down means the loop is working.
What the results tell you
| Symptom | Likely cause | Fix |
|---|---|---|
| Same test fails across runs and models | Task spec is ambiguous | Add acceptance criteria to the task file |
| Patch never applies | Wrong base commit or oversized task | Pin the base commit, shrink the task |
| Typecheck fails every time | Agent cannot see the type surface | Feed typecheck.log back into the next attempt |
| Failures are random on easy tasks | Model variance | Run the task several times and compare distributions |
The table is the argument in miniature, because most agent failures are diagnosed by looking at the loop rather than at a model card. When you treat the loop as the unit of analysis, the fix is usually a change in the task file, the base commit, or the feedback text — none of which requires a model upgrade.
Why this is an opinion, not a benchmark
The industry optimizes what it can measure, and model quality is easy to measure, while loop quality is repository-specific, slow to evaluate, and boring to write about, so nobody benchmarks it. That asymmetry is exactly why the leverage sits on the boring side, and why I keep arguing that teams should spend their token budget on loop experiments instead of model comparisons. Free model access changes the economics of this argument, because when each iteration costs nothing, you can afford to run the same task twenty times and tune the feedback instead of the prompt. A free server changes the logistics, because the loop runs nightly without your laptop, your VPN, or your attention, and that is the difference between an experiment and a habit.
This is also why MonkeyCode, an open-source project, is genuinely useful here: its free model access with 10 million tokens and its free server option are not the strongest tools in their category, but they make the loop sustainable long enough to produce data you can trust. I would rather have two weeks of nightly results from a free tier than one afternoon of impressive demos from a paid one.
Limitations
- The loop is only as good as your checks: a thirty-minute test suite makes it impractical, and a repository with no tests leaves you with weak signals from lint and typecheck alone.
- Free server tiers come with resource limits, so heavy builds will not fit, and the results describe your repository only, so they do not generalize to other codebases.
- Skip this approach if your team has no CI discipline, or if your agent work is long-running multi-file refactors where a single patch-apply cycle cannot capture the actual work.
- For that kind of task, the loop needs to be built around review and human judgment, not around an automated apply step.
If you want to see this loop running against real tasks, the free tier is enough to try it for a week; the script above is the entire setup.
Top comments (0)