Tool results are the most overlooked source of guidance for an LLM. Prompt engineering gets the attention, but the prompt states the job once — tool results arrive on every call, and each one is a chance to steer the model while it works. If you're building a harness, four changes to what your tools return will do more than most prompt revisions.
- Never raise — return something the model can read
- Put the near-miss inside the refusal
- Annotate success, not just failure
- Count the small failures and act on the pattern
Each of these exists to fix a problem the previous one leaves behind, so they are worth building in this order. Our harness is the worked example throughout — every mechanism described is one running in production.
Never raise — return something the model can read
The baseline problem is an agent that builds on garbage. A tool result that quietly failed gets trusted, and the error surfaces far from its cause; the blunt version is an unhandled exception, where the tool crashes, the loop dies, and the model never gets to react at all. Both failures share a root — the model was never handed anything it could read.
So make it a contract that every tool returns a structured result, never an exception. In ours, the completion and error paths both produce a result object with a success flag and, on failure, the error itself. A failed call becomes text the model reads on its next turn, and the run keeps its footing either way. That is the hard no — loud, explicit, and survivable.
Put the near-miss inside the refusal
A hard no alone still has a cost. A bare failure — "edit could not be applied" — forces the model to guess what went wrong, and the retry it burns is as blind as the first attempt. The retry is only ever as good as the information in the refusal, and your harness usually holds more information than it is passing on.
Edits are a concrete case. Ours applies them by matching an anchor block of text in the file and replacing it, which means a failed match comes with a near-miss attached — so the result carries the closest text that was actually found and a confidence score for how close it came. The model's second attempt is aimed at the real mismatch, usually a stale line or drifted whitespace, instead of being a fresh roll of the dice. An explanatory no turns a wasted retry into an aimed one.
Annotate success, not just failure
Once failures are handled, success becomes the place where the quieter damage accumulates. A write that lands cleanly can still leave problems behind that no compiler complains about — a new warning from the linter, the static checker that flags style errors and suspicious code, or a brace that lost its closing pair, or a TODO comment nobody asked for. None of that is visible in an "ok", which means success needs annotating just as much as failure does.
Attach the report to the success result itself. A successful write in our harness comes back stating how the write changed the linter's count of problems in that file, whether every bracket still balances, what TODO comments appeared, and the file's new length — so the model learns what its write left behind in the same result that confirms the write, without spending a turn re-reading the file. Reads work the same way, arriving with the linter's current findings and an outline of the file's functions and classes, so the state of the code stays continuously in front of the model rather than being something it must remember to check.
Give the feedback a standard to push against
Annotations alone are trivia. A lint warning only steers a model that owes somebody clean code — feedback and standards are two halves of one mechanism, and shipping either without the other does little.
Put the standard in the agent's identity, in the system prompt. Our code agent's includes lines to the effect of "you do not skip pre-commit checks" — the lint and test hooks that run before code is allowed into a commit — and "you do not modify tests to make them pass". Against that identity, a tool result reporting new lint warnings is not information — it is an unmet obligation the agent has already agreed to.
Count the small failures and act on the pattern
One gap remains, and it is the difference between an event and a pattern. A model can acknowledge a lint warning, fail to fix it, and move on — and with each result judged in isolation, nothing stops a slow leak of small failures from running the whole session aground.
So make the nos accumulate. Each of our agents carries circuit breakers — running counts of non-fatal issues kept across the whole run — and a count that gets high enough trips its breaker — lint failures on the third, type-check failures likewise, pattern violations after five. A tripped breaker doesn't abort the conversation; it changes how the agent proceeds from there. Retry budgets bound the loop from the other side, so a step that keeps failing is marked failed rather than retried forever. A single no is an event; the breakers are what turn a pattern of nos into a diagnosis.
What tool feedback cannot do
Lint measures shape, not judgment. A design mistake lints clean, and no verification report will flag a function that is well-formed, type-safe, and wrong. That class of problem is not solved by tool feedback in our harness or anyone's — it is why review exists as its own step, run by separate agents with their own gates.
The tool result is a prompt surface, in any harness. If your feedback only comes in "ok" and an exception, the model is steering on one bit per call, and everything between those two poles — the aimed retry, the yes-with-caveats, the accumulated pattern — is bandwidth you already paid for.
Where your harness stands
Three questions locate any harness on this map. When a tool fails, does the model receive something it can actually read? When an edit misses, does the refusal say how close it came? When a write succeeds, does anything tell the model what it left behind? Every no is a place a harness is spending retries and turns that graded feedback would hand back — and the quiet no, the one most harnesses skip entirely, is the one that pays the most.
Favur is our multi-agent software team — the harness these mechanisms come from. It is closed source and invite-only, but the repositories it produces are open, and you can drive a replay of a real run at https://favur.dev/go/devto/tool-feedback or see the same harness scored across models at https://evals.favur.dev/go/devto/tool-feedback. I work on it, so weigh the framing accordingly.
Top comments (0)