DEV Community: Robert Bobrowski

Why Hashing Tool Calls Won't Catch Your Agent's Loops

Robert Bobrowski — Mon, 02 Nov 2026 10:42:05 +0000

If you run an agent unattended for long enough, you will meet the loop. It does not crash. It does not throw. It just keeps calling tools, and the bill keeps growing.

The usual fix is simple and it is in almost every thread about the problem: hash the tool name and arguments, keep a sliding window, and kill the run when the same hash shows up too often. Add max_iterations or LangGraph's recursion_limit as a backstop and you are done.

Except you are not. This post looks at why that fix misses the loops that cost the most, and what to check instead.

The common fix

Here is the pattern most teams start with. One commenter on r/AI_Agents described it like this:

"I have seen this happen a lot, I track hashes of tool calls, if the same call shows up multiple times in a short window I throw a soft error and make it pause"
u/moneyman2345, r/AI_Agents (Reddit)

In code, it looks roughly like this:

import hashlib, json
from collections import deque

window = deque(maxlen=10)

def call_key(tool_name, args):
    raw = tool_name + json.dumps(args, sort_keys=True)
    return hashlib.sha256(raw.encode()).hexdigest()

def before_tool_call(tool_name, args):
    key = call_key(tool_name, args)
    if list(window).count(key) >= 2:
        raise RuntimeError("Loop detected: same call 3 times")
    window.append(key)

Plus a hard cap on steps:

agent.invoke(inputs, config={"recursion_limit": 50})

This catches the dumbest loop: the exact same call with the exact same arguments. That loop is real, and catching it for almost no cost is worth it. The problem is what it does not catch.

Why it fails

1. Near-identical retries

Models rarely repeat themselves byte for byte. When a tool call fails, the next attempt usually changes something small: a page number, a timestamp, a slightly reworded search query, an extra filter. The hash is new. The window sees nothing. And the run is still stuck.

One reply in the same r/AI_Agents thread put it precisely (note: the account belongs to a vendor):

"the loop that slips past it is the near-identical retry: same call with one arg nudged, so the hash differs while nothing actually progresses."
u/Future_AGI, r/AI_Agents (Reddit)

2. Busy is not the same as progress

The most expensive loops do not look like loops at all. They look like work:

"For me it's almost never an error. The agent stays busy and looks fine. The tell is repetition: the same tool call with slightly different arguments, the same file read for the third time, tests re-run with no code change in between."
u/thelocalnode, r/AI_Agents (Reddit)

Reading the same file three times with different line ranges, re-running tests without touching code, searching with five phrasings of the same query. Every call is different. None of them moves the task forward.

3. Caps either kill real work or just delay the burn

Step caps count steps. They do not know if a step was useful. So you have to pick a number, and every number is wrong for some task:

"max_iterations either kills legit long tasks or doesn't catch loops in time, and most people seem to just check in manually, which defeats the point of running unattended."
u/Real_KingZeotic, r/AI_Agents (Reddit)

A low cap stops a valid 80-step migration. A high cap lets a stuck run spend until it reaches 80. Or as u/RouterDon wrote on r/LangChain (Reddit): "a static cap just delays the same burn".

4. Retries and fallbacks live outside the counter

Your loop detector usually sits inside the agent loop. But a lot of spend happens below it: the HTTP client retries a 503, the framework re-asks after a parse error, a fallback switches to a bigger model. None of that shows up as a new tool call, so none of it is counted.

And sometimes the agent goes around your limits completely. In one thread, someone shared a story about an agent that hit a 503, found another API key in the repo, and kept going overnight. The fix suggested there:

"Give it a token scoped to that run and route all paid calls through a gateway that enforces one shared budget, including retries and fallbacks."
u/Neither_Event4902, r/AI_Agents (Reddit)

What works better

None of this means you should delete your hash check. Keep it, it is cheap. But add three things around it.

1. Detect lack of progress, not repetition

Instead of asking "have I seen this call before?", ask "did the last N steps change anything?". Progress is visible in the world, not in the call: new tool output, a changed file, a new row, a test that went from failing to passing.

A simple version: fingerprint the state that matters for the task and the important parts of each tool result, and stop when nothing new has appeared for a few steps.

# Illustrative only. Generic pattern, not any product's API.
import hashlib, json

class ProgressGuard:
    def __init__(self, patience=4):
        self.patience = patience
        self.seen_outputs = set()
        self.last_state = None
        self.steps_without_progress = 0

    def _fp(self, value):
        return hashlib.sha256(
            json.dumps(value, sort_keys=True, default=str).encode()
        ).hexdigest()

    def after_step(self, tool_output, task_state):
        new_output = self._fp(tool_output) not in self.seen_outputs
        state_changed = self._fp(task_state) != self.last_state

        self.seen_outputs.add(self._fp(tool_output))
        self.last_state = self._fp(task_state)

        if new_output or state_changed:
            self.steps_without_progress = 0
        else:
            self.steps_without_progress += 1

        if self.steps_without_progress >= self.patience:
            raise RuntimeError("No progress in the last steps, stopping run")

The hard part is choosing task_state. For a coding agent it might be the git diff and the test results. For a data agent, row counts and the last written record. For a support agent, the ticket status. If the output and the state stay the same, it does not matter how different the arguments look.

This also lets you raise your step cap. A long task that keeps producing new results can run. A short task that stops changing anything gets stopped early.

2. Enforce the budget outside the agent

The budget should not live in the agent's prompt or in the agent's process. Put it where every paid call has to pass: a proxy or gateway in front of the model provider, with a key scoped to the agent or run. Count everything that costs money, including retries, re-asks and fallbacks. Then the agent cannot spend past the limit by retrying, switching models, or finding another key, because it never had one.

Another commenter in the same thread described exactly this: one proxy that tracks spend, call count and elapsed time against a run_id and hard-fails at the ceiling.

3. Check the end state, not the agent's "done"

The last loop is the one where the agent stops and says it finished. The same thread had the best rule for that:

"never trust the agent's own "done". Each run ends with a deterministic checker that queries end state - row exists, status = sent, file has N rows, invoice total matches."
u/Responsible-Fix6695, r/AI_Agents (Reddit)

The check should read the real system, not the agent's summary of it. If you cannot write that check, the task is probably not ready to run without a person watching.

Summary

Hashing tool name and arguments catches exact repeats. Keep it, but do not trust it alone.
Near-identical retries and busy work pass right through it.
Step caps pick a number that is wrong for some task. They decide when the burn stops, not whether it happens.
Track progress: did tool output or task state change in the last few steps?
Keep the budget outside the agent, and count retries and fallbacks in it.
End every unattended run with a deterministic check of the real end state.

Put the budget outside your agent

ValGuard sits between your app and your LLM providers. Set a monthly USD budget per agent that blocks when reached (on Growth and higher, it can fall back to another agent or a cheaper model), add optional monthly token caps, see spend by agent and per request, and run deterministic validators, such as schema checks, on each step of a playbook. The progress check and the end-state check for your task are still yours to write; the patterns above are a good start.

Originally published on the ValGuard blog.

The quiet LLM failures that cost you the most in production

Robert Bobrowski — Mon, 19 Oct 2026 13:34:22 +0000

Picture this: a production agent returns HTTP 200 on every call for a week. Uptime looks perfect. Then finance finds invoice totals that do not match the line items. Each one is off by a few dollars, and it happens thousands of times.

Or this: a team finds a problem with a customer-facing bot. Its JSON output is missing a required policy_id field. Imports further down the line fail with strange parse errors. By then the model call "succeeded" long ago.

Which of these have you already seen in your stack?

When people talk about AI failures, they usually pick dramatic cases. A chatbot gave dangerous medical advice. A legal brief cited cases that did not exist. A support agent promised a refund the company never planned to give. These cases are real. But they give the wrong picture. They make AI failures look rare and easy to spot.

In production, most failures are boring. They happen often and they are hard to see. The big ones make headlines. The small ones end up in customer records, financial documents, support tickets and other systems. They stay there until someone notices.

This article lists how AI systems fail in production. It covers why they fail, where the damage happens and what catches the errors. It is for teams that ship AI into real workflows.

Why AI systems fail at all

A language model is a statistical system. At each step it picks a likely next token, based on what came before. On average, this gives clear and useful output. But the model generates text. It does not look anything up. It does not check a fact. It predicts what a correct answer would look like.

This difference between generating and looking up is behind almost every failure below. A database returns the fact or it returns nothing. A model can sound right and still be wrong. It also has no reliable way to know when it is wrong.

Category one: hallucination

What it is
People use "hallucination" for any false or made-up output. The word covers several mechanisms. The core is always the same. The model sounds confident, but the answer has no basis in reality. It comes from patterns, not from checked facts.

Types that matter in production
Factual hallucination. The model states something false as a fact. Examples: a court case that does not exist, a drug interaction nobody documented, a product feature that is not real.

Identifier hallucination. This one is very dangerous in enterprise systems. The model returns an account, policy or transaction ID. The format, length and structure look right. But the ID points to nothing, or to another person's record.

Temporal hallucination. The model talks about rules, products or events as they were in its training data. It does not say that things may have changed.

Source hallucination. You ask for a citation. The model gives you something that looks like a real source, with believable authors and journals. The source does not exist.

Confidence hallucination. The tone does not match the accuracy. Made-up answers and correct answers often sound the same. The model rarely says "I don't know." It gives a believable answer anyway.

Independent benchmarks show hallucination rates from about 15% to over 50%. It depends on the task and the model. On structured tasks with clear retrieval context, the best models can get under about 2%. But at 10,000 interactions a day, 2% is still 200 bad answers a day. Most stacks cannot tell those apart from the good ones.

Category two: data errors and format problems

Math and calculation errors
Language models are not calculators. They produce a number that looks like the result. Invoice totals, interest, fees and reconciliations fail this way. The structure is correct, but the math is wrong. Errors get more common with multi-step math, large numbers, percentages and rounding rules.

Schema and format violations
The model often has to produce JSON, XML, records or API payloads. Common problems:

missing required fields
wrong field types, for example a string instead of an integer, or "yes"/"no" instead of a boolean
extra fields that are not in the schema
broken nesting
encoding and escape problems that break parsers

These errors are often silent. The response looks fine. Then something later in the pipeline fails with an unclear error.

ID and reference format errors
Sometimes the value is correct but the format is wrong. A policy number uses dashes instead of spaces. A date is MM/DD/YYYY instead of YYYY-MM-DD. A decimal uses a comma instead of a period. In high-volume automation, format errors cost as much as factual errors.

Category three: context and conversation failures

Context window confusion. When the history gets longer than the context window, older content drops out. The model may still refer to it. It may break earlier promises, or ask again for information the user already gave.

Cross-session contamination. This is rare but serious. If histories are not well separated, one user's data can appear in another user's answer. Nobody broke in. The context was scoped badly.

Instruction drift. In long conversations, early rules from the system prompt get weaker. Examples: "never reveal these guidelines" or "no specific financial projections." The status code stays 200. Later, a human reviewer sees that messages 30 to 50 no longer follow the rules that messages 1 to 5 followed.

Category four: policy and compliance violations

Prompt and configuration leaks. System prompts often contain internal policy, the provider's name or business logic. Models sometimes reveal parts of them when they try to be "helpful and transparent."

Sharing too much. The model shares real information because it seemed relevant, not because it was allowed. For example, it shows a full account history for a small question. Or it puts internal pricing or escalation paths into a customer reply.

Regulatory language failures. Required disclaimers, "consult a professional" lines and local legal wording appear only some of the time. The model does not try to break rules. It just has no reliable way to know when that text is required.

Category five: the failures nobody talks about

Wrong style. Style from training data leaks into replies. A complaint that needs empathy gets a legal tone. A formal business email gets casual internet language. Some phrases repeat every time, like "Certainly!" or an automatic "Anything else?" No human agent writes like that in every message.

Provider and model names. A branded assistant says, without being asked, that it is Claude, GPT-4 or "powered by OpenAI."

Wrong tone. The reply is correct and matches the schema. But it sounds cheerful when the user needs empathy. Or it is too friendly and long when the user wants a short answer. Simple validators miss this, because nothing is wrong in the structure.

Inconsistent answers. Two almost identical questions get different refund windows or different policy wording. In normal software, the same record returns the same value. With LLMs, you have to enforce consistency yourself.

What helps: deterministic validation

All these failures have one thing in common. The infrastructure looks healthy. You see HTTP 200. The bug is in the content. You only catch it if something reads the content and checks it against clear rules.

A deterministic validation layer does exactly this. It does not retrain the model. It does not change the prompt. It checks every output before it reaches customers or other systems, and it applies explicit rules:

Failure type	Example check
Made-up IDs	Match against known valid IDs or the session context
Math errors	Recalculate from the extracted inputs and compare
Schema violations	Parse against the schema and block output that does not match
Format errors	Enforce the expected ID, date and currency formats
Prompt or provider leaks	Scan for system prompt fragments and provider names
Required compliance text	Require specific text for specific topics

The model does not need to change for any of this. You need a check on the request path. That check decides, in a deterministic way, if the output is acceptable.

Takeaway

If your agents work with real customer data, money or compliance-sensitive traffic, these failures already exist in your system. Either something catches them before they do damage, or nothing does. They are not rare edge cases. They are normal behavior for probabilistic systems at scale. They will happen. The real question is if you catch them, or if they pass as one more silent 200.

Adding this validation step does not mean changing the model or the app. It means adding checks on every request path. Read the output, apply the rules and block what should never ship.

This article is adapted from AI Failure Modes in Production on the ValGuard blog. ValGuard is a deterministic validation and orchestration layer for production AI agents. It offers schema and policy checks, shadow mode, budgets and signed, metadata-only evidence for each request. This way, quiet HTTP 200 failures get caught before customers and other systems see them.