Semantic Loop Detection: Catching Stuck AI Agents

It is 2am. The agent has burned 40k tokens and reverted the same file four times, and from where I am sitting it looks like it is working hard. That is the part that fooled me. It was busy. Every loop produced a new patch, a new diff, a new paragraph of reasoning about why this time would be different. The log scrolled. Things were happening. The agent just was not getting anywhere.

The task was a bug fix. Generate a patch, run the tests, watch them fail. Read the error, generate another patch with different variable names and different line numbers and the exact same underlying logic. Tests fail again. Third attempt, it wraps the fix in a try/except. Still fails. Same root cause, untouched. By attempt seven it had written and reverted the same file four times and was no closer than it had been on attempt one. I was watching a machine spend my tokens to stand perfectly still.

The thing I could not get past was that my loop detector said everything was fine.

The detector that lied to me

I had a guard for exactly this. A set of seen actions, each action hashed to a string, and the rule was simple: if the same action shows up twice, you are looping, halt. It had caught dumb loops before. So I trusted it.

It reported no loop. Not once across seven attempts. And it was technically correct, which is the worst kind of correct. Each action string really was different. "Change the timeout from 30 to 60." "Set the timeout to 60 seconds where it was 30." "Apply the same value at the call site instead." Three different strings, three different SHA-256 hashes, three entries the set saw as three distinct actions. The agent was not repeating itself, not in the only sense my detector understood. It was absolutely, structurally stuck, and my guard waved it through every single time.

That is the moment that got under my skin. The hash changes when the text changes. That is the entire failure mode. My detector was measuring whether the words moved, when the only thing I cared about was whether the situation moved. Those are not the same question, and I had been treating them as if they were.

I sat with the three timeout edits for a while. Look at what actually happened underneath them. The first two are the same edit with the prose rephrased. The third shifts where the edit lands but leaves the real bug, a hardcoded default upstream, completely untouched. The next test run would throw the identical error. Three "different" actions, one unchanged reality. The string was new. The progress was zero. My guard could only see the first one.

The thing I had been measuring wrong

Once I named it, I could not unsee it. There are two different things going on every time an agent acts, and I had collapsed them into one. There is syntactic novelty, which is just "the string changed." And there is semantic progress, which is "the underlying situation changed." Loop detection has to live at the second level. Hashing the action string only ever sees the first.

So the question stopped being "how do I make a better hash of the action" and became "what is the smallest honest description of what this agent is actually doing." Not how it described the work. What the work was. That reframing is the whole turn. Everything after it is bookkeeping.

This is where I stopped reinventing and went looking, and found that KARIMO (github.com/opensesh/KARIMO) had already published the shape of the answer: a four-dimensional fingerprint. Instead of hashing the action string, you describe the action along four axes and hash that. Each axis catches a different way an agent goes nowhere.

The first axis is a coarse action class. Not the full action string, just the category: a file edit, a test run, a shell command, a search. This is the one that would have saved me at 2am. My three timeout edits, with all their rephrased prose, collapse into the same category the instant you stop caring about the words. The phrasing difference simply disappears.

The second axis is where the work is landing, a normalized hash of the files involved. An agent that keeps touching the same two files on every attempt is structurally looping no matter what story it tells about each pass.

The third axis is the state of the world, supplied by the caller. A hash of whatever counts as the agent's environment: test results, file checksums, the context it is working against. This is the brutal one. If the state hash has not moved between two actions, then by definition nothing the agent did had any effect. Two identical state hashes across two different action strings is not a hint. It is proof.

The fourth axis is the one I wish I had had first, and it is the key to the bug-fix stall specifically. You take the error output and strip the parts that wander without meaning anything: line numbers collapse to a placeholder, memory addresses collapse, absolute paths reduce to bare filenames, timestamps collapse. What survives is the error type and the message template. So AssertionError at line 47 and AssertionError at line 52, after a refactor nudged the lines, normalize to the same thing. The agent is still hitting the same wall even when the traceback shifts under it. My old detector treated that shift as progress. It was noise.

Combine the four into one tuple, hash it, and you have a fingerprint that holds still under rephrasing and only moves when something real moves. That is the whole insight, and it is almost embarrassing how much calmer the problem felt once I had it.

Watching it catch the thing that fooled me

The first time I ran the new detector against the same stuck pattern, it was quietly satisfying in a way debugging rarely is.

Picture the five attempts. The same config file, the same timeout assertion failing, the environment state hash never moving an inch. The agent narrates a different story on every pass: change the timeout from 30 to 60, set it to 60 seconds, bump the request timeout parameter, update it in config, fix it again because the last fix was incomplete. Five confident, distinct sentences. But each one feeds the detector the same action class, the same file, the same state, the same normalized error. Five times.

The old seen-actions set looks at those five strings, counts five unique entries, and stays silent. The fingerprint looks at the same five and sees one identical tuple, over and over. And because detection without a response is just logging, the fingerprint is paired with a count. KARIMO's pattern is a two-step ladder. At three matches, escalate: the current approach is dead, but a stronger model or a different prompt might break the wall, so hand it up. At five, stop and surface to a human, because the stall has now survived escalation and is just burning money.

So the streak climbs. The first two attempts pass as routine. The third trips the escalate threshold and the detector says try something stronger. The fourth stays escalated. The fifth hits the halt and kicks it to me. Five syntactically distinct actions, one semantic fingerprint, and for the first time the guard actually saw what I had seen at 2am.

The two-step design is not arbitrary either, and I came around to why. The costs are asymmetric. A false alarm at three costs one unnecessary escalation, slightly more expensive, but the agent keeps going. A false alarm at five interrupts a run that might have finished, while a missed stall lets an agent burn resources with no path forward at all. That asymmetry is what makes five a defensible place to halt for most tasks and three a cheap early warning you can afford to be wrong about.

Where it still gets it wrong

I want to be straight about the limits, because this is a heuristic, not a truth oracle, and pretending otherwise is how you ship a guard that fights your own agent.

Intentional retries will trip it. Exponential backoff on a flaky API, or a polling loop waiting on a job, will look exactly like a stall if the error and the state stay stable across attempts. The fix is to loosen the threshold for those paths, reset the detector on known retry patterns, or hand them a different action class so they cannot pile onto the same streak.

Planned iteration is the friendlier case. A write-test, run, fix, run cycle produces alternating fingerprints and will not trip the streak, which is correct. But if the fix step itself stalls while the test and run steps keep cycling around it, the alternation hides the problem, and you may need a second detector scoped to just the fix-class actions.

And the thresholds are task-specific. An agent doing exploratory code search might legitimately re-examine the same files three times from three different angles. Three is far too tight there; seven or eight is more honest. Wire the thresholds through config and tune them per task type rather than baking in numbers that were only ever right for one kind of work.

None of that undoes the core move. The fingerprint pattern I leaned on here, the combination of action class, files touched, state, and normalized error, is published by KARIMO at github.com/opensesh/KARIMO; check the current license at the repo before you reuse it. What I took from it was not code. It was the reframing.

That is the thing I keep coming back to. The bug was never in the agent. The agent was doing what stuck things do, churning. The bug was in me, in what I had chosen to measure. I had built a detector that asked whether the words were changing, when the only question that ever mattered was whether anything real was. If your agent can spend 40k tokens looking productive while standing still, your loop detector is measuring the wrong thing, and it will keep lying to you politely until you teach it to look underneath the words.