I Scored Duplicate Tool Calls. Busy Was Not Progress.

The agent was not stuck. It said so. It had a tool trace, a confident summary, and three searches that only a slide deck would call iteration. I scored the repeats. Busy was not progress.

You know this transcript. The model calls search_docs with {"q": "rate limit"}. Then it calls it again. Then it swaps in "rate-limit" and walks the same hallway like the paint is new. Chat still reads as motion. The bill still reads as work. Was any of it?

I stopped arguing with the summary and started scoring the trace.

The rule is rude on purpose. If two tool calls share a name and the same arguments after canonical JSON, the second call is a stall. If the name matches and the arguments are a costume change—hyphens, plurals, the same question in a nicer coat—it might still be a stall. If the arguments actually move the search space, that is progress. Chat is not the judge of that. The trace is.

I wanted a harness a code review could replay. No cluster. No dashboard glow. A Python file, a folder of fixture transcripts, and a score that can fail CI. The fixtures are boring, which is the point. They grade the scorer. The scorer does not get to grade itself.

The hallway test

Think of an agent loop like a night guard. A useful patrol changes rooms. A useless one stamps the same doorway and files it as a shift. Token spend does not tell you which shift you bought. Duplicate tool calls do.

I split the score into two channels on purpose. Channel A is exact. It canonicalizes name plus json.dumps(args, sort_keys=True) and counts how many calls collide with an earlier key. Channel B is optional and meaner: a model that only answers whether the second argument blob explores anything new. Channel B does not own the gate. It writes an opinion. Disagreement is what I actually want in the log.

Here is channel A. It has no personality, which is why I trust it in CI.

# halt_score.py
from __future__ import annotations

import json
from pathlib import Path
from typing import Any

THRESHOLD = 0.34  # two clones in three calls is a habit, one retry can be jitter


def canonical(call: dict[str, Any]) -> tuple[str, str]:
    name = str(call.get("name") or "")
    args = call.get("args") or {}
    return name, json.dumps(args, sort_keys=True, separators=(",", ":"))


def duplicate_ratio(calls: list[dict[str, Any]]) -> float:
    seen: list[tuple[str, str]] = []
    dups = 0
    for call in calls:
        key = canonical(call)
        if key in seen:
            dups += 1
        seen.append(key)
    return dups / max(len(calls), 1)


def grounded_done(record: dict[str, Any]) -> bool:
    calls = record.get("tool_calls") or []
    claimed = bool(record.get("claimed_done"))
    if claimed and not calls:
        return False
    return True


def score(record: dict[str, Any]) -> dict[str, Any]:
    calls = record.get("tool_calls") or []
    ratio = round(duplicate_ratio(calls), 3)
    fail_dup = len(calls) >= 3 and ratio > THRESHOLD
    fail_ground = not grounded_done(record)
    return {
        "id": record.get("id"),
        "n_calls": len(calls),
        "duplicate_ratio": ratio,
        "fail_duplicates": fail_dup,
        "fail_ungrounded_done": fail_ground,
        "fail": fail_dup or fail_ground,
    }


def load_fixtures(dir_path: str) -> list[dict[str, Any]]:
    rows = []
    for path in sorted(Path(dir_path).glob("*.json")):
        rows.append(json.loads(path.read_text()))
    return rows


if __name__ == "__main__":
    for row in load_fixtures("fixtures"):
        print(json.dumps(score(row), indent=2))

Run that on fixtures you control and you get numbers a skeptic can recompute with python halt_score.py. I do not need a model to tell me that three identical searches are a loop wearing a lab coat.

The fixtures I used look like this. Copy them. Change them. If your scorer cannot fail these, it is a blog post, not a test.

{"id": "clone", "claimed_done": true, "tool_calls": [
  {"name": "search_docs", "args": {"q": "rate limit"}},
  {"name": "search_docs", "args": {"q": "rate limit"}},
  {"name": "search_docs", "args": {"q": "rate limit"}}
]}

{"id": "clean", "claimed_done": true, "tool_calls": [
  {"name": "search_docs", "args": {"q": "webhooks"}},
  {"name": "read_file", "args": {"path": "docs/webhooks.md"}},
  {"name": "apply_patch", "args": {"path": "app.py", "diff": "---"}}
]}

{"id": "hyphen", "claimed_done": true, "tool_calls": [
  {"name": "search_docs", "args": {"q": "rate limit"}},
  {"name": "search_docs", "args": {"q": "rate-limit"}},
  {"name": "search_docs", "args": {"q": "rate limits in API"}}
]}

{"id": "page", "claimed_done": true, "tool_calls": [
  {"name": "search_docs", "args": {"q": "webhooks", "page": 1}},
  {"name": "search_docs", "args": {"q": "webhooks", "page": 2}}
]}

{"id": "essay", "claimed_done": true, "tool_calls": []}

I ran the deterministic scorer against those five files. These numbers are fixture math, not a hosted-model bake-off. clone lands at duplicate_ratio=0.667 and fails. clean lands at 0.0 and passes. hyphen also lands at 0.0, which is the exact-match blind spot: the hallway changed spelling, not rooms. page stays at 0.0 because page actually moved. essay fails fail_ungrounded_done. That last one is not a duplicate problem. It is a done-claim with no fingerprints.

Exact match is honest. It is also a terrible literary critic. Engineers write queries like humans. Hyphens happen. Plurals happen. The same question shows up in a nicer coat and channel A waves it through.

The optional judge, kept on a leash

So I added channel B. Not as truth. As a suspect.

A free model can look at two argument blobs and say whether the second call explores anything new. That helps on the hyphen trick. It also launders a real refinement into a "semantic duplicate" that never happened. Pagination is the obvious landmine. Filters are next. Any field that is small in JSON and large in meaning will make a similarity judge flinch.

I keep that opinion off my laptop when I bother to run it. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option are the optional judge lane in this workflow—the fixtures and the exact-match scorer do not need them, and I will not pretend a one-off completion is a benchmark. If you want that lane off your workstation, their free server is how I isolated it. Steal the scorer either way.

The client is deliberately boring. OpenAI-shaped chat. Model name from the environment. No system prompt poetry.

# judge.py
from __future__ import annotations

import json
import os
import urllib.request

PROMPT = """You compare two tool argument blobs for the same tool name.
Reply with JSON only: {"duplicate": true or false, "reason": "<=20 words"}.
True means the second call does not explore a new search space.
Pagination, new paths, tighter filters, and new ids are NOT duplicates."""


def judge_pair(name: str, a: dict, b: dict) -> dict:
    body = json.dumps({
        "messages": [
            {"role": "system", "content": PROMPT},
            {"role": "user", "content": json.dumps({"tool": name, "first": a, "second": b})},
        ],
        "temperature": 0,
    }).encode()
    req = urllib.request.Request(
        os.environ["CHAT_URL"],
        data=body,
        headers={
            "Content-Type": "application/json",
            "Authorization": f"Bearer {os.environ.get('CHAT_TOKEN', '')}",
        },
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=30) as resp:
        payload = json.loads(resp.read().decode())
    content = payload["choices"][0][message_key(payload)]["content"]
    return json.loads(content)


def message_key(payload: dict) -> str:
    return "message"

I did not print a live-model leaderboard. Those numbers rot by lunch, and they would smuggle a product claim I cannot defend. What I did ship is a stubbed judge so you can rerun the disagreement table tonight, offline. The stub is the experiment control. Swap it for a live endpoint later if you want to measure that endpoint on these fixtures. That is evaluation. A vibe is not.

# test_halt_score.py
import json
from halt_score import score

CLONE = json.loads(open("fixtures/clone.json").read())
CLEAN = json.loads(open("fixtures/clean.json").read())
HYPHEN = json.loads(open("fixtures/hyphen.json").read())
PAGE = json.loads(open("fixtures/page.json").read())
ESSAY = json.loads(open("fixtures/essay.json").read())

STUB = {
    ("search_docs", "rate limit", "rate-limit"): True,
    ("search_docs", "rate limit", "rate limits in API"): True,
    ("search_docs", "webhooks|1", "webhooks|2"): True,  # intentional false stall
}


def stub_duplicate(first, second) -> bool:
    q1 = str(first.get("q"))
    q2 = str(second.get("q"))
    p1 = first.get("page")
    p2 = second.get("page")
    if p1 != p2:
        return STUB[("search_docs", "webhooks|1", "webhooks|2")]
    return STUB.get(("search_docs", q1, q2), False)


def test_clone_fails_exact():
    out = score(CLONE)
    assert out["duplicate_ratio"] == 0.667
    assert out["fail_duplicates"] is True


def test_clean_passes():
    assert score(CLEAN)["fail"] is False


def test_hyphen_is_exact_blind():
    out = score(HYPHEN)
    assert out["duplicate_ratio"] == 0.0
    assert out["fail_duplicates"] is False


def test_page_is_not_an_exact_clone():
    assert score(PAGE)["duplicate_ratio"] == 0.0


def test_essay_fails_grounding():
    assert score(ESSAY)["fail_ungrounded_done"] is True


def test_stub_judge_breaks_on_pagination():
    a, b = PAGE["tool_calls"]
    # This is the known break: a refinement that a similarity judge calls a stall.
    assert stub_duplicate(a["args"], b["args"]) is True

Read the stub like a crime board, not a vendor slide. Exact match nails clone and misses hyphen. The stub catches the hyphen trick and then, on page, calls a real refinement a stall. That is the break I care about. Semantic sameness is allergic to pagination. If you ship the judge as a gate, you will fail honest loops and call it quality.

So the workflow I actually use is meaner than "ask the model." CI fails on exact duplicates above 0.34 once there are at least three tool calls. One retry can be jitter. Two clones in three calls is a habit. The judge, when it exists, runs in warn mode. A human only looks when the two channels disagree. Disagreement is the product. Agreement is boring.

Why 0.34? Because I wanted a threshold I could explain without a spreadsheet religion. Two-in-three is 0.667. One-in-three is 0.333. I fail above a hair over one-in-three so a single retry does not page me and a repeating search does. If your tools are chatty by design, pick a different hair. Write it next to the scorer. Do not hide it in a prompt.

Where this approach lies to you

Canonical JSON does not know that {"path": "./a.py"} and {"path": "a.py"} are the same file. A model might. A model might also invent a third path and grade it with a straight face. I do not normalize paths in v1. I would rather see the stall than sand it off.

The stubbed judge is not evidence about any hosted model's quality. It is a canned false positive so the test stays deterministic. Point channel B at a live free endpoint and you are measuring that endpoint on five fixtures, on one day, with one prompt. That is a lab note. It is not a ranking.

Empty-tool essays fail a different check than duplicates. Do not mash them into one number and call it "agent quality." A loop that never calls tools and a loop that calls the same tool forever are different crimes. They just share a press release.

Who should not use this? If your tools are non-deterministic by design—clocks, random samples, live log tails—exact-match will scream every turn and you will hate the scorer instead of the loop. If your "agent" is a single function with no trace, you do not have a hallway to score. If you need the model judge to be the gate, you are buying flakiness and naming it evaluation. If you came here for a demo that always looks busy, this harness will embarrass you. Good.

I still read the summaries. I just refuse to let them grade the patrol. Next time the model says it investigated, will you score the doorway—or the press release?