Hash the Tool Catalog Before You Trust the Trace

The agent printed done after four tool calls. The JSONL file looked complete at a glance. One name in the log was run_tests.

The local harness never registered that tool name. It only exposed pytest, read_file, and grep. The runtime still accepted the invented call.

The runtime returned a one-line error string. The model apologized and called pytest next. The final assistant message still said green.

The session then burned forty minutes on prompt edits. The catalog itself never appeared in any span. That missing catalog set was the actual defect.

Why traces lie without a catalog

Most agent logs store name, arguments, and output. They skip the tool list the model received. Replay then compares calls against the wrong contract.

Three events get labeled as model failure. They are catalog events instead.

• Invented names that were never in the catalog
• Stale names left after a schema edit
• Result rows whose tool_call_id values do not match

Happy-path tool-calling writeups skip this gap. A complete trace needs the negative path. Unknown names should be first-class events.

Record a frozen catalog at run start

Freeze the catalog before the first completion request. Sort the names for a stable order. Hash the canonical blob once.

Store these fields on a single catalog JSONL event:

• run_id as a UUID string
• catalog_names as a sorted list
• catalog_sha256 over that name list
• schema_sha256 per tool, optional but useful
• started_at as a UTC timestamp

Every later span can join on catalog_sha256. Two runs with different hashes are not comparable. Diff those catalogs before you diff prompts.

Name hash versus schema hash

A name hash catches invented tools quickly. It misses field changes on the same name. Hash the JSON schema as well.

Use stable serialization for that second hash. Sort keys. Drop extra whitespace. Then SHA-256 the UTF-8 bytes.

If read_file gains an offset field, schema hash moves. The name hash stays put in that case. The debug loop can tell those two drifts apart.

Artifact: a JSONL catalog tracer

The script below is a labeled local example. It does not report any production metrics. It writes JSONL lines and a small summary table.

# catalog_trace.py
# Labeled local example. Not a measured production run.
from __future__ import annotations

import hashlib
import json
import uuid
from datetime import datetime, timezone
from pathlib import Path
from typing import Any

CATALOG = {
    "pytest": {
        "type": "object",
        "properties": {"args": {"type": "array", "items": {"type": "string"}}},
        "required": ["args"],
    },
    "read_file": {
        "type": "object",
        "properties": {"path": {"type": "string"}},
        "required": ["path"],
    },
    "grep": {
        "type": "object",
        "properties": {
            "pattern": {"type": "string"},
            "path": {"type": "string"},
        },
        "required": ["pattern", "path"],
    },
}


def utc_now() -> str:
    return datetime.now(timezone.utc).isoformat()


def sha256_bytes(data: bytes) -> str:
    return hashlib.sha256(data).hexdigest()


def canonical_json(obj: Any) -> bytes:
    return json.dumps(obj, sort_keys=True, separators=(",", ":")).encode("utf-8")


def catalog_event(run_id: str) -> dict[str, Any]:
    names = sorted(CATALOG)
    schemas = {
        name: sha256_bytes(canonical_json(schema))
        for name, schema in CATALOG.items()
    }
    return {
        "type": "catalog",
        "run_id": run_id,
        "started_at": utc_now(),
        "catalog_names": names,
        "catalog_sha256": sha256_bytes(canonical_json(names)),
        "schema_sha256": schemas,
    }


def call_event(
    run_id: str,
    catalog_sha: str,
    tool_call_id: str,
    name: str,
    arguments: dict[str, Any],
) -> dict[str, Any]:
    known = name in CATALOG
    flags = [] if known else ["UNKNOWN_TOOL"]
    schema_sha = None
    if known:
        schema_sha = sha256_bytes(canonical_json(CATALOG[name]))
    return {
        "type": "tool_call",
        "run_id": run_id,
        "ts": utc_now(),
        "catalog_sha256": catalog_sha,
        "tool_call_id": tool_call_id,
        "name": name,
        "known": known,
        "schema_sha256": schema_sha,
        "args_sha256": sha256_bytes(canonical_json(arguments)),
        "arguments": arguments,
        "flags": flags,
    }


def result_event(
    run_id: str,
    call_ids: set[str],
    tool_call_id: str,
    name: str,
    exit_code: int,
    output: str,
) -> dict[str, Any]:
    flags = []
    if tool_call_id not in call_ids:
        flags.append("ID_MISMATCH")
    return {
        "type": "tool_result",
        "run_id": run_id,
        "ts": utc_now(),
        "tool_call_id": tool_call_id,
        "name": name,
        "exit_code": exit_code,
        "output_bytes": len(output.encode("utf-8")),
        "output_head": output[:240],
        "flags": flags,
    }


def unused_event(run_id: str, catalog_names: list[str], used: set[str]) -> dict[str, Any]:
    unused = [name for name in catalog_names if name not in used]
    return {
        "type": "catalog_unused",
        "run_id": run_id,
        "unused_names": unused,
        "flags": ["UNUSED_TOOL"] if unused else [],
    }


def main() -> None:
    run_id = str(uuid.uuid4())
    out = Path("catalog_trace.jsonl")
    catalog = catalog_event(run_id)
    catalog_sha = catalog["catalog_sha256"]
    call_ids: set[str] = set()
    used: set[str] = set()
    rows = [catalog]

    # Invented name: not in CATALOG.
    unknown_id = "call_run_tests_1"
    rows.append(
        call_event(
            run_id,
            catalog_sha,
            unknown_id,
            "run_tests",
            {"path": "tests/"},
        )
    )
    call_ids.add(unknown_id)
    used.add("run_tests")
    rows.append(
        result_event(
            run_id,
            call_ids,
            unknown_id,
            "run_tests",
            2,
            "unknown tool: run_tests",
        )
    )

    # Registered name with a matching id.
    pytest_id = "call_pytest_1"
    rows.append(
        call_event(
            run_id,
            catalog_sha,
            pytest_id,
            "pytest",
            {"args": ["-q"]},
        )
    )
    call_ids.add(pytest_id)
    used.add("pytest")
    rows.append(
        result_event(
            run_id, call_ids, pytest_id, "pytest", 0, "2 passed in 0.40s"
        )
    )

    # Planted pairing bug: result id was never issued.
    rows.append(
        result_event(
            run_id,
            call_ids,
            "call_missing_9",
            "read_file",
            0,
            "# fake",
        )
    )

    rows.append(unused_event(run_id, catalog["catalog_names"], used))

    with out.open("w", encoding="utf-8") as handle:
        for row in rows:
            handle.write(json.dumps(row, sort_keys=True) + "\n")

    print(f"run_id={run_id}")
    print(f"catalog_sha256={catalog_sha}")
    print(f"wrote {out}")
    for row in rows:
        flags = row.get("flags") or []
        if flags:
            print(f"{row['type']:16} {row.get('name', '-'):12} {flags}")


if __name__ == "__main__":
    main()

How to run the example

python3 catalog_trace.py
wc -l catalog_trace.jsonl

The demo should print three flag groups. Treat them as fixtures, not live traffic.

1. UNKNOWN_TOOL on the run_tests call
2. ID_MISMATCH on the planted result row
3. UNUSED_TOOL for grep and read_file

Query the JSONL without a UI

python3 - <<'PY'
import json
from pathlib import Path

rows = [json.loads(line) for line in Path("catalog_trace.jsonl").read_text().splitlines()]
catalog = next(r for r in rows if r["type"] == "catalog")
print("catalog", catalog["catalog_sha256"], catalog["catalog_names"])

for row in rows:
    if row.get("flags"):
        print(row["type"], row.get("name"), row["flags"])

calls = {r["tool_call_id"]: r for r in rows if r["type"] == "tool_call"}
results = [r for r in rows if r["type"] == "tool_result"]
for result in results:
    matched = result["tool_call_id"] in calls
    print("paired" if matched else "orphan", result["tool_call_id"], result["name"])
PY

Optional jq checks if you already use it:

jq -r 'select(.type=="catalog") | .catalog_sha256' catalog_trace.jsonl
jq -c 'select(.flags != null and (.flags|length)>0)' catalog_trace.jsonl

A reusable debug loop

Do not start with the model card. Start with catalog identity. Then inspect pairing.

1. Write one catalog event before any completion.
2. Flag or reject names outside that frozen set.
3. Copy tool_call_id from each call into its result.
4. Flag ID_MISMATCH when those identifiers diverge.
5. Emit unused catalog names after the last span.
6. Compare two runs only after both hashes match.

Decision table

Symptom	First check	If catalog hash differs	If catalog hash matches
Invented name in the log	Membership in catalog_names	Wrong catalog shipped	Model invented a tool
Same name, rejected arguments	Per-tool schema_sha256	Tool definition drift	Argument validation bug
Result row without a call	tool_call_id index	Mixed JSONL files	Runtime pairing bug
Green summary after a failed tool	Unknown plus unused flags	Recovery hid a catalog miss	Prompt papered over the error

Keep argument hashes, not raw secrets. Redact tokens before you write JSONL. Hash the redacted object, not the live credential.

Parallel tool calls still need unique ids. Two read_file spans can share a name. They cannot share a tool_call_id.

Empty catalogs are events too. A zero-length name list should still hash. A later call against that hash is always UNKNOWN_TOOL.

Keep the tracer on the same host

Traces rot when they leave the machine that ran tools. Hash the catalog on that host. Store JSONL beside the prompt file.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode provides free model access and a free server option. That pairing is enough to keep catalog events, tool spans, and prompts on one box.

If you already schedule runs there, place catalog_trace.py next to the harness. The loop does not need a separate cluster.

What this loop does not catch

A catalog hash does not prove the tool body is correct. A registered pytest call can still run the wrong tests. You still need exit codes and file diffs for that class.

Name membership does not validate argument types at runtime. Add JSON Schema checks if your harness can afford them. Schema hashes only detect definition drift.

This tracer does not reconstruct truncated model streams. It records parsed objects after JSON decode. Mid-stream parse failures need a raw argument buffer.

It also does not fix clock skew across hosts. One host, one JSONL file, one clock. Split files across machines without a run id and pairing collapses.

Do not treat unused tools as proof of a bad prompt. Some catalogs include rare escape hatches. Mark those names as optional if that is the contract.

Who should skip this approach

Skip this loop if the agent has no tools. A chat-only completion has no catalog to hash. Log finish reasons instead.

Skip it if your runtime already emits OpenTelemetry spans. Those spans must already include catalog hash and call ids. Duplicating JSONL then adds noise.

Skip it for high-volume production without sampling. Full argument blobs will dominate disk. Keep hashes and flags, drop payloads.

Skip it when several agents share one file without a role field. Mix two catalogs in one JSONL and membership checks lie. Give each role its own run_id.

Close the loop in this order

Unknown tool names are trace events. They are not mysterious weight drift. Hash the catalog first. Then debug the model.

Match tool_call_id on every result row. List unused names at the end. Only then compare two runs that share a catalog hash.