Stop Scoring Prose. Start Diffing Tool Shapes.

Fluent agent answers are a weak merge signal today.
You should gate on tool-call shapes, not prose.
A schema-valid trace beats any polished summary.

Take a side

You are not testing an agent with chat transcripts.
You are testing a narrator that can call tools.
Engineering starts when those calls become contracts.

Vibe coding produces text that looks finished fast.
Calling that text a verified change remains the failure.
The model can lie in English without touching state.

Hot threads still argue about models beating developers.
That debate skips the only artifact you can freeze.
You cannot freeze charm, but you can freeze JSON shapes.

What a shape actually is

A shape is the JSON skeleton of each tool call.
It includes name, argument keys, types, and order.
It ignores free-text values that change every run.

You keep the skeleton and discard the essay around it.
You compare skeletons across runs, models, and branches.
A mismatch means the harness drifted, not the copy.

Store shapes as receipts beside the pull request.
Treat a missing receipt as a build failure immediately.
Do not accept screenshots of the chat as substitutes.

Why answer scores keep lying

Answer scores reward fluency and a confident tone.
Agents game both without doing the requested work.
A skipped tool still yields a soothing paragraph.

Unit tests written after the fact hide that skip.
Reviewers read the summary and then approve the skip.
The production path never sees the missing tool call.

Loop-engineering posts still warn about unbounded retries today.
Retries also mutate shapes while scores stay green.
Shape-diff catches that mutation while score dashboards miss it.

The merge rule you should adopt

This section is a proposal, not a measured case study.

1. Every agent run must emit a receipt file on disk.
2. Every receipt must validate against a frozen JSON schema.
3. CI must shape-diff that receipt against a golden file.
4. Prose in the PR body stays commentary, never evidence.

If step three fails, you do not merge the change.
You fix the harness or you update the frozen schema.
You never accept nicer wording as a substitute patch.

Pin the schema next to the tool definitions in git.
Require a human review when the schema itself changes.
That review is the real design discussion you wanted.

Freeze a receipt schema

The example below is illustrative and unexecuted here.
Copy it, then tighten types for your real tools.
Do not ship the sample tools into production paths.

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "title": "AgentReceipt",
  "type": "object",
  "required": ["schema_version", "run_id", "calls"],
  "properties": {
    "schema_version": { "const": "1" },
    "run_id": { "type": "string", "minLength": 8 },
    "calls": {
      "type": "array",
      "minItems": 1,
      "items": {
        "type": "object",
        "required": ["seq", "tool", "arg_keys", "arg_types"],
        "properties": {
          "seq": { "type": "integer", "minimum": 0 },
          "tool": { "type": "string", "pattern": "^[a-z][a-z0-9_]*$" },
          "arg_keys": {
            "type": "array",
            "items": { "type": "string" }
          },
          "arg_types": {
            "type": "array",
            "items": {
              "enum": ["string", "number", "boolean", "object", "array", "null"]
            }
          }
        },
        "additionalProperties": false
      }
    }
  },
  "additionalProperties": false
}

Notice the schema refuses extra keys on each call.
That refusal is the entire product of this gate.
Extra keys mean the harness drifted under your feet.
Pretty summaries cannot sneak a new argument through.

Strip values before you diff

Raw argument values poison the golden file too fast.
Timestamps, ids, and prose will never stay stable.
You strip values and keep names plus types only.

# Example only. Unexecuted in this article.
from __future__ import annotations

import json
from pathlib import Path
from typing import Any

TYPE_MAP = {
    str: "string",
    int: "number",
    float: "number",
    bool: "boolean",
    type(None): "null",
}


def json_type(value: Any) -> str:
    if isinstance(value, list):
        return "array"
    if isinstance(value, dict):
        return "object"
    return TYPE_MAP[type(value)]


def shape_of(call: dict[str, Any]) -> dict[str, Any]:
    args = call.get("arguments") or {}
    keys = sorted(args.keys())
    return {
        "seq": call["seq"],
        "tool": call["tool"],
        "arg_keys": keys,
        "arg_types": [json_type(args[k]) for k in keys],
    }


def receipt_shape(raw: dict[str, Any]) -> dict[str, Any]:
    return {
        "schema_version": "1",
        "run_id": raw["run_id"],
        "calls": [shape_of(c) for c in raw["calls"]],
    }


def dump_shape(src: Path, dest: Path) -> None:
    raw = json.loads(src.read_text())
    dest.write_text(json.dumps(receipt_shape(raw), indent=2) + "\n")

Keep run_id for humans and drop it from equality.
Compare the calls array only as the contract surface.
Sequence plus tool plus typed keys must stay identical.

Fail the build on a shape mismatch

Golden files belong in the same pull request always.
Update them in a dedicated schema-change commit only.
Never bury a shape change inside a feature commit.

# Example only. Unexecuted in this article.
import json
import sys
from pathlib import Path

from jsonschema import Draft202012Validator


def load(path: Path) -> dict:
    return json.loads(path.read_text())


def validate(schema_path: Path, receipt_path: Path) -> None:
    schema = load(schema_path)
    data = load(receipt_path)
    Draft202012Validator(schema).validate(data)
    print("SCHEMA_OK")


def diff_calls(golden_path: Path, actual_path: Path) -> int:
    left = load(golden_path)["calls"]
    right = load(actual_path)["calls"]
    if left != right:
        print("SHAPE_DIFF")
        print("golden:", json.dumps(left, indent=2))
        print("actual:", json.dumps(right, indent=2))
        return 1
    print("SHAPE_OK")
    return 0


if __name__ == "__main__":
    validate(Path("tools/receipt.schema.json"), Path("receipts/actual.shape.json"))
    raise SystemExit(
        diff_calls(Path("receipts/golden.shape.json"), Path("receipts/actual.shape.json"))
    )

Wire it as a boring command, not a dashboard widget.

python3 tools/shape_of.py receipts/actual.json receipts/actual.shape.json
python3 tools/shape_diff.py
test $? -eq 0

A non-zero exit code is the entire merge policy.
Do not parse the model essay after this step.
Do not negotiate with the log output either.

Fixture the tools, not the novel

The fixture below is a stub, not a live agent.
Replace emit_receipt() with your harness hook later.
Keep the tool names identical to production definitions.

# Example only. Unexecuted in this article.
import json
import uuid


def emit_receipt() -> dict:
    return {
        "schema_version": "1",
        "run_id": uuid.uuid4().hex,
        "calls": [
            {
                "seq": 0,
                "tool": "fetch_issue",
                "arguments": {"id": "ISSUE-1"},
            },
            {
                "seq": 1,
                "tool": "post_comment",
                "arguments": {"issue_id": "ISSUE-1", "body": "done"},
            },
        ],
    }


if __name__ == "__main__":
    print(json.dumps(emit_receipt(), indent=2))

Run that fixture until the golden file stops moving.
Then point the same hook at a hosted model runner.
The shape must not change when the host changes.

Run the worker off your laptop

Your laptop shares cache, secrets, and lucky timing.
A dedicated job host removes that shared luck.
Free model access belongs in that job, not demos.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode provides free model access and a free server option.
Point the shape-diff worker at that server when hosts are scarce.
Keep product features on your normal models and paid paths.

The free model only has to emit schema-valid receipts.
It does not have to win a style contest against anyone.
If the shape breaks, the model is the wrong job runner.

A minimal job sketch

This YAML is a template and not a vendor file.

# Example only. Unexecuted in this article.
name: shape-diff
on:
  pull_request:
jobs:
  receipt:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: emit receipt
        run: python3 tools/run_agent_fixture.py > receipts/actual.json
      - name: strip values
        run: python3 tools/shape_of.py receipts/actual.json receipts/actual.shape.json
      - name: validate and diff
        run: python3 tools/shape_diff.py

The fixture script should use a frozen prompt file.
It should use a frozen tool list and frozen fixtures.
It should not browse the public internet for truth.

Decision table

Use this table before you add another numeric score.

Situation	Trust the prose?	Trust the shape?	Action
You changed copy only	No	Yes, if calls match	Merge the copy
You added a tool argument	No	No, until schema updates	Review the schema
Model skipped a tool	Yes, it will sound fine	No	Block the merge
Model reordered two calls	Maybe	No	Block, then inspect
Values changed, keys did not	No	Yes	Merge after value tests
Schema file changed with no review	No	No	Reject the pull request

Print the table in the team handbook this week.
Do not keep it in a slide deck nobody opens.
The table is the policy and the script is the lock.

What this will not catch

Shape-diff ignores wrong values with the right types.
A string can still be a malicious URL or empty path.
You still need value tests for auth, money, and deletion.

It also ignores missing side effects outside listed tools.
A tool can return 200 and still write the wrong row.
Pair shape-diff with ordinary integration tests after it.

Free model access will not prove production quality here.
It only proves the harness still demands the same skeleton.
Do not publish those runs as public model leaderboards.

Who should not use this

Skip this if your agent has no structured tools.
A single-shot editor chat has no receipt to freeze.
Do not invent fake tools just to satisfy the linter.

Skip this if you cannot pin prompts and tool lists.
A daily prompt tweak without review breaks golden files.
You will spend the week blessing noisy receipt diffs.

Skip this during short exploratory research spikes too.
Exploration needs cheap failure, not a frozen skeleton.
Promote the spike into this gate only after tools stabilize.

Put the schema in CODEOWNERS

Prompt files and tool schemas are production code now.
They deserve owners, review, and a boring textual diff.
Leave the essay to the author and gate the skeleton.

# Example CODEOWNERS fragment
/prompts/ @harness-owners
/tools/*.schema.json @harness-owners
/receipts/golden.shape.json @harness-owners

When the golden file changes, named owners must speak.
Silence is not consent because the shape is the API.
You would not ship an unsigned OpenAPI change today.
Stop doing the same silent drift with agent tools.

Closing position

You do not lack a smarter model for this gate.
You lack a frozen skeleton for every tool call.
Until that skeleton exists, you are still vibing.

Keep the worker dull, local to git, and strict.
If you already emit receipts, run the shape-diff worker on a free server.
Keep the frozen schema in version control beside the tools.