Kill the Spike If the Schema Drifts

On a Tuesday afternoon, a backend pair watched an assistant invent a third argument for a lookup tool. The unit tests stayed green because they never pinned the JSON schema that the client would send. They needed a ship-or-kill answer inside a ninety-minute spike, not another hopeful chat transcript. The spike would freeze one tool contract in git and treat later field drift as an automatic kill.

Tool calling looks like ordinary function invocation until a generated handler silently reshapes the wire payload. Clients then send yesterday's fields while the server expects a renamed key that never existed in review. The analogy is a locksmith who keeps cutting new keys while the door hardware stays unchanged overnight. A time-boxed spike is useful only when it scores that lock, not the polish of the assistant's narration.

Public tutorials keep showing tool calling as a clean JSON envelope around an ordinary function in application code. The envelope is not the hard part; the hard part is stopping generated code from treating the envelope as clay. A spike that only demos a successful call will miss the renamed field that breaks the mobile client next week. Scoring the frozen schema is slower to clap for, and that slowness is the point of the ninety-minute wall.

The protocol below treats ninety minutes as a hard wall and one hypothesis as the only scored claim. Engineers write the hypothesis, freeze a JSON Schema, and refuse to expand scope after the clock starts. Evidence is a schema hash, a golden request, and an extra-properties test that must stay honest from red to green. Anything else, including refactors and comments, is noise until that contract either ships or dies.

Teams that isolate evaluation can park this spike on MonkeyCode's free server with free model access, keeping billing out of the clock.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The schema tests remain the judge if another host later runs the identical script and fixtures. Product convenience does not replace the kill rule when the frozen contract moves during the window. The spike still fails closed if the assistant edits the schema to make a green bar appear.

Before minute zero, the team records a single hypothesis in the repository rather than in a chat pane. The text names the tool, the schema file, the forbidden mutations, and the exact clock that will stop work. After that commit, new endpoints and extra stories are treated as contamination of the spike, not as progress. The wall clock starts only when the failing contract test is already present and executable on the server.

# Proposed file: HYPOTHESIS.md (commit this before starting the clock)
CLAIM: The generated lookup_widget handler validates request and response
against schema/v1.json. No property may be added, renamed, or dropped.
CLOCK: 90 minutes from first red pytest on tests/test_tool_contract.py.
SHIP: fingerprint match + golden fixture + additionalProperties stays false.
KILL: any schema edit, any extra field, or timeout without the triple.

The frozen schema is deliberately small so the assistant cannot hide drift inside optional bags of properties. additionalProperties stays false, required lists every field, and types stay narrow enough to fail loudly. A hash of that file becomes the fingerprint the spike must still match when the ninety minutes expire. If the assistant rewrites the schema to make tests pass, the fingerprint check kills the spike on purpose.

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "title": "LookupToolV1",
  "type": "object",
  "additionalProperties": false,
  "required": ["tool", "request", "response"],
  "properties": {
    "tool": { "const": "lookup_widget" },
    "request": {
      "type": "object",
      "additionalProperties": false,
      "required": ["widget_id", "locale"],
      "properties": {
        "widget_id": { "type": "string", "minLength": 1 },
        "locale": { "type": "string", "pattern": "^[a-z]{2}-[A-Z]{2}$" }
      }
    },
    "response": {
      "type": "object",
      "additionalProperties": false,
      "required": ["widget_id", "title", "in_stock"],
      "properties": {
        "widget_id": { "type": "string" },
        "title": { "type": "string" },
        "in_stock": { "type": "boolean" }
      }
    }
  }
}

The contract test should fail for extra keys, missing keys, and type changes before any handler exists. Golden fixtures live beside the test so replay does not depend on a model being available at assert time. Engineers run the file once to confirm red, then start the clock, then allow the assistant into the worktree. Green without a fingerprint match is still a kill, because the door hardware was swapped during the night.

# Proposed spike kit (unexecuted here). Copy into a disposable clone.
# tests/test_tool_contract.py
from pathlib import Path
import hashlib
import json
import subprocess
import sys

import pytest
from jsonschema import Draft202012Validator

ROOT = Path(__file__).resolve().parents[1]
SCHEMA_PATH = ROOT / "schema" / "v1.json"
FINGERPRINT_PATH = ROOT / "schema" / "v1.sha256"
GOLDEN_PATH = ROOT / "fixtures" / "lookup_widget.golden.json"
LOCKED = ("schema/v1.json", "schema/v1.sha256", "fixtures/lookup_widget.golden.json",
          "tests/test_tool_contract.py", "HYPOTHESIS.md")


def sha256(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()


def test_fingerprint_still_matches_frozen_schema():
    expected = FINGERPRINT_PATH.read_text().strip()
    actual = sha256(SCHEMA_PATH)
    assert actual == expected, f"schema drifted: {actual} != {expected}"


def test_locked_paths_were_not_edited_during_the_spike():
    diff = subprocess.check_output(
        ["git", "diff", "--name-only", "HEAD", "--", *LOCKED],
        cwd=ROOT,
        text=True,
    ).strip()
    assert diff == "", f"locked files moved:\n{diff}"


def test_golden_envelope_matches_schema_and_rejects_extra_keys():
    schema = json.loads(SCHEMA_PATH.read_text())
    validator = Draft202012Validator(schema)
    golden = json.loads(GOLDEN_PATH.read_text())
    errors = sorted(validator.iter_errors(golden), key=lambda e: e.json_path)
    assert errors == [], [e.message for e in errors]

    drifted = json.loads(json.dumps(golden))
    drifted["request"]["limit"] = 50  # the invented third argument
    extra = list(Draft202012Validator(schema).iter_errors(drifted))
    assert extra, "schema must reject extra properties; spike is otherwise lying"


def test_handler_round_trip_uses_the_frozen_envelope():
    # Handler is allowed to change; schema and fixtures are not.
    sys.path.insert(0, str(ROOT / "src"))
    from handler import handle_lookup_widget  # proposed module under test

    golden = json.loads(GOLDEN_PATH.read_text())
    body = handle_lookup_widget(golden["request"])
    envelope = {"tool": "lookup_widget", "request": golden["request"], "response": body}
    Draft202012Validator(json.loads(SCHEMA_PATH.read_text())).validate(envelope)
    assert body["widget_id"] == golden["request"]["widget_id"]

A tiny clock script keeps humans honest when the session feels productive and therefore worth extending. It prints remaining minutes, refuses to start without the red test, and writes a verdict file at the wall. The verdict is ship only when fingerprint, golden request, and extra-properties checks all pass together. Timeout without that triple is kill, even if the generated comments look careful and the demo path works.

# Proposed file: scripts/spike_clock.sh
set -euo pipefail
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
cd "$ROOT"
export PYTHONPATH="$ROOT"

if [[ ! -f schema/v1.sha256 ]]; then
  python3 - <<'PY'
from pathlib import Path
import hashlib
p = Path("schema/v1.json")
Path("schema/v1.sha256").write_text(hashlib.sha256(p.read_bytes()).hexdigest() + "\n")
PY
fi

echo "Confirming the contract test is red before minute zero..."
if pytest -q tests/test_tool_contract.py; then
  echo "KILL: tests were already green; the spike has no failing proof." >&2
  exit 2
fi

START="$(date +%s)"
DEADLINE="$((START + 90 * 60))"
echo "clock started at ${START}; kill at ${DEADLINE}"

while true; do
  NOW="$(date +%s)"
  LEFT="$((DEADLINE - NOW))"
  if [[ "${LEFT}" -le 0 ]]; then
    echo "KILL" > spike_verdict.txt
    echo "KILL: ninety minutes elapsed without a clean triple."
    exit 1
  fi
  if pytest -q tests/test_tool_contract.py; then
    echo "SHIP" > spike_verdict.txt
    echo "SHIP: fingerprint, golden envelope, and extra-key reject all passed."
    exit 0
  fi
  echo "still red; $((LEFT / 60)) minutes remaining"
  sleep 30
done

The commands below are labeled as a proposed local sequence and should be executed in a disposable clone. They assume pytest, jsonschema, and a python3 interpreter, and they claim no particular host size. Operators should replace the host with their own isolated machine whenever the evaluation lane is different. Copy the files, confirm the test is red, start the clock, then allow edits only under src/handler.py.

git clone . /tmp/lookup-spike && cd /tmp/lookup-spike
python3 -m pip install pytest jsonschema
printf '%s\n' '{"widget_id":"w-1","locale":"en-US"}' > /tmp/req.json
mkdir -p fixtures src schema tests scripts
# after copying schema/v1.json and the test file:
python3 - <<'PY'
from pathlib import Path
import hashlib, json
Path("fixtures").mkdir(exist_ok=True)
golden = {
  "tool": "lookup_widget",
  "request": {"widget_id": "w-1", "locale": "en-US"},
  "response": {"widget_id": "w-1", "title": "unused until handler exists", "in_stock": True},
}
Path("fixtures/lookup_widget.golden.json").write_text(json.dumps(golden, indent=2))
raw = Path("schema/v1.json").read_bytes()
Path("schema/v1.sha256").write_text(hashlib.sha256(raw).hexdigest() + "\n")
PY
printf '%s\n' 'def handle_lookup_widget(request): raise NotImplementedError' > src/handler.py
chmod +x scripts/spike_clock.sh
./scripts/spike_clock.sh

After the clock starts, the assistant may edit the handler, but it may not edit schema, fixtures, or tests. That boundary is the same idea as keeping the exam paper out of the student's backpack during grading. A simple git path check in the test file fails the spike if those locked files change their hashes. Humans still review the diff, because a passing contract can hide a hardcoded golden response with no lookup.

Ship means the frozen envelope still validates, extra keys still fail, and the handler returns the required three response fields. Kill means the assistant added limit, renamed locale, widened types, or ran out of clock while the NotImplementedError remained. Partial greens do not accumulate into a maybe; the spike records one word in spike_verdict.txt and then stops talking. That severity is what keeps a Tuesday demo from leaking a mutated client contract into Friday's mobile release.

This workflow does not measure latency, token cost, or literary quality of the generated implementation. It also does not prove authorization, rate limits, or what happens when upstream tools return partial errors. Teams still designing the product contract should not freeze a schema they expect to change before lunch. Incident response is the wrong place for a ninety-minute research clock and a single academic hypothesis.

Leaderboard hunters should skip this method because it refuses to rank models or publish comparative scores. Multi-service platforms cannot collapse their surface into one lookup tool without lying about coverage. Engineers without permission to isolate a server should not mix the spike with production credentials or data. If the hypothesis cannot be written in four lines, the spike is already too large to score cleanly.

The Tuesday pair shipped nothing that afternoon, and that kill was the useful artifact of the clock. They kept the frozen schema, the red-to-green test, and a short note explaining which field the assistant invented. The next spike can start from that note instead of from a regenerated sense of optimism about tool calling. Readers who want an isolated lane for the same clock can try MonkeyCode's free model access and free server.