A Pairing Hour on JSON That Parses but Lies: The Contract Canary We Kept

The bot returned valid JSON. That was the problem.

A diff had replaced a parameterized query with string concatenation, and the review bot flagged it — in a payload that parsed cleanly, matched the shape we expected, and said "severity": "warning". Our schema allowed low, medium, high. The map lookup returned undefined, the fallback caught it, and the finding landed in the queue as low. It surfaced in a log review later, not in an alert.

That log line is the reason this pairing hour happened at all. The senior who opened it did not start with the parser. She started with the boundary.

The questions that set the agenda

Four things got written on the whiteboard before anyone opened the editor:

• "Which layer is allowed to be wrong — the model, the parser, or the severity map?"
• "If the enum drifts again next Tuesday, what tells you before a human does?"
• "What is the cheapest request you can make that would have caught this specific failure?"
• "Who un-demotes a model, and what do they read before doing it?"

The third question is the one that produced the artifact below. It also killed three plausible fixes.

Three dead ends, and why each one died

1. Retry until the JSON parses. Parsing was never broken. Valid JSON with an out-of-vocabulary enum value passes every retry loop you can write, and each attempt spends a request. The failure is semantic, so a syntactic retry strategy is aimed at the wrong layer.

2. Regex the severity out of the raw text. This worked on the first try and still got deleted. The expression matched "severity": "warning" and could be mapped by hand, until a model wrote the severity here is not low inside a reason field. A regex turns a typed contract into a guess with better formatting.

3. Trust JSON mode as an enum guarantee. JSON mode generally constrains syntax; whether your specific vocabulary is enforced depends on the provider and the mode in use. That is a documentation question, not an assumption to build on. The harness verifies conformance instead of believing a header.

The decision we kept: a contract canary plus a demotion rule

The kept design is small. A fixed set of diff fixtures, each carrying a floor — the minimum severity a competent reviewer must assign. The canary runs every fixture against every candidate model, then grades three things: contract conformance, direction (did anything come back below its floor), and latency.

Two rules survived the hour:

1. Any direction failure demotes immediately. A silent downgrade of a real finding is the exact failure we are guarding, so it gets zero tolerance.
2. Two consecutive contract failures demote; one clean run resets the strike count. Promotion back is manual, always. Nobody wanted a model to be able to re-promote itself by getting lucky once.

The fixtures

Small on purpose. Fifteen fixtures finish in well under a minute even on a free instance.

[
  {
    "id": "sqli-string-concat",
    "path": "src/db/users.js",
    "floor": "high",
    "diff": "- const q = 'SELECT * FROM users WHERE id = $1';\n+ const q = 'SELECT * FROM users WHERE id = ' + req.params.id;"
  },
  {
    "id": "rename-only",
    "path": "src/report/summary.js",
    "floor": "low",
    "diff": "- const d = new Date();\n+ const createdAt = new Date();"
  }
]

The runner

No dependencies, no build step. It uses an OpenAI-compatible chat endpoint, so the same file works against any provider that exposes one.

// canary.mjs — node canary.mjs
import { readFile, writeFile } from "node:fs/promises";

const DIR = new URL("./", import.meta.url);
const fixtures = JSON.parse(await readFile(new URL("fixtures.json", DIR), "utf8"));
const statePath = new URL("router-state.json", DIR);
const state = JSON.parse(await readFile(statePath, "utf8").catch(() => "{}"));
const models = (process.env.CANARY_MODELS ?? "").split(",").filter(Boolean);

const RANK = { low: 0, medium: 1, high: 2 };
const CONTRACT = {
  required: ["severity", "path", "reason"],
  properties: {
    severity: { enum: ["low", "medium", "high"] },
    path: { type: "string", pattern: "^[\\w./-]+$" },
    reason: { type: "string", minLength: 12 }
  },
  additionalProperties: false
};

function validate(obj, schema) {
  if (typeof obj !== "object" || obj === null || Array.isArray(obj)) return ["not-an-object"];
  const errors = [];
  for (const key of schema.required) if (!(key in obj)) errors.push(`missing:${key}`);
  for (const [key, rule] of Object.entries(schema.properties)) {
    const v = obj[key];
    if (v === undefined) continue;
    if (rule.enum && !rule.enum.includes(v)) errors.push(`enum:${key}=${JSON.stringify(v)}`);
    if (rule.type === "string" && typeof v !== "string") errors.push(`type:${key}`);
    if (rule.pattern && typeof v === "string" && !new RegExp(rule.pattern).test(v)) errors.push(`pattern:${key}`);
    if (rule.minLength && typeof v === "string" && v.length < rule.minLength) errors.push(`minLength:${key}`);
  }
  if (schema.additionalProperties === false) {
    for (const key of Object.keys(obj)) if (!(key in schema.properties)) errors.push(`extra:${key}`);
  }
  return errors;
}

async function ask(model, fixture) {
  const res = await fetch(`${process.env.CANARY_BASE_URL}/chat/completions`, {
    method: "POST",
    headers: {
      "content-type": "application/json",
      authorization: `Bearer ${process.env.CANARY_API_KEY}`
    },
    body: JSON.stringify({
      model,
      temperature: 0,
      messages: [
        { role: "system", content: 'Review one diff. Reply with JSON only: {"severity":"low|medium|high","path":"<file>","reason":"<one sentence>"}' },
        { role: "user", content: `file: ${fixture.path}\n\n${fixture.diff}` }
      ]
    })
  });
  if (!res.ok) return { error: `http:${res.status}` };
  const body = await res.json();
  return { text: body.choices?.[0]?.message?.content ?? "" };
}

function gradeFixture(fixture, text) {
  const cleaned = text.replace(/^```
{% endraw %}
(?:json)?\s*|\s*
{% raw %}
```$/g, "").trim();
  let obj;
  try { obj = JSON.parse(cleaned); } catch { return { contract: false, direction: null, errors: ["unparseable"] }; }
  const errors = validate(obj, CONTRACT);
  if (errors.length) return { contract: false, direction: null, errors };
  const direction = RANK[obj.severity] >= RANK[fixture.floor];
  return { contract: true, direction, errors: [], severity: obj.severity };
}

function updateState(state, model, result) {
  const m = (state[model] ??= { strikes: 0, weight: 1, runs: 0, halted: false });
  m.runs += 1;
  m.lastRun = new Date().toISOString();
  m.lastResult = result;
  if (result.directionFailures > 0) { m.halted = true; m.weight = 0; return m; }
  if (result.contractFailures > 0) {
    m.strikes += 1;
    if (m.strikes >= 2) { m.halted = true; m.weight = 0; } else { m.weight = Math.max(1, m.weight - 1); }
    return m;
  }
  m.strikes = 0;
  m.weight = Math.min(3, m.weight + 1);
  return m;
}

let anyHalted = false;
for (const model of models) {
  let contractFailures = 0, directionFailures = 0, errors = 0;
  for (const fixture of fixtures) {
    const { text, error } = await ask(model, fixture);
    if (error) { errors += 1; continue; }
    const g = gradeFixture(fixture, text);
    if (!g.contract) { contractFailures += 1; continue; }
    if (g.direction === false) directionFailures += 1;
  }
  const result = { contractFailures, directionFailures, errors };
  const m = updateState(state, model, result);
  anyHalted ||= m.halted;
  console.log(JSON.stringify({ model, ...result, weight: m.weight, halted: m.halted }));
}

await writeFile(statePath, JSON.stringify(state, null, 2));
process.exitCode = anyHalted ? 1 : 0;

The decision table

Observation	Threshold (example)	Action
Severity below the fixture floor	1 fixture	Halt routing now, page the on-call
Schema/enum violation	1 fixture	One strike, weight -1
Schema/enum violation, second run in a row	2 runs	Halt routing, request manual review
Clean run	all fixtures	Strikes reset, weight +1 (cap 3)
HTTP or timeout errors	any	No strike — infrastructure, not contract

The last row matters more than it looks. A sleeping free instance produces connection errors that are indistinguishable from a broken model unless the runner separates them. It does.

Standing it up cheaply

The canary is a cron job with no database, no queue, and no dashboard. That profile fits a small always-on box, including the free server option the operator describes.

CANARY_MODELS=model-a,model-b \
CANARY_BASE_URL="$BASE_URL" \
CANARY_API_KEY="$API_KEY" \
node canary.mjs

17 6 * * * cd /srv/canary && /usr/bin/node canary.mjs >> canary.log 2>&1

A free instance may sleep between runs, so the first request of the morning can fail on connect before the process is warm. The runner treats that as an error rather than a strike, and a single retry after a short wait is enough in practice. The cost profile is the reason this design exists at all: comparing three or four candidate endpoints daily would otherwise need a budget line. Free model access on MonkeyCode is what this account uses to sweep several endpoints per run, and the free server hosts the cron.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Allowances and instance specs change; read the current offer page before planning around them rather than relying on a number quoted in any article, this one included.

What this does not measure

The canary catches contract breakage and downgrade drift. It does not measure review quality, and the two are easy to confuse. A model can pass all fifteen fixtures while giving shallow reasoning on everything else.

• Fifteen fixtures are a smoke test, not a benchmark. Missing classes stay missing until someone adds a fixture for them.
• History dies on prompt change. A system-prompt edit invalidates prior runs; reset the state file when the contract text changes.
• One run a day leaves a blind window. The runtime validator still guards every live response — it fails closed to the manual queue — but the canary only reports what happened at 06:17.
• Latency is recorded, price is not. The harness has no opinion on what a request costs.

Who should skip this

Teams that already route every diff through a human reviewer get little from a daily canary; the human is the guard. Teams with a hard latency SLA should not halt a model mid-shift on a single fixture failure without a fallback pool that can absorb the traffic. And anyone running a single model from a single vendor can replace the whole harness with a runtime schema check and an alert, which is cheaper to maintain.

The canary earns its keep at the awkward middle: several interchangeable endpoints, no budget for a monitoring tool, and a review bot whose output is trusted enough to skip reading.

If the next pairing hour is about picking which endpoints to sweep first, start with the ones you can call for free — then let the fixtures, not the marketing page, decide who keeps the traffic.