Fail Closed on Bad JSON: A Contract Gate for Free-Tier Model Endpoints

An endpoint that answers 200 OK with unparseable JSON is more dangerous than one that answers 503. The 503 fails closed and your code takes the error path. The malformed 200 fails open, sails through three layers of your pipeline, and writes garbage into a database that looks fine until a human reads it.

That asymmetry is why latency is the wrong first thing to measure on a free-tier model endpoint. Measure the contract first. In about ten minutes you can produce three numbers that decide whether a workflow is shippable: the clean-conformance rate, the schema-violation rate, and the recovery rate after exactly one repair attempt.

This is the checklist I use as a gate before pointing real users at a prototype. Every step produces evidence you can attach to a merge request, and every step can fail the gate.

What this gate proves, and what it does not

Be precise about the claim, or the checklist becomes theatre.

• It proves that, at the temperature you actually ship, the endpoint currently returns objects that satisfy your schema and your cross-field invariants.
• It does not prove latency, throughput, concurrency behaviour, cost ceilings, or availability next Tuesday.
• It is not a load test. Fifty trials is a smoke test with a documented result, not a statistical guarantee.

If you need an SLO, this gate is a prerequisite, not a substitute.

Step 1 — Write down what "valid" means before you call anything

Most teams skip this and end up debugging vibes. Do it first. You need four layers:

1. Shape — required keys, and their types.
2. Enums — the exact allowed string values, no synonyms tolerated.
3. Invariants — cross-field rules your domain depends on (severity == "high" must imply needs_human == true).
4. Presentation — no prose, no markdown fences. If you strip fences before parsing, you have a silent repair, and you need to know how often it fires.

Write these down in code, not in a wiki. The gate reads them.

Step 2 — The harness

Node 20+, zero dependencies. It calls an OpenAI-compatible /chat/completions endpoint and classifies every trial.

// contract-gate.mjs — Node 20+, no dependencies
import { createHash } from "node:crypto";
import { writeFileSync } from "node:fs";

const BASE   = process.env.BASE_URL;   // e.g. https://host/v1
const KEY    = process.env.API_KEY;
const MODEL  = process.env.MODEL;
const TRIALS = Number(process.env.TRIALS ?? 50);
const THRESH = Number(process.env.MIN_CONFORMANCE ?? 0.98);

const SYSTEM = [
  "Return exactly one JSON object. No prose. No markdown fences.",
  'Shape: {"ticket_id": string, "severity": "low"|"high", "needs_human": boolean}',
].join("\n");
const USER = 'Triage: "checkout button does nothing on Safari 17".';

const promptHash = createHash("sha256")
  .update(SYSTEM + "\n" + USER).digest("hex").slice(0, 12);

const violation = (o) => {
  if (o === null || typeof o !== "object" || Array.isArray(o)) return "not-an-object";
  if (typeof o.ticket_id !== "string" || o.ticket_id.trim() === "") return "ticket_id";
  if (o.severity !== "low" && o.severity !== "high") return "severity";
  if (typeof o.needs_human !== "boolean") return "needs_human";
  if (o.severity === "high" && o.needs_human !== true) return "invariant:high";
  return null;
};

async function callOnce(extra = []) {
  const res = await fetch(`${BASE}/chat/completions`, {
    method: "POST",
    headers: { "content-type": "application/json", authorization: `Bearer ${KEY}` },
    body: JSON.stringify({
      model: MODEL,
      temperature: 0,
      max_tokens: 200,
      messages: [
        { role: "system", content: SYSTEM },
        { role: "user", content: USER },
        ...extra,
      ],
    }),
    signal: AbortSignal.timeout(30_000),
  });
  return { status: res.status, body: await res.text() };
}

The single most important line in that file is temperature: 0. If you test at 0.7 and ship at 0, the evidence does not describe the system you run.

Step 3 — Classify failures, don't just count them

A pass rate with no failure taxonomy tells you nothing actionable. Tag each trial with exactly one primary stage, cheapest failure first:

function classify({ status, body }) {
  if (status !== 200) return { stage: "http", raw: body.slice(0, 160) };
  let content;
  try {
    content = JSON.parse(body).choices[0].message.content;
  } catch {
    return { stage: "envelope", raw: body.slice(0, 160) };
  }
  const trimmed = content.trim();
  const stripped = trimmed.replace(/^```
{% endraw %}
(?:json)?/i, "").replace(/
{% raw %}
```$/, "").trim();
  const fenced = stripped !== trimmed;
  let obj;
  try {
    obj = JSON.parse(stripped);
  } catch {
    return { stage: "parse", fenced, raw: stripped.slice(0, 160) };
  }
  const bad = violation(obj);
  if (bad) return { stage: "schema", fenced, detail: bad };
  return { stage: fenced ? "pass-after-fence-strip" : "clean", fenced };
}

The two things people forget to instrument:

• envelope — the API returned 200 but the body was an error page or an HTML interstitial. This happens constantly on free hosting.
• pass-after-fence-strip — your parser "works" only because something upstream tolerates ```json wrappers. Count it. It is a repair, and repairs break the day you change a system prompt.

Then loop, aggregate, and write evidence:

BASE_URL="$BASE" API_KEY="$KEY" MODEL="$MODEL" TRIALS=50 node contract-gate.mjs

The script exits 1 when clean conformance is below MIN_CONFORMANCE. Make CI consume that exit code. A gate that only prints is a dashboard, not a gate.

Step 4 — One repair attempt, measured separately

If a trial fails on parse or schema, send one follow-up turn containing the raw assistant output plus the exact violation string, and ask again. Cap it at one. Then record recovered per trial.

Why cap it at one? Because each retry is another billed call against whatever allowance you are on, and because a model that needs two repair turns has told you its contract adherence is unreliable, not that your loop needs depth. Retry depth hides the signal you are trying to collect.

Step 5 — Decide with a table you can defend

Observed on n=50 (temperature 0)	Read it as	Action
clean ≥ 0.98, zero fence strips	contract holds	ship; re-run on any model or prompt change
clean 0.90–0.98	intermittent drift	ship only with strict validation and a user-visible fallback path
clean < 0.90, repair recovery ≥ 0.8	model needs help	one bounded repair turn, then fail closed
clean < 0.90, repair recovery < 0.8	the contract is fiction	block the workflow
any invariant violation	silently wrong data	block immediately, ignore the rate

That last row is the whole point. A 0.99 pass rate that occasionally says "high severity" with needs_human: false has not passed anything.

Step 6 — Freeze the evidence so it expires honestly

Write one JSON file per run and commit it. Minimum fields:

• timestamp, endpoint_host, model_string, prompt_hash
• trials, clean, pass_after_fence_strip, http, envelope, parse, schema
• recovered_after_repair
• threshold, exit_code
• up to five raw failure snippets, truncated

Add two rules to the runbook: regenerate the evidence if the prompt hash changes, the model string changes, or the file is older than fourteen days. Free tiers move without changelogs. Evidence does not age gracefully.

Running it on a free allowance

This is where a free-tier environment earns its keep. MonkeyCode offers free model access and a free server option, which lets you run the gate in the same place you intend to run the prototype instead of on a laptop that has a different network path and a different DNS resolver.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The free server matters for a specific reason: envelope failures are environmental. A captive portal, an egress proxy, or a platform error page will look like a model failure if you only ever test locally. Run the harness where you ship, and the envelope count becomes real.

Budget the gate itself. A fixed two-message prompt with max_tokens: 200, fifty trials, plus up to fifty repair turns is a small, predictable spend. The published allowance on the free tier is 10M tokens and a free server option as of this writing; confirm the current figures in the console before you plan around them, because allowances are not contracts and this one can change.

Where this gate is the wrong tool

• Your product tolerates free text. If a human reads the output, schema conformance is overhead.
• You need latency or throughput numbers. This gate says nothing about either.
• You cannot fix the prompt. If engineering does not own the system prompt, the gate will keep failing for reasons you cannot act on.
• You are testing at non-zero temperature but shipping at zero. Fix the experiment before trusting the result.
• You need a statistical claim. n=50 is a smoke test. One failure is two percent. Say so in the merge request.

The compressed version

Define "valid" in code. Run fifty calls at the temperature you ship. Classify every failure into http, envelope, parse, schema, or invariant. Count fence stripping as a repair, not a pass. Allow one repair turn and measure recovery. Commit the evidence with a prompt hash and an expiry date. Block on any invariant violation regardless of the pass rate, and make the script exit non-zero so nobody can merge past a bad number.

If you want a place to run that harness without spending anything to find out whether it works, the free tier and free server are the cheapest way to get the first evidence bundle. Just run it before the demo, not after.