A 60-Line Contract Test for LLM JSON (and a Better AI Engineering Interview Answer)

When an LLM response drives a UI, a queue, or an automated decision, “the model usually returns JSON” is not a reliability strategy. Treat the response as untrusted input: parse it, validate the fields your program depends on, and keep a few bad examples in a regression suite.

This small Node.js exercise gives you a concrete way to discuss that design in an AI engineering interview. It also catches a failure that happy-path demos miss: an answer can look sensible to a person while still being unusable by software.

What contract are we actually protecting?

Imagine an incident-triage assistant. Your application needs three things from each response:

Field	Why the application needs it
summary	Something a human can scan quickly
risk	A bounded value that selects a workflow
next_action	A concrete, reviewable next step

That is a contract, not a prompt preference. A response such as "high risk, page the on-call" may be helpful prose, but it cannot safely drive a switch (risk) branch. Likewise, "urgent" is close to "high" in English but is not one of the values the rest of this application understands.

The useful boundary is:

model text → JSON parser → contract validator → application workflow
                  ↓                 ↓
             parse failure      validation failure

Each failure gets an observable result. Neither quietly becomes a plausible object.

Can we build the smallest useful validator?

Yes. The following file is deliberately dependency-free so that the testing idea is easier to inspect than the tooling. It accepts a triage object only when every required field is a non-empty string and risk belongs to the allowed set.

import assert from "node:assert/strict";

const schema = {
  required: ["summary", "risk", "next_action"],
  risk: new Set(["low", "medium", "high"]),
};

function validateTriage(value) {
  const errors = [];

  if (!value || typeof value !== "object" || Array.isArray(value)) {
    return ["result must be an object"];
  }

  for (const key of schema.required) {
    if (typeof value[key] !== "string" || value[key].trim() === "") {
      errors.push(`${key} must be a non-empty string`);
    }
  }

  if (!schema.risk.has(value.risk)) {
    errors.push("risk must be low, medium, or high");
  }

  return errors;
}

function parseAndValidate(text) {
  let value;
  try {
    value = JSON.parse(text);
  } catch (err) {
    return { ok: false, errors: [`invalid JSON: ${err.message}`] };
  }

  const errors = validateTriage(value);
  return { ok: errors.length === 0, value, errors };
}

There are two deliberate choices here.

First, parsing and validation are separate. JSON.parse only tells us whether text has JSON syntax. It does not tell us whether the JSON is the shape our program can use.

Second, the validator returns errors instead of inventing fallback values. A default risk: "low" would make a broken model response look safe. That is exactly the kind of bug an interviewer will probe: “What happens when the output is malformed?” The honest answer is that the application records the failure and uses an explicit recovery path, rather than silently taking an action on guessed data.

In production, I would put the same contract in a shared schema package and use the schema tool already adopted by the codebase. The important design decision is not this particular hand-written validator; it is having one executable source of truth at the boundary.

Which failures deserve a regression test?

The happy path proves almost nothing. Start with representative failures your downstream code could otherwise mistake for success:

const cases = [
  [
    "accepts a usable result",
    '{"summary":"payment retry spike","risk":"high","next_action":"pause deploy"}',
    true,
  ],
  [
    "rejects prose around JSON",
    'Here you go: {"summary":"x","risk":"low","next_action":"watch"}',
    false,
  ],
  [
    "rejects an unsupported enum",
    '{"summary":"x","risk":"urgent","next_action":"page on-call"}',
    false,
  ],
  [
    "rejects a missing action",
    '{"summary":"x","risk":"medium"}',
    false,
  ],
];

for (const [name, input, expected] of cases) {
  const result = parseAndValidate(input);
  assert.equal(result.ok, expected, `${name}: ${result.errors.join(", ")}`);
  console.log(`✓ ${name}`);
}

Run it with:

node contract-test.mjs

Expected output:

✓ accepts a usable result
✓ rejects prose around JSON
✓ rejects an unsupported enum
✓ rejects a missing action

The second case is worth keeping even if your provider offers a structured-output mode. Transport, middleware, model changes, and hand-edited fixtures can still hand your boundary something unexpected. Provider constraints can reduce the frequency of bad output; local validation defines how your system behaves when bad output arrives.

How should a real service recover?

Rejecting an invalid result is only half the design. The recovery policy should match the cost of the action:

Output use	Reasonable recovery
Draft text for a human	Show the validation error and offer a retry
UI classification	Render an “unable to classify” state, then retry within a small budget
Ticket or email creation	Hold for review; never send a fabricated substitute
Deployment, payment, or access change	Stop the automation and require an explicit human decision

Notice what is absent: an unlimited retry loop. A malformed-output retry needs a bounded attempt count, structured logs, and enough context to reproduce the failure without storing sensitive prompts or customer data by default.

That gives you a clean interview answer: “I treat model output as an integration boundary. I validate it locally, test both parse and semantic failures, and choose recovery based on the blast radius of the downstream action.”

What would I add after this exercise?

This tiny example omits several production concerns on purpose. The next increments should be driven by an actual risk, not by a desire to make the demo look enterprise-ready:

1. Add a version field when two consumers might evolve at different speeds.
2. Capture a redacted sample and a request identifier when validation fails.
3. Add contract fixtures whenever a prompt, provider, or schema changes.
4. Measure validation-failure rate separately from ordinary model latency.
5. Test the caller’s fallback path, not only the validator.

Those steps make the system easier to reason about because they preserve a simple rule: no unvalidated model text crosses into business logic.

How do you practice explaining this without sounding rehearsed?

Try narrating the four cases before looking at the code. Explain why prose around JSON fails, why an enum is a business rule rather than a parsing rule, and why a fallback is different for a draft than for a payment action.

For timed mock practice, aceround.app — an AI interview assistant can be useful as a neutral interviewer: ask it to challenge the recovery policy, then revise the explanation until you can connect the code to a real failure mode in two minutes.

The point is not to memorize a “correct” AI answer. It is to make a specific engineering judgment visible: models can generate useful text, but programs still need explicit contracts.

---

AI disclosure: AI assisted with outlining and copy editing. The code, failure cases, and technical claims were reviewed and verified before publication.

Comments

[Raju Dandigam]

Treating model JSON as untrusted input is still under-practiced, so I like how concrete this example is. The important follow-on, in my experience, is testing the recovery path with the same seriousness as the validator: what gets logged, what the user sees, and whether the next step retries, escalates, or safely degrades. Capturing a redacted bad sample plus request or trace IDs also pays off later when you need to separate prompt regressions from schema regressions. This is a strong interview exercise because it shows you understand operational behavior, not just parsing.