The Schema Repair Loop: A Take-Home Where the First Reply Is Allowed to Be Wrong

Structured output is the first place a cheap or free model breaks, so the interesting engineering is not the first reply but the recovery path after it. A take-home that grades only the happy path rewards a candidate for picking a cooperative prompt and a lucky sample. A take-home that grades the repair loop rewards the candidate for assuming drift, measuring it, and failing closed when repair stops working. This packet defines the brief, a weighted rubric, a reference implementation, and the failure modes that separate the two groups.

The brief handed to the candidate

Build an endpoint that turns a free-text support ticket into a validated record. The record must match a supplied JSON schema. The model may return invalid output, and your job is to recover without human help.

Hard constraints keep the exercise comparable across candidates:

1. One generation function is injected; the candidate may not swap providers mid-run.
2. A validation failure may trigger at most two repair attempts before the call returns a typed failure.
3. The endpoint must log the attempt count and the exact validation errors for every request.
4. The test suite must run offline against recorded fixtures, with no network access.
5. Fenced Markdown, prose preambles, and trailing commentary must be handled explicitly, not by a catch-all regex.

The schema is deliberately small but not trivial: ticket_id (string), severity (enum of low, high), components (array of strings), and customer_impact (boolean). Candidates who hand-roll a validator and candidates who pull a library are both acceptable, as long as the failure messages are specific enough to feed back into a repair prompt.

What the take-home does not ask for

The brief never asks for a wrapper around a single vendor SDK, and it never asks for retry logic that hides errors. Reviewers should treat an elaborate retry backoff as a distraction from the graded question, which is prompt repair quality. A candidate who adds exponential backoff without attempt telemetry has optimized the wrong variable.

Rubric, weighted for review speed

Weight	Criterion	What earns full credit
40%	Repair loop correctness	Parse and validation failures are distinguished; attempts are capped; failure is typed
25%	Failure specificity	Error messages name the path and the expectation, so a repair prompt is actionable
20%	Offline test suite	Fixtures cover clean, fenced, truncated, and unrecoverable outputs
15%	Operational honesty	Logs show attempt counts and the final status without leaking raw customer text

Two reviewers can apply this table in about fifteen minutes. The 40% line is the only one that requires reading code closely, and it is the one that predicts on-call behavior.

Reference solution

The snippet below is a starting point for reviewers, not a benchmarked result. It uses a hand-rolled validator so the exercise stays dependency-free, and it treats parsing and validation as separate failure classes.

// validator.js — minimal subset: type, required, properties, items, enum
export function validate(schema, value, path = '$') {
  const errors = [];
  const actual = Array.isArray(value) ? 'array' : typeof value;

  if (schema.type && actual !== schema.type) {
    return [`${path}: expected ${schema.type}, received ${actual}`];
  }
  if (schema.enum && !schema.enum.includes(value)) {
    errors.push(`${path}: expected one of ${schema.enum.join(' | ')}, received ${JSON.stringify(value)}`);
  }
  if (schema.type === 'object') {
    for (const key of schema.required ?? []) {
      if (!(key in value)) errors.push(`${path}.${key}: required field missing`);
    }
    for (const [key, sub] of Object.entries(schema.properties ?? {})) {
      if (key in value) errors.push(...validate(sub, value[key], `${path}.${key}`));
    }
  }
  if (schema.type === 'array' && schema.items) {
    value.forEach((item, index) => errors.push(...validate(schema.items, item, `${path}[${index}]`)));
  }
  return errors;
}

The loop keeps the task text stable and replaces only the error block, which keeps token use roughly flat across attempts.

// repair-loop.js
import { validate } from './validator.js';

const MAX_REPAIRS = 2;

export async function extractRecord({ generate, schema, task }) {
  const trace = [];
  let prompt = task;

  for (let attempt = 1; attempt <= MAX_REPAIRS + 1; attempt++) {
    const raw = await generate(prompt);
    const candidate = parseModelJson(raw);

    if (!candidate.ok) {
      trace.push({ attempt, stage: 'parse', errors: [candidate.error] });
      prompt = `${task}\n\nReturn a single JSON object. No prose, no code fences, no trailing text.`;
      continue;
    }

    const errors = validate(schema, candidate.value);
    trace.push({ attempt, stage: 'validate', errors });
    if (errors.length === 0) return { ok: true, value: candidate.value, attempts: attempt, trace };

    prompt = `${task}\n\nThe previous object failed validation:\n- ${errors.join('\n- ')}\nReturn a corrected JSON object only.`;
  }

  return { ok: false, error: 'validation_budget_exhausted', attempts: MAX_REPAIRS + 1, trace };
}

function parseModelJson(raw) {
  const text = String(raw ?? '').trim().replace(/^```
{% endraw %}
(?:json)?\s*|\s*
{% raw %}
```$/g, '');
  if (!text) return { ok: false, error: 'empty response' };
  try {
    return { ok: true, value: JSON.parse(text) };
  } catch (error) {
    return { ok: false, error: `json parse failed: ${error.message}` };
  }
}

Fixtures the suite must contain

• clean.json — a valid object on the first attempt; the loop must return attempts: 1.
• fenced.md — the same object wrapped in a JSON code fence; the loop must not consume a repair attempt.
• bad-enum.json — a severity value outside the enum followed by a corrected object.
• truncated.json — an unterminated object on every attempt; the loop must return validation_budget_exhausted.

The truncated case is the one reviewers should weigh most heavily, because it proves the candidate can stop. A loop that never gives up will pass every manual demo and fail in production under a degraded provider.

Where a free endpoint changes the exercise

Offline fixtures prove the control flow, but they say nothing about how a model reacts to an error block it has never seen. Running the same loop against a live endpoint exposes the difference between a repair prompt that names the failing path and one that restates the whole schema. Free model access is enough for this stage, since the volume is a few hundred short calls and the goal is drift, not throughput.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The operator describes free model access, a free server option, and a free allotment of ten million tokens; check the project page for current terms, because availability and limits change and this article does not restate them as permanent. The free server option matters for a different reason: it lets a reviewer hit the endpoint over HTTP instead of running the candidate's laptop, which makes the attempt logs observable during the review itself.

Common failure modes

1. Grading the first response. Candidates who score their own work on one clean sample will report success that the fixtures disprove.
2. Repair prompts that resend everything. Appending the schema, the task, and the candidate's full history inflates each attempt without improving the fix.
3. Silent fence stripping. A permissive regex that strips anything backticked hides parse failures instead of classifying them.
4. Unbounded loops. A while (true) around generation looks resilient and is the most common reason for a rejected submission.
5. Validator pass treated as truth. Schema conformance is not semantic correctness; a plausible ticket_id can still be wrong.
6. No attempt telemetry. Without per-attempt errors, nobody can tell whether repair failed or was never tried.

Who should not use this exercise

Teams hiring for UI-heavy roles get little signal here, and teams without a stable schema should fix the schema first. Candidates on constrained hardware should not be asked to host a model locally, since the graded artifact is the loop rather than the runtime. Anyone who needs deterministic byte-for-byte output should use a grammar-constrained decoder instead of a repair loop, because repair is a mitigation and not a guarantee.

If the goal is to see how someone behaves when a model misbehaves in a plausible way, this packet is cheap to run and hard to fake. The free model access and free server option are the shortest path to a live version of it; keep the rubric and the failure modes, and adjust the schema to your own domain.