Treat Prompt Changes Like Schema Migrations: A Free, Diffable LLM Smoke Test

Morgan Sun — Wed, 05 Aug 2026 07:04:55 +0000

Every team I've seen adopt an LLM feature goes through the same arc. Week one: the demo works. Week three: someone swaps a model version or rewrites the system prompt, half the downstream parsing quietly breaks, and nobody notices until a user complains. The uncomfortable truth is that prompts are configuration with production impact, yet we review them with less rigor than a CSS change.

This article is about borrowing a habit from database work: treat every prompt or model change like a schema migration. Before it merges, run a small, fixed smoke suite against it. Keep the raw outputs. Make the diff reviewable. I'll show a compact implementation that runs on whatever free compute you have available, including a genuinely free hosted option.

What we're actually testing

Forget leaderboard-style benchmarks for a moment. The question a smoke suite answers is much narrower: did the behavior I depend on survive this change? That breaks down into three properties worth pinning down:

Property	Example of breakage	Cheap way to check
Output shape	JSON answer now wrapped in markdown fences	Parse attempt + key presence
Content constraints	Phone number leaks into a summary	Blocklist substring/regex
Verbosity / tone drift	One-line answer becomes five paragraphs	Length budget, sentence count

None of these judge quality. They judge contract compliance — the same thing a unit test does for a function. Quality review stays a human job; the suite just makes sure the contract didn't silently change while nobody was looking.

The setup

Three pieces, no framework:

A YAML (or JSON) file of scenarios — input text plus expectations.
A runner script that hits any OpenAI-compatible chat endpoint and evaluates expectations.
A snapshots/ folder, committed to git, holding every raw response per run.

Here's a scenario file, scenarios.yaml, with different examples than the usual "extract a date" demos:

- name: ticket-triage-tags
  prompt: |
    Classify this support ticket with up to two tags from [billing, bug, account, feature].
    Reply with a comma-separated list only.
    Ticket: "I was charged twice after upgrading my plan yesterday."
  expect:
    - kind: allowed_values
      source: [billing, bug, account, feature]
      separator: ","
      max_items: 2

- name: no-internal-ids
  prompt: |
    Write a one-sentence customer-facing apology for order #88412 being delayed.
  expect:
    - kind: forbidden_substrings
      values: ["#88412", "88412"]
    - kind: max_sentences
      count: 1

- name: structured-slot-fill
  prompt: |
    Fill this template as strict JSON: {"city": ..., "unit": "celsius"|"fahrenheit"}
    Request: "What's the weather like in Berlin, in Fahrenheit?"
  expect:
    - kind: strict_json
    - kind: json_field_equals
      field: unit
      value: fahrenheit

Notice the checks are all mechanical: enumerations, forbidden strings, sentence counts, schema fields. Mechanical is the feature — a flaky semantic judge at temperature 0 is still flaky.

The runner (Node.js, no dependencies)

I wrote this one in Node with zero npm packages — fetch is built in since Node 18 — so it runs anywhere without an install step:

// smoke.mjs — run: LLM_BASE=... LLM_MODEL=... node smoke.mjs
import { readFileSync, writeFileSync, mkdirSync } from "node:fs";
import { execSync } from "node:child_process";

const BASE  = process.env.LLM_BASE;          // any OpenAI-compatible base URL
const KEY   = process.env.LLM_KEY ?? "";
const MODEL = process.env.LLM_MODEL;
const stamp = new Date().toISOString().replace(/[:.]/g, "-");
const outDir = `snapshots/${stamp}`;

// Tiny YAML-subset parser avoided for brevity: scenarios stored as JSON here.
const scenarios = JSON.parse(readFileSync("scenarios.json", "utf8"));

function evaluate(text, exp) {
  const t = text.trim();
  switch (exp.kind) {
    case "forbidden_substrings":
      return exp.values.every(v => !t.includes(v));
    case "max_sentences":
      return t.split(/[.!?]+/).filter(s => s.trim()).length <= exp.count;
    case "strict_json":
      try { JSON.parse(t); return true; } catch { return false; }
    case "json_field_equals":
      try { return JSON.parse(t)[exp.field] === exp.value; } catch { return false; }
    case "allowed_values": {
      const items = t.split(exp.separator).map(s => s.trim().toLowerCase());
      return items.length <= exp.max_items &&
             items.every(i => exp.source.includes(i));
    }
    default:
      throw new Error(`unknown check: ${exp.kind}`);
  }
}

async function ask(prompt) {
  const res = await fetch(`${BASE}/chat/completions`, {
    method: "POST",
    headers: { "content-type": "application/json",
               ...(KEY ? { authorization: `Bearer ${KEY}` } : {}) },
    body: JSON.stringify({
      model: MODEL,
      messages: [{ role: "user", content: prompt }],
      temperature: 0
    })
  });
  if (!res.ok) throw new Error(`HTTP ${res.status}: ${await res.text()}`);
  return (await res.json()).choices[0].message.content;
}

mkdirSync(outDir, { recursive: true });
let failures = 0;

for (const s of scenarios) {
  const output = await ask(s.prompt);
  const detail = s.expect.map(e => ({ ...e, ok: evaluate(output, e) }));
  const ok = detail.every(d => d.ok);
  writeFileSync(`${outDir}/${s.name}.json`,
    JSON.stringify({ output, detail }, null, 2));
  console.log(`${ok ? "ok  " : "FAIL"}  ${s.name}`);
  if (!ok) failures++;
}

console.log(`\n${scenarios.length - failures}/${scenarios.length} scenarios passed → ${outDir}/`);
try { execSync(`git add ${outDir}`); } catch {}
process.exit(failures ? 1 : 0);

(For simplicity the snippet reads scenarios.json; if you want YAML, one line with any YAML parser works — the harness logic is identical.)

Because the script exits non-zero on any failure, it drops straight into a pre-merge CI job. A prompt refactor that breaks the tag classifier now fails the pipeline the same way a broken test would. That's the entire trick: giving prompt changes a merge gate.

The snapshot folder is the real product

Pass/fail tells you that something moved. The committed snapshots tell you what moved, and git already knows how to show it to you:

diff -r snapshots/2026-08-01T09-00-00 snapshots/2026-08-04T14-22-10

Typical findings look like: the model now answers with a preamble sentence (sentence-count checks start flapping), it title-cases the tags (your lowercase comparison was too strict — good, the suite just taught you your own assumption), or it emits fenced code blocks around JSON on Mondays and plain JSON on Fridays. You would never catch any of that by eyeballing outputs once.

Three habits make this work:

Pin temperature: 0. Not a determinism guarantee — providers ship silent updates — but it removes the noise you control.
Run on a schedule, not just on edits. A weekly CI cron catches provider-side drift even when your repo is untouched.
Harvest scenarios from real incidents. Every time a model misbehaves in production, the fix ends with a new scenario in the file. The suite grows exactly the way a test suite should.

Running it for free

A full pass here is a handful of short completions, so cost is genuinely a rounding error — but "rounding error" still assumes you have an endpoint and a machine. Two operator-supplied facts make MonkeyCode a convenient fit for this workflow: it currently offers free model access, and it offers a free server option, so you can run the runner itself in a hosted environment instead of your laptop or CI minutes. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Concretely, that means setting LLM_BASE and LLM_MODEL to your MonkeyCode environment and running node smoke.mjs there. Nothing in the harness is MonkeyCode-specific, deliberately: a local Ollama, a colleague's vLLM box, or another provider's trial tier all work by changing two environment variables. If the free option disappears or its limits stop fitting your runs, you migrate by editing CI secrets, not code.

Where this approach breaks down

It cannot measure goodness. "Is this summary helpful?" needs human sampling or an LLM-as-judge layer, which costs money and adds its own variance. This suite guards contracts, not quality.
Tiny suites give false confidence. Five scenarios will miss subtle regressions. Treat the file as a living artifact, not a one-time deliverable.
Free capacity is a policy, not an SLA. Model availability, rate limits, and how long free access lasts are all subject to change. Keep the endpoint behind environment variables so a provider change is a config change.
Temperature 0 ≠ reproducible forever. Identical inputs still drift across provider updates. That drift is what the scheduled re-runs exist to surface.

Skip this if…

You need statistically meaningful comparisons across thousands of prompts — reach for a dedicated eval framework instead. Your model output already flows through a strict parser or validator downstream that fails loudly — you may already have the safety net. Or you're tuning one disposable prompt for a hackathon — the setup time exceeds the value.

The takeaway

The code above is about 70 lines. The durable part is the posture: prompts and model choices are production configuration, and configuration deserves a merge gate and a history. Three scenarios from bugs you've actually suffered, one CI job, a committed snapshot folder — and "seems fine" becomes a diff you can point at in code review. If you want a zero-cost place to start iterating, the free model and server access mentioned above is one way to fill in LLM_BASE; the discipline works the same wherever you point it.

DEV Community: Morgan Sun