Snapshot Your Prompts: A Golden-File Test for Free Model Endpoints

Prompt changes break silently. A new system message alters tone. A reworded instruction drops a constraint. Users notice before you do. Golden-file tests catch these regressions early.

This tutorial builds a complete golden-file harness for a free model endpoint. You capture a sample response. You write invariant checks. You run checks on a schedule. Every stage has a verification step. Total time: about 20 minutes.

Why golden files, not exact matches

LLM outputs are nondeterministic. Even at temperature 0, the same prompt can produce different text. Exact string matching fails constantly. Golden files store the structure of a good response. Checks test invariants, not exact text.

A golden-file test answers one question. Does the new response still look like the old one? It does not measure quality. It measures regression. That is the right job for a cheap, fast check.

What you need

• A free model endpoint with an API key
• A free server that can run a scheduled task
• Node.js 18 or newer
• curl for manual verification

This tutorial uses MonkeyCode's free model access and the free server option for the scheduled runner. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Both availability claims are operator-supplied. Check current quotas and limits before you rely on them.

Stage 1: Capture a golden sample

Create a project folder. Save the capture script.

// capture.mjs — save a golden sample from a free model endpoint
import { writeFile } from "node:fs/promises";

const ENDPOINT = process.env.MODEL_ENDPOINT;
const API_KEY = process.env.API_KEY;

const prompt = "Explain idempotency in one sentence.";

const response = await fetch(ENDPOINT, {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
    "Authorization": `Bearer ${API_KEY}`,
  },
  body: JSON.stringify({
    model: "default",
    messages: [{ role: "user", content: prompt }],
    temperature: 0,
  }),
});

if (!response.ok) {
  throw new Error(`HTTP ${response.status}: ${await response.text()}`);
}

const data = await response.json();
const text = data.choices?.[0]?.message?.content ?? "";

await writeFile("golden.json", JSON.stringify({
  prompt,
  capturedAt: new Date().toISOString(),
  text,
}, null, 2));

console.log(`Saved ${text.length} chars to golden.json`);

The script assumes an OpenAI-compatible chat response shape. If your endpoint differs, adjust the data.choices line. Do not skip this step.

Run it.

export MODEL_ENDPOINT="https://your-endpoint.example/v1/chat/completions"
export API_KEY="your-key"
node capture.mjs

Verify the output.

cat golden.json

You should see a JSON object. It holds the prompt, a timestamp, and the response text. The response should be one sentence about idempotency. If the response is empty, debug the endpoint first. Do not continue with a broken capture.

Stage 2: Write invariant checks

The golden file is a baseline. Now write checks that compare new responses against it.

// check.mjs — verify a new response against golden invariants
import { readFile } from "node:fs/promises";

const golden = JSON.parse(await readFile("golden.json", "utf8"));
const ENDPOINT = process.env.MODEL_ENDPOINT;
const API_KEY = process.env.API_KEY;

const response = await fetch(ENDPOINT, {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
    "Authorization": `Bearer ${API_KEY}`,
  },
  body: JSON.stringify({
    model: "default",
    messages: [{ role: "user", content: golden.prompt }],
    temperature: 0,
  }),
});

if (!response.ok) {
  console.error(`FAIL: HTTP ${response.status}`);
  process.exit(1);
}

const data = await response.json();
const text = data.choices?.[0]?.message?.content ?? "";
const failures = [];

if (!text) failures.push("empty response");
if (text.length < 20) failures.push("response too short");
if (text.length > golden.text.length * 3) failures.push("response 3x longer than golden");
if (!/idempoten/i.test(text)) failures.push("missing key term");
if (/sorry|as an ai/i.test(text)) failures.push("refusal pattern detected");

if (failures.length) {
  console.error(`FAIL: ${failures.join(", ")}`);
  process.exit(1);
}

console.log(`PASS: ${text.length} chars, ${text.split(" ").length} words`);

Pick invariants that match your real risks. The list above is a starting point, not a standard.

Run the check.

node check.mjs

Verify the exit code.

echo $?

You should see 0 and a PASS line. Now break a check on purpose. Change the regex to /zzzz/i. Run again. The exit code becomes 1. That proves the harness fails when it should.

Stage 3: Schedule the check on a free server

A manual check is not a regression test. You need a scheduled runner. The free server option runs a loop for you.

#!/usr/bin/env bash
set -euo pipefail
LOG=checks.log
while true; do
  if node check.mjs >> "$LOG" 2>&1; then
    echo "$(date -u +%FT%TZ) PASS" >> "$LOG"
  else
    echo "$(date -u +%FT%TZ) FAIL" >> "$LOG"
  fi
  sleep 3600
done

Save it as runner.sh. Make it executable.

chmod +x runner.sh
./runner.sh

Verify after one hour. Read the log.

cat checks.log

Each line has a UTC timestamp and a verdict. You should see at least one PASS. If you see FAIL, inspect the matching log line above it. The check script prints the reason. If your server offers cron, use it instead of the loop. A long-running process can die with the session.

Stage 4: Fail fast in CI (optional)

Scheduled checks catch drift within hours. CI catches it before merge. Add a job that runs the same check.

name: prompt-regression
on: [pull_request]
jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: node check.mjs
        env:
          MODEL_ENDPOINT: ${{ secrets.MODEL_ENDPOINT }}
          API_KEY: ${{ secrets.API_KEY }}

Verify by opening a pull request that changes a prompt. The check runs against the live endpoint. A failing invariant blocks the merge. A passing run gives you a green check.

What this catches and what it misses

The harness catches empty responses, truncated output, refusals, and length explosions. It catches a prompt edit that removes a required term. It catches endpoint changes that break the response shape.

It does not catch subtle quality loss. A response can pass every invariant and still be worse. It does not measure accuracy, tone, or usefulness. It does not replace human review or a real eval set.

Limitations

Free endpoints have quotas. A check every hour consumes tokens. Track your usage before you scale the schedule. Cold starts can add latency. A slow first call may time out. Set a timeout in the fetch call if your server enforces one.

The golden file ages. Prompts change. Product requirements change. Re-capture the golden file after intentional prompt updates. Otherwise the old baseline flags the new behavior as a regression.

Who should not use this

Skip this approach if you need exact-output guarantees. Skip it if you already run a full eval suite with labeled data. Skip it if your prompt changes daily. The harness earns its keep on stable prompts that must not drift.

A final check

Run the whole loop once more. Capture, check, schedule, verify. Four stages, four verification steps. That is the complete workflow.

If you build a golden-file harness, tell me which invariants caught real regressions. That is the data that matters.