Case Study: I Pinned 18 Prompts to a Free Model Endpoint. Two Broke Without a Code Change.

Free model endpoints drift. Your code can stay identical for a week and the answers still change. I built a 120-line C++ harness that pins prompts to expected output patterns; in six days it caught two silent regressions that the application itself would have tolerated. This is the story of that harness, the numbers it produced, and the rule I kept: pin prompts, not just code.

Background: a classifier that worked

The project was small: a C++17 tool that classifies git commit messages into bugfix, feature, refactor, or chore, using MonkeyCode's free model access for the classification. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

For two weeks, the label distribution looked sane. Then, on a Tuesday, the refactor bucket grew by eleven percentage points overnight. No commit. No deploy. No dependency change. The binary was the same one that had passed its tests the week before.

I had two explanations: the data changed, or the model changed. The data had not. The model had.

The goal: three properties

I wanted a prompt regression harness with three properties:

1. Pin a prompt to an expected pattern, not to a full golden string.
2. Fail loudly on timeout, empty body, or pattern mismatch.
3. Run on a schedule without my laptop.

Property three is why the free server option mattered. A one-off check is archaeology. A scheduled check catches drift the day it happens.

Implementation: promptpin

The harness is promptpin: a directory of JSON fixtures, one HTTP call per fixture, one validator per fixture. Fixture format:

{
  "name": "classify-json-bugfix",
  "prompt": "Classify this commit message. Reply with JSON: {\"label\": \"...\"}\n\nCommit: fix crash in cache eviction when TTL is zero",
  "expect_regex": "\"label\"\\s*:\\s*\"bugfix\"",
  "min_chars": 8
}

The core validation loop is short:

Result run_one(const Fixture& fx, const std::string& endpoint, long timeout_ms) {
  // POST fx.prompt to endpoint; capture body and HTTP code via libcurl.
  if (rc != CURLE_OK)
    return {fx.name, false, 0, elapsed_ms,
            std::string("transfer failed: ") + curl_easy_strerror(rc)};
  if (http_code != 200)
    return {fx.name, false, http_code, elapsed_ms, "HTTP " + std::to_string(http_code)};
  if (static_cast<int>(body.size()) < fx.min_chars)
    return {fx.name, false, http_code, elapsed_ms, "body too short"};
  std::regex re(fx.expect_regex);
  if (!std::regex_search(body, re))
    return {fx.name, false, http_code, elapsed_ms, "pattern not matched"};
  return {fx.name, true, http_code, elapsed_ms, "ok"};
}

Build and run:

g++ -std=c++17 promptpin.cpp -o promptpin -lcurl
./promptpin fixtures/ --endpoint "$MC_ENDPOINT" --timeout-ms 8000

Three implementation decisions matter.

1. Regex, not golden strings. Free models rephrase. A golden-string comparison produces false alarms. A regex pins the contract without over-fitting.
2. Hard timeout, not retry. A retry masks a hang. A timeout surfaces it. Any fixture over eight seconds is a failure, so a slow endpoint is visible in the table.
3. Nonzero exit on any failure. That makes the harness a gate. In CI, a regression fails the build.

The scheduled run

I ran the harness as a scheduled job on MonkeyCode's free server option. Three runs in six days: day 1, day 3, day 6. Each run took about four minutes for eighteen fixtures.

Run	Passed	Failed	Failure detail
Day 1	18	0	baseline
Day 3	17	1	JSON key casing changed: label → Label
Day 6	16	2	greeting prefix added: Here is the answer:

Both failures were invisible to the application. The JSON parser was case-insensitive, so Label worked. The one-word classifier took the last token, so the greeting was harmless. The harness flagged both anyway.

That is the point. The harness is stricter than the parser on purpose. You want to know the endpoint changed before it changes something your parser cannot tolerate.

Lessons learned

1. Pin prompts, not just code. Code review assumes a stable environment. With a free model endpoint, the environment is the product, and it changes without a changelog.
2. Validators must be stricter than parsers. If your parser tolerates drift, your tests will not see it. The harness is the early warning system; the parser is the last line of defense.
3. Scheduling is the feature. A harness you run once is archaeology. A harness that runs on a schedule is a gate.
4. Timeouts are not retries. Retrying a hang makes it someone else's problem later. Failing fast makes it your problem now, when it is cheap.

Limitations: who should not use this

This approach is not for everyone.

• If your endpoint is versioned and stable under a paid SLA, a prompt regression harness adds noise. Drift is already handled upstream.
• If your prompts change daily, you will spend more time maintaining fixtures than catching regressions. Pin stable prompts; do not pin experiments.
• A regex harness cannot measure quality. It detects format drift, not semantic decay. If the model starts returning plausible but wrong labels, this harness will pass it.
• The numbers here are one project, one endpoint, six days. They are evidence for the method, not a benchmark of any model.

Who should use this

Build a prompt regression harness if you ship code that calls a free model endpoint, you have no version pinning, and a silent behavior change would cost you debugging time. The cost is one C++ file and a fixtures directory. The payoff is knowing the day the endpoint changed, not the week after.

If you want to run this pattern yourself, MonkeyCode's free model access and free server option cover both halves: the model calls and the scheduled gate.