Case Study: Freeze the Error Taxonomy Before an Agent Writes the Error Mapper

Six slightly different 4xx answers for the same failure is exactly what you get when an agent writes your error mapper without a frozen taxonomy. Fixing it costs one contract file, one hash check, and a test that runs in under a second. This case study walks a small project end to end: four known failure modes, one generated mapper, and the freeze step that keeps the generator honest. Treat the listings below as a reference implementation you should run in your own repository before you adopt any of it.

Background: the mapper is downstream of decisions nobody wrote down

The service in this example sits behind a queue worker and answers three clients: a web app, a mobile app, and a partner integration. Each client decides independently whether to retry, which means each client needs a stable answer to two questions: is this failure retryable, and what should we show the user? Nobody had written those answers down, because they lived in the heads of the people who wrote the original catch blocks.

When an agent generates a mapper from that state, it produces something plausible and inconsistent. It picks 422 for one validation failure and 400 for its sibling, marks a rate-limit response non-retryable because the word "limit" reads like a permanent condition, and forwards err.message into the response body where the partner client can read a stack trace. Every one of those choices is defensible in isolation and wrong in aggregate.

The problem is not that the agent is careless. The problem is that you handed it a task with no acceptance criteria, so it optimized for readability instead of contract stability.

Goal: freeze the taxonomy, not the implementation

The freeze here is narrow and deliberate. You commit a machine-readable list of error codes with their HTTP status, retry semantics, message key, and log level; that file is the contract. You then let the agent write, rewrite, or throw away the mapper as often as it likes, because the mapper is now a pure function of a file it cannot edit.

Two properties make this work in practice.

1. Every consumer-visible field is present in the frozen file, so the mapper has no room to invent policy.
2. The frozen file is hash-checked in CI, so a helpful regeneration pass cannot quietly "improve" your taxonomy.

The artifact: contract/error-cases.json

Keep the file boring and dependency-free. JSON parses everywhere, diffs cleanly in review, and needs no YAML loader in your test runner.

{
  "version": 1,
  "frozen_at": "2026-09-15",
  "cases": [
    { "code": "AUTH_TOKEN_EXPIRED", "http": 401, "retryable": false, "message_key": "errors.auth.expired", "log_level": "info" },
    { "code": "RATE_LIMITED",       "http": 429, "retryable": true,  "message_key": "errors.rate_limited", "log_level": "warn" },
    { "code": "UPSTREAM_TIMEOUT",   "http": 504, "retryable": true,  "message_key": "errors.upstream.timeout", "log_level": "error" },
    { "code": "VALIDATION_FAILED",  "http": 422, "retryable": false, "message_key": "errors.validation", "log_level": "info" }
  ]
}

Store the digest next to it, and regenerate that digest only through a reviewed pull request:

sha256sum contract/error-cases.json | cut -d' ' -f1 > contract/error-cases.sha256

The contract test that does the actual freezing

This runs on Node 20 or newer with no third-party packages. It asserts conformance rather than implementation, so the agent stays free to restructure the mapper however it prefers.

// test/error-contract.test.ts
import { test } from "node:test";
import assert from "node:assert/strict";
import { readFileSync } from "node:fs";
import { createHash } from "node:crypto";

const contract = JSON.parse(readFileSync("contract/error-cases.json", "utf8"));
const { toProblem } = await import("../src/error-mapper.js");

test("contract hash is unchanged", () => {
  const digest = createHash("sha256")
    .update(readFileSync("contract/error-cases.json"))
    .digest("hex");
  assert.equal(digest, readFileSync("contract/error-cases.sha256", "utf8").trim());
});

test("every frozen code maps to its exact status and retry flag", () => {
  for (const c of contract.cases) {
    const p = toProblem({ code: c.code, message: c.code });
    assert.equal(p.type, c.code, `type drift for ${c.code}`);
    assert.equal(p.status, c.http, `status drift for ${c.code}`);
    assert.equal(p.retryable, c.retryable, `retryable drift for ${c.code}`);
    assert.equal(p.message_key, c.message_key, `message key drift for ${c.code}`);
  }
});

test("unknown codes never echo internal text to a client", () => {
  const p = toProblem({ code: "ECONNRESET", message: "connect ECONNRESET 10.0.0.7:5432" });
  assert.equal(p.type, "INTERNAL");
  assert.equal(p.status, 500);
  assert.equal(p.detail, undefined);
});

The mapper it constrains is short enough to review by eye, which is the second half of the trick.

// src/error-mapper.ts  (generated; safe to regenerate)
import { readFileSync } from "node:fs";

const contract = JSON.parse(readFileSync("contract/error-cases.json", "utf8"));
const byCode = new Map(contract.cases.map((c: any) => [c.code, c]));

export function toProblem(err: { code?: string; message?: string }) {
  const hit = byCode.get(err.code ?? "");
  if (!hit) {
    return { type: "INTERNAL", status: 500, retryable: false, log_level: "error" as const };
  }
  return {
    type: hit.code,
    status: hit.http,
    retryable: hit.retryable,
    message_key: hit.message_key,
    log_level: hit.log_level,
  };
}

Wiring the generation loop with free model access and a free server

For this case study I drafted the mapper with MonkeyCode's free model access and ran the throwaway harness on its free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Both availability claims are operator-supplied and worth re-checking against the project's current offer page: free model access, a free server option, and an operator-stated free allowance of 10,000,000 tokens as of this writing. Verify the live quota, model list, and terms yourself before you plan any workload around them, and expect those terms to move over time.

The prompt matters more than the model choice here. Keep it explicit about the boundary:

You may edit ONLY src/error-mapper.ts.
contract/error-cases.json and contract/error-cases.sha256 are frozen:
do not edit, reformat, or "fix" them.
Run: node --test test/error-contract.test.ts
Done means: all three tests pass, no new dependencies.

Then run this check before you accept the patch, because a generator that edits two files instead of one is the most common failure in this loop:

git diff --exit-code -- contract/ || { echo "contract touched; reject the patch"; exit 1; }

What the harness blocks (example run, illustrative)

The rows below describe the failure classes this test design is meant to intercept. They illustrate the design, not a benchmark of any particular model.

Failure mode	Symptom without the freeze	Caught by
Status drift	VALIDATION_FAILED returns 400 in one handler and 422 in another	per-case status assertion
Retry inversion	RATE_LIMITED marked non-retryable, so clients stop backing off	retryable assertion
Silent renames	errors.auth.expired becomes errors.auth.expired_token	message key assertion
Leak	raw socket error text reaches the partner client	unknown-code test
Taxonomy edits	a helpful agent adds a fifth code mid-task	hash assertion

Decision table: when freezing pays off

Situation	Freeze first?	Why
Three or more independent clients consume your errors	Yes	Retry policy has become a public API
One internal caller, error semantics still being redesigned	No	The contract would change every week
An agent generates code faster than you can review it	Yes	The freeze is your review budget
Error text is user-visible and localized	Yes	Message keys must stay stable for translators

Limitations and who should not use this

The hash check creates real friction. Adding a genuinely new error code now requires a pull request that touches the contract, regenerates the digest, and re-runs the suite; if your taxonomy is still in flux, that friction will push people to bypass the check instead of respecting it.

The tests also only prove conformance. A mapper can pass all three assertions and still log at the wrong level, drop correlation IDs, or leak data through a field the contract never names. Freeze the taxonomy first, then write separate tests for redaction and logging.

Do not use this approach if you cannot run a test suite in CI, if you are mid-redesign of your error semantics, or if your latency budget depends on hand-tuned serialization you would never regenerate. It is also a poor fit for one-off scripts where nothing retries and nobody parses your status codes.

Lessons learned

1. Freeze the smallest artifact that carries policy, and leave everything else to the generator.
2. Assertions on consumer-visible fields catch far more than snapshot tests of whole response bodies.
3. A hash check turns a silent edit into a loud CI failure, which is the entire point.
4. Keep the frozen file small enough to actually read during review; a 200-line taxonomy gets rubber-stamped.
5. Re-verify every free-tier claim before depending on it, including allowance, model access, and server terms.

If you want a zero-cost place to run this draft-and-test loop, MonkeyCode's free model access and free server option are what I used for this case study, and a four-case harness like this one is a reasonable first project to try there. Check the current offer page before you commit, then keep the workflow only if the freeze genuinely reduces your review churn.