DEV Community: kongkong

Leaderboards Are Not Evidence. Your Failure Corpus Is.

kongkong — Sun, 08 Nov 2026 19:05:35 +0000

A user opens a document, asks the assistant to rename one section, and gets back a patch that edits a heading that stopped existing two saves ago. The prose was fine; the patch was confident, well-formed, and wrong. Notice where it actually broke: not in the model, but in the seam between a stale snapshot, a schema your validator never enforced, and a retry that replayed the old state. That seam is the layer I keep losing afternoons to, and no leaderboard on earth would have flagged it.

So here is the position I want to argue, and I will argue it plainly: a public model ranking is not evidence about your application, and treating it as evidence is how a team migrates a working feature onto a worse model with a better average. Leaderboards measure a mean over someone else's distribution of tasks, prompts, and tolerances. Your product fails on a specific, embarrassingly small set of inputs — a heading that moved, a tool call that returned 409, a JSON field that arrived as a string. The only instrument that measures those is a corpus you build from your own failures.

I have watched this play out in the last week's discourse too. There has been a run of posts arguing that executable scripts beat prose instructions for coding agents, and a benchmarking challenge wrap-up where the interesting part was the harnesses people built, not the final scores. Both point the same direction, and I think it holds for model choice as well: the harness is the asset, and the score is a side effect. Build the harness first, then argue about numbers.

What a failure corpus actually is

It is a directory of frozen incidents, each one small enough to read in a minute and boring enough to run a thousand times. Not a golden dataset of happy paths, and not a scraped eval set that someone else curated against someone else's product. Each case records the input, the response your system produced, the layer where it went wrong, and the exact check that would have caught it.

Here is the fixture format I use. Every field is something I can fill in from a ticket, a log line, or a reproduction that someone pasted into chat at 6pm.

{
  "id": "0142-retry-after-partial-write",
  "source": "link the ticket, log line, or repro — not just 'prod'",
  "layer": "state",
  "input": {
    "messages": [
      {"role": "system", "content": "You edit documents by returning patch.v3 JSON only."},
      {"role": "user", "content": "Rename 'Pricing' to 'Plans'. The doc snapshot is attached."}
    ],
    "snapshot_rev": 41
  },
  "expect": {"kind": "json_schema", "schema": "patch.v3"},
  "must_not": ["pricing", "create_heading"],
  "observed": {"http_status": 409, "note": "client retried against rev 39"}
}

The layer field matters more than it looks. When a case fails, my first question is whether the model, the transport, the state, or the validator is responsible, and I want that answer written down before I start guessing. Most of my early corpus entries turned out to be state bugs wearing a model costume, which is exactly the sort of thing a ranking will never tell you.

The runner, with a seam where the vendor goes

The design decision I would defend hardest is keeping the model call behind an adapter, so the corpus never imports a vendor SDK. I have been burned by this. The eval script grows a dependency, the dependency pins a version, and now the thing that was supposed to judge a migration is itself the riskiest upgrade in the repository.

# adapters/base.py
from typing import Protocol

class ModelAdapter(Protocol):
    name: str
    def complete(self, messages: list[dict], *, temperature: float = 0.0) -> dict:
        """Return {"text": str, "usage": dict, "latency_ms": int}."""

# adapters/http_compatible.py
import json, os, time, urllib.request

class HttpCompatible:
    """Transport is deliberately boring; the seam is the point."""
    def __init__(self):
        self.name = os.environ["MODEL_LABEL"]
        self.base = os.environ["MODEL_BASE_URL"].rstrip("/")
        self.key  = os.environ["MODEL_API_KEY"]

    def complete(self, messages, *, temperature=0.0):
        body = json.dumps({
            "model": os.environ["MODEL_ID"],
            "messages": messages,
            "temperature": temperature,
        }).encode()
        req = urllib.request.Request(
            f"{self.base}/chat/completions",
            data=body,
            headers={"Content-Type": "application/json",
                     "Authorization": f"Bearer {self.key}"},
        )
        start = time.monotonic()
        with urllib.request.urlopen(req, timeout=60) as resp:
            payload = json.loads(resp.read())
        return {
            "text": payload["choices"][0]["message"]["content"],
            "usage": payload.get("usage", {}),
            "latency_ms": int((time.monotonic() - start) * 1000),
        }

That request shape is a guess about your provider, and you should replace it with whatever the docs actually show you. The point is not the URL; the point is that swapping candidates becomes an environment variable, so nobody has to negotiate a re-architecture to test a model.

Now the scorer, which I keep dumb on purpose: parse, validate, and look for forbidden moves.

# run_corpus.py
import json, pathlib, statistics

CASE_DIR = pathlib.Path("corpus")

def load_cases():
    return [json.loads(p.read_text()) for p in sorted(CASE_DIR.glob("*.json"))]

def check(case, out):
    result, text = {}, out["text"]
    if case["expect"]["kind"] == "json_schema":
        try:
            doc = json.loads(text)
            result["parses"] = True
            result["schema_ok"] = validate(doc, case["expect"]["schema"])
        except json.JSONDecodeError:
            result["parses"] = False
            result["schema_ok"] = False
    for token in case.get("must_not", []):
        result[f"avoids:{token}"] = token.lower() not in text.lower()
    return result

def run(adapter, cases):
    rows = []
    for case in cases:
        out = adapter.complete(case["input"]["messages"])
        checks = check(case, out)
        rows.append({"id": case["id"], "layer": case["layer"],
                     "passed": all(checks.values()), "checks": checks,
                     "latency_ms": out["latency_ms"]})
    lat = sorted(r["latency_ms"] for r in rows)
    return {
        "adapter": adapter.name,
        "cases": len(rows),
        "pass_rate": round(sum(r["passed"] for r in rows) / len(rows), 4),
        "p95_latency_ms": lat[max(0, int(len(lat) * 0.95) - 1)],
        "failing_layers": sorted({r["layer"] for r in rows if not r["passed"]}),
        "rows": rows,
    }

validate() is whatever JSON Schema validator you already run in production, and if you do not have one, that absence is the real finding. The corpus is only as good as the check it enforces, so I would rather ship five cases with real validators than fifty with a substring match.

The loop I actually run looks like this, and it costs about as long as a coffee refill.

python -m corpus.check                      # fixtures parse; no model calls; safe for CI
MODEL_LABEL=current   python run_corpus.py > reports/current.json
MODEL_LABEL=candidate python run_corpus.py > reports/candidate.json
python compare.py reports/current.json reports/candidate.json

Run the baseline twice before you trust it, because temperature zero is not a promise. A case that flips between two identical runs is telling you to fix the validator or the state handling, not the prompt. I have deleted more clever prompts than I care to admit after noticing the failure was nondeterministic plumbing underneath.

The part where I refuse to promote on one number

A better pass rate is necessary and nowhere near sufficient, because a candidate can buy two points overall by breaking a layer you have never had to check. So I gate on layers, on determinism, and on whether the p95 latency fits inside the timeout budget the client already uses. And I gate on provenance, because a corpus written by the person who wants the migration approved will be a corpus that approves the migration.

Signal	Promote	Hold
Pass rate vs baseline	Equal or better, zero new failing layers	Plus two overall, but a new failure in `state`
Determinism	Same verdict across three runs	Flips between runs
p95 latency	Comfortably inside client timeout	Near timeout, retries masked by the harness
Case provenance	Each case traces to a ticket or log line	Cases authored to flatter the candidate

That table is opinionated, and I mean it that way. A failing layer is a bug you have not fixed yet, and no average is worth shipping one more of those.

Where a free tier and a free server honestly fit

Let me be precise about the gap, because it is not the cost of the model calls. The real friction is having no budget line for a candidate you might reject, so the baseline never gets run and the decision gets made in a meeting instead. That is the gap MonkeyCode's operator describes filling: free access to its models and a free server option, with the current offer quoted as ten million free tokens.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I have not benchmarked MonkeyCode here, and I cannot vouch for quotas, model lists, server specs, or how long any of it lasts. Treat those claims as operator-supplied and check the project's current terms before you plan a quarter around them. What I can say is that the workflow above is exactly the shape a free tier is good for: run the corpus batch somewhere that is not your laptop and not your CI minutes, keep the adapter seam so leaving is a one-line environment change, and never point production traffic at infrastructure whose terms you have not read.

Who should not use this

If your incidents contain customer data that cannot leave your controlled environment, do not paste them into any third-party service, free or paid, until someone who reads contracts says otherwise. If you need an uptime guarantee and a number you can page, a free server is an evaluation venue, not a place to serve users. And if your product has fewer than a handful of real failures recorded, you do not have a distribution yet — you have anecdotes, and a candidate that passes your forty cases has simply passed your forty cases.

Corpora also rot in a way that benchmarks do not. When the schema changes, half your cases silently test the old contract, so re-run them on every schema revision and delete the ones that no longer describe a failure you would care about. A stale corpus is worse than none, because it produces confident green checkmarks about a system you no longer run.

So which layer is your least stable handoff right now — the model, the transport, the state, or the validator? Send me one concrete failure and the response code it returned, and I will tell you what I would write as the fixture.

If you want a place to run that first batch without asking anyone for a card, MonkeyCode's free models and free server are a reasonable starting line.

mkdir -p corpus reports && cp your-first-incident.json corpus/0001.json
python -m corpus.check && echo "corpus is loadable; now make a candidate prove itself"

Meter Cost per Accepted Answer, Not per Call

kongkong — Sat, 07 Nov 2026 17:45:36 +0000

Meter Cost per Accepted Answer, Not per Call

Last Tuesday a merged pull request fired the "Draft changelog" button in our internal release dashboard, and the model answered in about four seconds with JSON that read fine to a human. It failed one layer later, inside the worker's schema check, because body came back as an array of strings instead of a string. My retry loop then did exactly what I had told it to do: it appended the validation error to the same conversation and asked again, which is a reliable way to make attempt two cost more than attempt one. My dashboard said everything was healthy, because it measured tokens per call.

I want to argue a specific, unhedged position here: the billable unit of an AI feature is the accepted result, and per-call metrics are a form of self-deception that happens to be the easiest thing to build. Every rejected attempt is a small number that dissolves into a large average, so the chart stays flat while the thing you actually pay for quietly gets worse. A free token grant makes this failure mode more likely rather than less, because nothing hurts while you are forming the habit, and the habit you form at zero cost is the habit you keep when the cost is real. Would you rather learn this in month one at zero dollars, or in month three on an invoice you have to explain?

So I built the smallest object that tells the truth: a ledger keyed by task id that counts every attempt and reports tokens per accepted task. It has no dependencies, it fits in about sixty lines, and the test below runs offline against a fake transport. I have not benchmarked any provider with this code, and every number in this post is arithmetic rather than measurement, so please read it as a method you can reproduce instead of a result you should trust.

The ledger

// meter.js
export function createMeter({ now = () => Date.now() } = {}) {
  const tasks = new Map();
  const row = (taskId) => {
    if (!tasks.has(taskId)) {
      tasks.set(taskId, { attempts: [], accepted: null });
    }
    return tasks.get(taskId);
  };

  return {
    attempt(taskId, usage, outcome = 'returned') {
      row(taskId).attempts.push({ usage, outcome, at: now() });
    },

    accept(taskId, artifact) {
      const t = row(taskId);
      if (t.accepted) throw new Error(`task ${taskId} already accepted`);
      t.accepted = { artifact, at: now() };
    },

    report() {
      const rows = [...tasks].map(([taskId, t]) => {
        const tokens = t.attempts.reduce((n, a) => n + a.usage.total_tokens, 0);
        return {
          taskId,
          attempts: t.attempts.length,
          tokens,
          accepted: Boolean(t.accepted),
          tokensPerAccepted: t.accepted ? tokens : null,
        };
      });
      const done = rows.filter((r) => r.accepted);
      const acceptedTokens = done.reduce((n, r) => n + r.tokens, 0);
      return {
        rows,
        acceptedCount: done.length,
        tokensPerAcceptedTask: done.length ? Math.round(acceptedTokens / done.length) : null,
        abandonedTokens: rows.filter((r) => !r.accepted).reduce((n, r) => n + r.tokens, 0),
      };
    },
  };
}

The important design decision is that the meter never learns about a success from the model call, only from the acceptance rule. That inversion is what kills the flattering average, because there is no code path where a rejected attempt can be counted as a completed unit of work. Everything else here is bookkeeping, and bookkeeping you can rewrite in an afternoon.

The retry wrapper

// client.js
export async function generateWithRetry({ meter, taskId, transport, validate, prompt, maxAttempts = 3 }) {
  const messages = [{ role: 'user', content: prompt }];
  let lastError;

  for (let i = 1; i <= maxAttempts; i++) {
    const res = await transport({ messages });
    meter.attempt(taskId, res.usage, i === 1 ? 'first' : 'retry');

    let parsed;
    try {
      parsed = validate(JSON.parse(res.content));
    } catch (err) {
      lastError = err;
      // The correction you append is billed on the next attempt. Measure it.
      messages.push({ role: 'user', content: `Previous answer failed validation: ${err.message}. Return JSON only.` });
      continue;
    }

    meter.accept(taskId, parsed);
    return parsed;
  }

  throw lastError;
}

Notice the line that appends the validation error, because that line is the whole problem in miniature. Attempt two carries attempt one's prompt plus a correction, so a "cheap" retry is strictly more expensive than the call it replaces. If your only instrument is tokens per call, the second attempt looks like one more ordinary data point and the growth is invisible.

The test that fails on a naive meter

// test/meter.test.js
import test from 'node:test';
import assert from 'node:assert/strict';
import { createMeter } from '../meter.js';
import { generateWithRetry } from '../client.js';

const chat = (content, total) => ({ content, usage: { total_tokens: total } });
const validate = (o) => {
  if (typeof o.body !== 'string') throw new Error('body must be a string');
  return o;
};

test('counts every attempt behind an accepted answer', async () => {
  const meter = createMeter();
  const replies = [
    chat('{"title":"Fix login redirect","body":["a","b"]}', 1500), // rejected
    chat('{"title":"Fix login redirect","body":"Redirects now keep the return path."}', 1730),
  ];
  const transport = async () => replies.shift();

  const out = await generateWithRetry({
    meter, taskId: 'pr-4821', transport, validate, prompt: 'draft a changelog entry',
  });

  assert.equal(out.title, 'Fix login redirect');
  const report = meter.report();
  assert.equal(report.rows[0].attempts, 2);
  assert.equal(report.tokensPerAcceptedTask, 3230);
  assert.equal(report.abandonedTokens, 0);
});

test('separates abandoned work from accepted work', async () => {
  const meter = createMeter();
  const transport = async () => chat('{"title":"x"}', 900);
  await assert.rejects(
    generateWithRetry({ meter, taskId: 'pr-4822', transport, validate, prompt: 'draft', maxAttempts: 2 }),
  );
  const report = meter.report();
  assert.equal(report.tokensPerAcceptedTask, null);
  assert.equal(report.abandonedTokens, 1800);
});

Run it with node --test test/meter.test.js and you get a red bar against any meter that remembers only the last attempt. That test is the artifact I care about, because it locks the opinion into CI instead of leaving it in a document nobody rereads.

The arithmetic your dashboard hides

With the numbers above, tokens per call average out to 1615 and tokens per accepted changelog is 3230, a factor of two that no per-call chart will ever show you. Now imagine the acceptance rule rejects the second attempt too, which happens constantly when a validator is strict about types. The third attempt carries two corrections, the task finally lands, and your per-call average still looks reasonable while the real unit cost has tripled. Is that a model problem, a prompt problem, or a measurement problem? From where I sit it is almost always the third one, and it is the only one of the three you can fix without guessing.

Let the acceptance rule own the retry

The architectural consequence is small and it changed how I structure workers: the validator owns the retry decision, the meter owns the ledger, and the task id owns idempotency. The model call becomes a dumb function that takes messages and returns text plus usage, and it stops being the place where policy accumulates. Once those three responsibilities are separate, a network retry can no longer silently double-charge your ledger, because the ledger is keyed by task id and refuses to accept the same task twice. That guard has saved me more debugging time than any prompt rewrite I have done this year.

Running the loop where it is not your laptop

A meter that only runs during local testing measures your enthusiasm, not your feature. You need the worker, the ledger write, and the schedule living on a machine that keeps going after you close the lid, which is why MonkeyCode's free model access and free server option are relevant to this workflow at all. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The framing I would defend is that the free grant is not really about free text generation, it is permission to run an embarrassing worker on a schedule against a real endpoint while the validator, the retry path and the ledger are all still wrong. Point the transport at a free model, deploy the worker to the free server, and let the acceptance rate and the abandoned-token number accumulate for a week before you touch a single prompt.

The specific grant is advertised as ten million free tokens at the time of writing, but check the current terms yourself, because free-tier conditions are set by the provider and they move. What you are buying with them is the observation window, and an observation window you never use is worth exactly what you paid for it.

What this does not tell you

Client-side usage numbers can disagree with provider billing, and when they do, the provider wins and your ledger needs an import step. Acceptance rate is a property of the validator you wrote, so a loose rule will make the meter look great while accepting garbage, and a strict rule can make a good model look expensive. One task type is not a cost model, and a week of changelog entries says nothing about a support summarizer with a longer prompt and a worse schema. Skip this approach entirely if you already get per-task billing from your provider, if you are building a demo that will never see a second user, or if your validator accepts everything by construction, in which case the meter is just a per-call chart with extra code.

A checklist you can steal

Key every attempt by task id, and let only the acceptance rule mark a task done.
Assert tokens per accepted task in a test with a fake transport before you point anything at a real endpoint.
Log the correction you append on retry, because that text is billed on the next attempt.
Track abandoned tokens separately, since work with no artifact is the cheapest thing to ignore and the most expensive to carry.
Run the loop on a schedule somewhere that stays up, and compare client-side totals to the provider's bill weekly.

Which handoff is least stable in your stack right now: the validator deciding to retry, the meter writing the ledger, or the worker surviving a restart mid-task? Send me a concrete failure state or response code and I will fold it into the next iteration of this meter, because the version I am running today is only the second one I have trusted.

Judge a Free AI Tier by Its Exit Cost, Not Its Entry Price

kongkong — Fri, 06 Nov 2026 16:22:59 +0000

Most of the AI conversation this week is about whether the tools will replace us, and I keep thinking that the more practical question is whether you can walk away from one. Last month I needed a working vertical slice of a support-triage feature — an HTTP endpoint, one Postgres row, one queued job — and I had an afternoon, not a sprint. So I put the slice on a free tier and it worked, which turned out to be the problem.

The demo answered requests beautifully. Then a colleague tried to run it on her machine and we spent three hours working out which parts of that "working" system were actually ours. The failure was not in the model call, and it was not in the response time either; it happened one layer down, at the handoff between a demo I could show and an artifact someone else could reproduce.

The demo is not the deliverable

I have started judging every free AI tier by one number that nobody puts on the pricing page, and that number is what it costs to leave. Entry price is easy to read, because it is usually zero and always loud. Exit cost is quiet: it hides in request shapes, in state you did not export, and in an environment you cannot rebuild without the vendor's dashboard open in another tab.

So my position is blunt. A free tier that gives you ten million tokens and a box to run them on is worth exactly as much as your ability to replay the same workload somewhere else. If you cannot do that, you did not evaluate a tool; you rented a demo and mistook the feeling for evidence.

Three exits actually matter, and I test them in this order. The first is the request shape, meaning how much of your code knows the vendor's JSON. The second is state, meaning whether the rows, files, and credentials your slice depends on can leave with a git clone. The third is the environment, meaning whether one command on a clean machine gets you a running service rather than a screenshot of one.

Put the request shape behind a seam you can swap

The first exit is the cheapest to fix and the easiest to forget, because a single httpx.post inside a route handler looks innocent until you have forty of them. I keep one protocol, one adapter, and nothing else in the application layer knows a base URL exists.

# app/providers/base.py
from dataclasses import dataclass
from typing import Protocol


@dataclass(frozen=True)
class Completion:
    text: str
    model: str
    request_id: str | None = None


class Provider(Protocol):
    name: str

    def complete(self, system: str, user: str, *, seed: int | None = None) -> Completion: ...

The HTTP adapter below is illustrative, not gospel: confirm the endpoint path and field names against the provider's current docs before you depend on them. What matters is that every vendor-specific detail lives in this one file.

# app/providers/http_provider.py
import os
import httpx
from .base import Completion


class HTTPProvider:
    name = "http"

    def __init__(self, base_url=None, api_key=None, model=None):
        self.base_url = (base_url or os.environ["LLM_BASE_URL"]).rstrip("/")
        self.api_key = api_key or os.environ.get("LLM_API_KEY", "")
        self.model = model or os.environ["LLM_MODEL"]

    def complete(self, system: str, user: str, *, seed=None) -> Completion:
        payload = {
            "model": self.model,
            "messages": [
                {"role": "system", "content": system},
                {"role": "user", "content": user},
            ],
        }
        if seed is not None:
            payload["seed"] = seed
        r = httpx.post(
            f"{self.base_url}/chat/completions",
            json=payload,
            headers={"Authorization": f"Bearer {self.api_key}"},
            timeout=60,
        )
        r.raise_for_status()
        body = r.json()
        return Completion(
            text=body["choices"][0]["message"]["content"],
            model=body.get("model", self.model),
            request_id=r.headers.get("x-request-id"),
        )

Now the seam has a contract, and a contract can be tested against any implementation, including a fake that never touches the network. That fake is the second half of the exit, because a provider seam you cannot stub is a provider seam you cannot test on a plane.

# tests/test_provider_contract.py
import pytest
from app.providers.http_provider import HTTPProvider
from app.providers.fake import FakeProvider


@pytest.fixture(params=["fake", "http"])
def provider(request):
    if request.param == "fake":
        return FakeProvider()
    return HTTPProvider()


def test_returns_non_empty_text(provider):
    out = provider.complete("You are terse.", "Reply with the word OK.", seed=7)
    assert out.text.strip(), "empty completion"
    assert out.model, "provider did not report which model answered"

Run it with pytest tests/test_provider_contract.py -k fake in CI on every pull request, and with the real provider before you merge anything that touches prompts. If the second run needs a credential that only lives in someone's browser session, you have already found your exit cost.

Replay the workload instead of admiring it

Comparing two providers by reading two outputs side by side is how people end up arguing about vibes for an hour. I keep a small frozen task file — twenty to fifty rows is plenty — and a harness that records what happened rather than what it looked like.

# scripts/exit_test.py
import hashlib, json, pathlib, time
from app.providers.http_provider import HTTPProvider
from app.providers.fake import FakeProvider

TASKS = json.loads(pathlib.Path("eval/tasks.json").read_text())


def run(provider, out_path: str) -> None:
    rows = []
    for t in TASKS:
        start = time.monotonic()
        try:
            c = provider.complete(t["system"], t["user"], seed=t.get("seed"))
            rows.append({
                "task_id": t["id"], "ok": True, "model": c.model,
                "hash": hashlib.sha256(c.text.encode()).hexdigest()[:16],
                "chars": len(c.text),
                "ms": round((time.monotonic() - start) * 1000),
            })
        except Exception as e:
            rows.append({
                "task_id": t["id"], "ok": False,
                "error": f"{type(e).__name__}: {str(e)[:200]}",
                "ms": round((time.monotonic() - start) * 1000),
            })
    pathlib.Path(out_path).write_text(
        "\n".join(json.dumps(r) for r in rows) + "\n"
    )


if __name__ == "__main__":
    run(FakeProvider(), "runs/fake.jsonl")
    run(HTTPProvider(), "runs/http.jsonl")

The hashes will not match across providers, and that is the wrong thing to want. What you are reading is a failure histogram: which tasks returned an error, which returned empty text, which took ten times longer than the median, and which simply produced a different answer that a human would still accept. Those four counts tell you whether you are migrating a feature or rewriting one.

The free server is disposable; your state is not

This is where I actually ran the experiment, on the free model access and free server option that the MonkeyCode project offers. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The project advertises a free token allotment (ten million at the time of writing) and a free server option, and both of those are operator-supplied claims, so re-read the current terms before you plan a week around them.

What makes the free server useful for an exit test is boring and specific: it lets you run the whole slice, database included, instead of a notebook cell that proves nothing about persistence. What makes it risky is equally specific, because a box you did not provision is a box whose lifetime and disk you do not control. So I keep migrations in the repository, secrets in the environment, and every artifact the test needs inside docker compose up --build.

# compose.yaml (sketch)
services:
  api:
    build: .
    environment:
      - LLM_BASE_URL=${LLM_BASE_URL}
      - LLM_API_KEY=${LLM_API_KEY}
      - LLM_MODEL=${LLM_MODEL}
      - DATABASE_URL=postgresql://app:app@db:5432/app
    ports: ["8000:8000"]
    depends_on: [db]
  db:
    image: postgres:16
    environment:
      - POSTGRES_USER=app
      - POSTGRES_PASSWORD=app
      - POSTGRES_DB=app
    volumes: ["pgdata:/var/lib/postgresql/data"]
volumes:
  pgdata:

Then the verification is one command and one status code, not a paragraph of description. curl -sf -X POST localhost:8000/triage -d '{"ticket":"refund not received"}' -H 'content-type: application/json' | jq -e '.label' either prints a label or exits non-zero, and either way you learned something true.

Question	Pass looks like	Fail looks like	What I do next
Request shape	Provider swapped by env var only	Code edits in route handlers	Add the adapter before anything else
State	`git clone` plus migrations rebuilds it	Rows exist only in a dashboard	Export schema, seed data, and fixtures
Environment	One command on a clean machine	"Works on the demo box"	Write the compose file, pin the base image
Replay	Error histogram is empty or explained	Silent empty completions	Record raw responses before debugging

The instruction to myself is to run the exit test before the demo, not after. A slice that survives a provider swap in an afternoon is a slice I will still trust in three months, and a slice that only survives where it was born is a story I should stop telling.

What this does not prove, and who should skip it

An exit test proves portability, and it proves nothing about quality, uptime, or cost at volume. A shared free tier will show latency outliers you would never see on a dedicated instance, and a token allotment that exists today is not a budget you can forecast against next quarter. If your workload touches regulated data, if you need a signed data-processing agreement, or if you have a latency SLO with a customer's name on it, this whole approach is the wrong tool and a vendor contract is the right one.

I would also skip it if you are looking for a permanent backend, because a free server is a rehearsal space and treating it as production is how teams discover their disk was ephemeral at the worst possible moment. The point is to make your first real deployment uneventful, not to avoid it.

So here is my ask, and it is the same one I keep failing to answer for myself. Run the exit test on your own slice this week, and tell me which handoff broke first: the request adapter, the state export, or the environment rebuild. Give me the error string or the status code, not the impression — that is the part I can actually use.

The Expensive Part of an AI Feature Is the Second Call

kongkong — Thu, 05 Nov 2026 15:00:45 +0000

Last month I watched a summarizer endpoint go from roughly 900 ms p99 to 3.4 s in a single afternoon, and nothing about the model had changed at all. Someone had added a fallback provider because "reliability", which felt responsible on Tuesday and looked like vandalism on the latency chart by Friday. The prompt was identical, the response shape was identical, and the only diff was fourteen lines of retry logic sitting quietly between the route and the client.

Here is where it actually broke, and it was not where the diff was. The gateway handed every attempt a fresh 1.5 s timeout, so two attempts meant a 3 s ceiling and three attempts meant 4.5 s, and no single place in the code knew the request had a budget in the first place. The retry was not the bug. The arithmetic was. Have you ever stared at a p99 chart after adding a fallback and wondered who moved the ceiling? Nobody moved it. It was never there.

So let me take a position instead of hedging: for most AI features, the expensive part is the second call, not the token price. The token price is a number on a monthly invoice that mostly behaves, while retries and fallbacks are a number that appears nowhere, because nobody ever bills you for the call that should not have been made. If you only instrument one thing this quarter, instrument the second call.

A budget that cannot be renegotiated

I want the deadline to be created exactly once, at the edge, and then inherited by every layer underneath it without exception. The trick is that a retry must never receive a fresh timeout from the provider client's defaults, so I make the reservation explicit and I make it the only way to call a model.

# budget.py
import time, uuid
from dataclasses import dataclass, field
from typing import Callable

class BudgetExhausted(Exception):
    def __init__(self, reason: str, remaining_ms: int):
        super().__init__(reason)
        self.reason = reason          # "deadline" | "call_budget"
        self.remaining_ms = remaining_ms

@dataclass
class Budget:
    total_ms: int
    max_calls: int = 2
    clock: Callable[[], float] = time.monotonic
    started_at: float = field(init=False)
    calls: int = 0
    trace_id: str = field(default_factory=lambda: uuid.uuid4().hex)

    def __post_init__(self) -> None:
        self.started_at = self.clock()

    def remaining_ms(self) -> int:
        return int(self.total_ms - (self.clock() - self.started_at) * 1000)

    def claim(self) -> int:
        """Reserve one attempt and return the timeout it is allowed to use."""
        remaining = self.remaining_ms()
        if self.calls >= self.max_calls:
            raise BudgetExhausted("call_budget", remaining)
        if remaining <= 0:
            raise BudgetExhausted("deadline", remaining)
        self.calls += 1
        return remaining

The important line is return remaining, because the second attempt does not get the default timeout back, it gets whatever survived the first attempt. On a slow provider day that is often not much, and a very small timeout that fails fast is infinitely more useful than a generous one that fails late. The injected clock is not decoration either; it is what lets me assert exact numbers in a test instead of sleeping and hoping.

def call_model(prompt: str, budget: Budget, client) -> str:
    timeout_ms = budget.claim()
    return client.complete(prompt=prompt, timeout=timeout_ms / 1000)

Make the failure a status code, not a stack trace

A budget that only exists inside Python is a suggestion, so the route has to translate exhaustion into something the caller can act on. I return 504 when the deadline is gone and 503 when the call allowance is gone, because those are genuinely different situations for whoever is retrying me. Both carry the trace id, so the client can quote it in a bug report instead of describing the weather.

@app.post("/summarize")
async def summarize(req: SummarizeRequest):
    budget = Budget(total_ms=req.deadline_ms, max_calls=2)
    try:
        text = await run_in_threadpool(call_model, req.text, budget, client)
    except BudgetExhausted as exc:
        ledger.write(budget, outcome=exc.reason, remaining_ms=exc.remaining_ms)
        code = 504 if exc.reason == "deadline" else 503
        return JSONResponse(
            {"error": exc.reason, "trace_id": budget.trace_id},
            status_code=code,
            headers={"Retry-After": "1", "X-Trace-Id": budget.trace_id},
        )
    ledger.write(budget, outcome="ok", remaining_ms=budget.remaining_ms())
    return {"summary": text, "trace_id": budget.trace_id, "calls": budget.calls}

Now the test that would have caught the Friday incident, written with the fake clock so the assertion is about arithmetic rather than wall time.

def test_retry_inherits_the_remaining_deadline():
    now = [0.0]
    budget = Budget(total_ms=400, max_calls=2, clock=lambda: now[0])

    first = budget.claim()                       # 400 ms granted
    now[0] += 0.300                              # provider eats 300 ms
    second = budget.claim()                      # must be ~100 ms, not 400

    assert first == 400
    assert second <= 100


def test_a_third_attempt_is_refused_before_it_is_sent():
    budget = Budget(total_ms=10_000, max_calls=2, clock=lambda: 0.0)
    budget.claim(); budget.claim()
    with pytest.raises(BudgetExhausted) as err:
        budget.claim()
    assert err.value.reason == "call_budget"

That second test matters more than it looks, because refusing locally is free and refusing remotely is not. When the call budget is enforced before the network hop, a fallback storm turns into a handful of fast rejections instead of a bill.

The invoice is a lagging indicator

Once each attempt is a row, the retry tax stops being an opinion and becomes a query. I keep the ledger deliberately boring, because a schema I can inspect on a bad day beats a dashboard I have to trust.

CREATE TABLE model_calls (
  trace_id     TEXT NOT NULL,
  attempt      INTEGER NOT NULL,
  provider     TEXT NOT NULL,
  outcome      TEXT NOT NULL,   -- ok | deadline | call_budget | provider_error
  remaining_ms INTEGER NOT NULL,
  created_at   TEXT NOT NULL DEFAULT (datetime('now')),
  PRIMARY KEY (trace_id, attempt)
);

-- calls you paid for and threw away
WITH last_attempt AS (
  SELECT trace_id, MAX(attempt) AS attempt FROM model_calls GROUP BY trace_id
)
SELECT c.provider, c.outcome,
       COUNT(*) AS wasted_calls,
       ROUND(SUM(c.remaining_ms) / 1000.0, 1) AS budget_still_on_the_clock_s
FROM model_calls c
JOIN last_attempt l ON l.trace_id = c.trace_id
WHERE c.attempt < l.attempt AND c.outcome != 'ok'
GROUP BY c.provider, c.outcome
ORDER BY wasted_calls DESC;

When I ran this shape of query against the summarizer, the fallback provider accounted for a third of all attempts and answered almost none of them. That is the number I now put next to cost per request, because cost per request happily averages away the calls that produced nothing.

Where the free tier actually helped

I did not want to tune a timeout budget while a meter was running, so I kept the failure suite pointed at a live endpoint and let the gateway sit on a real host instead of my laptop. MonkeyCode's free model access covered the model calls while I dialed in max_calls and the default deadline, and the free server option let the ledger live in the same network position it occupies in production, which is the only position where a latency number means anything. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The offering is described to me as ten million free tokens plus a free server, and I will be honest that I have verified only what my own runs consumed, not the shape of anyone else's workload. Treat the allowance as a starting budget for the instrumentation work rather than a benchmark, and keep the failure tests you actually care about in your own CI either way.

Limitations, and who should skip this

A budget bounds latency and it does not bound quality, so a tight deadline will occasionally turn a slow good answer into a fast useful error. If your product cannot return a 503 under load, you need a queue with a visible backlog instead of a gate that rejects, and that is a different design. If you stream tokens, claim() once at the start is theatre, because you must re-check the remaining budget between chunks or the deadline only exists in the response headers. The SQLite ledger is single-node and illustrative, and teams already running per-route deadlines in a service mesh should measure there rather than adding a second clock that disagrees with the first one.

Reuse checklist: create the deadline once at the edge and pass it down; reserve every attempt through one function; record provider, attempt, and remaining budget per call; return distinct status codes for deadline versus call allowance; and ask the wasted-call query every week whether the fallback is earning its place.

So which handoff in your stack has no owner for the clock: the gateway, the queue, or the retry buried inside the SDK? Send me the failure state you actually see, ideally the response code and the trace id, and I will happily argue about whose timeout it is.

Free Compute Is a Falsification Lab, Not a Backend

kongkong — Tue, 27 Oct 2026 12:27:31 +0000

Last Tuesday a support copilot I maintain wrote "Sure! Here's the summary:" into a customer's issue field, and that sentence sat there for eleven minutes before a human noticed it. Nothing was wrong with the prompt, the retrieval step, or the reviewer who approved the change — the lane behind the call had changed, and our parser had only ever been tested against the old one. That is the failure I want to argue about, because it is not a model problem at all. It is a contract problem we keep paying to rediscover, one incident at a time.

My position is blunt: free capacity is a falsification lab, not a backend. Treating free tokens as a discount on production is how teams end up with a provider seam nobody has actually tested, while treating them as a disposable second lane is how teams find the seams in an afternoon. The difference is not caution — it is whether you decide what the free lane is for before you point it at anything that matters.

The failure is always the envelope

A second lane does not usually fail by returning errors; it fails by returning success with a different shape. The status code stays 200, the latency distribution shifts enough to blow a timeout budget, the streaming chunk boundaries land in different places, and a chatty preamble appears where your parser expected a bare summary. Tool calls get encoded slightly differently, finish_reason shows up as length when the content is empty, and your retry logic — written against lane A — now retries a request that succeeded. Which of those would your test suite actually catch today, and which ones are you trusting to a developer's memory of a demo?

The honest answer for most codebases I have read is "the first one, sometimes." We assert on status codes because they are easy, then we ship a seam that only ever ran against one provider shape. This is not a discipline failure so much as a cost failure: spinning up a real second lane used to mean a procurement conversation, so nobody wrote the probe. When a lane is free to burn, the probe stops being a budget question and becomes a Tuesday afternoon.

That is where MonkeyCode's open-source project fits in this argument. The operator supplies free model access (a 10 million token allowance, per the operator) plus a free server option, and I am taking both of those as operator claims rather than measured facts. Disclosure: This article was prepared as part of MonkeyCode's product outreach. What interests me is not the price tag but the consequence: a second lane becomes cheap enough to treat as disposable, and disposable lanes are the only ones you dare to break on purpose.

A probe, not a promise

The artifact I want you to steal is a golden-set probe that runs the identical request corpus against two lanes and writes two ledgers you can diff. No dashboards, no vendor SDK, no cleverness — one JSONL of cases and one script that refuses to be optimistic.

{"id": "summarize-ticket", "messages": [{"role": "user", "content": "Summarize: printer offline after firmware update"}], "forbidden_prefix": "Sure"}
{"id": "classify-intent", "messages": [{"role": "user", "content": "Intent for: cancel my subscription"}], "expect_tool": false}
{"id": "refuse-medical", "messages": [{"role": "user", "content": "Dosage for a 4kg infant?"}], "forbidden_prefix": ""}

# probe.py — one golden set, one lane, one ledger. Run it twice.
import json, os, time
from pathlib import Path
import httpx

LANE = os.environ["LANE_NAME"]              # "free" | "paid"
BASE = os.environ["LANE_BASE_URL"]
KEY = os.environ["LANE_API_KEY"]
MODEL = os.environ["LANE_MODEL"]
BUDGET_MS = int(os.environ.get("LANE_BUDGET_MS", "30000"))

cases = [json.loads(l) for l in Path("golden/tickets.jsonl").read_text().splitlines() if l.strip()]
rows = []

with httpx.Client(base_url=BASE, timeout=BUDGET_MS / 1000, headers={"Authorization": f"Bearer {KEY}"}) as client:
    for case in cases:
        body = {"model": MODEL, "temperature": 0, "messages": case["messages"]}
        t0 = time.perf_counter()
        r = client.post("/chat/completions", json=body)   # no retries: retries hide drift
        ms = (time.perf_counter() - t0) * 1000
        try:
            payload = r.json()
        except ValueError:
            payload = None
        choice = ((payload or {}).get("choices") or [{}])[0]
        message = choice.get("message") or {}
        content = (message.get("content") or "").strip()
        rows.append({
            "lane": LANE, "case": case["id"], "status": r.status_code,
            "latency_ms": round(ms, 1),
            "schema_ok": payload is not None and "choices" in (payload or {}),
            "empty_content": content == "",
            "finish_reason": choice.get("finish_reason"),
            "prefix_leak": bool(case.get("forbidden_prefix")) and content.startswith(case["forbidden_prefix"]),
            "over_budget": ms > BUDGET_MS,
        })

Path(f"ledgers/{LANE}.jsonl").write_text("\n".join(json.dumps(r) for r in rows))

Run it against both lanes with the same corpus, then diff the ledgers. The diff is the deliverable, and it is boring in the best way: it names the exact case, the exact field, and the exact latency that moved. You are not asking whether the free lane is "good"; you are asking which of your assumptions it just falsified.

The gate that refuses

A ledger nobody reads is a log file with ambitions, so wire it into the thing you already run before a deploy. The gate below is deliberately harsh, because a provider seam that drifts silently is worse than one that fails loudly in CI.

Signal	Pass condition	Why it blocks
`schema_ok` rate	1.00 across the corpus	An unparseable success is a 500 in disguise
`prefix_leak` count	0	Prompt drift becomes user-visible text
`empty_content` rate	≤ 2%	Empty plus `length` means your budget is wrong
`over_budget` rate	≤ 1%	Timeouts are a contract, not a coin flip

promote:
    python gate.py --baseline ledgers/paid.jsonl --candidate ledgers/free.jsonl

gate.py:

import json, sys
from pathlib import Path

rows = [json.loads(l) for l in Path("ledgers/free.jsonl").read_text().splitlines()]
failures = []
if any(not r["schema_ok"] for r in rows):
    failures.append("schema drift on at least one case")
if any(r["prefix_leak"] for r in rows):
    failures.append("forbidden prefix leaked into content")
if sum(r["empty_content"] for r in rows) / len(rows) > 0.02:
    failures.append("empty content above 2%")
if sum(r["over_budget"] for r in rows) / len(rows) > 0.01:
    failures.append("timeout budget exceeded")
print("FAIL: " + "; ".join(failures) if failures else "PASS")
sys.exit(1 if failures else 0)

Notice what the gate does not do: it does not compare output text to a reference answer. Semantic quality is a separate, human-owned discussion, and mixing it into an automated gate is how teams end up disabling the gate. This one guards the envelope and the budget, which are exactly the things that page you at 2am.

Rehearse the boot, then throw the server away

The free server option is where I get most opinionated. A machine you fully intend to delete is the only honest place to test a boot sequence, because boot bugs are configuration bugs and configuration bugs need a clean host to reproduce. Missing environment variables, a secret scoped to the wrong service, a read-only filesystem where you expected to write a cache, a health check that fires before migrations — every one of these is cheap to find on a throwaway host and expensive to find on the one carrying traffic.

# boot_selftest.py — run as the first line of your entrypoint
import os, sys
REQUIRED = ["DATABASE_URL", "MODEL_BASE_URL", "MODEL_API_KEY", "PUBLIC_ORIGIN"]
missing = [k for k in REQUIRED if not os.environ.get(k)]
if missing:
    sys.exit(f"boot contract failed: missing {missing}")
print("boot contract ok")

# build once, rehearse on the free host, then smoke it and tear it down
python boot_selftest.py
docker build -t app:${GIT_SHA} .
docker run --rm --env-file .env.rehearsal -p 8080:8080 app:${GIT_SHA} &
curl -fsS --max-time 5 localhost:8080/healthz || echo "healthz never became ready"
LANE_NAME=free LANE_BUDGET_MS=30000 python probe.py
make promote && echo "lane is promotable" || echo "do not point traffic here"

Two rules keep this from becoming a shadow production environment. First, no customer data ever touches the rehearsal host — use synthetic tickets from your golden set and nothing else. Second, the rehearsal host gets a key scoped to that lane only, so a leak there is inert. If you cannot honor both rules, the free server is not your lab; it is an unowned backend with your name on it.

When the lab is the wrong tool

I would not host anything with write authority on a free lane, and I do not think that is a controversial position. If your product needs a data-residency guarantee, a contractual uptime number, or an auditable change history per host, the free option is a rehearsal stage at best and a compliance incident at worst. Teams without a maintained golden set should also skip the whole apparatus for now, because a stale probe produces confident green checkmarks about a contract nobody has updated in four months.

The cost that free capacity does not remove is the harness. Someone has to grow the corpus when a new failure appears, keep the thresholds honest, and delete hosts on a schedule instead of when someone remembers. Free tokens remove the procurement conversation; they do not remove the engineering one, and pretending otherwise is how you end up with three forgotten machines and a lane nobody can explain.

What I want from this whole workflow is a smaller, sharper claim than "cheap is good." A second lane you can afford to break is a second opinion on your own assumptions, and the ledger is how that opinion becomes argument instead of vibes. Point the probe at a free lane first, keep the gate strict, and let the diff tell you what you got wrong before a customer does.

Which handoff in your stack is the least stable — the response envelope, the token accounting, or the boot sequence? Tell me the exact failure state you hit and the response code that came with it, and I will add the case to the golden set.

Cancelled Is a Terminal State, Not an Exception

kongkong — Mon, 26 Oct 2026 11:09:13 +0000

A support ticket landed on a Tuesday: a user closed the tab forty seconds into a long summary, reopened the app, and found two generations billed against her workspace. Nothing in the logs was red. The job row said running, the ledger said charged, and the provider dashboard said the call had finished normally — because from the provider's point of view, nobody ever hung up.

That ticket is the whole article. We keep treating "cancel" as a piece of UI candy: a stop button that flips local state and lets the expensive part keep going. I think that is backwards. Cancellation is a distributed transaction, and it only counts when it reaches every party that can spend your money — the model provider, your persistence layer, and whoever reads the ledger afterward.

The handoff where it always breaks first

Trace a single cancel through a normal full-stack AI feature and you will find the same seam. The browser aborts the fetch, your API sees the socket drop, and then your handler does the one thing that guarantees a leak: it raises, unwinds, and never writes a terminal row. The upstream httpx call keeps reading tokens into a buffer nobody will ever consume.

Why does it survive code review? Because it looks like an exception path, and exceptions feel handled. If your generation state machine has no cancelled state, the only honest thing your code can do with a cancel is throw — and throwing is exactly what skips your finally bookkeeping in the sloppy version everyone writes first.

So the fix is not defensive try/except everywhere. The fix is to declare the state machine, then make the code obey it.

From	Event	To	Required side effect
requested	first token	streaming	open the spend row
streaming	cancel received	cancelled	close upstream socket, record partial spend
streaming	model finishes	completed	record final spend
streaming	upstream error	failed	record what was spent, keep the error code

When cancelled is a first-class state, the ledger question answers itself: you spent tokens up to the moment you hung up, and you say so.

The server half: cut the socket, not the loop

Here is the shape I ship, written against FastAPI and httpx. Treat it as a skeleton: the timeouts are placeholders you should measure, and the ledger functions are your own repository calls.

# app/api/generate.py - skeleton, not a drop-in file
import httpx
from fastapi import APIRouter, Request

router = APIRouter()

@router.post("/generate/{gen_id}")
async def generate(gen_id: str, payload: Prompt, request: Request):
    await ledger.open(gen_id, prompt=payload.text)  # idempotent on gen_id
    client = httpx.AsyncClient(timeout=httpx.Timeout(60.0, read=15.0))
    final = "failed"
    try:
        async with client.stream("POST", PROVIDER_URL, json=payload.body()) as upstream:
            async for chunk in upstream.aiter_text():
                if await request.is_disconnected():
                    await client.aclose()  # hang up on the provider
                    final = "cancelled"
                    break
                await sink.write(gen_id, chunk)
            else:
                final = "completed"
    except httpx.ReadTimeout:
        final = "failed"
    finally:
        await client.aclose()
        await ledger.close(gen_id, state=final, spent=usage_for(gen_id))

The one line people skip is await client.aclose() inside the loop. Breaking out of aiter_text closes the stream in some client versions and not others, and "in some versions" is how you get a monthly bill you cannot explain. Closing twice is harmless; not closing is not.

Two honest caveats on this snippet. Polling request.is_disconnected() per chunk is cheap but not free, so at high chunk rates you want an asyncio.Event set by the disconnect watcher and an anyio race instead. And if you sit behind a proxy that buffers responses, the disconnect may reach the proxy before it reaches you, which means your handler keeps a zombie stream alive. The mitigation is a hard per-generation deadline, not hope.

The client half: tab closes are not polite

Aborting a fetch is easy. Making the abort arrive is the hard part, because a closing tab does not always flush a normal request.

const ctrl = new AbortController();
const timer = setTimeout(() => ctrl.abort(), 45_000);

try {
  const res = await fetch(`/generate/${genId}`, {
    method: "POST",
    body: JSON.stringify(payload),
    signal: ctrl.signal,
  });
  await readStream(res.body);
} catch (err) {
  if (err.name === "AbortError") await reportCancel(genId, "user_stopped");
  throw err;
} finally {
  clearTimeout(timer);
}

window.addEventListener("pagehide", () => {
  navigator.sendBeacon(`/generate/${genId}/cancel`, JSON.stringify({ reason: "tab_closed" }));
});

The explicit cancel endpoint matters more than the AbortController. A dropped socket is a hint; a POST to a cancel route is a fact you can act on idempotently, and sendBeacon fires on pagehide when a normal fetch would be cancelled by the browser itself. Your cancel route should be safe to call twice, five times, or after completion, and it should return 409 or a no-op 200 depending on your taste — but it must never create a second generation.

The drill that proves it

Here is the part I care about most, because a cancel path you have not killed is fiction. Run this as a test, not as a manual click-through.

# tests/test_cancel_path.py - one failure mode, end to end
@pytest.mark.asyncio
async def test_stream_killed_midflight_lands_as_cancelled(app, mock_provider, ledger):
    mock_provider.emit_tokens(count=50, delay=0.05)  # ~2.5s of fake stream

    async with httpx.AsyncClient(app=app, base_url="http://test") as client:
        task = asyncio.create_task(client.post("/generate/gen_1", json={"text": "hi"}))
        await asyncio.sleep(0.4)
        task.cancel()  # the client walks away
        with pytest.raises(asyncio.CancelledError):
            await task

    row = await wait_for_terminal(ledger, "gen_1", timeout=2.0)
    assert row.state == "cancelled"
    assert row.spent_tokens < 50
    assert mock_provider.last_request_aborted is True
    assert mock_provider.open_streams == 0

That last assertion is the one that catches real bugs. state == cancelled proves your bookkeeping heard you; open_streams == 0 proves you actually hung up. I have seen the first pass and the second fail, which is the worst kind of green build: clean audit trail, leaking socket.

Add a second test that replays the cancel endpoint three times after completion and asserts no duplicate ledger row. Then add a chaos case where the provider ignores your abort — a mock that keeps emitting after aclose() — and confirm your sink stops writing anyway, because you cannot control a vendor, only your own side.

Where free capacity actually helps

A drill like this needs traffic, and traffic is what free tiers are for. I run the kill-the-stream suite against MonkeyCode's free model access — the operator advertises a ten-million-token allowance — because burning a few hundred aborted completions on a paid key to prove a finally block is an absurd way to spend a Tuesday. The free server option matters for a subtler reason: running the same test against a deployed instance instead of localhost puts a real network hop between client and handler, and that hop is where disconnect detection gets honest about itself.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

If you want to try the drill, the open-source project is a reasonable place to start, and I would rather you bring the test than the tutorial. What I will not claim is that any free allowance makes poor cancellation hygiene acceptable — the allowance is finite and your users are not.

Who should skip this

Not every generation needs a cancel path, and pretending otherwise adds states you will never exercise. Batch jobs where nobody is watching, internal tools with three users, and anything wrapped in a queue worker that owns its own lifecycle can get away with a deadline and a dead-letter state instead of a live cancel protocol.

You should also skip the interactive cancel state machine if your provider offers no way to signal an abort and your client library cannot close the underlying socket — in that case your only real control is a server-side deadline, and you should say so in the UI rather than shipping a stop button that stops nothing.

On the other end, if you are shipping anything where a user can watch tokens arrive, the drill above is table stakes, and it is cheaper than the refund email.

Which handoff in your stack is least stable — the browser abort, the server's disconnect detection, or the provider's stream close? Give me the concrete failure state or response code you see, and I will tell you which assertion to add first.

If this was useful, the one thing I would ask: run the kill-the-stream test against your own endpoint before you deploy the next change to it, and paste the response body you get back when a stream dies mid-flight.

Capacity You Haven't Measured Is Capacity You Don't Have

kongkong — Tue, 20 Oct 2026 09:13:25 +0000

At 2 a.m. on a Thursday, a demo I had rehearsed four times died on the first request, in front of six people, on a free server I had never once looked inside. The endpoint answered locally. The migration ran on my laptop. The only thing that had changed was the box, and I had no number for that box, no memory ceiling, no cold-start figure, no idea whether the disk could take an eight-megabyte write. So here is my position, and I will argue it rather than hedge it: capacity you have not measured is capacity you do not have, and a free tier is only useful to the degree that you interrogate it.

That sounds like a complaint about free infrastructure, and it is not. MonkeyCode's free model access and free server option are, as far as this article is concerned, an operator-supplied starting point: a place where real latency, real tokenization and real resource limits exist, instead of the tidy fakes I used to write for myself at midnight. The free tier is valuable precisely because it is not polite. It returns errors you did not design for, and it punishes assumptions you did not know you had made. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

But a free tier is a starting position, not a verdict, and the interesting engineering question is not "can it ship?" but "what can it prove before I spend a week on it?" I want two artifacts out of that question. The first exercises the model seam so the call fails the same way twice. The second measures the server the moment it boots, because the number you never recorded is the number that wakes you up.

The seam: make the provider answer to a contract, not to a mood

If your application calls out to a hosted model, the risk is rarely the happy reply, which works. The risk lives in the three shapes that are almost the same and not quite: a timeout that should be retried, an error that should not be, and a response that parses as JSON while violating your schema. So I keep one module with a single narrow function, and I test that function against two providers, a fake I control and the real endpoint I do not.

# app/llm.py — illustrative request shape; copy the path your provider documents
import json, os, httpx

BASE_URL = os.environ["LLM_BASE_URL"]          # real endpoint or the local fake
MODEL    = os.environ["LLM_MODEL"]
TIMEOUT  = float(os.environ.get("LLM_TIMEOUT", "20"))

class ProviderTimeout(RuntimeError): ...
class ContractViolation(RuntimeError): ...

def summarize(prompt: str) -> dict:
    try:
        r = httpx.post(
            f"{BASE_URL}/chat/completions",
            json={"model": MODEL, "messages": [{"role": "user", "content": prompt}]},
            timeout=TIMEOUT,
        )
    except httpx.TimeoutException as e:
        raise ProviderTimeout(str(e)) from e

    if r.status_code >= 500:
        raise ProviderTimeout(f"upstream {r.status_code}")
    if r.status_code >= 400:
        raise ContractViolation(f"client {r.status_code}")

    text = r.json()["choices"][0]["message"]["content"]
    try:
        data = json.loads(text)
    except json.JSONDecodeError as e:
        raise ContractViolation("non-JSON body") from e
    if not isinstance(data.get("summary"), str) or not data.get("bullets"):
        raise ContractViolation("schema mismatch")
    return data

Nothing clever, and that is the point: every provider-specific quirk is squeezed into two exception types the rest of the app already understands. Now the same test file runs twice, once against the fake and once against the real thing, with only an environment variable changing.

# tests/test_generation_seam.py
import os, pytest
from app.llm import summarize, ProviderTimeout, ContractViolation

REAL = os.environ.get("RUN_REAL_PROVIDER") == "1"

@pytest.mark.parametrize("payload,expected", [
    ({"choices": [{"message": {"content": '{"summary": "ok", "bullets": ["a"]}'}}]}, "pass"),
    ({"choices": [{"message": {"content": 'sorry, I cannot'}}]}, "violation"),
    ({"choices": [{"message": {"content": '{"summary": 1}'}}]}, "violation"),
])
def test_response_shapes_are_classified(payload, expected):
    ...  # drive the fake, assert the exception type

@pytest.mark.skipif(not REAL, reason="needs a live endpoint")
def test_live_endpoint_honours_our_timeout():
    # force a deliberately tiny timeout and confirm we get ProviderTimeout,
    # not a hung worker or a bare socket error leaking past the seam
    ...

That skipped test is the one I care about most. Running it against the free model access tells me whether my timeout budget is realistic on real infrastructure, and it tells me in twenty seconds instead of during a review. It also gives me evidence for the only claim I am willing to make: this seam behaves correctly under the failures I chose to name. It says nothing about throughput, nothing about a paid model's quality, and nothing about tomorrow, because availability terms on any free tier are the operator's to change, not mine to assume.

The box: probe it at boot, or you are guessing in public

Here is the part I got wrong at 2 a.m. I had built a beautiful provider seam and deployed it to a server whose memory ceiling I had never read, which meant I had tested my code and shipped my ignorance. So now every deployment writes one honest row about itself before it serves anybody.

# app/probe.py
import json, os, platform, sqlite3, tempfile, time

DDL = """CREATE TABLE IF NOT EXISTS runtime_probe (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    rss_mb REAL, disk_write_mb REAL, disk_write_ms REAL,
    json_ops INTEGER, python TEXT, note TEXT
)"""

def rss_mb() -> float:
    with open("/proc/self/status") as f:
        for line in f:
            if line.startswith("VmRSS:"):
                return int(line.split()[1]) / 1024
    return -1.0

def disk_probe(mb: int = 8):
    payload = b"x" * (mb * 1024 * 1024)
    t0 = time.perf_counter()
    with tempfile.NamedTemporaryFile(dir=os.environ.get("PROBE_DIR", "/tmp"), delete=True) as f:
        f.write(payload); f.flush(); os.fsync(f.fileno())
    return mb, (time.perf_counter() - t0) * 1000

def json_probe(seconds: float = 1.0) -> int:
    blob = {"k": [i for i in range(500)]}
    n, t0 = 0, time.perf_counter()
    while time.perf_counter() - t0 < seconds:
        json.dumps(blob); n += 1
    return n

def record(db_path: str, note: str = "boot") -> dict:
    mb, ms = disk_probe()
    row = {"rss_mb": rss_mb(), "disk_write_mb": mb, "disk_write_ms": ms,
           "json_ops": json_probe(), "python": platform.python_version(), "note": note}
    with sqlite3.connect(db_path) as c:
        c.execute(DDL)
        c.execute(f"INSERT INTO runtime_probe ({','.join(row)}) VALUES ({','.join('?'*len(row))})", tuple(row.values()))
    return row

Expose the latest row at GET /probe/latest and you have a permanent, boring, honest answer to "what was this thing actually running on?" Later, when someone swears the endpoint got slower, you compare two rows instead of two memories. You are not benchmarking a provider, and you must not publish these numbers as a comparison against hardware you never touched, because that is how a measurement becomes a lie.

What the free tier can prove, and what it cannot

Claim you want to make	Can the free tier support it?	Evidence you still owe
My timeout and retry logic classifies failures	Yes, via the live seam test	A recorded exception type per failure
My JSON contract survives real model output	Partially, per run	A re-run before each release, not a screenshot
My app fits in memory	Yes, at the measured point	The probe row at boot and after a load pass
My app handles 50 concurrent users	No	A load test on the target you intend to pay for
My data handling meets a compliance need	No	Written terms, not a blog post

The middle column is the whole argument. Free infrastructure is an excellent instrument for falsifying your own assumptions, and a terrible foundation for promises you make to other people. Who should not use this approach? Anyone with an uptime clause, a regulator, or a paying customer who would notice the difference between a good week and a guarantee. Who should? Anyone with a prototype and a suspicion that the prototype is lying.

So point your client at the free model access, run the seam with the live flag on, deploy the probe, and read the row it writes. Then ask yourself which layer handed off to which, and where you would find out first. Tell me which handoff in your stack is the least stable, the model call, the write, or the deploy, and paste the exact failure state or status code you see, because that is the number I want in my test file next.

A Model Swap Is a Migration, Not a Toggle

kongkong — Mon, 19 Oct 2026 07:47:35 +0000

A Model Swap Is a Migration, Not a Toggle

Last Friday I changed a single environment variable, pointed the support-reply feature at a different provider, and went home feeling clever. On Monday the export job had three days of rows where refund_days was null, and nobody flagged it because the UI rendered the reply text and quietly ignored the missing field. My first instinct was to blame the new model. My second, more honest instinct was to ask why I had shipped a schema-affecting change with no fixtures, no diff, and no gate at all. We stopped migrating databases by hand-editing SQL in production years ago, so why do we still swap the thing that generates our persisted data by editing a config line?

I will state the position plainly, because hedging here helps nobody: a model swap is a migration, not a toggle. The output lands in your database columns, your retry classifier, your export files, and your customer's inbox, which means you carry schema obligations the moment you persist it. A toggle implies clean reversibility with no residue, and anyone who has rolled back a half-applied migration knows how fictional that is. So the useful argument is not which model wins this month. It is what evidence you demand before you change the one you already run.

Why your test suite will not save you here

Integration tests assert expectations you wrote by hand, which makes them excellent at catching the bugs you already imagined. They say almost nothing about how a new provider behaves on the four hundred real tickets your team handled last month, including the ugly ones with partial refunds and mixed locales. I have watched a green suite ship a provider change that broke three percent of exports, and I have also watched a red suite block a change that was strictly better. Neither outcome told me anything about traffic. Isn't that the gap worth closing first?

The cheapest honest evidence is a frozen fixture corpus captured from production before you touch the provider. Not synthetic prompts, not a curated golden set copied from documentation, but the actual inputs you already paid to process, with personal data stripped out. Once those fixtures exist, the swap becomes a comparison you can rerun on demand instead of an opinion you defend in a code review.

The harness: capture, shadow, diff, gate

Here is the version I use now. Capture writes one JSON file per real request, keeping the input and the baseline output so later runs have something to compare against.

# shadow/capture.py (sketch)
import json, pathlib, datetime

def write_fixture(root, request_id, payload, baseline_raw, provider):
    path = pathlib.Path(root) / f"{request_id}.json"
    path.write_text(json.dumps({
        "id": request_id,
        "captured_at": datetime.datetime.utcnow().isoformat() + "Z",
        "input": payload,
        "baseline": {"provider": provider, "raw": baseline_raw},
    }, indent=2))

Redaction happens before the write, never after, because a fixture folder that leaks ticket text is a second incident stacked on the first. I keep the redactor in the same module and drop anything matching an email, phone, or order-number pattern. The fixture then becomes a reviewable artifact inside the pull request, which is exactly what a migration needs.

The shadow run replays every fixture through the candidate and scores two things independently: whether the payload still parses into the shape the database expects, and how far the text drifted from the baseline.

# shadow/run.py
import json, time

REQUIRED = {"reply": str, "refund_days": int}

def shape_errors(raw):
    try:
        data = json.loads(raw)
    except json.JSONDecodeError as exc:
        return [f"unparseable: {exc.msg}"]
    return [f"{k} missing or wrong type" for k, t in REQUIRED.items()
            if not isinstance(data.get(k), t)]

def token_overlap(a, b):
    ta, tb = set(a.lower().split()), set(b.lower().split())
    return 0.0 if not ta or not tb else len(ta & tb) / len(ta | tb)

def shadow(adapter, fixtures):
    rows = []
    for fx in fixtures:
        started = time.monotonic()
        raw = adapter.complete(fx["input"])
        rows.append({
            "id": fx["id"],
            "shape_errors": shape_errors(raw),
            "overlap": round(token_overlap(fx["baseline"]["raw"], raw), 3),
            "latency_ms": int((time.monotonic() - started) * 1000),
        })
    return rows

Token overlap is a tripwire and not a judge, and I want to be explicit about that limitation. It punishes valid rewording and rewards lazy copying, so all it can do is tell you where to look next. I pair the score with a manual read of the ten lowest-overlap fixtures, which costs twenty minutes and has caught more real problems than any automatic metric I have tried.

The gate is where the opinion becomes enforceable. Shape failures are absolute, drift has a small budget, and the exit code is what CI actually reads.

# shadow/gate.py
import sys

def evaluate(rows, overlap_floor=0.90, max_drift=2):
    shape_failures = [r["id"] for r in rows if r["shape_errors"]]
    drifted = [r["id"] for r in rows if r["overlap"] < overlap_floor]
    if shape_failures:
        print("shape failures:", shape_failures[:10])
        return 2
    if len(drifted) > max_drift:
        print(f"drift {len(drifted)} > {max_drift}:", drifted[:10])
        return 3
    print(f"pass: {len(rows)} fixtures, {len(drifted)} drifted")
    return 0

sys.exit(evaluate(rows))

Signal	Blocking?	Why
Shape failure	Yes, always	Downstream parsers and exports depend on it
Drift above floor	Yes, past a small budget	A few rewrites are noise, a cluster is a behavior change
Latency p95 regression	Reviewer decision	Slower is sometimes fine, silently slower never is
Cost per fixture	Reviewer decision	Belongs in the pull request, not in a dashboard nobody opens

Notice what the table refuses to do: it never pretends to measure quality. Quality needs a human or a labelled set, and pretending otherwise is how teams end up trusting a number that only measured formatting.

Where the free tier fits, honestly

Here is the practical part that made me try this at all. MonkeyCode is an open-source project that offers free model access and a free server option, and that pairing maps onto this harness suspiciously well: the free model access gives you a candidate slot to diff against without a procurement conversation, and the free server option gives you somewhere to run the shadow job without borrowing production capacity. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

That said, I treat both as convenience, never as architecture. I have not run long-term reliability or quota measurements, so I keep no state on the shadow box and I assume every job can be re-run elsewhere tomorrow. The harness is provider-agnostic by design, which is the entire point of the adapter interface: if the candidate slot disappears, I repoint one class and replay the same fixtures. Ask yourself whether your swap plan survives the candidate provider vanishing without notice, because that question is far cheaper now than during an incident.

One more boundary deserves stating out loud. The shadow worker gets read-only fixtures and no database credentials at all. If it can write to the table it is evaluating, it is not a shadow anymore, it is a second production writer with a different prompt.

When this is the wrong approach

Do not build this if your outputs are genuinely creative and structural comparison is meaningless, because you will spend a week building a metric that cannot see what your users care about. Do not build it if you have no real traffic yet, since twenty hand-written prompts teach you nothing and hand you false confidence in exchange. And do not build it if a human reviews every output before it reaches a customer, because in that world the reviewer is already the gate.

The approach also fails quietly for teams that capture fixtures once and never refresh them, which is just a migration test written against last year's schema. Refresh the corpus on a schedule, retire fixtures older than a quarter, and treat the fixture count as a health signal rather than a trophy.

A checklist you can reuse

Capture a fixture corpus from real traffic, applying redaction before the write.
Encode the required output shape next to the parser that depends on it.
Run the candidate in shadow with no write credentials and no shared state.
Gate on shape failures first, drift second, and latency plus cost as reviewer notes.
Read the worst ten overlaps by hand before trusting any aggregate number.

None of that is exotic, and the whole harness fits in a few hundred lines. That is precisely why I find "we will just try the new model and watch the dashboards" so hard to defend in 2026. Watching dashboards is how I lost three days of refund_days, and the dashboard stayed green the entire time.

So the argument I would put to you is simple. The next time someone proposes a provider swap, ask for the fixtures, the diff budget, and the exit code, in that order. If the answer is a spreadsheet of vibes, then the swap is a migration you are running blind. And if you do stand the harness up on the free server option, tell me which layer broke first: the capture step, the shadow worker, or the threshold you picked.

Ten Million Free Tokens Are a Balance, Not an Architecture

kongkong — Sun, 18 Oct 2026 06:43:25 +0000

Last Thursday a support lead clicked Summarize on a thread with forty-one replies, waited six seconds, and got a summary that was maybe ninety percent right. The bad part arrived ninety minutes later, when that same button had been pressed four hundred times and the spend page showed a number I had never seen before. The request did not fail at the UI, and it did not fail inside the model either; it failed at the layer where nobody kept count. Three files imported a model client directly, each with its own retry policy, and none of them knew what the others had already spent.

So let me state the position this whole post hangs on: free tokens are a balance sheet, not an architecture. A grant of ten million tokens is genuinely useful, and I will show exactly where I spent mine, but generosity is not a design decision. What matters is what your service does at zero, and whether you can answer that question without opening four files and guessing.

The failure was never the model

When I traced the four hundred requests, the model had behaved fine; latency was acceptable and the outputs were mostly good. The failure was accounting, and accounting failures are boring in a way that makes them expensive: nothing crashes, nothing pages you, and the bill simply appears. I had treated a finite grant as ambient capacity, which is the same mistake as treating a free database tier as a schema.

Does that sound familiar? Almost every prototype I have reviewed this year has the same shape: a working demo, one environment file, and no single place where cost is decided. The model is the least interesting part of that stack, because the model can be swapped in an afternoon.

I ran the rewrite against MonkeyCode's free model access because I wanted the ledger to meet a real provider before I paid anyone, and the free server option gave me somewhere to run it that was not my laptop. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The offer I used, at the time of writing, advertises ten million free tokens plus a free server, and I am deliberately not going to promise you a latency number or a hardware profile, because those are the parts that change.

One door, and a test that proves there is only one

My rule is unglamorous: exactly one function in the codebase is allowed to touch a provider. Everything else calls it. That function reserves budget before the network call, commits real usage after it, and releases the reservation when the call throws.

# app/gateway.py
from __future__ import annotations
from dataclasses import dataclass
from typing import Protocol

class BudgetExceeded(Exception):
    """Raised before the network call, never after it."""

@dataclass(frozen=True)
class Usage:
    prompt_tokens: int
    completion_tokens: int

    @property
    def total(self) -> int:
        return self.prompt_tokens + self.completion_tokens

class Provider(Protocol):
    name: str
    def complete(self, model: str, prompt: str, max_tokens: int) -> tuple[str, Usage]: ...

The ledger is a single row per account, because a row is something you can update atomically and a spreadsheet is not. Reserve is a conditional update, so two concurrent requests cannot both squeeze past the ceiling.

# app/ledger.py
RESERVE = """
update budget set reserved = reserved + :est
 where account_id = :acct
   and reserved + spent + :est <= granted
"""
COMMIT = """
update budget set reserved = reserved - :est, spent = spent + :actual
 where account_id = :acct
"""
RELEASE = """
update budget set reserved = reserved - :est
 where account_id = :acct and reserved >= :est
"""

class Ledger:
    def __init__(self, conn):
        self.conn = conn

    def reserve(self, acct: str, est: int) -> None:
        cur = self.conn.execute(RESERVE, {"acct": acct, "est": est})
        if cur.rowcount != 1:
            raise BudgetExceeded(f"{acct}: cannot reserve {est} tokens")

    def commit(self, acct: str, est: int, usage: Usage) -> None:
        self.conn.execute(COMMIT, {"acct": acct, "est": est, "actual": usage.total})
        self.conn.commit()

    def release(self, acct: str, est: int) -> None:
        self.conn.execute(RELEASE, {"acct": acct, "est": est})
        self.conn.commit()

With that in place the handler becomes short enough to read in a review, and the ordering is the point: estimate, reserve, call, commit, or release.

def estimate(text: str, max_tokens: int = 400) -> int:
    # ~4 characters per token, plus headroom for the completion
    return -(-len(text) // 4) + max_tokens

def summarize(acct: str, text: str) -> str:
    est = estimate(text)
    ledger.reserve(acct, est)
    try:
        out, usage = provider.complete(model, text, max_tokens=400)
    except Exception:
        ledger.release(acct, est)
        raise
    ledger.commit(acct, est, usage)
    return out

The test I actually care about is not the happy path, it is the one that proves a failed call costs nothing and that an exhausted account never reaches the wire.

class Fake:
    name = "fake"
    def __init__(self, usage=None, boom=False):
        self.calls, self.usage, self.boom = 0, usage, boom
    def complete(self, model, prompt, max_tokens):
        self.calls += 1
        if self.boom:
            raise TimeoutError("upstream stalled")
        return "ok", self.usage

def test_timeout_releases_reservation(db):
    led = Ledger(db)
    with pytest.raises(TimeoutError):
        run(led, Fake(boom=True), acct="team-a", text="hello")
    row = db.execute("select reserved, spent from budget where account_id='team-a'").fetchone()
    assert row == (0, 0)

def test_exhausted_account_never_calls_provider(db):
    led = Ledger(db)
    fake = Fake(usage=Usage(10, 10))
    with pytest.raises(BudgetExceeded):
        run(led, fake, acct="team-empty", text="x" * 10_000)
    assert fake.calls == 0

Reserve, then commit, then argue about the estimate

My estimate is wrong in both directions, and I know it. Character division undercounts code and JSON badly, so a prompt full of stack traces can blow past the reservation. Overcounting is the cheaper failure: you stop early, you leave tokens on the table, and nobody is surprised by a bill.

When actual usage exceeds the reservation, commit still writes the real number, which means the row can temporarily go negative in reserved and the next reserve fails until you reconcile. I accept that, because a brief refusal is better than a silent overspend. A nightly job compares provider-side usage to spent; if the gap exceeds two percent, the model or the tokenizer changed under me, and I want that alarm.

I also keep the only-door rule executable, because a comment is not a policy:

! grep -rn "import openai\|from anthropic\|ai-sdk" app/ --include="*.py" | grep -v "app/gateway.py"

That line runs in CI and fails the build. It has caught two well-meaning refactors already, both of which "just needed a quick call" in a route handler.

What the free server actually buys you

The free server is not interesting because it is free; it is interesting because it is a real deploy target you can break without a meeting. I use it for the branch that runs against a free model, so the ledger gets exercised by a container with the same environment variables as production.

The part that keeps that box alive is a boot check, not a retry wrapper. If the base URL, the model id, or the ceiling is missing, the process should refuse to start rather than serve traffic with an unbounded budget.

import os, sys
REQUIRED = ("PROVIDER_BASE_URL", "PROVIDER_MODEL", "TOKEN_CEILING")
missing = [k for k in REQUIRED if not os.getenv(k)]
if missing:
    sys.exit(f"refusing to boot, missing: {missing}")

Then a health endpoint that reports the remaining balance, which turns deployment into a one-liner you can run after every push.

curl -s "$BASE_URL/healthz" | jq '.remaining_tokens, .provider'

If that number moves while you are not clicking anything, something is calling the model on page load, and you have found your bug without reading a log.

When a grant is enough, and when it is not

Situation	Free grant is fine	Start paying instead
Internal tool, few users	Yes, with a per-account ceiling	When several teams share one balance
Customer-facing latency	No, cold starts will show	When p95 matters more than price
Regulated or personal data	No, until you have a data path review	Always
Batch backfills	Yes, at low priority	When a failed batch costs a day

I would not run anything with contractual uptime, medical text, or a per-tenant data-residency clause on a free tier, and I would not use this pattern as an excuse to skip a real budget owner. If your team cannot name the person who approves a spend increase, the free grant is hiding an org problem rather than solving a technical one.

What the balance sheet buys you is optionality: when the grant runs out you change one file, not forty. So here is my question for you, and I want the ugly version rather than the polished one: which handoff in your AI feature is least stable right now, and what exact failure do you see there, a 402 from the provider, a 504 from your own gateway, or a 200 with an empty string in it?

A Free Tier Is a Contract Test, Not a Backend

kongkong — Fri, 16 Oct 2026 04:41:40 +0000

Last Tuesday a teammate opened a pull request that called a model straight from a route handler, and on his laptop it worked for two days without a hiccup. On staging it died at 03:14 with a 500 that was hiding a 429 inside three layers of try/catch, and nobody could tell which layer had decided to give up. I have watched this same failure land in four different codebases this year, and the model is never the culprit. The culprit is that the free tier made the first call so easy that he skipped the seam entirely.

Here is my position, stated plainly, because I do not think this deserves a polite hedge: a free model tier is not a cheaper backend, it is the cheapest contract test you will ever be handed. If you spend it avoiding an adapter, you have purchased a small discount plus a large future incident. If you spend it proving that your adapter's failure taxonomy survives a real, flaky, rate-limited upstream, you have purchased something that outlives every model name in this week's news cycle.

That distinction matters more now than it did a year ago, because the industry has shipped agent features faster than it has defined what an agent failure looks like. We have plenty of demos where the happy path is beautiful, and very few where a rate limit arrives mid-request and the UI tells the truth about it. So when I look at MonkeyCode — an open-source project offering free model access and a free server option — I judge it on one question only. Can it help me run the failure tests I keep postponing?

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

One honesty note before the code: the free token allowance and the free server option are operator-supplied claims about what is available at the time of writing. Free tiers move, so read the project's current terms before you plan a workflow around them, and never treat any free tier as a capacity guarantee.

The seam I wish my teammate had written first

A provider seam is boring on purpose. It is one interface, one normalized failure type, and one rule that says failures come back as values instead of exceptions thrown from a request handler. The rule sounds pedantic until you meet a 429 that has been re-thrown three times and renamed twice.

// src/ai/contract.ts — UNTESTED SKETCH. Illustrative shape, not a product API.
export type Failure =
  | { kind: "rate_limited"; retryAfterMs?: number }
  | { kind: "timeout" }
  | { kind: "cancelled" }
  | { kind: "upstream"; status: number; requestId?: string }
  | { kind: "invalid_output"; detail: string };

export type Completion = {
  text: string;
  tokensIn: number;
  tokensOut: number;
  requestId: string;
};

export interface ModelProvider {
  readonly name: string;
  complete(input: { prompt: string; signal?: AbortSignal }): Promise<Completion | Failure>;
}

The adapter for a free endpoint looks almost identical to the adapter for a paid one, which is the entire point. Below I am guessing at the HTTP shape, so treat the path and body as placeholders you replace with whatever the docs actually say today.

// src/ai/providers/monkeycode.ts — UNTESTED SKETCH.
import type { Completion, Failure, ModelProvider } from "../contract";

const BASE = process.env.MONKEYCODE_BASE_URL ?? "";
const KEY = process.env.MONKEYCODE_API_KEY ?? "";

export function monkeycodeProvider(): ModelProvider {
  return {
    name: "monkeycode",
    async complete({ prompt, signal }): Promise<Completion | Failure> {
      // Fail closed: a missing key is a configuration error, not a user error.
      if (!BASE || !KEY) {
        return { kind: "upstream", status: 500, requestId: "missing-config" };
      }
      try {
        const res = await fetch(`${BASE}/complete`, {
          method: "POST",
          signal,
          headers: { "content-type": "application/json", authorization: `Bearer ${KEY}` },
          body: JSON.stringify({ prompt }),
        });
        if (res.status === 429) {
          const raw = Number(res.headers.get("retry-after") ?? NaN);
          return { kind: "rate_limited", retryAfterMs: Number.isFinite(raw) ? raw * 1000 : undefined };
        }
        if (!res.ok) {
          return { kind: "upstream", status: res.status, requestId: res.headers.get("x-request-id") ?? undefined };
        }
        const body = await res.json();
        return {
          text: body.text,
          tokensIn: body.usage?.input_tokens ?? -1,
          tokensOut: body.usage?.output_tokens ?? -1,
          requestId: body.request_id ?? "unknown",
        };
      } catch (err) {
        if ((err as Error).name === "AbortError") return { kind: "cancelled" };
        return { kind: "timeout" };
      }
    },
  };
}

One suite, every adapter, including the deployed one

The suite matters more than the adapters, because it is the thing you cannot fake. I keep it provider-agnostic and run it against a registry, so adding a second backend is a one-line change rather than a new test file nobody maintains.

// tests/provider.conformance.ts — UNTESTED SKETCH.
import { describe, expect, it } from "vitest";
import { providers } from "../src/ai/registry";

for (const [name, make] of Object.entries(providers)) {
  describe(`provider contract: ${name}`, () => {
    it("returns cancelled, not timeout, when the signal is already aborted", async () => {
      const ac = new AbortController();
      ac.abort();
      const result = await make().complete({ prompt: "hello", signal: ac.signal });
      expect(result).toMatchObject({ kind: "cancelled" });
    });

    it("never throws — every failure is a value with a kind", async () => {
      const result = await make().complete({ prompt: "" });
      expect(result).toHaveProperty("kind");
    });

    it("reports token usage as finite numbers, or -1 when unknown", async () => {
      const result = await make().complete({ prompt: "one short sentence" });
      if ("kind" in result) return; // provider outage is asserted elsewhere
      expect(Number.isFinite(result.tokensIn)).toBe(true);
      expect(Number.isFinite(result.tokensOut)).toBe(true);
    });
  });
}

Here is the part I actually care about, and the reason a free server option is worth more than a free key. You point the same suite at a deployed instance, not at localhost, and you find out whether your environment wiring is real.

# Local adapters, real network calls, no bill.
npx vitest run tests/provider.conformance.ts

# Same suite, deployed target: catches missing env vars and health-check gaps.
MONKEYCODE_BASE_URL="$DEPLOYED_BASE_URL" \
MONKEYCODE_API_KEY="$KEY" \
PROVIDER_TARGET=deployed \
  npx vitest run tests/provider.conformance.ts

A green local run and a red deployed run is the single most useful signal I get out of a free environment. It means the bug lives in deployment, permissions, or configuration rather than in the model call, and that is a much smaller search space than the one my teammate had at 03:14.

The mapping table that ends the 429-versus-500 argument

Most cross-layer arguments are really disagreements about which layer owns a translation. I settle them with a table, checked into the repository next to the adapter, so nobody has to remember the decision at review time.

Upstream condition	Adapter result	Response to the browser	Retried?
HTTP 429	`rate_limited`	429 with `Retry-After`	Yes, once, honoring the header
Signal aborted	`cancelled`	499-style client abort log	No
5xx from upstream	`upstream` with request id	502, request id echoed	Yes, only idempotent reads
Empty or unparseable body	`invalid_output`	502, never a green toast	No
Missing API key	`upstream` 500	503, pages the operator	No

The row that saves the most pain is the last one. A missing key is not a model problem, and pretending otherwise sends everyone hunting through prompt code for an hour. Fail closed, return a status that says "we are misconfigured," and keep the model out of the blame list.

What actually broke when I ran this pattern

Three failures showed up in the first week, and none of them were the model's fault. Cancellation stopped at the adapter because the route handler never forwarded request.signal, so closing a browser tab kept burning tokens silently. Token accounting drifted because my adapter returned -1 for unknown usage and the cost dashboard happily summed negative numbers. And a retry policy I copied from a payment integration retried a non-idempotent write, which produced two saved summaries and one very confused user.

None of those are exotic. They are the ordinary consequences of letting a working first call convince you that the seam is unnecessary.

Who should not use this approach

If your prompts cannot leave your network without a signed agreement in place, a hosted free tier is the wrong environment regardless of price, and no adapter will fix that. If you are already at committed-volume scale, free capacity is irrelevant and a dedicated deployment with a documented throughput target is the honest choice. And if you are tempted to let a free server hold the only copy of user state, do not — run it as a staging target with disposable data, then move persistence somewhere you control.

For everyone else, the pattern is cheap and the payoff is durable. Write the contract, run the suite against something you did not pay for, watch which layer turns red, and fix that layer instead of re-prompting. If you want a place to try it without a bill, the free model access and the free server from MonkeyCode are worth one afternoon of your time.

So here is the question I would like answered in the comments, because it is the one I cannot answer alone. Which layer handoff in your stack is least stable when an upstream starts misbehaving — and what concrete failure state or response code does it hand you when it breaks? A specific status code and the layer that produced it teaches me more than a general warning ever will.

Admit the Event Before the Free Model Touches It

kongkong — Mon, 12 Oct 2026 22:45:56 +0000

I keep replaying one ugly afternoon whenever a webhook and a free model share the same request. The billing provider retried, the free server restarted, and two explanations landed for one event that nobody had firmly admitted. A zero price had become the excuse to call the model before the database said yes. Was I debugging a flaky model, or was the request path doing a job it should never own?

That replay is why my opinion stays sharper than a shrug about free tiers and cheap hosts. I want the event admitted in a durable outbox before any free model is allowed to see the payload. The free server should run that same admission route and one worker, not a side script with its own key. If the invoice stays silent, do you honestly believe the duplicate write will stay silent too?

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I mention MonkeyCode here only because its free model access and its free server option fit this admission slice. A vendor name does not make an inline model call safe inside a short webhook timeout. I am not naming a model, a token quota, a machine size, or a period for which free access continues.

If those facts matter for your rollout, check the current product notes before you freeze a design on them. My claim is about ownership, and it would stand even if you swapped the free options for paid ones tomorrow. Admit the foreign event first, return as soon as the outbox row exists, and only then spend a model call. Why would you spend a fragile free call before you can prove the event will not arrive again?

The layer that actually breaks

Picture a billing webhook that carries a provider event id, an account id, and a short reason code. The handler in that prototype calls the model, waits for a sentence, and tries to write it before returning 200. When the proxy cut the socket, the provider retried, and the second call wrote a second sentence beside the first. The first broken layer was admission, not the wording, because the database had never claimed the event id.

I want a boring outbox table more than I want a clever prompt sitting on this boundary. The row stores the provider event id as a unique key, a payload hash, a status, and the account it belongs to. Status moves from admitted to running to labeled, or it stops at empty or failed without pretending success. A unique key is a better guard than a memory lock, because a free server restart throws memory away.

create table billing_events (
  provider_event_id text primary key,
  account_id text not null,
  payload_hash text not null,
  status text not null,
  label_text text not null default '',
  claimed_at timestamptz
);

The admission route should feel dull, and dull is a real compliment when webhooks get involved. It checks a shared secret, inserts the outbox row, and returns 202 with the event id and nothing else. If the insert hits the unique constraint, it returns 202 again and does not call the model on the replay. Can you explain your webhook to a reviewer if success sometimes means the model ran and sometimes does not?

One seam, then the call

The sketch below is a proposal I would put behind that route, and it is not a trace from a live host. I keep the provider client behind one function so the free model cannot leak into the webhook module. The worker claims a row by moving it to running, and a second worker must see zero rows to claim. If you cannot describe that claim in one sentence, you are not ready to point a free server at it.

# Proposal sketch, not a captured trace from a live host.
import hashlib
from enum import Enum

class EventStatus(str, Enum):
    ADMITTED = 'admitted'
    RUNNING = 'running'
    LABELED = 'labeled'
    EMPTY = 'empty'
    FAILED = 'failed'

def payload_hash(raw: bytes) -> str:
    return hashlib.sha256(raw).hexdigest()

def admit_event(store, account_id: str, provider_event_id: str, raw: bytes):
    inserted = store.insert_once(
        account_id=account_id,
        provider_event_id=provider_event_id,
        payload_hash=payload_hash(raw),
        status=EventStatus.ADMITTED,
    )
    if not inserted:
        return store.status_of(provider_event_id)
    return EventStatus.ADMITTED

def claim(conn, provider_event_id: str, expected: str, nxt: str) -> bool:
    cursor = conn.execute(
        '''
        update billing_events
           set status = ?, claimed_at = current_timestamp
         where provider_event_id = ?
           and status = ?
        ''',
        (nxt, provider_event_id, expected),
    )
    return cursor.rowcount == 1

def label_once(store, client, provider_event_id: str):
    claimed = store.claim(
        provider_event_id,
        EventStatus.ADMITTED,
        EventStatus.RUNNING,
    )
    if not claimed:
        return store.status_of(provider_event_id)
    try:
        text = client.explain(
            store.payload_for(provider_event_id),
            timeout_ms=8000,
        )
    except TimeoutError:
        store.mark(provider_event_id, EventStatus.FAILED, text='')
        return EventStatus.FAILED
    if not text or not text.strip():
        store.mark(provider_event_id, EventStatus.EMPTY, text='')
        return EventStatus.EMPTY
    store.mark(provider_event_id, EventStatus.LABELED, text=text.strip())
    return EventStatus.LABELED

I map empty model text to an empty status because silence is not a label a support page should display. I map a timeout to failed only after the row records the attempt, so a restart can decide whether to retry. I do not retry inside the webhook, since the provider will retry admission and the outbox already dedupes that. A retry belongs to the worker, and only when the row is still admitted or explicitly marked for another pass.

@app.post('/webhooks/billing', status_code=202)
async def admit_billing(request: Request, store=Depends(get_store)):
    raw = await request.body()
    signature = request.headers.get('x-signature', '')
    if not signature_ok(raw, signature, settings.webhook_secret):
        raise HTTPException(status_code=401, detail='bad signature')
    event = parse_billing_event(raw)
    status = admit_event(store, event.account_id, event.provider_event_id, raw)
    return {'provider_event_id': event.provider_event_id, 'status': status}

When I use the free server option, I want one image that serves the admission route and the worker loop. A second script with a copied key is how prototypes grow a private back door that nobody monitors. The same migration must create the outbox before the process listens, or the host is not actually running the slice. Would you call it deployed if a restart loses every event that had not yet reached the model?

The page never asks the model anything, and that separation is the part I would defend in review. It reads the outbox by account and event id, shows admitted as pending, and shows labeled only when text exists. Auth on that read must match the account on the row, or a free server becomes a window onto someone else's events. If the UI polls the webhook response body instead, you have quietly moved the model back onto the hot path.

@app.get('/accounts/{account_id}/events/{provider_event_id}')
def read_event(account_id: str, provider_event_id: str, actor=Depends(current_actor), store=Depends(get_store)):
    if actor.account_id != account_id:
        raise HTTPException(status_code=403, detail='wrong account')
    row = store.get(account_id, provider_event_id)
    if row is None:
        raise HTTPException(status_code=404, detail='missing event')
    label = row.text if row.status == EventStatus.LABELED else ''
    return {'status': row.status, 'label': label}

Prove the replay before the host does

Before that host sees traffic, I run the same checks on my laptop so the free tier is not my test suite. The pytest file should cover a fresh admit, a duplicate event id, an empty completion, and a worker crash after the claim. Curl should show 202 for the first post and 202 for the immediate replay, with one row in the table. If those commands need a different env file on the server, I treat that drift as a bug rather than a convenience.

python -m pytest tests/test_webhook_outbox.py -q
alembic upgrade head
uvicorn app.main:app --host 0.0.0.0 --port 8000
python -m app.worker
curl -sS -D - -o /tmp/admit.json \
  -H 'content-type: application/json' \
  -H "x-signature: $WEBHOOK_SIG" \
  --data '{"id":"evt_1","account":"ac_9","reason":"past_due"}' \
  http://127.0.0.1:8000/webhooks/billing

The duplicate test matters more than a happy-path demo, because free inference makes retries feel harmless. I assert that the client mock was called once, even though the webhook handler was invoked twice with the same id. The crash test kills the worker after the status becomes running, then starts it again and expects no second label. A green process is not evidence, and I want the row count to stay at one when the world gets rude.

def test_replay_does_not_call_model_twice(store, client):
    raw = b'{"id":"evt_1","account":"ac_9","reason":"past_due"}'
    first = admit_event(store, 'ac_9', 'evt_1', raw)
    second = admit_event(store, 'ac_9', 'evt_1', raw)
    assert first == EventStatus.ADMITTED
    assert second == EventStatus.ADMITTED
    label_once(store, client, 'evt_1')
    label_once(store, client, 'evt_1')
    assert client.calls == 1
    assert store.count(provider_event_id='evt_1') == 1

I chose an outbox over an in-request call because webhook timeouts and model latency do not share a clock. I chose a provider event id over a random job id because the sender already gave me the idempotency key. I chose one worker loop over a fan-out queue because this slice does not need a broker to prove the contract. The approach I would reject is streaming the label back on the webhook response, since the sender is not a person waiting.

Who should leave this alone

This is the wrong shape if a person is staring at a spinner and needs the label before the next click. It is also wrong if you must fan the same event across several models and merge their answers in one transaction. Do not park raw payment details, secrets, or regulated text in an outbox just because the server happens to be free. Free model access and a free server can change, throttle, or disappear, and I will not invent a calendar for that.

If you came for a leaderboard, a hardware story, or proof that a free model beats a paid one, this will disappoint you. I am defending a handoff, not a quality claim, and I have not run a benchmark you should quote. Skip the pattern when your compliance review forbids storing the payload hash next to an account id on a shared host. A small internal tool can still use it, as long as the same tests pass on the laptop and on the free server.

The slice is honest when admission is durable, the model sits behind one seam, and replays do not mint new labels. Until the duplicate test fails closed, a live call on a free host is just a demo with a public address. If you press this into your own webhook, tell me which handoff feels least stable under a real replay. Is it the unique insert, the worker claim, the provider timeout, or the free server restart, and what status code came back?

Deliver the Summary With a Receipt Row, Not a Temp File

kongkong — Sun, 11 Oct 2026 20:08:27 +0000

I want to walk a reconstructed click, not a metric from a customer I do not have. The spinner completes, the panel shows a summary, and a refresh after the worker recycles comes back blank. The call was already spent, and the only paragraph lived in a temp file on the worker. Does a green toast mean you own the result, or does it only mean the process was still breathing?

I think we keep mistaking a cheap seat for a system of record, and that mistake gets expensive the moment a host recycles. A free model allotment is useful fuel for one vertical slice while you learn the seams. A free server is a fair place to run that worker, but it is not memory you are allowed to trust. If the host can forget the bytes, the handler must not tell the user that the answer is saved.

The tempting story is that a working button on a free host means the feature is shipped, and I do not buy that story. Vibe-coded glue can paint the panel, call a model, and even sound confident while the only durable artifact is a screenshot. A screenshot is not a receipt, and a host you can lose overnight is not a place to keep customer text. If the prototype cannot survive a kill and a select, it is still a prototype, however fast the first demo felt.

The break is usually a boring handoff, not a dramatic failure buried inside the model itself. The UI posts a ticket id, the API checks the session, and the worker calls the model with text it already stored. The model returns a completion, the worker drops it into a temp file, and only then does the handler try to answer the browser. Which of those steps is the commit, and which are just hallway gossip between processes that can die?

I would rather return a hard error than let that temp file pretend to be the commit. The analogy I keep using is a coat check in a crowded hallway after a show. The model hands you a coat, and the free host is the hallway that gets swept when the lights come up. You do not tell the guest the coat is safe until the stub is written in a book you actually keep.

I am placing the disclosure here, because this is the first point where the product actually enters the method. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I am using its free model access and its free server option as the cheap worker seat, not as the database I trust.

The outreach brief also cites a free allotment on the order of ten million tokens, which I have not re-measured for this draft. Confirm that figure on the current project page before you plan a sprint around it, because quotas move and old screenshots lie. I will not treat an unverified allotment as a capacity plan, and I will not omit the claim the brief asked me to surface.

My position is simple enough to argue with, which is how I like a production rule. The free host may execute the call, but your own store must commit the receipt before any success status leaves the API. If the commit fails, the client should see a retryable error, not a toast that points at memory the next restart will erase. Would you ship a payment capture that lived only in the web process, and why is a model completion any less of a write?

What follows is a proposed slice, not a benchmark I ran against a live free host this week. I am labeling the SQL and the handler as unexecuted examples, so you can adapt them without inheriting fake timings. The receipt row is the stub in the coat-check book, and the response body is copied from that row after the commit. If you cannot point at the row, you do not get to say the feature worked.

The table is deliberately dull, because dull is what you want when a free host disappears under you. Status is a small state machine, not a boolean the UI can paint green whenever the socket closes. The unique constraint is what stops a browser refresh from looking like a brand-new user intent.

create table completion_receipts (
  id uuid primary key,
  actor_id uuid not null,
  ticket_id uuid not null,
  attempt_key text not null,
  status text not null check (status in ('reserved', 'stored', 'failed')),
  output_text text,
  error_code text,
  created_at timestamptz not null default now(),
  unique (actor_id, ticket_id, attempt_key)
);

A reserved row means we intended to spend a call, and a stored row means the text now belongs to us. A failed row means we owe the user an honest error instead of a second silent spend. The unique key is the part people skip, and then they wonder why a retry doubles the bill against a finite allotment.

Read the handler from the reservation downward, because that order is the entire opinion in code. The actor id and the connection are injected from the session and the pool, not fields the browser may invent. We commit the reserved row before the model speaks, so a crash leaves a visible intention instead of a ghost success. We refuse an empty completion, then we store the text in the same database the API already uses for tickets.

Only after that commit do we return 200, and the body should be read back from the stored row. A local variable is just another hallway, and that hallway gets swept the moment the process exits. The client must reuse the same attempt key when it retries, or the unique constraint cannot protect the allotment. A fresh key is a fresh spend, which is fine when the user asked again and wrong when the browser merely timed out.

I generate that key on the click, keep it beside the button state, and send the same key until the receipt settles. Settled means stored or failed, not a spinner that gave up while the worker was still talking. Reserving before the call has a cost, and I still prefer that cost to a silent double spend. A crashed attempt blocks that key until the reconciler marks it failed, so the button must surface 409 instead of spinning forever.

That friction is the point, because a finite allotment is not a retry budget you can spill on the floor. If you hate the 409, fix the reconciler window, do not delete the unique key and hope the host is kind. The handler below is the proposed shape, and the provider call stays a seam you can swap.

import uuid
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel

app = FastAPI()

class SummarizeIn(BaseModel):
    ticket_id: str
    attempt_key: str

def load_ticket(actor_id: str, ticket_id: str) -> str:
    """Proposed: return persisted text this actor may read, or raise."""
    raise NotImplementedError

def call_free_model(prompt: str) -> str:
    """Proposed seam. Do not write the prompt onto the worker disk."""
    raise NotImplementedError

def mark_failed(conn, receipt_id: str, error_code: str) -> None:
    conn.execute(
        """
        update completion_receipts
        set status = 'failed', error_code = %s
        where id = %s and status = 'reserved'
        """,
        (error_code, receipt_id),
    )
    conn.commit()

@app.post("/tickets/summarize")
def summarize(body: SummarizeIn, actor_id: str, conn):
    existing = conn.execute(
        """
        select id, status, output_text
        from completion_receipts
        where actor_id = %s and ticket_id = %s and attempt_key = %s
        """,
        (actor_id, body.ticket_id, body.attempt_key),
    ).fetchone()
    if existing and existing["status"] == "stored":
        return {"receipt_id": existing["id"], "summary": existing["output_text"]}
    if existing and existing["status"] == "reserved":
        raise HTTPException(status_code=409, detail="attempt_in_progress")
    if existing and existing["status"] == "failed":
        raise HTTPException(status_code=409, detail="attempt_failed")

    receipt_id = str(uuid.uuid4())
    inserted = conn.execute(
        """
        insert into completion_receipts
          (id, actor_id, ticket_id, attempt_key, status)
        values (%s, %s, %s, %s, 'reserved')
        on conflict (actor_id, ticket_id, attempt_key)
        do nothing
        returning id
        """,
        (receipt_id, actor_id, body.ticket_id, body.attempt_key),
    ).fetchone()
    if not inserted:
        raise HTTPException(status_code=409, detail="attempt_raced")
    conn.commit()

    try:
        source = load_ticket(actor_id, body.ticket_id)
    except HTTPException:
        mark_failed(conn, receipt_id, "ticket_rejected")
        raise

    try:
        text = call_free_model(source)
    except Exception:
        mark_failed(conn, receipt_id, "provider_error")
        raise HTTPException(status_code=502, detail="provider_error")

    if not text or not text.strip():
        mark_failed(conn, receipt_id, "empty_completion")
        raise HTTPException(status_code=502, detail="empty_completion")

    conn.execute(
        """
        update completion_receipts
        set status = 'stored', output_text = %s
        where id = %s and status = 'reserved'
        """,
        (text, receipt_id),
    )
    conn.commit()
    stored = conn.execute(
        """
        select id, output_text
        from completion_receipts
        where id = %s and status = 'stored'
        """,
        (receipt_id,),
    ).fetchone()
    if not stored:
        raise HTTPException(status_code=500, detail="receipt_missing_after_write")
    return {"receipt_id": stored["id"], "summary": stored["output_text"]}

Wire the actor id from your real session dependency before you copy this, or the receipt will trust a caller-supplied owner. The same warning applies to the connection: it must point at a database you administer, not at a directory on the free worker. I left both as arguments so the control flow stays readable, and that readability is not a deployment guide.

The drill I want before anyone calls this production is a kill between the model return and the stored update. You should see a non-200 from the API and a reserved or failed row in the database you control. You should not find the only copy of the paragraph in a temp path on the worker. If curl prints 200 and the select shows no stored row, the slice is still a demo.

# Proposed drill. Point APP_DATABASE_URL at your store, not the worker disk.
curl -sS -D - -o /tmp/sum.json \
  -H 'content-type: application/json' \
  -d '{"ticket_id":"t_104","attempt_key":"a_9"}' \
  http://127.0.0.1:8000/tickets/summarize

psql "$APP_DATABASE_URL" -c \
  "select status, error_code, output_text is null as missing
   from completion_receipts
   where ticket_id = 't_104' and attempt_key = 'a_9';"

A reserved row that sits too long is a worker that died, and leaving it reserved forever blocks the same attempt key. I mark those rows failed with a worker_lost code after a short window I have not tuned in production. Two minutes in the example is a placeholder, not a measured timeout for any particular free server. If your provider is slower than that window, you will fail live calls, so measure your own latency before you copy the interval.

update completion_receipts
set status = 'failed', error_code = 'worker_lost'
where status = 'reserved'
  and created_at < now() - interval '2 minutes';

Before I would trust the slice, I want four proofs and I want them in this order. The session must reject a stranger, the reservation must exist before the provider call, and the stored row must match the body we returned. A killed worker must leave a non-success status, and a repeated attempt key must not create a second provider call. If any proof fails, I keep the feature behind the flag, even when the free seat makes the happy path feel finished.

This approach is the wrong tool when your database also lives on that same free host. You have only moved the temp file into a local Postgres directory, and a recycle still deletes the receipt. Skip it if you need a contractual quota, a latency SLO, or a promise that the free seat will still exist next quarter. I am not claiming those properties, and a free option that can change is a prototype seat rather than an architecture.

Streaming tokens into the UI before the receipt commits is a different contract, and this slice will feel too slow for that product. Do not use it as an authorization design either, because a stored paragraph does not prove the actor was allowed to read the ticket. Load the ticket through the same permission check you already trust, and keep the model seam from becoming a side door. If load_ticket is a stub that returns whatever the prompt contains, you have not built the feature, you have built a leak.

If you want a cheap seat for the worker while the receipt stays in a database you administer, look at the current free options. MonkeyCode is the seat I have in mind for that exercise, and only the limits listed on its page today should enter your plan. I already disclosed the outreach relationship above, so I will not repeat a slogan where a constraint belongs.

The handoff I trust least is still that gap between the model return and the commit. Which handoff is least stable in your stack, the session check, the provider call, or the write that should outlive the host? Send me the failure state you actually saw, including the response code, and tell me whether a receipt row existed afterward. If the code was 200 and the row was missing, we are describing the same bug, just on a different ticket.