DEV Community: kongkong

Capacity You Haven't Measured Is Capacity You Don't Have

kongkong — Tue, 20 Oct 2026 09:13:25 +0000

At 2 a.m. on a Thursday, a demo I had rehearsed four times died on the first request, in front of six people, on a free server I had never once looked inside. The endpoint answered locally. The migration ran on my laptop. The only thing that had changed was the box, and I had no number for that box, no memory ceiling, no cold-start figure, no idea whether the disk could take an eight-megabyte write. So here is my position, and I will argue it rather than hedge it: capacity you have not measured is capacity you do not have, and a free tier is only useful to the degree that you interrogate it.

That sounds like a complaint about free infrastructure, and it is not. MonkeyCode's free model access and free server option are, as far as this article is concerned, an operator-supplied starting point: a place where real latency, real tokenization and real resource limits exist, instead of the tidy fakes I used to write for myself at midnight. The free tier is valuable precisely because it is not polite. It returns errors you did not design for, and it punishes assumptions you did not know you had made. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

But a free tier is a starting position, not a verdict, and the interesting engineering question is not "can it ship?" but "what can it prove before I spend a week on it?" I want two artifacts out of that question. The first exercises the model seam so the call fails the same way twice. The second measures the server the moment it boots, because the number you never recorded is the number that wakes you up.

The seam: make the provider answer to a contract, not to a mood

If your application calls out to a hosted model, the risk is rarely the happy reply, which works. The risk lives in the three shapes that are almost the same and not quite: a timeout that should be retried, an error that should not be, and a response that parses as JSON while violating your schema. So I keep one module with a single narrow function, and I test that function against two providers, a fake I control and the real endpoint I do not.

# app/llm.py — illustrative request shape; copy the path your provider documents
import json, os, httpx

BASE_URL = os.environ["LLM_BASE_URL"]          # real endpoint or the local fake
MODEL    = os.environ["LLM_MODEL"]
TIMEOUT  = float(os.environ.get("LLM_TIMEOUT", "20"))

class ProviderTimeout(RuntimeError): ...
class ContractViolation(RuntimeError): ...

def summarize(prompt: str) -> dict:
    try:
        r = httpx.post(
            f"{BASE_URL}/chat/completions",
            json={"model": MODEL, "messages": [{"role": "user", "content": prompt}]},
            timeout=TIMEOUT,
        )
    except httpx.TimeoutException as e:
        raise ProviderTimeout(str(e)) from e

    if r.status_code >= 500:
        raise ProviderTimeout(f"upstream {r.status_code}")
    if r.status_code >= 400:
        raise ContractViolation(f"client {r.status_code}")

    text = r.json()["choices"][0]["message"]["content"]
    try:
        data = json.loads(text)
    except json.JSONDecodeError as e:
        raise ContractViolation("non-JSON body") from e
    if not isinstance(data.get("summary"), str) or not data.get("bullets"):
        raise ContractViolation("schema mismatch")
    return data

Nothing clever, and that is the point: every provider-specific quirk is squeezed into two exception types the rest of the app already understands. Now the same test file runs twice, once against the fake and once against the real thing, with only an environment variable changing.

# tests/test_generation_seam.py
import os, pytest
from app.llm import summarize, ProviderTimeout, ContractViolation

REAL = os.environ.get("RUN_REAL_PROVIDER") == "1"

@pytest.mark.parametrize("payload,expected", [
    ({"choices": [{"message": {"content": '{"summary": "ok", "bullets": ["a"]}'}}]}, "pass"),
    ({"choices": [{"message": {"content": 'sorry, I cannot'}}]}, "violation"),
    ({"choices": [{"message": {"content": '{"summary": 1}'}}]}, "violation"),
])
def test_response_shapes_are_classified(payload, expected):
    ...  # drive the fake, assert the exception type

@pytest.mark.skipif(not REAL, reason="needs a live endpoint")
def test_live_endpoint_honours_our_timeout():
    # force a deliberately tiny timeout and confirm we get ProviderTimeout,
    # not a hung worker or a bare socket error leaking past the seam
    ...

That skipped test is the one I care about most. Running it against the free model access tells me whether my timeout budget is realistic on real infrastructure, and it tells me in twenty seconds instead of during a review. It also gives me evidence for the only claim I am willing to make: this seam behaves correctly under the failures I chose to name. It says nothing about throughput, nothing about a paid model's quality, and nothing about tomorrow, because availability terms on any free tier are the operator's to change, not mine to assume.

The box: probe it at boot, or you are guessing in public

Here is the part I got wrong at 2 a.m. I had built a beautiful provider seam and deployed it to a server whose memory ceiling I had never read, which meant I had tested my code and shipped my ignorance. So now every deployment writes one honest row about itself before it serves anybody.

# app/probe.py
import json, os, platform, sqlite3, tempfile, time

DDL = """CREATE TABLE IF NOT EXISTS runtime_probe (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    rss_mb REAL, disk_write_mb REAL, disk_write_ms REAL,
    json_ops INTEGER, python TEXT, note TEXT
)"""

def rss_mb() -> float:
    with open("/proc/self/status") as f:
        for line in f:
            if line.startswith("VmRSS:"):
                return int(line.split()[1]) / 1024
    return -1.0

def disk_probe(mb: int = 8):
    payload = b"x" * (mb * 1024 * 1024)
    t0 = time.perf_counter()
    with tempfile.NamedTemporaryFile(dir=os.environ.get("PROBE_DIR", "/tmp"), delete=True) as f:
        f.write(payload); f.flush(); os.fsync(f.fileno())
    return mb, (time.perf_counter() - t0) * 1000

def json_probe(seconds: float = 1.0) -> int:
    blob = {"k": [i for i in range(500)]}
    n, t0 = 0, time.perf_counter()
    while time.perf_counter() - t0 < seconds:
        json.dumps(blob); n += 1
    return n

def record(db_path: str, note: str = "boot") -> dict:
    mb, ms = disk_probe()
    row = {"rss_mb": rss_mb(), "disk_write_mb": mb, "disk_write_ms": ms,
           "json_ops": json_probe(), "python": platform.python_version(), "note": note}
    with sqlite3.connect(db_path) as c:
        c.execute(DDL)
        c.execute(f"INSERT INTO runtime_probe ({','.join(row)}) VALUES ({','.join('?'*len(row))})", tuple(row.values()))
    return row

Expose the latest row at GET /probe/latest and you have a permanent, boring, honest answer to "what was this thing actually running on?" Later, when someone swears the endpoint got slower, you compare two rows instead of two memories. You are not benchmarking a provider, and you must not publish these numbers as a comparison against hardware you never touched, because that is how a measurement becomes a lie.

What the free tier can prove, and what it cannot

Claim you want to make	Can the free tier support it?	Evidence you still owe
My timeout and retry logic classifies failures	Yes, via the live seam test	A recorded exception type per failure
My JSON contract survives real model output	Partially, per run	A re-run before each release, not a screenshot
My app fits in memory	Yes, at the measured point	The probe row at boot and after a load pass
My app handles 50 concurrent users	No	A load test on the target you intend to pay for
My data handling meets a compliance need	No	Written terms, not a blog post

The middle column is the whole argument. Free infrastructure is an excellent instrument for falsifying your own assumptions, and a terrible foundation for promises you make to other people. Who should not use this approach? Anyone with an uptime clause, a regulator, or a paying customer who would notice the difference between a good week and a guarantee. Who should? Anyone with a prototype and a suspicion that the prototype is lying.

So point your client at the free model access, run the seam with the live flag on, deploy the probe, and read the row it writes. Then ask yourself which layer handed off to which, and where you would find out first. Tell me which handoff in your stack is the least stable, the model call, the write, or the deploy, and paste the exact failure state or status code you see, because that is the number I want in my test file next.

A Model Swap Is a Migration, Not a Toggle

kongkong — Mon, 19 Oct 2026 07:47:35 +0000

A Model Swap Is a Migration, Not a Toggle

Last Friday I changed a single environment variable, pointed the support-reply feature at a different provider, and went home feeling clever. On Monday the export job had three days of rows where refund_days was null, and nobody flagged it because the UI rendered the reply text and quietly ignored the missing field. My first instinct was to blame the new model. My second, more honest instinct was to ask why I had shipped a schema-affecting change with no fixtures, no diff, and no gate at all. We stopped migrating databases by hand-editing SQL in production years ago, so why do we still swap the thing that generates our persisted data by editing a config line?

I will state the position plainly, because hedging here helps nobody: a model swap is a migration, not a toggle. The output lands in your database columns, your retry classifier, your export files, and your customer's inbox, which means you carry schema obligations the moment you persist it. A toggle implies clean reversibility with no residue, and anyone who has rolled back a half-applied migration knows how fictional that is. So the useful argument is not which model wins this month. It is what evidence you demand before you change the one you already run.

Why your test suite will not save you here

Integration tests assert expectations you wrote by hand, which makes them excellent at catching the bugs you already imagined. They say almost nothing about how a new provider behaves on the four hundred real tickets your team handled last month, including the ugly ones with partial refunds and mixed locales. I have watched a green suite ship a provider change that broke three percent of exports, and I have also watched a red suite block a change that was strictly better. Neither outcome told me anything about traffic. Isn't that the gap worth closing first?

The cheapest honest evidence is a frozen fixture corpus captured from production before you touch the provider. Not synthetic prompts, not a curated golden set copied from documentation, but the actual inputs you already paid to process, with personal data stripped out. Once those fixtures exist, the swap becomes a comparison you can rerun on demand instead of an opinion you defend in a code review.

The harness: capture, shadow, diff, gate

Here is the version I use now. Capture writes one JSON file per real request, keeping the input and the baseline output so later runs have something to compare against.

# shadow/capture.py (sketch)
import json, pathlib, datetime

def write_fixture(root, request_id, payload, baseline_raw, provider):
    path = pathlib.Path(root) / f"{request_id}.json"
    path.write_text(json.dumps({
        "id": request_id,
        "captured_at": datetime.datetime.utcnow().isoformat() + "Z",
        "input": payload,
        "baseline": {"provider": provider, "raw": baseline_raw},
    }, indent=2))

Redaction happens before the write, never after, because a fixture folder that leaks ticket text is a second incident stacked on the first. I keep the redactor in the same module and drop anything matching an email, phone, or order-number pattern. The fixture then becomes a reviewable artifact inside the pull request, which is exactly what a migration needs.

The shadow run replays every fixture through the candidate and scores two things independently: whether the payload still parses into the shape the database expects, and how far the text drifted from the baseline.

# shadow/run.py
import json, time

REQUIRED = {"reply": str, "refund_days": int}

def shape_errors(raw):
    try:
        data = json.loads(raw)
    except json.JSONDecodeError as exc:
        return [f"unparseable: {exc.msg}"]
    return [f"{k} missing or wrong type" for k, t in REQUIRED.items()
            if not isinstance(data.get(k), t)]

def token_overlap(a, b):
    ta, tb = set(a.lower().split()), set(b.lower().split())
    return 0.0 if not ta or not tb else len(ta & tb) / len(ta | tb)

def shadow(adapter, fixtures):
    rows = []
    for fx in fixtures:
        started = time.monotonic()
        raw = adapter.complete(fx["input"])
        rows.append({
            "id": fx["id"],
            "shape_errors": shape_errors(raw),
            "overlap": round(token_overlap(fx["baseline"]["raw"], raw), 3),
            "latency_ms": int((time.monotonic() - started) * 1000),
        })
    return rows

Token overlap is a tripwire and not a judge, and I want to be explicit about that limitation. It punishes valid rewording and rewards lazy copying, so all it can do is tell you where to look next. I pair the score with a manual read of the ten lowest-overlap fixtures, which costs twenty minutes and has caught more real problems than any automatic metric I have tried.

The gate is where the opinion becomes enforceable. Shape failures are absolute, drift has a small budget, and the exit code is what CI actually reads.

# shadow/gate.py
import sys

def evaluate(rows, overlap_floor=0.90, max_drift=2):
    shape_failures = [r["id"] for r in rows if r["shape_errors"]]
    drifted = [r["id"] for r in rows if r["overlap"] < overlap_floor]
    if shape_failures:
        print("shape failures:", shape_failures[:10])
        return 2
    if len(drifted) > max_drift:
        print(f"drift {len(drifted)} > {max_drift}:", drifted[:10])
        return 3
    print(f"pass: {len(rows)} fixtures, {len(drifted)} drifted")
    return 0

sys.exit(evaluate(rows))

Signal	Blocking?	Why
Shape failure	Yes, always	Downstream parsers and exports depend on it
Drift above floor	Yes, past a small budget	A few rewrites are noise, a cluster is a behavior change
Latency p95 regression	Reviewer decision	Slower is sometimes fine, silently slower never is
Cost per fixture	Reviewer decision	Belongs in the pull request, not in a dashboard nobody opens

Notice what the table refuses to do: it never pretends to measure quality. Quality needs a human or a labelled set, and pretending otherwise is how teams end up trusting a number that only measured formatting.

Where the free tier fits, honestly

Here is the practical part that made me try this at all. MonkeyCode is an open-source project that offers free model access and a free server option, and that pairing maps onto this harness suspiciously well: the free model access gives you a candidate slot to diff against without a procurement conversation, and the free server option gives you somewhere to run the shadow job without borrowing production capacity. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

That said, I treat both as convenience, never as architecture. I have not run long-term reliability or quota measurements, so I keep no state on the shadow box and I assume every job can be re-run elsewhere tomorrow. The harness is provider-agnostic by design, which is the entire point of the adapter interface: if the candidate slot disappears, I repoint one class and replay the same fixtures. Ask yourself whether your swap plan survives the candidate provider vanishing without notice, because that question is far cheaper now than during an incident.

One more boundary deserves stating out loud. The shadow worker gets read-only fixtures and no database credentials at all. If it can write to the table it is evaluating, it is not a shadow anymore, it is a second production writer with a different prompt.

When this is the wrong approach

Do not build this if your outputs are genuinely creative and structural comparison is meaningless, because you will spend a week building a metric that cannot see what your users care about. Do not build it if you have no real traffic yet, since twenty hand-written prompts teach you nothing and hand you false confidence in exchange. And do not build it if a human reviews every output before it reaches a customer, because in that world the reviewer is already the gate.

The approach also fails quietly for teams that capture fixtures once and never refresh them, which is just a migration test written against last year's schema. Refresh the corpus on a schedule, retire fixtures older than a quarter, and treat the fixture count as a health signal rather than a trophy.

A checklist you can reuse

Capture a fixture corpus from real traffic, applying redaction before the write.
Encode the required output shape next to the parser that depends on it.
Run the candidate in shadow with no write credentials and no shared state.
Gate on shape failures first, drift second, and latency plus cost as reviewer notes.
Read the worst ten overlaps by hand before trusting any aggregate number.

None of that is exotic, and the whole harness fits in a few hundred lines. That is precisely why I find "we will just try the new model and watch the dashboards" so hard to defend in 2026. Watching dashboards is how I lost three days of refund_days, and the dashboard stayed green the entire time.

So the argument I would put to you is simple. The next time someone proposes a provider swap, ask for the fixtures, the diff budget, and the exit code, in that order. If the answer is a spreadsheet of vibes, then the swap is a migration you are running blind. And if you do stand the harness up on the free server option, tell me which layer broke first: the capture step, the shadow worker, or the threshold you picked.

Ten Million Free Tokens Are a Balance, Not an Architecture

kongkong — Sun, 18 Oct 2026 06:43:25 +0000

Last Thursday a support lead clicked Summarize on a thread with forty-one replies, waited six seconds, and got a summary that was maybe ninety percent right. The bad part arrived ninety minutes later, when that same button had been pressed four hundred times and the spend page showed a number I had never seen before. The request did not fail at the UI, and it did not fail inside the model either; it failed at the layer where nobody kept count. Three files imported a model client directly, each with its own retry policy, and none of them knew what the others had already spent.

So let me state the position this whole post hangs on: free tokens are a balance sheet, not an architecture. A grant of ten million tokens is genuinely useful, and I will show exactly where I spent mine, but generosity is not a design decision. What matters is what your service does at zero, and whether you can answer that question without opening four files and guessing.

The failure was never the model

When I traced the four hundred requests, the model had behaved fine; latency was acceptable and the outputs were mostly good. The failure was accounting, and accounting failures are boring in a way that makes them expensive: nothing crashes, nothing pages you, and the bill simply appears. I had treated a finite grant as ambient capacity, which is the same mistake as treating a free database tier as a schema.

Does that sound familiar? Almost every prototype I have reviewed this year has the same shape: a working demo, one environment file, and no single place where cost is decided. The model is the least interesting part of that stack, because the model can be swapped in an afternoon.

I ran the rewrite against MonkeyCode's free model access because I wanted the ledger to meet a real provider before I paid anyone, and the free server option gave me somewhere to run it that was not my laptop. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The offer I used, at the time of writing, advertises ten million free tokens plus a free server, and I am deliberately not going to promise you a latency number or a hardware profile, because those are the parts that change.

One door, and a test that proves there is only one

My rule is unglamorous: exactly one function in the codebase is allowed to touch a provider. Everything else calls it. That function reserves budget before the network call, commits real usage after it, and releases the reservation when the call throws.

# app/gateway.py
from __future__ import annotations
from dataclasses import dataclass
from typing import Protocol

class BudgetExceeded(Exception):
    """Raised before the network call, never after it."""

@dataclass(frozen=True)
class Usage:
    prompt_tokens: int
    completion_tokens: int

    @property
    def total(self) -> int:
        return self.prompt_tokens + self.completion_tokens

class Provider(Protocol):
    name: str
    def complete(self, model: str, prompt: str, max_tokens: int) -> tuple[str, Usage]: ...

The ledger is a single row per account, because a row is something you can update atomically and a spreadsheet is not. Reserve is a conditional update, so two concurrent requests cannot both squeeze past the ceiling.

# app/ledger.py
RESERVE = """
update budget set reserved = reserved + :est
 where account_id = :acct
   and reserved + spent + :est <= granted
"""
COMMIT = """
update budget set reserved = reserved - :est, spent = spent + :actual
 where account_id = :acct
"""
RELEASE = """
update budget set reserved = reserved - :est
 where account_id = :acct and reserved >= :est
"""

class Ledger:
    def __init__(self, conn):
        self.conn = conn

    def reserve(self, acct: str, est: int) -> None:
        cur = self.conn.execute(RESERVE, {"acct": acct, "est": est})
        if cur.rowcount != 1:
            raise BudgetExceeded(f"{acct}: cannot reserve {est} tokens")

    def commit(self, acct: str, est: int, usage: Usage) -> None:
        self.conn.execute(COMMIT, {"acct": acct, "est": est, "actual": usage.total})
        self.conn.commit()

    def release(self, acct: str, est: int) -> None:
        self.conn.execute(RELEASE, {"acct": acct, "est": est})
        self.conn.commit()

With that in place the handler becomes short enough to read in a review, and the ordering is the point: estimate, reserve, call, commit, or release.

def estimate(text: str, max_tokens: int = 400) -> int:
    # ~4 characters per token, plus headroom for the completion
    return -(-len(text) // 4) + max_tokens

def summarize(acct: str, text: str) -> str:
    est = estimate(text)
    ledger.reserve(acct, est)
    try:
        out, usage = provider.complete(model, text, max_tokens=400)
    except Exception:
        ledger.release(acct, est)
        raise
    ledger.commit(acct, est, usage)
    return out

The test I actually care about is not the happy path, it is the one that proves a failed call costs nothing and that an exhausted account never reaches the wire.

class Fake:
    name = "fake"
    def __init__(self, usage=None, boom=False):
        self.calls, self.usage, self.boom = 0, usage, boom
    def complete(self, model, prompt, max_tokens):
        self.calls += 1
        if self.boom:
            raise TimeoutError("upstream stalled")
        return "ok", self.usage

def test_timeout_releases_reservation(db):
    led = Ledger(db)
    with pytest.raises(TimeoutError):
        run(led, Fake(boom=True), acct="team-a", text="hello")
    row = db.execute("select reserved, spent from budget where account_id='team-a'").fetchone()
    assert row == (0, 0)

def test_exhausted_account_never_calls_provider(db):
    led = Ledger(db)
    fake = Fake(usage=Usage(10, 10))
    with pytest.raises(BudgetExceeded):
        run(led, fake, acct="team-empty", text="x" * 10_000)
    assert fake.calls == 0

Reserve, then commit, then argue about the estimate

My estimate is wrong in both directions, and I know it. Character division undercounts code and JSON badly, so a prompt full of stack traces can blow past the reservation. Overcounting is the cheaper failure: you stop early, you leave tokens on the table, and nobody is surprised by a bill.

When actual usage exceeds the reservation, commit still writes the real number, which means the row can temporarily go negative in reserved and the next reserve fails until you reconcile. I accept that, because a brief refusal is better than a silent overspend. A nightly job compares provider-side usage to spent; if the gap exceeds two percent, the model or the tokenizer changed under me, and I want that alarm.

I also keep the only-door rule executable, because a comment is not a policy:

! grep -rn "import openai\|from anthropic\|ai-sdk" app/ --include="*.py" | grep -v "app/gateway.py"

That line runs in CI and fails the build. It has caught two well-meaning refactors already, both of which "just needed a quick call" in a route handler.

What the free server actually buys you

The free server is not interesting because it is free; it is interesting because it is a real deploy target you can break without a meeting. I use it for the branch that runs against a free model, so the ledger gets exercised by a container with the same environment variables as production.

The part that keeps that box alive is a boot check, not a retry wrapper. If the base URL, the model id, or the ceiling is missing, the process should refuse to start rather than serve traffic with an unbounded budget.

import os, sys
REQUIRED = ("PROVIDER_BASE_URL", "PROVIDER_MODEL", "TOKEN_CEILING")
missing = [k for k in REQUIRED if not os.getenv(k)]
if missing:
    sys.exit(f"refusing to boot, missing: {missing}")

Then a health endpoint that reports the remaining balance, which turns deployment into a one-liner you can run after every push.

curl -s "$BASE_URL/healthz" | jq '.remaining_tokens, .provider'

If that number moves while you are not clicking anything, something is calling the model on page load, and you have found your bug without reading a log.

When a grant is enough, and when it is not

Situation	Free grant is fine	Start paying instead
Internal tool, few users	Yes, with a per-account ceiling	When several teams share one balance
Customer-facing latency	No, cold starts will show	When p95 matters more than price
Regulated or personal data	No, until you have a data path review	Always
Batch backfills	Yes, at low priority	When a failed batch costs a day

I would not run anything with contractual uptime, medical text, or a per-tenant data-residency clause on a free tier, and I would not use this pattern as an excuse to skip a real budget owner. If your team cannot name the person who approves a spend increase, the free grant is hiding an org problem rather than solving a technical one.

What the balance sheet buys you is optionality: when the grant runs out you change one file, not forty. So here is my question for you, and I want the ugly version rather than the polished one: which handoff in your AI feature is least stable right now, and what exact failure do you see there, a 402 from the provider, a 504 from your own gateway, or a 200 with an empty string in it?

A Free Tier Is a Contract Test, Not a Backend

kongkong — Fri, 16 Oct 2026 04:41:40 +0000

Last Tuesday a teammate opened a pull request that called a model straight from a route handler, and on his laptop it worked for two days without a hiccup. On staging it died at 03:14 with a 500 that was hiding a 429 inside three layers of try/catch, and nobody could tell which layer had decided to give up. I have watched this same failure land in four different codebases this year, and the model is never the culprit. The culprit is that the free tier made the first call so easy that he skipped the seam entirely.

Here is my position, stated plainly, because I do not think this deserves a polite hedge: a free model tier is not a cheaper backend, it is the cheapest contract test you will ever be handed. If you spend it avoiding an adapter, you have purchased a small discount plus a large future incident. If you spend it proving that your adapter's failure taxonomy survives a real, flaky, rate-limited upstream, you have purchased something that outlives every model name in this week's news cycle.

That distinction matters more now than it did a year ago, because the industry has shipped agent features faster than it has defined what an agent failure looks like. We have plenty of demos where the happy path is beautiful, and very few where a rate limit arrives mid-request and the UI tells the truth about it. So when I look at MonkeyCode — an open-source project offering free model access and a free server option — I judge it on one question only. Can it help me run the failure tests I keep postponing?

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

One honesty note before the code: the free token allowance and the free server option are operator-supplied claims about what is available at the time of writing. Free tiers move, so read the project's current terms before you plan a workflow around them, and never treat any free tier as a capacity guarantee.

The seam I wish my teammate had written first

A provider seam is boring on purpose. It is one interface, one normalized failure type, and one rule that says failures come back as values instead of exceptions thrown from a request handler. The rule sounds pedantic until you meet a 429 that has been re-thrown three times and renamed twice.

// src/ai/contract.ts — UNTESTED SKETCH. Illustrative shape, not a product API.
export type Failure =
  | { kind: "rate_limited"; retryAfterMs?: number }
  | { kind: "timeout" }
  | { kind: "cancelled" }
  | { kind: "upstream"; status: number; requestId?: string }
  | { kind: "invalid_output"; detail: string };

export type Completion = {
  text: string;
  tokensIn: number;
  tokensOut: number;
  requestId: string;
};

export interface ModelProvider {
  readonly name: string;
  complete(input: { prompt: string; signal?: AbortSignal }): Promise<Completion | Failure>;
}

The adapter for a free endpoint looks almost identical to the adapter for a paid one, which is the entire point. Below I am guessing at the HTTP shape, so treat the path and body as placeholders you replace with whatever the docs actually say today.

// src/ai/providers/monkeycode.ts — UNTESTED SKETCH.
import type { Completion, Failure, ModelProvider } from "../contract";

const BASE = process.env.MONKEYCODE_BASE_URL ?? "";
const KEY = process.env.MONKEYCODE_API_KEY ?? "";

export function monkeycodeProvider(): ModelProvider {
  return {
    name: "monkeycode",
    async complete({ prompt, signal }): Promise<Completion | Failure> {
      // Fail closed: a missing key is a configuration error, not a user error.
      if (!BASE || !KEY) {
        return { kind: "upstream", status: 500, requestId: "missing-config" };
      }
      try {
        const res = await fetch(`${BASE}/complete`, {
          method: "POST",
          signal,
          headers: { "content-type": "application/json", authorization: `Bearer ${KEY}` },
          body: JSON.stringify({ prompt }),
        });
        if (res.status === 429) {
          const raw = Number(res.headers.get("retry-after") ?? NaN);
          return { kind: "rate_limited", retryAfterMs: Number.isFinite(raw) ? raw * 1000 : undefined };
        }
        if (!res.ok) {
          return { kind: "upstream", status: res.status, requestId: res.headers.get("x-request-id") ?? undefined };
        }
        const body = await res.json();
        return {
          text: body.text,
          tokensIn: body.usage?.input_tokens ?? -1,
          tokensOut: body.usage?.output_tokens ?? -1,
          requestId: body.request_id ?? "unknown",
        };
      } catch (err) {
        if ((err as Error).name === "AbortError") return { kind: "cancelled" };
        return { kind: "timeout" };
      }
    },
  };
}

One suite, every adapter, including the deployed one

The suite matters more than the adapters, because it is the thing you cannot fake. I keep it provider-agnostic and run it against a registry, so adding a second backend is a one-line change rather than a new test file nobody maintains.

// tests/provider.conformance.ts — UNTESTED SKETCH.
import { describe, expect, it } from "vitest";
import { providers } from "../src/ai/registry";

for (const [name, make] of Object.entries(providers)) {
  describe(`provider contract: ${name}`, () => {
    it("returns cancelled, not timeout, when the signal is already aborted", async () => {
      const ac = new AbortController();
      ac.abort();
      const result = await make().complete({ prompt: "hello", signal: ac.signal });
      expect(result).toMatchObject({ kind: "cancelled" });
    });

    it("never throws — every failure is a value with a kind", async () => {
      const result = await make().complete({ prompt: "" });
      expect(result).toHaveProperty("kind");
    });

    it("reports token usage as finite numbers, or -1 when unknown", async () => {
      const result = await make().complete({ prompt: "one short sentence" });
      if ("kind" in result) return; // provider outage is asserted elsewhere
      expect(Number.isFinite(result.tokensIn)).toBe(true);
      expect(Number.isFinite(result.tokensOut)).toBe(true);
    });
  });
}

Here is the part I actually care about, and the reason a free server option is worth more than a free key. You point the same suite at a deployed instance, not at localhost, and you find out whether your environment wiring is real.

# Local adapters, real network calls, no bill.
npx vitest run tests/provider.conformance.ts

# Same suite, deployed target: catches missing env vars and health-check gaps.
MONKEYCODE_BASE_URL="$DEPLOYED_BASE_URL" \
MONKEYCODE_API_KEY="$KEY" \
PROVIDER_TARGET=deployed \
  npx vitest run tests/provider.conformance.ts

A green local run and a red deployed run is the single most useful signal I get out of a free environment. It means the bug lives in deployment, permissions, or configuration rather than in the model call, and that is a much smaller search space than the one my teammate had at 03:14.

The mapping table that ends the 429-versus-500 argument

Most cross-layer arguments are really disagreements about which layer owns a translation. I settle them with a table, checked into the repository next to the adapter, so nobody has to remember the decision at review time.

Upstream condition	Adapter result	Response to the browser	Retried?
HTTP 429	`rate_limited`	429 with `Retry-After`	Yes, once, honoring the header
Signal aborted	`cancelled`	499-style client abort log	No
5xx from upstream	`upstream` with request id	502, request id echoed	Yes, only idempotent reads
Empty or unparseable body	`invalid_output`	502, never a green toast	No
Missing API key	`upstream` 500	503, pages the operator	No

The row that saves the most pain is the last one. A missing key is not a model problem, and pretending otherwise sends everyone hunting through prompt code for an hour. Fail closed, return a status that says "we are misconfigured," and keep the model out of the blame list.

What actually broke when I ran this pattern

Three failures showed up in the first week, and none of them were the model's fault. Cancellation stopped at the adapter because the route handler never forwarded request.signal, so closing a browser tab kept burning tokens silently. Token accounting drifted because my adapter returned -1 for unknown usage and the cost dashboard happily summed negative numbers. And a retry policy I copied from a payment integration retried a non-idempotent write, which produced two saved summaries and one very confused user.

None of those are exotic. They are the ordinary consequences of letting a working first call convince you that the seam is unnecessary.

Who should not use this approach

If your prompts cannot leave your network without a signed agreement in place, a hosted free tier is the wrong environment regardless of price, and no adapter will fix that. If you are already at committed-volume scale, free capacity is irrelevant and a dedicated deployment with a documented throughput target is the honest choice. And if you are tempted to let a free server hold the only copy of user state, do not — run it as a staging target with disposable data, then move persistence somewhere you control.

For everyone else, the pattern is cheap and the payoff is durable. Write the contract, run the suite against something you did not pay for, watch which layer turns red, and fix that layer instead of re-prompting. If you want a place to try it without a bill, the free model access and the free server from MonkeyCode are worth one afternoon of your time.

So here is the question I would like answered in the comments, because it is the one I cannot answer alone. Which layer handoff in your stack is least stable when an upstream starts misbehaving — and what concrete failure state or response code does it hand you when it breaks? A specific status code and the layer that produced it teaches me more than a general warning ever will.

Admit the Event Before the Free Model Touches It

kongkong — Mon, 12 Oct 2026 22:45:56 +0000

I keep replaying one ugly afternoon whenever a webhook and a free model share the same request. The billing provider retried, the free server restarted, and two explanations landed for one event that nobody had firmly admitted. A zero price had become the excuse to call the model before the database said yes. Was I debugging a flaky model, or was the request path doing a job it should never own?

That replay is why my opinion stays sharper than a shrug about free tiers and cheap hosts. I want the event admitted in a durable outbox before any free model is allowed to see the payload. The free server should run that same admission route and one worker, not a side script with its own key. If the invoice stays silent, do you honestly believe the duplicate write will stay silent too?

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I mention MonkeyCode here only because its free model access and its free server option fit this admission slice. A vendor name does not make an inline model call safe inside a short webhook timeout. I am not naming a model, a token quota, a machine size, or a period for which free access continues.

If those facts matter for your rollout, check the current product notes before you freeze a design on them. My claim is about ownership, and it would stand even if you swapped the free options for paid ones tomorrow. Admit the foreign event first, return as soon as the outbox row exists, and only then spend a model call. Why would you spend a fragile free call before you can prove the event will not arrive again?

The layer that actually breaks

Picture a billing webhook that carries a provider event id, an account id, and a short reason code. The handler in that prototype calls the model, waits for a sentence, and tries to write it before returning 200. When the proxy cut the socket, the provider retried, and the second call wrote a second sentence beside the first. The first broken layer was admission, not the wording, because the database had never claimed the event id.

I want a boring outbox table more than I want a clever prompt sitting on this boundary. The row stores the provider event id as a unique key, a payload hash, a status, and the account it belongs to. Status moves from admitted to running to labeled, or it stops at empty or failed without pretending success. A unique key is a better guard than a memory lock, because a free server restart throws memory away.

create table billing_events (
  provider_event_id text primary key,
  account_id text not null,
  payload_hash text not null,
  status text not null,
  label_text text not null default '',
  claimed_at timestamptz
);

The admission route should feel dull, and dull is a real compliment when webhooks get involved. It checks a shared secret, inserts the outbox row, and returns 202 with the event id and nothing else. If the insert hits the unique constraint, it returns 202 again and does not call the model on the replay. Can you explain your webhook to a reviewer if success sometimes means the model ran and sometimes does not?

One seam, then the call

The sketch below is a proposal I would put behind that route, and it is not a trace from a live host. I keep the provider client behind one function so the free model cannot leak into the webhook module. The worker claims a row by moving it to running, and a second worker must see zero rows to claim. If you cannot describe that claim in one sentence, you are not ready to point a free server at it.

# Proposal sketch, not a captured trace from a live host.
import hashlib
from enum import Enum

class EventStatus(str, Enum):
    ADMITTED = 'admitted'
    RUNNING = 'running'
    LABELED = 'labeled'
    EMPTY = 'empty'
    FAILED = 'failed'

def payload_hash(raw: bytes) -> str:
    return hashlib.sha256(raw).hexdigest()

def admit_event(store, account_id: str, provider_event_id: str, raw: bytes):
    inserted = store.insert_once(
        account_id=account_id,
        provider_event_id=provider_event_id,
        payload_hash=payload_hash(raw),
        status=EventStatus.ADMITTED,
    )
    if not inserted:
        return store.status_of(provider_event_id)
    return EventStatus.ADMITTED

def claim(conn, provider_event_id: str, expected: str, nxt: str) -> bool:
    cursor = conn.execute(
        '''
        update billing_events
           set status = ?, claimed_at = current_timestamp
         where provider_event_id = ?
           and status = ?
        ''',
        (nxt, provider_event_id, expected),
    )
    return cursor.rowcount == 1

def label_once(store, client, provider_event_id: str):
    claimed = store.claim(
        provider_event_id,
        EventStatus.ADMITTED,
        EventStatus.RUNNING,
    )
    if not claimed:
        return store.status_of(provider_event_id)
    try:
        text = client.explain(
            store.payload_for(provider_event_id),
            timeout_ms=8000,
        )
    except TimeoutError:
        store.mark(provider_event_id, EventStatus.FAILED, text='')
        return EventStatus.FAILED
    if not text or not text.strip():
        store.mark(provider_event_id, EventStatus.EMPTY, text='')
        return EventStatus.EMPTY
    store.mark(provider_event_id, EventStatus.LABELED, text=text.strip())
    return EventStatus.LABELED

I map empty model text to an empty status because silence is not a label a support page should display. I map a timeout to failed only after the row records the attempt, so a restart can decide whether to retry. I do not retry inside the webhook, since the provider will retry admission and the outbox already dedupes that. A retry belongs to the worker, and only when the row is still admitted or explicitly marked for another pass.

@app.post('/webhooks/billing', status_code=202)
async def admit_billing(request: Request, store=Depends(get_store)):
    raw = await request.body()
    signature = request.headers.get('x-signature', '')
    if not signature_ok(raw, signature, settings.webhook_secret):
        raise HTTPException(status_code=401, detail='bad signature')
    event = parse_billing_event(raw)
    status = admit_event(store, event.account_id, event.provider_event_id, raw)
    return {'provider_event_id': event.provider_event_id, 'status': status}

When I use the free server option, I want one image that serves the admission route and the worker loop. A second script with a copied key is how prototypes grow a private back door that nobody monitors. The same migration must create the outbox before the process listens, or the host is not actually running the slice. Would you call it deployed if a restart loses every event that had not yet reached the model?

The page never asks the model anything, and that separation is the part I would defend in review. It reads the outbox by account and event id, shows admitted as pending, and shows labeled only when text exists. Auth on that read must match the account on the row, or a free server becomes a window onto someone else's events. If the UI polls the webhook response body instead, you have quietly moved the model back onto the hot path.

@app.get('/accounts/{account_id}/events/{provider_event_id}')
def read_event(account_id: str, provider_event_id: str, actor=Depends(current_actor), store=Depends(get_store)):
    if actor.account_id != account_id:
        raise HTTPException(status_code=403, detail='wrong account')
    row = store.get(account_id, provider_event_id)
    if row is None:
        raise HTTPException(status_code=404, detail='missing event')
    label = row.text if row.status == EventStatus.LABELED else ''
    return {'status': row.status, 'label': label}

Prove the replay before the host does

Before that host sees traffic, I run the same checks on my laptop so the free tier is not my test suite. The pytest file should cover a fresh admit, a duplicate event id, an empty completion, and a worker crash after the claim. Curl should show 202 for the first post and 202 for the immediate replay, with one row in the table. If those commands need a different env file on the server, I treat that drift as a bug rather than a convenience.

python -m pytest tests/test_webhook_outbox.py -q
alembic upgrade head
uvicorn app.main:app --host 0.0.0.0 --port 8000
python -m app.worker
curl -sS -D - -o /tmp/admit.json \
  -H 'content-type: application/json' \
  -H "x-signature: $WEBHOOK_SIG" \
  --data '{"id":"evt_1","account":"ac_9","reason":"past_due"}' \
  http://127.0.0.1:8000/webhooks/billing

The duplicate test matters more than a happy-path demo, because free inference makes retries feel harmless. I assert that the client mock was called once, even though the webhook handler was invoked twice with the same id. The crash test kills the worker after the status becomes running, then starts it again and expects no second label. A green process is not evidence, and I want the row count to stay at one when the world gets rude.

def test_replay_does_not_call_model_twice(store, client):
    raw = b'{"id":"evt_1","account":"ac_9","reason":"past_due"}'
    first = admit_event(store, 'ac_9', 'evt_1', raw)
    second = admit_event(store, 'ac_9', 'evt_1', raw)
    assert first == EventStatus.ADMITTED
    assert second == EventStatus.ADMITTED
    label_once(store, client, 'evt_1')
    label_once(store, client, 'evt_1')
    assert client.calls == 1
    assert store.count(provider_event_id='evt_1') == 1

I chose an outbox over an in-request call because webhook timeouts and model latency do not share a clock. I chose a provider event id over a random job id because the sender already gave me the idempotency key. I chose one worker loop over a fan-out queue because this slice does not need a broker to prove the contract. The approach I would reject is streaming the label back on the webhook response, since the sender is not a person waiting.

Who should leave this alone

This is the wrong shape if a person is staring at a spinner and needs the label before the next click. It is also wrong if you must fan the same event across several models and merge their answers in one transaction. Do not park raw payment details, secrets, or regulated text in an outbox just because the server happens to be free. Free model access and a free server can change, throttle, or disappear, and I will not invent a calendar for that.

If you came for a leaderboard, a hardware story, or proof that a free model beats a paid one, this will disappoint you. I am defending a handoff, not a quality claim, and I have not run a benchmark you should quote. Skip the pattern when your compliance review forbids storing the payload hash next to an account id on a shared host. A small internal tool can still use it, as long as the same tests pass on the laptop and on the free server.

The slice is honest when admission is durable, the model sits behind one seam, and replays do not mint new labels. Until the duplicate test fails closed, a live call on a free host is just a demo with a public address. If you press this into your own webhook, tell me which handoff feels least stable under a real replay. Is it the unique insert, the worker claim, the provider timeout, or the free server restart, and what status code came back?

Deliver the Summary With a Receipt Row, Not a Temp File

kongkong — Sun, 11 Oct 2026 20:08:27 +0000

I want to walk a reconstructed click, not a metric from a customer I do not have. The spinner completes, the panel shows a summary, and a refresh after the worker recycles comes back blank. The call was already spent, and the only paragraph lived in a temp file on the worker. Does a green toast mean you own the result, or does it only mean the process was still breathing?

I think we keep mistaking a cheap seat for a system of record, and that mistake gets expensive the moment a host recycles. A free model allotment is useful fuel for one vertical slice while you learn the seams. A free server is a fair place to run that worker, but it is not memory you are allowed to trust. If the host can forget the bytes, the handler must not tell the user that the answer is saved.

The tempting story is that a working button on a free host means the feature is shipped, and I do not buy that story. Vibe-coded glue can paint the panel, call a model, and even sound confident while the only durable artifact is a screenshot. A screenshot is not a receipt, and a host you can lose overnight is not a place to keep customer text. If the prototype cannot survive a kill and a select, it is still a prototype, however fast the first demo felt.

The break is usually a boring handoff, not a dramatic failure buried inside the model itself. The UI posts a ticket id, the API checks the session, and the worker calls the model with text it already stored. The model returns a completion, the worker drops it into a temp file, and only then does the handler try to answer the browser. Which of those steps is the commit, and which are just hallway gossip between processes that can die?

I would rather return a hard error than let that temp file pretend to be the commit. The analogy I keep using is a coat check in a crowded hallway after a show. The model hands you a coat, and the free host is the hallway that gets swept when the lights come up. You do not tell the guest the coat is safe until the stub is written in a book you actually keep.

I am placing the disclosure here, because this is the first point where the product actually enters the method. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I am using its free model access and its free server option as the cheap worker seat, not as the database I trust.

The outreach brief also cites a free allotment on the order of ten million tokens, which I have not re-measured for this draft. Confirm that figure on the current project page before you plan a sprint around it, because quotas move and old screenshots lie. I will not treat an unverified allotment as a capacity plan, and I will not omit the claim the brief asked me to surface.

My position is simple enough to argue with, which is how I like a production rule. The free host may execute the call, but your own store must commit the receipt before any success status leaves the API. If the commit fails, the client should see a retryable error, not a toast that points at memory the next restart will erase. Would you ship a payment capture that lived only in the web process, and why is a model completion any less of a write?

What follows is a proposed slice, not a benchmark I ran against a live free host this week. I am labeling the SQL and the handler as unexecuted examples, so you can adapt them without inheriting fake timings. The receipt row is the stub in the coat-check book, and the response body is copied from that row after the commit. If you cannot point at the row, you do not get to say the feature worked.

The table is deliberately dull, because dull is what you want when a free host disappears under you. Status is a small state machine, not a boolean the UI can paint green whenever the socket closes. The unique constraint is what stops a browser refresh from looking like a brand-new user intent.

create table completion_receipts (
  id uuid primary key,
  actor_id uuid not null,
  ticket_id uuid not null,
  attempt_key text not null,
  status text not null check (status in ('reserved', 'stored', 'failed')),
  output_text text,
  error_code text,
  created_at timestamptz not null default now(),
  unique (actor_id, ticket_id, attempt_key)
);

A reserved row means we intended to spend a call, and a stored row means the text now belongs to us. A failed row means we owe the user an honest error instead of a second silent spend. The unique key is the part people skip, and then they wonder why a retry doubles the bill against a finite allotment.

Read the handler from the reservation downward, because that order is the entire opinion in code. The actor id and the connection are injected from the session and the pool, not fields the browser may invent. We commit the reserved row before the model speaks, so a crash leaves a visible intention instead of a ghost success. We refuse an empty completion, then we store the text in the same database the API already uses for tickets.

Only after that commit do we return 200, and the body should be read back from the stored row. A local variable is just another hallway, and that hallway gets swept the moment the process exits. The client must reuse the same attempt key when it retries, or the unique constraint cannot protect the allotment. A fresh key is a fresh spend, which is fine when the user asked again and wrong when the browser merely timed out.

I generate that key on the click, keep it beside the button state, and send the same key until the receipt settles. Settled means stored or failed, not a spinner that gave up while the worker was still talking. Reserving before the call has a cost, and I still prefer that cost to a silent double spend. A crashed attempt blocks that key until the reconciler marks it failed, so the button must surface 409 instead of spinning forever.

That friction is the point, because a finite allotment is not a retry budget you can spill on the floor. If you hate the 409, fix the reconciler window, do not delete the unique key and hope the host is kind. The handler below is the proposed shape, and the provider call stays a seam you can swap.

import uuid
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel

app = FastAPI()

class SummarizeIn(BaseModel):
    ticket_id: str
    attempt_key: str

def load_ticket(actor_id: str, ticket_id: str) -> str:
    """Proposed: return persisted text this actor may read, or raise."""
    raise NotImplementedError

def call_free_model(prompt: str) -> str:
    """Proposed seam. Do not write the prompt onto the worker disk."""
    raise NotImplementedError

def mark_failed(conn, receipt_id: str, error_code: str) -> None:
    conn.execute(
        """
        update completion_receipts
        set status = 'failed', error_code = %s
        where id = %s and status = 'reserved'
        """,
        (error_code, receipt_id),
    )
    conn.commit()

@app.post("/tickets/summarize")
def summarize(body: SummarizeIn, actor_id: str, conn):
    existing = conn.execute(
        """
        select id, status, output_text
        from completion_receipts
        where actor_id = %s and ticket_id = %s and attempt_key = %s
        """,
        (actor_id, body.ticket_id, body.attempt_key),
    ).fetchone()
    if existing and existing["status"] == "stored":
        return {"receipt_id": existing["id"], "summary": existing["output_text"]}
    if existing and existing["status"] == "reserved":
        raise HTTPException(status_code=409, detail="attempt_in_progress")
    if existing and existing["status"] == "failed":
        raise HTTPException(status_code=409, detail="attempt_failed")

    receipt_id = str(uuid.uuid4())
    inserted = conn.execute(
        """
        insert into completion_receipts
          (id, actor_id, ticket_id, attempt_key, status)
        values (%s, %s, %s, %s, 'reserved')
        on conflict (actor_id, ticket_id, attempt_key)
        do nothing
        returning id
        """,
        (receipt_id, actor_id, body.ticket_id, body.attempt_key),
    ).fetchone()
    if not inserted:
        raise HTTPException(status_code=409, detail="attempt_raced")
    conn.commit()

    try:
        source = load_ticket(actor_id, body.ticket_id)
    except HTTPException:
        mark_failed(conn, receipt_id, "ticket_rejected")
        raise

    try:
        text = call_free_model(source)
    except Exception:
        mark_failed(conn, receipt_id, "provider_error")
        raise HTTPException(status_code=502, detail="provider_error")

    if not text or not text.strip():
        mark_failed(conn, receipt_id, "empty_completion")
        raise HTTPException(status_code=502, detail="empty_completion")

    conn.execute(
        """
        update completion_receipts
        set status = 'stored', output_text = %s
        where id = %s and status = 'reserved'
        """,
        (text, receipt_id),
    )
    conn.commit()
    stored = conn.execute(
        """
        select id, output_text
        from completion_receipts
        where id = %s and status = 'stored'
        """,
        (receipt_id,),
    ).fetchone()
    if not stored:
        raise HTTPException(status_code=500, detail="receipt_missing_after_write")
    return {"receipt_id": stored["id"], "summary": stored["output_text"]}

Wire the actor id from your real session dependency before you copy this, or the receipt will trust a caller-supplied owner. The same warning applies to the connection: it must point at a database you administer, not at a directory on the free worker. I left both as arguments so the control flow stays readable, and that readability is not a deployment guide.

The drill I want before anyone calls this production is a kill between the model return and the stored update. You should see a non-200 from the API and a reserved or failed row in the database you control. You should not find the only copy of the paragraph in a temp path on the worker. If curl prints 200 and the select shows no stored row, the slice is still a demo.

# Proposed drill. Point APP_DATABASE_URL at your store, not the worker disk.
curl -sS -D - -o /tmp/sum.json \
  -H 'content-type: application/json' \
  -d '{"ticket_id":"t_104","attempt_key":"a_9"}' \
  http://127.0.0.1:8000/tickets/summarize

psql "$APP_DATABASE_URL" -c \
  "select status, error_code, output_text is null as missing
   from completion_receipts
   where ticket_id = 't_104' and attempt_key = 'a_9';"

A reserved row that sits too long is a worker that died, and leaving it reserved forever blocks the same attempt key. I mark those rows failed with a worker_lost code after a short window I have not tuned in production. Two minutes in the example is a placeholder, not a measured timeout for any particular free server. If your provider is slower than that window, you will fail live calls, so measure your own latency before you copy the interval.

update completion_receipts
set status = 'failed', error_code = 'worker_lost'
where status = 'reserved'
  and created_at < now() - interval '2 minutes';

Before I would trust the slice, I want four proofs and I want them in this order. The session must reject a stranger, the reservation must exist before the provider call, and the stored row must match the body we returned. A killed worker must leave a non-success status, and a repeated attempt key must not create a second provider call. If any proof fails, I keep the feature behind the flag, even when the free seat makes the happy path feel finished.

This approach is the wrong tool when your database also lives on that same free host. You have only moved the temp file into a local Postgres directory, and a recycle still deletes the receipt. Skip it if you need a contractual quota, a latency SLO, or a promise that the free seat will still exist next quarter. I am not claiming those properties, and a free option that can change is a prototype seat rather than an architecture.

Streaming tokens into the UI before the receipt commits is a different contract, and this slice will feel too slow for that product. Do not use it as an authorization design either, because a stored paragraph does not prove the actor was allowed to read the ticket. Load the ticket through the same permission check you already trust, and keep the model seam from becoming a side door. If load_ticket is a stub that returns whatever the prompt contains, you have not built the feature, you have built a leak.

If you want a cheap seat for the worker while the receipt stays in a database you administer, look at the current free options. MonkeyCode is the seat I have in mind for that exercise, and only the limits listed on its page today should enter your plan. I already disclosed the outreach relationship above, so I will not repeat a slogan where a constraint belongs.

The handoff I trust least is still that gap between the model return and the commit. Which handoff is least stable in your stack, the session check, the provider call, or the write that should outlive the host? Send me the failure state you actually saw, including the response code, and tell me whether a receipt row existed afterward. If the code was 200 and the row was missing, we are describing the same bug, just on a different ticket.

Bind the Rewrite to One Id, Not the Box

kongkong — Sat, 10 Oct 2026 16:07:40 +0000

Bind the Rewrite to One Id, Not the Box

I can walk the late demo in my head, because the rewrite looks done and the room nods before anyone opens the database. A visitor clicks Rewrite bio, the spinner stops, and a polished paragraph appears in a toast that fades before anyone checks the row. The profile row is still the old sentence, the model log has no visitor id, and the access line only says 200. If that box disappeared before morning, which click would you replay, and which response would you trust?

I do not think the model failed, and I do not think the toast is the feature the visitor actually received. The first broken layer is the missing stamp that should have tied the click, the handler, the provider call, and the draft row together. Without that stamp, a free server is a stage with no script, and a free model is a voice with no cue sheet. Would you ship a payment form that could not name the exact charge it had just created for that visitor?

My position is blunt, and I am not interested in a softer version that leaves both doors politely open. Do not keep the rehearsal box as the source of truth, and do not let the model write the canonical profile first. Stamp one request id through the UI, the API, the provider seam, and the draft row, then prove a replay returns the stored draft. The free server is useful only while you are willing to wipe it and run the same entrypoint from the repository again.

The stamp has to cross the click

I put the stamp in the browser because the click is the only event a visitor can point at when the paragraph looks wrong. The handler inserts a pending draft with that id before it calls the model, so a crash still leaves a row you can find. The provider client must send the same id as metadata, or the model log becomes a novel you cannot join back to the click. Only after the draft is stored does the UI replace the toast with the saved text, not a hope the row never held.

The browser should mint the id once per click and show it beside the error, because a hidden header abandons the visitor. I keep that id on the button until the response is stored, so a double click reuses the stamp instead of minting another rewrite. If the fetch throws, the toast should name the status and the id, not a cheerful sentence that implies the profile changed. Can your current UI answer which click produced the paragraph, or can it only say that something pleasant happened?

async function rewriteBio(userId, currentBio, instruction, button) {
  const requestId = button.dataset.requestId || crypto.randomUUID();
  button.dataset.requestId = requestId;
  const response = await fetch("/bio/rewrites", {
    method: "POST",
    headers: {
      "content-type": "application/json",
      "x-request-id": requestId,
    },
    body: JSON.stringify({
      user_id: userId,
      current_bio: currentBio,
      instruction,
    }),
  });
  const body = await response.json();
  if (!response.ok) {
    throw new Error(`${response.status} ${body.error} ${requestId}`);
  }
  return { requestId, proposal: body.proposal, replayed: body.replayed };
}

The slice below is a proposal you can run with the Python standard library, not a benchmark I measured on a shared host. I would rather fail with 409 when the same id arrives with a different body than silently start a second provider call. A 422 means the stamp or the body was missing, and a 502 means the seam failed after the pending row existed. A 200 means the draft row was stored, which is the only success I am willing to paint for the visitor who clicked.

#!/usr/bin/env python3
"""Proposed rehearsal server. Unmeasured example: swap fake_complete before any real provider."""
import hashlib
import json
import os
import sqlite3
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

DB = os.environ.get("DRAFT_DB", "drafts.sqlite")
PORT = int(os.environ.get("PORT", "8091"))

def connect():
    conn = sqlite3.connect(DB)
    conn.execute(
        """
        create table if not exists bio_drafts (
          request_id text primary key,
          user_id text not null,
          body_hash text not null,
          status text not null,
          proposal text,
          provider_calls integer not null default 0
        )
        """
    )
    return conn

def fake_complete(text, instruction, request_id):
    # Seam only. Free model access, if you use it, belongs here and nowhere else.
    return f"{text} ({instruction}) [{request_id[-4:]}]"

class Handler(BaseHTTPRequestHandler):
    def _send(self, code, payload):
        raw = json.dumps(payload).encode()
        self.send_response(code)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(raw)))
        self.end_headers()
        self.wfile.write(raw)

    def do_POST(self):
        if self.path != "/bio/rewrites":
            return self._send(404, {"error": "not_found"})
        request_id = self.headers.get("X-Request-Id", "")
        if len(request_id) < 8:
            return self._send(422, {"error": "missing_request_id"})
        length = int(self.headers.get("Content-Length", "0"))
        try:
            payload = json.loads(self.rfile.read(length) or b"{}")
            user_id = payload["user_id"]
            current = payload["current_bio"]
            instruction = payload["instruction"]
        except (KeyError, json.JSONDecodeError, TypeError):
            return self._send(422, {"error": "bad_body", "request_id": request_id})
        digest = hashlib.sha256(
            f"{user_id}|{current}|{instruction}".encode()
        ).hexdigest()
        conn = connect()
        row = conn.execute(
            "select body_hash, status, proposal, provider_calls from bio_drafts where request_id = ?",
            (request_id,),
        ).fetchone()
        if row and row[0] != digest:
            conn.close()
            return self._send(409, {"error": "id_reused", "request_id": request_id})
        if row and row[1] == "stored":
            conn.close()
            return self._send(200, {
                "request_id": request_id,
                "status": "stored",
                "proposal": row[2],
                "provider_calls": row[3],
                "replayed": True,
            })
        if not row:
            conn.execute(
                "insert into bio_drafts(request_id, user_id, body_hash, status) values (?, ?, ?, 'pending')",
                (request_id, user_id, digest),
            )
            conn.commit()
        proposal = fake_complete(current, instruction, request_id)
        if not proposal.strip():
            conn.execute(
                "update bio_drafts set status = 'failed' where request_id = ?",
                (request_id,),
            )
            conn.commit()
            conn.close()
            return self._send(502, {"error": "empty_proposal", "request_id": request_id})
        conn.execute(
            """
            update bio_drafts
               set status = 'stored', proposal = ?, provider_calls = provider_calls + 1
             where request_id = ?
            """,
            (proposal, request_id),
        )
        conn.commit()
        calls = conn.execute(
            "select provider_calls from bio_drafts where request_id = ?",
            (request_id,),
        ).fetchone()[0]
        conn.close()
        self._send(200, {
            "request_id": request_id,
            "status": "stored",
            "proposal": proposal,
            "provider_calls": calls,
            "replayed": False,
        })

if __name__ == "__main__":
    ThreadingHTTPServer(("0.0.0.0", PORT), Handler).serve_forever()

Replay it, then throw the room away

I would start it with a sqlite file beside the script, because a memory dict vanishes when the free box recycles the process. The first curl should return provider_calls set to 1 and replayed set to false, which tells you the seam actually ran. The second curl, with the same id and the same body, should return replayed true and leave provider_calls at 1. If the second call increments the counter, that is not a replay, it is a second rewrite in the first id's coat.

DRAFT_DB=drafts.sqlite python3 rewrite_stamp.py
# second shell
curl -sS -D - http://127.0.0.1:8091/bio/rewrites \
  -H 'content-type: application/json' \
  -H 'X-Request-Id: req-1001' \
  -d '{"user_id":"u1","current_bio":"I build APIs.","instruction":"shorter"}'
curl -sS http://127.0.0.1:8091/bio/rewrites \
  -H 'content-type: application/json' \
  -H 'X-Request-Id: req-1001' \
  -d '{"user_id":"u1","current_bio":"I build APIs.","instruction":"shorter"}'
curl -sS -D - http://127.0.0.1:8091/bio/rewrites \
  -H 'content-type: application/json' \
  -H 'X-Request-Id: req-1001' \
  -d '{"user_id":"u1","current_bio":"I build APIs.","instruction":"longer"}'

I would kill the process, start it again with the same DRAFT_DB, and send the same curl before I trusted any host. The row should still be stored, the proposal text should match, and the provider counter should refuse to climb. Change only the instruction, keep the id, and you should get 409, because an id is a key, not a suggestion. That failure is the one I want in the pull request, since a silent overwrite is how a demo eats a visitor's real bio.

There is a hole I will not wallpaper over, because the pending window can still double-call if you crash after insert and before store. A retry in that window has a row, no proposal, and a temptation to call the provider again under the same id. The honest fix is to make the provider seam idempotent on that id, or to store a reservation token the seam can recognize. Until you have that, the script proves the happy replay, not the crash replay, and pretending otherwise is how vibe-coded features rot.

Think of the free server as a rented rehearsal room, not as the orchestra's archive of scores. You can play the piece there, invite a colleague to hear that same request id, and lock the door when the slot ends. You cannot leave the only copy of the score on the piano and call that a release. The repository, the sqlite contract or its real successor, and the curl that expects 200 are the score.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. Free model access belongs behind fake_complete, as a seam you can replace without teaching the route a vendor SDK. The free server option is only a place to run this same entrypoint, replay the curl, and wipe the box on purpose. I will not freeze a token quota, a model name, or a hardware shape here, because those settings move.

Read the current project page before you budget a rehearsal around them, and treat any older number you remember as untrusted. A free allowance is a budget you can exhaust, not a reason to skip the draft row or the replay check. If the project later changes what free includes, the stamp and the database contract should still boot from the repository. That is the tradeoff I care about: cost can change overnight, while an untraceable rewrite stays broken forever.

Who should walk away

This slice leaves auth out so the replay is easy to read, and that is not a suggestion to ship an open route. The session should own user_id, or a caller who guesses a stamp can store a draft against someone else's profile. I would reject a mismatch with 403 before the pending insert, so a bad caller never creates a row you must later explain. Permissions are part of the feature, not a coat you add after the free server has already impressed the room.

This approach is the wrong gate if the rewrite must become legal, billing, or medical copy without a human apply step after the draft. Skip it if your canonical store cannot upsert on request_id, because a free server will only hide that gap until the first restart. Skip it if a proxy strips X-Request-Id, since a stamp that dies at the edge never reaches the draft row you planned to query. If you need a multi-region write path this week, do not pretend one sqlite file on a free box is that path.

The stamp proves you can find the proposal, not that the proposal is good, kind, or safe to publish on a profile. A provider that drops metadata will still return text, and your join will fail even though the toast looks perfect. I have not measured latency, cost, or concurrency here, and I will not invent those numbers to make the slice sound finished. Swap in a real provider only after the replay stays at one call, or you will debug taste instead of identity.

Before I trust the slice, I want those checks in the pull request, as commands a teammate can paste without a tour. A missing id should return 422, and a changed body with the same id should return 409 rather than a fresh paragraph. A repeat should return 200 with provider_calls still at 1, including after you kill the process and start it on the same database file. If any check needs a file that exists only on the rehearsal box, the check is theater.

Which handoff is least stable in your stack: the browser stamp, the handler insert, the provider metadata, or the draft read after restart? Send the failure status you actually get, not the status you hoped the toast was hiding. I will take a 409 and a request id over a paragraph that no log can name.

Fail the Save on an Empty Completion, Not a Green Toast

kongkong — Fri, 09 Oct 2026 11:48:23 +0000

Picture the moment a teammate clicks save on a vibe-coded invoice summary and the toast turns green. The notes column is an empty string, yet every layer reported success in its own narrow way. The browser trusted the API, the API trusted the provider, and the provider returned choices with no text. Have you ever debugged that kind of quiet success, or do you still treat 200 as done?

I take a hard position on this, and I do not think the polite middle is useful. An empty completion is a failed user action, even when the free server stayed up and the model call did not throw. Shipping the green toast is how a small prototype turns into a product that lies to the person who clicked. If the text you promised to store is missing, then the save did not happen, no matter what the toast said.

The toast is not the feature

The first break is rarely the button itself, and it usually sits in the handler that maps a clean status onto a finished feature. Vibe-coded stacks love that shortcut because the happy path demos cleanly on a laptop and nobody asserts the blank payload. Then a provider returns a stop with no content, a proxy strips the body, or a restart drops the note you never wrote. Which of those three failures have you actually asserted in a test, and which one do you only remember after a user complains?

I want one contract object that every layer shares, instead of three independent booleans that all happen to be named ok. The UI may render a spinner, the API may speak JSON, and the provider may speak a schema you do not own. The user action still has one outcome, and that outcome is saved, failed, or still running. Failed must include empty text and truncated text, because anything softer invites the toast to lie.

A gate before the write

Here is the gate I drop in before the repository write, and I am labeling it as an unexecuted example. Run it inside your own application before you trust the shape, the names, or the transaction boundary. The point stays small: refuse to persist a blank summary, and return a reason the client can show without guessing. I keep the model client outside this function so a test can pass a fixture completion without spending a call.

# Unexecuted example. Adapt names to your stack before you rely on it.
from dataclasses import dataclass

@dataclass
class Completion:
    text: str
    finish_reason: str | None

def failure_reason(completion: Completion) -> str | None:
    text = (completion.text or "").strip()
    if not text:
        return "empty_completion"
    if completion.finish_reason == "length":
        return "truncated_completion"
    return None

def save_summary(repo, invoice_id: str, completion: Completion) -> dict:
    reason = failure_reason(completion)
    if reason:
        repo.mark_failed(invoice_id, reason)
        return {"status": "failed", "reason": reason}
    repo.save_notes(invoice_id, completion.text.strip())
    return {"status": "saved", "reason": None}

The architectural choice is deliberate, and the extra write on the failure path is the cost I am willing to pay. I mark the row failed inside the same boundary as the would-be notes write, so a retry cannot observe a half-saved note. I do not call the model again from that failure path, because a second call hides the first defect and can double spend. What would your handler do if mark_failed itself raised, and have you exercised that branch with a repository that is already down?

The client should read the action status, not the transport code alone, or the green toast will come back. A 200 that carries failed is honest, while a 204 with an empty body is how this bug hides in production logs. I keep the draft visible and show the reason beside the notes field, so the clicker can see that nothing was stored. If you only log the reason on the server, the person who clicked save still believes the invoice was summarized.

The browser side of that contract is just as small, and I am again labeling the sketch as unexecuted. It reads the action status from the JSON body, and it refuses to celebrate merely because fetch resolved. The fixture field belongs only in a probe build, and a production call should omit it so the guard on the server stays meaningful. If your UI framework swallows non-OK statuses before this function runs, fix that interceptor first or the toast will lie again.

// Unexecuted client sketch. Do not toast on transport success alone.
async function summarize(invoiceId, requestId, probe) {
  const payload = probe ? { fixture: "empty_completion" } : {};
  const response = await fetch(`/invoices/${invoiceId}/summarize`, {
    method: "POST",
    headers: {
      "content-type": "application/json",
      "x-request-id": requestId,
    },
    body: JSON.stringify(payload),
  });
  const body = await response.json();
  if (body.status !== "saved") {
    showDraft(body.reason || "unknown_failure");
    return;
  }
  showSaved(body);
}

I probe the running handler with a fixture flag so I can force the empty path without mocking the entire provider. The request id travels in a header so the application log and the host log can be joined later. If your route only accepts live model calls, add the fixture behind a test setting and refuse it when that setting is off. Would you rather discover a missing fixture guard in review, or after a client learns how to skip the model?

# Unexecuted probe. Expect status=failed when the fixture completion is blank.
curl -sS -X POST http://127.0.0.1:8000/invoices/inv_1/summarize \
  -H 'content-type: application/json' \
  -H 'x-request-id: demo-empty-1' \
  -d '{"fixture":"empty_completion"}' | jq '{status,reason}'

I also want a test that fails the build when someone simplifies the handler back into a blind write. This pytest sketch uses a fake repository so you can run it without a network and without a model key. It remains a proposal until you point it at your real session wrapper and watch it go red for the right reason. A test that never fails on your main branch is not protecting the contract, and it may only be protecting your mood.

# Unexecuted test sketch. Replace FakeRepo with your session wrapper.
def test_empty_completion_does_not_save_notes():
    repo = FakeRepo()
    result = save_summary(
        repo,
        "inv_1",
        Completion(text="  ", finish_reason="stop"),
    )
    assert result["status"] == "failed"
    assert result["reason"] == "empty_completion"
    assert repo.notes.get("inv_1") is None
    assert repo.failures["inv_1"] == "empty_completion"

Rehearse the empty path, then leave

Disclosure: This article was prepared as part of MonkeyCode's product outreach. Free model access and a free server option are enough to rehearse this slice on a real request path. You need a provider response and a process that can restart, not another architecture diagram on a slide. I would point the same contract test at that host, force the empty fixture, and confirm the row records failed. I would not treat that host as the system of record, and I would not assume allowance or uptime will match production.

A failure I want this gate to catch is the mismatch between provider silence and application silence in one catch block. Some clients collapse a missing text field and a network timeout together, then show saved because the catch only wraps the renderer. A timeout means you do not know, so the row should stay running or become unknown, not saved. Empty text means you do know, so the row should become failed with the reason empty_completion and no notes write.

Another failure is truncated output that still parses into an object whose text field is an empty string or a cut-off clause. The model hit a length limit, the client library returned a partial message, and your dataclass accepted it without protest. That is why finish_reason sits in the gate beside the stripped text, instead of living only in a debug log. Would you rather show a failed badge than a confident half-sentence next to a payable amount on an invoice?

What this gate will not do

There is a hard limit to this pattern, and I do not want it stretched into a general content judge. It does not score factual accuracy, toxicity, or whether the summary actually matches the invoice lines you uploaded. It only stops the save when the completion is missing or cut off, which is a narrower promise than a full review workflow. Teams that need grounding against source documents still need a separate check, a human review, or both working together.

If your product can tolerate a blank note, you do not need this gate, and adding it would only create noise for operators. Batch jobs that already store raw provider payloads for later repair can log the empty case without failing the whole file. Local scripts that never persist a user-visible field do not need a failed row, and a rehearsal host will not make them safer. Multi-tenant production should not park customer text on a free rehearsal host just because that path felt convenient that afternoon.

If I leave you one reusable check, it is a short sequence you can run without turning the article into a poster. Seed an invoice, force an empty completion, assert the notes column is untouched, and assert a failure reason was stored. Repeat the run with a truncated finish_reason, then with a normal sentence that should land in the notes column. Run it on your laptop and on the rehearsal host, with the same request id in both logs.

I am not interested in another architecture sketch that ends at the model call and calls itself a full-stack feature. The feature is the saved note, or an honest failure the user can see without opening a log aggregator. Which handoff is least stable in your app right now, the provider payload, the repository write, or the toast? Tell me the status code and the body you actually received, not the one you hoped the demo would show.

Lease the Model Call to the Actor, Not the Tab

kongkong — Thu, 08 Oct 2026 09:02:30 +0000

I want you to hold one scene before we talk about hosts, models, or how cheap a call can be. Someone clicks summarize on an open invoice, closes the laptop, and assumes the tab died with the click. A late completion still arrives from the server, and the invoice row now holds a summary that nobody confirmed. Would you defend that write just because the model call itself was inexpensive and easy to retry?

I do not think the model is the villain here, and I do not think a faster host would have saved the row. The failure sits in the handoff between a browser tab and a write that outlives the person who started it. A free model makes that handoff feel harmless, because a bad summary costs little until it lands on the wrong record. If the tab can vanish without a trace, why should its in-flight call still be allowed to commit a write?

My position is blunt: lease every model call to the signed-in actor, and let the open tab be only a view. The lease is a short row that names the actor, the action, and a deadline, and the worker may honor only that token. A completion that arrives after the lease has already expired should stay a log line, not become a mutation. A completion that names a different actor is a bug you should fail closed, not a helpful surprise.

People often reach for a background job here, and a job does solve the held-connection problem, but it does not solve identity. A job can finish faithfully for whoever created it, including a forged client id that a vibe-coded handler trusted. I want the lease checked at create time, at generation time, and again at apply time, because each hop can lie. Have you ever seen a worker trust the user id that arrived inside the JSON body from the browser?

When I want a cheap lane for this slice, I use MonkeyCode's free model access and free server option only as staging. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I treat that lane as a place to rehearse the lease, not as a promise about models, limits, or how long it lasts. The useful part is narrower: you can exercise the lease against a live model without pretending the host is your production boundary.

Picture the click as a claim check at a coat room, not as a promise that the coat will walk home by itself. The browser posts the invoice id and a purpose, and the API derives the actor from the session cookie rather than from the body. It inserts an open lease, returns the lease id, and refuses to call the model inline on that same request. The tab may poll or subscribe for status, but the tab does not own the right to write the invoice row.

Put the claim on a row

The sketch below is a proposal for a FastAPI service, and I have not executed it against a live host in this draft. Treat the SQL and the handlers as a reproducible shape you can adapt, not as a benchmark or a captured log. I keep the table boring on purpose, because a clever schema usually hides the exact check I want a reviewer to see. If the lease row cannot explain who may apply the result, the rest of the stack is only decoration.

CREATE TABLE inference_lease (
  id UUID PRIMARY KEY,
  actor_id UUID NOT NULL,
  invoice_id UUID NOT NULL,
  purpose TEXT NOT NULL,
  status TEXT NOT NULL CHECK (
    status IN ('open', 'proposed', 'applied', 'expired', 'rejected')
  ),
  deadline_at TIMESTAMPTZ NOT NULL,
  proposal JSONB,
  created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

I derive the actor inside current_actor because a body field named user id is just a rumor the client can edit. The purpose allow-list is equally deliberate, since a free-form prompt from the tab turns the lease into an open proxy. Ninety seconds is a product choice for this invoice summary, not a universal timeout, and you should pick a deadline your users can still see. If generation regularly exceeds that window, shorten the prompt or move the work, but do not silently extend the lease after the tab has gone.

from datetime import datetime, timedelta, timezone
from uuid import UUID, uuid4

from fastapi import APIRouter, Depends, HTTPException
from pydantic import BaseModel

router = APIRouter()

class LeaseIn(BaseModel):
    invoice_id: UUID
    purpose: str

class LeaseOut(BaseModel):
    lease_id: UUID
    deadline_at: datetime
    status: str

@router.post('/leases', response_model=LeaseOut)
def open_lease(body: LeaseIn, actor=Depends(current_actor), db=Depends(get_db)):
    if body.purpose not in {'summarize_invoice'}:
        raise HTTPException(status_code=422, detail='purpose_not_allowed')
    lease_id = uuid4()
    deadline = datetime.now(timezone.utc) + timedelta(seconds=90)
    db.execute(
        '''
        INSERT INTO inference_lease
          (id, actor_id, invoice_id, purpose, status, deadline_at)
        VALUES (%s, %s, %s, %s, 'open', %s)
        ''',
        (lease_id, actor.id, body.invoice_id, body.purpose, deadline),
    )
    db.commit()
    return LeaseOut(lease_id=lease_id, deadline_at=deadline, status='open')

The worker's second read is the part I would not skip, even though it looks redundant on a quiet laptop. The model call is the slow hop, and a cancel, a logout, or a deadline can land while the free server is still generating. I update to proposed only while status is open and the actor still matches, so a late packet cannot reopen a closed lease. What would your queue do if that completion arrived after the user had already signed out of the app?

def generate_for_lease(lease_id: UUID, db, model_client) -> None:
    row = db.fetch_one(
        'SELECT * FROM inference_lease WHERE id = %s FOR UPDATE',
        (lease_id,),
    )
    now = datetime.now(timezone.utc)
    if row is None or row['status'] != 'open' or row['deadline_at'] <= now:
        db.execute(
            '''
            UPDATE inference_lease
            SET status = 'expired'
            WHERE id = %s AND status = 'open'
            ''',
            (lease_id,),
        )
        db.commit()
        return
    proposal = model_client.complete(
        purpose=row['purpose'],
        invoice_id=str(row['invoice_id']),
        actor_id=str(row['actor_id']),
    )
    fresh = db.fetch_one(
        'SELECT status, deadline_at, actor_id FROM inference_lease WHERE id = %s',
        (lease_id,),
    )
    if fresh['status'] != 'open' or fresh['deadline_at'] <= datetime.now(timezone.utc):
        db.execute(
            '''
            UPDATE inference_lease
            SET status = 'expired'
            WHERE id = %s AND status = 'open'
            ''',
            (lease_id,),
        )
        db.commit()
        return
    db.execute(
        '''
        UPDATE inference_lease
        SET status = 'proposed', proposal = %s
        WHERE id = %s AND status = 'open' AND actor_id = %s
        ''',
        (proposal, lease_id, row['actor_id']),
    )
    db.commit()

Apply is a separate route on purpose, because seeing a proposal is not the same decision as writing the invoice. I return 404 when the actor does not match, so a guessed lease id does not confirm that some other row exists. I return 409 when the status is wrong or the deadline has passed, and I expire the row in the same transaction as the refusal. The invoice update also requires owner id to equal the actor, because a lease bug should not become a cross-tenant write.

@router.post('/leases/{lease_id}/apply')
def apply_lease(lease_id: UUID, actor=Depends(current_actor), db=Depends(get_db)):
    row = db.fetch_one(
        'SELECT * FROM inference_lease WHERE id = %s FOR UPDATE',
        (lease_id,),
    )
    if row is None or row['actor_id'] != actor.id:
        raise HTTPException(status_code=404, detail='lease_not_found')
    if row['status'] != 'proposed':
        raise HTTPException(status_code=409, detail='lease_' + row['status'])
    if row['deadline_at'] <= datetime.now(timezone.utc):
        db.execute(
            '''
            UPDATE inference_lease
            SET status = 'expired'
            WHERE id = %s AND status = 'proposed'
            ''',
            (lease_id,),
        )
        db.commit()
        raise HTTPException(status_code=409, detail='lease_expired')
    db.execute(
        '''
        UPDATE invoice
        SET summary = %s
        WHERE id = %s AND owner_id = %s
        ''',
        (row['proposal']['summary'], row['invoice_id'], actor.id),
    )
    db.execute(
        "UPDATE inference_lease SET status = 'applied' WHERE id = %s",
        (lease_id,),
    )
    db.commit()
    return {'status': 'applied', 'lease_id': str(lease_id)}

@router.post('/leases/{lease_id}/cancel')
def cancel_lease(lease_id: UUID, actor=Depends(current_actor), db=Depends(get_db)):
    updated = db.execute(
        '''
        UPDATE inference_lease
        SET status = 'rejected'
        WHERE id = %s AND actor_id = %s AND status IN ('open', 'proposed')
        ''',
        (lease_id, actor.id),
    )
    db.commit()
    if updated.rowcount != 1:
        raise HTTPException(status_code=409, detail='lease_not_cancelable')
    return {'status': 'rejected', 'lease_id': str(lease_id)}

The test I care about is not a happy-path screenshot of a polished summary sitting on a demo invoice. It inserts a proposed lease whose deadline is already past, posts apply, and asserts both the 409 and an untouched invoice summary. I would add a second case where the session actor differs from actor id and expect 404, because leakage loves a 403 that admits the row. Run that pair before you point any button at a live model, or you will debug identity with real completions and a noisy log.

def test_expired_lease_does_not_apply(client, db, actor):
    lease_id = uuid4()
    past = datetime.now(timezone.utc) - timedelta(seconds=5)
    db.execute(
        '''
        INSERT INTO inference_lease
          (id, actor_id, invoice_id, purpose, status, deadline_at, proposal)
        VALUES (%s, %s, %s, 'summarize_invoice', 'proposed', %s, %s)
        ''',
        (lease_id, actor.id, actor.invoice_id, past, {'summary': 'late text'}),
    )
    db.commit()
    response = client.post(f'/leases/{lease_id}/apply')
    assert response.status_code == 409
    assert response.json()['detail'] == 'lease_expired'
    invoice = db.fetch_one(
        'SELECT summary FROM invoice WHERE id = %s',
        (actor.invoice_id,),
    )
    assert invoice['summary'] is None

Prove the race before the wording

Locally I would boot the API, hit the lease route with a session cookie, and only then enqueue generate_for_lease. If curl returns a lease id and pytest stays green, the contract holds even while the model client is still a fake. Swap in the free model client only after that fake proves the race, because a live completion will distract you with wording. Did the summary sound smart to you, or did the invoice row stay unchanged when the lease was already dead?

uvicorn app:app --reload
pytest tests/test_inference_lease.py -q
curl -s -X POST http://127.0.0.1:8000/leases \
  -H 'Content-Type: application/json' \
  -H 'Cookie: session=dev-session' \
  -d '{"invoice_id":"6f1c2a40-0b0e-4a7d-9c2e-1b6d0a9e4c11","purpose":"summarize_invoice"}'

The model client is a narrow seam with one method, and the route handlers never import a vendor SDK directly. That is how a free staging model stays replaceable when you later bring your own host, without editing the invoice write. I pass purpose, invoice id, and actor id, and I do not pass the session cookie or any secret into the prompt. If the provider wants a key, it belongs in the worker environment, not in the browser bundle that opened the tab.

class ProposalClient:
    def complete(self, purpose: str, invoice_id: str, actor_id: str) -> dict:
        if purpose != 'summarize_invoice':
            raise ValueError('purpose_not_allowed')
        return {'summary': f'proposal-for-{invoice_id}', 'actor_id': actor_id}

The first version I sketched trusted the tab to send cancel and did nothing if that request never arrived. Laptops sleep and proxies drop, so a missing cancel is the common case, and the deadline has to win when the client stays silent. The second mistake was storing the proposal on the invoice immediately, which made rollback a second write with its own races. Keeping the proposal on the lease until apply means a rejected summary never touches the record a finance user is editing.

Who should not bother

This lease will not save you if the model client can call tools that mutate other systems before apply ever runs. It will not help a batch job with no signed-in actor, and a nameless service actor just recreates the open tab. I would not use this shape for a chat panel whose product is partial text, because one apply is the wrong metaphor there. If you cannot name the actor and the single row the completion may change, you are not ready for this gate.

Skip this if you are still exploring prompt wording alone and no row can change, because a lease would only slow that notebook. Skip it if your compliance story requires a human approval queue with named reviewers, since a self-apply by the same actor is not that control. I also would not point production traffic at a free server just because the lease tests passed there. A staging lane can prove the race, but your production host, secrets, and retention rules still have to be yours.

I keep coming back to the same question whenever a demo starts to feel done a little too early. Which handoff in your app is least stable today, the session-to-lease check, the post-model re-read, or the apply write? If you have a concrete failure, tell me the status code and whether the invoice changed after the tab died. A free model lane is enough to rehearse that race, and it is not a substitute for the lease that should guard the write.

Add a Boot Probe Behind a Deploy Contract, Not a Free Host

kongkong — Thu, 08 Oct 2026 06:19:11 +0000

I want to walk you through a Friday failure I keep arguing about, and it never starts inside the model call itself. Someone has a generated app that looks finished on a laptop, then a fresh host answers with a blank 502. The page had already talked to a model, saved nothing durable, and assumed a dotenv file would appear by magic. Does that sound like a model problem to you, or like a boot problem wearing a model costume?

I take a hard position on this handoff, and I am not interested in a soft maybe. A free host is a staging bench until a deploy contract passes, and a passing laptop demo is not that contract. If your first live check is a human refreshing the homepage, you already skipped the only test that matters. Why would you trust a process you have not asked to name its own missing pieces out loud?

Think of the free server as a borrowed workbench in a shared shop, not as a house you already own. You can set your tools down and make a cut, but the clamps, the power drop, and the lock were never implied. A generated Dockerfile is the tool roll, while the contract is the agreement about power, identity, and what happens when the shop closes. People paste the tool roll onto the bench and then act surprised when the lights cut mid-cut.

The contract I want is small enough to fail in one request, and boring enough that a reviewer can read it aloud. It names required environment keys without printing their values, and it refuses to continue when one name is absent. It checks that the migration head matches the revision the running code expects, because a fresh disk will not forgive a guessed schema. It also proves the process can answer a ready route before any product router accepts a user request.

People have been arguing lately that a generated interface is enough to show your work, and I think that claim skips the boot. The generator optimized for a happy path on one machine, so it hid configuration inside a local file and a long-lived dev server. A free host starts cold, drops your process when idle, and will not smuggle your laptop secrets into the environment for you. Have you ever watched an app demand a database URL in production while the only copy lived in a gitignored file on one laptop?

The failure then climbs the stack in a way that fools the person who is watching the browser. The UI treats every slow response as the model thinking, so a 502 becomes a spinner and a 503 becomes a retry toast. Meanwhile the API never recorded who owned the boot, and the database never confirmed that this build knows this schema. I would rather show a dull not-ready page with a code than a chat box on a process that cannot name its database.

So I would not point any traffic at the host until a boot probe owns that boot. The module below is a proposal you can run yourself, and I am not claiming I timed it on any particular machine. It writes a single decision in memory, returns a stable error code, and leaves the product routes closed when the contract is red. That result is less glamorous than another chat wrapper, and that dullness is the point.

# Unexecuted example: a boot contract, not a measured production trace.
from __future__ import annotations

import os
from dataclasses import dataclass

REQUIRED_KEYS = ('DATABASE_URL', 'SESSION_SECRET', 'MIGRATION_HEAD')

@dataclass(frozen=True)
class ProbeResult:
    ok: bool
    code: str
    missing: tuple[str, ...]
    migration_head: str | None

def read_contract(expected_head: str) -> ProbeResult:
    missing = tuple(key for key in REQUIRED_KEYS if not os.environ.get(key))
    if missing:
        return ProbeResult(False, 'config_missing', missing, None)
    actual = os.environ['MIGRATION_HEAD']
    if actual != expected_head:
        return ProbeResult(False, 'migration_mismatch', (), actual)
    return ProbeResult(True, 'ready', (), actual)

I keep the product surface behind that result on purpose, because a green function nobody calls is just a comment. The ready route is the only public door until the code is ready, and every other route can stay unmounted. If you mount the chat route first, you have already decided that a missing secret is a user-facing incident. Would you let a customer hit checkout while the cash drawer is still in a box on the sidewalk?

# Unexecuted example: product routes stay unmounted until the probe is green.
from fastapi import FastAPI, HTTPException

app = FastAPI()
EXPECTED_HEAD = '20261008_01'

@app.get('/ready')
def ready() -> dict:
    result = read_contract(EXPECTED_HEAD)
    if not result.ok:
        # 503 tells the host and the UI this build is up but not eligible.
        raise HTTPException(
            status_code=503,
            detail={'code': result.code, 'missing': list(result.missing)},
        )
    return {'code': result.code, 'migration_head': result.migration_head}

The matching test is the part I actually want you to keep, because a contract without a failing test is a wish. This one deletes a key, expects config_missing, and never asserts on a secret value you might later leak. You can add the mismatch case in the same style, and you should, before you argue about which model to call. A test that only covers the happy path will smile at you on the laptop and then vanish on the fresh host.

# Unexecuted example. Requires pytest; monkeypatch is the built-in fixture.
def test_missing_config_stays_dark(monkeypatch) -> None:
    monkeypatch.delenv('DATABASE_URL', raising=False)
    monkeypatch.setenv('SESSION_SECRET', 'dev-only')
    monkeypatch.setenv('MIGRATION_HEAD', '20261008_01')
    result = read_contract('20261008_01')
    assert result.ok is False
    assert result.code == 'config_missing'
    assert 'DATABASE_URL' in result.missing

# Unexecuted local dry run. Do not commit these placeholder values.
# Terminal A:
# uvicorn app:app --host 127.0.0.1 --port 8000
# Terminal B:
export DATABASE_URL='postgres://app:local@127.0.0.1:5432/app'
export SESSION_SECRET='dev-only-not-a-real-secret'
export MIGRATION_HEAD='20261008_01'
python -m pytest tests/test_boot_probe.py -q
curl -sS -D - http://127.0.0.1:8000/ready

Read the status line before you interpret the prose in the body, because the code is the contract. A 503 with code config_missing means the process booted and the gate held, which is a success even though the app is not live. A connection reset or a platform 502 means your code never ran, so stop blaming the model client and inspect the start command. A 200 with code ready means you may mount the next route, and only that next route, after a person reads the body.

I also want the model client behind a seam that this probe does not import at all. The boot check should pass on a machine with no model credential, because host configuration is a different job from prompt completion. If your ready route opens a socket to a model provider, you have coupled two failure domains and you will page the wrong person. Can you explain, in one sentence, which credential the probe is allowed to touch, and why that list is so short?

This is where a free model path and a free server option actually earn a place in the workflow. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project that, as described for this article, offers free model access and a free server option. I will not print a token count, a hardware shape, or a duration, because those figures move and go stale.

Read the current project docs before you design a budget around any allowance, including a free token figure an outreach note mentioned. My opinion on that free capacity is narrow, and it is not a slogan about shipping faster than your review. Use it as a cheap bench where a red probe is affordable, then exercise one read path that cannot mutate customer state. A free server will not invent your migration head, and a free model call will not rotate a session secret for you.

If the docs show a token allowance, treat it as a ceiling you re-check, not as proof of isolation. A ceiling is not a deed, and it does not make the host isolated, retained, or yours to keep. There is a plan step and an apply step, and I would not collapse them because a button labeled deploy feels nicer. Plan is the probe on the free host, with product routes unmounted, and the only artifact is the status code plus the code string.

Apply is a second change that mounts one route after a human reads that result and agrees the host may take a request. If a generated script both boots the host and flips traffic, you have hidden the apply inside a tool you did not review. Is the person who clicked deploy the same person who can read a 503 body without asking a model to translate it? Who should walk away from this approach before they copy the probe into a repository they do not own?

If you already run a reviewed cluster with secret injection and a migration gate, keep the probe and leave the free host alone. If the workload holds customer data, payment details, or private prompts, do not park it on a free server you have not reviewed. An unreviewed free server is not a softer kind of production, and I will not pretend the logs are private. If you want a one-command portfolio that never shows a failure code, this probe will feel obstinate on purpose.

A static page is the honest alternative when you do not want a status code in the review. If your work cannot tolerate a cold start, do not pretend a free server is a worker fleet. Would you rather explain a red probe on Monday, or explain a vanished job that never had an owner? The example has limits I do not want you to sand off in the name of a cleaner demo.

It checks that key names exist and that a head string matches, but it does not prove a role can migrate. A private network path is a separate review, and this probe will not vouch for that path at all. A green ready can still die on the first real query, so the next test should be one read against a scratch schema with writes refused. The snippets are unexecuted proposals, so adapt the key names, the framework, and the head format before you trust them in a review.

I am not offering a benchmark, a hardware note, or a claim that any host will keep a process warm for you. Before I call a host eligible, I want the pull request to answer the gate in sentences a reviewer can challenge. The required key names are listed, values are absent, and the expected migration head is pinned to this exact build. The ready route returns 503 with config_missing or migration_mismatch when someone breaks it on purpose in the test log.

Product routes stay unmounted until that probe is green, and the model client is not imported by the probe module. A human records the status code from the free host before any traffic flip, and customer data stays off that host. A separate review has to say the host is acceptable before any private prompt is stored, replayed, or sent to a model. If those sentences are missing, the deploy is not done, no matter how polished the generated interface looks in a screenshot.

If you remember one argument, remember that a free host is a cheap place to fail the contract, not a pardon for skipping it. Which handoff is least stable in your app right now, the missing environment name, the migration head, or the ready route? Reply with the status code and the first line of the body, and leave every secret value out of the comment. If you want a low-cost bench for that probe, check the current MonkeyCode docs, then come back with the failure you actually hit.

Pin the Schema Before a Free Model Can Write

kongkong — Thu, 24 Sep 2026 18:00:57 +0000

I keep walking a small summarizer from the browser into a real save, and the pretty paragraph is never the part that breaks. Someone pastes a customer note, the completion looks clean, and the save returns a 500 before any row changes. The handler is doing a naked JSON parse on a string that starts with a markdown fence. So did the model fail, or did we skip the contract that should have rejected that string?

A popular thread this week is really about whether short functions are enough to make behavior obvious. I buy the worry, but the summarizer handler was already short, and the save behavior was still opaque. Nobody could point to a rule for which completions were allowed to become a saved note. So is the fix another line of commentary above the parse, or a contract the test can fail?

I think the free model is the wrong thing to blame first, and the wrong thing to trust next. A cheaper completion is still an untrusted upstream, closer to a flaky webhook than to a function in your repo. If the shape can drift, the write path should fail closed before a row is touched. Would you let an unmarked HTTP client insert into orders just because the demo looked fluent?

Where the save actually breaks

The architectural choice I want is a versioned schema between the provider response and the database, owned by the API. The UI may render a preview, but that preview is not authority, and a second client can always skip it. I tried the other way in a prototype, repairing fences in the browser so the demo kept moving. That repair hid the drift, a scripted client posted the raw string, and the note column filled with backticks.

Here is the contract I would pin before any model swap, including a move onto free inference. The example below is a proposed fixture, not a trace from a production incident, and you should run it yourself. I am not claiming a benchmark, a latency number, or a saved dollar from this snippet. If the bounds feel strict, that is the opinion, because a loose parser is how drift becomes data.

import json
from pydantic import BaseModel, Field, ValidationError

FENCE = "`" * 3

class SummaryV1(BaseModel):
    schema_version: str = Field(pattern=r"^1$")
    summary: str = Field(min_length=1, max_length=500)
    confidence: float = Field(ge=0.0, le=1.0)

def parse_completion(raw: str) -> SummaryV1:
    text = raw.strip()
    if text.startswith(FENCE):
        raise ValueError("fenced_markdown")
    try:
        data = json.loads(text)
    except json.JSONDecodeError as exc:
        raise ValueError("invalid_json") from exc
    return SummaryV1.model_validate(data)

The fence check is deliberate, because a helpful model often wraps an object and still sounds correct to a reader. I would rather return 422 with a stable error code than spend another prompt trying to shave the backticks. Extra keys can be ignored by configuration, missing keys should reject, and a truncated object should never become a null summary. Have you ever watched a UI show saved while the column stored an empty string that passed a truthiness check?

Pin a version, then fail closed

The route then stays boring, which is the whole point of putting the contract in one place. It loads the note, calls the provider behind one seam, and persists only after the schema accepts the body. A rejected completion is stored as a failed attempt with the schema version, not as a successful note. If you skip that attempt record, the next on-call engineer sees only a 500 and a prompt that already vanished.

def save_summary(note: str, raw_completion: str, db) -> dict:
    try:
        parsed = parse_completion(raw_completion)
    except (ValueError, ValidationError) as exc:
        db.attempts.insert({
            "schema_version": "1",
            "status": "rejected",
            "error": str(exc),
        })
        return {"status": 422, "error": "contract_rejected"}
    db.notes.update_summary(note_id=note, summary=parsed.summary)
    return {"status": 200, "schema_version": parsed.schema_version}

That insert is pseudocode for the seam, not a claim that your ORM or your tables look like this. The idea travels to SQL or a document store, because rejection is data and success is a separate write. I want 422 when the shape is wrong, and 502 only when the provider failed before a body existed. Mixing those codes sends people into the wrong layer, which is how a parser bug gets filed as an outage.

You can prove the gate without a network call, and I would not skip that step to save an afternoon. A fixture with a fenced string should fail, a valid object should pass, and an empty summary should fail the length bound. I keep those three cases beside the route so a model change cannot land as a silent prompt edit. Why would you trust a live swap if the repo cannot already reject the last bad body you saw?

python -m pytest tests/test_summary_contract.py -q

def test_fenced_markdown_is_rejected():
    fence = "`" * 3
    payload = '{"schema_version":"1","summary":"ok","confidence":0.5}'
    raw = fence + "json\n" + payload + "\n" + fence
    try:
        parse_completion(raw)
    except ValueError as exc:
        assert str(exc) == "fenced_markdown"
    else:
        raise AssertionError("fence should fail closed")

Prove it locally before the free endpoint

Only after that local gate is green do I point the same parser at a live free endpoint. The credential stays in the server environment, and the browser never learns the provider base URL. A demo page that calls the model directly has already skipped the schema and exposed the key path. I would rather see a 401 from my own API than a fluent completion that never passed validation.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The brief I was given describes MonkeyCode as an open-source project with free model access and a free server option. That pair is useful as a second environment for the same fixture, not as permission to persist. I am not stating a token allowance, a hardware size, a duration, or a model name here.

Those terms move quickly, and I have not re-checked a primary page before writing this draft. Read the current project terms yourself, and treat any number remembered from chat as stale until you do. If the free option is gone or narrowed, the local fixture still stands, which is why it lives in the repo. Would you drop the schema pin just because a posted allowance changed between two weekly deploys?

The browser sends the note to your API, your API holds the credential, and the free server is another base URL behind that seam. A session header on that curl is a stand-in, not a real token, and you should use your own auth cookie. I log the provider status beside the schema version so a 429 is not mistaken for a bad summary. If the body is prose apologizing for the format, quarantine it and stop, because a retry will not grow a schema.

curl -sS -X POST http://127.0.0.1:8000/summaries \
  -H 'content-type: application/json' \
  -H 'authorization: Bearer dev-session' \
  -d '{"note":"Customer asked to move delivery to Friday."}'

Point the server at the free option with environment variables, and keep the secret out of the repo. The host below is a placeholder, not a documented endpoint, so replace it only after you read the current project docs. I refuse to paste a live host or a sample key into a post that will be copied into terminals.

export PROVIDER_BASE_URL="https://example.invalid/free-server"
export PROVIDER_API_KEY="from-your-secret-store"

What usually fails next is rarely the tidy JSON object you practiced against while writing the prompt. A free pool can return 429, an empty content field, or a 200 whose body is an apology. The fixture should record the status and the first hundred characters, then stop without a blind retry. Retrying a 200 that failed the schema only burns the allowance and teaches the handler to be generous.

Limits I will not talk past

There is a maintainability tradeoff I accept on purpose, and it will annoy anyone chasing a magical demo. A strict schema will reject some completions a human would have repaired, so the first week looks less fluent. That cost is smaller than a column full of fences, and cheaper than a migration after three clients stored three shapes. Performance stays predictable because you parse a small object and you do not hold a transaction open across the provider call.

When version two arrives, I would add a second model and backfill stored notes, not reinterpret old JSON. Old records keep their schema version, so a reader can see which contract produced the summary. A flag in the component that swaps prompts without a migration is how you get two meanings in one column. Have you already found a text field that means different things depending on which week it was written?

This approach is a poor fit when the note holds secrets or regulated data you cannot send to an external host. It is a poor fit if you need a latency guarantee, because free capacity is not an SLA I will pretend to own. Teams that want the model to invent new fields every week should not pin version one, or they will fight their own gate. If the request has no authenticated user, do not add this write, since a public summarizer becomes an open proxy.

Before I call the change done, the fence fixture has to fail and then pass on a valid body. The empty summary has to be rejected, and a 422 has to stay distinct from a provider 502. The provider key has to be absent from the browser bundle, which you can check with a simple search. That is the reusable gate, and it does not depend on which model happened to answer the note.

I would ship that gate before I ever allow a free completion to update a customer note. The free server is only a second environment for the same test, not a reason to widen what you persist. Call, then validate, then write, in that order, even when the endpoint costs you nothing today. Which status came back first on your last bad save, a 422 from the gate or a 500 from a naked parse?

Add Model Spend Behind a Budget Row, Not a Browser Agent

kongkong — Wed, 23 Sep 2026 15:58:30 +0000

The support console fired a rewrite request at 4:11 p.m., and the widget treated a model response like a database grant. The browser held a provider key in local storage, posted a completion, then PATCHed the invoice with whatever paragraph came back. Did anyone ask whether that tenant still had inference budget for ticket rewrites this month? Did a second tab retry the same click and persist a second, slightly different version of the legal text?

I think the fashionable answer this week is wrong, and I will not hedge it. An agent that never leaves the browser is a demo of latency, not a delivery path for customer writes. The first layer that fails is not the model, because the model did return usable prose. The missing budget row should have refused the write, and nobody on the path asked it.

Free tokens are a subsidy, not a permission

Free inference is a prototype subsidy, and it is not permission to mutate invoices, comments, or stored schema. If your UI can call a model and your API can persist the result, you already gave write authority to a meter you do not own. Why do we keep wiring it the other way around, week after week, in otherwise serious codebases? The happy path looks cheap until a retry doubles spend and the bad paragraph is already in Postgres.

Can you roll that write back from a chat transcript after the toast has already gone green? I cannot, and I do not believe a streaming DOM node will save you either. The contract I want is boring on purpose, which is usually a sign it will survive contact with production. Every model call that can affect stored state must present a tenant, a purpose, and a budget row. If the row cannot cover the call, the handler returns 402 and nothing is written.

Follow the click until the handoff dies

Walk the request with me as a vertical slice, not as an architecture poster on a wiki. A human clicks rewrite, and the browser should post a command with ticket_id, intent, and an idempotency key. That client should not hold a provider key, and it should not pick a model name from a dropdown somebody added during a hack day. Where does this fail first when the spend envelope is not a real row?

It fails right after the model returns text, because nothing in the path is allowed to say no. Auth said the user could edit the ticket, and the model said here is a paragraph, so Postgres returned 200. The budget said nothing at all, because it was never asked as a locked row. That is not an artificial intelligence problem hiding in a vendor SDK. That is a missing constraint at the only layer that can refuse a write.

The budget row as a pasteable slice

I am labeling the next blocks as a reference implementation you can run locally, not as a souvenir from a named cluster. You should keep that label in your own repo until the 402 path is proven with a real status code. The unique constraint on the idempotency key is the whole point of the second table. Network retries must not mint a second spend event and a second invoice rewrite.

CREATE TABLE inference_budgets (
  id            uuid PRIMARY KEY,
  tenant_id     uuid NOT NULL,
  purpose       text NOT NULL,
  period_start  timestamptz NOT NULL,
  period_end    timestamptz NOT NULL,
  token_limit   integer NOT NULL,
  token_spent   integer NOT NULL DEFAULT 0,
  UNIQUE (tenant_id, purpose, period_start)
);

CREATE TABLE inference_spend_events (
  id               uuid PRIMARY KEY,
  budget_id        uuid NOT NULL REFERENCES inference_budgets(id),
  idempotency_key  text NOT NULL UNIQUE,
  provider         text NOT NULL,
  input_tokens     integer NOT NULL,
  output_tokens    integer NOT NULL,
  status           text NOT NULL,
  created_at       timestamptz NOT NULL DEFAULT now()
);

Have you watched a mobile client replay a POST because the toast never rendered on a flaky radio? The second write is almost always the one that lands in support later. A FastAPI handler can refuse the business write when a locked budget row cannot cover a conservative estimate. This sketch estimates input tokens plus a hard output cap, then reconciles after the provider responds, which is the actual seam.

# Reference implementation: prove the 402 locally. Not production gospel.
from fastapi import APIRouter, Depends, HTTPException
from pydantic import BaseModel, Field

router = APIRouter()
ESTIMATED_OUTPUT_CAP = 800

class RewriteCommand(BaseModel):
    ticket_id: str
    intent: str = Field(pattern="^rewrite_friendly$")
    idempotency_key: str
    tenant_id: str

@router.post("/tickets/{ticket_id}/rewrite")
def rewrite_ticket(
    ticket_id: str,
    cmd: RewriteCommand,
    db=Depends(get_db),
    models=Depends(get_model_client),
):
    if cmd.ticket_id != ticket_id:
        raise HTTPException(409, "ticket_id mismatch")

    existing = db.fetch_spend_event(cmd.idempotency_key)
    if existing:
        return {"status": existing.status, "spend_id": existing.id, "replayed": True}

    budget = db.lock_budget(cmd.tenant_id, purpose="ticket_rewrite")
    if budget is None:
        raise HTTPException(403, "no budget row for purpose")

    estimate = estimate_input_tokens(ticket_id) + ESTIMATED_OUTPUT_CAP
    if budget.token_spent + estimate > budget.token_limit:
        raise HTTPException(402, "inference budget exhausted")

    reservation_id = db.reserve_tokens(budget.id, estimate, cmd.idempotency_key)
    try:
        result = models.complete(
            purpose="ticket_rewrite",
            prompt=load_ticket_prompt(ticket_id),
            max_tokens=ESTIMATED_OUTPUT_CAP,
        )
    except ProviderError as exc:
        db.release_reservation(reservation_id, reason=str(exc))
        raise HTTPException(502, "provider failed; budget released") from exc

    db.finalize_spend(
        reservation_id,
        provider=result.provider,
        input_tokens=result.input_tokens,
        output_tokens=result.output_tokens,
    )
    db.apply_ticket_rewrite(ticket_id, result.text, spend_id=reservation_id)
    return {"status": "applied", "spend_id": reservation_id}

Notice what the handler refuses to do, even when the process is running on a scratch box you stood up for a demo. It does not accept a model name from the client, and it does not stream tokens into a websocket that also applies SQL. The purpose string is the product contract, not a log field you add later for analytics. Why would playground_chat and ticket_rewrite share one pool when they do not share one blast radius?

Prove the 402 on a scratch server

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A scratch environment is the right place to teach the ledger to say no, because that experiment should not ride a paid provider bill. MonkeyCode's free model access and free server option can host that slice while you are still proving the refusal path. Treat the open-source project as a sandbox for the contract, not as a substitute for the contract. If you cannot get a 402 from the free box, you will not get a real refusal in production either.

python -m venv .venv
source .venv/bin/activate
pip install fastapi uvicorn "psycopg[binary]"
uvicorn app:app --reload --port 8080

psql "$DATABASE_URL" -c "insert into inference_budgets
  (id, tenant_id, purpose, period_start, period_end, token_limit)
  values ('00000000-0000-0000-0000-000000000001',
          '11111111-1111-1111-1111-111111111111',
          'ticket_rewrite', now(), now() + interval '30 days', 200);"

curl -sS -X POST "$API/tickets/T-42/rewrite" \
  -H "content-type: application/json" \
  -H "authorization: Bearer $DEV_TOKEN" \
  -d '{"ticket_id":"T-42","intent":"rewrite_friendly",
       "idempotency_key":"rewrite-T-42-01",
       "tenant_id":"11111111-1111-1111-1111-111111111111"}'

The first call may apply if the estimate still fits inside that tiny envelope. The second call, with a new idempotency key, should return 402 and leave the ticket body untouched. Replaying the same key should return the original status and still not double-write the invoice. If your status codes do not split that way, the seam is theater, not a contract.

I also want a test that the browser never saw a provider key in headers or payloads. That is the difference between a tab-resident agent and a feature that survives a stolen laptop. The test is ugly on purpose, because pretty tests tend to skip the overdraw and only assert that JSON looks friendly.

def test_rewrite_does_not_overdraw_or_write(client, db):
    db.set_token_limit(tenant="t1", purpose="ticket_rewrite", limit=50)
    before = db.ticket_body("T-42")
    first = client.post("/tickets/T-42/rewrite", json=command("T-42", "k1"))
    assert first.status_code in {200, 402}
    second = client.post("/tickets/T-42/rewrite", json=command("T-42", "k2"))
    assert second.status_code == 402
    if first.status_code == 402:
        assert db.ticket_body("T-42") == before
    assert "sk-" not in str(second.headers)
    replay = client.post("/tickets/T-42/rewrite", json=command("T-42", "k1"))
    assert replay.json().get("replayed") in {True, None} or replay.status_code == 402

Is that test going to impress a design critique on a clean-code thread? No, and I do not need it to. Spend is a row, writes are a later row, and the model is a vendor behind a purpose. The browser is a client, not a wallet, and it should not get a vote on either row.

The failures that show up when you skip the row

The failure mode looks like this, and it is more common than a clever browser agent demo. A handler temporarily reads max_tokens from the client, and the client sends a larger cap after the first 402. A streaming UI applies a partial sentence when the socket dies at eighty percent, then applies again on resume. A free-server compose file lands in production with the budget check commented out because it made the pitch deck jitter.

The model layer looks healthy, and the persist layer looks healthy, so the dashboard stays green. The handoff is an if statement somebody deleted on a Friday so the demo would not blink. Does that sound like a model-quality incident to you? It is a permissions incident with extra tokens attached.

Performance testing of the API does not save you if every scenario still has tokens left in the envelope. A load test that never exhausts budget is a load test of the happy path, which you already knew worked. Can you show me the 402 p95, and can you show that ticket bodies did not change on those requests? If you cannot, you have not tested the contract you claim to have.

Who should not copy this

If your product is a local playground with no persistence, a budget row is ceremony, and you should not add it for the aesthetic. If you cannot name a tenant, a purpose, and a period, you are not ready to attach a ledger. A free model will only hide that gap until the first shared demo key leaks into a write path. Also skip this sketch if you need hard multi-region accounting today, because it assumes one database and a row lock.

That starting contract is not a billing platform, and I will not pretend otherwise. People who only need a read-only eval slice should stay on the read path until a human grants write authority. Do not use this article as cover for giving a chat widget schema rights. The whole point is the opposite move: prove refusal before you celebrate the prose.

Prove the click, the 402, the replay, and the absent provider key before you ship the rewrite. Keep model names on the server, keep purposes distinct, and keep the free server in the eval path until the ledger refuses correctly. Which layer handoff is least stable in your stack right now, budget to model or model to persist? Send a failure state or a response code, not a model name, because I care whether the write happened after the budget already said no.