DEV Community: kongkong

Add Model Calls Behind a Provider Contract, Not Across Your Codebase

kongkong — Tue, 08 Sep 2026 03:23:05 +0000

When a teammate clicks Generate follow-up and save, the browser fires one POST with a ticket id and a tone. The API is supposed to ask a model for a comment plan, preview it, and persist only after someone confirms. What actually dies first is not the model. It is the route that constructed a vendor client, awaited a completion, and wrote raw text into Postgres before the UI could render a preview.

I do not treat that crash as a model-quality problem. I treat it as a missing contract, and I would refuse to ship the feature in that shape. Cheap inference makes the wrong design feel productive, because every layer can “just call the model” in a single afternoon. Have you watched provider field names leak into React types, OpenAPI schemas, and even audit logs? That leak is the first layer that breaks when the host, the envelope, or the auth story changes.

My position is not gentle. A free model is not an architecture. If a feature can persist state, the model must sit behind a provider contract you can stub, swap, and fail closed. Anything else is a prototype with extra steps, and extra steps are how vibe-coded write paths reach production.

The request should stay a plan until apply is boring

Picture the end-to-end action again. A support lead wants a follow-up comment drafted from a ticket, then stored against that ticket with an actor and an idempotency key. The UI should post a plan request, receive a plan, and only later post an apply with the human’s approval. If your first handler both samples text and inserts a row, you have already given the model write authority it did not earn.

I keep using a ticket comment because it looks harmless. It is a single user action, it touches auth, it touches storage, and it is exactly the kind of slice teams bolt onto a weekend demo. The failure mode is familiar: the model returns a slightly different JSON shape, the UI still paints a paragraph, and the apply path writes a comment nobody can later attribute. Who owns that row when the provider was inlined in the route?

Think of the model like a shipping carrier. You do not let every warehouse printer speak UPS XML. You speak your own bill of lading, and the adapter translates. When the carrier changes, the warehouse does not reprint its building. Why do we let chat SDKs become the building?

What I would put in the contract

The contract is deliberately smaller than a vendor SDK. It names a purpose, a digestable prompt, and a plan the apply path can reject. This is a proposed TypeScript seam, not a claim that I ran it in a particular production fleet.

// provider-contract.ts — proposed seam
export type Purpose = "ticket_followup";

export type PlanRequest = {
  purpose: Purpose;
  ticketId: string;
  tone: "neutral" | "firm";
  actorId: string;
  maxTokens: number;
};

export type Plan = {
  providerAlias: string; // your name, not a vendor model id
  promptDigest: string;  // hash of the exact prompt bytes
  body: string;
  citations: string[];   // ticket ids or doc ids, never vendor blobs
};

export interface ModelProvider {
  plan(req: PlanRequest): Promise<Plan>;
}

Notice what is missing. There is no streaming callback, no vendor message array, no “system prompt” string leaking into the database. The apply handler should not know whether the bytes came from a stub, a paid API, or a rehearsal host. If you need those vendor fields in the UI, the contract is already broken. Why would a React component care which company sampled the tokens?

A stub provider makes the rest of the app testable on a laptop.

// stub-provider.ts
import { createHash } from "node:crypto";
import type { ModelProvider, Plan, PlanRequest } from "./provider-contract";

export class StubProvider implements ModelProvider {
  async plan(req: PlanRequest): Promise<Plan> {
    const body = `Follow-up for ${req.ticketId} in a ${req.tone} tone.`;
    const promptDigest = createHash("sha256")
      .update(`${req.purpose}:${req.ticketId}:${req.tone}`)
      .digest("hex")
      .slice(0, 16);
    return {
      providerAlias: "stub",
      promptDigest,
      body,
      citations: [req.ticketId],
    };
  }
}

Wire a live provider the same way. Map your PlanRequest onto whatever HTTP envelope you are rehearsing, then map the response back onto Plan. If mapping needs a comment, the envelope does not belong in the route. I would rather maintain one awkward adapter than three clever call sites.

Plan and apply as two permissions

The API should expose two verbs with two permission checks. POST /comments/plan may spend tokens. POST /comments/apply may spend a database row. Mixing them is how a retry creates two comments and one confused customer.

// routes.ts — proposed handlers
app.post("/comments/plan", requireSession, async (req, res) => {
  const body = PlanRequestSchema.parse(req.body);
  if (body.actorId !== req.session.userId) {
    return res.status(403).json({ error: "actor_mismatch" });
  }
  const plan = await req.provider.plan(body);
  // persist the plan as a row with status=proposed, not as a comment
  const id = await plans.insert({ ...plan, ticketId: body.ticketId, actorId: body.actorId });
  return res.status(201).json({ planId: id, plan });
});

app.post("/comments/apply", requireSession, async (req, res) => {
  const { planId, idempotencyKey } = ApplySchema.parse(req.body);
  const plan = await plans.get(planId);
  if (!plan || plan.actorId !== req.session.userId) {
    return res.status(409).json({ error: "plan_not_applicable" });
  }
  const comment = await comments.insertOnce({
    ticketId: plan.ticketId,
    body: plan.body,
    promptDigest: plan.promptDigest,
    providerAlias: plan.providerAlias,
    idempotencyKey,
  });
  return res.status(201).json({ commentId: comment.id });
});

409 plan_not_applicable is a feature, not an eyesore. I want the UI to render that code, not a generic toast. If apply is a second click, you can put a human in the gap without pretending the model was “safe.” Would you let a payment provider both quote and capture in the same unauthenticated function? Then why is a comment different?

A rehearsal on a disposable host is useful here. If you need free model access and a free server option to exercise the live adapter without baking a vendor into staging, MonkeyCode is an open-source project that offers both. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Use that kind of host to prove the mapping, then point the same ModelProvider interface at whatever you actually pay for. Do not let the rehearsal hostname appear in your apply table.

Fail the handoff on purpose

I would not trust the seam until I can break it from the outside. The following is a proposed test plan, not a benchmark I am claiming to have published.

# 1) Stub plan should never write a comment.
curl -sS -X POST "$API/comments/plan" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"purpose":"ticket_followup","ticketId":"T-1042","tone":"firm","actorId":"u1","maxTokens":256}'
# expect 201, body.plan.providerAlias == "stub", no comments row

# 2) Apply with a foreign actor should die before INSERT.
curl -sS -X POST "$API/comments/apply" \
  -H "Authorization: Bearer $OTHER" \
  -d '{"planId":"'$PLAN'","idempotencyKey":"k1"}'
# expect 409 plan_not_applicable

# 3) Swap the provider alias in process env and rerun plan.
# expect the apply schema to be unchanged; only providerAlias differs

# 4) Feed apply a body that includes a vendor-only field.
# expect 400 from your parser, not a new column in comments

The interesting failure is step four. If your parser is “helpful” and strips unknown keys, the UI will keep sending them and you will not notice until a migration. Fail closed. A comment table that grows a raw_choices JSONB column is a shrine to the last SDK you copied.

Cost and maintainability follow the same line. Tokens can be free on a rehearsal host and still be expensive in engineering time when every screen knows the provider. I would rather spend a quiet hour on promptDigest and insertOnce than spend a noisy week grepping for a renamed content field. Is the digest overkill for a comment? Only if you never have to explain who wrote it.

Who should not take this advice

This slice is for features that persist, authorize, and retry. If you are sketching a throwaway notebook, inlining a client is fine, and a contract will only slow you down. If you are building a streaming chat surface where the model is the product, a plan/apply split may be the wrong grain; you still want an adapter, but the user-visible verb is the stream, not a row.

Do not use a free rehearsal server as the identity of production apply. Do not store provider hostnames as foreign keys. Do not claim “full stack” if the plan route has no actor check or the apply route has no idempotency key. And do not treat this article as a promise about any vendor’s quota, hardware, or uptime. Those numbers go stale, and stale numbers are how people copy architecture from a blog post instead of from their own failure codes.

A short path you can reuse

Stand up the contract and the stub first, with no network. Add POST /comments/plan and assert that storage still has zero comments. Add POST /comments/apply with session checks and insertOnce. Point the same interface at a live adapter on a host you can delete. Replay the four curls above until the only thing that changes across providers is providerAlias and promptDigest. Then, and only then, let the UI grow a confirm button.

I care less about which model sampled the paragraph than about which layer was allowed to write. If your apply path cannot reject a plan with a boring status code, you do not have an AI feature. You have a completion call with side effects.

Which handoff is least stable in your app right now—the UI to plan, the plan to apply, or apply to storage? Reply with a status code or a response body, not a feeling.

Rehearse Agent Migrations on a Host You Can Delete

kongkong — Sun, 06 Sep 2026 20:22:56 +0000

Last Tuesday our product manager asked for unique emails on accounts, which sounded like a boring ALTER TABLE. I pasted the request into the coding agent and watched it emit a migration that looked textbook-clean. Staging then returned SQLSTATE 23505 because three seeded users already shared the address dev@example.com. Was the model hallucinating SQL, or did we aim a write-capable agent at a database we refuse to rebuild?

I think we keep picking the wrong host. Shared staging is a museum of forgotten seeds, half-applied branches, and indexes nobody wants to own. An agent cannot tell your leftover duplicate from a real production invariant, and you cannot smash the box when the apply goes sideways. If the rehearsal environment cannot die, the rehearsal is theater.

That is the opinion I will defend here, without a polite hedge in the middle. An agent-authored migration should first run on compute you are willing to destroy, with credentials that cannot reach staging or production. Only after the scratch host accepts the apply twice, and a schema digest stays stable, should a human paste anything into a pipeline. Familiar data on staging is not honesty. Familiar is just leftover occupancy wearing a production-shaped coat.

People treat staging like a dress rehearsal because the column names look like production and the dashboards already exist. Familiar is not the same as a controlled before-and-after. When the agent proposes ADD CONSTRAINT accounts_email_key UNIQUE (email), the first failure is rarely the parser sitting in the model. The first failure is occupancy: rows that violate the new rule, a Postgres version that rejects constraint syntax you assumed, or a role that can SELECT but cannot LOCK the table. Staging mixes those failures with yesterday’s failed deploy, so you debug ghosts instead of contracts.

A disposable host isolates the only question that matters before write authority. Did this migration apply cleanly against a known dump? Did it apply cleanly a second time without changing catalogs again? Did pg_constraint and pg_index move only in the ways the request described? Those are boring questions, and boring is exactly what you want before anyone receives a staging URL. If you cannot delete the box, you will negotiate with it, and agents are terrible negotiators.

Here is the working path I want you to steal, starting from one user action rather than a model name. The request is simple: make emails unique, keep existing users, and do not hold accounts longer than a brief lock. The agent may draft SQL. The agent may not receive a connection string that points at staging, production, or any replica that stores customer mail. Persistence starts as a checked-in dump. Permissions start as a role that can ALTER only inside one scratch database.

I keep the dump tiny and synthetic on purpose. Real addresses do not belong on a host you plan to throw away, and a rehearsal that contains production mail is not a rehearsal. Load three colliding rows on purpose so the first apply is supposed to fail until a cleanup step exists. If your agent “succeeds” against empty tables, you learned nothing about the constraint you claimed to add.

-- dumps/scratch_accounts.sql
CREATE TABLE accounts (
  id BIGSERIAL PRIMARY KEY,
  email TEXT NOT NULL,
  created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

INSERT INTO accounts (email) VALUES
  ('dev@example.com'),
  ('dev@example.com'),
  ('ok@example.com');

The cleanup is part of the artifact, not a chat message you hope someone remembers. Deduplicate first, then add the constraint, and make both steps safe to retry. I want IF NOT EXISTS on the constraint name where the server version allows it, and I want the dedupe to use a stable rule so a second apply does not invent new winners.

-- migrations/20260907_unique_email.sql
BEGIN;

DELETE FROM accounts a
USING accounts b
WHERE a.email = b.email
  AND a.id > b.id;

DO $$
BEGIN
  IF NOT EXISTS (
    SELECT 1 FROM pg_constraint
    WHERE conname = 'accounts_email_key'
  ) THEN
    ALTER TABLE accounts
      ADD CONSTRAINT accounts_email_key UNIQUE (email);
  END IF;
END$$
;

COMMIT;

Now the host. Create a role that cannot wander. Do not hand the agent your personal superuser URL because the model asked nicely, and do not put staging credentials in the same dotenv file “just for convenience.” Convenience is how write authority leaks across environments that were supposed to be seams.

# commands I actually run against scratch Postgres, never against staging
psql "$SCRATCH_DATABASE_URL" -v ON_ERROR_STOP=1 -c "CREATE DATABASE rehearsal;"
psql "$SCRATCH_DATABASE_URL" -v ON_ERROR_STOP=1 <<'SQL'
CREATE ROLE rehearsal_applier LOGIN PASSWORD 'scratch-only';
GRANT CONNECT ON DATABASE rehearsal TO rehearsal_applier;
SQL
psql "postgres://rehearsal_applier:${SCRATCH_PASSWORD}@${SCRATCH_HOST}:5432/rehearsal" \
  -v ON_ERROR_STOP=1 -f dumps/scratch_accounts.sql

The apply script is the contract. It refuses to start if the URL hostname looks like staging or prod, it records a catalog digest before and after, and it runs the migration a second time expecting a zero diff. I do not want a green checkbox from the agent. I want two identical digests after the second apply, plus a probe that proves the colliding email is gone.

# rehearsal/apply.py — example gate, not a hosted service
import hashlib, json, os, sys, urllib.parse
import psycopg

FORBIDDEN_HOST_PARTS = ("staging", "prod", "rds.amazonaws.com")
DIGEST_SQL = """
SELECT c.conname, c.contype, pg_get_constraintdef(c.oid)
FROM pg_constraint c
JOIN pg_class t ON t.oid = c.conrelid
WHERE t.relname = 'accounts'
ORDER BY 1;
"""

def refuse_shared_hosts(url: str) -> None:
    host = (urllib.parse.urlparse(url).hostname or "").lower()
    if any(part in host for part in FORBIDDEN_HOST_PARTS):
        raise SystemExit(f"refusing rehearsal against host {host}")

def digest(conn) -> str:
    rows = conn.execute(DIGEST_SQL).fetchall()
    blob = json.dumps(rows, default=str, separators=(",", ":"))
    return hashlib.sha256(blob.encode()).hexdigest()

def main() -> None:
    url = os.environ["SCRATCH_DATABASE_URL"]
    refuse_shared_hosts(url)
    migration = open("migrations/20260907_unique_email.sql").read()
    with psycopg.connect(url, autocommit=False) as conn:
        before = digest(conn)
        conn.execute(migration)
        conn.commit()
        mid = digest(conn)
        conn.execute(migration)
        conn.commit()
        after = digest(conn)
        remaining = conn.execute(
            "SELECT email, COUNT(*) FROM accounts GROUP BY 1 HAVING COUNT(*) > 1"
        ).fetchall()
    if remaining:
        raise SystemExit(f"duplicates survived apply: {remaining}")
    if mid != after:
        raise SystemExit("second apply mutated catalog; migration is not idempotent")
    print(json.dumps({"before": before, "after": after, "idempotent": True}))
    if before == after:
        print("warning: digest did not change; did the constraint already exist?", file=sys.stderr)

if __name__ == "__main__":
    main()

Run it twice in your head. The first process must fail on the stock dump until the DELETE lands, which is the point of loading collisions. After the cleanup exists, the first apply changes the digest and the second apply must not. If the second apply still rewrites catalogs, you do not have a migration. You have a script that depends on mood.

python rehearsal/apply.py
# expected on a clean scratch host after the DELETE is present:
# {"before": "...", "after": "...", "idempotent": true}

I also keep a pytest that is allowed to use the scratch URL and is forbidden from importing production settings. This is not coverage theater. It is the cross-layer failure test that staging keeps stealing from you, because staging is already dirty. When this test fails, the response you want is a process exit, not a chatbot apology.

# tests/test_rehearsal_idempotent.py
import json, os, subprocess, sys

def test_second_apply_is_digest_stable():
    env = os.environ.copy()
    assert "staging" not in env["SCRATCH_DATABASE_URL"]
    proc = subprocess.run(
        [sys.executable, "rehearsal/apply.py"],
        check=True, capture_output=True, text=True, env=env,
    )
    payload = json.loads(proc.stdout.strip().splitlines()[-1])
    assert payload["idempotent"] is True
    assert payload["before"] != payload["after"]

What failed along the way, at least in this shape of incident, was not “the model is bad at Postgres.” The agent wrote reasonable SQL. The architectural mistake was handing it a URL whose data I could not reset, whose leftover rows I could not attribute, and whose failure codes I could not tell apart from last week’s hotfix. Staging 23505 and scratch 23505 are not the same event. One is occupancy you refuse to own. The other is a contract you can rerun after deleting the database.

Version drift still bites, and you should not pretend otherwise. A throwaway host on Postgres 15 will happily accept patterns that staging on 14 will reject, especially around constraint IF NOT EXISTS folklore people copy from blogs. Pin the rehearsal image to the same major version as production, or your digest is a souvenir. Disposable does not mean random.

Permissions belong in the same story. The applier role should not create users, should not read other databases, and should not be reused by the application runtime. If your agent can print SCRATCH_DATABASE_URL into a README, it can print a worse URL tomorrow. I treat the apply script as the only process that sees the secret, and I rotate the scratch password when the host is rebuilt. That sounds heavy until you remember how many “temporary” dotenv files become permanent.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I am not going to invent model names, token ceilings, or hardware claims, because none of those rescue a migration that is not idempotent. The useful part is narrower. If you need a model lane that does not bill you while you iterate on the SQL, and a scratch server you can point this rehearsal script at, MonkeyCode’s free model access and free server option are one way to keep that loop off shared staging. If CI already gives you throwaway Postgres, use that and skip the product. The opinion does not require a vendor. It requires a host you can delete.

Who should not use this approach? Anyone whose only available server still holds real customer mail, tokens, or backups you cannot legally copy. Anyone who would promote the scratch URL into Helm values because “it worked.” Anyone expecting a free box to prove lock duration under production traffic, which this rehearsal will never show. And anyone hoping the agent can own rollback. Rollback is a human pipeline step with a reviewed inverse, not a second prompt.

Production caveats stay blunt. A stable digest is not a load test. A unique constraint still needs a query plan check on the real table size, which you do in a change window, not in chat. DELETE ... USING on a huge accounts table can lock far longer than the product manager’s “brief” story, so the scratch dump should include a realistic row count even when the addresses are fake. If you cannot build that dump without production PII, stop. You do not have a rehearsal problem. You have a data-handling problem.

I still want a short gate you can reuse before anyone pastes DATABASE_URL into an agent thread. Confirm the request names the table, the constraint, and the failure you expect on dirty data. Confirm the dump is synthetic and the role cannot see other databases. Confirm the apply ran twice with a digest that moved once. Confirm the hostname would make refuse_shared_hosts happy. If any of those are vibes instead of output, the agent does not get write authority. It gets another draft.

So here is my non-neutral close. Stop using shared staging as a kindness to the model. Staging is where humans accumulate unexplained state, and unexplained state is catnip for an agent that wants to look finished. Give the migration a host that can die, then let the pipeline commit. Which layer handoff is least stable in your shop when an agent proposes schema change—the dump, the role, the first apply, or the second? Send the SQLSTATE or the exact response body, not the story that the model “mostly got it.”

Trust the Agent Only After You Can Kill Its Server

kongkong — Sat, 05 Sep 2026 19:02:24 +0000

Last Friday a teammate walked me through an invoice summarizer that looked finished inside a long-lived notebook kernel. He typed one customer identifier, watched a tidy paragraph appear, and then declared the whole feature done. I asked him to restart the kernel and run the same cell without pasting any secrets again. Could that request still succeed, and if it could, whose invoice text was still sitting in memory?

That is the only user action I trust as a starting point for an AI feature now. A signed-in user posts an invoice identifier and expects a summary without leaking the last tenant. The first layer that fails is almost never the model, which is a hard thing to admit. It is the sticky process that still holds last week's key and last customer's raw text.

I am going to take a position that will annoy people who live inside chat windows all day. An AI feature is not a feature until you can kill the process and the HTTP contract still holds. Free model access does not buy you production, no matter how generous the trial feels this week. A restartable server that you are willing to crash on purpose is the cheaper and more honest teacher.

When I need that crashable box without standing up a billable cluster, I reach for MonkeyCode's free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach, and I am using that context as a rehearsal environment, not a bake-off. The project is open, and it currently offers free model access plus that free server path for this kind of drill. I do not need a named model or a benchmark chart to learn whether my feature remembers tenants after a restart.

Here is the architectural choice that notebooks keep hiding from you until a pager goes off. The model must sit behind a tiny client contract, and the invoice text must sit behind authorization. If either of those lives as a module global, you do not have a feature, you have a souvenir. Why do we keep pretending a kernel is an application just because the JSON looks pretty in stdout?

The proposed slice is deliberately boring, because boring is what survives a restart. One process serves HTTP, one store owns invoice bytes, and one client interface owns model calls. Auth is a bearer token that maps to a tenant, not a comment in a notebook that says remember to switch customers. Persistence is a row with tenant_id, not a Python dict that outlives the engineer who created it.

# proposed seam, not a vendor SDK tour
from typing import Protocol

class ModelClient(Protocol):
    def complete(self, *, system: str, user: str) -> str: ...

class HttpModelClient:
    def __init__(self, base_url: str, api_key: str, timeout_s: float = 20.0):
        self.base_url = base_url.rstrip("/")
        self.api_key = api_key
        self.timeout_s = timeout_s

    def complete(self, *, system: str, user: str) -> str:
        # Wire this to whatever free-model endpoint you actually have.
        # The rest of the app should never import a provider SDK.
        raise NotImplementedError("bound at process boot, not at call sites")

Boot the client once from environment variables, then refuse to read those variables again inside request handlers. That sounds fussy until a hot reload leaves a stale key in memory and your "quick retry" bills the wrong account. I want the process to be disposable, which means secrets live in the supervisor, not in a cell that nobody will re-run. If the server cannot start without MODEL_BASE_URL and DATABASE_URL, good: that is a feature, not ceremony.

# proposed FastAPI slice
from fastapi import FastAPI, Depends, Header, HTTPException
from pydantic import BaseModel

app = FastAPI()
# model_client and db are built in create_app(), never at import time

class SummaryOut(BaseModel):
    invoice_id: str
    summary: str
    model_ok: bool

def tenant_from(authorization: str | None = Header(default=None)) -> str:
    if not authorization or not authorization.startswith("Bearer "):
        raise HTTPException(status_code=401, detail="missing_bearer")
    token = authorization.removeprefix("Bearer ").strip()
    tenant = lookup_tenant(token)  # hits the store, not a global dict
    if tenant is None:
        raise HTTPException(status_code=401, detail="unknown_token")
    return tenant

@app.post("/invoices/{invoice_id}/summarize", response_model=SummaryOut)
def summarize(invoice_id: str, tenant: str = Depends(tenant_from)):
    row = db.fetch_invoice(tenant_id=tenant, invoice_id=invoice_id)
    if row is None:
        raise HTTPException(status_code=404, detail="invoice_not_found")
    text = model_client.complete(
        system="Summarize the invoice for the authenticated tenant only.",
        user=row["body"],
    )
    db.insert_summary(tenant_id=tenant, invoice_id=invoice_id, summary=text)
    return SummaryOut(invoice_id=invoice_id, summary=text, model_ok=True)

Notice what this endpoint refuses to do. It does not accept raw invoice text from the client, because that is how yesterday's paste becomes today's leak. It does not let the model choose a tenant, because models are optimistic narrators, not access-control layers. It writes the summary back to the same tenant-scoped row, so a restart cannot resurrect a paragraph that never landed in storage. If the model call fails, the user should see 503 with model_unavailable, not a half sentence that looks authoritative.

The interesting test is not "did the prose sound smart." Kill the server, start it again, and replay the same request with the same bearer token. I want 200 and the same tenant's invoice, or I want a clean 401/404, not a ghost of the previous kernel. Cross-tenant replay should stay 404, even if the invoice identifier is easy to guess. If you cannot name the status code you expect after SIGTERM, you are still demoing.

# proposed restart drill; run this against localhost or a free server you control
uvicorn app:app --port 8080 &
PID=$!
curl -sS -o /tmp/a.json -w "%{http_code}\n" \
  -H "Authorization: Bearer tenant-a-token" \
  -X POST http://127.0.0.1:8080/invoices/inv_100/summarize
kill $PID
wait $PID 2>/dev/null || true
uvicorn app:app --port 8080 &
PID=$!
curl -sS -o /tmp/b.json -w "%{http_code}\n" \
  -H "Authorization: Bearer tenant-a-token" \
  -X POST http://127.0.0.1:8080/invoices/inv_100/summarize
curl -sS -o /tmp/c.json -w "%{http_code}\n" \
  -H "Authorization: Bearer tenant-b-token" \
  -X POST http://127.0.0.1:8080/invoices/inv_100/summarize
kill $PID

What failed for me, in this kind of slice, was never the fancy sampling settings. A module-level LAST_SUMMARY cache made the second tenant look served when the model was actually down. A connection pool created at import time kept an old role after I rotated database credentials. One helper logged the first two hundred characters of invoice body, which is how a "debug print" becomes a compliance incident. The model sounded confident through all of that, which is exactly why I stopped treating fluency as evidence.

Cheap generated code makes this worse, not better, because the cost of typing dropped faster than the cost of owning state. If an assistant can scaffold an endpoint in twenty seconds, it can also hide a global in twenty seconds. The antidote is not a longer prompt. The antidote is a process you can murder, a store that still has the row, and a client you can swap without touching the route. That is a provider seam plus a restart, not a vibe.

Use the free model access as a noisy dependency, not as a personality. Slow responses, empty strings, and 429 from the upstream should already have mapped status codes in your handler. The free server option matters because it gives you a machine that is allowed to die, which laptops running notebooks are emotionally unwilling to do. I would rather rehearse a crash on a complimentary box than discover implicit memory on the first paying tenant.

This approach is wrong for some people, and I will say that plainly. If you are training weights, ranking models, or chasing latency charts, a killable HTTP slice will feel like a stall. If your app never stores user data, the tenant checks are theater. If you cannot put a bearer token in front of the route, do not put a model behind it either, because you just built a public summarizer for whoever guesses an identifier.

Production caveats stay unromantic. Cap the invoice body you send upstream, because free model access is not an excuse to ship entire PDF dumps as prompt stuffing. Keep timeouts shorter than your load balancer, or users will retry and double-write summaries. Make summarization idempotent on (tenant_id, invoice_id) so a restart plus a client retry does not create two competing paragraphs. And please stop holding API keys on the object that renders the UI; the browser is not a vault, even when the demo is internal.

If you want a crashable box without a purchase order, that free server path is enough to run this restart drill once. After that, the work is local: env at boot, tenant on every read, model behind a protocol, store as the only memory. I do not care how pretty the first summary looked in a notebook. I care whether the second process still knows who paid the invoice.

Which layer handoff is least stable in your app when the process dies: auth, persistence, or the model client? Send the status code you actually get after a restart, not the one the happy-path screenshot promised.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Add Agent Writes Behind a Schema Digest, Not a Hopeful Prompt

kongkong — Fri, 04 Sep 2026 16:57:20 +0000

A user clicks Export on the weekly revenue report and waits for a CSV that never arrives. The agent renamed amount_cents to amount because a UI mock spoke dollars, while the API still serialized integer cents. The cached client still posted to /reports/export, and the worker still selected a column that no longer existed. Where did that request die first: the button handler, the cached contract, or the database row?

I do not treat that mismatch as a prompting issue anymore, and I do not think you should either. An agent that assumes last week's contract is not being helpful; it is writing through a subway map from a closed station. Cheap generation makes those assumptions cheaper to print, so the blast radius grows every time someone says "just let the model fix it." My position is blunt: fail closed on schema drift before any model receives write authority.

The first layer that actually fails is almost never the model, even when the dashboard blames the prompt. It is the handshake you skipped between the UI, the API, and the migration that already landed. If the agent can POST without proving it saw the current schema digest, you handed it write paths across the codebase. Why would a retry help here, if that retry still carries yesterday's field names into the worker?

The opinion, without the hedge

I think the phrase "the agent will notice" is not an architecture, no matter how confident the demo looked. Noticing is a vibe, and production needs a digest, an If-Match header, and a 412. You can wrap tools and even ask the model to recite the OpenAPI file from memory. It will still assume a column that marketing renamed on Thursday, because recitation is not a handshake.

The honest response to an assumed write is not a clever repair; it is a rejected request that names the contract it missed. This is the same lesson cache people already learned in brownfield SPAs, just wearing an agent costume. A generated client that still believes v3 of a report shape is a stale cache, even if no CDN is involved. If you would not let a browser replay a PUT with a stale ETag, why allow a tool loop to do it?

A working handshake you can run

I want a vertical slice you can boot, not a diagram that pretends to be an implementation. The user action is POST /reports/export, and that is the only story I will keep on the table. The contract lives at GET /_contract/reports, returns a JSON Schema plus a hex digest, and every write must echo that digest. Auth is a scoped token, not a global admin key the agent found in an example README.

Persistence means the digest is computed from the same schema file the migration already used. So the database and the HTTP surface cannot drift in silence, which is the whole point of pinning. Here is a small Python module you can drop next to your API, labeled as a sketch you should run locally. It is not a claim about a fleet I secretly operate, and you should treat the numbers as fixtures.

# schema_digest.py
from __future__ import annotations

import hashlib
import json
from pathlib import Path
from typing import Any

SCHEMA_PATH = Path(__file__).parent / "contracts" / "reports.export.v4.json"


def load_schema() -> dict[str, Any]:
    return json.loads(SCHEMA_PATH.read_text(encoding="utf-8"))


def digest_for(schema: dict[str, Any]) -> str:
    canonical = json.dumps(schema, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(canonical.encode("utf-8")).hexdigest()


def current_digest() -> str:
    return digest_for(load_schema())

The contract file is boring on purpose, because boring is what you want a write path to be. Additional properties are closed, required fields are explicit, and cents stay cents instead of a hopeful float. If your agent cannot fill this shape, it is not ready to export anything that finance will trust.

{
  "$id": "https://example.local/contracts/reports.export.v4.json",
  "type": "object",
  "additionalProperties": false,
  "required": ["range_start", "range_end", "amount_cents", "currency"],
  "properties": {
    "range_start": { "type": "string", "format": "date" },
    "range_end": { "type": "string", "format": "date" },
    "amount_cents": { "type": "integer", "minimum": 0 },
    "currency": { "type": "string", "enum": ["USD", "EUR"] }
  }
}

The API then publishes that digest and refuses writes that do not match, before the ORM session even opens. I like Starlette-style middleware because it sits in front of every agent tool, not only the happy-path router. Does this look ceremonial from the routing table, like a bouncer checking IDs at a quiet door?

# handshake.py
from starlette.middleware.base import BaseHTTPMiddleware
from starlette.requests import Request
from starlette.responses import JSONResponse

from schema_digest import current_digest, load_schema

WRITE_PATHS = {"/reports/export"}
WRITE_METHODS = {"POST", "PUT", "PATCH", "DELETE"}


class SchemaHandshakeMiddleware(BaseHTTPMiddleware):
    async def dispatch(self, request: Request, call_next):
        if request.url.path == "/_contract/reports" and request.method == "GET":
            schema = load_schema()
            digest = current_digest()
            return JSONResponse(
                {"schema": schema, "digest": digest, "permission": "reports:read"},
                headers={"ETag": f'"{digest}"'},
            )

        if request.url.path in WRITE_PATHS and request.method in WRITE_METHODS:
            token_scope = request.headers.get("x-permission", "")
            if token_scope != "reports:write":
                return JSONResponse({"error": "forbidden", "code": 403}, status_code=403)

            incoming = request.headers.get("if-match", "").strip().strip('"')
            expected = current_digest()
            if incoming != expected:
                return JSONResponse(
                    {
                        "error": "schema_mismatch",
                        "code": 412,
                        "expected_digest": expected,
                        "hint": "GET /_contract/reports and retry with If-Match",
                    },
                    status_code=412,
                )

        return await call_next(request)

Ceremony is cheaper than a migration that the agent "helpfully" inverted while nobody watched the logs. The UI side is equally strict, because a cached fetch of the contract is how this story usually starts. I would rather pay one extra GET than debug a CSV that contains the wrong currency for a week.

// exportReport.js
export async function exportReport(payload, { token }) {
  const contractRes = await fetch("/_contract/reports", {
    headers: { Authorization: `Bearer ${token}` },
    cache: "no-store",
  });
  if (!contractRes.ok) {
    throw new Error(`contract_unreadable:${contractRes.status}`);
  }
  const { digest } = await contractRes.json();

  const res = await fetch("/reports/export", {
    method: "POST",
    headers: {
      Authorization: `Bearer ${token}`,
      "Content-Type": "application/json",
      "If-Match": digest,
      "X-Permission": "reports:write",
    },
    body: JSON.stringify(payload),
  });

  if (res.status === 412) {
    const body = await res.json();
    throw new Error(`stale_schema:${body.expected_digest}`);
  }
  if (!res.ok) {
    throw new Error(`export_failed:${res.status}`);
  }
  return res.json();
}

Notice the cache: "no-store" on that contract fetch, because a warm cache is just a polite assumption. If you let a service worker keep the digest, you rebuilt the stale map with nicer syntax and a shrug. Ask yourself whether your agent runtime treats 412 as a signal, or as an insult to be retried blindly.

The test that makes the opinion real

A handshake nobody fails is just documentation with extra YAML, and documentation does not return status codes. This test is the artifact I want in CI, and it should fail loud when an agent-shaped client skips the digest. If the harness cannot parse 412 as control flow, keep the model on a dry-run tool that cannot touch rows.

# test_schema_handshake.py
from starlette.applications import Starlette
from starlette.responses import JSONResponse
from starlette.routing import Route
from starlette.testclient import TestClient

from handshake import SchemaHandshakeMiddleware
from schema_digest import current_digest


async def fake_export(request):
    return JSONResponse({"ok": True, "id": "rpt_1"})


app = Starlette(routes=[Route("/reports/export", fake_export, methods=["POST"])])
app.add_middleware(SchemaHandshakeMiddleware)
client = TestClient(app)


def test_write_without_digest_is_precondition_failed():
    response = client.post(
        "/reports/export",
        headers={"x-permission": "reports:write"},
        json={"range_start": "2026-08-01", "range_end": "2026-08-07"},
    )
    assert response.status_code == 412
    assert response.json()["error"] == "schema_mismatch"


def test_write_with_current_digest_passes_the_gate():
    digest = current_digest()
    response = client.post(
        "/reports/export",
        headers={"x-permission": "reports:write", "if-match": digest},
        json={"range_start": "2026-08-01", "range_end": "2026-08-07"},
    )
    assert response.status_code == 200
    assert response.json()["ok"] is True


def test_write_with_read_scope_is_forbidden_even_with_digest():
    digest = current_digest()
    response = client.post(
        "/reports/export",
        headers={"x-permission": "reports:read", "if-match": digest},
        json={},
    )
    assert response.status_code == 403

Run it like this, then break the schema on purpose and watch the first test stay red until the client learns the new digest.

python -m pip install starlette httpx pytest
python -m pytest test_schema_handshake.py -q

That command is the whole ceremony: install the thin test stack, then prove the gate without standing up a warehouse. If both digest tests pass while a client still omits If-Match in production, your deploy skipped the middleware, not the model. Which layer would you trust after that miss, the prompt diff sitting in chat or the 412 counter?

Where free models and a free server actually help

I do not want the model proposing migrations against the production database while you watch to see what it does. You need a scratch box that serves the same /_contract/reports fixture, plus a loop allowed to draft patches and forbidden from applying them. MonkeyCode is an open-source environment with free model access and a free server option for this kind of scratch box.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
The useful workflow stays narrow on purpose, because a rehearsal room should not grow a shadow product.

Check out the contract file and boot the handshake on the free server before any model sees a write tool. Then let a free model propose a client change after you flip amount_cents into a money object. If the proposal still POSTs without If-Match, you learned the failure on a box that cannot hurt billing data. If it GETs the contract and retries with the new digest, you actually earned a write path.

Would I use that box to soak overnight retries of confused clients, instead of the region that stores exports? Yes, because a 412 storm is cheaper there than in the cluster that holds customer revenue files. Would I treat the free tier as proof the design is fast, durable, or cheap at scale? No, it is a rehearsal room, not a capacity plan, and I will not pretend otherwise.

What failed when softer gates looked tempting

Putting the schema into the system prompt failed first, because prompts do not have ETags and nobody invalidates them. Generating TypeScript types from a staging URL failed next, because a feature flag made staging lie about the live shape. Letting the agent migrate forward after a 500 failed after that, because a 500 is smoke rather than a contract. The only gate that kept its teeth was the digest check that returned 412 before the ORM woke up.

There is an analogy I cannot shake when people call that retry a form of intelligence. You would not let a payment provider retry a capture with a stale idempotency key and clap for the creativity. An agent retry with a stale schema is the same retry, just more polite in the logs. If the handoff between UI, API, and storage cannot name its digest, it is not a handoff; it is a rumor.

Production caveats, and who should not bother

This pattern assumes you can pin a schema file to a migration, which not every team can honestly claim. If you are on a schemaless event pile with no canonical shape, a digest will only freeze your confusion in place. If you are still discovering the product during a spike, a 412 will slow you down, and that friction is the point. You may not want that friction yet, and I would not bolt this onto a weekend prototype that still changes field names hourly.

If your agent must write during a network partition, you need an outbox with the digest recorded beside the payload. A prompt that says be careful is not an outbox, and it will not replay cleanly when the socket returns. Do not use this handshake as a substitute for backups, row-level auth, or a real job queue that you can drain. A matching digest does not mean the caller may export someone else's revenue; it only means the shape agreed.

Also do not stash the digest in localStorage and call that offline support for money movement. Offline support for writes is a ledger with identities, and that is a different argument I will not smuggle in here. A short checklist you can reuse lives in one breath, because checklists that become posters stop being run. Publish GET /_contract/... with a sha256 of the canonical schema, require If-Match on every agent write, and return 412 with the expected digest. Keep cache: no-store on the contract fetch, and assert 412 in CI with a client that forgets the header. Compute the digest from the same file the migration uses, and scope the token to the resource instead of the whole app.

I care less about which model drafted the client than about which status code the write received. If you already wired a handshake, which layer handoff is least stable for you when the names move? Send the exact failure state or response code you saw, not a vibe about the model being almost right. If you want a throwaway rehearsal for that 412 path, try MonkeyCode's free server option and keep production out of the story.

Your AI Feature Doesn't Fail at the Model. It Fails at the Handoff.

kongkong — Fri, 04 Sep 2026 13:43:57 +0000

Last week I watched a demo where an agent drafted a database migration. The chat UI streamed the plan token by token, and then the save button returned a 413: the JSON body had outgrown the proxy's limit. The model had done its job flawlessly, and the handoff had not. Nobody in the room blamed the model, yet everybody went back to comparing models.

Here is my position, stated plainly: the model is the most reliable component in your AI feature. The handoffs around it are where the feature actually dies. Auth, storage, streaming, retries, idempotency — those layers fail in ways the model never will, because they are your code and your deployment. We keep treating AI features as model problems when they are integration problems wearing a model costume.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I have been rehearsing this argument with the open-source MonkeyCode project, which gives me two things that make handoff rehearsal cheap: ten million free tokens and a free server to run the slice against. I am not claiming the tokens make the model smarter, because they do not; they make the rehearsal cheaper, and that is the entire point.

Here is the smallest slice I could build that still crosses every boundary. It is a boring FastAPI proxy that takes a chat request, checks the auth header, rejects oversized bodies, calls the upstream model, and streams the response back. The model call is about ten lines, and the handoff is everything else.

# handoff.py — the boring proxy that owns every boundary
from fastapi import FastAPI, Header, HTTPException
from fastapi.responses import StreamingResponse
import httpx

app = FastAPI()

UPSTREAM = "https://provider.example/v1/chat/completions"  # replace with your endpoint
TOKEN = "your_free_token"
MAX_MESSAGES = 50

@app.post("/chat")
async def chat(payload: dict, authorization: str = Header(default="")):
    if authorization != f"Bearer {TOKEN}":
        raise HTTPException(status_code=401, detail="missing or bad token")
    if len(payload.get("messages", [])) > MAX_MESSAGES:
        raise HTTPException(status_code=413, detail="payload too large")
    async with httpx.AsyncClient(timeout=60) as client:
        upstream = await client.post(
            UPSTREAM,
            json=payload,
            headers={"Authorization": authorization},
        )
        upstream.raise_for_status()
        return StreamingResponse(upstream.aiter_bytes(), media_type="text/event-stream")

Now deploy that proxy to the free server and run this rehearsal script against it. Every row is a boundary, not a benchmark, and the model never sees the failing requests.

# rehearse.py — every row is a boundary, not a benchmark
import httpx

BASE = "https://your-free-server.example"   # the free server
TOKEN = "your_free_token"

def check(name, headers, body, expected):
    r = httpx.post(f"{BASE}/chat", headers=headers, json=body, timeout=30)
    verdict = "PASS" if r.status_code == expected else f"FAIL (got {r.status_code})"
    print(f"{name:24} -> {verdict}")

big = {"messages": [{"role": "user", "content": "x" * 200_000}]}
small = {"messages": [{"role": "user", "content": "hello"}]}

check("missing auth", {}, small, 401)
check("wrong token", {"Authorization": "Bearer nope"}, small, 401)
check("oversized body", {"Authorization": f"Bearer {TOKEN}"}, big, 413)
check("happy path", {"Authorization": f"Bearer {TOKEN}"}, small, 200)

Run it once and you will learn more about your feature than a week of prompt tuning will teach you. The missing-auth case proves your gateway actually enforces identity, the oversized case proves your proxy has a real limit and not an aspirational one, and the happy path proves the free tokens reach the model through your code and not around it. Each of those is a handoff, and each of them can break in production without the model changing at all. The script takes about thirty seconds to run, and it gives you a table you can paste straight into a PR description.

Why does the server need to be free? Because the whole point of rehearsal is that you run it often, and you will not run it often if every run costs you money or a ticket. Localhost flatters you: TLS works, DNS works, cold starts do not exist, and the proxy limit is whatever you imagine it to be. A real server, even a free one, has real network boundaries, real timeouts, and real proxy layers between you and the model, and those are exactly the layers that fail in production. Think of the free server as a rehearsal room rather than a stage; you are not performing for users, you are practicing the parts that usually go wrong.

You might object that the model is the unpredictable part, and you are right that its output is unpredictable. But unpredictability is not the same as unreliability: the model returns a string, your proxy returns a 502, and one of those is your code. How many teams are spending their week comparing model names when their real failure is a 413 they have never once triggered? The cheapest failure is the one you have already seen, because you have already written the fix for it.

Now the honest limits, because this approach is not for everyone. If your bottleneck is genuinely prompt quality, meaning you are doing research rather than shipping, free tokens will not fix your prompt design and rehearsal will not help you. If you operate under data-residency rules, do not put customer data on a free server just because it is free, because compliance does not care about your budget. And do not build a production business on a free allowance, because a free allowance is a rehearsal budget, not a business model; treat the ten million tokens as fuel for finding handoff bugs and meter your real traffic separately. The same logic applies to the free server: it is a staging tool, so treat it like one and keep your real infrastructure separate.

Here is the checklist I reuse on every AI slice now. Deploy the proxy to the free server before you tune the prompt. Run the rehearsal script on every deploy, not just the first one. Treat every non-200 as a handoff bug until you have proven otherwise. Spend the free tokens on rehearsals rather than demos, and tear the server down when the sprint ends so the next sprint starts from a clean slate.

If you want to see where your own feature actually fails, deploy the proxy, run the script, and read the table with fresh eyes. The model will be fine, and the question is whether your handoffs are; MonkeyCode's free tokens and free server are a cheap place to start that rehearsal, and the project is open source if you want to look under the hood. Which handoff is least stable in your stack, and what response code does it return when it breaks?

MonkeyCode provides free models that can run this workflow.

The Free Token Quota Is a Failure Budget, Not a Demo Budget

kongkong — Fri, 04 Sep 2026 12:43:22 +0000

Last week a teammate demoed an AI-assisted migration planner, and the room loved it. The agent produced a clean plan, the UI rendered it in seconds, and nobody asked the one question that actually matters. I asked what happens when the model provider returns a 429 at 9:42 on a Tuesday, and the room went quiet. That silence is the real state of most AI integrations I see: the happy path is polished, and the failure path is an afterthought. The first layer that fails in production is never the model; it is the handoff between the model route and everything downstream.

Here is my position, stated plainly. Free token quotas and free servers are not a discount on the workload you already run; they are a license to break things at zero marginal cost. Teams that spend them on happy-path demos are wasting the only infrastructure that pays you to be destructive.

MonkeyCode's open-source project currently pairs a free token quota — ten million tokens as of this writing — with a free server option. It sounds like a marketing bundle; it is actually a failure-testing playground. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I am writing about it because the combination genuinely supports the workflow below, not because the numbers are impressive. Quotas like this move, so verify the current terms before you rely on them.

The reason this matters is that production failures in model-backed features almost never live in the model itself. They live in the handoffs: the rate limiter that returns 429, the gateway that times out at thirty seconds, the JSON that arrives truncated, the cold start that eats your first request. Localhost hides all of these, which is why the free server is the more valuable half of the deal. A free server gives you a real network path, a real reverse proxy, and real cold starts. The free token quota gives you permission to burn tokens on requests you fully expect to fail. The two together are the cheapest honest staging environment you will ever rent.

So here is the workflow I now recommend to anyone building on a free tier. Deploy the smallest possible FastAPI app to the free server, wrap the model call behind a provider seam, and then deliberately break it. The seam matters because it forces you to decide what failure looks like before the provider decides for you. Without it, a 429 is just a number; with it, a 429 becomes a typed error that your route, your queue, and your UI are forced to acknowledge.

# provider.py — the seam that makes failure injection possible
from abc import ABC, abstractmethod

class RateLimitedError(Exception):
    """The provider said slow down, and we made it a first-class citizen."""

class ProviderTimeoutError(Exception):
    """The provider took too long, and we refused to hang forever."""

class ModelProvider(ABC):
    @abstractmethod
    async def complete(self, prompt: str) -> str:
        """Return the completion text or raise a typed error."""

class MonkeyCodeProvider(ModelProvider):
    async def complete(self, prompt: str) -> str:
        # one HTTP call, one timeout budget, one retry policy
        # a 429 becomes RateLimitedError, a slow response becomes ProviderTimeoutError
        ...

The route then becomes a thin translation layer between the provider's failure language and the user's experience. That is the whole point of the exercise: you are deciding, in advance, that a rate limit is a 503 with a retry hint, not a stack trace.

# app.py — the smallest honest deployment
from fastapi import FastAPI, HTTPException
from provider import MonkeyCodeProvider, RateLimitedError

app = FastAPI()
provider = MonkeyCodeProvider()

@app.post("/api/chat")
async def chat(prompt: str | None = None):
    if prompt is None:
        raise HTTPException(status_code=422, detail="prompt is required")
    try:
        return {"text": await provider.complete(prompt)}
    except RateLimitedError:
        raise HTTPException(status_code=503, detail="model busy, retry later")

Before writing the injection script, run this curl once to confirm the route is alive. The script assumes the happy path works, because you are testing the failure path, not debugging the deployment.

curl -s -o /dev/null -w "%{http_code} in %{time_total}s\n" \
  -X POST https://your-free-server.example/api/chat \
  -H "Content-Type: application/json" \
  -d '{"prompt": "hello"}'

The cheapest way to start is to deploy this two-file app to MonkeyCode's free server and run the script against it; the whole loop takes an afternoon and costs nothing but quota. Now the fun part: run a failure-injection script against the deployed URL, never against localhost. The script is deliberately dumb; it fires requests designed to provoke specific failure modes and records what the stack actually does. Each probe costs a few hundred tokens, which is the entire economic argument of this article.

# inject_failures.py — spend the quota on the paths that break
import asyncio, time
import httpx

SCENARIOS = [
    ("rate_limited", "/api/chat", {"prompt": "hello"}, "provoke a 429"),
    ("oversized", "/api/chat", {"prompt": "x" * 100_000}, "provoke a context overflow"),
    ("malformed", "/api/chat", {"prompt": None}, "provoke a validation error"),
    ("slow", "/api/chat", {"prompt": "write a very long story"}, "provoke a gateway timeout"),
]

async def run(client, name, path, payload, intent):
    started = time.monotonic()
    try:
        response = await client.post(path, json=payload, timeout=30)
        elapsed = time.monotonic() - started
        print(f"{name:12s} {intent:35s} -> {response.status_code} in {elapsed:.2f}s")
    except httpx.TimeoutException:
        elapsed = time.monotonic() - started
        print(f"{name:12s} {intent:35s} -> CLIENT TIMEOUT after {elapsed:.2f}s")

async def main():
    base = "https://your-free-server.example"  # the deployed URL, not localhost
    async with httpx.AsyncClient(base_url=base, timeout=30) as client:
        for name, path, payload, intent in SCENARIOS:
            await run(client, name, path, payload, intent)

asyncio.run(main())

The output is the artifact, and the first run is usually embarrassing in a useful way. My first run taught me that the client would hang for the full timeout while the server silently retried upstream. The user saw a spinner for ninety seconds and then an error with no context. The second run taught me that an oversized prompt produced a 413 from the proxy before it ever reached the model. That is a failure mode no model benchmark will ever show you. The third run taught me that a malformed payload passed validation at the route and only failed inside the provider call. My error handling lived in the wrong layer entirely.

That last one is the pattern worth internalizing. When you inject failures on a free server, you are not testing the model; you are testing every handoff between the browser and the model. The free quota is what makes the test affordable. A single 429 probe costs a few hundred tokens, while a single demo prompt costs thousands, so the economics are on the side of destruction. Teams that treat the quota as a failure budget end up with retry policies, timeout budgets, and error contracts. They build these before they ever spend a dollar on production traffic. Teams that treat it as a demo budget end up with a beautiful video and a pager.

Now the caveats, because this approach is not for everyone. If you already have a staging environment with production-like traffic shaping, the free tier adds little. If you are building a customer-facing demo that must not fail, the free tier is the wrong place to demo it. The biggest trap is extrapolation. The failure envelope on a free server is not the failure envelope on your paid infrastructure, so treat what you learn as a lower bound, not a prediction. The quotas themselves are also a moving target, so check the current terms before you architect around them. Who should use this? Teams that are still deciding whether a model-backed feature is worth shipping. Also teams that have been burned by happy-path demos and want to see the ugly path for free.

My closing question is the same one I asked that teammate, aimed at you. Which layer handoff in your stack is least stable, and what does the user actually see when it fails? I want the concrete failure state or the response code, not the architecture diagram. If you do not know the answer, you have just found your first failure test, and the free quota is the cheapest place to run it.

MonkeyCode provides free models that can run this workflow.

Replay Recorded Tool-Call Traces Before a New Model Touches State

kongkong — Fri, 04 Sep 2026 11:43:00 +0000

A model that refuses the delete_project call in a clean prompt often behaves differently once the same prompt includes a search result, a user role, and a 12-row tool schema. The current agent-tool security discussions on DEV have the right instinct—treat tool access as a gatekeeper problem—but most quick checks test a single conversational transcript. Those transcripts do not contain the context conditions that actually flip an agent into a permission violation. The missing artifact is a replayer that turns recorded tool-call traces into a regression dataset and runs them against any model change before that model can write state.

The output should be three counters, not a vibe score: forbidden calls, missed required actions, and calls outside the case contract. That is enough to make a merge decision for a model substitution, and it gives you data you can show a reviewer.

Build the replay harness

I’m using MonkeyCode here as the open-source host for the free model route and server option. The operator reports a free 30M token allowance, which is enough to replay several thousand small tool-call traces without provisioning a GPU. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The script below expects traces.jsonl. Each line is a recorded case:

{"case_id":"a1","messages":[...],"required_actions":["search_projects"],"forbidden_actions":["delete_project"],"tools":[...]}

The replay function is deliberately small: send the original messages, force temperature=0 to reduce sampling noise, collect the returned tool_calls, then compare their function names to the case contract. Token usage is accumulated so you can stop before a free tier budget is exhausted.

import json, os
from pathlib import Path
from openai import OpenAI

client = OpenAI(
    base_url=os.environ["MODEL_BASE_URL"],
    api_key=os.environ["MODEL_API_KEY"],
)

BUDGET = int(os.environ.get("TOKEN_BUDGET", 30_000_000))

def report_case(case, tool_calls):
    required = set(case.get("required_actions", []))
    forbidden = set(case.get("forbidden_actions", []))
    called = {call.function.name for call in tool_calls}
    return {
        "forbidden_calls": sorted(called & forbidden),
        "missed_required": sorted(required - called),
        "unexpected_calls": sorted(called - required - forbidden),
    }

total = {"tokens": 0, "cases": 0,
         "forbidden_calls": 0, "missed_required": 0, "unexpected_calls": 0}

for line in Path("traces.jsonl").read_text().splitlines():
    case = json.loads(line)
    response = client.chat.completions.create(
        model=os.environ["MODEL_NAME"],
        messages=case["messages"],
        tools=case.get("tools", []),
        temperature=0,
    )
    usage = response.usage.total_tokens
    total["tokens"] += usage
    total["cases"] += 1
    calls = response.choices[0].message.tool_calls or []
    problems = report_case(case, calls)
    for key in problems:
        total[key] += len(problems[key])
    if total["tokens"] > BUDGET * 0.9:
        print(f"stop at 90% budget: {total['tokens']}/{BUDGET} tokens")
        break

print(json.dumps(total, indent=2))

The budget guard is not cosmetic. Replay evaluation can consume tokens quickly when each test case includes a long system message and ten tool definitions.

Read the report as a drift signal

A run against 48 hand-picked production traces might produce this shape:

Metric	Value
cases	48
tokens	184,392
forbidden_calls	2
missed_required	7
unexpected_calls	3

The two forbidden calls are the blocking signal: merge nothing until you understand why the new model called an action the old policy disallowed. The seven missed required actions are less urgent but still a product risk because silent omissions are harder to notice than loud denials. The three unexpected calls usually mean your trace metadata is incomplete—either you forgot an allowed action or the tool contract changed. Handle them by updating the test contract, not by ignoring them.

This is the difference between a model demo and a deployment gate. A demo shows one success. A replay report shows which failure mode appears when the same context is replayed across many cases with deterministic tool selection.

A simple decision table for evaluation layers

Layer	Evidence	Use when
One-shot deny test	Pass/fail on one transcript	Prototype, sanity check
Recorded trace replay	Three drift counters	You preserved user/service prompts
Live shadow canary	Real traffic without writes	Traces are sparse or biased
Gateway contract	Runtime deny in production	You already trust the model route

The replay layer is cheap and repeatable; the canary layer is more representative but needs live instrumentation. They are not substitutes. A model can pass all three and still be unsafe if your gateway contract does not enforce tool policy in production.

Limitations

This harness only fails a model on cases you already collected. It cannot prove safety; it can only show that a specific new model diverges from a specific recorded behavior set. If your tool names or parameter schemas change between model candidates, the traces will produce false unexpected_calls. If your recorded cases are all happy-path requests, the replay will inherit that blind spot. It also does not evaluate reasoning quality, latency, structured output accuracy, or prompt injection resistance. Those need separate tests.

Do not use this approach for a one-off prototype that has no stored tool-call history. Do not use it if your team cannot keep trace metadata in sync with the tool registry. Start instead with a smaller golden set and one read-only tool.

For a cheap first run, point the script at a recorded trace from one service and keep the budget limit enabled. The output gives you a concrete merge conversation: which tool route is allowed, which one failed, and whether the new model changed behavior under context the original conversation test never saw.

A free server option is enough to reproduce the setup.

Build Dual-Read Before Giving the Agent Drop Authority

kongkong — Fri, 04 Sep 2026 10:42:33 +0000

When the Slack ping landed at 4:51 on a Thursday, the request sounded almost reasonable and small. Could the coding agent just rename status to account_state so billing would stop lying to finance? I watched the generated diff compile, the laptop tests pass, and the settings page render the new field as if Postgres had already grown up. Have you ever shipped a rename that only existed inside the process that ran the migration?

That is the failure I keep seeing now that generation is cheap and confidence is free. The agent assumes the write it proposed already happened on every replica, cache, and phone. Replicas, CDNs, and the billing export job do not share that optimism, and they will not start sharing it because a prompt was sure. I am tired of treating a live schema change like a local refactor just because one model can emit Alembic and a TypeScript type in one breath.

Here is my position, and I am not sanding it down for politeness. You should not let an agent drop, rename, or rewrite a column until a dual-read path has survived production traffic. If the generated migration contains RENAME COLUMN or DROP COLUMN against a live API, it is not a migration. It is a coordinated outage with extra steps, and the first layer that fails is almost never the model call. It is the serializer still emitting the old key into a cache that will happily serve mixed JSON for the next ninety seconds.

Think about a bridge crew that dynamites the old span while the new deck is still a drawing on a tablet. Why do we let a coding agent cut the old column the moment the new model file exists on disk? Because the prototype loop rewards a green local test, and local tests do not have rolling deploys, read replicas, or a React Query key the same agent also invented. Cheap code moves the debt out of functions and into unsigned contracts between UI, API, and storage. That is not architecture being precious. That is the only way a rename fails closed instead of failing in support tickets.

The working path I want is boring on purpose, which is exactly why generated code tries to skip it. Freeze a response contract that can carry both field names without lying about which one is canonical yet. Serve reads from both columns while you backfill. Write only to the new column after the expand is deployed. Delete the old column only after every caller, including the forgotten CSV export, has stopped asking for it. An agent may propose the expand. It does not get drop authority until dual-read tests say the old shape still round-trips.

I keep that rule in code, not in a prompt appendix nobody enforces at apply time. The read model dual-reads, the write model refuses the old column, and the apply endpoint rejects forbidden operations with 409 instead of a cheerful 200.

# proposed pattern: dual-read serializer + forbidden apply ops
# labeled example — run it as a contract test, not as folklore
from typing import Any, Literal

from pydantic import BaseModel, Field, model_validator

class AccountStateRead(BaseModel):
    id: str
    status: str | None = None          # old wire key, still served
    account_state: str | None = None   # new wire key
    canonical: Literal["status", "account_state"]

    @model_validator(mode="after")
    def both_shapes_or_honest_null(self) -> "AccountStateRead":
        if self.status is None and self.account_state is None:
            raise ValueError("dual-read produced no value on either column")
        if self.canonical == "account_state" and self.account_state is None:
            raise ValueError("canonical column missing while old column still exists")
        return self


class SchemaPlan(BaseModel):
    table: str
    old_column: str
    new_column: str
    operations: list[str] = Field(default_factory=list)
    dual_read_proven: bool = False

    @model_validator(mode="after")
    def no_drop_before_dual_read(self) -> "SchemaPlan":
        forbidden = {"DROP COLUMN", "RENAME COLUMN"}
        hits = [op for op in self.operations if op.upper() in forbidden]
        if hits and not self.dual_read_proven:
            raise ValueError(f"drop/rename before dual-read: {hits}")
        return self

The API should make that failure a status code, not a log line a human might miss during a Friday rollout. I want the agent to be able to POST a plan, and I want apply to be a different verb with a different permission. If you give the same bearer token to plan and apply, you did not build a gate. You built a comment.

# proposed FastAPI slice — plan is readable, apply is capability-gated
from fastapi import FastAPI, Header, HTTPException
from fastapi.testclient import TestClient

app = FastAPI()
PROVEN = {"accounts.status->account_state": False}

@app.post("/schema/plans")
def create_plan(plan: SchemaPlan):
    try:
        plan.dual_read_proven = PROVEN.get(
            f"{plan.table}.{plan.old_column}->{plan.new_column}", False
        )
        SchemaPlan.model_validate(plan.model_dump())
    except ValueError as exc:
        raise HTTPException(status_code=409, detail=str(exc)) from exc
    return {"id": "plan_123", "status": "accepted", "apply": False}

@app.post("/schema/plans/{plan_id}/apply")
def apply_plan(plan_id: str, x_capability: str | None = Header(default=None)):
    if x_capability != "schema:apply":
        raise HTTPException(status_code=403, detail="plan cannot apply itself")
    raise HTTPException(status_code=409, detail="dual-read not proven")

Does that feel slow compared with letting the agent run Alembic on the primary? Good. Speed is how the bad rename got into staging last time. The cross-layer test I actually care about is not “did the model emit SQL.” It is whether the UI can still hydrate from the old key while a replica is one statement behind, and whether apply stays 403 or 409 when the capability header is missing. If your only test is a unit test on the generator, you tested the printer, not the press.

def load_account(row: dict[str, Any]) -> AccountStateRead:
    new = row.get("account_state")
    old = row.get("status")
    return AccountStateRead(
        id=str(row["id"]),
        status=old if old is not None else new,
        account_state=new if new is not None else old,
        canonical="account_state" if new is not None else "status",
    )


def test_dual_read_survives_replica_lag():
    primary = {"id": "u1", "status": None, "account_state": "active"}
    replica = {"id": "u1", "status": "active", "account_state": None}  # lag
    assert load_account(primary).account_state == "active"
    assert load_account(replica).status == "active"


def test_apply_without_capability_is_403():
    client = TestClient(app)
    plan = {
        "table": "accounts",
        "old_column": "status",
        "new_column": "account_state",
        "operations": ["ADD COLUMN"],
    }
    assert client.post("/schema/plans", json=plan).status_code == 200
    denied = client.post("/schema/plans/plan_123/apply")
    assert denied.status_code == 403


def test_drop_before_dual_read_is_409():
    client = TestClient(app)
    plan = {
        "table": "accounts",
        "old_column": "status",
        "new_column": "account_state",
        "operations": ["DROP COLUMN"],
    }
    res = client.post("/schema/plans", json=plan)
    assert res.status_code == 409
    assert "dual-read" in res.json()["detail"]

Run that file against a real HTTP process, not against a mocked router in a notebook. I use a scratch box so the 409 is an actual wire response the UI must handle, because optimistic caches are where generated frontends hide the failure. Disclosure: This article was prepared as part of MonkeyCode's product outreach. When I need a disposable model to propose nasty migrations and a free server to prove the API rejects them, MonkeyCode's free model access and free server option are the scratch lane I actually use, not a place I apply DDL.

The trick is the job you give the free tier. Ask it to generate the migration a tired Friday agent would generate, including RENAME COLUMN status TO account_state in one shot. Then POST that plan at your contract endpoint and record the status code. If you get 200, your gate is theater. If you get 409 with a detail the UI can show, you just spent free tokens on the only rehearsal that matters: making the agent stop assuming the schema it invented already exists everywhere.

# proposed rehearsal — generate a bad plan, then prove the server refuses it
python -m pip install fastapi pydantic uvicorn pytest httpx
pytest -q test_schema_contract.py
uvicorn app:app --port 8080 &

curl -sS -D - http://127.0.0.1:8080/schema/plans \
  -H 'content-type: application/json' \
  -d '{"table":"accounts","old_column":"status","new_column":"account_state","operations":["DROP COLUMN"]}'

# expect HTTP/1.1 409 Conflict, not a migration id

Production still has caveats this pattern will not hug away. Dual-read doubles the chance you leak an old value after a new write if you forget to write-through both during the expand, so I write new-only and backfill in batches with a lag check. JSON documents inside jsonb columns will not be saved by a table-level dual-read, and mobile clients that pin an old protobuf will ignore your pretty serializer. Feature flags that only hide a button do not count as authorization, because the agent will call apply without the button. And if your ORM session is shared between the request identity and the tool runner, the capability header is a costume.

Who should not use this? Do not drag expand and contract into a throwaway prototype with one user and a database you can drop at lunch. Do not use a free scratch server as your production migrator, and do not treat a passing local pytest as proof that replicas converged. If you have no way to version the wire contract, this whole argument collapses into comments in a pull request. Comments are not a contract. Status codes are.

I still want a short checklist you can paste beside the generated Alembic file before anyone types apply. Dual-read serializer emits both keys and fails if both are empty. Plan endpoint accepts expand operations and returns 409 on drop or rename while dual_read_proven is false. Apply requires a capability the planner does not hold, and missing it is 403. One test sends the old UI payload against a lagged replica shape. One test hits a real HTTP port, not a mocked app object. If any box is missing, the agent does not get drop authority, no matter how clean the diff looks.

So I will keep arguing the same unpopular thing when the next rename shows up in Slack at 4:51. Cheap generation did not make schema design optional. It made unsigned handoffs cheaper to ship, which is worse. Which layer handoff is least stable in your app right now, and what status code does it return when the agent assumes the new column already exists—200 with mixed JSON, 403 on apply, or a quiet 500 after the replica catches up?

Put a Ledger Between the Model and the Row

kongkong — Thu, 03 Sep 2026 11:35:05 +0000

A billing operator approved an AI-suggested pause on sub_9f21. The UI returned 200. Finance later found the subscription already paused from a browser retry ten seconds earlier. Slack blamed “the model.” The second request never called a model. Generation and mutation had shared one POST, so a normal retry became a second write.

Cheap completion APIs make that mash-up feel like a feature. A handler that loads a row, asks a vendor, and updates the same row in one shot looks tidy in a screenshot. It is also how you lose the ability to say whether a 409 was a stale snapshot, a permission miss, or a double spend. Intelligence is not a GRANT. The missing piece is a ledger the apply path can spend without thinking.

Advice is not a transaction

I treat every model-backed mutation as two products that happen to share a domain object:

A suggestion is a durable record of what the caller was allowed to see, what the model proposed, and which row version that proposal was built on.
A commit is a separate permissioned action that spends that record against the current row, once.

If those two live in one function, you cannot replay the advice, you cannot expire it, and you cannot keep retries from inventing a second story. The model never belongs in the write transaction. Consultants do not get the checkbook.

This is not a prompt-quality essay. It is a rehearsal you can paste into a small FastAPI app: subscriptions instead of a chat demo, a suggestion table that cannot update billing state, and an apply route that is forbidden from calling a provider.

Schema first, because the prompt will not stop a write

Ship the ledger in a migration that has no UPDATE on subscriptions. If the generated handler “needs a column to write,” that is the smell. Two deploys is cheaper than a week of explaining extra version bumps.

-- 001_subscription_suggestions.sql
CREATE TABLE subscription_suggestions (
  suggestion_id TEXT PRIMARY KEY,
  subscription_id TEXT NOT NULL,
  actor_id TEXT NOT NULL,
  snapshot_json TEXT NOT NULL,
  change_json TEXT NOT NULL,
  seen_version INTEGER NOT NULL,
  provider TEXT NOT NULL,
  lifecycle TEXT NOT NULL,
  created_at TEXT NOT NULL
);
-- This file must not mutate subscriptions.

The snapshot is the lock, not a timestamp the model wrote in a paragraph. When people argue that “the model was wrong,” they are usually arguing about stale context. If you cannot reload the exact read, you cannot blame the provider and you cannot defend the commit.

Typed records, no optional write smuggled in

Keep the contract small. Optional fields are how apply sneaks into a suggest payload “for later.”

# suggestion_contract.py
from typing import Literal, Optional
from pydantic import BaseModel, Field

class SubscriptionView(BaseModel):
    id: str
    state: Literal["active", "paused", "canceled", "past_due"]
    plan_code: str
    version: int
    last_invoice_cents: int
    account_note: Optional[str] = None

class SuggestedChange(BaseModel):
    change: Literal["pause", "resume", "cancel", "no_op"]
    rationale: str = Field(min_length=8, max_length=500)
    seen_version: int

class SuggestionRecord(BaseModel):
    suggestion_id: str
    subscription_id: str
    actor_id: str
    snapshot: SubscriptionView
    change: SuggestedChange
    provider: str
    lifecycle: Literal["draft", "spent", "rejected", "expired"]

seen_version is copied from the view the provider received. If the wrapper invents a version, fail the suggestion. Do not “helpfully” apply prose to the wrong generation of the row.

Suggest is a query-shaped POST

Creating a suggestion authenticates, authorizes a read, snapshots the subscription, calls a seam, and inserts a ledger row. It does not flip state, even when the recommendation looks obvious. If that feels slow for a demo, that slowness is the rehearsal.

# suggest.py
from fastapi import APIRouter, Depends, HTTPException
from .auth import require_user
from .subscriptions import SubscriptionRepo, Forbidden, NotFound
from .suggestions import SuggestionRepo, new_id
from .advisor import Advisor

router = APIRouter()

@router.post("/subscriptions/{subscription_id}/suggestions")
def create_suggestion(subscription_id: str, user=Depends(require_user)):
    try:
        snapshot = SubscriptionRepo.get_for_read(subscription_id, actor_id=user.id)
    except NotFound:
        raise HTTPException(status_code=404, detail="subscription not found")
    except Forbidden:
        raise HTTPException(status_code=403, detail="read not allowed")

    change = Advisor.recommend(snapshot)
    if change.seen_version != snapshot.version:
        raise HTTPException(
            status_code=409,
            detail="advisor echoed a version it was not given",
        )

    return SuggestionRepo.insert(
        suggestion_id=new_id(),
        subscription_id=subscription_id,
        actor_id=user.id,
        snapshot=snapshot,
        change=change,
        provider=Advisor.name,
        lifecycle="draft",
    )

The 409 on a mismatched seen_version is the first place a sloppy SDK wrapper will lie. I would rather fail the draft than let confident JSON land on the wrong generation. Can your current logs tell that failure apart from “row changed under us” on apply?

The advisor may not see a repository

People paste a vendor client into a module that already imported the ORM, then call the paste “architecture.” The seam takes a SubscriptionView and returns a SuggestedChange. No request object, no session, no bearer token in a prompt log you will one day export.

# advisor.py
class Advisor:
    name = "env-complete"

    @staticmethod
    def recommend(snapshot: SubscriptionView) -> SuggestedChange:
        # Keep SubscriptionRepo out of this file on purpose.
        return client_from_env().complete(
            system=(
                "Return JSON for SuggestedChange. "
                "Copy seen_version from the snapshot. Never invent a version."
            ),
            user=snapshot.model_dump_json(),
            schema=SuggestedChange.model_json_schema(),
        )

I am not naming a model. The product is not the incident, and the incident is whether complete() can change billing state. In this module it cannot, because there is no repository. If you cannot keep the repository out, you are not ready to argue about latency or which free tier you used.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I treat MonkeyCode's free model access and free server option as a throwaway box for hosting this suggestion API and its test runner, not as a substitute for production permission checks. A disposable box is enough to feel 401, 403, and 409 before they touch a live subscription.

Prove the ledger cannot move the row

Do not assert that the rationale sounds wise. Assert that the slice fails closed, that a retry does not become a commit, and that the subscription version is unchanged after suggest. Run these against the process you actually booted, including auth middleware. A notebook that imported the route module and skipped middleware is not a rehearsal.

# test_suggestion_ledger.py
def test_suggest_without_credentials_is_401(client):
    res = client.post("/subscriptions/sub_9f21/suggestions")
    assert res.status_code == 401

def test_suggest_without_read_scope_is_403(client, observer_token):
    res = client.post(
        "/subscriptions/sub_9f21/suggestions",
        headers=auth(observer_token),
    )
    assert res.status_code == 403

def test_draft_does_not_bump_subscription_version(client, owner_token, sub_9f21):
    before = sub_9f21.version
    res = client.post(
        "/subscriptions/sub_9f21/suggestions",
        headers=auth(owner_token),
    )
    assert res.status_code == 200
    assert res.json()["lifecycle"] == "draft"
    assert sub_9f21.reload().version == before

def test_idempotent_create_returns_the_same_suggestion_id(client, owner_token, idem_key):
    headers = auth(owner_token, idem_key)
    first = client.post("/subscriptions/sub_9f21/suggestions", headers=headers)
    second = client.post("/subscriptions/sub_9f21/suggestions", headers=headers)
    assert first.status_code == 200
    assert second.status_code == 200
    assert first.json()["suggestion_id"] == second.json()["suggestion_id"]

If you keep one assertion, keep the version check. A suggest endpoint that increments the subscription version is a write wearing a read costume, and it will poison apply with conflicts you invented. Idempotency belongs on create because browsers retry. A new draft on every retry is how one subscription collects three incompatible stories.

export BASE=http://127.0.0.1:8000

curl -s -o /tmp/unauth.json -w "unauth:%{http_code}\n" \
  -X POST "$BASE/subscriptions/sub_9f21/suggestions"

curl -s -o /tmp/forbidden.json -w "observer:%{http_code}\n" \
  -H "Authorization: Bearer $OBSERVER" \
  -X POST "$BASE/subscriptions/sub_9f21/suggestions"

curl -s -D - -o /tmp/draft.json \
  -H "Authorization: Bearer $OWNER" \
  -H "Idempotency-Key: rehearsal-7c2e" \
  -X POST "$BASE/subscriptions/sub_9f21/suggestions"

python -c "import json; print(json.load(open('/tmp/draft.json'))['lifecycle'])"
# draft — SELECT on subscriptions.version must match the value from before the POST

Commit spends the ledger; it does not think

Only after those checks stay green do I add apply, as a different resource with a different permission. The handler loads the suggestion, checks write scope, refuses anything that is not draft, compares versions, and updates one row in one transaction. It does not call the advisor again. A retry must not become a different change. Calling the model during apply is how a pause becomes a cancel while the network is merely being honest.

# commit.py
@router.post("/suggestions/{suggestion_id}/commit")
def commit_suggestion(suggestion_id: str, user=Depends(require_user)):
    record = SuggestionRepo.get(suggestion_id)
    if record is None:
        raise HTTPException(status_code=404, detail="suggestion not found")
    if not SubscriptionRepo.can_write(record.subscription_id, actor_id=user.id):
        raise HTTPException(status_code=403, detail="write not allowed")
    if record.lifecycle != "draft":
        raise HTTPException(
            status_code=409,
            detail=f"suggestion already {record.lifecycle}",
        )

    try:
        SubscriptionRepo.apply_change(
            subscription_id=record.subscription_id,
            change=record.change.change,
            expected_version=record.change.seen_version,
            actor_id=user.id,
        )
    except Conflict:
        raise HTTPException(
            status_code=409,
            detail="subscription changed since this draft was stored",
        )

    SuggestionRepo.mark_spent(suggestion_id)
    return {"ok": True, "suggestion_id": suggestion_id}

Think of the draft as a single-use voucher. Spending it twice is a 409, not a second generation. Streaming tokens from the commit route to make the button feel alive is how you hide a write behind a progress bar.

Operational rules that are not footnotes

Persist the suggestion in the same system that owns the subscription, or you will commit a plan whose snapshot has already vanished.
Expire drafts when version moves, or operators will apply archaeology with a straight face.
Write suggestion_id onto the billing event. An audit trail that is a chat screenshot in a ticket is not an audit trail.
Keep suggest and commit in separate deploys if that is what it takes to keep the first migration write-free.

Skip this pattern if the model only writes copy that never touches a row. A ledger would be ceremony. Skip it if the domain has no conflict token and no permission bit; you have a larger problem than a wrapper. If you need a multiplayer sketch this afternoon, keep the API unable to mutate until the contract exists in code. This is for teams who already know the feature will write and do not want to discover authorization in production.

I still hear “which model is smart enough to hold a write lock?” as if fluency were a database privilege. That question keeps producing one handler that does everything badly. Trust lives in the handoff: the read check, the snapshot you stored, the version you compared, and the commit that is not allowed to think. Generated code made that handler easy. It did not make rollback cheap, and it did not make a silent 200 that mutated twice acceptable.

Reuse the path before anyone debates a prompt. Boot the API with a real database and an advisor that cannot import SubscriptionRepo. Create one actor who can read sub_9f21 and one who cannot. Hit POST /subscriptions/{id}/suggestions until 401, 403, 404, and 200 are boring. Prove the version did not move, replay the idempotency key, then enable commit against a stale draft that must return 409 without calling the model.

Which boundary in your stack is least stable on retry right now—the read, the ledger insert, or the spend—and what status code does it actually return? Send a concrete 403, a 409, or a 200 that wrote twice. I care less about the wording of the rationale than about whether the advice can survive being wrong without taking the row with it.

Stop Letting Your Model Remember Things—Use Free Tiers to Enforce Statelessness

kongkong — Tue, 01 Sep 2026 12:39:49 +0000

I spent an afternoon last week chasing a bug where an AI endpoint returned one user's email address to another user. The model was innocent; my code had a module-level list that accumulated every request's context. It wasn't a model-memory problem, it was an application-memory problem, and it made me rethink how we treat context.

We keep talking about AI models having memory, but the real issue is that we give them memory by accident. Every global variable, every cached conversation, every 'smart' retry that tacks on previous messages turns a stateless inference call into a hidden state machine. And a hidden state machine is the last thing you want in production.

Here's where free tiers come in. A free model with a small context window won't let you stuff the entire conversation history into the prompt. A free server gives you just enough compute to build a context-assembly layer without overprovisioning. Both forces push you toward a stateless design: pass the minimum context the model needs, every time, and let your application own the state.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I've been using MonkeyCode's free model access and free server option for this exact workflow, and it fits the pattern beautifully.

MonkeyCode is an open-source project that offers a free tier with a generous token allowance (currently 10M in my plan) and a free server target for your own services. I won't quote latency or uptime numbers because they change; what matters is the architectural discipline it lets you practice.

Here's the pattern I use. The client sends a request and a user ID. The server—running on a free MonkeyCode server—loads only the facts relevant to that user, constructs a one-shot prompt, and calls the model. No session state, no hidden history.

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
import db, llm  # your storage and MonkeyCode client

app = FastAPI()

class AskRequest(BaseModel):
    user_id: str
    question: str

class AskResponse(BaseModel):
    answer: str

@app.post("/ask", response_model=AskResponse)
async def ask(req: AskRequest):
    # Load only the facts needed for this question
    context = db.get_relevant_facts(req.user_id, req.question)
    if not context:
        raise HTTPException(status_code=404, detail="No context for this user")
    # One-shot, stateless call
    answer = llm.call(
        "You answer with facts only. Reply using the given context.",
        f"Context: {context}\n\nQuestion: {req.question}"
    )
    return AskResponse(answer=answer)

Because the model call is one-shot, you can test it in isolation. Two requests with the same context produce the same result (modulo temperature). And because the server owns the context, you can change models, add guardrails, or version your prompts without touching the client.

The free tier forces this discipline. A 10M-token allowance is generous, but it's still finite; you won't waste it on giant prompts. A free server is also finite; you'll write code that runs within its limits. These constraints are features, not bugs.

To verify you're actually stateless, write a test that replays the same request twice and asserts the same answer (with temperature set to 0). Then delete the 'context' and see if the model still returns something—it shouldn't, if your system is honest. Here's a minimal pytest suite that captured my regression:

def test_stateless_ask():
    c1 = client.post("/ask", json={"user_id": "42", "question": "What's my plan?"})
    c2 = client.post("/ask", json={"user_id": "42", "question": "What's my plan?"})
    assert c1.json() == c2.json()

def test_missing_context_rejected():
    r = client.post("/ask", json={"user_id": "nobody", "question": "hi"})
    assert r.status_code == 404

The second test might surprise you. A model that 'remembers' would return something even with no context. My stateless service refuses. That's the point.

But this pattern isn't for every AI feature. If you're building a chatbot that genuinely needs multi-turn memory, you'll need to store that state somewhere—a database, a vector store, or a cache. The trick is to make that state explicit and retrievable, not implicit in the model call. Also, free tiers have rate limits and no SLA; don't build a critical path on them without a fallback. You've been warned.

Here's my short checklist for adopting this pattern: make sure every model call is one-shot with all needed context in the prompt, the server explicitly loads that context, and a test proves the endpoint rejects missing context. That's the contract.

MonkeyCode's free tier is a great way to try this without spinning up your own GPU. Check the repo, and let me know which layer of your AI stack is the most stateful—I'm curious.

Your AI Reviewer Needs a Review. Use Free Tokens and a Free Server to Build One.

kongkong — Mon, 31 Aug 2026 11:39:21 +0000

Last month, we merged an AI-generated pull request that quietly removed a retry loop from our payment service. Two days later, a customer's payment failed silently, and we only found out because the support team happened to ask. The reviewer on that PR had said "looks fine," but nobody had ever tested whether that reviewer could catch an AI's confident hallucination. That's the moment I started treating the reviewer as a piece of software that needs its own test suite.

Free model tokens and a free server are everywhere these days, and most people use them to build a chatbot that tells you the weather. That's fine for a weekend, but it doesn't help you ship anything safer. What you actually need is a way to verify the automated reviewer you already trust, because if the reviewer is wrong, you're just moving the risk from the code to the review process.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I've been using MonkeyCode's free model access and its free server option to run exactly this kind of experiment, and they're a practical fit if you want to move beyond demos. Instead of treating the free tier as a toy, I use it to host a reviewer bot that blocks PRs with suspicious changes, and then I run a second bot that tries to fool it.

Here's the core loop I run on that free server. The attacker bot generates a fake PR by mutating a real one, and then we check whether the reviewer catches the mutation. It's like penetration testing for your code review process, not for your API.

# adversary.py — simplified for readability
import os, requests

def mutate_diff(diff):
    lines = diff.splitlines()
    for i, line in enumerate(lines):
        if line.startswith("+") and "status_code" in line:
            lines[i] = line.replace("200", "502")
            break
    return "\n".join(lines)

def run_reviewer(diff):
    resp = requests.post(
        os.getenv("REVIEWER_URL"),
        json={"diff": diff},
        timeout=30
    )
    return resp.json()["verdict"]

original = open("sample.diff").read()
for _ in range(500):
    adversarial = mutate_diff(original)
    verdict = run_reviewer(adversarial)
    if verdict == "approved":
        print("Reviewer missed a mutation!")
        break
    else:
        print("Caught it, continuing...")

That script gives you a quantitative signal: how many mutations your reviewer lets through before it blocks them. If you run this once and see a dozen missed mutations, you know your rule-based heuristic is too narrow. If you run it and see zero, you might be too strict, which is fine for a gate but not for developer productivity. The key is to treat this as an experiment, not a one-off script.

The free server part matters because you want this loop running on a schedule, not just when you remember to type python. I use a cron job that fires every hour, writes the results to a small SQLite database, and alerts me if the block rate drops below a threshold. That way I'm measuring the reviewer's health continuously, which is exactly what you'd do for any other service in production.

Of course, production is more nuanced than a 500-iteration loop. Model output is probabilistic, so your gate should return "requires human review" when confidence is low, not a clean yes or no. Your mutation set needs to reflect actual failure modes you've seen in your codebase, otherwise you'll polish a bot that only catches fake problems. And the free server option you're using will have cold starts and network hiccups, so your bot must handle timeouts and retries with a sane fallback.

Who should not use this approach? If you don't have a clear contract or a set of known historical failures, the adversarial test will give you confidence mostly in your own creativity. If you're just looking for a chat widget, the free tokens are better spent elsewhere. And if you can't commit to periodically auditing the reviewer's decisions, you'll slowly trust a screen that has no accountability behind it. For teams that already have a heavy manual review process, this might feel like overkill, but if AI is generating most of your code, you cannot rely on the same human attention that worked for hand-written PRs.

Here's a short checklist I reuse whenever I set up one of these reviewer gates: make the reviewer output structured JSON with a confidence score; seed your mutation library with real bugs from your last quarter; run the reviewer in CI as a gate, but keep a manual override path; log every verdict and mutation so you can reconstruct why it passed or failed. None of this requires a big budget, but it does require treating the free resources as infrastructure, not as a toy.

So before you merge the next AI-generated PR, ask yourself a simple question: when was the last time you tested the reviewer? If you can't answer, you already have your next experiment.

Free Models and a Free Server Are Enough to Validate an AI Endpoint

kongkong — Sun, 30 Aug 2026 10:49:26 +0000

A support team wants to triage incoming tickets automatically. Management asks for a proof of concept. No budget exists yet. Most engineers reach for a paid API and worry about the invoice later. That order is wrong. Cost should be tested before code is written.

Free models let you experiment without commitment. A free server lets you run the experiment in a realistic environment. MonkeyCode provides both. Their free tier includes model access and a container host. Disclosure: This article was prepared as part of MonkeyCode's product outreach. You can build a working HTTP service without a credit card. You can also tear it down and start over without regret.

This article walks through a small ticket triage service. It uses FastAPI, the OpenAI SDK, and environment variables for configuration. The same image runs against a local model, a paid provider, or MonkeyCode's free model. The only difference is the variable value. That separation is the first step toward production.

The service accepts a ticket text and returns a priority and a category. The model call uses a generic client. Here is the core implementation.

import os
from typing import Literal
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel, Field
from openai import OpenAI

app = FastAPI()

client = OpenAI(
    base_url=os.getenv("LLM_BASE_URL"),
    api_key=os.getenv("LLM_API_KEY"),
)
model = os.getenv("LLM_MODEL", "default")

class Ticket(BaseModel):
    text: str = Field(..., min_length=1, max_length=2000)

class Triage(BaseModel):
    priority: Literal["low", "medium", "high"]
    category: str

@app.post("/triage", response_model=Triage)
async def triage(ticket: Ticket):
    prompt = (
        "Classify the support ticket. Return two lines: "
        "priority (low/medium/high) and category (one word). "
        f"Ticket: {ticket.text}"
    )
    try:
        response = client.chat.completions.create(
            model=model,
            messages=[{"role": "user", "content": prompt}],
            temperature=0,
            max_tokens=30,
        )
        content = response.choices[0].message.content.strip()
        lines = content.split("\n")
        if len(lines) < 2:
            raise ValueError("model returned fewer than two lines")
        priority = lines[0].lower()
        category = lines[1].strip()
        if priority not in ("low", "medium", "high"):
            raise ValueError("unknown priority")
        if not category or len(category) > 50:
            raise ValueError("invalid category")
        return Triage(priority=priority, category=category)
    except Exception as exc:
        raise HTTPException(status_code=502, detail=f"Model failure: {exc}")

Several decisions matter here. The client lives at module level, so connections are reused. The model name and endpoint come from the environment. That makes provider switching trivial. The prompt is short and explicit. Temperature is zero for deterministic output. Max tokens is thirty because the answer is tiny. A tight limit keeps latency down on a shared free server.

The validation block is the most important part. It assumes the model returns exactly two lines. If the model returns unexpected text, the service returns a 502. This is a contract. The prompt is the contract. The parser enforces it. Many AI prototypes fail because they trust the model's formatting. This one does not.

Now the service needs to run somewhere. A Docker container makes deployment uniform. The Dockerfile is deliberately lean.

FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY app.py .
EXPOSE 8000
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]

The requirements file contains only three packages:

fastapi
uvicorn[standard]
openai

Build the image and push it to MonkeyCode's free server. The platform accepts standard container images. Check the current dashboard for the registry URL and credentials. Commands change too frequently to quote them here. Follow the documented flow.

Once the service is online, real testing begins. A simple shell script drives traffic with hey. This gives concrete latency and error numbers.

#!/usr/bin/env bash
set -euo pipefail
URL="${URL:-http://localhost:8000}"
DATA='{"text":"Payment failed twice. Need refund now."}'
for c in 1 10 20; do
  echo "=== concurrency $c ==="
  hey -n 200 -c "$c" -m POST \
    -H "Content-Type: application/json" \
    -d "$DATA" "$URL/triage"
done

Run this against the deployed endpoint. Expect surprises. A free server runs on shared CPU and limited memory. Requests may time out. Some may return 502 when the model is slow. Latency may climb as concurrency rises. These are not bugs. They are measurements.

The free model endpoint also behaves differently from a local one. Token limits, response speed, and formatting can vary. Your parser may break on an unexpected newline. That is a legitimate production failure. It happens only when you test against the actual endpoint.

A decision table helps choose between the free tier and paid infrastructure.

Condition	Free model + free server	Paid infrastructure
Proof of concept	Yes	No
Low-traffic internal tool	Yes	No
Staging environment	Yes	Maybe
Customer-facing SLA	No	Yes
High concurrency (>50)	No	Yes
Strict latency budget	No	Yes

The free tier is not a permanent replacement. It is a validation tool. It tells you whether the feature works and where the bottlenecks are. That knowledge is worth more than the seven dollars you would have spent on a trial run.

Who should avoid this approach? Teams already at scale need real capacity. A free server is not a place for a production database. Companies with strict data policies may not allow external model calls. Applications with a hard latency budget need dedicated resources. Use the free tier only for experiments, staging, or very light internal tools.

The lesson is simple. Free models and a free server remove the financial excuse for skipping integration tests. The next time an AI feature is proposed, deploy it on the free tier first. Measure the failures. Fix the contract. Then decide whether a paid layer is justified. MonkeyCode gives you free model access and a free server in one place. Try them together and see where your assumptions break.