DEV Community: kongkong

Stop Building AI Gatekeepers. Start Shipping Boring Proxies.

kongkong — Thu, 20 Aug 2026 01:01:05 +0000

Last week a teammate asked me to review a new "gatekeeper" service that sat in front of every agent tool call. It was four hundred lines of clever permission logic, and it still let a malformed tool response crash the whole request. Nobody had tested the handoff between the model and the storage layer, and I closed the review with one question: what happens when the model returns the right shape but the wrong answer?

There is a popular discussion this week about stopping AI agents from using tools and building a gatekeeper instead, and the instinct is correct. Agents with tools are dangerous, and every team I know is bolting on permission checks after the first incident. But I think the architecture is backwards: a gatekeeper is middleware with a policy problem, and policy belongs in the contract, not in the request path.

What an agent feature actually needs is a boring proxy with three things: a contract test, a budget ledger, and a kill switch. Everything else is decoration. The contract test pins the response shape so a model upgrade cannot silently break the UI, and the budget ledger answers who pays before the bill arrives. The kill switch is just a flag that stops the request path when the model misbehaves, which is more useful than a permission matrix you will forget to maintain.

Here is the proxy I keep coming back to, written as a single FastAPI file:

# boring_proxy.py
import os
import httpx
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel, Field

app = FastAPI()

class Prompt(BaseModel):
    text: str = Field(min_length=1, max_length=2000)
    user_id: str

LEDGER: dict[str, int] = {}  # staging-only ledger, resets on restart
DAILY_BUDGET = int(os.getenv("DAILY_BUDGET", "50000"))
MODEL_URL = os.getenv("MODEL_URL")
MODEL_KEY = os.getenv("MODEL_KEY")
KILL_SWITCH = os.getenv("KILL_SWITCH", "false").lower() == "true"

def estimate_tokens(s: str) -> int:
    return max(1, len(s) // 4)

@app.post("/v1/complete")
async def complete(p: Prompt):
    used = LEDGER.get(p.user_id, 0)
    if KILL_SWITCH:
        raise HTTPException(status_code=503, detail="route disabled")
    if used >= DAILY_BUDGET:
        raise HTTPException(status_code=429, detail="budget exhausted")
    if not MODEL_URL or not MODEL_KEY:
        raise HTTPException(status_code=503, detail="model route not configured")
    try:
        async with httpx.AsyncClient(timeout=30) as client:
            r = await client.post(
                MODEL_URL,
                json={"prompt": p.text, "user_id": p.user_id},
                headers={"Authorization": f"Bearer {MODEL_KEY}"},
            )
            r.raise_for_status()
            data = r.json()
    except httpx.TimeoutException:
        raise HTTPException(status_code=504, detail="model timed out")
    except httpx.HTTPStatusError as e:
        raise HTTPException(status_code=502, detail=f"model returned {e.response.status_code}")
    LEDGER[p.user_id] = used + estimate_tokens(p.text) + estimate_tokens(data.get("text", ""))
    return {"text": data["text"], "user_id": p.user_id}

Now the test that makes the proxy honest:

# test_contract.py
from fastapi.testclient import TestClient
from boring_proxy import app, LEDGER

def test_response_shape_is_stable():
    LEDGER.clear()
    r = TestClient(app).post("/v1/complete", json={"text": "hello", "user_id": "u1"})
    assert r.status_code == 200
    assert set(r.json()) == {"text", "user_id"}

def test_budget_exhaustion_returns_429_not_a_crash():
    LEDGER["u1"] = 10**9
    r = TestClient(app).post("/v1/complete", json={"text": "hello", "user_id": "u1"})
    assert r.status_code == 429

That second test is the one people skip, and it is the one that matters. A gatekeeper guards the tool layer, but the failures I actually debug are timeouts, exhausted budgets, and response shapes that drift after a provider update. The proxy turns all three into response codes a frontend can handle, which is exactly what a production incident needs.

This is where a free tier stops being a marketing line and becomes a staging environment. MonkeyCode's free model access and free server give you a place to run this proxy against a real model endpoint without opening a corporate card. Disclosure: This article was prepared as part of MonkeyCode's product outreach. At the time of writing, the free tier includes a token allowance (the current bucket is 10 million tokens) and a free server for exactly this kind of staging work, and the project itself is open source.

The workflow I actually recommend is short. Deploy the proxy to the free server with MODEL_URL and MODEL_KEY set, run the contract tests against the deployed URL, then shadow a few real requests before you enable any write path. If the provider changes the response shape, the contract test fails before a user ever sees the breakage, and the budget ledger keeps one runaway prompt from burning the whole allowance.

Who should not use this approach? If you are handling multi-tenant secrets, PHI, or PCI data, a shared free server is not your staging ground, so run the same proxy in your own VPC instead. Free tiers have rate limits and no SLA, which means the allowance is a staging resource rather than a production commitment. And the in-memory ledger resets on restart, so move it to Redis or Postgres before anything real depends on it.

The checklist I hand to every team that asks about agent safety:

[ ] Deploy the proxy to the free server with the model route configured
[ ] Run the contract test against the deployed URL, not just locally
[ ] Set DAILY_BUDGET below the free allowance so the 429 path is exercised
[ ] Shadow one real user request before enabling any write authority
[ ] Replace the in-memory ledger before production

The gatekeeper debate is fun, but shipping is boring, and boring is what survives an incident. Build the boring proxy first, put it on the free server, and let the contract test argue with the model instead of arguing with your middleware. If you want to see the pattern in action, MonkeyCode's free server is a fine place to start, though the discipline matters more than the vendor. Which handoff is least stable in your stack right now, and what response code does it actually return when it fails?

Stop Mocking the Model. Run CI Against the Free Tier Instead.

kongkong — Wed, 19 Aug 2026 18:45:59 +0000

Last month, a colleague's PR merged cleanly because every test mocked the model as returning perfect JSON. Then the real model wrapped its answer in markdown code fences on the first production request, and the parser died instantly. The post-mortem read like a confession: we had tested everything except the one thing that could actually fail. That is when I stopped believing that mocking a model is a testing strategy, and the free tier became the only honest test double available.

Here is the position I want to argue: free AI compute is not a production discount, and it is not a demo budget either. It is a CI test runner that happens to speak HTTP, and the teams that understand this are the ones whose AI features survive contact with reality. The teams that deploy free tiers to serve real traffic are building on a foundation that can vanish mid-request. The difference is not model quality; it is the honesty of the environment you test against.

MonkeyCode's open-source gateway and free server option fit this pattern because they give you a real endpoint with a real token allowance. Disclosure: This article was prepared as part of MonkeyCode's product outreach. As of this writing, the free tier includes a ten-million-token allowance and a deployable server, which is enough for thousands of CI runs but nowhere near enough for production traffic. That asymmetry is not a limitation; it is a specification for where the resource belongs.

The artifact I want to share is a test suite that treats the model as an external service with a contract, not as a function you can mock away. The first test verifies that the response schema matches what the frontend expects, the second enforces a latency budget, and the third deliberately exhausts the quota to confirm the error path is structured and parseable. None of these tests are possible with a mock, because a mock cannot produce a 429, a slow response, or a malformed JSON body.

# tests/test_ai_contract.py
import os
import time
import httpx
import pytest

GATEWAY_URL = os.environ.get("GATEWAY_URL", "http://localhost:8000")

@pytest.fixture(scope="session")
def client():
    with httpx.Client(base_url=GATEWAY_URL, timeout=20.0) as c:
        yield c

def test_response_schema_matches_frontend(client):
    payload = {
        "messages": [{"role": "user", "content": "Summarize this PR in one sentence."}],
        "max_tokens": 128,
    }
    r = client.post("/chat", json=payload)
    assert r.status_code == 200, r.text
    body = r.json()
    assert "choices" in body, f"missing choices key: {body.keys()}"
    content = body["choices"][0]["message"]["content"]
    assert isinstance(content, str) and len(content) > 0

def test_latency_stays_under_budget(client):
    start = time.monotonic()
    r = client.post("/chat", json={
        "messages": [{"role": "user", "content": "Reply with OK."}],
        "max_tokens": 16,
    })
    elapsed = time.monotonic() - start
    assert r.status_code == 200
    assert elapsed < 10.0, f"latency breach: {elapsed:.1f}s"

def test_quota_exhaustion_returns_structured_error(client):
    r = client.post("/chat", json={
        "messages": [{"role": "user", "content": "hi"}],
        "max_tokens": 10_000_000,
    })
    assert r.status_code in (429, 503)
    assert "retry" in r.text.lower() or "exhaust" in r.text.lower()

The CI workflow is where this pattern earns its keep, because it runs on every pull request without anyone remembering to do anything. You start the gateway as a service container, point the test suite at it, and let the free allowance absorb the cost of catching regressions early. The first time I wired this up, the suite caught a breaking prompt-template change within three minutes of the commit, which is faster than any code review I have ever attended.

# .github/workflows/ai-contract.yml
name: ai-contract
on:
  pull_request:
    paths: ["app/**", "prompts/**"]
jobs:
  contract:
    runs-on: ubuntu-latest
    services:
      gateway:
        image: ghcr.io/your-org/ai-gateway:latest
        ports: ["8000:8000"]
        env:
          PRIMARY_URL: ${{ secrets.PRIMARY_URL }}
          PRIMARY_KEY: ${{ secrets.PRIMARY_KEY }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements-dev.txt
      - run: pytest tests/ -v
        env:
          GATEWAY_URL: http://localhost:8000

Before you copy this pattern, your gateway needs to meet three contract requirements, and the workflow above assumes you have containerized it and pushed it to a registry your runner can pull from. It must translate provider errors into structured HTTP responses, it must expose a token counter that resets on a fixed window, and it must fail with a 503 when every provider is exhausted. If your gateway returns raw provider exceptions, the test suite will still pass on the happy path, and you will have learned nothing about your real failure modes.

Now the honest limitations, because this pattern is not for everyone. If your feature requires deterministic outputs, like a grading rubric or a financial summary, a live model in CI will flake, and you need a recorded-replay approach instead. If your organization blocks outbound traffic from CI runners, the free server becomes a manual step, and manual steps get skipped, and skipped tests are worse than no tests because they create false confidence.

The teams that should not use this pattern are the ones that need a guarantee, because a free tier provides none. You cannot promise a stakeholder that the model will respond within two seconds when the provider can throttle you at any moment. You also cannot promise that the server will be warm when the pipeline starts, so your tests must treat cold starts as a normal state. What you can promise is that your code handles every one of those failures gracefully, and that promise is worth more than any benchmark score.

Here is the checklist I run before wiring any free model tier into a pipeline. First, verify the gateway translates provider errors into structured responses with status codes your frontend understands. Second, confirm the test suite fails loudly when the model is down, so a green build actually means something. Third, set a per-run token budget so one pathological test cannot drain the monthly allowance, and fourth, document who owns the gateway configuration, because a shared endpoint with no owner is an incident waiting for a victim.

So the next time someone argues that free AI compute is too unreliable for real work, agree with them, and then ask why they are trying to serve production traffic from a test environment. The free tier is not a server that happens to be cheap; it is a test runner that happens to be free. The teams that understand that distinction are the ones whose AI features survive contact with the real world. If you want to see the failure modes yourself, point this test suite at MonkeyCode's free server and let the 429s teach you what your mocks have been hiding all along.

Migrate Models by Re-recording the Contract, Not by Re-running the Benchmark

kongkong — Wed, 19 Aug 2026 00:32:09 +0000

Last month I watched a friend switch an internal API from one model provider to another after a ten-minute look at two public benchmarks. By the second hour, the support channel filled with reports that the agent was returning malformed tool calls. The benchmark scores had differed by two points. The tool call contract had changed underneath us, and nobody had recorded the old contract before it disappeared.

A model migration is not a leaderboard race. It is a contract migration, and most teams skip the only step that would catch the breakage before production. They treat a provider switch as a performance decision, then discover that the word function now appears in a different JSON shape or that a tool name stopped being returned entirely. Performance benchmarks cannot see those failures because they are scored on generic tasks, not on your prompts, your tool definitions, or your output schema.

That is where MonkeyCode's free model access and free server option become relevant, not as a cheap place to run production traffic but as a record-and-replay bench. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The reported 30 million token free allocation is enough to record a representative set of requests and replay them against a candidate model before you pay a single invoice. You should not think of it as a discount on production work; you should think of it as the cost of building the contract fixture that your next migration will desperately need.

The argument I want to make is simple. Free infrastructure is wasted when it is used to run one more ad-hoc evaluation. It becomes valuable when it produces a persistent artifact that survives the model swap. That artifact is a small JSON Lines file containing the prompts you rely on, the expected presence of text, and the expected tool call structure. Once you have it, every future model change becomes a pass-or-fail test against your own behavior instead of a vibes-based comparison against a public benchmark.

Let me show the code I would run on a free server. The first script records a contract fixture by sending a set of prompts through the model and storing only the structural facts that matter for your integration. You do not need to store exact text because exact text is a poor contract; models legitimately rephrase answers. You need to store whether a text response exists and whether tool calls arrive in the shape your code expects.

import json
import os
import sys
from pathlib import Path
import httpx

BASE_URL = os.getenv('MODEL_BASE_URL', 'https://api.monkeycode.example')
API_KEY = os.getenv('MODEL_API_KEY')
MODEL = os.getenv('MODEL_NAME', 'free-model')

def load_prompts(path):
    return [json.loads(line) for line in Path(path).read_text().splitlines()]

def record_contract(prompts, output_path):
    with httpx.Client(base_url=BASE_URL, headers={'Authorization': f'Bearer {API_KEY}'}, timeout=60) as client:
        records = []
        for p in prompts:
            resp = client.post('/v1/chat/completions', json={
                'model': MODEL,
                'messages': [{'role': 'user', 'content': p['prompt']}],
                'tools': p.get('tools', []),
                'temperature': 0.0,
            })
            resp.raise_for_status()
            data = resp.json()
            message = data['choices'][0]['message']
            records.append({
                'prompt': p['prompt'],
                'expected_text_present': bool(message.get('content')),
                'expected_tool_call_count': len(message.get('tool_calls', [])),
                'expected_tool_names': [tc['function']['name'] for tc in message.get('tool_calls', [])],
            })
        Path(output_path).write_text('\n'.join(json.dumps(r, ensure_ascii=False) for r in records))

if __name__ == '__main__':
    record_contract(load_prompts(sys.argv[1]), sys.argv[2])

You can run this once against the free model with a set of prompts that exercise the corners of your app: a request that should not trigger a tool call, one that should trigger a single tool call, one that asks for multiple tools in sequence, and one that contains a deliberately malformed argument so the model must refuse or ask for clarification. That last case matters more than most people admit, because a contract is not only about happy paths. It is about how the model fails.

The second script verifies a candidate model against that saved contract. I deliberately keep the checks primitive: text presence, tool call count, and tool name equality. Those are the things your route handler likely depends on. If any check fails, you stop the migration before it reaches a user.

def verify_contract(contract_path, base_url, api_key, model):
    failures = []
    with httpx.Client(base_url=base_url, headers={'Authorization': f'Bearer {api_key}'}, timeout=60) as client:
        for line in Path(contract_path).read_text().splitlines():
            contract = json.loads(line)
            resp = client.post('/v1/chat/completions', json={
                'model': model,
                'messages': [{'role': 'user', 'content': contract['prompt']}],
                'temperature': 0.0,
            })
            resp.raise_for_status()
            message = resp.json()['choices'][0]['message']
            text_ok = bool(message.get('content')) == contract['expected_text_present']
            tool_count_ok = len(message.get('tool_calls', [])) == contract['expected_tool_call_count']
            tool_names_ok = [tc['function']['name'] for tc in message.get('tool_calls', [])] == contract['expected_tool_names']
            if not all([text_ok, tool_count_ok, tool_names_ok]):
                failures.append({'prompt': contract['prompt'], 'text_ok': text_ok, 'tool_count_ok': tool_count_ok, 'tool_names_ok': tool_names_ok})
    return failures

The beauty of this approach is that the contract fixture becomes a repeatable regression suite. When your prompt engineering changes, you re-record. When a provider deprecates a model, you verify the replacement before you route a single customer to it. When you onboard a new teammate, they can run the verification without needing access to production credentials or an explanation of tribal knowledge. That is what a contract does: it turns the implicit assumptions of an integration into a file that can be executed.

I need to be honest about the limits. A contract fixture built on a free model will not predict human-quality output, and it will not tell you whether the candidate model handles long-context reasoning better than the old one. Free-tier rate limits and latency are not representative of a paid production endpoint, so this is a functional test, never a load test. If your application depends heavily on fine-grained tone or style, structural equality tests will pass while your users still complain that the writing feels different. And if the free model was trained with different tool calling conventions, using it as the reference can hide defects that a production-grade model would expose. This is why the fixture should be re-recorded from the model currently serving your real traffic whenever possible.

The opinion I will defend is this: the expensive part of a model migration is not the token bill. It is the moment you discover that the new model changed the shape of a tool call after your route handler has already shipped. Free tokens are wasted if they are spent on another model battle. They are well spent if they buy you the contract file that catches that shape change in a controlled environment. If you have an internal tool that currently works with one model, do not start the next migration with a benchmark sweep. Start by recording a live contract on the free tier, then replay it against every candidate before you let a single request through. MonkeyCode's free model route and server option make that first recording possible without a credit card, and the reported 30 million token allocation is enough for a weekend of record-and-replay work. The next time a provider switch breaks your integration, the failure report should land in your terminal, not in the support channel.

Your Free Token Quota Is a Contract Sandbox, Not a Production Discount

kongkong — Mon, 17 Aug 2026 23:33:06 +0000

A few weeks ago I watched a small team wire an internal support agent to a free model endpoint and then skip the authorization check because, in their words, the tokens cost nothing. By the end of the afternoon the agent was retrying write requests into a staging database, and the only thing that stopped it was a missing column on the target table. The tokens were free; the near-miss was not.

Free model access changes the cost conversation, but it does not remove the need for a boundary. If anything, a free tier is the ideal place to make that boundary explicit and test it before any real data is involved. That is why I keep treating free model access as a contract sandbox rather than a hosting discount.

I have been looking at MonkeyCode's open-source project, which offers free model access and a free server option at the time of writing. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The stated free allowance is 30 million tokens, but you should verify current terms before you build any billing or capacity assumption on them.

My argument is not that free tiers are unsafe. My argument is that they are too often treated as permission to skip the exact controls that make them safe. The token price can be zero and the blast radius can still be enormous.

The failure starts at the handoff

When I trace an AI feature from UI to storage, the first unstable handoff is usually not the model itself. It is the point where a model output crosses into a side effect: a database write, a file upload, an email, or a provider retry. A free quota can hide that boundary because the immediate billing signal is zero, so nobody feels the cost of an unbounded agent path.

The free server option is a good place to make that boundary explicit. I want a local gate that rejects work before it reaches a write path, records which owner consumed what, and stops retrying transient provider failures on state-changing routes. That is a small piece of infrastructure, but it changes the conversation from saving money to enforcing a contract.

A contract gate with teeth

Here is a deliberately small Node.js gate that I can run on any machine. It does not depend on a specific provider, so the same adapter works whether I point it at a local mock, a paid endpoint, or MonkeyCode's free server. The contract stays on my side.

import { createServer } from 'node:http';
import { readFileSync, writeFileSync } from 'node:fs';

const BUDGET_TOKENS = 10_000;
const LEDGER_FILE = './ledger.json';

function loadLedger() {
  try {
    return JSON.parse(readFileSync(LEDGER_FILE, 'utf8'));
  } catch {
    return { owners: {} };
  }
}

function saveLedger(ledger) {
  writeFileSync(LEDGER_FILE, JSON.stringify(ledger, null, 2));
}

function remainingFor(ledger, owner) {
  return BUDGET_TOKENS - (ledger.owners[owner]?.tokens ?? 0);
}

function guard(owner, requestedTokens) {
  const ledger = loadLedger();
  const remaining = remainingFor(ledger, owner);
  if (requestedTokens > remaining) {
    throw new Error(`owner ${owner} exceeds contract budget`);
  }
  ledger.owners[owner] = {
    tokens: (ledger.owners[owner]?.tokens ?? 0) + requestedTokens,
    lastUsedAt: new Date().toISOString(),
  };
  saveLedger(ledger);
  return { owner, requestedTokens, remaining: remaining - requestedTokens };
}

const server = createServer((req, res) => {
  const owner = req.headers['x-owner'] ?? 'anonymous';
  const requestedTokens = Number(req.headers['x-tokens'] ?? 0);

  if (req.method === 'POST' && req.url === '/v1/chat') {
    try {
      const ticket = guard(owner, requestedTokens);
      // In a real adapter this is where the provider call happens.
      // If the provider returns 429 or a write path is involved, do not retry blindly.
      res.writeHead(200, { 'content-type': 'application/json' });
      res.end(JSON.stringify({ ok: true, ticket }));
    } catch (err) {
      res.writeHead(429, { 'content-type': 'application/json' });
      res.end(JSON.stringify({ ok: false, error: err.message }));
    }
    return;
  }

  res.writeHead(404).end();
});

server.listen(8787, () => console.log('contract gate on 8787'));

Run it with node contract-gate.mjs and then make two requests. The first one should be refused before any model provider is called:

curl -i -X POST http://localhost:8787/v1/chat -H 'x-owner: staging-agent' -H 'x-tokens: 12000'

The second one should pass and leave a small ledger entry:

curl -i -X POST http://localhost:8787/v1/chat -H 'x-owner: staging-agent' -H 'x-tokens: 100'

This gate is not protecting the provider's free quota. It is protecting the system behind the model by forcing every request to declare an owner and fit inside a budget before it can continue. A free server is a rehearsal room, but a rehearsal room still needs a stage manager.

What I would not do with a free tier

I would not store production secrets in a free model sandbox, retry failed write paths merely because the first attempt returned a transient 429, or treat a stated token allowance as a standing production capacity plan. A free tier is a rehearsal stage, and a rehearsal stage is a terrible place to run payroll.

The local gate also has real limits. It trusts the caller-supplied owner header, so it will not stop a malicious caller from lying. It enforces only a token budget, not prompt injection, tool authorization, or response validation. If your use case involves protected data, this is not a substitute for server-side identity and permissions.

Developers who need production reliability should not point a free server at user-facing traffic and call it a launch. Teams that cannot version their model contracts will still have the same failure, just with cheaper tokens. The free tier is most useful when you treat it as a controlled rehearsal environment, not as a discount that deletes architectural work.

If you are already using MonkeyCode's free server, try the gate on the next small integration and ask yourself which handoff is least stable when the model oversteps its budget. That question costs less than the cleanup.

Shadow Traffic Is the Only Honest Free Model Evaluation

kongkong — Mon, 17 Aug 2026 17:12:28 +0000

The demo replay played in the dim glow of a conference room, and everyone agreed the free model sounded good. The same four prompts produced crisp summaries, polite error handling, and no obvious hallucination, so the conversation drifted toward replacing our paid endpoint before lunch. I sat through it with an uncomfortable question that nobody asked: why were we trusting a handful of canned examples in front of a projector instead of letting the model read the messy, malformed, real-world input our production system receives every day? A free model can charm you in a controlled slide deck and still fall apart the moment a user sends a 90-line stack trace with a question mark.

That is why I now insist on shadow traffic before a free model gets promoted to any real path. MonkeyCode is an open-source project that currently advertises a free tier of 30 million tokens and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The offer is useful not because it replaces a paid provider, but because those free tokens and the free server give you enough room to send real production requests through the model without ever letting its answer reach a user.

A shadow deployment means copying a fraction of live requests to a secondary model, recording what it returns, and comparing that against the primary model's behavior. You are not evaluating the free model; you are evaluating the difference between the free model and the thing your customers already tolerate. The free model gets authentic traffic, including the gibberish, the half-finished prompts, and the user who pastes a full database schema into a text field. The production path never pauses, the user never notices, and you finally have a data set that can say whether the free model is a viable alternative or just a good demo actor.

The easiest way to implement this is a small middleware that duplicates the request into two executions. One call goes to the primary model and returns to the user as usual; the other call goes to the free model, waits for its response in the background, and writes both responses to a sidecar log or database. The key is that the shadow call must not touch the response that reaches the client, and it must share the same request identifier so you can pair the two outputs later.

import asyncio
import json
from dataclasses import dataclass
from datetime import datetime, timezone

@dataclass
class ShadowResult:
    request_id: str
    primary_response: str
    shadow_response: str
    primary_latency_ms: int
    shadow_latency_ms: int
    shadow_error: str | None
    recorded_at: str

class ShadowRecorder:
    def __init__(self, primary, shadow, store) -> None:
        self.primary = primary
        self.shadow = shadow
        self.store = store

    async def generate(self, request_id: str, prompt: str, max_tokens: int) -> str:
        start = asyncio.get_event_loop().time()
        primary_response = await self.primary.generate(prompt, max_tokens)
        primary_ms = int((asyncio.get_event_loop().time() - start) * 1000)

        async def shadow_task():
            shadow_start = asyncio.get_event_loop().time()
            shadow_response = None
            shadow_error = None
            try:
                shadow_response = await self.shadow.generate(prompt, max_tokens)
            except Exception as exc:
                shadow_error = type(exc).__name__
            shadow_ms = int((asyncio.get_event_loop().time() - shadow_start) * 1000)
            result = ShadowResult(
                request_id=request_id,
                primary_response=primary_response,
                shadow_response=shadow_response or "",
                primary_latency_ms=primary_ms,
                shadow_latency_ms=shadow_ms,
                shadow_error=shadow_error,
                recorded_at=datetime.now(timezone.utc).isoformat(),
            )
            await self.store.write(result)

        asyncio.create_task(shadow_task())
        return primary_response

The primary response returns immediately, while the shadow comparison happens in a task that is deliberately fire-and-forget. The free server may fail, time out, or produce an empty result, and none of that matters to the user. It only matters to the data you collect. That data becomes the basis for a decision you can finally defend in a meeting.

After a week of shadow traffic, the evaluation is no longer a matter of opinion. You can count the number of times the free model produced a materially different answer from the primary, the number of times it failed outright, the average latency difference, and the distribution of token consumption. The honest conversation moves from “it felt fine” to “it disagreed with our current model on 14 percent of support-ticket summaries and failed twice as often under burst traffic.” Those numbers do not tell you whether the free model is good in an absolute sense; they tell you whether it is good enough to sit next to the thing you already run.

I also record a small set of hash-based fingerprints for each response so the sidecar storage can be scanned for exact duplication and for suspiciously short outputs. A model that matches the primary on most prompts but melts into a single vague sentence on five percent of requests deserves a different kind of scrutiny. The free tokens are useful here because they let you run this side-by-side on real traffic for days without a surprise invoice.

The biggest mistake in this entire exercise is treating the shadow model's nice output as permission to switch. A free model in shadow traffic is still invisible; it has not experienced a cold start, a retry storm, a schema change, or a customer support ticket. The next step is a gradual release that begins with a read-only canary or an internal dashboard before any write path is involved, but that is a separate conversation about deployment rather than evaluation. Shadow traffic answers the question “is this model any good?” It does not answer the question “can this model survive my infrastructure?”

There are clear limits to what a shadow comparison can teach you. Shadow calls are asynchronous, so the recorded latency includes the overhead of the background task and may not represent the true end-to-end delay a user would experience if the free model were primary. The sample is biased toward whatever traffic your production system sends, which means the free model may shine on your current user base and still fail on a new market segment. The evaluation metrics also have blind spots: token overlap and failure rates do not capture subtle semantic drift, tone, safety, or the quality of references. You still need a separate human review process for meaning, not just structure.

You should not run shadow traffic if your production data contains PII, medical records, or financial details that you have not legally cleared for a second provider. A free server is not a license to ignore the boundaries your data-protection team already enforced, and copying sensitive prompts into a new model's log is exactly the kind of quiet expansion that creates a compliance incident. You should also skip this method if your team refuses to store shadow responses as first-class artifacts; invisible data that no one ever reviews is not evaluation, it is waste.

MonkeyCode's 30 million free tokens and free server option are enough to run a rigorous shadow evaluation on real traffic before anyone has to commit. The point is not to save a few dollars on the next invoice. The point is to stop making infrastructure decisions in front of a projector, where every model sounds perfect and every failure is invisible. So before you promote that charming free model, ask your team one question: are we evaluating it on real requests, or are we just voting on a demo?

Try MiniMax H3 Without Hardcoding It Into Your Free Server

kongkong — Mon, 17 Aug 2026 12:30:23 +0000

When MiniMax H3 started showing up in my feed last week, I did the exact thing I warn other developers not to do. I copied a snippet for an OpenAI-compatible endpoint, pasted it into a route, and called it a day. The demo worked for a few hours, but the moment another service wanted the same model, the configuration was already scattered across two repositories and a Jupyter notebook. That is the real cost of a hot model release: the model itself is easy, and the integration is what slowly becomes unmanageable.

I do not have any privileged information about MiniMax H3's weights or published benchmarks, so I treated it as an unknown provider with an OpenAI-compatible surface. That is not skepticism; it is the only sane way to evaluate a new model without rewriting the application later. The first thing I wanted was a small provider seam that could hold the model name, endpoint, and timeout in one place. A seam sounds like architecture jargon, but it is really just a boring boundary that lets you swap one implementation for another without disturbing the callers.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used MonkeyCode's free model access and free server option as the deployment target for this experiment, because a free server is exactly where rushed integrations tend to accumulate. When nobody has to provision hardware to try a promising model, the convenience can hide the fact that every client now owns a slightly different configuration. The adapter I am about to show does not care which vendor sits behind it, so the experiment remains reversible.

The core idea is to represent a model call as a small value object and hide the HTTP details behind a protocol. In Python, a protocol is just a shape that any provider implementation can satisfy. I started with a request dataclass that carries the prompt and generation parameters, because those are the only fields the rest of the application should know about. Then I wrote a thin client for any OpenAI-compatible endpoint, since that pattern covers the MiniMax H3 snippets I was seeing without making me trust a single vendor forever.

from dataclasses import dataclass
from typing import Protocol
import httpx

@dataclass(frozen=True)
class ModelRequest:
    prompt: str
    max_tokens: int = 256
    temperature: float = 0.2

class Provider(Protocol):
    def complete(self, request: ModelRequest) -> str:
        ...

class OpenAICompatibleProvider:
    def __init__(self, base_url: str, api_key: str, model: str):
        self.base_url = base_url.rstrip("/")
        self.api_key = api_key
        self.model = model

    def complete(self, request: ModelRequest) -> str:
        response = httpx.post(
            f"{self.base_url}/chat/completions",
            headers={"Authorization": f"Bearer {self.api_key}"},
            json={
                "model": self.model,
                "messages": [{"role": "user", "content": request.prompt}],
                "max_tokens": request.max_tokens,
                "temperature": request.temperature,
            },
            timeout=30,
        )
        response.raise_for_status()
        return response.json()["choices"][0]["message"]["content"]

This is deliberately small, and that is the point. The adapter does not solve prompt engineering, output validation, or cost accounting; it only ensures that the rest of the service can call a model without knowing where the model lives. I have seen teams try to make this layer smarter by adding retries and caching, but that usually turns a clean boundary into a junk drawer. The first version should be boring enough to read in one pass.

Once the provider client exists, the FastAPI route becomes almost trivial. The application accepts a prompt, constructs a ModelRequest, and delegates to whatever provider was configured at startup. That means a developer can test MiniMax H3 on a free server without pasting credentials into every service that needs generated text. I ran the route locally with a single command, then pointed a temporary deployment at the same code without changing the route itself.

from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()
provider = OpenAICompatibleProvider(
    base_url="https://your-provider.example/v1",
    api_key="replace-me",
    model="your-model-name",
)

class Prompt(BaseModel):
    text: str

@app.post("/complete")
def complete(prompt: Prompt):
    return {"text": provider.complete(ModelRequest(prompt=prompt.text))}

The command I used locally was uvicorn main:app --host 0.0.0.0 --port 8000, then I exercised the route with a curl request. The response itself was less interesting than the fact that I could swap the base URL and model string by changing environment variables instead of editing application code. That is the difference between evaluating a model and adopting a model.

curl -s http://127.0.0.1:8000/complete -H 'Content-Type: application/json' -d '{"text":"Explain the value of a provider seam in a model deployment."}'

The next failure I wanted to catch was not a slow response; it was a silent ownership problem. When a model route lives behind a seam, you can ask a precise question before enabling write access: who is allowed to change the provider configuration, and how would you know if a change broke the service? I added a tiny health check that returns the configured model name, not because it proves the model works, but because it makes drift visible. If someone points the adapter at a different model, the health check tells you which one is actually serving traffic.

@app.get("/health")
def health():
    return {"status": "ok", "model": provider.model}

The open-source spirit I care about here is not a license on a particular repository. It is the permission to test a new model without forking my application, and to leave the seam in place after the trend fades. MonkeyCode's free model access and free server option fit that spirit because they lower the cost of running a reversible experiment, not because they promise a permanent home for every model I try. The code I wrote still works if I move the provider behind a paid gateway tomorrow, and that independence is what keeps the integration honest.

This approach is not for everyone. If you only need a model for one evening of prompt testing, a notebook is faster and an adapter is overhead. If you are evaluating a provider that requires fine-grained permissions or data residency checks, the adapter does not replace reading the terms and controlling where prompts are sent. And if your application already has a stable model gateway, adding a second seam can create two sources of truth.

The question I would ask before adopting any hot model is simpler than it sounds: which layer would break first if you changed providers this afternoon? For me it was configuration, because the endpoint had leaked into too many places. The adapter fixed that first, and then the actual MiniMax H3 evaluation became a configuration change rather than a migration.

Put the Free Token Quota Behind a Per-Project Ledger, Not a Shared Counter

kongkong — Mon, 17 Aug 2026 10:34:31 +0000

A 30M-token quota looks generous until five people share it. At 2,000 tokens per call and 200 calls per person per day, the quota lasts 15 working days. The first failure is not the model or the server; it is the shared counter that cannot tell you which feature consumed 80% of the budget. That is why you should not build a quota tracker that increments a single number. Build a ledger that attributes every token to a project, feature, and caller before the team starts treating the free tier as unlimited.

I am evaluating the MonkeyCode open-source project's free model access and free server option, which includes a 30M-token quota. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The quota and free-server terms are operator-supplied and can change; verify current limits before building around them.

The current AI news cycle is full of watermarking debates and agent gatekeepers, but the mechanical failure many teams hit is quieter: nobody knows which feature burned the shared quota. The token total does not answer that. A ledger does.

The shared counter hides the cost of each feature

A single integer that decreases on every API call is easy to implement and useless to debug. It tells you how much is left, not what consumed it. When the quota hits zero, the team is left arguing about which feature was responsible. That argument is expensive: it happens after the outage, under time pressure, and with no data.

A free tier is not a blank check. It is a finite shared resource with an invisible price: every under-tested feature, retry loop, or exploratory script draws from the same pool. Without attribution, the most aggressive consumer wins by default, and the most careful teammate gets blamed when the meter stops.

The fix is not a better dashboard. It is a schema that records every token against a project, a feature, and a caller. Once those columns exist, the team can answer cost questions before the quota is gone.

A ledger is a schema, not a dashboard

Start with a SQLite table. In a local or development environment, SQLite is enough. In a production gateway, replace it with a log sink like ClickHouse or a managed analytics store.

CREATE TABLE token_usage (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    occurred_at TEXT NOT NULL,
    project TEXT NOT NULL,
    feature TEXT NOT NULL,
    caller TEXT NOT NULL,
    model TEXT NOT NULL,
    prompt_tokens INTEGER NOT NULL,
    completion_tokens INTEGER NOT NULL,
    total_tokens INTEGER NOT NULL,
    status_code INTEGER NOT NULL,
    latency_ms INTEGER NOT NULL DEFAULT 0
);

CREATE INDEX idx_token_usage_project_time
    ON token_usage(project, occurred_at);

CREATE INDEX idx_token_usage_feature_time
    ON token_usage(feature, occurred_at);

The key design choice is to store project, feature, and caller as explicit columns. Do not store a single label string and parse it later. Parsing a comma-separated column in a hot path leads to the same blame game you were trying to avoid.

After each API call, insert one row. The snippet below wraps a generic chat completion function so every call is recorded without forcing the caller to remember bookkeeping.

import sqlite3
import time
import json
import os

LEDGER_DB = os.environ.get('LEDGER_DB', 'token_usage.db')

def record_token_usage(record: dict) -> None:
    conn = sqlite3.connect(LEDGER_DB)
    conn.execute(
        """
        INSERT INTO token_usage (
            occurred_at, project, feature, caller, model,
            prompt_tokens, completion_tokens, total_tokens,
            status_code, latency_ms
        ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
        """,
        (
            time.strftime('%Y-%m-%dT%H:%M:%SZ', time.gmtime()),
            record.get('project', 'unknown'),
            record.get('feature', 'unknown'),
            record.get('caller', 'unknown'),
            record.get('model', 'unknown'),
            record.get('prompt_tokens', 0),
            record.get('completion_tokens', 0),
            record.get('total_tokens', 0),
            record.get('status_code', 0),
            record.get('latency_ms', 0),
        )
    )
    conn.commit()
    conn.close()

# Example of a wrapped provider call
def chat_completion(project: str, feature: str, caller: str, messages: list, max_tokens: int = 256):
    started = time.perf_counter()
    # Replace this with the actual provider request from the MonkeyCode free server
    # The response body should include usage.prompt_tokens, usage.completion_tokens, usage.total_tokens
    response = {  # illustrative stub
        'status_code': 200,
        'usage': {'prompt_tokens': 120, 'completion_tokens': 80, 'total_tokens': 200}
    }
    latency_ms = int((time.perf_counter() - started) * 1000)
    record_token_usage({
        'project': project,
        'feature': feature,
        'caller': caller,
        'model': 'issued-model',
        'prompt_tokens': response['usage']['prompt_tokens'],
        'completion_tokens': response['usage']['completion_tokens'],
        'total_tokens': response['usage']['total_tokens'],
        'status_code': response['status_code'],
        'latency_ms': latency_ms,
    })
    return response

Do not wait for the quota to run out before adding this. Add the schema and the record_token_usage call in the first pull request that introduces the free route. If you add it later, the early consumption data is already lost.

Policy questions the ledger can answer

Once the rows exist, a few SQL queries turn the shared counter into a burn-down report.

-- Which project consumed the most tokens in the last 7 days?
SELECT project, SUM(total_tokens) AS tokens
FROM token_usage
WHERE occurred_at >= datetime('now', '-7 days')
GROUP BY project
ORDER BY tokens DESC;

-- Which feature had the highest error rate in the last 24 hours?
SELECT feature,
       SUM(CASE WHEN status_code >= 400 THEN 1 ELSE 0 END) AS errors,
       COUNT(*) AS calls,
       ROUND(100.0 * SUM(CASE WHEN status_code >= 400 THEN 1 ELSE 0 END) / COUNT(*), 2) AS error_pct
FROM token_usage
WHERE occurred_at >= datetime('now', '-1 day')
GROUP BY feature
ORDER BY error_pct DESC;

-- Daily burn rate for the last 14 days
SELECT date(occurred_at) AS day, SUM(total_tokens) AS tokens
FROM token_usage
WHERE occurred_at >= datetime('now', '-14 days')
GROUP BY day
ORDER BY day;

The first query answers the blame question. The second catches a failing feature that is burning tokens on retries. The third lets you forecast when the quota will run out: divide the remaining tokens by the average daily burn.

The table below summarizes four common allocation policies you can enforce with the ledger data. Pick one before the quota becomes scarce.

Allocation policy	How it works	Best for	Failure mode if ignored
Equal per project	Cap each project at a fixed share of the quota	Teams with several independent prototypes	One project starves the rest
Priority reserved	Keep a reserve for high-value features; let the rest share the remainder	A single flagship feature plus experiments	The flagship hits zero first
Feature flag limit	Pause low-priority features at a threshold	Teams that ship behind flags	No kill switch exists at the threshold
Time-window budget	Allow a fixed token amount per day, roll over unused	Batch jobs or CI runs	A weekend spike exhausts the window

The policy is not the point. The point is that the ledger gives you the data to choose a policy instead of guessing.

Limits and who should not use this

The free server is likely a shared, best-effort slot. A clean ledger does not make the provider stable. Quotas and free-server availability can change, so do not treat the ledger as committed capacity. Do not send private data. Do not grant write authority based on token attribution alone; correctness and permissions need their own checks.

Skip this approach if you need p99 below two seconds for user-facing workloads, if the route handles regulated data, or if your team cannot tolerate provider terms changing. Use a committed endpoint or a self-hosted model instead. The free server is most useful for read-only shadow traffic, contract tests, and failure-state discovery before you pay for committed capacity.

If your team is sharing a free token quota, stop tracking it as one number. Add a project and feature column to the first few calls, run the weekly query, and see which feature consumes the most. That observation is more actionable than the headline quota.

Right-Size an Internal Model Route by Requests per User Before You Provision the Server

kongkong — Sun, 16 Aug 2026 21:55:10 +0000

30,000,000 tokens breaks down to 1,000,000 tokens per day. At a routed generation of 2,800 tokens, that is 357 requests per day. Assign those requests across a 20-person product team and each person gets about 17 calls per day. That number, rather than the dashboard balance, decides whether an internal assistant can be rolled out to the whole team or remains a single-user experiment.

This is an overlooked layer in the current agent-tool conversation. Teams are building permission gates for tools, but many skip the capacity gate underneath: can the free model route support the actual user population without the first owner spending the entire allowance by Thursday? MonkeyCode's open-source project advertises a free model route and a free server option. The outreach brief for this article states that the current free tier includes a 30,000,000 monthly token allowance. I treat the 30M number as an operator-supplied claim to verify in the dashboard before relying on it. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The method here converts a token balance into requests per user, exposes that arithmetic as a small API, and uses the free server option as the host for that API. The model route remains free for generations; the calculator consumes no tokens.

Why a token balance hides the real constraint

A dashboard balance answers “how much is left.” It does not answer “can this team use the feature without one power user exhausting the shared allowance.” The difference matters when the route moves from a single developer's script to a shared internal service.

Planning view	Question it answers	Blind spot
Provider token balance	How many tokens remain in the cycle?	Request size variance, per-person distribution, peak demand
Requests per user per day	Can this group adopt the route?	Prompt drift toward longer inputs
Per-user quota policy	Who is blocked first when capacity is exhausted?	Shared-key workarounds
Synthetic workload on a candidate endpoint	How does it behave under modeled demand?	Real usage that diverges from the fixture

The free server option does not resolve those blind spots by itself. It gives you a place to run the deterministic budget API so the whole team sees the same decision instead of each person doing napkin math.

Convert the allowance into a repeatable calculator

The arithmetic is small enough to get wrong in a meeting. I move it into a script with four inputs: monthly tokens, users, tokens per request, and the desired requests per user per day. The output says whether the requested workload fits.

import argparse

def parse_args():
    parser = argparse.ArgumentParser()
    parser.add_argument('--monthly-tokens', type=int, default=30_000_000)
    parser.add_argument('--users', type=int, required=True)
    parser.add_argument('--tokens-per-request', type=int, default=2800)
    parser.add_argument('--requests-per-user-day', type=int, default=8)
    return parser.parse_args()

def main():
    args = parse_args()
    days = 30
    daily_requests = args.monthly_tokens / days / args.tokens_per_request
    per_user = daily_requests / args.users
    requested_tokens = args.users * args.requests_per_user_day * args.tokens_per_request * days
    headroom = args.monthly_tokens - requested_tokens
    print(f'daily_requests={daily_requests:.1f}')
    print(f'budget_per_user_per_day={per_user:.1f}')
    print(f'requested_tokens={requested_tokens:,}')
    print(f'headroom_tokens={headroom:,}')
    print(f'viable={requested_tokens <= args.monthly_tokens}')

if __name__ == '__main__':
    main()

Run it for a 20-person team at 10 calls per person per day:

python capacity_budget.py --users 20 --requests-per-user-day 10 --tokens-per-request 2800

The tool returns viable=True with headroom_tokens=13,200,000. That number is useful only because the input workload was made explicit. Once the workload is explicit, the team can negotiate which variable changes: fewer users, shorter prompts, or a lower calls-per-day target.

Expose the same math as an API on the free server

The command-line version works for one engineer. The API version works for an engineering group that keeps asking “can we add another integration?” If the free server plan allows a small Python process, deploy this FastAPI endpoint there:

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel

class Budget(BaseModel):
    monthly_tokens: int = 30_000_000
    users: int
    tokens_per_request: int = 2800
    requests_per_user_day: int = 8
    days: int = 30

app = FastAPI()

@app.post('/budget')
def budget(body: Budget):
    daily_requests = body.monthly_tokens / body.days / body.tokens_per_request
    per_user = daily_requests / body.users
    requested_tokens = body.users * body.requests_per_user_day * body.tokens_per_request * body.days
    viable = requested_tokens <= body.monthly_tokens

    return {
        'daily_requests': round(daily_requests, 1),
        'per_user_per_day': round(per_user, 1),
        'requested_tokens': requested_tokens,
        'headroom_tokens': body.monthly_tokens - requested_tokens,
        'viable': viable,
    }

A colleague can ask the same question without opening a spreadsheet:

curl -X POST https://your-free-server/budget \
  -H 'Content-Type: application/json' \
  -d '{"users":20,"requests_per_user_day":10,"tokens_per_request":2800}'

This endpoint never calls the model route, so it does not spend generation tokens. If the request is not viable, return 429 from this endpoint or from the gateway so a bad rollout request fails before it burns the shared allowance:

if not viable:
    raise HTTPException(status_code=429, detail='requested workload exceeds free allowance')

Use a scenario table to catch capacity cliffs

The most dangerous scenario is not the one that obviously fails. It is the one that fits by a small margin. A 4% headroom plan will fail under prompt drift, reranking, or a teammate turning on a new tool.

Scenario	Users	Tokens/request	Calls/user/day	Monthly tokens	Fits 30M?	Headroom
Small eng team	6	2,800	8	4,032,000	yes	25,968,000
Full product team	20	2,800	8	13,440,000	yes	16,560,000
Heavy usage	20	2,800	20	33,600,000	no	-3,600,000
High-turnover support	50	1,400	10	21,000,000	yes	9,000,000

The full product team looks comfortable until someone moves from short classification prompts to longer summarization prompts. If that team's average request grows from 2,800 to 3,800 tokens, the same call schedule becomes 18,240,000 tokens per month. That still fits but shaves more than 4.5M tokens off the headroom. The scenario table should be rerun every time a prompt changes shape, not once at project setup.

Limitations and who should skip

A monthly token allowance is a current claim, not a permanent capacity target. Verify the number in the provider dashboard and do not build a hard quota from a third-party summary. Request sizes should come from measured percentiles, not the optimistic average; I use the 90th percentile tokens per request in the calculator when available.

The free server option may impose idle limits or cold starts, so verify that the plan can host a small always-available API before making it the source of truth for rollout decisions. If it cannot, the same endpoint can run locally or in a CI job. The value is the shared arithmetic, not the host.

Skip this approach if you have dedicated billing, per-project budgets, and an existing cost model. Skip it too when the free tier is a temporary learning sandbox rather than a shared production dependency. The calculator is for teams deciding whether a free route can become a shared internal service without silently exhausting the allowance.

At what user count did your free route stop looking free: ten, twenty, or the first time someone integrated it into a batch job?

Measure a Free Token Server by Its Tail Latency, Not Its Ceiling

kongkong — Sun, 16 Aug 2026 15:30:01 +0000

A 30M-token allowance sounds enormous until you divide it by your observed output length. At 200 tokens per generated response the ceiling is roughly 150,000 requests; at 800 tokens it is about 37,500. On a free server, the more useful number is tail latency under queuing, because a generous quota does not prevent connection stalls, 429s, or truncated streams. MonkeyCode advertises free model access, a free server tier, and an open-source project path that removes per-call cost while you measure this. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Start with the capacity you will actually consume

Output tokens per request	Requests before a 30M-token ceiling
100	300,000
200	150,000
500	60,000
800	37,500
1,200	25,000

Token ceilings are a procurement number. Your route has a different request ceiling once chunking, RAG context, function-call schemas, and follow-ups are included. Record the average prompt and completion tokens from 50 representative calls in your integration, not the vendor's sample. That average determines whether you are testing a route that can absorb 25,000 user turns or 300,000 user turns.

The test harness

Point this at any OpenAI-compatible streaming endpoint. It issues REQUESTS calls with CONCURRENCY workers and records first-token time, total duration, status, and streamed chunk count. This is the artifact; run it before and after changing route code.

import os, json, statistics, time
from concurrent.futures import ThreadPoolExecutor
import httpx

BASE_URL = os.environ['MODEL_BASE_URL'].rstrip('/')
API_KEY = os.environ['MODEL_API_KEY']
MODEL = os.environ['MODEL_NAME']
CONCURRENCY = int(os.environ.get('CONCURRENCY', '8'))
REQUESTS = int(os.environ.get('REQUESTS', '40'))

def one_call(i):
    started = time.perf_counter()
    first = None
    chunks = 0
    usage = None
    with httpx.stream(
        'POST',
        f'{BASE_URL}/chat/completions',
        headers={'Authorization': f'Bearer {API_KEY}'},
        json={
            'model': MODEL,
            'messages': [{'role': 'user', 'content': 'List five ways to reduce duplicate API calls.'}],
            'max_tokens': 128,
            'stream': True,
        },
        timeout=60,
    ) as r:
        status = r.status_code
        if status >= 400:
            return {'status': status, 'ttft_ms': None, 'duration_ms': (time.perf_counter() - started) * 1000, 'chunks': 0, 'usage': r.text[:200]}
        for line in r.iter_lines():
            if not line.startswith('data:'):
                continue
            payload = line[5:].strip()
            if payload == '[DONE]':
                break
            try:
                event = json.loads(payload)
            except json.JSONDecodeError:
                continue
            if first is None:
                first = time.perf_counter()
            chunks += 1
            if event.get('usage'):
                usage = event['usage']
        return {
            'status': status,
            'ttft_ms': (first - started) * 1000 if first else None,
            'duration_ms': (time.perf_counter() - started) * 1000,
            'chunks': chunks,
            'usage': usage,
        }

with ThreadPoolExecutor(max_workers=CONCURRENCY) as pool:
    results = list(pool.map(one_call, range(REQUESTS)))

errors = [r for r in results if r['status'] >= 400]
ok = [r for r in results if r['status'] < 400]
ttfts = sorted(r['ttft_ms'] for r in ok if r['ttft_ms'] is not None)
durations = sorted(r['duration_ms'] for r in ok)

def pct(xs, p):
    if not xs:
        return None
    k = min(len(xs) - 1, int(len(xs) * p))
    return round(xs[k], 1)

print(json.dumps({
    'requests': len(results),
    'error_rate': round(len(errors) / len(results), 4),
    'p50_ttft_ms': pct(ttfts, 0.50),
    'p95_ttft_ms': pct(ttfts, 0.95),
    'p95_duration_ms': pct(durations, 0.95),
    'chunks_per_request': round(statistics.mean(r['chunks'] for r in ok), 1) if ok else 0,
}, indent=2))

A streamed chunk is not necessarily a token; treat chunk count as a proxy for progress and prefer the provider's final usage object when present.

Read the two numbers that actually block a rollout

Signal	Measured value	Example fail threshold	Why it matters
Rejections	HTTP 429/5xx error rate	>2%	Shared free infrastructure often queues requests as rate limits
Time to first token	p95	>4 seconds	Users abandon before the token ceiling matters
Total duration	p95	>15 seconds	Long streams hold worker slots and amplify retries
Truncation	chunks per request	stopping before [DONE]	Proxy or streaming bugs can silently end output

These thresholds are starting controls, not vendor guarantees. Keep the actual numbers in the PR alongside a date, region, concurrency, and model identifier so later failures are attributable to a change instead of a vibes-based regression.

Decision table for the read-only slice

Observed behavior	Action
p95 TTFT <=4s and error rate <=2%	Keep the route as a read-only shadow and begin adding persistence
p95 TTFT 4-8s or error rate 2-5%	Add client-side timeout and one idempotent retry, but do not grant write authority
p95 TTFT >8s or error rate >5%	Reduce concurrency to 2; if it does not recover, use the free tier only as a load-test substrate
Any 5xx on a write path without an idempotency key	Stop and fix idempotency before merging

What this does not prove

The harness produces a point-in-time tail-latency report. It does not prove correctness, data safety, capacity guarantees, or that a free server will behave the same way next week. Free capacity can be rebalanced, rate limits can shift, and queue depth can change with tenant load. Pair this artifact with a golden-set evaluation for quality and a permission check before any write. Do not use a free server for regulated data, stable customer-facing traffic, or any payload where a truncated answer can cause harm. If you need a contractual SLO or isolated throughput, a paid endpoint is the correct boundary; the free tier is for validation and controlled slices.

Who should skip this: teams that need a hard SLA, production writes without idempotency, or PII/regulated text. The harness will not make a shared free tier production-safe.

If you want to avoid merging a route on quota vibes, point the harness at MonkeyCode's free endpoint first and keep the five-number report next to the PR.

Put a Free Model Route Behind an Error Budget, Not a Retry Storm

kongkong — Sat, 15 Aug 2026 20:09:36 +0000

429 is a signal, not a bug. A route that responds with 429 Too Many Requests is already telling the client to back off. A wrapper that immediately retries treats that signal as noise and spends remaining capacity faster. With the current wave of agent-driven workflows, one user action can fan out into many model calls, so a free model route is usually the first layer to saturate.

I used MonkeyCode's free model access and free server option as a disposable integration seam for this test. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I do not need a specific model name or quota to make the point: the problem shows up at any HTTP route that enforces rate limits.

The failure mode

Most integration wrappers start with a simple loop: call the endpoint, and if the status is 429 or 5xx, sleep and retry. That loop has three failure modes.

Signal	What it means	Naive reaction	Breaker reaction
`429`	Quota or rate limit reached	Retry immediately, spend more quota	Open until `Retry-After` elapses
`5xx`	Provider-side overload	Retry and add more load	Count against an error budget
Timeout	Unknown state, maybe slow downstream	Retry the same slow call	Record as unknown failure, open at threshold

The worst case is a shared free route with many callers. A single process that retries on 429 can double or triple the original request volume before a human looks at the dashboard. If the caller is an agent loop, the same pattern can repeat for each tool invocation.

A small error-budget breaker

The breaker below reacts to three inputs: status code, Retry-After, and failure rate over a rolling window. It opens when either a 429 includes Retry-After, or the window contains enough samples and the failure rate crosses the configured limit. A half-open state sends one probe to test recovery.

from enum import Enum
import time

class State(Enum):
    CLOSED = 'closed'
    OPEN = 'open'
    HALF_OPEN = 'half_open'

class ErrorBudgetBreaker:
    def __init__(self, window_seconds=60.0, failure_rate_limit=0.20,
                 cooldown_seconds=30.0, min_samples=10):
        self.window_seconds = window_seconds
        self.failure_rate_limit = failure_rate_limit
        self.cooldown_seconds = cooldown_seconds
        self.min_samples = min_samples
        self.state = State.CLOSED
        self.window_start = time.monotonic()
        self.total = 0
        self.failures = 0
        self.opened_at = 0.0

    def _is_failure(self, status):
        return status is None or status == 429 or status >= 500

    def allow(self):
        if self.state == State.OPEN:
            if time.monotonic() - self.opened_at >= self.cooldown_seconds:
                self.state = State.HALF_OPEN
                return True
            return False
        return True

    def record(self, status, retry_after=None):
        now = time.monotonic()
        if now - self.window_start >= self.window_seconds:
            self.window_start = now
            self.total = 0
            self.failures = 0

        self.total += 1
        failed = self._is_failure(status)
        if failed:
            self.failures += 1

        if status == 429 and retry_after:
            self.state = State.OPEN
            self.opened_at = now - self.cooldown_seconds + retry_after
            return self.state

        if self.state == State.HALF_OPEN:
            if failed:
                self.state = State.OPEN
                self.opened_at = now
            else:
                self.state = State.CLOSED
                self.failures = 0
            return self.state

        if self.total >= self.min_samples and self.failures / self.total >= self.failure_rate_limit:
            self.state = State.OPEN
            self.opened_at = now

        return self.state

A production wrapper calls allow() before sending a request and record() after receiving a response.

def parse_retry_after(value):
    try:
        return float(value)
    except (TypeError, ValueError):
        return None

def call_with_budget(client, url, payload, breaker, timeout=10.0):
    if not breaker.allow():
        return {'error': 'circuit_open', 'state': breaker.state.value}

    try:
        resp = client.post(url, json=payload, timeout=timeout)
    except Exception as exc:
        breaker.record(None)
        raise exc

    breaker.record(resp.status_code, parse_retry_after(resp.headers.get('Retry-After')))

    if resp.status_code == 429 or resp.status_code >= 500:
        return {'error': 'provider_unavailable', 'status': resp.status_code, 'state': breaker.state.value}

    resp.raise_for_status()
    return resp.json()

Deterministic test plan

Rather than trusting a synthetic benchmark, I run the breaker against a deterministic sequence of statuses and check the state transitions. The table below uses the same code and parameters.

Input sequence	min_samples	failure_rate_limit	cooldown	Expected states
`[200, 200, 429]`	3	0.20	30s	closed, closed, open
`[200, 200, 200]`	3	0.20	30s	closed, closed, closed
`[429 with Retry-After: 2s, 200]`	5	0.20	30s	open, skipped until header elapsed, then half-open
`[500, 500, 500]`	3	0.20	30s	closed, closed, open

This is the minimum signal check I would want before wiring a new model route into a larger service. It verifies that the breaker sees rate limits differently from transient 5xx errors and that it does not stay open forever.

Where to place it

The breaker works best as a route-level seam shared by all callers to the same provider endpoint. If multiple workers use separate in-memory breakers, a noisy worker can still produce a retry storm while a quiet worker is open. In a single-process service, an in-memory instance is enough. In a multi-instance deployment, move the state to a small store or use coordinated half-open probes.

I keep the breaker outside the model call itself. The model code remains a simple HTTP call; the breaker decides whether that call is allowed to happen. That separation means the same wrapper can protect a free model route, a free server slot, or any OpenAI-compatible endpoint without changing the rest of the request path.

What this does and does not fix

An error-budget breaker prevents retry storms and makes rate-limit pressure visible. It does not change the provider limit, repair a misconfigured key, or stop an over-committed quota from being exhausted by other callers. It also does not replace logs, budget ledgers, or golden-set evaluation; those are separate seams for separate failure modes.

A 429 can arrive before the window has enough samples. The code opens immediately only when a Retry-After header is present. If the provider sends 429 without Retry-After, the call only contributes to the rolling failure rate. For routes that omit that header, I usually add a static cooldown or count a bare 429 as an immediate open signal; the decision depends on the provider contract, not on the wrapper.

This approach is not useful for one-off scripts with a single request, offline batch jobs where retry delay is cheap, or callers that cannot tolerate a half-open probe. It is most useful when the route is shared, the call volume is high, and the cost of a failed call is larger than the cost of waiting.

Run this harness against your own rate limit and print the state transitions before promoting a free route into production. The cheapest way to learn a provider's failure contract is with a breaker in front of it, not after a retry storm has already consumed the quota.

Meter a Free Model Route as a Read-Only Canary Before It Writes State

kongkong — Sat, 15 Aug 2026 07:01:01 +0000

The current AI feature cycle has moved from "which model answers best" to "which tool is allowed to change state." Recent agent-tooling discussions keep landing on the same failure boundary: write access. A model can produce a bad summary in a sandbox and you lose nothing. The same model updating a ticket status, deleting a row, or publishing a reply creates cleanup work that can dwarf the token cost.

Start with budget arithmetic because it changes the canary design. Suppose a grounded RAG call uses 2,400 prompt tokens and 700 completion tokens. That is 3,100 tokens per call, so a 30M-token allowance is about 9,700 calls. If an agent path retries or re-plans an average of five calls per task, the same budget covers fewer than 2,000 tasks. A free server slot is not the scarce resource in that scenario; token burn and side effects are.

For the sandbox below I used MonkeyCode's free model access and free server option, which the operator describes as including a 30M-token allowance and a free server slot. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I treat those two availability claims as operator-supplied rather than a permanent quota or hardware guarantee.

Decide what is read-only before the model runs

The first gate is not a prompt. It is an HTTP method allowlist. Most writes arrive as POST, PATCH, PUT, or DELETE. A read-only canary should reject every one of those verbs in the default mode, while still allowing model-backed GET routes such as summarization, classification, or entity extraction. That keeps the model useful without allowing it to change application state.

Route intent	Default canary mode	What belongs in the ledger
Summarize a support thread	`GET`, read only	route, prompt tokens, completion tokens, timestamp
Classify or extract fields	`GET`, read only	route, token split, accuracy sample ID if available
Draft a reply but do not send	`GET`, read only	route, token split, draft length
Update a CRM status	blocked unless `ALLOW_WRITE=1`	separate write event with actor and review status
Delete or archive	never enabled on a free canary	destructive guard, not token-only

The table forces the decision to be explicit. If a route is not safe to call with GET, it should not run in the default canary deployment. If it is safe, it still needs to record token usage so the free allowance does not disappear silently.

The artifact: a read-only guard plus a token ledger

The reproduction is small: a FastAPI app, a SQLite ledger, and a guard that disallows write methods unless an environment variable is set.

import os
import sqlite3
import time
from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse
import httpx

app = FastAPI()
DB = os.getenv("LEDGER_DB", "canary.db")
ALLOW_WRITE = os.getenv("ALLOW_WRITE") == "1"


def init_db():
    with sqlite3.connect(DB) as conn:
        conn.execute(
            """
            CREATE TABLE IF NOT EXISTS token_events (
                id INTEGER PRIMARY KEY,
                route TEXT NOT NULL,
                prompt_tokens INTEGER NOT NULL,
                completion_tokens INTEGER NOT NULL,
                created_at INTEGER NOT NULL
            )
            """
        )


@app.middleware("http")
async def read_only_guard(request: Request, call_next):
    if not ALLOW_WRITE and request.method not in ("GET", "HEAD", "OPTIONS"):
        return JSONResponse(
            {"error": "write_path_blocked", "mode": "read_only_canary"},
            status_code=403,
        )
    return await call_next(request)


async def call_model(route: str, messages: list[dict]) -> dict:
    base_url = os.getenv("MODEL_BASE_URL")
    if not base_url:
        raise RuntimeError("MODEL_BASE_URL is not set")

    resp = httpx.post(
        f"{base_url}/chat/completions",
        json={"messages": messages, "temperature": 0},
        timeout=30,
    )
    resp.raise_for_status()
    data = resp.json()

    usage = data.get("usage") or {}
    prompt_tokens = int(usage.get("prompt_tokens") or 0)
    completion_tokens = int(usage.get("completion_tokens") or 0)

    with sqlite3.connect(DB) as conn:
        conn.execute(
            "INSERT INTO token_events(route, prompt_tokens, completion_tokens, created_at) "
            "VALUES (?, ?, ?, ?)",
            (route, prompt_tokens, completion_tokens, int(time.time())),
        )

    return data


@app.get("/read/ticket-summary/{ticket_id}")
async def ticket_summary(ticket_id: str):
    response = await call_model(
        "summary",
        [
            {"role": "system", "content": "Summarize the ticket without taking an action."},
            {"role": "user", "content": f"Read ticket {ticket_id} and summarize only."},
        ],
    )
    return {"ticket_id": ticket_id, "summary": response["choices"][0]["message"]["content"]}

The example assumes the provider endpoint accepts an OpenAI-style messages body and returns a usage object. If your provider uses a different HTTP shape, replace call_model without changing the ledger. The ledger is the part that matters.

After a few calls, inspect where the allowance is going:

SELECT route,
       COUNT(*) AS calls,
       SUM(prompt_tokens) AS prompt_tokens,
       SUM(completion_tokens) AS completion_tokens,
       SUM(prompt_tokens + completion_tokens) AS total_tokens
FROM token_events
GROUP BY route
ORDER BY total_tokens DESC;

For a budget check, subtract the recorded total from the operator-supplied allowance rather than hardcoding the number as a permanent guarantee.

def remaining_budget(budget=30_000_000) -> int:
    with sqlite3.connect(DB) as conn:
        row = conn.execute(
            "SELECT COALESCE(SUM(prompt_tokens + completion_tokens), 0) FROM token_events"
        ).fetchone()
    return budget - row[0]

Test before you grant write access

A useful canary run should prove three things before ALLOW_WRITE=1 is ever set.

Default mode rejects writes. POST, PATCH, PUT, and DELETE should return 403 with write_path_blocked. If they do not, the method guard is not the first gate.
The ledger matches provider-reported usage. Run the same read-only request three times and compare prompt_tokens and completion_tokens. Wide variance means the route is nondeterministic, the provider is retrying silently, or the prompt is being expanded in ways you did not expect.
The remaining budget decreases by the expected amount. Check remaining_budget before and after a known request. The change should equal the reported usage minus any provider rounding.

The write path should be a separate deployment, not a flag flip in the canary environment. Keep the canary at ALLOW_WRITE=0, then create a second release with explicit write authorization and a different database or tenant.

Limits and who should not use this

The read-only guard only checks HTTP verbs. It cannot catch a GET route that has a hidden side effect in application code, so keep read-only routes pure. The ledger also depends on provider-reported token counts; a local tokenizer may disagree, and that difference is part of the evaluation.

This is also not a security boundary against prompt injection. A model that can read private data and return it to an aggressive prompt is still a data-exfiltration risk, even if the route cannot write. Treat the canary as a release gate, not as an access-control substitute.

Skip this approach if you already run an LLM gateway with budget alerts and audit logs, or if every write is already human-reviewed before execution. If I were building a new evaluation sandbox, however, I would use a disposable free server slot for exactly this read-only stage before asking for write permission.

Turn a Free Token Quota Into a Rejection Rate Before You Merge the Route

kongkong — Sat, 15 Aug 2026 00:31:00 +0000

30 million free tokens sounds like capacity. The number that actually changes your architecture is how many of those tokens you can pull per minute before the endpoint starts returning 429s, resetting connections, or silently truncating a completion. This article walks through a token-aware load test that turns a free-tier quota into two engineering numbers: rejections per second and p95 latency at a fixed token rate.

MonkeyCode is an open-source project that, according to operator-supplied materials, currently offers a free model endpoint and a free server option with a reported 30 million-token allocation. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I did not verify the current quota on the project page, so treat the 30 million figure as a claim to test, not a fact to import into your capacity plan. The method below works for any hosted model endpoint.

Measure rejection, not tokens

A token quota is a budget, not a throughput guarantee. A 30M-token allocation tells you how much you can spend, but it says nothing about how fast you can spend it. Three separate limits determine whether a free route can serve real traffic:

Signal	What it measures	Why it matters
Token quota	Total tokens allowed over a period	Predicts monthly cost, not latency
Request rate limit	Requests per minute or second	Determines concurrency ceiling
Token accounting	How prompt and completion tokens are counted	Changes cost per request and overage risk
Failure mode	429 vs timeout vs dropped connection	Decides retry and fallback behavior

A useful load test targets the last two rows. You do not need to consume the entire 30M allocation to learn what happens at the boundary; you need to hold a fixed token rate and observe when the endpoint starts rejecting work.

A token-aware load generator

The following Python harness uses httpx and a configurable token estimate. Replace the endpoint, key, and model string with the values from the current project page. Run it for five to fifteen minutes so you can see steady-state behavior rather than a cold-start spike.

import asyncio
import os
import time
import httpx

ENDPOINT = os.getenv("MODEL_ENDPOINT", "https://your-endpoint.example/v1/chat/completions")
API_KEY = os.getenv("MODEL_API_KEY", "")
MODEL = os.getenv("MODEL_NAME", "replace-me")
CONCURRENCY = 8
TARGET_TOKENS_PER_REQUEST = 240

async def one_request(client, sem):
    async with sem:
        payload = {
            "model": MODEL,
            "messages": [
                {"role": "user", "content": "x" * 400}
            ],
            "max_tokens": 80,
        }
        started = time.monotonic()
        try:
            resp = await client.post(
                ENDPOINT,
                headers={"Authorization": f"Bearer {API_KEY}"},
                json=payload,
                timeout=30.0,
            )
            latency = time.monotonic() - started
            body = resp.text
            status = resp.status_code
            prompt_tokens = len(payload["messages"][0]["content"]) // 4
            completion_tokens = min(len(body) // 4, 80)
            total_tokens = prompt_tokens + completion_tokens
            print(f"{status} latency={latency:.2f}s tokens={total_tokens}")
            return status, latency, total_tokens
        except Exception as exc:
            latency = time.monotonic() - started
            print(f"ERR {type(exc).__name__} latency={latency:.2f}s")
            return "ERR", latency, 0

async def main():
    sem = asyncio.Semaphore(CONCURRENCY)
    async with httpx.AsyncClient() as client:
        tasks = [one_request(client, sem) for _ in range(200)]
        results = await asyncio.gather(*tasks)
    statuses = [r[0] for r in results]
    latencies = [r[1] for r in results]
    total_tokens = sum(r[2] for r in results)
    print(f"statuses={ {s: statuses.count(s) for s in set(statuses)} }")
    print(f"total_tokens={total_tokens} mean_latency={sum(latencies)/len(latencies):.2f}s")
    print(f"p95_latency={sorted(latencies)[int(0.95*len(latencies))]:.2f}s")

if __name__ == "__main__":
    asyncio.run(main())

This is not a benchmark of MonkeyCode; it is a probe you can run against the endpoint you intend to use. The token estimate is deliberately coarse because provider tokenizers differ. For a stricter test, replace the // 4 heuristic with the provider's tokenizer so your rejection count aligns with billable usage.

Decision table and limits

Use the measured rejection rate to decide whether the free server belongs in development, staging, or nowhere near production.

Observed behavior	Safe use	Unsafe use
Rejection rate < 1% at target token rate	Dev/staging shadow route, internal demos	User-facing traffic without a paid fallback
Rejection rate 1-10%	Background batch jobs with retries	Synchronous request path
Intermittent timeouts or dropped connections	Canary experiments with strict deadlines	Workloads with strict SLOs
Token accounting is opaque or changes daily	Cost estimation only	Budget forecasting

The free server option becomes most useful as a shadow tier: send a copy of non-critical traffic, compare outputs and failure modes against your primary route, and keep the paid route as the fallback. That mirrors the shadow-contract pattern, but the missing piece is usually rate-limit measurement, which is what this harness adds.

Do not use the approach if you are handling regulated data, need guaranteed tail latency, or cannot tolerate a promotional quota changing before you finish your test. A free allocation is a starting condition, not a service-level agreement. Check the project page for the current token amount and server terms before relying on either. If you run the harness, share the failure state you hit first: 429, timeout, or token accounting that did not match the request you sent.