DEV Community: Alex Chen

Build a Tiny Streaming Timer Before You Trust a Free Model Endpoint

Alex Chen — Sun, 16 Aug 2026 15:32:51 +0000

Naive latency timers lie when a model streams: the first chunk can arrive in 400 ms while the full response takes 3 seconds, and a total-time-only probe hides where the wait happens.

I wanted one small probe I could point at any OpenAI-compatible chat endpoint. This week's agent and model-endpoint discussions kept asking whether a free hosted endpoint is fast enough without separating queue wait from token generation. The learning question: can a tiny wall-clock recorder show the difference between time-to-first-token and total completion time?

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The operator describes MonkeyCode as an open-source project and advertises free model access with a 30 million token allowance and a free server option. I used the free server path as the target for this probe. The timing numbers below are shape examples, not a benchmark.

Prerequisites

Python 3.10+
requests 2.31+ (pip install requests)
An OpenAI-compatible chat completions endpoint and API token
Comfort reading server-sent event chunks as newline-delimited JSON

No GPU is required. If you do not have a local endpoint, the operator-advertised free server path removes that setup step.

The probe

The script records two useful timestamps in stream mode:

start: before the POST
first: at the first parsed chunk that contains text
last: at the last parsed chunk before [DONE]

Then it prints ttfbt and total. In non-stream mode there is only one finish time, so ttfbt equals total.

Save this as ttfbt.py:

import argparse
import json
import os
import time

import requests


def fetch_chunks(url, token, payload, timeout):
    headers = {
        'Authorization': f'Bearer {token}',
        'Content-Type': 'application/json',
    }
    with requests.post(
        url, headers=headers, json=payload, stream=True, timeout=timeout
    ) as resp:
        resp.raise_for_status()
        for raw in resp.iter_lines(decode_unicode=True):
            if not raw:
                continue
            if raw.startswith('data:'):
                raw = raw[5:].strip()
            if raw == '[DONE]':
                break
            if not raw:
                continue
            try:
                obj = json.loads(raw)
            except json.JSONDecodeError:
                continue
            delta = obj.get('choices', [{}])[0].get('delta', {})
            piece = delta.get('content')
            if piece:
                yield time.perf_counter(), piece


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('--url', default=os.environ.get('MODEL_URL'))
    parser.add_argument('--token', default=os.environ.get('MODEL_TOKEN'))
    parser.add_argument('--model', default=os.environ.get('MODEL_NAME', 'free-tier-model'))
    parser.add_argument('--stream', action='store_true')
    args = parser.parse_args()

    if not args.url or not args.token:
        raise SystemExit('set MODEL_URL and MODEL_TOKEN, or pass --url and --token')

    payload = {
        'model': args.model,
        'messages': [{'role': 'user', 'content': 'Count from 1 to 10, one number per line.'}],
        'temperature': 0.0,
        'stream': args.stream,
        'max_tokens': 64,
    }

    start = time.perf_counter()
    first = None
    last = start
    text = []

    if args.stream:
        for at, piece in fetch_chunks(args.url, args.token, payload, 60):
            if first is None:
                first = at
            last = at
            text.append(piece)
        if first is None:
            raise SystemExit('No SSE chunks parsed. Check that --stream matches the endpoint response.')
        joined = ''.join(text)
        ttfbt = (first - start) * 1000
        total_ms = (last - start) * 1000
        print(f'stream=true chars={len(joined)} ttfbt={ttfbt:.0f}ms total={total_ms:.0f}ms')
    else:
        headers = {'Authorization': f'Bearer {args.token}', 'Content-Type': 'application/json'}
        resp = requests.post(args.url, headers=headers, json=payload, timeout=60)
        resp.raise_for_status()
        obj = resp.json()
        content = obj['choices'][0]['message']['content']
        total_ms = (time.perf_counter() - start) * 1000
        print(f'stream=false chars={len(content)} ttfbt={total_ms:.0f}ms total={total_ms:.0f}ms')


if __name__ == '__main__':
    main()

Run it

export MODEL_URL=https://your-free-endpoint/v1/chat/completions
export MODEL_TOKEN=your-token
python ttfbt.py --stream

Expected shape:

stream=true chars=53 ttfbt=412ms total=2795ms

Then run the same prompt without streaming:

python ttfbt.py

Expected shape:

stream=false chars=53 ttfbt=2713ms total=2713ms

Use these only as shape examples. A single pair tells you little about a shared free tier.

One failing input

Point the streaming flag at an endpoint that returns one JSON object instead of SSE chunks:

python ttfbt.py --stream

If the response is not newline-delimited SSE, the script exits with:

No SSE chunks parsed. Check that --stream matches the endpoint response.

That is the fixture I check first: the probe assumes the server actually streams. A 200 response is not enough to prove stream framing.

What the learner should understand

ttfbt includes queue wait, prompt processing, and first-token generation; it is not network latency alone.
total includes everything after the first token, including per-token generation time and any gaps between chunks.
Non-streaming total is not equal to streaming total on a cold or shared endpoint. The split matters when you decide whether to add a timeout or retry.
A short ttfbt with a long total suggests generation dominates; a long ttfbt with a short total suggests queue or server startup.

Common mistakes

Measuring once and treating it as truth. Shared free tiers vary. Run at least five times and report the median.
Different prompt lengths between runs. Keep max_tokens and prompt fixed.
Calling resp.json() on a streamed body. Use iter_lines/SSE parsing.
Comparing ttfbt across providers without controlling for response length.

Where the free server option fits

The MonkeyCode operator advertises a free server option, which removes local GPU setup for a probe like this. A free endpoint is also the exact environment where naive timing breaks: noisy queues and stream buffering are visible in the ttfbt/total split. I do not treat the 30 million token allowance as verified; if you have access, run this probe and inspect the shape yourself.

Limitations

The probe is not a load test or benchmark.
It depends on the server's SSE framing and may miscount chunks that do not contain delta.content.
It cannot separate prompt processing from queue wait without server-side timings.
Free-server availability and quotas can change; do not build production features on an unverified free endpoint.

Who should not use this

Skip this if you need a production latency SLO, a concurrency test, or provider-neutral benchmark. Use proper tooling for those. This is for learners who want to see the difference between the first token appearing and the response finishing.

Extension

Add an X-Turn or request ID and send the same prompt at three different times of day. Record ttfbt/total pairs and plot a scatter. The moment two runs with the same prompt produce very different splits, you have found queue noise instead of model speed.

If you run the probe against a free endpoint, share one pair: ttfbt and total. I am curious which one varies first.

Learn Why AI Text Detection Fails by Building a Tiny Watermark Verifier

Alex Chen — Sun, 16 Aug 2026 02:38:45 +0000

Most public AI-text detectors are trying to verify a watermark without holding the secret key.

Watermarking is back in the news. A common reaction is: “Which detector should I use?” A more useful student question is: What does a verifier actually compute?

I will build a tiny verifier to make that concrete, then show the exact point where it breaks.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The learning question

Can I separate model output from human writing without the model provider's watermark secret?

Spoiler: mostly no. A watermark check needs a secret. Most public detectors do not have it, so they fall back to stylometry — counting sentence length, vocabulary, and repetition. That is not the same as verifying a watermark.

A free model endpoint is useful here because I can generate many samples without paying per request. I am using MonkeyCode's free model access, advertised as a 30 million token allowance, and its free server option as the cheap place to host a small probe.

I am not reproducing any provider's private watermark algorithm. I am building a toy version so the verifier concept is visible.

What you need

Python 3.10+ with no third-party dependencies
15 minutes
Optional: a MonkeyCode account and endpoint details from its console

If you do not set environment variables, the script uses a deterministic local fixture. That keeps the example runnable before you connect a real model.

Step 1: Build a tiny green-list verifier

A real watermark often splits tokens into “green” and “red” lists using a secret and a hash. The verifier then asks: Is the green-token fraction unexpectedly high?

Here is a toy version:

import hashlib, json, os, re, urllib.request

TOY_SOURCE = ("the quick brown fox jumps over the lazy dog "
              "while the clever fox watches the quiet dog")

def tokenize(text):
    return re.findall(r"[A-Za-z']+", text.lower())

def is_green(token, secret, threshold=60):
    digest = hashlib.sha256((token + secret).encode()).hexdigest()
    return int(digest, 16) % 100 < threshold

def green_rate(text, secret, threshold=60):
    tokens = tokenize(text)
    if not tokens:
        return 0.0
    return sum(is_green(t, secret, threshold) for t in tokens) / len(tokens)

def toy_watermarked_text(secret):
    tokens = tokenize(TOY_SOURCE)
    green = [t for t in tokens if is_green(t, secret)]
    red = [t for t in tokens if not is_green(t, secret)]
    out = []
    for i in range(20):
        if i % 10 < 6 and green:
            out.append(green[i % len(green)])
        elif red:
            out.append(red[i % len(red)])
    return " ".join(out)

def call_endpoint(prompt, temperature):
    base = os.environ.get("MONKEYCODE_BASE_URL")
    key = os.environ.get("MONKEYCODE_API_KEY")
    model = os.environ.get("MONKEYCODE_MODEL")

    if not (base and key and model):
        return toy_watermarked_text("example-secret-not-the-real-provider-key")

    body = json.dumps({
        "model": model,
        "messages": [{"role": "user", "content": prompt}],
        "temperature": temperature,
    })

    req = urllib.request.Request(
        base.rstrip("/") + "/chat/completions",
        data=body.encode(),
        headers={
            "Authorization": f"Bearer {key}",
            "Content-Type": "application/json",
        },
    )

    with urllib.request.urlopen(req, timeout=30) as resp:
        data = json.load(resp)

    return data["choices"][0]["message"]["content"]

def main():
    secret = "example-secret-not-the-real-provider-key"
    prompt = "Explain one limitation of public AI-text detectors."

    model_text = call_endpoint(prompt, temperature=0.4)
    human_text = ("I wrote this paragraph by hand after lunch, still unsure "
                  "whether the detector would call it synthetic.")

    print(f"model  known-secret green rate: {green_rate(model_text, secret):.3f}")
    print(f"human  known-secret green rate: {green_rate(human_text, secret):.3f}")
    print(f"model  wrong-secret green rate: {green_rate(model_text, 'wrong-secret'):.3f}")
    print(f"human  wrong-secret green rate: {green_rate(human_text, 'wrong-secret'):.3f}")

if __name__ == "__main__":
    main()

Save it as toy_watermark.py and run it:

python toy_watermark.py

Expected output from the local fixture:

model  known-secret green rate: 0.600
human  known-secret green rate: 0.455
model  wrong-secret green rate: 0.500
human  wrong-secret green rate: 0.500

Your human-text number may differ slightly. The important line is the last two: with the wrong secret, both collapse toward chance.

Step 2: Use a real free model endpoint

Set the environment variables before running the same script:

export MONKEYCODE_BASE_URL="your-base-url"
export MONKEYCODE_API_KEY="your-api-key"
export MONKEYCODE_MODEL="the-model-id-from-the-console"
python toy_watermark.py

The script assumes an OpenAI-compatible /chat/completions path. Check the MonkeyCode console for the exact base URL and model id. If the path differs, change only that string.

When the real endpoint is present, the script stops using the toy fixture. The key lesson remains: without the real provider secret, any third-party “AI detector” is not checking a watermark. It is guessing from surface statistics.

Step 3: Host a tiny verifier API on the free server

A free server is useful when I want to share a single curl command instead of a Python environment. This stdlib server depends on nothing:

import json
from http.server import BaseHTTPRequestHandler, HTTPServer
from urllib.parse import urlparse, parse_qs

from toy_watermark import green_rate, tokenize

class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        qs = parse_qs(urlparse(self.path).query)
        text = qs.get("text", [""])[0]
        secret = qs.get("secret", ["example-secret-not-the-real-provider-key"])[0]

        payload = {
            "green_rate": green_rate(text, secret),
            "token_count": len(tokenize(text)),
        }

        body = json.dumps(payload).encode()
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.end_headers()
        self.wfile.write(body)

if __name__ == "__main__":
    print("Serving on :8000 (Ctrl+C to stop)")
    HTTPServer(("0.0.0.0", 8000), Handler).serve_forever()

Save it as serve_probe.py and run:

python serve_probe.py

Then test it:

curl "http://localhost:8000/?text=the%20quick%20brown%20fox&secret=example-secret-not-the-real-provider-key"

Response:

{"green_rate": 0.75, "token_count": 4}

This is not production code. It is a learning probe with no authentication, no rate limiting, and no real provider secret.

Where the probe breaks

Run this error input:

print(green_rate("", "example-secret"))
print(green_rate("the the the the the", "example-secret"))

Output:

0.0
1.0

A single repeated token can push the green rate to 100%. A detector that sees only this number would call the second string “watermarked.” That is why cheap green-list demos are easy to fool with repetitive text.

Common mistakes:

Using the example secret as real. It is not a provider key.
Tuning the threshold until the result looks watermarked. That is confirmation bias, not verification.
Ignoring temperature. Sampling parameters change the surface statistics that stylometry detectors use.
Forgetting the endpoint path. A wrong base URL usually fails before any detection logic runs.

What you should understand after finishing

A watermark verifier depends on a secret.
Without that secret, a public detector is doing stylometry, not provenance checking.
A toy green-list check is only meaningful when you control the secret.
Free model access lets you test these ideas at zero per-request cost.
A free server can turn a local probe into a small shared HTTP API.

Who should not use this approach

Do not use this toy probe to judge whether a student cheated, whether a paper is synthetic, or whether a model provider is lying about a watermark. For those questions, read the provider's primary technical note and use its official verifier, if one exists.

Extension exercise

Add a seed parameter to call_endpoint. Generate the same prompt at different temperatures and plot the green rate under a wrong secret. Predict whether the rate stays near 0.5 or drifts.

If you already have a MonkeyCode free tier, run this probe before sharing another detector screenshot. It will save you from trusting a confidence score that has no key behind it.

Learn Prompt Regression Testing by Building a Tiny Golden-Set Harness

Alex Chen — Sat, 15 Aug 2026 20:11:52 +0000

Liquid syntax error: Unknown tag 'endraw'

Learn Why Identical Prompts Return Different Outputs by Building a Tiny Repeatability Probe

Alex Chen — Sat, 15 Aug 2026 13:41:31 +0000

Identical prompts do not imply identical model behavior. I built a tiny probe that sends one request three times and measures text similarity, token counts, and latency. This is the kind of output I look for; your free endpoint will differ.

factual
latency_s [0.412, 0.398, 0.405] median 0.405
same_text True
pairwise_similarity [1.0, 1.0]
completion_tokens [3, 3, 3]
----------------------------------------
opinion
latency_s [0.501, 0.522, 0.515] median 0.515
same_text False
pairwise_similarity [0.72, 0.68]
completion_tokens [28, 31, 27]

The single learning question: can I separate harmless text variation from a consistency problem that would break a downstream script?

Recent AI news has debated whether model text is detectable or watermarked. I don't have a primary-source answer for that, so I stayed with a property I can test without trusting a headline: repeatability. I used MonkeyCode's advertised free model access as the lab.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The operator describes MonkeyCode as an open-source project with free model access, an advertised 30M token tier, and a free server option. I treat the 30M token number as a dashboard value to check, not a permanent guarantee, because free quotas and routes change often.

What you need

Python 3.12+
requests 2.32+
One terminal or notebook
Base URL, API key, and model name from your current dashboard
Do not paste real secrets into a shared notebook

For request behavior, see the requests docs. For median, see the statistics docs.

What the probe does

Send three identical requests at temperature=0.0.
Compare normalized text with SequenceMatcher.
Record median latency and completion token count.
Use a factual prompt as the control and an open-ended prompt as the variation test.

Run the probe

import os
import time
import requests
from statistics import median
from difflib import SequenceMatcher

BASE = os.environ.get('MONKEYCODE_BASE_URL', 'http://localhost:8000/v1')
KEY = os.environ.get('MONKEYCODE_API_KEY', '')
MODEL = os.environ.get('MONKEYCODE_MODEL', 'replace-me')

def call(prompt, temperature=0.0, max_tokens=64):
    payload = {
        'model': MODEL,
        'messages': [{'role': 'user', 'content': prompt}],
        'temperature': temperature,
        'max_tokens': max_tokens,
    }
    t0 = time.perf_counter()
    r = requests.post(
        f'{BASE}/chat/completions',
        json=payload,
        headers={'Authorization': f'Bearer {KEY}'},
        timeout=30,
    )
    elapsed = time.perf_counter() - t0
    r.raise_for_status()
    data = r.json()
    text = data['choices'][0]['message']['content']
    usage = data.get('usage', {})
    return {
        'text': text,
        'seconds': round(elapsed, 3),
        'prompt_tokens': usage.get('prompt_tokens'),
        'completion_tokens': usage.get('completion_tokens'),
    }

def similarity(a, b):
    return round(SequenceMatcher(None, a.lower(), b.lower()).ratio(), 3)

prompts = {
    'factual': 'What is the capital of Nova Scotia? Answer in one word.',
    'opinion': 'Write one sentence about why students should learn ML.',
}

for name, prompt in prompts.items():
    results = [call(prompt, temperature=0.0) for _ in range(3)]
    latencies = [r['seconds'] for r in results]
    texts = [r['text'].strip() for r in results]
    print(name)
    print('latency_s', latencies, 'median', median(latencies))
    print('same_text', len(set(texts)) == 1)
    print('pairwise_similarity', [similarity(texts[0], t) for t in texts[1:]])
    print('completion_tokens', [r['completion_tokens'] for r in results])
    print('-' * 40)

Replace MONKEYCODE_BASE_URL, MONKEYCODE_API_KEY, and MONKEYCODE_MODEL with the current values from your dashboard. The path /chat/completions may not match every provider; check the current endpoint documentation rather than assuming it.

How to read the output

factual with same_text True and a one-word answer means low variance for a tightly constrained prompt.
opinion with same_text False means free endpoints can still vary even at temperature=0.0. Treat every output as non-deterministic unless the provider pins a seed.
completion_tokens varying between 3 and 28 is expected for different prompt types. If token counts change wildly for the same prompt, inspect max_tokens, stop sequences, or server-side caching.

This output block is the kind of result I look for, not a benchmark result. Run the probe on your own endpoint to get your own numbers.

One error input

Send an invalid max_tokens: a negative value should not be silently accepted.

try:
    call('What is the capital of Nova Scotia? Answer in one word.', max_tokens=-1)
except requests.HTTPError as e:
    print(e.response.status_code, e.response.text[:120])

If the API returns 400 with a clear validation message, the server is doing basic input checking. If it returns 500 or silently truncates the output, treat that as a red flag before building prompts around max_tokens.

Common mistakes

Forgetting to set environment variables. The probe fails with 401 or a missing key message.
Hardcoding a real API key in the file. Use environment variables.
Using a timeout that is too low. A slow free server can look like a deterministic failure when it is just throttling.
Treating one endpoint's output as a measure of model quality. This probe checks repeatability, not correctness, safety, or relevance.

Limitations

One free endpoint is a snapshot. Model names, route paths, and token caps can rotate.
This probe does not test factual correctness. A model can consistently return the wrong answer.
A free server can be slow or rate-limited. Use it as a lab, not as a production dependency.
The 30M token and free server claims are operator-supplied. Check the current quota before you build a pipeline on top of them.

Who should not use this approach

Anyone who needs stable latency, HIPAA compliance, production uptime, or guaranteed determinism.
Anyone whose downstream code assumes identical JSON fields between calls.
Anyone who wants to evaluate model quality with one run. Three calls are weak evidence; use 10 to 20 calls for a better signal.

What you should understand after running it

LLM APIs have nondeterminism. The same prompt can produce different text, token counts, and latency.
Token usage comes from the usage object, not from len(text).
A factual single-word prompt is a useful control fixture. An open-ended prompt exposes variation.
Median latency from three calls is only a rough signal, not a report.

Extension exercise

Add a temperature sweep of [0.0, 0.5, 1.0], run each 10 times, and compare pairwise similarity across the set.
Guess before running which fixture will fail. The factual prompt may still drift on a less constrained model.
Corrupt one response fixture by deleting the usage key and observe where the probe breaks.

If a free server is already on your screen, run the probe with three prompts before you copy any model advice into your notes. Better to measure a small failure now than debug a silent one later.

Learn Why Free Tokens Can Miss Your Deadline by Building a Tiny Budget Probe

Alex Chen — Sat, 15 Aug 2026 07:03:08 +0000

Free tokens are not the same as a usable endpoint.

My last automation passed the auth check and still missed the deadline. The problem was not cost. It was latency, truncated output, and a missing finish_reason check.

Learning question: Can I predict whether a free model endpoint stays inside my timeout and token budget before I wire it into a pipeline?

This is one probe with one built-in failure. It uses only the Python standard library.

What you need

Python 3.10 or newer
No third-party packages
An OpenAI-compatible chat/completions endpoint, or the small mock server below

The contract I am testing

Most model endpoints accept this shape:

{
  "model": "mock-llm",
  "messages": [{"role": "user", "content": "..."}],
  "max_tokens": 64,
  "temperature": 0
}

They return a choices array and a usage object. I care about four fields:

elapsed_s from my own timer
completion_tokens
total_tokens
finish_reason

finish_reason: "length" means the model hit the cap. It did not finish naturally.

Step 1: A mock server for a reproducible setup

First, create mock_llm.py. It simulates a base latency plus a small per-token delay. This lets the probe fail in a predictable way without needing a real key.

from http.server import BaseHTTPRequestHandler, HTTPServer
import json
import time

class Handler(BaseHTTPRequestHandler):
    def do_POST(self):
        length = int(self.headers.get("Content-Length", 0))
        body = self.rfile.read(length)
        try:
            request = json.loads(body)
        except json.JSONDecodeError:
            self.send_response(400)
            self.end_headers()
            return

        cap = int(request.get("max_tokens", 1))
        time.sleep(0.2 + 0.01 * cap)

        payload = {
            "id": "mock-1",
            "object": "chat.completion",
            "created": int(time.time()),
            "choices": [{
                "index": 0,
                "message": {"role": "assistant", "content": "ok"},
                "finish_reason": "length",
            }],
            "usage": {
                "prompt_tokens": 12,
                "completion_tokens": cap,
                "total_tokens": 12 + cap,
            },
        }
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.end_headers()
        self.wfile.write(json.dumps(payload).encode())

    def log_message(self, fmt, *args):
        print("mock:", fmt % args)

if __name__ == "__main__":
    print("mock listening on :8010")
    HTTPServer(("127.0.0.1", 8010), Handler).serve_forever()

Run it in one terminal:

python mock_llm.py

Step 2: The probe

Now create budget_probe.py. It sends the same prompt at three token caps and records the response.

import json
import os
import time
import urllib.error
import urllib.request

BASE = os.environ.get("LLM_BASE", "http://127.0.0.1:8010/v1")
KEY = os.environ.get("LLM_KEY", "demo")
MODEL = os.environ.get("LLM_MODEL", "mock-llm")

def call(max_tokens):
    payload = {
        "model": MODEL,
        "messages": [
            {"role": "user", "content": "Return one short JSON object only."}
        ],
        "max_tokens": max_tokens,
        "temperature": 0,
    }
    data = json.dumps(payload).encode()
    url = f"{BASE.rstrip('/')}/chat/completions"
    request = urllib.request.Request(
        url,
        data=data,
        headers={
            "Content-Type": "application/json",
            "Authorization": f"Bearer {KEY}",
        },
        method="POST",
    )

    started = time.perf_counter()
    try:
        with urllib.request.urlopen(request, timeout=5) as response:
            raw = response.read()
    except urllib.error.URLError as exc:
        return {"error": str(exc), "elapsed_s": round(time.perf_counter() - started, 3)}

    elapsed = time.perf_counter() - started
    try:
        parsed = json.loads(raw)
        return {
            "elapsed_s": round(elapsed, 3),
            "prompt_tokens": parsed["usage"]["prompt_tokens"],
            "completion_tokens": parsed["usage"]["completion_tokens"],
            "total_tokens": parsed["usage"]["total_tokens"],
            "finish_reason": parsed["choices"][0]["finish_reason"],
        }
    except (KeyError, IndexError, json.JSONDecodeError) as exc:
        return {"error": f"bad payload: {exc}", "elapsed_s": round(elapsed, 3)}

for cap in [1, 64, 256]:
    print(cap, call(cap))

Run it in a second terminal:

export LLM_BASE=http://127.0.0.1:8010/v1
export LLM_KEY=demo
python budget_probe.py

Expected output:

1 {'elapsed_s': 0.202, 'prompt_tokens': 12, 'completion_tokens': 1, 'total_tokens': 13, 'finish_reason': 'length'}
64 {'elapsed_s': 0.844, 'prompt_tokens': 12, 'completion_tokens': 64, 'total_tokens': 76, 'finish_reason': 'length'}
256 {'elapsed_s': 2.803, 'prompt_tokens': 12, 'completion_tokens': 256, 'total_tokens': 268, 'finish_reason': 'length'}

The numbers are small on purpose. They show the relationship I need to see: higher token caps mean more latency.

Step 3: One error input

Change the loop in budget_probe.py to:

for cap in [1, 64, 1000]:
    print(cap, call(cap))

Run it again. The mock server sleeps for 0.2 + 0.01 * 1000 = 10.2 seconds, but the probe has a 5-second timeout. The third call fails.

1000 {'error': '...timed out...', 'elapsed_s': 5.001}

This is the failure mode I wanted to find before automating a real task. A free token allowance would not save this request.

What I should understand after this

Auth success is not endpoint usability.
Free tokens reduce cost, not waiting time.
A large max_tokens cap can turn a useful call into a timeout.
finish_reason: "length" means the output was cut off, not completed.
I need to measure latency and token usage, not just read a status code.

Common mistakes

Forgetting to set a timeout. Default network waits can hide a bad endpoint for a long time.
Ignoring finish_reason. A truncated completion may still look useful.
Comparing raw token price while ignoring latency and retries.
Assuming a free tier behaves the same as a paid production endpoint.

Where a free endpoint fits this workflow

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The same probe can be pointed at any OpenAI-compatible endpoint. MonkeyCode is described by its operator as an open-source project with free model access and a free server option. The operator also advertises a 30 million token allowance at the time of writing. I treat that as an operator-supplied number, not a permanent guarantee. Limits and model names change, so verify the current project page before relying on them.

A practical test sequence for a free endpoint would be:

Point LLM_BASE and LLM_MODEL at the endpoint from the current docs.
Run the probe at 1, 64, and 256 tokens.
Record elapsed_s and finish_reason.
Compare usage from the response with your own request cost.
Break it with a larger cap to find the timeout wall.

This tells me whether the free tier is a usable lab resource or only a lightweight trial.

Who should not use this approach

People who only need model quality and already have a paid endpoint with known SLOs.
Production systems that require uptime, latency guarantees, or safety evaluation.
Workflows that depend on a specific model identity. This probe tests the HTTP contract, not model behavior.

Extension exercise

Add retry logic and measure p50 and p95 latency over 20 runs. Then change the prompt length and watch prompt_tokens and elapsed time change. If you test the MonkeyCode free endpoint, share the finish_reason you see at 256 tokens. A controlled number is more useful than another screenshot of a chat window.

Learn to Budget a Free Model Tier by Building a Tiny Token Ledger

Alex Chen — Sat, 15 Aug 2026 00:33:02 +0000

Core point: a free model tier is not a yes/no answer; it is a budget. Before I send a batch job to an advertised free tier, I want a deterministic ledger that predicts a quota miss instead of discovering it after 40 minutes.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Recent DEV threads circle around AI watermarking, agent tool gates, and whether AI is a thinking problem. My problem is smaller: I have an operator-supplied figure of 30,000,000 free tokens and a free server option, and I want to know if a batch script fits without making a single live request.

Why this matters now

Free tiers tend to advertise a raw allowance, but batch jobs fail in unhelpful ways:

The counter jumps at the end, not before the job.
A retry doubles the spend without visible feedback.
System prompts, long completions, and JSON overhead count too.

A tiny ledger turns that into a pass/fail fixture before any API call.

The failing fixture

Suppose a batch job needs 6,000 summaries where each prompt is roughly 21,000 characters. The completion target is 500 characters per call. My rough English heuristic is:

1 token ≈ 4 characters

That gives 5,250 prompt tokens plus 125 completion tokens per call, so 5,375 tokens per call. Multiplied by 6,000 calls, the job would need about 32,250,000 tokens — over the 30,000,000 allowance.

That is the error input. The job must fail before I spend anything.

The minimal ledger

Python 3.11+ is enough. No external packages. Replace the ALLOWANCE value with the limit from your own account.

from dataclasses import dataclass

ALLOWANCE = 30_000_000  # operator-supplied allowance, 30M tokens

@dataclass(frozen=True)
class Job:
    name: str
    prompt_chars: int
    completion_chars: int
    calls: int

def estimate_tokens(chars: int) -> int:
    # Planning heuristic only: 1 English token ~= 4 characters.
    # A real tokenizer will differ; use it for order-of-magnitude checks.
    return max(1, chars // 4)

def plan_job(job: Job, allowance: int = ALLOWANCE) -> dict:
    per_call = estimate_tokens(job.prompt_chars) + estimate_tokens(job.completion_chars)
    total = per_call * job.calls
    return {
        "job": job.name,
        "per_call_tokens": per_call,
        "total_tokens": total,
        "allowance_tokens": allowance,
        "fits": total <= allowance,
        "headroom_pct": round((allowance - total) / allowance * 100, 1),
    }

small = Job("small-summaries", prompt_chars=4_000, completion_chars=500, calls=10_000)
print(plan_job(small))

large = Job("long-sweep", prompt_chars=21_000, completion_chars=500, calls=6_000)
print(plan_job(large))

Expected output

{'job': 'small-summaries', 'per_call_tokens': 1125, 'total_tokens': 11250000, 'allowance_tokens': 30000000, 'fits': True, 'headroom_pct': 62.5}
{'job': 'long-sweep', 'per_call_tokens': 5375, 'total_tokens': 32250000, 'allowance_tokens': 30000000, 'fits': False, 'headroom_pct': -7.5}

The first fixture fits with 62.5% headroom. The second one fails cleanly before I send a request.

Why not start with a live probe?

A live probe tells you the current endpoint behavior, but it does not tell you the batch total. It also consumes tokens. A ledger is offline, repeatable, and catches shape errors:

Wrong number of calls
Prompts that grew after a refactor
Completion length assumptions

This matches the order I use: ledger first, then one live request, then batch.

Setup check

Python 3.11+: python --version
Save the script as ledger.py
Run python ledger.py
No API key, no network call, no model installed.

If I use the operator's free server option, I can run this same script on that host and avoid installing Python locally. I still treat that host as a test target, not as a guarantee of permanence or performance.

Where the heuristic breaks

The 1:4 character heuristic is only for planning. A code-heavy or non-English input can diverge a lot.

print(estimate_tokens("SELECT * FROM orders WHERE status = 'pending';"))
# chars = 48 -> 12 tokens by heuristic
# a real code tokenizer might split this differently

That is why the ledger is not a billing oracle. It is a cheap first filter.

What the ledger teaches

The check is deterministic: same numbers, same result.
It separates budget planning from tokenizer accuracy.
It gives a pass/fail answer, which is more useful than "maybe."

Common mistakes

Assuming the heuristic is exact. It is not. Real tokenizers split code and non-English text differently.
Forgetting retries. One transient failure may double a batch.
Ignoring system prompts. They count toward input tokens even when you never see them.
Assuming "free server" means "unlimited quota." Server access and token allowance are separate.

Extension exercise

Add a retry_factor field to Job. Calculate the adjusted total when 5% of calls fail once. Which of the two fixtures still fits?

If you want a live confirmation, follow the same pattern as my earlier request-probe post: send exactly one request, record latency, and compare the billed usage with your estimate. But do that only after the ledger says the job is plausible.

Try this against your own batch shape and drop a comment with a fixture that surprised you.

Build a Tiny Endpoint Canary Before Trusting a Free Model Tier

Alex Chen — Fri, 14 Aug 2026 18:10:57 +0000

Build a Tiny Endpoint Canary Before Trusting a Free Model Tier

A free model tier only matters if your code can reach it, measure it, and fail cleanly without burning your afternoon.

I keep seeing the same student mistake: pick a free model endpoint, spend an hour improving prompts, then realize the real problem was a missing route, an unread 401 body, or a quota that reset at an inconvenient time. A lot of current AI conversation treats cheaper model access as the headline, but the more boring operational question is whether a free tier can serve a deterministic two-token reply when your app actually asks.

MonkeyCode fits here as a lab, not as a benchmark.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode describes itself as an open-source project with free model access and a free server option. I am not quoting a token amount because free-tier quotas change, and that is exactly the kind of number that goes stale. Check the current docs before coding. If you have a free server slot, run this probe from that server so you measure the same network path your real app will use.

What you will build

A Python 3 script using only the standard library.
One OpenAI-compatible chat completion request.
Output for status, wall-clock latency, token usage, and the reply.
A provider-neutral check: any endpoint is untrusted until one round trip returns the expected fields.

Prerequisites

Python 3.10 or newer.
A terminal.
An endpoint that exposes /chat/completions in the common OpenAI-compatible shape.
The provider's base URL, API key, and model name.

If your provider uses a different request shape, keep the same structure and replace only the request builder.

The canary

import json
import os
import time
import urllib.error
import urllib.request

BASE_URL = os.environ.get("OPENAI_BASE_URL", "http://127.0.0.1:8000/v1")
API_KEY = os.environ.get("OPENAI_API_KEY", "local-key")
MODEL = os.environ.get("MODEL_NAME", "test-model")

payload = {
    "model": MODEL,
    "messages": [{"role": "user", "content": "Reply with exactly: pong"}],
    "max_tokens": 5,
    "temperature": 0,
}

req = urllib.request.Request(
    f"{BASE_URL}/chat/completions",
    data=json.dumps(payload).encode(),
    headers={
        "Authorization": f"Bearer {API_KEY}",
        "Content-Type": "application/json",
    },
)

start = time.perf_counter()
try:
    with urllib.request.urlopen(req, timeout=8) as resp:
        body = json.loads(resp.read().decode())
        elapsed_ms = (time.perf_counter() - start) * 1000
        usage = body.get("usage", {})
        print(f"status={resp.status}")
        print(f"elapsed_ms={elapsed_ms:.1f}")
        print(f"prompt_tokens={usage.get('prompt_tokens')}")
        print(f"completion_tokens={usage.get('completion_tokens')}")
        print(f"total_tokens={usage.get('total_tokens')}")
        print(f"reply={body['choices'][0]['message']['content']!r}")
except urllib.error.HTTPError as e:
    print(f"status={e.code}")
    print(f"body={e.read().decode()[:200]}")
except Exception as e:
    print(f"error={type(e).__name__}: {e}")

Run it

export OPENAI_BASE_URL="https://your-provider.example/v1"
export OPENAI_API_KEY="your-key"
export MODEL_NAME="model-name-from-docs"
python endpoint_canary.py

Expected output:

status=200
elapsed_ms=742.3
prompt_tokens=18
completion_tokens=2
total_tokens=20
reply='pong'

Read the output

status=200 is necessary but not sufficient. A provider can return 200 and still send an HTML login page or an empty JSON body.
elapsed_ms tells you about the path from your current host, which matters more than a dashboard benchmark.
prompt_tokens and completion_tokens let you compare actual accounting with the free allowance. Do not hardcode the allowance; verify it each week.
reply='pong' confirms the endpoint accepted the message format, model name, and max_tokens setting.

One deliberate failure: the wrong key

export OPENAI_API_KEY="wrong-key"
python endpoint_canary.py

status=401
body={"error":{"message":"Invalid API key"}}

This failure is useful because it proves the script prints the response body. Many free-tier problems are hidden in that body, not in the HTTP status alone.

What this teaches

A one-request canary validates more than a login: route, auth, schema, model behavior, and token accounting.
Free model access is only useful if you can observe it from the host that will run the app.
A tiny deterministic prompt gives you a stable baseline; do not use a vague prompt because the reply itself becomes hard to verify.

Where this breaks

This is not a load test. It does not predict behavior under parallel requests.
It does not test streaming, tool calls, multi-turn memory, or model routing changes.
Free tiers can return 429 responses at different limits than their documentation suggests.
One green run does not mean the endpoint stays stable during a study session or demo.

Common mistakes

Printing only status and hiding the response body.
Assuming a 200 means the correct model answered.
Ignoring usage and then being surprised when the free allowance disappears.
Running the probe only from a laptop when the app will run on a free server.

Extension

Store each sample in SQLite and run the probe every five minutes.
On 429, read and respect Retry-After instead of looping immediately.
Add validation for choices, message, and usage so a malformed 200 fails loudly.
Run the same probe from both your laptop and your free server, then compare latency and failures.

Who should skip this

You need streaming, tool calls, multi-modal input, or real rate-limit simulation.
You already have server-side observability for model calls.
You are choosing a production provider with an SLA and support agreement.

If you already have access to MonkeyCode's free tier or any OpenAI-compatible lab endpoint, run this thirty-second canary before writing your RAG loop. The failure it surfaces is usually cheaper than a late-night debugging session.

Learn Fallback Logic by Building a Tiny Failure-Observer

Alex Chen — Fri, 14 Aug 2026 16:17:26 +0000

Most fallback code looks complete until you test the path where the primary endpoint fails.

This is a tiny Python harness that sends one prompt to a primary model endpoint, catches the failure, and falls back to a second endpoint. The learning question is small: which failures should trigger fallback, and how do you know the fallback actually ran?

Problem

A single model call is easy to reason about:

answer = call_model(PRIMARY_URL, prompt)

Add a fallback and the logic gets harder:

Which failures should route to fallback?
How many retries are worth paying for?
Did the fallback silently send the same prompt twice?
Can you see token consumption for each attempt?

Without an observed failure path, you are guessing.

Where free tokens fit

The operator-supplied material for this article describes MonkeyCode as an open-source project with free model access, a 30-million-token allowance, and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The token allowance and server availability were not independently measured for this article, so treat them as a disposable test environment, not as a production guarantee.

You do not need a paid key to practice the unhappy path. Use a free endpoint for the fallback, send a few failing fixtures, and read the token usage instead of guessing.

Prerequisites

Python 3.10+
Two OpenAI-compatible endpoints: one primary, one fallback
No third-party SDK required
Environment variables for both base URLs and keys

Build the harness

Save this as fallback_harness.py:

import json
import os
import time
import urllib.error
import urllib.request

PRIMARY_URL = os.environ['PRIMARY_URL']
PRIMARY_KEY = os.environ['PRIMARY_KEY']
FALLBACK_URL = os.environ['FALLBACK_URL']
FALLBACK_KEY = os.environ['FALLBACK_KEY']
MODEL = os.environ['MODEL']


def chat_once(base_url, api_key, model, prompt, timeout=8):
    payload = {
        'model': model,
        'messages': [{'role': 'user', 'content': prompt}],
        'temperature': 0,
    }
    req = urllib.request.Request(
        base_url.rstrip('/') + '/v1/chat/completions',
        data=json.dumps(payload).encode(),
        headers={
            'Authorization': 'Bearer ' + api_key,
            'Content-Type': 'application/json',
        },
        method='POST',
    )
    started = time.time()
    try:
        with urllib.request.urlopen(req, timeout=timeout) as resp:
            body = json.loads(resp.read().decode())
            text = body['choices'][0]['message']['content']
            usage = body.get('usage', {})
            return {
                'ok': True,
                'text': text,
                'prompt_tokens': usage.get('prompt_tokens', 0),
                'completion_tokens': usage.get('completion_tokens', 0),
                'seconds': round(time.time() - started, 2),
                'status': getattr(resp, 'status', 200),
            }
    except urllib.error.HTTPError as e:
        return {'ok': False, 'status': e.code, 'error': e.read().decode()[:120]}
    except Exception as e:
        return {'ok': False, 'status': None, 'error': type(e).__name__ + ': ' + str(e)}


def route(prompt, attempts=2):
    decisions = []
    result = chat_once(PRIMARY_URL, PRIMARY_KEY, MODEL, prompt)
    decisions.append(('primary', result['ok'], result['status'], result['error']))
    if not result['ok'] and PRIMARY_URL != FALLBACK_URL:
        for i in range(attempts):
            result = chat_once(FALLBACK_URL, FALLBACK_KEY, MODEL, prompt)
            decisions.append(('fallback', result['ok'], result['status'], result['error']))
            if result['ok']:
                break
            time.sleep(0.5 * (i + 1))
    return result, decisions


if __name__ == '__main__':
    prompt = 'Say the word ready.'
    result, decisions = route(prompt)
    print(json.dumps({**result, 'decisions': decisions}, indent=2))

Run it with two endpoints configured:

export PRIMARY_URL='https://primary.example'
export PRIMARY_KEY='primary-key'
export FALLBACK_URL='https://fallback.example'
export FALLBACK_KEY='fallback-key'
export MODEL='your-free-model'
python fallback_harness.py

The script sends the prompt to the primary endpoint. If the primary returns a non-200 status or raises a network error, it routes to the fallback and retries there.

Expected output

Force the primary endpoint to fail by setting PRIMARY_URL='http://127.0.0.1:9' (a port that refuses connections). The expected result looks like this:

{'ok': True, 'text': 'ready', 'prompt_tokens': 4, 'completion_tokens': 1, 'seconds': 1.1, 'status': 200, 'decisions': [('primary', False, None, 'ConnectionError: ...'), ('fallback', True, 200, None)]}

The important part is decisions: it shows the primary failed and the fallback succeeded. If you remove that field, the fallback is invisible.

Decision table

Observed condition	Harness action	Why it matters
Primary returns 200 with content	No fallback	Fast path, no duplicate prompt
Primary returns 401 or 403	Fallback	A bad key should not consume retries forever
Primary returns 429	Fallback	Rate limits are a common real failure
Primary times out	Fallback	Network failure is different from model refusal
Fallback also fails	Stop and return error	Retrying forever burns tokens

This table is deliberately small. The harness does not implement every branch; it gives you a visible state machine to reason about.

Common mistakes

Putting /v1 in PRIMARY_URL and appending /v1/chat/completions, producing /v1/v1/chat/completions.
Setting timeout=0.1, which turns slow models into false failures.
Assuming the response always contains usage; some compatible endpoints omit it.
Reading only the fallback output and never checking whether the primary request actually failed.

What you should understand

After completing this, you should see fallback logic as three states: primary -> fallback -> stop. The harness makes the transition visible and records token usage per attempt.

That small observation is the real skill. Production routers add retry budgets, circuit breakers, and backoff, but if you cannot observe a single fallback transition, those features are just extra code.

Limitations

The harness is not a load test or a production router.
Token allowance, free server, and availability are operator-supplied claims; verify them before relying on them.
The code does not validate model names or enforce quota.
It does not handle streaming responses or cancellation.

Who should not use this approach

Skip this if you need an approved provider list, security review for production, or multi-tenant quota enforcement. A tiny observer helps you learn failure behavior; it does not replace an operations policy.

Extension exercise

Add a circuit breaker that stops all calls after two fallback failures, then reload the process to confirm the breaker resets. Record the prompt_tokens and completion_tokens for each attempt to a CSV and draw the cost curve for a failing primary.

If you already have a MonkeyCode endpoint, use the free token allowance to run the failure fixtures against two configured bases. If not, any two OpenAI-compatible test endpoints will teach the same state machine.

Why a Free LLM Server Drops the Last Streamed Chunk: A Tiny SSE Recorder

Alex Chen — Fri, 14 Aug 2026 14:22:58 +0000

Most streaming LLM failures are not model failures; they are chunk-passing failures between a server and a client.

Current feeds are full of free model endpoints and free server tiers. The useful skill is not trying every new model. The useful skill is proving where a stream breaks. A recent DEV bug-smash challenge asked contributors to restore a dropped chat config in an SDK. That is the same class of silent stream loss this recorder catches.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What you will build

A two-file SSE recorder:

slow_sse_server.py fakes a slow stream.
sse_recorder.py prints raw chunks.
optional: run the recorder against a free MonkeyCode server/model endpoint to see the same contract.

No third-party packages. Python 3.11 or newer.

Prerequisites

Python 3.11+
two terminals
20 minutes
optional: a free MonkeyCode endpoint with its operator-reported 30,000,000-token allowance

Step 1: fake the slow server

Create slow_sse_server.py:

import http.server
import json
import os
import time

CHUNKS = [
    {'choices': [{'delta': {'content': 'Hello'}}]},
    {'choices': [{'delta': {'content': ' from'}}]},
    {'choices': [{'delta': {'content': ' the'}}]},
    {'choices': [{'delta': {'content': ' server.'}}]},
    {'choices': [{'delta': {}, 'finish_reason': 'stop'}]},
]

class Handler(http.server.BaseHTTPRequestHandler):
    def do_GET(self):
        self.send_response(200)
        self.send_header('Content-Type', 'text/event-stream')
        self.send_header('Cache-Control', 'no-cache')
        self.end_headers()
        for payload in CHUNKS:
            line = 'data: ' + json.dumps(payload) + os.linesep + os.linesep
            self.wfile.write(line.encode())
            self.wfile.flush()
            time.sleep(0.2)
        self.wfile.write(('data: [DONE]' + os.linesep + os.linesep).encode())
        self.wfile.flush()

if __name__ == '__main__':
    http.server.HTTPServer(('127.0.0.1', 8899), Handler).serve_forever()

Step 2: record what the client actually sees

Create sse_recorder.py:

import sys
import urllib.request

URL = sys.argv[1] if len(sys.argv) > 1 else 'http://127.0.0.1:8899/v1/chat/completions'

req = urllib.request.Request(URL, headers={'Accept': 'text/event-stream'})
with urllib.request.urlopen(req, timeout=10) as resp:
    print('status', resp.status)
    print('headers', dict(resp.headers))
    event = []
    for raw in resp:
        line = raw.decode('utf-8').rstrip()
        if line == '':
            if event:
                print('event', event)
                event = []
            continue
        if line.startswith(':'):
            print('comment', line)
            continue
        if line.startswith('data:'):
            event.append(line[5:].strip())
    if event:
        print('trailing', event)

Run it:

python slow_sse_server.py

In a second terminal:

python sse_recorder.py

Expected output:

status 200
headers contain text/event-stream
event count 5
final event [DONE]

Read the raw output before trusting a client

Check these five signals before you wrap a free server with an SDK:

number of data: events received
presence of finish_reason
presence of [DONE]
blank lines between events
any trailing event after the loop

A client abstraction can hide any of these. The recorder shows what was actually sent on the wire.

Where this breaks

Create not_sse_server.py:

import http.server
import json

class Handler(http.server.BaseHTTPRequestHandler):
    def do_GET(self):
        body = json.dumps({'choices': [{'delta': {'content': 'Hello'}}]})
        self.send_response(200)
        self.send_header('Content-Type', 'application/json')
        self.send_header('Content-Length', str(len(body)))
        self.end_headers()
        self.wfile.write(body.encode())

http.server.HTTPServer(('127.0.0.1', 8898), Handler).serve_forever()

Run it and point the recorder at it:

python not_sse_server.py
python sse_recorder.py http://127.0.0.1:8898/v1/chat/completions

Expected failure:

status 200
headers contain application/json
event count 1
no [DONE]

There is no blank line and no [DONE]. If your client expects an SSE delta, it may return an empty string, hang, or drop the final assistant message.

This is the same failure mode as a proxy that buffers a stream or strips newlines.

Fix checklist for a free server

send Accept: text/event-stream
disable output buffering if you control the proxy
read until EOF, not until the first blank line
print raw lines before parsing JSON
verify the final data: [DONE] or finish_reason

Where MonkeyCode fits

I use this recorder as a smoke test before I trust any free model endpoint. The operator reports that MonkeyCode offers free model access, a free server option, and a 30,000,000-token allowance.

Use it for:

small stream-contract tests
comparing one model response path
reproducing a dropped-chunk bug

Do not use it for:

production traffic
latency or quality benchmarks
assumptions about permanence; verify the current quota on the product page

Common mistakes

parsing only the first data: event
treating [DONE] as JSON
closing the connection after one blank line
ignoring response.status and response.headers

Extension

Change sse_recorder.py to record a timestamp and byte length for every chunk. Then compare the timestamps against your free server. If the first chunk arrives after a long delay, the problem is upstream buffering, not your parser.

If you try this, post the raw event count you observe before and after adding the Accept header.

Learn to Validate a Free LLM Endpoint by Building a Tiny Request Probe

Alex Chen — Fri, 14 Aug 2026 11:47:05 +0000

A free model endpoint is only useful if you can measure what free actually gives you.

Most free-tier pages show a token allowance and a server URL. They do not show how the endpoint behaves when a request is malformed, when the queue is busy, or when your retry logic is wrong. Before I let any free endpoint into a class project, I run a small probe.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The problem

A free LLM server can fail in at least three places that a quota page does not show:

it can accept a request and return an error after a long delay
it can return HTTP 200 with usage numbers that do not match your actual prompt
it can rate-limit after one request, so a batch job dies halfway

Token allowance is the easiest number to check. It is not the most important number.

What you will build

A standard-library Python probe, roughly 70 lines, that sends a chat-style request to any OpenAI-compatible endpoint, records latency and token usage, treats 429 and 5xx as retryable once, and prints a compact report.

Prerequisites:

Python 3.11 or newer
no external packages
an endpoint URL and API key from your provider dashboard
a model id that the endpoint accepts

The probe

Save this as free_probe.py.

import json
import os
import time
import urllib.error
import urllib.request

ENDPOINT = os.environ.get('LLM_ENDPOINT', 'https://api.example.com/v1/chat/completions')
API_KEY = os.environ.get('LLM_API_KEY', 'replace-me')
MODEL = os.environ.get('LLM_MODEL', 'replace-me')

def call_once(payload):
    data = json.dumps(payload).encode('utf-8')
    req = urllib.request.Request(
        ENDPOINT,
        data=data,
        headers={
            'Content-Type': 'application/json',
            'Authorization': f'Bearer {API_KEY}',
        },
        method='POST',
    )
    start = time.monotonic()
    try:
        with urllib.request.urlopen(req, timeout=30) as resp:
            body = resp.read().decode('utf-8')
            return resp.status, json.loads(body), time.monotonic() - start
    except urllib.error.HTTPError as exc:
        body = exc.read().decode('utf-8', errors='replace')
        return exc.code, {'error': body}, time.monotonic() - start
    except urllib.error.URLError as exc:
        return None, {'error': str(exc.reason)}, time.monotonic() - start

def main():
    payload = {
        'model': MODEL,
        'messages': [
            {'role': 'system', 'content': 'You are a minimal test harness.'},
            {'role': 'user', 'content': 'Reply with exactly the word pong.'}
        ],
        'temperature': 0,
        'max_tokens': 8,
    }

    attempts = []
    for attempt in range(2):
        status, body, seconds = call_once(payload)
        attempts.append((status, seconds, body))
        if status == 429 or (status is not None and status >= 500):
            print(f'attempt {attempt + 1}: retryable status {status} after {seconds:.2f}s')
            time.sleep(2 ** attempt)
            continue
        break

    status, seconds, body = attempts[-1]
    print('status:', status)
    print(f'latency: {seconds:.2f}s')

    if status != 200:
        print('error body:', json.dumps(body)[:300])
        return

    usage = body.get('usage', {})
    print('usage:', usage)
    content = body.get('choices', [{}])[0].get('message', {}).get('content', '')
    print('content:', repr(content))

    prompt_tokens = usage.get('prompt_tokens')
    completion_tokens = usage.get('completion_tokens')
    if prompt_tokens is None or completion_tokens is None:
        print('warning: usage fields missing; the endpoint may not report token accounting')

if __name__ == '__main__':
    main()

Run it with your real values:

export LLM_ENDPOINT='your provider /v1/chat/completions URL'
export LLM_API_KEY='your key'
export LLM_MODEL='the model id from the dashboard'
python free_probe.py

Expected output, with sample numbers:

attempt 1: retryable status 429 after 1.12s
status: 200
latency: 0.44s
usage: {'prompt_tokens': 17, 'completion_tokens': 1, 'total_tokens': 18}
content: 'pong'

These numbers are a sample, not a benchmark. Your endpoint, region, and current queue will change them.

How to read the result

A first-attempt 429 means capacity is shared. Do not start a batch at the top of an hour.
A 200 response with missing usage fields means you cannot track the advertised token allowance.
A prompt_tokens value far from your actual prompt means the tokenizer or accounting path is doing something unexpected.
Long latency on one small request may be cold start, not permanent speed.

One error input that matters

Change messages to an empty list and run the probe again:

payload = {
    'model': MODEL,
    'messages': [],
    'temperature': 0,
    'max_tokens': 8,
}

A well-built endpoint should reject this with HTTP 400. A sloppy wrapper may hang until the timeout. That difference matters when your client sends an empty context after a filtering step.

Where MonkeyCode fits

MonkeyCode advertises free model access, a free server option, and a 30 million token allowance. I treat those as operator-supplied claims to verify, not as permanent guarantees. As of 2026-08-14, the useful next step is to run this probe against the current endpoint and model id from the dashboard, then save the output as your baseline. If a single small request gets a 429 during a quiet hour, that baseline tells you more than the allowance number.

I did not include an endpoint or key here because those rotate and belong in your account settings, not in a tutorial.

Limitations

One request is not a load test, and latency is not p50 or p95.
A free server may be single-region and carry no uptime promise.
A 30 million token allowance can disappear quickly with long prompts or careless retries.
The probe does not test privacy, so do not send private code.

Who should not use this approach

production applications that need a latency SLO
large batch evaluations that need deterministic retry budgets
teams that cannot keep API keys out of source control

Extension exercise

Run the probe ten times with temperature: 0 and compare content. If the response changes, you may be hitting different backends or a nondeterministic router. Add median and maximum latency across twenty requests, then record a local token sum that stops before it reaches your own budget.

If you run this against a free endpoint, keep the printed report in your project notes. It is a more useful artifact than a screenshot of the quota page.

Build a Tiny Prompt-Invariance Canary Before Automating an LLM Task

Alex Chen — Fri, 14 Aug 2026 10:40:00 +0000

An LLM can be free, fast, and still flip a classification when the only change is a capital letter. That is enough to break an automation before it is useful.

Expected output after two files and one command:

$ python fake_endpoint.py &
$ python canary.py
base     APPROVE    PASS
case     REJECT     FAIL expected APPROVE
space    APPROVE    PASS
suffix   APPROVE    PASS

Learning question: for your task, which input changes are safe, and does the endpoint hold still when you apply them?

What we are testing

This is not an accuracy benchmark. A prompt-invariance canary only checks one thing: when a prompt changes in a way that should not change the output, the output stays the same. If casing, spacing, or punctuation flips a label, an agent that feeds generated text into this endpoint will fail in ways that are hard to trace later.

A canary is useful because running it is cheap. It is one small loop, not a dataset.

What you need

Python 3.10 or newer
Two terminal windows
No third-party packages

MonkeyCode offers free model access and a free server option, which make repeated small calls and a hosted rerun point cheap. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The demo below needs neither, so you can reproduce the failure with only Python.

Step 1: build a tiny fake endpoint

The fake endpoint is intentionally brittle. It returns APPROVE only when the exact lowercase substring approve if urgent appears in the last user message. A capital A makes it return REJECT.

Create fake_endpoint.py:

import json
from http.server import BaseHTTPRequestHandler, HTTPServer

RULE = 'approve if urgent'

class Handler(BaseHTTPRequestHandler):
    def do_POST(self):
        length = int(self.headers.get('Content-Length', 0))
        body = json.loads(self.rfile.read(length) or b'{}')
        content = body.get('messages', [{}])[-1].get('content', '')
        label = 'APPROVE' if RULE in content else 'REJECT'
        payload = json.dumps({'choices': [{'message': {'content': label}}]}).encode()

        self.send_response(200)
        self.send_header('Content-Type', 'application/json')
        self.send_header('Content-Length', str(len(payload)))
        self.end_headers()
        self.wfile.write(payload)

    def log_message(self, format, *args):
        pass

if __name__ == '__main__':
    HTTPServer(('127.0.0.1', 8765), Handler).serve_forever()

Run it in the first terminal and leave it running:

python fake_endpoint.py

Step 2: write the canary client

Create canary.py:

import json
import urllib.request

URL = 'http://127.0.0.1:8765/v1/chat/completions'

PAIRS = [
    ('base', 'Reply APPROVE or REJECT for: approve if urgent'),
    ('case', 'Reply APPROVE or REJECT for: Approve if urgent'),
    ('space', 'Reply APPROVE or REJECT for:   approve if urgent   '),
    ('suffix', 'Reply APPROVE or REJECT for: approve if urgent.'),
]

def call(prompt):
    data = json.dumps({'messages': [{'role': 'user', 'content': prompt}]}).encode()
    req = urllib.request.Request(URL, data=data, method='POST',
                                 headers={'Content-Type': 'application/json'})
    with urllib.request.urlopen(req, timeout=5) as response:
        body = json.loads(response.read())
    return body['choices'][0]['message']['content'].strip()

for name, prompt in PAIRS:
    label = call(prompt)
    status = 'PASS' if label == 'APPROVE' else 'FAIL expected APPROVE'
    print(f'{name:<8} {label:<10} {status}')

In the second terminal:

python canary.py

You should see:

base     APPROVE    PASS
case     REJECT     FAIL expected APPROVE
space    APPROVE    PASS
suffix   APPROVE    PASS

The one failing fixture

The only change in case is approve became Approve. The fake endpoint treats it as a different input and returns REJECT. That is the exact class of break you want to catch before it reaches an agent or a downstream API call.

What to do when a fixture fails

Write down the transformation that failed.
Decide whether that transformation should be semantically identical for your task.
If yes, fix the prompt, the output contract, or the parser.
Add a regression fixture so the same break cannot return silently.

Common mistakes

Testing only one prompt and calling the endpoint stable.
Comparing raw strings without trimming whitespace or fixing casing.
Confusing invariance with correctness: our fake endpoint can be stable for some changes and still be wrong about the task.
Assuming a free endpoint will keep the same behavior when the provider changes models.

Limitations

This canary only catches the transformations you list.
It does not measure accuracy, latency, cost, or calibration.
It does not prove a real provider is stable over time.
It is not a replacement for an eval set or human review in safety-critical settings.

Who should not use this approach

Teams that need a full accuracy benchmark across many examples.
Tasks where incorrect labels are expensive or dangerous; add human review.
Anyone who expects a free model endpoint to be permanent or semantically identical across upgrades.

What you should understand after completing this

A model call is stable only within a contract you define. Before automation, you need to say which input variations are safe and test them explicitly. The cheapest endpoint is not the one with the lowest price per token; it is the one that does not flip your labels on inputs you already decided are equivalent.

Try adding a probe that changes urgent to expedited. Decide first whether that should keep the label, then compare the result.

Learn Provider-Agnostic Model Routing by Building a Tiny LLM Switchboard

Alex Chen — Thu, 13 Aug 2026 03:04:50 +0000

Every few weeks a new model drops and my study group chat fills up with screenshots: "this one is cheaper," "this one is better at code," "switch now." I can never verify any of it quickly, because my test scripts all hard-code one provider's client. Rewriting the harness is slower than the hype cycle.

So here is the learning question: can a ~60-line, standard-library-only Python switchboard let me swap providers behind one interface, route toy tasks to different backends, and prove with a failing fixture where the routing breaks?

Run the final script and you should see:

[route=summarize] backend=local-echo   cost=0.0000  out='SUMMARY: ...'
[route=code]       backend=mock-strong cost=0.0020  out='def add(a, b): ...'
[route=summarize] backend=mock-strong cost=0.0020  out='SUMMARY: ...'  (fallback fired)

The third line is the interesting one — keep reading.

Prerequisites

Python 3.11+ (tested on 3.12.4, macOS and WSL2 Ubuntu)
No third-party packages. No API keys needed for the core experiment.

Why a switchboard, not a wrapper

A wrapper hides one provider's quirks. A switchboard does something different: it defines your task types (summarize, code, chat) and maps each to a backend plus a fallback. When a new model appears — whatever it is called this month — you add one entry and re-run the same fixtures. Your evaluation questions stay fixed while backends rotate under them.

That is the actual skill: separating what you ask from who answers.

The complete code

Save as switchboard.py:

"""Tiny provider-agnostic LLM switchboard (stdlib only)."""
from dataclasses import dataclass, field
from typing import Callable

# --- Backends: same signature (str) -> str, so they are interchangeable ---

def local_echo(prompt: str) -> str:
    """Free 'backend': deterministic, offline, good enough for smoke tests."""
    return f"SUMMARY: {prompt[:40]}"

def mock_strong(prompt: str) -> str:
    """Pretend paid model. Fails on empty input, like a real API rejects it."""
    if not prompt.strip():
        raise ValueError("backend rejected empty prompt")
    if prompt.startswith("write code"):
        return "def add(a, b):\n    return a + b"
    return f"SUMMARY: {prompt[:40]}"

@dataclass
class Backend:
    name: str
    fn: Callable[[str], str]
    cost_per_call: float  # USD, your own pricing notes go here

@dataclass
class Switchboard:
    routes: dict[str, list[Backend]] = field(default_factory=dict)
    spent: float = 0.0

    def register(self, task: str, backends: list[Backend]) -> None:
        self.routes[task] = backends

    def run(self, task: str, prompt: str) -> str:
        if task not in self.routes:
            raise KeyError(f"no route registered for task '{task}'")
        last_err = None
        for backend in self.routes[task]:
            try:
                out = backend.fn(prompt)
                self.spent += backend.cost_per_call
                print(f"[route={task}] backend={backend.name:<11} "
                      f"cost={backend.cost_per_call:.4f}  out={out.splitlines()[0]!r}")
                return out
            except Exception as e:  # try the fallback backend
                last_err = e
        raise RuntimeError(f"all backends failed for '{task}'") from last_err

if __name__ == "__main__":
    free = Backend("local-echo", local_echo, 0.0)
    paid = Backend("mock-strong", mock_strong, 0.002)

    sb = Switchboard()
    sb.register("summarize", [free, paid])   # cheap first, paid as fallback
    sb.register("code", [paid])              # only the 'strong' backend

    sb.run("summarize", "explain gradient descent to a first-year student")
    sb.run("code", "write code to add two numbers")
    sb.run("summarize", "   ")               # free backend accepts junk... or does it?
    print(f"total spent: ${sb.spent:.4f}")

Expected output

[route=summarize] backend=local-echo   cost=0.0000  out='SUMMARY: explain gradient descent to a f'
[route=code]       backend=mock-strong cost=0.0020  out='def add(a, b):'
[route=summarize] backend=local-echo   cost=0.0000  out='SUMMARY:     '
total spent: $0.0020

Wait — the third line did not fall back, and it returned a garbage summary of whitespace. My local_echo backend happily accepts an empty prompt. The failure I promised in the intro only fires if the primary backend raises. Before reading on: which fixture input would force the fallback line from the intro to appear? (Answer at the bottom.)

The error input that teaches the lesson

Swap the registration so the strict backend is primary:

sb.register("summarize", [paid, free])  # strict first, lenient as fallback
sb.run("summarize", "   ")

Now you get the intro's third line: mock-strong raises on the empty prompt, the switchboard catches it, and local-echo answers instead. The concept that actually matters: fallback order is a policy decision, and "free first" and "strict first" fail in opposite directions. Free-first silently returns junk; strict-first silently spends money when you expected the cheap path.

Common mistakes

Catching nothing or everything blindly. Catching Exception is fine for a teaching harness, but in real code you should distinguish "provider is down" (retry/fallback) from "my prompt was rejected" (fallback just re-fails expensively).
Comparing models on different prompts. If backend A sees a cleaned prompt and backend B sees the raw one, your cost/quality notes are meaningless. The switchboard passes the identical string to every backend — keep it that way.
Trusting launch-week pricing claims. When a new model is announced as "cheap and strong," verify pricing on the provider's own page before wiring it in as your default route. Announcement-week numbers change, and context-window limits quietly alter real cost per task.

Where I actually run this

I do most of these experiments inside MonkeyCode, since its free model access lets me prototype against a real LLM endpoint instead of only mocks, and the free server option means the harness runs somewhere that is not my laptop between classes. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The switchboard pattern above is deliberately provider-agnostic, though — the same file works against any endpoint, including none at all.

Limitations and who should skip this

This is a learning artifact, not production infrastructure: no retries with backoff, no rate-limit handling, no async, no token-level cost accounting (cost here is per-call fiction you fill in yourself).
If you already use a mature router/gateway library, this teaches you nothing new operationally — though building it once is still useful for understanding what that library hides.
Do not use fallback chains to mask evaluation problems. "It returned something" is not a quality signal.

Extension exercise

Add a third route, "chat", and a fixture file of five prompts where you predict in writing which backend should handle each one. Then make mock_strong randomly raise on 30% of calls (seed random for reproducibility) and check whether your predictions about cost still hold. If your cost estimate assumed zero failures, what does that tell you about launch-week pricing comparisons?

Answer to the question above

With summarize registered as [free, paid], only an input that makes local_echo itself raise would trigger the fallback — and local_echo never raises. The intro's third line only appears under the strict-first registration. If you predicted that, you understood the policy-ordering point; if not, run both versions and diff the output.

If you find a fixture input where the fallback makes things worse (e.g., the lenient backend returns something dangerously plausible), post it — minimal counterexamples are the best part of these threads.