DEV Community: Jordan Huang

Five Cold-Start Myths About Free Model Servers (and a Wake-Up Test)

Jordan Huang — Sun, 30 Aug 2026 15:45:06 +0000

The first time I curled a free model server, I expected a fast reply. I got a 2.4-second pause. The response itself was fine. The wait was not. So I ran a week of experiments. Now I want to share the five myths that keep tripping up teams.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Free model servers are not always-on APIs. They sleep between requests. When a request arrives, the server wakes up first. That wake-up changes your latency profile completely.

Myth 1: The server is always running

Most free tiers shut down idle instances. The first request after a quiet period pays a cold-start penalty. I measured consecutive calls to a fresh instance. The first took 2.4 seconds. The next took 310ms. Same endpoint. Same prompt. Two different worlds.

Myth 2: Cold starts are tiny

Some teams dismiss cold starts as a 100ms blip. My tests say no. With two minutes of idle time, latencies jumped to 1.8–3.1 seconds. Model size mattered. Small models woke faster. Large models took forever. Even the small ones spiked past 1.5 seconds.

Idle time	Median response	Overhead
0 min	320ms	1x
2 min	2.1s	~6x
5 min	2.7s	~8x

Don't trust my table. Run your own. Providers change timeouts, pools, and routing weekly.

Myth 3: One warm-up request fixes it

I see keep-alive loops everywhere. A request every 30 seconds. That works only if the idle timeout is longer. Many free servers idle out after 1–5 minutes. Your keep-alive interval might be too sparse. Or the load balancer sends you to a different instance. Your warm-up call hit a warm one. The real request hit a cold one.

Myth 4: Stable latency means a stable server

Free instances share CPU with neighbors. A busy neighbor steals cycles. I fired 10 parallel requests at my "reliable" endpoint. Latency climbed from 210ms to 4.9 seconds. Some requests failed completely. Nothing in the logs explained it. That is the reality of shared capacity.

Myth 5: Free means production-ready

Free model servers are best-effort. There's no SLA, no guaranteed capacity, no 99.9% uptime. They throttle under load. They drop silent requests. They restart without notice. If you route real user traffic through one, you are gambling with latency.

The Wake-Up Test

I wrote a small script to measure the true cost of waking a server. It needs an OpenAI-compatible endpoint. It sends one request to warm the instance. Then it sleeps and sends again. Compare the times.

#!/bin/bash
# wake-up-test.sh — measure cold starts on a free model server
ENDPOINT="${1:?ENDPOINT needed}"
API_KEY="${2:?API_KEY needed}"
IDLE="${3:-120}"

call() {
  curl -s -o /dev/null -w "%{time_total}" \
    -H "Authorization: Bearer $API_KEY" \
    -H "Content-Type: application/json" \
    -d '{"model":"your-model","messages":[{"role":"user","content":"ping"}],"max_tokens":1}' \
    "$ENDPOINT/v1/chat/completions"
}

echo "Warm request: $(call) sec"
sleep 5
echo "Hot request:  $(call) sec"

for round in 1 2 3; do
  sleep "$IDLE"
  echo "After ${IDLE}s idle: $(call) sec"
done

Replace the model name, endpoint, and key. Then watch the last three lines. If the "after idle" times are several times higher, you have cold starts.

Reading the Results

What should you look for?

Warm request under 500ms: the server is alive.
Hot request similar: no immediate penalty.
After 120s idle: expect 2x–8x jumps.
After 300s idle: often worse.

What if the idle time makes no difference? That's useful too. Either the server never sleeps, or someone else keeps it warm. You still learned something.

Limitations

Free tiers change quickly. My numbers are not yours. A provider can tighten idle timing next week. Another can add more capacity. Treat this article as a method, not a verdict.

Who Should Skip Free Servers

Do not put a free model server behind a live user interface. Do not use it for customer-facing chatbots or any latency-sensitive workflow. These servers are for prototypes, demos, and personal experiments. If you need predictable p99, pay for a guaranteed tier.

The Honest Take

Free model access is a gift. It lets you test prompts, build a proof of concept, or learn a new SDK. But the "free" part is the price, not the performance. Plan for cold starts. Measure them. Then decide if that's acceptable for your use case.

MonkeyCode's free model access and free server option are one place where this test works. So is any other free tier you trust. Run the script. Look at the numbers. Then build accordingly.

Your Model Calls Got Slower. Run git bisect Before You Blame the Server

Jordan Huang — Sun, 30 Aug 2026 12:42:55 +0000

Say your dashboard looks like this. Tuesday: 180 ms per call. Wednesday: 1.4 s. No model upgrade. No endpoint change. Retries grew. The dashboard blamed the server. The server blamed the weekend.

Model latency regressions are rarely model regressions. They are request-shape regressions. A prompt template moved from v2 to v3. A cache key started including a timestamp. A middleware appended four lines to your system prompt. Your dashboard sees a slow server. Your git history sees a guilty commit.

This is a three-myth FAQ plus a script. The myths fix your mental model. The script automates the blame. It works with any OpenAI-compatible chat endpoint, including MonkeyCode's free model routes. When the free server gets noisy, move the probe to a quieter one before calling any commit guilty. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Three myths to drop

1. "Slow calls mean slow infrastructure."

Shared free servers have noisy neighbors. I wrote about that separately. Neighbor noise rarely creates a clean step function. A jump that lines up with a merge is your code, not the host.

Run a two-call probe. First call cold. Second call warm. If the warm call is fast, your cache key changed. Cache keys change inside commits. That is bisectable.

2. "The dashboard will name the commit."

Dashboards aggregate by time, not by commit. They show a spike at 14:02. They do not show the merge at 14:01.

You must build that mapping. Treat dashboards as alarms. Treat git history as the detective.

3. "git bisect only finds compile failures."

False. git bisect run executes any script. Your script exits 0 for good. It exits non-zero for bad. Slow responses are observable states. So are cache misses, token counts, and prompt sizes.

History becomes a binary search. The cause gets isolated in O(log n) checkouts.

The reproducibility script

Save this as ~/bin/latency_bisect.sh. Keep it outside the repository you are probing, or bisect will migrate your test tool into the feedback loop.

#!/usr/bin/env bash
# usage:
#   ENDPOINT=... MODEL=... TOKEN=... THRESHOLD_MS=1200 bash ~/bin/latency_bisect.sh
set -euo pipefail

: "${ENDPOINT:?set ENDPOINT}"
: "${MODEL:?set MODEL}"
: "${TOKEN:?set TOKEN}"
THRESHOLD_MS="${THRESHOLD_MS:-1200}"

now_ms() { perl -MTime::HiRes=time -e 'printf "%d\n", time * 1000'; }
body_file="${TMPDIR:-/tmp}/lb_body_$$.json"

# Optional: use the repo's committed request generator.
if [ -f ./build_request.sh ]; then
  bash ./build_request.sh "$MODEL" > "$body_file"
else
  printf '{"model":"%s","messages":[{"role":"user","content":"say ok"}]}' "$MODEL" > "$body_file"
fi

start=$(now_ms)
if ! curl_out=$(curl -sS -o "$body_file.resp" \
  --max-time 30 \
  -H "Authorization: Bearer ${TOKEN}" \
  -H "Content-Type: application/json" \
  --data @"$body_file" \
  -w "%{http_code} %{time_starttransfer} %{time_total}" \
  "$ENDPOINT"); then
  echo "curl failed on this commit" >&2
  exit 125 # tell git bisect to skip this commit
fi
end=$(now_ms)
read -r http_code ttfb_s total_s <<< "$curl_out"

ttfb_ms=$(awk -v x="$ttfb_s" 'BEGIN{printf "%d", x * 1000}')
total_ms=$(( end - start ))

echo "http=$http_code ttfb_ms=$ttfb_ms total_ms=$total_ms"

# Exit 0 means "good enough"; non-zero means "this commit is bad".
[ "$http_code" = "200" ] && [ "$total_ms" -lt "$THRESHOLD_MS" ]

Why Perl? macOS ships it, while date +%s%N needs GNU coreutils. Why awk? curl prints decimal seconds, and shells cannot do float math. The exit 125 line tells git bisect run to skip a commit when the network fails. Accidentally slow is not the same as untestable.

Assumptions: model names are simple identifiers. For JSON-heavy names, replace the printf with jq -n --arg model "$MODEL" .... I keep the fallback short on purpose.

The three-step bisect

Start from a clean worktree. git bisect run does not overwrite local edits.
Mark your brackets. Good is a known fast commit. Bad is current main.

git status --porcelain

git bisect start
git bisect bad main
git bisect good HEAD~50
git bisect run bash ~/bin/latency_bisect.sh

Read the printed commit id. Save the evidence.

git bisect log > bisect-evidence.txt
git bisect reset

Reading the two numbers

Symptom	Likely cause	Look at
`ttfb_ms` high	Queueing or server-side load	Quota, neighbor traffic, retry storms
`total_ms` high, `ttfb_ms` fine	Longer generation	`max_tokens`, prompt length, cache misses
First call slow, second fast	Prefix-cache invalidation	Recent template or cache-key commits
Both high everywhere	Network path	DNS, proxy, rotated token

Do not trust one sample. This is a blame divider, not a benchmark. When the free server looks nervous, run the script twice per commit and keep the lower value.

What this script will not catch

Prompt-shape-only changes. The fixed fallback always asks "say ok". If the slow case needs a 4k-token contract, the sample stays fast on every commit. Fix: commit a build_request.sh inside the repo under test. The test script runs it when present, so the old generator travels with old commits.

Neighbor effects. They are time-based, not commit-based. A busy neighbor makes the whole run noisy. Move to a quieter window or a quieter server first.

Slow drift. Bisect needs a step change. If latency creeps upward for months, there is no clean bad commit. Fix the baseline before you bisect.

Throughput and quota curves. This sends one request per commit. It does not stress test. It does not measure concurrency. It finds the first commit that made a single call slower.

Who should not use this

Teams with no known-good commit. Without a bracket, bisect has no answer.
Gateway-only users. If you cannot modify the request path, you will bisect someone else's code.
Anyone storing tokens in the repository. Endpoint secrets belong in env files, not in the history being searched.

The correction

Slow calls are symptoms. Commits are causes. Dashboards summarize. Git bisect decides.

Next time the p50 jumps, run the script first. Let the binary search find the guilty merge. Then blame the server with evidence it cannot argue with.

Your Free Model Quota Is a Bank Account. Build a Budget Proxy on a Free Server

Jordan Huang — Sun, 30 Aug 2026 11:37:16 +0000

Your team got a free model API key last month. Everyone loved it. Then, one morning, every call started returning 429. No warning. No dashboard. No idea which project burned the quota.

I've been there. Free tiers are generous until they aren't. The fix isn't another monitoring SaaS. It's a tiny proxy that counts tokens before they vanish.

This article walks you through a reproducible budget proxy. You'll run it on a free server, give it a daily token limit, and watch it reject expensive calls automatically. All with open-source Python.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode offers free models and a free server tier for experiments like this. That means you can test the whole workflow below without paying for infrastructure.

Why a Budget Proxy?

Raw API calls are dangerous. Every dev on your team sends their own prompts. Some are small. Some dump a 50k-token context. You have no central place to say: “we've spent enough today.”

The proxy sits between your code and the model API. It sees every request, reads the response usage, stores the running total, and blocks new calls when the daily cap is hit. Simple, cheap, auditable.

Step 1: Understand the `usage` Field

Most OpenAI-compatible endpoints return a usage object in each response. It looks like this:

{
  "usage": {
    "prompt_tokens": 1200,
    "completion_tokens": 350,
    "total_tokens": 1550
  }
}

That total_tokens is your meter. The proxy reads it and adds it to a local counter. No guessing, no heuristics.

Step 2: Build the Proxy

We'll use Flask because it's minimal. Save this as budget_proxy.py:

import json
import os
import time
import requests
from flask import Flask, request, jsonify

app = Flask(__name__)

# Change these constants for your environment
DAILY_BUDGET = 500_000  # tokens
USAGE_FILE = "usage.json"


def load_usage():
    """Return today's spent tokens and the date string."""
    today = time.strftime("%Y-%m-%d")
    if os.path.exists(USAGE_FILE):
        with open(USAGE_FILE) as f:
            data = json.load(f)
        if data.get("date") == today:
            return data["spent"]
    return 0


def save_usage(spent):
    with open(USAGE_FILE, "w") as f:
        json.dump({"date": time.strftime("%Y-%m-%d"), "spent": spent}, f)


@app.route("/v1/chat/completions", methods=["POST"])
def proxy():
    spent = load_usage()
    if spent >= DAILY_BUDGET:
        return jsonify({"error": "Daily budget exceeded"}), 429

    payload = request.json
    # Forward to your model endpoint. Replace URL with the real one.
    upstream = "https://api.example.com/v1/chat/completions"
    headers = {"Authorization": request.headers.get("Authorization")}

    try:
        resp = requests.post(upstream, json=payload, headers=headers, timeout=60)
        data = resp.json()
        usage = data.get("usage", {})
        tokens = usage.get("total_tokens", 0)
        if tokens:
            save_usage(spent + tokens)
        return jsonify(data), resp.status_code
    except requests.exceptions.Timeout:
        return jsonify({"error": "Upstream timeout"}), 504


if __name__ == "__main__":
    app.run(host="0.0.0.0", port=5000)

The proxy stores usage in a JSON file. For a low-traffic personal setup, that's enough. If you expect many concurrent requests, swap JSON for SQLite.

Step 3: Add a Reset Policy

The load_usage() function already resets the counter when the date changes. The first call of each day starts from zero. That's the default behavior. If you need a rolling 24-hour window instead, adjust the logic with a timestamp instead of a date string.

Step 4: Test Locally

Run the proxy:

python budget_proxy.py

In a second terminal, send a fake request:

curl http://localhost:5000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_KEY" \
  -d '{"model": "gpt-3.5-turbo", "messages": [{"role": "user", "content": "Hello"}]}'

You'll get a response from the upstream API (or a timeout error if the endpoint is invalid). Check usage.json:

cat usage.json

You should see today's date and the token count from your test.

Step 5: Deploy to a Free Server

MonkeyCode's free server tier runs small Python apps like this without a container registry. Clone your repo, set up the environment, and start the process.

Here's a minimal requirements.txt:

flask==2.3.3
requests==2.31.0

Then run:

pip install -r requirements.txt
gunicorn -w 2 -b 0.0.0.0:8000 budget_proxy:app

Gunicorn gives you multiple workers. Note: the JSON file approach has a race condition when two workers write at once. For a serious deployment, switch to SQLite or a simple lock.

Step 6: Point Your Code at the Proxy

Now, every client should call your proxy URL instead of the model API directly. In OpenAI's Python SDK:

from openai import OpenAI

client = OpenAI(
    base_url="https://your-proxy.example.com/v1",
    api_key="your-api-key"
)

response = client.chat.completions.create(
    model="your-model",
    messages=[{"role": "user", "content": "Hello"}]
)

That's it. Every request now passes through your budget gate.

What This Proxy Does NOT Do

It doesn't sanitize prompts or block prompt injection.
It doesn't rotate API keys.
It doesn't give you a metrics dashboard.
It doesn't handle upstream rate limits separately.

If you need those, extend the proxy or pair it with a proper gateway.

When Should You Not Use This?

Skip the proxy when:

You already have an enterprise gateway with usage tracking.
You're building a high-throughput production service where latency and reliability matter.
Your free server is too small to handle the extra hop.

For experiments, internal tools, and small teams, it's perfect.

Final Thoughts

A free model quota disappears faster than you expect. The trick is to see it coming. This proxy gives you a daily stop-loss without adding complex infrastructure.

Try it with MonkeyCode's free model access and free server. Deploy the proxy, set a conservative budget, and let it protect you from your own over-eager prompts. Your future self will thank you.

Who Owns the Deadline? Four Timeout Myths That Break Free Model Calls

Jordan Huang — Sun, 30 Aug 2026 09:39:13 +0000

Your service calls a free model endpoint. The client timeout is 30 seconds. The model answers in 35.

Who pays the extra five seconds?

Your user does. So does every service behind you.

I run small services against free model servers. MonkeyCode's free model access and free server option keep the barrier low. Low cost is good. Low cost is not low complexity.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

This post is a myth-busting FAQ. Each myth is a claim I keep seeing in code. Each myth gets evidence you can reproduce. There is a runnable probe at the end.

The Myths, In One List

A client timeout is a deadline.
Longer timeouts are safer.
Only the model call needs a timeout.
A connect timeout is enough.

Here is the corrected mental model: timeouts are local limits. Deadlines are end-to-end promises. You need both. Most code only has the first.

Myth 1: A Client Timeout Is a Deadline

What I hear: 'The timeout is 30s, so the function can't hang.'

True. A function cannot hang forever. That is a weak guarantee.

A deadline comes from outside. A user waits two seconds. A queue pops after five. An upstream SLO fails at ten. Your function has no opinion about any of those.

Now chain a few services:

Service A waits for Service B. Timeout: 30s.
Service B waits for the model. Timeout: 30s.
The model is slow. It takes 35s.

What happens? The model finishes at 35s. B abandoned the call at 30s. B returns an error. A waits for a useful answer but now has no time left. The user left after 10s.

The timeout protected nobody. It was local. The deadline was global.

Correction: name the deadline first. Then divide it into slices. Only then pick relative timeouts.

Myth 2: Longer Timeouts Are Safer

What I hear: 'Free servers are slow. Give them a full minute.'

Run the arithmetic. Two nested calls, 60s each. Worst case before your code learns: 120 seconds.

That 120-second window is not free. You hold a connection. You hold a worker. You might hold a database handle. Queueing delays multiply downstream.

A generous timeout does not fix a slow tail. It just waits for the whole tail.

Correction: a timeout is a budget cap. Want to be safer? Fail faster with a useful fallback. A clear 504 beats a late 200.

Myth 3: Only the Model Call Needs a Timeout

What I hear: 'Everything around the call is fast. DNS, JSON, logs — no risk.'

Really? DNS can hang. A cold database can hang. Logs can block under pressure.

Every hop between request and response is latency. None of them is protected by the model timeout.

Correction: measure every hop. Allocate each one a slice. Slice 1: DNS lookup. Slice 2: preprocessing. Slice 3: the model call. Slice 4: validation and response write.

Slices are not guesses. They come from a probe.

Myth 4: A Connect Timeout Is Enough

What I hear: 'If the server is down, connect fails. Timeout on connect only.'

Here is the gap: a server can accept a TCP connection instantly. Then it thinks for a long time. Then it sends the first token.

Or it never sends anything.

Your connect timeout never fires. Your code sleeps on the read. Nobody set a read timeout.

Correction: timeouts on connect, read, and write are separate. A total budget is separate again. In async Python, wrap the whole step in asyncio.timeout.

The Mental Model That Fixes All Four

Write it down before you code.

A request arrives. Set a deadline: request_start + 2.0s.
Pass the deadline down the stack as an absolute time. Relative timeouts become slices.
Each hop asks: How much budget is left?
If the answer is zero, stop. Return 504 with the reason.

Here is a small async template:

import asyncio
from time import monotonic

class BudgetExhausted(Exception):
    pass

def remaining(deadline: float) -> float:
    return deadline - monotonic()

async def call_model(client, payload: dict, deadline: float):
    left = remaining(deadline)
    if left <= 0:
        raise BudgetExhausted()
    # The whole I/O step gets one slice of the budget.
    async with asyncio.timeout(left):
        return await client.post('/chat', json=payload)

Not magic. The template forces one question: How much time is really left? Relative timeouts never ask that.

A Probe You Can Run

This is the reproducible part. Point it at any HTTP endpoint you use. It warms up and prints per-hop times. You decide the slices from the output, not from a blog post.

import asyncio
import json
import os
import time
import urllib.request

URL = os.environ.get('MODEL_URL', 'http://127.0.0.1:8000/chat')

def one_call(payload: dict) -> float:
    data = json.dumps(payload).encode()
    req = urllib.request.Request(URL, data=data, method='POST')
    t0 = time.perf_counter()
    with urllib.request.urlopen(req, timeout=30) as resp:
        resp.read()
    return time.perf_counter() - t0

async def main():
    payload = {'prompt': 'ping', 'max_tokens': 8}
    # Warm-up beats first-call surprises.
    for _ in range(3):
        one_call(payload)
    times = sorted(one_call(payload) for _ in range(10))
    median = times[len(times) // 2]
    p90 = times[int(len(times) * 0.9) - 1]
    print(f'median={median*1000:.0f}ms p90={p90*1000:.0f}ms')

asyncio.run(main())

Change the URL. Change the payload. Run it twenty times.

If your p90 is triple your median, a fixed 30-second client timeout is a lottery ticket. You want a deadline budget that includes the tail.

Decision Table: When the Budget Runs Out

The hard part of a deadline budget is the response.

Situation	Evidence in your probe	Action
Connect fails fast	OSError within 50ms	Retry once. Connect failures are often transient.
Read times out, first token arrived	Partial body streamed	Do not retry. State is ambiguous. Return 504.
Read times out, nothing arrived	0 bytes in the window	Retry once only if the caller allows it.
Total budget exhausted	Median and p90 confirm it	Stop. Degrade to a fallback or an error.

The table keeps decisions where the data lives. This script measures. It does not retry. Recovery is where your product policy starts.

Who Should Skip This Approach

This is not a universal law. Skip it when:

You run one-shot batch jobs. A 60-second wait is fine at 2 AM.
You have no user-facing request. No deadline exists.
Your chain has one hop. One timeout equals one deadline.

Otherwise the claim holds: your code needs deadlines, not just timeouts. Free model servers are a great place to practice, because the tail is visible. If you handle the budget there, your paid-tier calls will feel boring — and boring is good.

Next time you adjust a timeout, write the deadline down in code. Then time every hop. The answer often surprises you. It will also surprise your users less.

Streaming Responses Have a Silent Gap: A Three-Myth FAQ

Jordan Huang — Sun, 30 Aug 2026 06:36:41 +0000

I was watching a chat UI print one character, stall, print two, stall again. The first guess is a dead server. The second guess is a broken proxy. Both guesses miss.

The stream is alive. The tokens just arrive in the wrong rhythm.

This FAQ busts three streaming myths I keep seeing in free-tier integrations. MonkeyCode has a free model access path and a free server option, which makes an experiment like this cheap to run. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

You do not need MonkeyCode to run the probe. Any endpoint that streams SSE events works.

Myth 1: Streaming always makes the response faster

Streaming moves the first byte earlier. It does not promise a shorter total trip. A stream can take longer than a buffered response and still feel faster.

Measure two numbers, not one:

First-token latency: time from request until the first real SSE data line.
Inter-token gap: time between consecutive SSE data lines.

Both matter. Chat UIs care about the first. Linters and validators care about the second. A whole paragraph can arrive, then silence. That silence is part of the latency budget.

Correct mental model: streaming fast means low first-token latency. Total duration belongs to the buffered path. Do not call it faster until you measure both.

Myth 2: The end of the stream means the answer is complete

An SSE stream ends when the server closes the connection. That is not the same as a finished model response. Proxies, CDNs, and free-tier gateways can close long-lived streams.

Many chat-completions providers use a [DONE] sentinel as their end-of-stream marker. Check for it before you treat the response as complete. The stream may also end inside a JSON fragment. Count the closing brace, or prefer a client that understands event boundaries.

Correct mental model: HTTP response complete is not model output complete. A stream without its sentinel is an interrupted candidate, not a result.

Myth 3: A quiet stream is a slow one, so wait longer

A quiet stream can be a blocked stream. The socket is open, but no token is coming. The server might be scheduling work, GCing, waiting on a queue, or stuck behind a hidden buffer.

The fix is an idle timeout, not a total timeout. Track the time since the last SSE event. If that gap passes your threshold, abort and plan a retry. A total timeout kills busy streams that still make progress.

Try this rule: if one inter-token gap is more than five times the median gap, your client should record it. The mean hides it. The max catches it too late.

Correct mental model: silence is a metric, not a verdict. Record the gap length, abort deliberately, and classify the failure before retrying.

The probe

I wrote a small Node script that watches a streaming chat completion. It parses SSE data lines, records inter-token gaps, and prints the distribution. Run it against an endpoint you own.

// stream-probe.mjs
// ENV: ENDPOINT, TOKEN, MODEL (optional), IDLE_TIMEOUT_MS (optional), MAX_MS (optional)
import { performance } from 'node:perf_hooks';

const {
  ENDPOINT,
  TOKEN,
  MODEL = 'placeholder-model',
  IDLE_TIMEOUT_MS = 5000,
  MAX_MS = 120000
} = process.env;

if (!ENDPOINT || !TOKEN) {
  console.error('Set ENDPOINT and TOKEN.');
  process.exit(1);
}

const controller = new AbortController();
const started = performance.now();

let bytes = 0;
let firstEventAt = null;
let lastEventAt = null;
let doneSeen = false;
const gaps = [];
let buffer = '';
let idleTimer;
let timer = setTimeout(() => abort(`stream longer than ${MAX_MS}ms`), Number(MAX_MS));

function abort(reason) {
  if (!controller.signal.aborted) controller.abort(new Error(reason));
}

function resetIdleTimer() {
  clearTimeout(idleTimer);
  idleTimer = setTimeout(() => abort(`no SSE event within ${IDLE_TIMEOUT_MS}ms`), Number(IDLE_TIMEOUT_MS));
}

function handleSse(raw) {
  buffer += raw;
  let sep;
  while ((sep = buffer.indexOf('\n\n')) !== -1) {
    const event = buffer.slice(0, sep);
    buffer = buffer.slice(sep + 2);
    const data = event.split('\n').find((line) => line.startsWith('data:'))?.slice(5).trimStart();
    if (!data) continue;
    const now = performance.now();
    if (firstEventAt === null) firstEventAt = now - started;
    gaps.push(now - (lastEventAt ?? started));
    lastEventAt = now;
    if (data === '[DONE]') doneSeen = true;
    resetIdleTimer();
  }
}

const response = await fetch(ENDPOINT, {
  method: 'POST',
  headers: {
    'content-type': 'application/json',
    authorization: 'Bearer ' + TOKEN
  },
  body: JSON.stringify({
    model: MODEL,
    messages: [{ role: 'user', content: 'Write one short paragraph about measuring streams.' }],
    stream: true
  }),
  signal: controller.signal
});

if (!response.ok) throw new Error('HTTP ' + response.status + ' ' + response.statusText);

const reader = response.body.getReader();
const decoder = new TextDecoder();

try {
  resetIdleTimer();
  while (true) {
    const { done, value } = await reader.read();
    if (done) break;
    bytes += value.byteLength;
    handleSse(decoder.decode(value, { stream: true }));
  }
} catch (err) {
  if (err.name !== 'AbortError') throw err;
  console.error('aborted:', err.message);
} finally {
  handleSse('\n\n');
  clearTimeout(idleTimer);
  clearTimeout(timer);
}

const durationMs = performance.now() - started;
const sorted = [...gaps].sort((a, b) => a - b);
const n = sorted.length;
const median = n ? n % 2 ? sorted[Math.floor(n / 2)] : (sorted[n / 2 - 1] + sorted[n / 2]) / 2 : 0;
const p90 = n ? sorted[Math.min(n - 1, Math.floor(n * 0.9))] : 0;

console.log(JSON.stringify({
  status: response.status,
  durationMs: Math.round(durationMs),
  firstEventMs: firstEventAt === null ? null : Math.round(firstEventAt),
  bytes,
  complete: doneSeen,
  gapsMs: {
    n,
    min: Math.round(sorted[0] ?? 0),
    median: Math.round(median),
    p90: Math.round(p90),
    max: Math.round(sorted[n - 1] ?? 0)
  }
}, null, 2));

Reading the output

The script prints JSON. The shape looks like this:

{
  "status": 200,
  "durationMs": 9120,
  "firstEventMs": 312,
  "bytes": 4102,
  "complete": true,
  "gapsMs": {
    "n": 74,
    "min": 8,
    "median": 12,
    "p90": 35,
    "max": 4200
  }
}

Treat those numbers as a shape, not a benchmark. Your provider will show a different story. Look at the relationship between the fields:

High firstEventMs means the server queued the whole response before sending.
One big max is a scheduling hiccup. Record it, keep going.
Many high inter-token gaps mean a stalling stream.
complete: false means the sentinel never arrived, even if HTTP says 200.

Decision table

You see	Likely cause	What helps
High first-event latency	Network or server queue	Warm up, then stream again
One huge gap	Server scheduling or GC	Log it; do not retry blindly
Repeated large gaps	Proxy buffering or read backpressure	Shorten prompt, read chunks faster
`complete: false`	Interrupted stream	Idle timeout, surface interrupted state

Limitations

This probe measures what your client receives, not what the model generates. A CDN that batches tokens will inflate your gaps. That is still useful for UX, but it is not a server benchmark.

Do not use it as a load test. One request at a time. Free tiers will not enjoy a hammering client.

Do not use streaming when you are writing output to a file. Buffer the whole JSON and validate it. Streaming earns its complexity in chat, typeahead, and long-running generation.

Bottom line

Before you file an outage ticket, ask for the inter-token gap histogram. A stream can be alive and still be unusable. Measure the rhythm, set an idle timeout, and treat an ending without a sentinel as incomplete.

The myth is that a stream either works or fails. The reality is that a stream degrades between pixels. That is where you debug next.

The Neighbor Effect: A Measured FAQ About Free Model Servers

Jordan Huang — Sun, 30 Aug 2026 03:32:39 +0000

Your free model server is a shared parking lot. Other cars arrive all day. Their engines change your commute. Most latency posts pretend the road is empty. Shared free tiers never are. Here is a myth-busting FAQ. It comes with one reproducible probe. You can run it in an afternoon. No vendor, no hype.

Why bother? Because your client code makes assumptions. A solo run teaches you nothing about contention. Under a neighbor burst, assumptions fail. The failure shows up in production, not in tests. This probe makes contention visible early.

The probe

The probe sends the same prompt at rising concurrency. One call, then two, then four, then eight. It records latency for every request. Run it at different hours. Watch the spread, not just the average.

# shared_fate_probe.py
# Probe a shared free model endpoint without hammering it.
# Run one concurrency level at a time. Ask the endpoint owner first.

import asyncio
import json
import os
import time
import urllib.request
from typing import List

ENDPOINT = os.environ.get("FREE_MODEL_URL", "https://example.invalid/v1/chat/completions")
TOKEN = os.environ.get("FREE_MODEL_TOKEN", "")
PROMPT = "Reply with one word: ready."

def one_call() -> float:
    payload = json.dumps({
        "model": "sample-model",
        "messages": [{"role": "user", "content": PROMPT}],
        "temperature": 0,
        "max_tokens": 4,
    }).encode("utf-8")
    req = urllib.request.Request(ENDPOINT, data=payload, headers={
        "Content-Type": "application/json",
        "Authorization": f"Bearer {TOKEN}",
    })
    start = time.perf_counter()
    with urllib.request.urlopen(req, timeout=10) as resp:
        resp.read()
    return (time.perf_counter() - start) * 1000

async def run_batch(concurrency: int) -> List[float]:
    return await asyncio.gather(
        *(asyncio.to_thread(one_call) for _ in range(concurrency))
    )

async def main() -> None:
    for level in (1, 2, 4, 8):
        samples: List[float] = []
        for _ in range(5):
            samples.extend(await run_batch(level))
            await asyncio.sleep(1)  # Your sleeping spot in the lot.
        samples.sort()
        p50 = samples[len(samples) // 2]
        p95 = samples[max(0, int(len(samples) * 0.95) - 1)]
        print(
            f"concurrency={level} n={len(samples)} "
            f"p50={p50:6.0f}ms p95={p95:7.0f}ms max={max(samples):7.0f}ms"
        )

if __name__ == "__main__":
    asyncio.run(main())

Sample output, marked clearly as illustrative:

concurrency=1 n=5  p50=  812ms p95= 1043ms max= 1102ms
concurrency=2 n=10 p50=  901ms p95= 1298ms max= 1401ms
concurrency=4 n=20 p50= 1128ms p95= 2104ms max= 2234ms
concurrency=8 n=40 p50= 1906ms p95= 3841ms max= 4107ms

That output is not a benchmark of any product. Your numbers will differ. The shape matters more than the sizes.

How to read the output

Start at the concurrency=1 line. That is your solo reference. Compare p95 growth across levels.

If p95 stays flat, the endpoint tolerates parallel calls.
If p95 bends upward, you hit the shared edge.
If max spikes without warning, a neighbor burst arrived.
If p50 and p95 move together, the queue is growing.

Keep every run in a file. One screenshot teaches nothing. A week of runs reveals the pattern. Save the time window with each sample.

Now the three myths.

Myth 1: "My latency is my own"

Queueing theory says it plainly. Delay grows as arrivals speed up. A shared line includes other tenants. Their load becomes your tail.

Evidence in your own data: run the probe at two different hours. Keep the prompt and concurrency identical. Watch p95 move while nothing on your side changes. That is your neighbor fingerprint.

Corrected mental model: think of a shared buffer, not a private pipe. Your result is a distribution. The shape moves with outside demand. Label every sample with its time window. A snapshot is not a baseline.

Myth 2: "More concurrency means more throughput"

Parallel calls can increase speed. On a shared server, that stops somewhere. Extra requests add queue time after the knee. The probe table shows it. p50 rises, p95 rises, maximum rises. Throughput stops climbing. You are borrowing from your neighbors.

Corrected mental model: concurrency is a loan, not a budget. Test in small steps. Find the level where tail latency bends upward. Back down one level. Use that as your hard cap.

Myth 3: "A healthy endpoint stays healthy for my whole job"

Endpoint health has a half-life. Free tiers share capacity with many tenants. A quiet block can turn crowded in minutes. Your earlier success is not current load.

You will likely meet this pattern. Probe at 9am, and the tail looks flat. At 3pm, another tenant starts a batch job. p50 stays similar, p95 triples. This is an illustration, not a promise. The lesson still holds.

Corrected mental model: refresh measurements before important jobs. Do not reuse a week-old health label. A short smoke test beats a stale baseline. Measurement is rent, not property.

Where a free server honestly helps

This is where MonkeyCode's free model access and free server option fit. Free model access gives you an endpoint for the probe. The free server can host scheduled runs. You check contention without paying a VM bill. Keep the schedule light and polite.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Who should not use this method

The probe is a diagnostic tool, not a benchmark. Several people should skip it.

Teams with contract SLAs and audit requirements.
Production traffic measurement for customer-facing SLOs.
Users probing an endpoint without the owner's permission.
Engineers who need one clean number for a slide.

The last group should use a load-testing tool behind a private queue. Shared-fate probes are for learning. They are not for revenue tables.

Limitations

The probe measures one variable: latency spread. It cannot separate scheduling, model routing, or network paths. It never shows all tenants. It shows your luck at that exact moment. Treat outputs as relative signals. Run the probe across several days before drawing conclusions. Also note: p50 is a summary, not a promise. The tail carries the neighbor story.

The corrected mental model

Latency is shared state.
Concurrency is a loan.
Health is a timed sample.
Retest before you trust an old number.

A free model server is never empty. Every millisecond you spend is borrowed. Stop asking "is the server slow?" Ask "what did my neighbors just do?" Then measure your answer.

Your AI Reviewer Is a Black Box: A Response Audit FAQ

Jordan Huang — Sun, 30 Aug 2026 00:28:59 +0000

Every developer just got promoted. AI writes code. We review the diff. But nobody reviews the AI server's responses.

I call this the reviewer blind spot. I see it in CI all the time. A script calls a free model endpoint. It receives some JSON.

And we call that an automated code review. So I built a response audit. This FAQ breaks down five myths about server-side model output.

Prerequisites

It's a simple Linux toolchain. Make sure these are installed.

curl for HTTP requests
jq for JSON parsing
git for diff generation

My examples work in any CI runner. You can adapt them to GitHub Actions or GitLab CI.

Myth 1: A `200 OK` Means the Review Worked

The claim: "I got a 200, so the model saw my diff."

The evidence: HTTP status codes describe transport. They don't describe logical correctness. A 200 can mean the gateway accepted your request and the model crashed or returned a blank string.

Check the body. My script flags empty responses immediately.

RESPONSE=$(curl -s -w "\n%{http_code}" "$ENDPOINT" -H "Content-Type: application/json" -d @payload.json)
HTTP_CODE=$(tail -n1 <<< "$RESPONSE")
BODY=$(sed '$d' <<< "$RESPONSE")

if [ "$HTTP_CODE" != "200" ]; then
  echo "Transport failed: $HTTP_CODE"
  exit 1
fi

The corrected mental model: Status codes are handshakes, not verdicts. You must parse the payload. The payload is the contract.

Myth 2: "The Output Looks Good" Is Enough

The claim: "I read the model's comment. It approved my changes."

The evidence: Reading is the new rubber stamp. If you don't extract a concrete verdict, your pipeline is just a fancy chat window.

I enforce a JSON schema. Here is the rule for my review bot.

VERDICT=$(echo "$BODY" | jq -r '.choices[0].message.content')
echo "$VERDICT" | jq -e 'has("rating") and has("summary")' > /dev/null
if [ $? -ne 0 ]; then
  echo "Malformed AI feedback"
  exit 1
fi

This catches partial responses. Free servers truncate unexpectedly. This catches a response that is just "LGTM" with no structure.

The corrected mental model: If you can't jq an output, you can't automate a review. Extract the verdict or ignore the comment.

Myth 3: LLMs Are Great at Finding Bugs in Dependencies

The claim: "The model knows every CVE. It will check my versions."

The evidence: This is close to a hallucination. LLMs are a reading comprehension engine. They don't execute your lockfile.

I let the model handle the nuanced diff. Static tools handle the deterministic checks.

npm audit --omit=dev > /dev/null
if [ $? -ne 0 ]; then
  echo "Dependency audit failed. Skipping LLM review."
  exit 1
fi

The corrected mental model: Run tools first. Send the model the remaining logic. Never ask an LLM to do math it cannot compute.

Myth 4: A Free Server Can Absorb My CI's Concurrent Bursts

The claim: "I'll fire fifty diffs at the endpoint simultaneously."

The evidence: Network sockets aren't infinite. Free server endpoints handle limited connections. Bursts trigger timeouts instantly.

I run my own CI queue. The server remains predictable.

seq 1 5 | xargs -P 2 -I {} curl -s "$ENDPOINT" -d @payload.json -o /dev/null -w "%{http_code}\n"

The corrected mental model: Free doesn't mean unlimited. Use xargs or a simple retry calculation. Keep your concurrency below the visible limit.

Myth 5: My CI Is Slower Because of the AI Reviewer

The claim: "The network call adds too much latency to my pipeline."

The evidence: You are sending too much context. Free tier models don't need the full repo. They need the relevant diff chunk.

I measure the diff size first. I skip the model entirely on tiny changes.

LINES=$(git diff --unified=5 HEAD~1 | wc -l)
if [ "$LINES" -lt 50 ]; then
  echo "Tiny change. Skipping LLM review."
  exit 0
fi

The corrected mental model: The AI reviewer is a scalpel. It is not a chainsaw. Optimize payload size or drop the tool entirely.

A Simple Decision Matrix

I now run my prompt through this decision table.

Change Type	Static Tools	LLM Review	Rationale
Package manager lockfile	Yes	No	CVEs are numeric facts
TypeScript types	Yes	No	Compiler catches it
Business logic	No	Yes	Context-heavy nuance
CSS styling	No	Yes	Taste and consistency
Security boundary	Both	Both	Defense in depth

This table saves me tokens and time. It also saves my free server from pointless requests.

Why This Workflow Is a Real Audit

I run this probe against MonkeyCode's free model access and free server option. The setup exposes a vital network bottleneck.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Testing against a real free server is the only way to learn its behavior. Local simulations lie.

Limitations

This audit is not a security boundary. It does not catch every prompt injection. It doesn't validate model safety.

You shouldn't use this for legal or medical decisions. You shouldn't use this on mission-critical infrastructure without human review. My script is a starting point, not a final gate.

Closing

Your AI reviewer is a black box. Mine isn't. I parse the verdict and check the transport.

Have you audited your AI review pipeline? What does your response checker look like?

Free Model Tiers: Five Myths Your Team Repeats (and a Scorecard)

Jordan Huang — Sat, 29 Aug 2026 21:25:08 +0000

Everybody says free model tiers are dangerous. They talk about rate limits. They blame latency. I think the real danger is your mental model.

Let me bust five myths. Each one has a test. You can run that test yourself.

I used MonkeyCode's free model access and the free server option to host this probe. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The Setup

A probe is a small script. It sends realistic prompts to an endpoint. It records latency and parse errors. Then it decides: free or paid.

Here is the core command. It measures one request with curl:

for i in $(seq 1 5); do
  curl -s -o /dev/null \
    -w "latency %{time_total}s\n" \
    -H "Authorization: Bearer $FREE_KEY" \
    "$FREE_URL"
done

Latency alone is not enough. You also need output validity. This scorecard adds that.

Myth #1: Free Tiers Can't Handle Production

The claim: Production demands an SLA. Free tiers never have one.

Why it spreads: We confuse “no SLA” with “will crash.” Shared queues look chaotic.

Corrected model: Production means defined failure behavior. You can define a fallback path. Use free for tolerant paths. Use paid for critical paths.

Test: Break the free endpoint in your script. Measure how fast your fallback kicks in. That recovery time is your real SLA.

Myth #2: Free Means Bad Model Quality

The claim: Free models output garbage. Paid models output gold.

Why it spreads: A few bad responses poison your memory. You stop measuring.

Corrected model: Quality is a function of task shape. A free model can classify text well. A paid model may still fail on ambiguous prompts.

Test: Send the same five prompts to both endpoints. Compare JSON parse rates. You will probably find the gap is smaller than you think.

Myth #3: A Free Server Can't Run Real Jobs

The claim: Free servers are for demos. They restart, throttle, and vanish.

Why it spreads: You once lost a container and blamed the whole category.

Corrected model: Idempotent jobs survive restarts. Write tasks that can repeat safely. Then a free server becomes a batch worker.

Test: Schedule a cron job that writes a heartbeat file every minute. Add an idempotent counter. Let it run for a week. Count interruptions.

Myth #4: Free Tier Costs Nothing

The claim: $0 invoice means $0 cost.

Why it spreads: We ignore engineering hours. We celebrate saved dollars.

Corrected model: Free shifts cost from money to time. You debug rate limits. You tune prompts. You add abstraction.

Test: Track every hour you spend working around a free-tier quirk. Multiply by your hourly rate. That is the real price.

Myth #5: You Should Wait Until You're Big Enough

The claim: Free tiers lock you into bad habits.

Why it spreads: Lock-in is scary. People treat price as the only protection.

Corrected model: Lock-in is managed by abstraction. Write a thin client layer. Swap endpoints when you need to.

Test: Put your API calls behind one function. Change the base URL. If you edit more than three lines, your architecture owns you.

The Scorecard

This script replaces opinion with a decision. Save it as freetier_score.py.

#!/usr/bin/env python3
"""freetier_score.py - decide if your workload fits a free model tier."""
import os, time, json, requests

PROMPTS = [
    "Classify sentiment as JSON: {'positive': true}",
    "Extract name and date: 'Sam joined on 2026-08-30'",
    "Summarize this sentence in one line: 'Free tiers need fallbacks'.",
]

def invoke(url, key, prompt):
    start = time.time()
    r = requests.post(
        url,
        headers={"Authorization": f"Bearer {key}"},
        json={"messages": [{"role": "user", "content": prompt}]},
        timeout=30,
    )
    latency = time.time() - start
    try:
        content = r.json()["choices"][0]["message"]["content"]
        json.loads(content)  # raises if invalid
        valid = True
    except Exception:
        valid = False
        content = r.text[:80]
    return {"latency": latency, "valid": valid, "chars": len(content)}

def main():
    url = os.environ["ENDPOINT_URL"]
    key = os.environ["ENDPOINT_KEY"]
    results = [invoke(url, key, p) for p in PROMPTS]
    valid_ratio = sum(r["valid"] for r in results) / len(results)
    avg_latency = sum(r["latency"] for r in results) / len(results)

    print(f"valid_ratio: {valid_ratio:.2f}")
    print(f"avg_latency: {avg_latency:.2f}s")

    if valid_ratio >= 0.9 and avg_latency <= float(os.getenv("LATENCY_BUDGET", "5.0")):
        print("USE FREE")
    else:
        print("PAY")

if __name__ == "__main__":
    main()

Run it with:

ENDPOINT_URL="https://your-endpoint" \
ENDPOINT_KEY="your-key" \
LATENCY_BUDGET="5.0" \
python3 freetier_score.py

The Decision Table

Parse-error rate	Average latency	Verdict
< 10%	< 5s	Use free
> 10%	< 5s	Fix prompts or pay
< 10%	> 5s	Batch jobs or pay
> 10%	> 5s	Pay or redesign

Tune the numbers to your business. There is no universal threshold.

Limitations

This is a smoke test, not a benchmark. It uses three prompts. Your workload may look completely different.

Results vary by provider, time, and model. A free tier can shine at 2 a.m. and choke at 2 p.m.

Do not use this scorecard to pick a vendor. Use it to challenge assumptions about cost and reliability.

Who Should Not Use This

Skip this approach if you need regulatory latency guarantees. Skip it if you can never retry a failed request. Skip it if you have no fallback plan.

Free infrastructure rewards flexible designers. It punishes teams that want one fixed path.

Takeaway

Free model tiers are not a trap. Your unmeasured assumptions are the trap.

Run the probe. See what breaks. Then decide with data.

Your mental model will thank you.

No Response Is a Response: Five Silent-Drop Myths on Free Model Servers

Jordan Huang — Sat, 29 Aug 2026 18:22:22 +0000

Some failures never raise an error. Your client waits. The deadline passes. You blame the model server. But what if the request never arrived? What if it arrived and then vanished? That is a silent drop. Retry counters will not catch it. Logs usually won't either.

The failure no status code describes

An HTTP error is honest. 429 says back off. 503 says try later. 500 says something broke. A silent drop says nothing. You get no status code. You get no body. You get a timeout that you invented yourself. That distinction matters.

A timeout is a client-side decision. A status code is a server-side statement. Confusing them is where the myths start.

What I mean by silent drop

A silent drop is a request that disappears between your process and the model. The server may never see it. The server may see it and then lose it. Either way, your application sees nothing. No answer. No code. No trace.

Free-tier endpoints hide these drops well. The symptom looks exactly like slowness. That's why retry loops and timeout tweaks don't fix it. You are measuring the wrong layer.

To measure this properly, you need an endpoint that accepts arbitrary JSON. I use MonkeyCode's free model access and its free server option as a test target. That gives me a real model endpoint and a server I can restart. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The probe

Stop guessing. Build a probe. The probe sends requests with a unique request ID. Then it compares client results against server logs. The gap between those two is your drop rate.

You can test locally first. No cloud account required. Two files. Five minutes.

Part 1 - A drop server

This small server simulates four failure modes. echo is the healthy control. hold/N delays the answer past the client deadline. cut accepts the request, then closes the connection without answering. phantom drops the request before the app ever records it.

Run it in terminal one:

# drop_server.py
import asyncio, json

SEEN = []

def http_response(body: bytes, status: str = "200 OK") -> bytes:
    return (
        f"HTTP/1.1 {status}\r\n"
        f"Content-Length: {len(body)}\r\n"
        f"Connection: close\r\n\r\n"
    ).encode() + body

async def handle(reader, writer):
    try:
        header_bytes = await asyncio.wait_for(
            reader.readuntil(b"\r\n\r\n"), timeout=3
        )
        headers = header_bytes.decode(errors="replace").split("\r\n")
        request_line = headers[0] if headers else ""
        parts = request_line.split(" ")
        path = parts[1] if len(parts) > 1 else "/"

        request_id = None
        for line in headers:
            if line.lower().startswith("x-request-id:"):
                request_id = line.split(":", 1)[1].strip()

        # Simulate drops that the app never sees.
        if path.startswith("/phantom"):
            await asyncio.sleep(0.2)
            writer.close()
            return

        if request_id:
            SEEN.append({"request_id": request_id, "path": path})

        if path.startswith("/hold/"):
            seconds = float(path.split("/")[-1])
            await asyncio.sleep(seconds)
            data = json.dumps({"ok": True}).encode()
            writer.write(http_response(data))
            await writer.drain()
        elif path == "/cut":
            # Claim receipt, then close without replying.
            await asyncio.sleep(0.1)
            writer.close()
            return
        elif path == "/echo":
            data = json.dumps({"ok": True}).encode()
            writer.write(http_response(data))
            await writer.drain()
        elif path == "/logs":
            data = json.dumps(SEEN).encode()
            writer.write(http_response(data))
            await writer.drain()
        else:
            writer.write(http_response(b"{}", "404 Not Found"))
            await writer.drain()
    except Exception:
        pass
    finally:
        try:
            writer.close()
        except Exception:
            pass

async def main():
    server = await asyncio.start_server(handle, "127.0.0.1", 8001)
    print("drop server running on http://127.0.0.1:8001")
    async with server:
        await server.serve_forever()

if __name__ == "__main__":
    asyncio.run(main())

Part 2 - The client

The client sends N requests with a request ID. Then it asks the server which IDs arrived. The table shows the quiet ones. Run it in terminal two:

# probe.py
import json, time, uuid
import requests
from concurrent.futures import ThreadPoolExecutor

BASE = "http://127.0.0.1:8001"
PAYLOAD = {"model": "probe-model", "prompt": "ping"}

def send_one(request_id: str, path: str):
    headers = {"X-Request-ID": request_id, "Content-Type": "application/json"}
    started = time.monotonic()
    result = "ok"
    try:
        r = requests.post(f"{BASE}{path}", json=PAYLOAD, headers=headers, timeout=5)
        if r.status_code != 200:
            result = f"http_{r.status_code}"
    except requests.exceptions.Timeout:
        result = "timeout"
    except requests.exceptions.ConnectionError as exc:
        result = "reset" if "reset" in str(exc).lower() else "connection_error"
    elapsed = round(time.monotonic() - started, 2)
    return request_id, result, elapsed

def run_probe(path: str, count: int = 12, workers: int = 4):
    ids = [f"req-{i:03d}" for i in range(count)]
    with ThreadPoolExecutor(max_workers=workers) as pool:
        results = list(pool.map(lambda rid: send_one(rid, path), ids))
    seen = requests.get(f"{BASE}/logs", timeout=3).json()["seen"]
    seen_ids = {item["request_id"] for item in seen}
    print(f"{'request_id':<12} {'client':<14} {'elapsed':<8} {'server_log'}")
    for rid, result, elapsed in results:
        in_log = "yes" if rid in seen_ids else "no"
        print(f"{rid:<12} {result:<14} {elapsed:<8} {in_log}")

if __name__ == "__main__":
    print("--- echo: healthy control ---")
    run_probe("/echo")
    print("--- hold/10: slow beyond client deadline ---")
    run_probe("/hold/10")
    print("--- cut: arrived, then vanished ---")
    run_probe("/cut")
    print("--- phantom: dropped before the app ---")
    run_probe("/phantom")

Sample output shape (your timing will differ):

--- echo: healthy control ---
request_id   client         elapsed   server_log
req-000      ok             0.21      yes
req-001      ok             0.19      yes

--- hold/10: slow beyond client deadline ---
req-002      timeout        5.01      yes

--- cut: arrived, then vanished ---
req-003      reset          0.11      yes

--- phantom: dropped before the app ---
req-004      reset          0.20      no

How to read the results

client saw	server logged	meaning	move
ok	yes	healthy round trip	keep going
timeout	yes	arrived, response lost	raise read timeout or stream
timeout/reset	no	dropped before the app	recreate connection, retry with ID
reset	yes	connection killed mid-flight	check body size and keep-alive
4xx/5xx	maybe	explicit error	handle status codes normally

Do not count all timeouts as drops. A timeout with a server log is a slow response. A timeout without a log is a true silent drop. The distinction changes your fix. Slow means wait longer. Dropped means resend. The request ID makes that choice safe.

Five myths, corrected

Myth 1: A timeout means the server is thinking

Timeouts fire before inference even starts. DNS can stall. TCP can stall. A proxy can close an idle connection. Your timeout is a measurement, not a diagnosis. It cannot see inside the network path. Treat it as a symptom, not proof.

Myth 2: An empty server log means a clean bill of health

Some servers log only completed work. Some gateways drop before the app sees the request. Log sampling hides rare events. Absence of evidence is not evidence of absence. That is why this probe checks the server side explicitly.

Myth 3: Retry after timeout is always safe

Is it safe when the first request arrived? No. You create a duplicate. The server may process both. Request IDs put a pin in that problem. Without an ID, a retry is a guess. With one, it is an idempotent replay.

Myth 4: Drops on free tiers are random noise

Free-tier traffic is not uniform. Drops cluster around idle connections. They cluster around large prompts and long generations. Classify your samples before calling them random. Patterns show up once you look.

Myth 5: No error means no problem

The quiet failures are the expensive ones. One drop is noise. Five percent is a user-visible bug. If your dashboard only watches 4xx and 5xx, it is blind. Add the no-response category to your monitoring.

When this probe is not enough

This probe is a spot check, not a full observability stack. It misses trace-level issues and slow cascading failures. For high throughput, use distributed tracing. For security boundaries, use a proper gateway. And remember: the probe creates its own load. Watch for new 429s. They are data, not bugs. If you cannot add a request ID to the request, this method falls apart.

Run it

Start the drop server. Point the probe at your endpoint. Let it run for one hour. Then read the table with fresh eyes. The empty rows tell more than the error codes. I run this before trusting any new free-tier endpoint. It takes five minutes and saves a day of confusion.

SSE Myths That Make Your Free Model API Feel Slower

Jordan Huang — Sat, 29 Aug 2026 15:18:49 +0000

You blame the free model server. I get it.

Fifty percent of the time, the server is fine. Your client code is eating the tokens.

Let's talk about streaming. Specifically, the myths around Server-Sent Events (SSE).

Myth One: SSE Is Just JSON Over WebSocket

I see this everywhere. People say, "Streaming needs websockets. It's two-way, right?"

No.

SSE is HTTP. The MIME type is text/event-stream. Websockets require a protocol upgrade. SSE works over plain HTTP.

One direction. Server to client. That's it.

For AI chat, that's exactly what you need. You send the prompt up. The model streams the response down.

Using WebSocket here adds overhead. You're managing: ws, onopen, onmessage. Why? HTTP works fine.

Myth Two: `await res.json()` Is Fine For Streaming

This is the biggest one.

When you use fetch, that data is a stream. Calling res.json() buffers the entire response before it resolves.

You are not streaming. You are waiting for the file to end.

The fix? Read the body stream chunk by chunk.

const resp = await fetch("/v1/chat/completions", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify(payload),
});

if (!resp.ok) throw new Error(`HTTP ${resp.status}`);

const reader = resp.body.getReader();
const decoder = new TextDecoder();
let buffer = "";

while (true) {
  const { value, done } = await reader.read();
  if (done) break;

  buffer += decoder.decode(value, { stream: true });
  const events = buffer.split("\n\n");

  // Last chunk might be incomplete. Keep it in the buffer.
  buffer = events.pop();

  for (const evt of events) {
    if (!evt.startsWith("data:")) continue;
    const raw = evt.slice(5).trim();
    if (raw === "[DONE]") continue;

    const json = JSON.parse(raw);
    const token = json.choices?.[0]?.delta?.content || "";
    process.stdout.write(token);
  }
}

Do you see the difference?

The first token appears on your screen. It doesn't wait for the last token.

Myth Three: Nginx Reverse Proxy and Edge Caching Buffers Are Some Mystery

Your proxy is probably buffering everything to disk before sending it.

Nginx has a feature called X-Accel-Buffering. FastAPI, Node, or your reverse proxy trusts it. If your proxy buffers chunks, your fellow devs.

Your low TTFB means nothing if the first byte you see hits a 500 ms boundary.

Try adding this header before streaming:

res.setHeader("X-Accel-Buffering", "no");
res.setHeader("Cache-Control", "no-cache");
res.setHeader("Connection", "keep-alive");

If you use FastAPI, you also need StreamingResponse:

from fastapi.responses import StreamingResponse

def stream_result(prompt: str):
    for token in call_free_model(prompt):
        yield token

@app.post("/v1/chat/completions")
def chat_route(payload: dict):
    return StreamingResponse(
        stream_result(payload["messages"]),
        media_type="text/event-stream",
        headers={"X-Accel-Buffering": "no"},
    )

If you skip the header, your model feels slow. It was never slow. Your infrastructure just said "hold on, I'll hold your beer, let me buffer first."

Myth Four: `curl` Shows The Real Behavior

I have an old habit. I run this:

curl -N https://api.example.com/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"free","stream":true}'

See the -N flag? That disables buffering.

A lot of people omit it. Then curl buffers. Then they tell me the API is slow.

Again: it's critical to separate the server's behavior from the client's behavior.

Here's a small executable script. It measures the time to the first token versus the time to the full response.

import time
import httpx
import statistics

def time_to_first_token(url, payload, runs=5):
    first = []
    total = []

    for _ in range(runs):
        start = time.monotonic()
        with httpx.stream("POST", url, json=payload, timeout=60) as resp:
            seen_first = False
            for line in resp.iter_lines():
                if line.startswith("data:") and not seen_first:
                    first.append(time.monotonic() - start)
                    seen_first = True
        total.append(time.monotonic() - start)

    return statistics.median(first), statistics.median(total)

# Example
url = "http://localhost:8000/v1/chat/completions"
payload = {"model": "my-free-model", "stream": True, "messages": [{"role": "user", "content": "hi"}]}

first_ms, total_ms = time_to_first_token(url, payload)
print(f"First token (median): {first_ms * 1000:.1f} ms")
print(f"Full output (median): {total_ms * 1000:.1f} ms")

If first_ms is in the 80-120 ms range but total_ms is 8 seconds, your server is actually streaming. The user perceives it as slow because the model spends 7 seconds generating tokens.

The MonkeyCode Proxy Check

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

When I run this against a free model endpoint, it helps to have the proxy in the loop. MonkeyCode's free server option lets you spin up a small proxy. That proxy can hide the model endpoint by design.

But if you set that proxy up, respect the streaming path. Don't let your proxy controller collect the whole payload and then forward it. That's a very common failure: your cloud function is invisible, but it kills the streaming.

I keep the proxy file simple:

service: proxy
build:
  command: python -m app

Again, don't add an intermediate layer that buffers. Your free model API becomes one giant lie. The endpoint responds in 200 ms, but your gateway shows 9 seconds.

Who Should Ignore This Advice

Not everyone needs to chase first-token latency.

If you're writing a CLI tool that processes a batch job, call res.json() honestly. It's easier to read. It's simpler to maintain.

If you're building a UI with a progress bar or chat bubble, you need streaming. You need first-token observability.

If you're building a debugging test suite, write the script I shared above. Keep it in your CI. Protect your proxy from buffer regressions.

The Correct Mental Model

Every SSE increment is a TCP packet.

Every buffer is a lie waiting to happen.

Your free model is not your car. It's a broker. If you send a response in the wrong way, the broker is the traffic jam.

Use -N in curl. Use getReader() in the browser. Use iter_lines() in Python. Force X-Accel-Buffering: no on your gateway.

Then you can complain about the model honestly—because sometimes it is genuinely slow.

But at least you'll have the numbers to prove it.

Stop Timing the First Token: A Streaming FAQ for Free Model Servers

Jordan Huang — Sat, 29 Aug 2026 12:13:55 +0000

Your chat UI feels instant. Your batch job crawls. Same model. Same server.

Streaming is usually the difference. Streaming changes what your client sees first. It does not change how fast the model thinks.

Most teams time one number: the first token. That number explains maybe a third of the total wait. The rest hides in the gaps between tokens.

This is a myth-busting FAQ. Claims first. Evidence second. Corrected mental model third. Run the probe at the end and check me.

Myth 1: Streaming makes the request faster

The claim: "Set stream=True and the whole call speeds up."

The evidence: Total time stays roughly flat. Streaming moves the first byte earlier. The generation work is identical. Clients feel speed. Stopwatches do not.

The corrected model: Streaming is a perception tool. It is not a throughput tool. Use it where a human waits. Drop it where a pipeline measures.

Myth 2: First-token latency is the only metric that matters

The claim: "My SLA is p95 TTFT. Everything else is noise."

The evidence: Total response time has two parts. First comes the first token. Then comes the stream after it. A 20-token answer? TTFT owns the budget. A 500-token answer? The gaps own it. Tracking one number hides the second half.

The corrected model: Track three numbers. TTFT, median inter-token gap, total time. Which one you optimize depends on where the user waits.

Myth 3: More chunks means a faster-feeling response

The claim: "We flush one chunk per token. The UI feels alive."

The evidence: Every chunk costs a network trip, a JSON parse, and a UI update. Tiny chunks serialize badly under load. Many free servers batch tokens anyway. Chunk count and throughput are different numbers.

The corrected model: Measure tokens per second. Not chunks per second. Sometimes buffering two or three tokens feels smoother than spraying them one by one.

Myth 4: An open stream can't time out

The claim: "I skip timeouts. The stream keeps the connection alive."

The evidence: Stream timeouts are idle-based. A server that emits a token every 25 seconds kills a 15-second idle timeout. Worse: a long stream holds a connection-pool slot. Other requests queue behind your free stream. A quiet jam.

The corrected model: Time out on the gap. Not the connection. If the provider sends keep-alive pings, measure them. Separate connection idle from generation active.

Myth 5: Aborting a stream saves tokens

The claim: "The user clicks stop. We close the socket. Money saved."

The evidence: Closing your socket does not stop the server's loop. Some providers finish generating and bill the orphaned tail. Others check abort signals between chunks, not inside them. Your client stopped. The server did not.

The corrected model: Treat abort as UX. Not cost control. Cap generation server-side with max_tokens. Then compare what the API reported as usage. Assume nothing.

The probe: measure your own endpoint

This script runs the same prompt in both modes. It prints median TTFT, median total time, and median token gap. You need Python 3 and the requests package. Roughly ten minutes per endpoint.

import statistics
import time

import requests

ENDPOINT = "https://your-endpoint/v1/chat/completions"
MODEL = "your-model"
HEADERS = {"Authorization": "Bearer $OPENAI_API_KEY"}
PROMPT = "Explain, in 150 words, why free-tier LLM latency is noisy."
TRIALS = 5


def measure(stream: bool) -> dict:
    payload = {
        "model": MODEL,
        "messages": [{"role": "user", "content": PROMPT}],
        "max_tokens": 300,
        "stream": stream,
    }
    started = time.perf_counter()
    first = None
    gaps = []
    last = started

    resp = requests.post(ENDPOINT, json=payload, headers=HEADERS, stream=True)

    if not stream:
        # Time until the first byte, then read the body.
        for _ in resp.iter_content(chunk_size=1):
            first = time.perf_counter() - started
            break
        resp.json()
        total = time.perf_counter() - started
        return {"ttft": first, "total": total, "gap_median": None}

    for line in resp.iter_lines():
        if not line or not line.startswith(b"data:"):
            continue
        if line[5:].strip() == b"[DONE]":
            break
        now = time.perf_counter()
        if first is None:
            first = now - started
        else:
            gaps.append(now - last)
        last = now

    total = time.perf_counter() - started
    return {
        "ttft": first,
        "total": total,
        "gap_median": statistics.median(gaps) if gaps else None,
    }


for stream in (False, True):
    rows = [measure(stream) for _ in range(TRIALS)]
    label = "stream" if stream else "plain"
    print(f"--- {label} ({TRIALS} trials) ---")
    print(f"median TTFT : {statistics.median(r['ttft'] for r in rows):.2f}s")
    print(f"median total: {statistics.median(r['total'] for r in rows):.2f}s")
    gaps = [r["gap_median"] for r in rows if r["gap_median"]]
    if gaps:
        print(f"median gap  : {statistics.median(gaps):.2f}s")

If a provider batches several tokens per chunk, gap_median measures chunks, not tokens. That is still the number your UI sees. So it is the honest metric.

An output shape looks like this. Illustrative, not a benchmark. Run it on your endpoint first.

metric	plain	stream
median TTFT	0.9s	0.4s
median total	3.8s	3.9s
median token gap	—	0.11s

Notice the pattern. First token gets faster. Total time stays flat. That is the whole argument in one table.

When streaming wins (and when it does not)

Your user	TTFT matters?	Total matters?	Verdict
Interactive chat	Yes	No	Stream
Batch summarizer	No	Yes	Plain
Unit test / contract check	Yes	Yes	Stream, then assert total
Proxy fanning out many calls	No	Yes	Plain or buffer chunks

My rule of thumb: keep pipelines plain. Stream only the UI-facing path.

Where I run this probe

I point the probe at MonkeyCode's free model access through the free server option first. Free tiers deserve measurement before trust. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The script is provider-blind. Any OpenAI-compatible endpoint works.

Who should not use this probe

You are quota-starved. Ten calls can eat a day of free-tier budget. Set TRIALS = 2.
Your provider has no SSE support. The stream path will hang. Test one call first.
You need server-side metrics. This is a client-side ruler. Use the provider dashboard for the other half.

Limitations

Five trials is a sample, not a census. Free servers are noisy by design. Raise TRIALS to twenty when your budget allows. Results do not transfer between regions, models, or hours. Rerun on your own endpoint. Never cite the table above as a benchmark. It is a shape, not proof.

Next time a new free model drops, run this probe before you wire it in. The gap between the first token and the last one will surprise you.

Your users wait for the last token. So should your dashboards.

Contract Tests for Model Output: The JSON Safety Net You're Missing

Jordan Huang — Sat, 29 Aug 2026 09:09:51 +0000

One day the model stops returning the shape you expected. You did not change your code. You did not change your prompt. But your parser starts throwing KeyError in production.

You could patch the parser again. Most teams do. The better move is to write a contract test for the one API that changes without a version bump: the model itself.

You already write contract tests for your REST endpoints. The model deserves the same treatment, especially when you run it on a free tier alongside your regular stack.

The Silent Contract Break

Large language models return text. Your application expects JSON. Between those two facts sits a fragile contract with three unnamed clauses: keys must match, types must match, and the payload must survive transport.

Each clause breaks in its own way. A model starts returning amount as a string. It adds a new field you never asked for. It wraps the JSON in markdown fences because a previous prompt told it to be chatty.

None of these break the API call. All of them break your downstream code.

A Contract Test for Text

A contract test for model output is a small script that does three things:

Send a fixed prompt to the model.
Parse the raw text response.
Validate the result against a schema.

Run it in CI. Run it on a schedule. Treat a failure the way you treat a broken API contract: loudly.

You can run this against any endpoint you can reach. I use MonkeyCode's free model access as a cheap target for exactly this kind of suite. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The same test works against any OpenAI-compatible endpoint.

The Schema First

Start with the smallest schema your application actually needs. Not the ideal output. The contract.

For a receipt extractor, that might look like this:

{
  "type": "object",
  "required": ["vendor", "amount", "date", "items"],
  "properties": {
    "vendor": {"type": "string"},
    "amount": {"type": "number"},
    "date": {"type": "string", "pattern": "^\\d{4}-\\d{2}-\\d{2}$"},
    "items": {"type": "array", "items": {"type": "string"}}
  },
  "additionalProperties": false
}

Notice additionalProperties: false. That is the clause most people forget. It catches the day the model starts inventing fields.

The Three-Layer Test

Here is the reusable pytest scaffold. It keeps three concerns separate: calling the model, cleaning the output, and validating the shape.

import json
import os
import urllib.request

import jsonschema
import pytest

SCHEMA = {
    'type': 'object',
    'required': ['vendor', 'amount', 'date', 'items'],
    'properties': {
        'vendor': {'type': 'string'},
        'amount': {'type': 'number'},
        'date': {'type': 'string', 'pattern': '^\\d{4}-\\d{2}-\\d{2}$'},
        'items': {'type': 'array', 'items': {'type': 'string'}},
    },
    'additionalProperties': False,
}


def call_model(prompt):
    body = json.dumps({
        'messages': [{'role': 'user', 'content': prompt}],
        'temperature': 0,
    }).encode()
    req = urllib.request.Request(
        os.environ['MODEL_URL'],
        data=body,
        headers={
            'Authorization': f"Bearer {os.environ['MODEL_TOKEN']}",
            'Content-Type': 'application/json',
        },
    )
    with urllib.request.urlopen(req, timeout=60) as resp:
        data = json.loads(resp.read())
    return data['choices'][0]['message']['content']


def parse_model_output(raw):
    raw = raw.strip()
    if raw.startswith('```

'):
        raw = raw.strip('`')
        if raw.startswith('json'):
            raw = raw[4:]
    return json.loads(raw)


@pytest.mark.parametrize('fixture', ['receipt-1.txt', 'receipt-2.txt', 'receipt-3.txt'])
def test_model_output_matches_contract(fixture):
    with open(fixture) as f:
        prompt = f.read()
    raw = call_model(prompt)
    parsed = parse_model_output(raw)
    jsonschema.validate(parsed, SCHEMA)
    assert isinstance(parsed['amount'], (int, float))
```

The three fixtures are your baseline receipts. They should be boring, realistic examples, not edge cases. Boring fixtures catch drift; edge cases test cleverness.

## The Cleaning Function Is a Boundary

Note the `parse_model_output` function. It exists because models wrap JSON in prose, markdown fences, or both.

Do not let the cleaning function grow forever. If you find yourself adding a fourth rule, stop and change the prompt instead. The cleaning function should handle known noise, not absorb unknown model behavior.

## Running It on a Schedule

A contract test run once in CI is useful. A contract test run every hour is a different tool.

Put the suite on a timer. I run it on MonkeyCode's free server option when I want a long-lived observer without babysitting my laptop. The free tier of the model access keeps the cost near zero, and the server handles the scheduling.

The goal is a history of responses. When the model changes shape, the failure is a data point, not a mystery.

## When the Contract Fails

Here is the decision table I use when a model output fails validation:

| Failure | Likely cause | Move |
| --- | --- | --- |
| `additionalProperties` error | Model added a new key | Add the key to the schema or tighten the prompt; never let the parser ignore it silently |
| `amount` is a string | Type drift | Fix the prompt first, add a conversion layer only as a temporary guard |
| Date pattern mismatch | Format drift | Constrain the prompt with an example and re-run the suite |
| Markdown fences appear | Prompt leakage or model habit | Handle in cleaning, then log the frequency so it does not become normal |

## What This Test Will Not Catch

The schema validates shape. It does not validate meaning or safety.

The model can return `{'amount': 12.5, 'vendor': 'Not A Vendor', 'date': '2026-08-29', 'items': []}` and pass every assertion. Your business logic still needs its own checks.

This test also cannot detect prompt-injection attempts in the fixtures themselves. Treat this as a stability net, not a security boundary.

## Who Should Skip This

If your model output is a single sentence displayed to a user, skip the schema. A string is already a contract.

If you only call a model once in a script, skip the full suite. A one-off `json.loads` inside a try block is enough.

But if you have more than one parser, more than one prompt template, or a cron job that turns model output into database rows, write the contract test first.

The cost of this suite is tiny. The cost of a silent shape change is a page at 2 a.m.

## The First Step

Pick one model output you depend on. Write one schema, three fixtures, and a cleaning function. That is the whole project.

The model will change without telling you. At least your tests will be paying attention.

DEV Community: Jordan Huang

Five Cold-Start Myths About Free Model Servers (and a Wake-Up Test)

Myth 1: The server is always running

Myth 2: Cold starts are tiny

Myth 3: One warm-up request fixes it

Myth 4: Stable latency means a stable server

Myth 5: Free means production-ready

The Wake-Up Test

Reading the Results

Limitations

Who Should Skip Free Servers

The Honest Take

Your Model Calls Got Slower. Run git bisect Before You Blame the Server

Three myths to drop

1. "Slow calls mean slow infrastructure."

2. "The dashboard will name the commit."

3. "git bisect only finds compile failures."

The reproducibility script

The three-step bisect

Reading the two numbers

What this script will not catch

Who should not use this

The correction

Your Free Model Quota Is a Bank Account. Build a Budget Proxy on a Free Server

Why a Budget Proxy?

Step 1: Understand the usage Field

Step 2: Build the Proxy

Step 3: Add a Reset Policy

Step 4: Test Locally

Step 5: Deploy to a Free Server

Step 6: Point Your Code at the Proxy

What This Proxy Does NOT Do

When Should You Not Use This?

Final Thoughts

Who Owns the Deadline? Four Timeout Myths That Break Free Model Calls

The Myths, In One List

Myth 1: A Client Timeout Is a Deadline

Myth 2: Longer Timeouts Are Safer

Myth 3: Only the Model Call Needs a Timeout

Myth 4: A Connect Timeout Is Enough

The Mental Model That Fixes All Four

A Probe You Can Run

Decision Table: When the Budget Runs Out

Who Should Skip This Approach

Streaming Responses Have a Silent Gap: A Three-Myth FAQ

Myth 1: Streaming always makes the response faster

Myth 2: The end of the stream means the answer is complete

Myth 3: A quiet stream is a slow one, so wait longer

The probe

Reading the output

Decision table

Limitations

Bottom line

The Neighbor Effect: A Measured FAQ About Free Model Servers

The probe

How to read the output

Myth 1: "My latency is my own"

Myth 2: "More concurrency means more throughput"

Myth 3: "A healthy endpoint stays healthy for my whole job"

Where a free server honestly helps

Who should not use this method

Limitations

The corrected mental model

Your AI Reviewer Is a Black Box: A Response Audit FAQ

Prerequisites

Myth 1: A 200 OK Means the Review Worked

Myth 2: "The Output Looks Good" Is Enough

Myth 3: LLMs Are Great at Finding Bugs in Dependencies

Myth 4: A Free Server Can Absorb My CI's Concurrent Bursts

Myth 5: My CI Is Slower Because of the AI Reviewer

A Simple Decision Matrix

Why This Workflow Is a Real Audit

Limitations

Closing

Free Model Tiers: Five Myths Your Team Repeats (and a Scorecard)

The Setup

Myth #1: Free Tiers Can't Handle Production

Myth #2: Free Means Bad Model Quality

Myth #3: A Free Server Can't Run Real Jobs

Myth #4: Free Tier Costs Nothing

Step 1: Understand the `usage` Field

Myth 1: A `200 OK` Means the Review Worked

Myth Two: `await res.json()` Is Fine For Streaming

Myth Four: `curl` Shows The Real Behavior