DEV Community: Riley Lin

Trust Boundaries in AI Coding's Free Servers

Riley Lin — Sun, 30 Aug 2026 11:20:57 +0000

Every AI coding tool that offers a free server is quietly moving your trust boundary to a machine you don't control. I've spent the last few weeks mapping where my data actually travels in these workflows, and the most overlooked point isn't the prompt text. It's the sandbox where the model's suggestions get executed.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source assistant that provides free model access along with a free server for lightweight demos. The server option means you can deploy a small web app directly from the tool, which is convenient for testing ideas quickly.

But convenient is not the same as contained. Think about three zones that your code crosses. Your local machine holds source files, environment variables, and maybe a .env you forgot to ignore. The model API receives the snippets you paste or accept, and those snippets may be logged for training or debugging. The free server receives your whole application, including any secrets baked into the code or configuration.

Each zone has a different owner, a different retention policy, and a different attacker profile. The typical failure I see in code reviews is the same one I found in my own experiments. A developer copies a working directory into the cloud sandbox without checking what's inside.

A single test fixture with a real-looking key, a debug log that prints request bodies, or a dependency that calls home with a token will turn your harmless demo into a data leak. The model isn't the villain; the boundary is just wider than you think.

Let me show you a practical gate. Before I let MonkeyCode's free server run any code, I execute a small shell script that scans for likely secrets and red flags. Here's the version I use:

#!/usr/bin/env bash
# leakguard.sh - scan a directory before sending it to a remote sandbox
TARGET="${1:-.}"
THRESHOLD="${2:-0}"
echo "Scanning $TARGET for sensitive patterns..."
issues=0

# Known key shapes: AWS, GitHub, Slack, Stripe, generic PEM
grep -rE --include='*.{env,yml,yaml,json,js,ts,py,go,rb}' \
  -E '(AKIA[0-9A-Z]{16}|gh[pousr]_[0-9a-zA-Z]{36}|xox[baprs]-[0-9a-zA-Z-]{20,}|sk_live_[0-9a-zA-Z]{24,}|-----BEGIN [A-Z ]*PRIVATE KEY-----)' \
  "$TARGET" 2>/dev/null && ((issues+=1))

# Common secret-bearing filenames
for f in .env .env.production id_rsa deploy_key.pem; do
  if [ -f "$TARGET/$f" ]; then echo "Found sensitive file: $f"; ((issues+=1)); fi
done

# High-entropy strings and assignment patterns (heuristic)
git -C "$TARGET" grep -n -E '(password|secret|token)\s*[:=]' -- ':!*.md' 2>/dev/null && ((issues+=1))

echo "----"
if [ "$issues" -gt "$THRESHOLD" ]; then
  echo "Aborting: $issues suspicious item(s) found."
  exit 1
else
  echo "Clean enough (issues: $issues). Proceed."
fi

This script isn't a silver bullet; it's a tripwire. It catches obvious key formats and tells you about files that almost certainly shouldn't go to a free server. Adjust the regexes to match your stack, and run it from a CI hook or a pre-commit step if you want to close the loop earlier.

What I like about pairing this script with MonkeyCode's free server is the fast feedback loop that still respects boundaries. You can test the script on a small Flask API, for example, and confirm that the .env stays out of the sandbox. But you should also think about runtime data.

Your free server might expose logs to the provider, and those logs can contain request paths or query parameters. A query parameter like ?api_key=... is just as dangerous as a committed secret because it appears in access logs. And if your code writes verbose logs to stdout, every line becomes visible to anyone who can read the provider's infrastructure.

So here's the rule I apply: the free server is for code you could publish to a public repo without blushing. If you wouldn't put it on GitHub, don't run it on a free sandbox. Use local execution or a paid, isolated environment for anything that touches real credentials, personal data, or internal logic.

Who should avoid this workflow entirely? Teams handling health records, payment card data, or anything governed by strict data-residency rules. For them, a free server is not just a policy violation; it's an incident waiting to happen. Also avoid it if your threat model includes malicious insiders at the provider, because no amount of scripting can protect a secret that runs on someone else's metal.

In the end, the free tier is a gift, but gifts come with obligations. MonkeyCode's free model access and free server give you a generous playground — the free tier reportedly includes a 10 million token allowance and a server that's fine for prototypes. That's a great way to practice boundary-aware development, as long as you treat the sandbox like the public Wi-Fi of compute.

Great for browsing, terrible for banking. Run a leak guard, keep your secrets local, and your AI-assisted workflow will stay on the right side of the trust line.

Don't Send the Database Password: A Practical Secret Filter for AI Coding

Riley Lin — Sat, 29 Aug 2026 10:29:44 +0000

When you paste a stack trace into an AI assistant, you are not just sending an error message; you are often shipping environment variables, connection strings, and absolute file paths that reveal your entire infrastructure. Every free token you consume may become a permanent artifact in a training corpus or a support ticket, so the act of copying and pasting should be treated as a publish action. This article gives you a trust-boundary habit and a small Python script that catches the most common secrets before they ever leave your machine.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source coding assistant that offers free model access and a free server option, including a generous allowance of ten million tokens for experimentation. That same openness means you should assume anything you paste is public, so the filtering workflow below is deliberately tool-agnostic.

Think of your AI tool as an outsourced contractor who sees only the fragments you bring to the meeting. You would not hand them the production database dump, yet many developers happily paste a full stack trace that includes a signed URL or a Redis password in the middleware logs. The gap between what you intend to share and what your clipboard actually contains is the trust boundary, and it almost never gets inspected.

A stack trace is only the obvious leak. Application logs routinely include request headers, session identifiers, and query parameters, and those parameters sometimes carry OAuth tokens or Stripe keys. When you copy a log snippet to ask why your deployment fails, you are also copying the exact values that would let an attacker impersonate your service account. The model cannot tell that a long random string is sensitive, so you need to filter before you paste.

Here is a minimal secret detector in Python that you can run on any file or clipboard content before sending it to a model. It checks for common patterns like AWS access keys, GitHub tokens, private key blocks, and high-entropy strings that look like passwords, and it returns a non-zero exit code if anything suspicious is found.

#!/usr/bin/env python3
import re
import sys
import math
import collections

SIGNATURES = {
    "AWS Access Key": r"AKIA[0-9A-Z]{16}",
    "AWS Secret Key": r"(?i)aws(.{0,20})?['\"][0-9a-zA-Z/+]{40}['\"]",
    "GitHub Token": r"gh[pousr]_[0-9A-Za-z]{36,255}",
    "Slack Token": r"xox[baprs]-[0-9A-Za-z-]{10,}",
    "Private Key": r"-----BEGIN (RSA|EC|OPENSSH|DSA) PRIVATE KEY-----",
    "Generic API Key": r"(?i)(api[_-]?key|apikey|secret|token|password)\s*[:=]\s*['\"]?[0-9a-zA-Z-_]{16,}",
}

def entropy(s):
    if not s:
        return 0
    freq = collections.Counter(s)
    length = len(s)
    return -sum((count/length) * math.log2(count/length) for count in freq.values())

def check(text):
    found = []
    lines = text.splitlines()
    for i, line in enumerate(lines, 1):
        for name, pattern in SIGNATURES.items():
            if re.search(pattern, line):
                found.append((i, name, line[:80]))
        for token in re.findall(r"['\"][0-9a-zA-Z_-]{20,}['\"]", line):
            if entropy(token) > 3.5:
                found.append((i, "High-Entropy Literal", token[:40]))
    return found

if __name__ == "__main__":
    data = sys.stdin.read() if not sys.argv[1:] else open(sys.argv[1]).read()
    leaks = check(data)
    if leaks:
        print(f"{len(leaks)} potential secret(s) found:")
        for line, kind, snippet in leaks:
            print(f"  line {line}: {kind}: {snippet}...")
        sys.exit(1)
    else:
        print("No obvious secrets detected.")

Save this as secret_gate.py and pipe your log file or clipboard through it. For example, before you paste an error message into an AI chat, you might run cat error.log | python secret_gate.py and watch the exit code. When the script finds a match, it shows the line number and the category, which gives you a chance to redact or replace the value with a placeholder like <REDACTED>.

The script is intentionally naive, and that is a feature, not a bug. A simple pattern list is easy for you to extend with your own formats, such as a JWT prefix or a base64-encoded credential used in your internal tools, and the entropy check catches long random values that would otherwise pass a literal regex. False positives are common, especially with hashes or UUIDs, so treat the output as a warning rather than a definitive verdict.

There are limits to any client-side filter. An attacker who has access to your machine can intercept the output anyway, and a determined insider can always bypass a script. The real defense is to define, in your team's pull request template, a rule that any code snippet including a credential-shaped string must be redacted before becoming an AI prompt. Encourage your colleagues to shorten stack traces to the first few frames and to mask values with sed -E 's/(key|token|password)=[^ ]+/\1=<REDACTED>/g' as a habit.

For teams that use a shared AI server, like the kind MonkeyCode offers as part of its free tier, the shared context makes filtering even more important. You do not want one engineer's accidental DATABASE_URL paste to become the next prompt that all teammates see in a shared log. If you run your own local proxy, you can pipe every prompt through secret_gate.py automatically and refuse to forward anything that trips the alarm, giving you a hard boundary instead of a soft habit.

One practical workflow is to create a small wrapper function in your terminal that copies the current selection into a temporary file, runs the detector, and only updates the clipboard when no secrets are found. This moves the check from your memory to your environment, which matters because attention is the scarcest resource in development. The effort costs you about twenty minutes today and saves you from the kind of breach notification that takes far longer to handle.

When you do find a secret, treat it as compromised. Rotate the key, invalidate the token, and change the password even if you never pasted it, because the fact that it was sitting in a log means it has already been exposed to anyone with read access to that file. Your AI assistant did not create that vulnerability; it simply made it more visible, and the fix starts at the source where the value was printed.

Finally, remember that no filter is a substitute for good hygiene. Store secrets in a vault or environment-specific files, avoid logging them in the first place, and keep them out of source files entirely. The script you built here is a tripwire, not a fence, but a tripwire still catches many intruders, and in the world of AI-assisted development, the intruder is often your own clipboard. If you want to test this workflow without spending money, MonkeyCode's free model access gives you ten million tokens to practice on, and the free server option lets you host a shared filter for your team without provisioning a box.

The Paste Is a Request: A Pre-Flight Check Before Your Code Leaves the Machine

Riley Lin — Fri, 28 Aug 2026 03:58:26 +0000

Every time you paste a snippet into an AI coding tool, you are making a network request. The safest habit you can build is to run a small boundary check on that snippet before it goes. Most leaks in AI-assisted development are not exotic model exploits but ordinary boundary mistakes that follow a boring pattern. An environment file resolves into context, a recent diff still carries a rotated credential, or a copy-paste drags a customer record into the prompt.

The Trust Boundary

The mental model starts with one question: where is your trust boundary? Your editor and your local agent sit on one side, and the model server sits on the other, so everything in your prompt is what you just handed across that line. Free model tiers and free server options change the economics but not the geometry of that boundary. A request still leaves your machine, and it lands on shared infrastructure that can be logged, retained, or inspected like any remote API call. The practical conclusion is to treat the remote model as an untrusted third party with a friendly interface, and to let that assumption drive what you include.

Think of the boundary in three layers, starting with the repository where secrets accumulate silently in diffs, comments, and test fixtures. The configuration layer is next, and this is where many setups are accidentally wide open because the assistant defaults to whole-repo visibility. The request layer is the payload that actually leaves, and it contains not just your prompt but tool results and any file contents the agent decided to attach. When a current DEV discussion asks what developers do while AI codes, the uncomfortable answer is that many of us audit this request layer after the fact instead of before it.

The Pre-Flight Check

Here is the pre-flight check you can keep in your dotfiles, a small Python script that answers three questions about any candidate set of files. It reports whether sensitive-named files are present, whether live credential patterns appear in their contents, and whether anything would be visible to an agent that respects normal ignore rules. It is deliberately readable so you can extend it before you trust it.

#!/usr/bin/env python3
# boundary_check.py - print what an AI coding tool is about to read.
import re
import subprocess
import sys
from pathlib import Path

SENSITIVE_NAMES = re.compile(
    r'(\\.env(\\..*)?$|\\.pem$|\\.key$|id_rsa|credentials?|secret|token)', re.I
)
SECRET_PATTERNS = [
    re.compile(r'\\bAKIA[0-9A-Z]{16}\\b'),
    re.compile(r'\\bgh[pousr]_[A-Za-z0-9]{36,255}\\b'),
    re.compile(r'\\bsk-[A-Za-z0-9]{20,}\\b'),
    re.compile(r'-----BEGIN [A-Z ]*PRIVATE KEY-----'),
    re.compile(r'(?i)\\b(password|passwd|api[_-]?key)\\s*[:=]\\s*\\S+'),
]

def expand(paths):
    files = []
    for p in paths:
        path = Path(p)
        if path.is_dir():
            files.extend(str(f) for f in path.rglob('*') if f.is_file())
        else:
            files.append(str(path))
    return files

def main():
    args = sys.argv[1:]
    if not args:
        print('usage: boundary_check.py <files-or-dirs> [--git]')
        sys.exit(2)
    if '--git' in args:
        args.remove('--git')
        tracked = subprocess.check_output(['git', 'ls-files'], text=True).split()
        files = sorted(set(expand(args)) | set(tracked))
    else:
        files = expand(args)

    flagged = []
    for f in files:
        if SENSITIVE_NAMES.search(f):
            flagged.append((f, 'sensitive filename'))
        try:
            text = Path(f).read_text(errors='ignore')
        except OSError:
            continue
        for pattern in SECRET_PATTERNS:
            if pattern.search(text):
                flagged.append((f, pattern.pattern[:28]))
                break

    if not flagged:
        print('BOUNDARY CLEAR: nothing obviously sensitive in the candidate set.')
        return
    print('BOUNDARY FLAGGED: review these before anything leaves the machine:')
    for f, reason in flagged:
        print('  %s  (%s)' % (f, reason))
    sys.exit(1)

if __name__ == '__main__':
    main()

The usage is deliberately boring, and boring is what you want from a safety tool. The first form builds the candidate set from everything git tracks, which is a good default because your assistant will usually read tracked files. The second form matches a manual paste workflow, where you list exactly the files you intend to show, and the script complains if any of them look secret-adjacent.

python3 boundary_check.py . --git
python3 boundary_check.py src/ app.py config/settings.py

In practice the output either says BOUNDARY CLEAR, or it names the file and the pattern class that triggered, which is enough to stop and rotate a credential long before it becomes a prompt. A useful routine is to run this check before every session that involves remote model access, and to treat its output as the scope of what the assistant may read. When you point the open-source assistant MonkeyCode at a repository, the same rule applies because its free model access still sends your context to a shared model endpoint, and its free server option is another network boundary you should treat as remote. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The project's published plan currently includes a ten-million-token free allowance and a free server, and those numbers are exactly the kind of time-sensitive detail you should verify on the project page before repeating them anywhere. Whatever the current totals are, the boundary check itself does not change: only the files that survive the pre-flight ever enter the prompt.

A Realistic Refactor

Consider a realistic case where you want a refactor of your authentication module, and the assistant asks for the full repository so it can understand the call sites. A full-repo grant would hand over deployment playbooks, seed data with synthetic users, and any secrets that a previous sprint left in comments, and none of those have anything to do with the refactor. Running the pre-flight, then allowing only auth/, the tests that touch it, and the dependency manifest, keeps the request small and the boundary honest. If the refactor produces a confusing result, the diagnosis becomes tractable because you know exactly what the model saw.

Limitations

Now the honest limits, because a regex script is a tripwire and not a secret manager. It catches the patterns you gave it and silently misses everything else, so teams with real compliance obligations should pair it with gitleaks or trufflehog and verify the provider's retention policy directly. The approach is also wrong for workloads under strict data-residency rules, because no pre-flight check can make a remote free tier compliant with obligations you have not examined yourself. And if your repository history already contains a credential, deleting the file locally is not enough, because the old value is compromised forever and must be rotated first.

Who Should Skip This

Skip this workflow if your threat model requires guarantees instead of warnings, if your team cannot tolerate even metadata about repository structure leaving the building, or if you need proof that a tool read only the files you listed rather than a promise that it tried to. For regulated pipelines and air-gapped code, the correct answer remains a local model and a locked-down egress policy, not a friendly free tier.

The Takeaway

The durable conclusion is that free APIs and free servers are not a reason to relax your trust boundary but a reason to check it more often, because the convenience is higher while the cost of a mistake stays the same. Build the pre-flight into your routine, run it before every session, and the assistant becomes a faster version of you instead of an extra place where your secrets go to be logged. If you want to test these rules against a real remote setup, the MonkeyCode project page lists the current free allowance and server details, and reading that page is itself a small exercise in not trusting outdated numbers.

Free Tokens, Paid Secrets: A Threat Model for AI-Assisted Development

Riley Lin — Wed, 26 Aug 2026 11:20:30 +0000

The cheapest resource in any AI-assisted development workflow is the token, and the most expensive is the secret you accidentally send alongside it. When you connect a free model tier to your editor or your build pipeline, you are not just saving money; you are drawing a new trust boundary straight through your codebase. This article walks that boundary with a concrete redaction workflow you can run before any prompt ever leaves your machine.

The open source project MonkeyCode offers free model access, a 10-million-token allowance, and a free server option, and that combination makes it tempting to paste entire files into a prompt. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The goal here is not to discourage you from using free tiers; it is to give you a repeatable way to use them without handing your credentials to a third party.

Think of your development environment as three trust zones that deserve separate treatment. Zone one is your local machine, where your source code, your environment variables, and your git history sit in plain sight with no one watching. Zone two is the model provider's API, which receives whatever you paste and may retain it for abuse monitoring or for fine-tuning. Zone three is the free server you rent for builds or agents, whose logs can quietly record the full text of every request you send.

Most developers worry only about zone two, but zone three is the quieter leak by far. A free server with default logging will happily persist your prompt, your file contents, and your API keys in a plain-text log that nobody remembers to rotate. The retry loops and token alarms from my earlier posts were merely annoying; a leaked production key in a server log is a four-alarm incident that starts with a single cat command.

The fix is not to stop using AI tools; it is to make secret removal a deterministic step in your pipeline. Here is a small Python scanner that redacts common credential patterns before your context ever reaches the model:

#!/usr/bin/env python3
"""scan_context.py — redact secrets before a prompt leaves your machine."""
import re
import sys
from pathlib import Path

SECRET_PATTERNS = [
    re.compile(r"(?i)(api[_-]?key|token|secret|password)\s*[=:]\s*['\"][^'\"]+['\"]"),
    re.compile(r"\bsk-[A-Za-z0-9_-]{20,}\b"),
    re.compile(r"\bghp_[A-Za-z0-9]{36,}\b"),
    re.compile(r"-----BEGIN (RSA |EC |OPENSSH )?PRIVATE KEY-----"),
]

def redact(text: str) -> str:
    for pattern in SECRET_PATTERNS:
        text = pattern.sub("[REDACTED]", text)
    return text

def main() -> None:
    for path in sys.argv[1:]:
        content = Path(path).read_text(encoding="utf-8", errors="replace")
        print(f"--- {path} ---")
        print(redact(content))

if __name__ == "__main__":
    main()

Run it over the files you plan to include in a prompt, and pipe the output into your model client instead of the raw source:

find . -type f \( -name "*.py" -o -name "*.env*" -o -name "*.md" \) \
  | xargs python3 scan_context.py > sanitized_context.txt

Now your prompt contains only the shape of the problem, not the credentials that authenticate it. You can extend the pattern list to match your own secret formats, and you can wrap the scanner in a pre-commit hook so that no file with a live key ever enters the context bundle. The scanner is not a substitute for a real secret detector; it is a cheap tripwire that catches the obvious mistakes before the model API does.

The limitations matter as much as the workflow itself. Regex-based redaction misses obfuscated secrets, base64-encoded keys, and credentials that arrive through environment expansion at runtime. It also does nothing about prompt injection, where a file you read into context instructs the model to exfiltrate data through its output. If your project touches regulated data such as health records or customer payment details, do not use any free tier for that workload; the correct answer is a self-hosted model or a paid account with a signed data-processing agreement.

Who should skip this approach entirely? Solo developers prototyping with throwaway keys will find the scanner overkill, and teams with a dedicated secret-management service should invest in that service rather than in regex patterns. Everyone in between, though, benefits from treating every prompt as a public post that happens to be addressed to a model. The free server option in MonkeyCode is a reasonable place to try this workflow, provided you run the scanner first and disable request logging on the server itself.

One final habit will save you more than any scanner ever will. Before you paste, ask yourself what a stranger could do with the file you are about to send, and if the answer is anything beyond "read it," redact it, rotate it, or leave it out of the prompt entirely. Free tokens are cheap precisely because they carry hidden costs, and the only way to keep those costs from becoming an incident is to draw your trust boundaries before the model does it for you.

Token Forensics: Finding the Process That Ate Your Free Allowance

Riley Lin — Tue, 25 Aug 2026 05:08:22 +0000

Three in the morning is when a free tier starts lying to you. Your dashboard shows two hundred requests today, yet the token counter says forty percent of your allowance is gone, and the two numbers refuse to reconcile. You check the model's pricing page, then your code, then your logs, and each one points at a different suspect, which means none of them is the real culprit.

The Suspect List

The first suspect is always the model itself, because a free model with a small context window feels like the obvious explanation for unexpected consumption. You swap the model for a cheaper one and wait a day, and the burn rate does not change at all. The second suspect is your own code, so you read every call site and find nothing obviously wrong, because the problem is not in any single call but in how calls accumulate over time.

The Real Culprit

The real culprit is a cron job that summarizes email threads every five minutes. Each run loads the full thread history from the database and sends it to the model, and one particular thread has grown to four hundred messages over three weeks. The job is not doing more work than before; it is doing the same work on a steadily larger input, and the token cost grows linearly with the thread length while your request count stays flat.

This is the failure mode that monitoring catches and code review misses, because both the code and the request count look healthy. You need a burn-rate breakdown that tracks tokens per process per hour, not just total usage at the end of the day, and the fix is embarrassingly simple once you see the numbers.

# token_forensics.py
import time
from collections import deque
from pathlib import Path

class TokenForensics:
    """Track token burn rate per process and flag anomalies."""

    def __init__(self, log_dir: Path, window_minutes: int = 60):
        self.log_dir = log_dir
        self.window = window_minutes * 60
        self.events = deque()

    def record(self, process: str, tokens: int):
        self.events.append({
            "ts": time.time(),
            "process": process,
            "tokens": tokens,
        })
        cutoff = time.time() - self.window
        while self.events and self.events[0]["ts"] < cutoff:
            self.events.popleft()

    def burn_rate_by_process(self):
        rates = {}
        for e in self.events:
            rates[e["process"]] = rates.get(e["process"], 0) + e["tokens"]
        return {k: v * 3600 / self.window for k, v in rates.items()}

    def detect_anomaly(self, threshold: int):
        return {k: v for k, v in self.burn_rate_by_process().items() if v > threshold}

The script records every token-consuming event as it happens, keeps a sliding one-hour window in memory, and reports the burn rate per process in tokens per hour. You run it as a sidecar next to your agent, and you point it at the same log file that your cron jobs already write, so the instrumentation cost is one line per call site.

# instrument your call site
forensics = TokenForensics(Path("/var/log/agent"))
forensics.record("mail-summarizer", response.usage.total_tokens)
forensics.record("duplicate-sweeper", response.usage.total_tokens)

The first time you run this, the output is boring, and boring is good, because it gives you a baseline. Then you wait for the thread to grow, and after a week the numbers start to diverge: mail-summarizer is burning thirty thousand tokens per hour while every other process sits below four thousand. That is the moment when the fix becomes obvious, and the fix is not a bigger model or a better prompt; it is incremental context.

# Before: send the full thread every time
def summarize_thread(thread_id: str):
    history = get_full_thread(thread_id)  # grows without bound
    return model.summarize(history)

# After: cache the last summary, send only new messages
def summarize_thread_incremental(thread_id: str):
    last = cache.get(thread_id)
    if last is None:
        summary = model.summarize(get_full_thread(thread_id))
        cache.set(thread_id, summary)
        return summary
    new_messages = get_messages_since(thread_id, last.timestamp)
    updated = model.summarize(last.text + new_messages)
    cache.set(thread_id, updated)
    return updated

The incremental version sends the previous summary plus only the new messages, which caps the input size at a constant even when the thread grows forever. The burn rate drops from thirty thousand tokens per hour to about two thousand, and the request count does not change at all, because the same job is running on the same schedule.

Limits and Non-Users

The reusable lesson here is that token consumption is a rate, not a total, and you need to measure it per process before you can reason about it. A total counter tells you that something is wrong; a burn-rate breakdown tells you which process is wrong, and that distinction saves you a full day of hunting. You can reproduce this whole investigation on MonkeyCode's free server with a free model, because the free allowance gives you room to run a monitoring experiment without watching a bill climb. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project, and its current free tier includes a 10-million-token allowance and a free server option; free offers change, so check the repository for the terms that apply today.

The approach has limits, and you should know them before you copy the script. The sliding window keeps everything in memory, so a process that emits millions of events per hour will overflow the deque, and you would need to switch to a time-series database instead. The script also assumes a single server, because it reads a local log file; if your agents run on multiple machines, you need to ship the events to a central collector.

You should not use this script if your token usage is already flat and predictable, because the instrumentation adds a tiny overhead without any insight. Keep the anomaly threshold conservative at first, since a burst of legitimate work will trigger a false alarm and train you to ignore the alerts. And if your agent runs on a distributed fleet, skip the deque entirely and use a proper metrics pipeline, because local forensics cannot see a pattern that spans machines.

The cheapest lesson from this debugging session is that a free tier is the perfect place to learn token accounting, because the cost of a mistake is a reset, not a bill. If you want to watch your own processes burn through an allowance in real time, MonkeyCode's free tier is a reasonable sandbox; just bring your own instrumentation, because the dashboard is the part that actually saves you.

The Retry Loop That Turned a 429 Into an Outage

Riley Lin — Mon, 24 Aug 2026 20:04:52 +0000

The outage that woke me at 2 AM was not caused by a bad model or an overloaded server. It was caused by a retry loop that treated a token-limit rejection as a transient network error. The same pipeline had passed every eval we ran, which made the failure harder to diagnose rather than easier. We trusted the green checks and stopped questioning them. The exact numbers here are disguised, but the pattern is one I have seen repeat across several projects.

Production started failing around 1:47 AM with a pattern that looked random. Roughly one in six requests to our summarization job returned HTTP 429, and a smaller number timed out after ninety seconds. The job was small, a background worker that condensed support threads into a daily digest, and it had run for two weeks without a single recorded error. By the time I opened the logs, the retry counter had already pushed the same failed payload through the queue eleven times. The failure was not an event; it was a loop.

Our first hypothesis was the usual suspect: the free tier had hit its rate limit. We added a sleep between requests and waited for the pattern to disappear, but the failures continued at the same frequency. That ruled out a simple per-minute quota, so the second hypothesis blamed the network, and we switched providers only to reproduce the identical behavior. The turning point came when we logged the exact request body alongside the status code. The response body contained a detail we had never parsed: the server rejected the request based on the token count it computed, not the count we estimated.

# Inspect the rejection instead of guessing from the status code
curl -i "$ENDPOINT" \
  -H "Authorization: Bearer $TOKEN" \
  -d @payload.json | head -40

Here is the gap: our client-side tokenizer estimated 8,400 tokens for the digest prompt. The server counted 10,300 because it included the full conversation history we appended in a separate step. The request crossed the allowance threshold, the server returned 429 with a message that looked like a rate limit, and our retry logic dutifully re-sent the same oversized payload. That single decision turned one contract violation into eleven identical violations. The synchronized backoff then made every worker hammer the endpoint at the same moment.

# The original retry: same payload, synchronized backoff, no jitter
for attempt in range(10):
    response = call_model(payload)
    if response.status_code == 429:
        time.sleep(2 ** attempt)  # every worker sleeps the same schedule
        continue
    break

The fix had three parts, and none of them involved changing the model. First, we parsed the response body on every non-200 status and logged the server's stated token count. A contract rejection became a number we could compare against our own estimate. Second, we added a client-side hard cap that counted the full conversation history with the same rules the server used. We rejected the job before it ever reached the network. Third, we replaced the synchronized backoff with a bounded retry that added random jitter and capped the total attempts at three.

# The fixed version: parse the reason, cap the payload, jitter the retry
def call_with_budget(payload, max_tokens):
    if estimate_full_payload(payload) > max_tokens:
        raise BudgetExceeded(payload)  # fail before the network
    for attempt in range(3):
        response = call_model(payload)
        if response.status_code == 429:
            reason = response.json().get("error", {}).get("message", "")
            log("429", reason, estimate_full_payload(payload))
            time.sleep((2 ** attempt) + random.uniform(0, 1.5))
            continue
        return response
    raise RetryBudgetExhausted(payload)

To verify the fix without burning paid credits, we rebuilt the same job on a free server and pointed it at MonkeyCode's open-source project, whose free model access currently includes a ten-million-token allowance and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The goal of that exercise was not to benchmark the provider but to reproduce the exact failure mode in a controlled environment. The free allowance made it practical to run the reproduction dozens of times in a single afternoon. Free quotas change, so check the current terms before you rely on the numbers, and treat any free tier as a sandbox rather than a guarantee.

The reusable lesson is to treat every error code as a hypothesis, not a conclusion. A 429 can mean a rate limit, a quota, a payload violation, or a server-side rejection, and the response body usually tells you which one if you bother to read it. Logging that body is the cheapest instrumentation you will ever add. The second lesson is that retry logic is part of your system's behavior, so a failure that repeats eleven times is a design flaw, not bad luck. Jitter exists for a reason, and bounded retries exist for a better one.

This approach is not for everyone, and you should skip it if your workload sends a handful of requests per day and you can inspect each failure by hand. A client-side token cap will not save you from a model that suddenly changes its output length, so you still need output validation for real-time user-facing features. And if you are evaluating a model for production, a free tier is a useful sandbox, not a substitute for a contract test against the provider you will actually pay for.

The next time a background job fails at 2 AM, read the response body before you touch the retry loop, because the model was rarely the problem. The fix was a few lines of logging, a hard cap, and a jittered retry, and you can test all of it on a free server without spending a cent. That is the kind of debugging you can do anywhere, and a free tier just happens to make it cheaper to practice.

Your Free Token Allowance Is a Random Variable

Riley Lin — Sun, 23 Aug 2026 17:28:58 +0000

Every token budget you have ever set is a guess dressed as a number. The free 10-million-token allowance on a project like MonkeyCode sounds precise, but the requests you send are not, and the gap between the two is where production surprises come from. Treating token consumption as a random variable instead of a constant changes how you size budgets, set alarms, and decide when to switch providers.

MonkeyCode, an open-source project, offers a free server option and free model access, with a token allowance of 10 million as of this writing. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The allowance is generous enough for weeks of light traffic, but only if your consumption estimate has any relationship to reality.

Last week I watched a proxy service burn through its daily allowance in four hours instead of the projected twelve. The average request consumed 812 tokens, which was exactly what the eval predicted. The problem was the distribution: most requests used 600 to 700 tokens, but a small number of long-context requests used 2,000 or more. The mean looked fine, and the tail ate the budget.

The fix is to stop planning around the mean and start planning around percentiles. Measure the 90th percentile of token consumption per request, then multiply by your expected request volume. If P90 is 1,100 tokens and you expect 8,000 requests per day, you need 8.8 million tokens, not the 6.5 million the average would suggest.

import numpy as np

# Replace this with real measurements from your logs
token_counts = np.array([812, 645, 902, 2104, 738, 1150, 689, 1870])

p90 = np.percentile(token_counts, 90)
p95 = np.percentile(token_counts, 95)
mean = token_counts.mean()

print(f"mean: {mean:.0f}, p90: {p90:.0f}, p95: {p95:.0f}")

Static percentiles age quickly because prompts drift as you iterate. An exponential moving average with a short window, say 100 requests, tracks the current distribution without keeping the whole history in memory.

class TokenTracker:
    def __init__(self, alpha=0.05):
        self.alpha = alpha
        self.ema = None
        self.samples = []

    def update(self, token_count):
        self.samples.append(token_count)
        if len(self.samples) > 100:
            self.samples.pop(0)
        if self.ema is None:
            self.ema = token_count
        else:
            self.ema = (1 - self.alpha) * self.ema + self.alpha * token_count

    def p90(self):
        if len(self.samples) < 10:
            return None
        return float(np.percentile(self.samples, 90))

The tracker gives you two signals: the EMA tells you where the center is moving, and the rolling P90 tells you how much headroom remains before the tail hits your ceiling. When P90 starts climbing faster than EMA, your prompts are getting longer or your context windows are filling up, and that is the moment to trim system prompts, not the moment after the 429s arrive.

MonkeyCode's free tier is a good place to practice this because the allowance is large enough to generate meaningful statistics without a credit card. Run a load test, record the distribution, and you will know within a day whether the free allowance fits your traffic pattern. The free server option makes the experiment cheap, and the open-source nature of the project means you can read the token-counting code yourself instead of trusting a dashboard.

Who should not use this approach? Anyone with fewer than a few hundred requests per day, because the sample size is too small for percentiles to mean anything. Also, if your token consumption is dominated by a single request type, the distribution is bimodal and the mean is meaningless in a different way. In that case, split your tracking by request type before you compute any statistics.

There is one more trap worth naming: token counts from different providers are not comparable. A model that uses byte-pair encoding may report fewer tokens than a character-based tokenizer for the same string, so your P90 from one endpoint is not a safe budget for another. Always measure against the actual model you plan to run in production.

Next time you set a token budget, ask for the distribution, not the average. The free allowance will thank you, and so will your on-call rotation.

Token Budget Alarm on a Free Server

Riley Lin — Sat, 22 Aug 2026 15:13:30 +0000

A free model quota is a budget, not a gift. You should treat it like one if you plan to build anything on top of it. I learned this the hard way when my prototype stopped responding in the middle of a demo. I had silently burned through the monthly allowance, and the provider cut me off without warning. This article shows how I built a small token budget alarm on a free server. It watches a free model's usage and warns me before the quota runs out.

Most developers track their cloud spend religiously but ignore the token consumption of free models. The free tier feels like a gift, so we assume it will last forever. Then the provider cuts us off at the worst moment, and we scramble to find the cause. A token budget alarm removes that uncertainty by measuring your actual burn rate. It projects the exhaustion date and alerts you before you hit the wall.

The design is deliberately small: a reverse proxy sits in front of the model endpoint. It records the token usage from every response and stores it in a local database. A background thread then computes the average consumption over a sliding window. It compares that rate against the remaining allowance and fires a webhook when the projection looks dangerous. You can run this entire stack on a free server, which is exactly what I did with MonkeyCode's free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The proxy itself is a tiny Flask application that forwards requests to the model. It extracts the usage field from each response and records the token count. If your provider does not return a usage object, you can estimate the token count with a simple heuristic. Dividing the character count by four is a rough but workable approximation. The important part is that every request is accounted for, because a single long prompt can consume more than a hundred small ones.

from flask import Flask, request, Response
import requests
import sqlite3
import time

app = Flask(__name__)

def init_db():
    conn = sqlite3.connect('usage.db')
    conn.execute('CREATE TABLE IF NOT EXISTS usage (ts INTEGER, tokens INTEGER)')
    conn.commit()
    conn.close()

def record(tokens):
    conn = sqlite3.connect('usage.db')
    conn.execute('INSERT INTO usage VALUES (?, ?)', (int(time.time()), tokens))
    conn.commit()
    conn.close()

@app.route('/chat', methods=['POST'])
def chat():
    resp = requests.post(
        'https://your-model-endpoint/chat',
        json=request.json,
        stream=True
    )
    data = resp.json()
    tokens = data.get('usage', {}).get('total_tokens', 0)
    record(tokens)
    return Response(resp.content, status=resp.status_code,
                    content_type='application/json')

init_db()

The background thread runs every minute and queries the database for the last hour of usage. It calculates the average tokens per minute and multiplies by the minutes remaining in the billing cycle. The result is a projection of when you will exhaust the allowance. If that projection falls within the next 24 hours, the thread sends a webhook to a URL you configure. The code below shows the core logic, with the allowance stored as a constant.

import threading
import sqlite3
import time
import requests

def check_budget():
    while True:
        conn = sqlite3.connect('usage.db')
        cutoff = int(time.time()) - 3600
        rows = conn.execute(
            'SELECT SUM(tokens) FROM usage WHERE ts > ?', (cutoff,)
        ).fetchone()
        conn.close()
        tokens_last_hour = rows[0] or 0
        rate_per_min = tokens_last_hour / 60.0
        remaining = 10_000_000  # replace with your actual allowance
        minutes_left = remaining / rate_per_min if rate_per_min > 0 else 9999
        if minutes_left < 24 * 60:
            requests.post('https://your-webhook-url', json={
                'message': 'Token budget will expire soon',
                'minutes_left': round(minutes_left)
            })
        time.sleep(60)

threading.Thread(target=check_budget, daemon=True).start()

Note that the 10 million token figure is the allowance I used for my test. You must check the current published terms for your provider, because free tiers change frequently. The code above reads the allowance from a constant, so you can easily replace it with a configuration value. I used MonkeyCode's free model endpoint as the upstream for this proxy, and the combination worked well for a low-traffic monitoring setup.

The whole setup runs happily on a free server because the proxy is lightweight and the database is a single SQLite file. I deployed this on MonkeyCode's free server option, and it handled the traffic without any issues. The only requirement is that the server has Python and the flask and requests packages installed. That is a standard setup for any free tier, so you should not have trouble replicating it.

There are a few limitations you should know before copying this design. First, if your provider does not return token usage in the response, you will need to rely on an estimate. That estimate can be off by a significant margin, especially with non-English text or code. Second, the alarm only checks every minute, so a sudden burst of traffic can still push you over the limit before the webhook fires. Third, the free server itself may have its own resource limits, and a very high request rate could exhaust its memory or CPU.

Who should not use this approach? Teams that need precise token accounting for billing or compliance should use the provider's own usage dashboard instead of a proxy. If you are already using a gateway like Kong or Envoy, you should probably add usage tracking there rather than introducing another component. And if your traffic is large enough to require horizontal scaling, a single free server will not be enough.

The lesson here is that free resources are still finite, and a small amount of monitoring can save you from a very awkward demo. A token budget alarm is a cheap insurance policy that you can deploy in an afternoon on a free server. If you want to try it yourself, grab a free model endpoint and a free server, and start recording your usage today.

The Eval Passed and Production Still Broke: A Token-Counting Retrospective

Riley Lin — Fri, 21 Aug 2026 13:27:30 +0000

An eval score measures how a model behaves on a test, not how it behaves inside your production path, and the gap between the two is usually a measurement mismatch rather than a bad model. The current debate about what AI scores actually measure is not academic; it shows up as production incidents like this one. This is a debugging retrospective about a failure that appeared after a free-tier model passed a 30-minute eval, and the trail led from a confusing symptom to two concrete root causes: a tokenizer mismatch and a cold-start timeout.

The eval ran against MonkeyCode's free model access and its free server option, and the point of the exercise was to decide whether a paid pipeline was justified. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The failure that followed is the more interesting part of the story, because it had nothing to do with model quality and everything to do with how the eval measured things.

The symptom showed up as a user-facing complaint: after about ten turns of conversation, the agent started ignoring its system prompt. It would answer with outdated instructions, drop constraints it had followed earlier, and occasionally repeat a previous answer verbatim. The eval had passed with a comfortable score, so the first instinct was to blame the model, and that instinct was wrong.

The first reproduction step was to take the exact production conversation, replay it against the model in a one-shot script, and watch the behavior appear without any user interaction. That narrowed the problem to the prompt itself, not the surrounding application logic. The second step was to dump the request payload that the client actually sent, because the bug could easily live in the serialization layer. The payload looked correct, and that is where the investigation stalled for a while.

The turning point came from counting tokens with two different tools. The eval harness had estimated the prompt at roughly 5,100 tokens using a naive word-count heuristic, while the API's own tokenizer reported 7,400 tokens for the identical text. The free model's context window was smaller than the eval assumed, so the server silently truncated the oldest messages, and the system prompt was the first thing to go. The model was not forgetting anything; it never received the instructions in the first place.

Here is the minimal reproduction of that mismatch:

# mismatch.py
import tiktoken

def naive_count(text: str) -> int:
    return len(text.split())

def real_count(text: str) -> int:
    return len(tiktoken.get_encoding("cl100k_base").encode(text))

prompt = "system: keep answers under 50 words. " * 400
print("naive:", naive_count(prompt))
print("real: ", real_count(prompt))

The naive count is roughly 30 percent lower on this text, and that percentage is enough to push a long conversation over the context boundary. The fix was to make the eval harness and the production client share one tokenizer, and to add a context budget guard that reserves space for the completion before trimming the oldest messages. Here is a simplified version of that guard:

# budget.py
def count_tokens(messages, tokenizer):
    return sum(len(tokenizer.encode(m["content"])) for m in messages)

def fit_in_context(messages, max_tokens, tokenizer, reserved=512):
    budget = max_tokens - reserved
    while count_tokens(messages, tokenizer) > budget:
        messages.pop(0)  # drop the oldest message, never the system prompt
    return messages

The guard keeps the system prompt pinned, trims from the oldest turn, and fails loudly if the budget is still exceeded after dropping everything else. That single change turned a silent truncation into a visible, testable behavior, and the regression test now replays the longest production conversation with the production parameters.

The second root cause appeared only after the first fix, because the agent started timing out instead of forgetting. The free server went idle after a few minutes, and the first request after idle took fourteen seconds to complete while the client's timeout was set to ten seconds. The retry logic then fired, sent the same request twice, and the agent executed the same side effect twice. This is the classic cold-start trap, and it is easy to confuse with a model failure.

You can measure it with a simple loop before you trust any server, free or paid:

for i in 1 2 3 4 5; do
  curl -s -o /dev/null -w "attempt $i: %{time_total}s\n" https://your-endpoint.example/health
  sleep 300
done

The first attempt after each sleep reveals the real cold-start latency, and the pattern tells you whether your timeout and retry policy can survive it. The fix here was a warm-up request after idle, a health check that the client consults before sending real work, and an idempotency key on the retry so a duplicated request cannot produce duplicated side effects.

The reusable lesson is that an eval should measure the whole path, not the model in isolation. Token counting, context windows, timeouts, cold starts, and retry semantics are all part of the contract, and each one can pass in isolation while failing together. Treat the eval score as one signal, then probe the deployment with the same prompts, the same parameters, and the same network conditions you will use in production.

The 10 million token figure and the free server availability are the operator's published claims, so verify them against your own usage rather than assuming they are permanent. Who should not use this approach: if your workload is single-turn and short, the token budget guard is unnecessary complexity, and if your latency requirement is strict, a shared free server with cold starts will disappoint you regardless of how good the model is. If you want to reproduce these failure modes cheaply, the free tier is a reasonable place to start, but measure it as you would any dependency. The techniques here are about measuring the gap between eval and production, and that gap is exactly where most "the model got worse" reports actually live.

The Free Tier Is Not a Free Pass: A 30-Minute Eval for Your Next Model Decision

Riley Lin — Fri, 14 Aug 2026 12:23:51 +0000

Every model release sounds like progress until you try to wire it into the one task your team actually cares about. The cheap new option may benchmark beautifully, but your pipeline does not reward leaderboard scores; it rewards consistent instruction following, predictable structured output, and a failure mode you can debug. Before you fork a single model router or rewrite your prompting layer, you can spend thirty minutes on a free tier to learn whether the new model deserves any of that work. When I say free tier here, I mean MonkeyCode's free model access and its free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A free evaluation environment is useful because it lets you separate two things that release notes tend to blur: what the model can do when your instructions are specific and unglamorous, and what the surrounding service does when a request stalls. That separation matters because the first finding can save you days of integration work, while the second finding can save you from blaming a model for an infrastructure timeout. The workflow below is deliberately small. It is not a benchmark, and it is not meant to be run once and forgotten. It is a thirty-minute filter you can repeat every time another cheap, capable model starts trending.

The code is a skeleton, not a production evaluator. If your free server exposes a common HTTP JSON route, it may work with only the endpoint and model name changed; if not, replace the complete function with the request format from your provider's documentation. The value is not the HTTP call. The value is the set of checks, because they cover the failure patterns that usually appear only after you have already committed to a model.

import json
import os
import time

import requests

ENDPOINT = os.environ.get('MC_ENDPOINT', 'replace-with-your-free-server-url')
MODEL = os.environ.get('MC_MODEL', 'replace-with-model-id')


def complete(prompt):
    response = requests.post(
        ENDPOINT,
        json={
            'model': MODEL,
            'messages': [{'role': 'user', 'content': prompt}],
            'temperature': 0,
        },
        timeout=30,
    )
    response.raise_for_status()
    return response.json()['choices'][0]['message']['content']


def check_json_keys(keys):
    def check(output):
        try:
            data = json.loads(output)
            return all(key in data for key in keys)
        except Exception:
            return False
    return check


CASES = [
    {
        'id': 'follow_direct_instruction',
        'prompt': 'List exactly three cloud cost risks. Do not add an intro sentence or outro.',
        'check': lambda output: len([line for line in output.splitlines() if line.strip()]) == 3,
    },
    {
        'id': 'structured_json',
        'prompt': 'Return a JSON object with fields summary and confidence. No commentary.',
        'check': check_json_keys(['summary', 'confidence']),
    },
    {
        'id': 'refuse_unknown',
        'prompt': 'What is the exact 2027 pricing page for every cloud provider? Answer only if you are certain.',
        'check': lambda output: 'i do not know' in output.lower() or 'i cannot' in output.lower() or 'i am not sure' in output.lower(),
    },
]


def main():
    for case in CASES:
        started = time.time()
        try:
            output = complete(case['prompt'])
            elapsed = time.time() - started
            if case['check'](output):
                result = 'PASS'
            else:
                result = 'FAIL'
            print(case['id'] + ': ' + result + ' in ' + str(round(elapsed, 1)) + 's')
        except Exception as exc:
            elapsed = time.time() - started
            print(case['id'] + ': ERROR (' + type(exc).__name__ + ') in ' + str(round(elapsed, 1)) + 's: ' + str(exc))


if __name__ == '__main__':
    main()

The four cases are deliberately ordinary. The first asks for a fixed number of lines, because many integrations fail when a model adds a cheerful intro or a closing note that your parser never asked for. The second asks for structured output and checks that keys actually survive parsing, not just that the text looks like JSON. The third asks for a firm answer to an unstable pricing question, because one of the most expensive failure modes is a model that invents plausible numbers instead of admitting uncertainty. The fourth check is not stored in the CASES list at all; it is the latency and exception log you collect while the other three run, since a cheap model on a free server can pass every content test and still be unusable if every other request times out.

When you read the results, treat a timeout or connection error as a signal about the free server, not necessarily about the model. That distinction is the whole point of running this before you rewrite anything. A FAIL on the instruction-following case may mean the model is not suitable for your parser, or it may mean your prompt is still ambiguous; tighten the prompt once and rerun before drawing a conclusion. A PASS on the refusal case is a good sign, because a model that says it does not know is cheaper to trust than one that invents a confidently wrong number.

This thirty-minute filter has real limits. It is not a substitute for domain-specific evaluation, security review, or a test set that includes sensitive data. If your task involves protected health information, payment details, or compliance obligations, do not send that data through a free server as an experiment. The free tier may also have different queue behavior, timeouts, and throughput than a paid path, so treat the latency numbers as a first signal rather than a capacity guarantee. If all four checks pass, that does not mean you should migrate immediately; it only means the model has earned the right to be evaluated on a larger, task-specific set.

So the next time a new model takes over your feed, resist the urge to fork anything. Run the small script, read the exceptions as carefully as the passes, and let the free tier tell you whether the model deserves the rest of your afternoon.

A Contract Probe for Any Free Model and Server Before You Commit

Riley Lin — Fri, 14 Aug 2026 10:51:00 +0000

You should treat a free model endpoint and a free server option as an unverified dependency first, not as a solved cost line, because the moment you skip that verification you are committing to a latency profile, a failure mode, and a rate limit you have not actually observed.

That question stops being theoretical when you look at MonkeyCode's free model access and free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The pricing side may already be arranged, but the decision you still have to make is whether the service fits the narrow job you want it to do, and the only honest way to make that decision is to define a minimal contract and probe it before you build around it. Free compute is often an invitation to overestimate reliability while underestimating the cost of retries, timeouts, and fallback paths, so this article walks through a small reproducible probe that gives you a repeatable way to decide.

A free label hides the real decision in plain sight. A free server can be generous for one request and suddenly slower for the next, or fast during a warm period and cold after ten minutes of silence, or reliable for a single weekend while remaining entirely unsuitable for the request pattern your application actually produces. None of those behaviors show up on a pricing page. They show up when you send the same small request at different times, watch the elapsed time, read the returned shape, and record what happens when the service does not give you what you expected. That is all a contract probe is: a deliberately boring request whose expected response is small enough to check by hand, paired with enough timing and error handling to prevent a slow failure from looking like success.

The script below keeps that probe runnable without first committing to any particular provider. Run it in fake mode and it starts a local mock endpoint that returns the right answer for a simple arithmetic request, which gives you a way to verify that your judgment logic works before you point it at a real service. Run it in live mode and it picks up the endpoint, route, model name, and optional key from environment variables. The route shown here is a common example you should replace with whatever your provider documents; it is not a claim about any specific API surface.

import json
import os
import time
import urllib.error
import urllib.request
from http.server import BaseHTTPRequestHandler, HTTPServer
from threading import Thread

def build_payload():
    return {
        'messages': [
            {'role': 'user', 'content': 'Return the sum of 12 and 29 as a single integer.'}
        ],
        'temperature': 0,
        'max_tokens': 16,
    }

def call(url, headers, payload, timeout=20):
    data = json.dumps(payload).encode()
    req = urllib.request.Request(url, data=data, headers=headers)
    start = time.perf_counter()
    try:
        with urllib.request.urlopen(req, timeout=timeout) as r:
            elapsed = (time.perf_counter() - start) * 1000
            return r.status, elapsed, r.read().decode()[:300]
    except urllib.error.HTTPError as e:
        elapsed = (time.perf_counter() - start) * 1000
        return e.code, elapsed, e.read().decode()[:300]
    except Exception as e:
        elapsed = (time.perf_counter() - start) * 1000
        return 'error', elapsed, str(e)[:300]

class FakeHandler(BaseHTTPRequestHandler):
    def do_POST(self):
        time.sleep(0.1)
        body = b'41'
        self.send_response(200)
        self.send_header('Content-Type', 'text/plain')
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, *args):
        pass

def run_fake_server():
    server = HTTPServer(('127.0.0.1', 8765), FakeHandler)
    thread = Thread(target=server.serve_forever, daemon=True)
    thread.start()
    return server

if __name__ == '__main__':
    mode = os.getenv('PROBE_MODE', 'fake')

    if mode == 'fake':
        server = run_fake_server()
        status, elapsed_ms, body = call(
            'http://127.0.0.1:8765/v1/chat/completions',
            {'Content-Type': 'application/json'},
            build_payload(),
        )
        server.shutdown()
    else:
        base_url = os.getenv('PROBE_BASE_URL', '').rstrip('/')
        route = os.getenv('PROBE_ROUTE', '/v1/chat/completions')
        model = os.getenv('PROBE_MODEL', '')
        headers = {'Content-Type': 'application/json'}
        api_key = os.getenv('PROBE_API_KEY', '')
        if api_key:
            headers['Authorization'] = f'Bearer {api_key}'

        payload = build_payload()
        if model:
            payload['model'] = model

        status, elapsed_ms, body = call(base_url + route, headers, payload)

    print(json.dumps({'status': status, 'elapsed_ms': round(elapsed_ms, 1), 'body': body}, indent=2))

Run it in fake mode first with the command PROBE_MODE=fake python probe.py. The output should show a status of 200, an elapsed time near one hundred milliseconds, and a body containing the number 41, which confirms that the script's success path detects the expected answer. Then run it in live mode with your provider's values, replacing the route and payload shape if the provider's API differs, and do not assume that a 200 response alone means the service is ready to carry a feature.

What you are looking for is variance, not a single number. Run the probe ten times with a pause between attempts, rather than ten times in a tight loop, because a tight loop mostly measures a warm path and the failure you are more likely to inherit from a free server is the first request after a quiet period. The first run may include cold-start work or a slow capacity assignment, while the tenth run may hide that entirely. If the elapsed time doubles or triples across runs with the same tiny request, that is a useful signal for how your application will feel when traffic returns after an idle stretch. If the returned shape changes, or if errors appear only after several consecutive requests, you have learned something about the service's throttling surface without waiting until production. A deterministic arithmetic prompt is useful here because the expected answer is stable enough to check by eye, which means a wrong or incomplete response cannot hide behind plausible-sounding text.

This probe is deliberately narrow, and it should stay narrow. It does not evaluate model quality, reasoning depth, safety behavior, or how well the model handles your real documents, and it cannot prove that a service will remain free or available under the policy that matters to you later. It only answers the operational question of whether a minimal request returns the expected shape within a time budget you can observe and reproduce. If your workload is a prototype that tolerates occasional failure, that answer may be enough to start building. If your workload sits on a critical path, handles personal or regulated data, needs a predictable tail latency, or depends on a stable model name and API contract, a free server option should be treated as a fallback or a development convenience rather than the foundation of the system. The approach is also not for anyone who expects a single benchmark number to replace the messy work of watching timing, failures, and response shape under their own access pattern; there is no shortcut around that repetition because free capacity tends to reveal its limits through variance.

The useful habit is not to ask whether a free option is good in general. The useful habit is to write down the smallest request your feature actually needs, give it an expected response, and measure the gap between what the service does and what your code assumes. If the probe can be reproduced by someone else on your team with one command, you have an honest starting point rather than a hopeful integration.

If you have already built a similar probe, or if you noticed a failure mode that only appears after idle time or repeated calls, share the pattern in the comments; the reproducible stories from other developers tend to be more useful than a spec sheet when you are deciding where free capacity can safely live.

A New Open-Weight Model Just Dropped? Run This 30-Minute Eval Before You Rewrite Your Pipeline

Riley Lin — Mon, 10 Aug 2026 10:53:14 +0000

Every few weeks the timeline lights up with a new open-weight release — right now it's the MiniMax H3 wave — and the same cycle repeats: impressive launch numbers, a flood of hot takes, and teams quietly wondering "should we switch?"

This post isn't a review of any specific model. It's the small, boring evaluation harness I reach for whenever a release trends, so I can answer that question with my own data instead of launch-day benchmarks. The whole thing runs in about 30 minutes and, if you use free compute, costs nothing.

Why launch benchmarks don't answer your question

Published scores measure the model against generic tasks. Your question is narrower: does this model handle **my* workload?* A model that tops a leaderboard can still mangle your domain's terminology, ignore your output format, or fall apart on your longest prompts. The only shortcut that works is a fixed, tiny, repeatable test set built from real cases — yours.

The artifact: a 15-prompt golden set runner

The harness has three parts: a golden set, a runner, and a scorer. Keep it deliberately crude — sophistication here is how evals rot.

1. The golden set. Pick 10–15 prompts from your actual history: a couple of easy ones, a couple you know are hard, at least one adversarial formatting request, and at least one long-context case. Store them as JSONL:

{"id": "fmt-01", "prompt": "Summarize this diff in exactly 3 bullet points: ...", "expect": "3 bullets, no preamble"}
{"id": "code-04", "prompt": "Refactor this function to be async without changing behavior: ...", "expect": "compiles, same return values"}
{"id": "long-02", "prompt": "Given the 8k-token log below, find the first error: ...", "expect": "identifies line 1142"}

2. The runner. One file, provider-agnostic via an environment variable so you can point it at any endpoint:

# runner.py — illustrative example, adapt endpoint/payload to your provider
import json, os, time, urllib.request

ENDPOINT = os.environ["MC_ENDPOINT"]   # any OpenAI-compatible chat endpoint
API_KEY  = os.environ.get("MC_API_KEY", "")
MODEL    = os.environ["MC_MODEL"]      # the model id you want to test

def ask(prompt: str) -> tuple[str, float]:
    body = json.dumps({
        "model": MODEL,
        "messages": [{"role": "user", "content": prompt}],
        "temperature": 0,
    }).encode()
    req = urllib.request.Request(
        ENDPOINT, data=body,
        headers={"Content-Type": "application/json",
                 "Authorization": f"Bearer {API_KEY}"})
    t0 = time.time()
    with urllib.request.urlopen(req, timeout=120) as r:
        out = json.load(r)["choices"][0]["message"]["content"]
    return out, time.time() - t0

results = []
for line in open("golden.jsonl"):
    case = json.loads(line)
    try:
        answer, latency = ask(case["prompt"])
        results.append({**case, "answer": answer, "latency_s": round(latency, 2), "error": None})
    except Exception as e:
        results.append({**case, "answer": None, "latency_s": None, "error": str(e)})

with open(f"results-{MODEL.replace('/', '_')}.json", "w") as f:
    json.dump(results, f, indent=2)

Temperature zero matters: you want run-to-run comparability, not creativity. Save raw answers, not just scores — you'll want to read the failures.

3. The scorer. Start manual. With 15 cases, eyeballing pass/fail against your expect notes takes ten minutes and catches things regexes never will (confident tone, wrong-but-plausible code). Once the set grows past ~50 cases, then consider a judge model or scripted checks.

Where to run it for free

The friction in this workflow used to be access: new releases land on hosted endpoints first, and spinning up a GPU box to self-host a fresh checkpoint is a weekend project, not a 30-minute eval.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

This is where MonkeyCode is a practical fit. It offers free model access and a free server option, so you can point the runner above at a newly released open-weight model without paying for a hosted tier or provisioning hardware. Because the harness only needs an endpoint and a model id, swapping in the trending release of the week is an environment-variable change — which is exactly what an eval harness should demand of its infrastructure.

There's also a cultural point worth making. The reason this harness works at all is the open-weight ecosystem: teams publishing checkpoints the community can inspect, host, and benchmark independently. Tooling that lowers the cost of trying open models — free access tiers, free servers, standard endpoints — is part of that same open-source spirit. Openness only compounds if people can actually run the models, not just read the model cards.

A decision table for "should we switch?"

After the run, score the new model against your incumbent on four axes:

Axis	Question	Weight
Accuracy on golden set	Pass count vs. incumbent	High
Failure shape	Are failures loud (refusals, errors) or silent (plausible wrong answers)?	High
Latency on your longest case	Within your UX budget?	Medium
Ops cost	Endpoint stability, migration effort, lock-in risk	Medium

Silent failures are the real killer. A model that's wrong 20% of the time but obviously wrong is often safer in production than one that's wrong 5% of the time with total confidence.

Limitations, honestly

Fifteen prompts is a smoke test, not a benchmark. It catches regressions and deal-breakers; it cannot rank two close models. Treat a pass as "worth a real pilot," not "ship it."
Free tiers change. Free model access and free server availability are what's offered today; quotas, model catalogs, and duration can shift, so re-verify before building a recurring pipeline on them.
Judge-model scoring is circular if the judge is the model under test or its sibling. Keep a human in the loop for the golden set.

Who should skip this

If you're in a regulated environment where outputs need formal validation, a hobby harness is the wrong tool entirely. If you don't yet have 10 real prompts from production, fix that first — the golden set is the asset; the runner is disposable. And if your current model already passes everything and nothing in the new release addresses a pain you actually have, the correct eval result is do nothing.

If you do run it, I'd be curious what your golden set looks like — the shape of people's test cases tends to reveal more about their product than any benchmark does. And if access cost is what's been stopping you from trying new open releases, MonkeyCode's free model and free server options are an easy first endpoint to point the runner at.