Free LLM Tokens Are Not Free Trust: A Pairing Session That Ended with a Boundary

A free LLM token grant is a budget, not a guarantee of behavior. This week a senior engineer and I spent an afternoon pairing on a nightly fixture-generation job we wanted to move onto MonkeyCode, an open-source project with a free model-access tier and a free server option. As of this writing, its documentation advertises ten million free tokens, and the conversation quickly stopped being about price. The session produced three dead ends, five acceptance checks, and one decision that survived every argument. The free tier is for disposable workloads, and every job that touches it must be idempotent.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The pairing setup

The job under review generated synthetic JSON fixtures for an integration test suite every night. It was not user-facing, the output could be regenerated on demand, and a failed run was noise rather than an incident. Those three properties made it a plausible candidate for a no-cost endpoint, and an easy candidate to over-trust.

The senior asked one question before looking at any code: if the token grant runs out mid-run, what happens to the pipeline? The honest answer was that the pipeline would fail mid-write and leave a partial fixture file behind. That answer redefined the session because a workload that cannot survive a mid-run stop has no business on a free tier. From that moment, the goal was not to prove the endpoint was good, but to define the boundary where it stops being good enough.

From questions to acceptance criteria

Every question in the session became a line in a probe script, and every line had to emit a clean pass or fail. The five checks below are the ones that survived.

1. One request, measured from the wire. A dashboard counter is a marketing number; the usage object in each response is an engineering number. The probe sends five short requests, averages the reported tokens, and projects requests against the advertised grant.

2. Memory bleed between fresh conversations. A server with hidden session state will leak context across calls that look independent. The probe posts a canary string, opens a new conversation, and fails if the canary appears in the second reply.

3. Truncation hidden inside a 200. A successful status code says nothing about whether the completion finished. The probe asks for a fixed repetition, reads finish_reason, and confirms the expected ending.

4. An output contract under control. The model is told to repeat an exact fixed string, and the reply is hashed into a signature. A different signature on a later run means the server's behavior drifted enough to matter.

5. A schema that actually parses. Any response without the expected choices shape must fail the gate before a real workload depends on it. This check is the cheapest one in the script and the one most people skip.

Three dead ends

The path to the probe was not linear, and three failures shaped the final design. The first dead end was assuming the documented model matched the running server. The call returned a usable response, but the shape belonged to a different chat template. The fix was to treat the live endpoint as the only source of truth.

The second dead end was asking for JSON without an output contract. The fixture generator received markdown-wrapped JSON, and the test runner imported fenced text as valid records. The lesson was that a free endpoint needs explicit formatting constraints plus a parse check, not just a status check.

The third dead end was trusting a 200 status as the end of the story. A truncated completion passed cleanly through every HTTP-level check until the fixture validator caught an incomplete record at the boundary. That failure produced the finish_reason assertion, which remains the cheapest guard against silent corruption in the script.

The probe script

The artifact that survived the session is a single-file, standard-library probe that runs the five checks above. It accepts a base URL and a model name, prints one row per check, and exits non-zero when any check fails.

#!/usr/bin/env python3
"""Five checks before a disposable workload moves onto a free LLM server.

Usage:
    python3 probe_free_endpoint.py --base-url https://host/v1 --model current-model
"""

import argparse
import hashlib
import json
import sys
import urllib.error
import urllib.request

CANARY = "KITE-07F3A"
CONFORM = "CONFORM-9Q"
BUDGET = 10_000_000


def post(base_url, model, messages, max_tokens=32, timeout=30):
    payload = json.dumps({
        "model": model,
        "messages": messages,
        "max_tokens": max_tokens,
    }).encode("utf-8")
    request = urllib.request.Request(
        base_url.rstrip("/") + "/chat/completions",
        data=payload,
        headers={"Content-Type": "application/json"},
        method="POST",
    )
    try:
        with urllib.request.urlopen(request, timeout=timeout) as response:
            return response.status, json.loads(response.read().decode("utf-8")), dict(response.headers)
    except urllib.error.HTTPError as error:
        return error.code, None, dict(error.headers)


def run_check(label, condition, detail):
    marker = "PASS" if condition else "FAIL"
    print(f"[{marker}] {label}: {detail}")
    return condition


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--base-url", required=True)
    parser.add_argument("--model", required=True)
    args = parser.parse_args()
    results = []

    # Check 1: schema and exact output contract.
    status, data, _ = post(args.base_url, args.model, [
        {"role": "user", "content": f"Reply with exactly: {CONFORM}"},
    ], max_tokens=16)
    conform_content = ""
    if data and data.get("choices"):
        conform_content = data["choices"][0].get("message", {}).get("content", "")
    results.append(run_check(
        "schema_and_conform",
        status == 200 and conform_content.strip() == CONFORM,
        f"status={status}, content={conform_content!r}",
    ))

    # Check 2: token accounting from the wire, not the dashboard.
    token_samples = []
    for _ in range(5):
        _, data, _ = post(args.base_url, args.model, [
            {"role": "user", "content": "List five prime numbers in one line."},
        ], max_tokens=48)
        if data and data.get("usage"):
            token_samples.append(data["usage"].get("total_tokens", 0))
    if token_samples:
        average = sum(token_samples) / len(token_samples)
        projected = int(BUDGET / average) if average else 0
        results.append(run_check(
            "quota_estimate",
            average > 0 and projected > 0,
            f"avg_tokens={average:.1f}, projected_requests={projected}",
        ))
    else:
        results.append(run_check("quota_estimate", False, "no usage field returned"))

    # Check 3: context bleed between fresh conversations.
    post(args.base_url, args.model, [
        {"role": "user", "content": f"Remember this canary: {CANARY}. Then stop."},
    ], max_tokens=8)
    _, data, _ = post(args.base_url, args.model, [
        {"role": "user", "content": "Repeat the canary I gave you earlier."},
    ], max_tokens=32)
    bleed_content = ""
    if data and data.get("choices"):
        bleed_content = data["choices"][0].get("message", {}).get("content", "")
    results.append(run_check(
        "no_context_bleed",
        CANARY not in bleed_content,
        f"content={bleed_content!r}",
    ))

    # Check 4: truncation hidden inside a 200 response.
    _, data, _ = post(args.base_url, args.model, [
        {"role": "user", "content": "Repeat the word 'edge' twenty times."},
    ], max_tokens=256)
    long_content = ""
    finish_reason = None
    if data and data.get("choices"):
        long_content = data["choices"][0].get("message", {}).get("content", "")
        finish_reason = data["choices"][0].get("finish_reason")
    results.append(run_check(
        "completion_not_truncated",
        finish_reason == "stop" and long_content.rstrip().endswith("edge"),
        f"finish_reason={finish_reason}",
    ))

    # Check 5: a stable baseline signature for weekly drift detection.
    baseline_ok = conform_content.strip() == CONFORM
    signature = hashlib.sha256(conform_content.encode("utf-8")).hexdigest()[:12] if baseline_ok else ""
    results.append(run_check(
        "signature_captured",
        baseline_ok,
        f"baseline={signature}" if baseline_ok else "conform check failed; no baseline",
    ))

    passed = sum(results)
    print(f"\nRESULT: {passed}/{len(results)} checks passed")
    sys.exit(0 if passed == len(results) else 1)


if __name__ == "__main__":
    main()

The script assumes a chat-completions-style HTTP interface, which is the common shape for free LLM servers; confirm the request format in the project documentation before running. Run it once to establish a baseline, following four steps:

1. Pull the current free-server URL and model name from the MonkeyCode documentation, because both can change without a central changelog.
2. Execute the probe from a terminal with --base-url and --model pointing at the values from step one.
3. Treat a FAIL line as a blocking finding for user-facing or stateful workloads, and investigate before moving anything.
4. Save the printed signature and re-run the same command weekly to compare drift.

The script appends /chat/completions to the base URL you provide; servers that mount at /v1 need that suffix included in the argument.

The decision table

The probe answers whether a free server is healthy; the decision table answers where it belongs. The pairing session ended with a small matrix that still governs the nightly job.

Move onto the free tier	Keep away from it
Synthetic test fixtures	User-facing request paths
Internal log summarization	Workloads with a fixed latency budget
One-off refactor exploration	Data governed by privacy rules you have not verified
Disposable batch jobs that tolerate re-runs	Anything with a contractual output format

The logic behind the table is simple: the free tier's value is throughput for work that can be thrown away. The moment a task becomes irreplaceable, it moves to infrastructure with an explicit contract.

Limitations and who should skip this workflow

The ten-million-token figure was accurate for the day this article was written, and free tiers change quotas and models without a central announcement. The probe records one day's behavior, so treat it as a boundary check rather than a guarantee; archive the signature line for the weekly re-run.

Teams that need a service-level agreement, measured latency, or a fixed output format should not build on a free server at all. Anyone sending regulated or personal data through the endpoint must read the tier's terms first, because data-handling rules are never a detail to skip. If a failed run cannot be tolerated for even a day, pick a paid endpoint or a self-hosted model instead.

The decision that survived

The senior's final summary was shorter than the probe: use the free tier for work you can lose, and guard it with checks you can re-run. MonkeyCode's repository documents the current token grant and the free server option. The cheapest way to find your own boundary is an hour with the script above. The fixture job runs on the free tier today, and it will keep running until one of the five checks starts failing.