Learn Why Free Tokens Can Miss Your Deadline by Building a Tiny Budget Probe

Free tokens are not the same as a usable endpoint.

My last automation passed the auth check and still missed the deadline. The problem was not cost. It was latency, truncated output, and a missing finish_reason check.

Learning question: Can I predict whether a free model endpoint stays inside my timeout and token budget before I wire it into a pipeline?

This is one probe with one built-in failure. It uses only the Python standard library.

What you need

• Python 3.10 or newer
• No third-party packages
• An OpenAI-compatible chat/completions endpoint, or the small mock server below

The contract I am testing

Most model endpoints accept this shape:

{
  "model": "mock-llm",
  "messages": [{"role": "user", "content": "..."}],
  "max_tokens": 64,
  "temperature": 0
}

They return a choices array and a usage object. I care about four fields:

• elapsed_s from my own timer
• completion_tokens
• total_tokens
• finish_reason

finish_reason: "length" means the model hit the cap. It did not finish naturally.

Step 1: A mock server for a reproducible setup

First, create mock_llm.py. It simulates a base latency plus a small per-token delay. This lets the probe fail in a predictable way without needing a real key.

from http.server import BaseHTTPRequestHandler, HTTPServer
import json
import time

class Handler(BaseHTTPRequestHandler):
    def do_POST(self):
        length = int(self.headers.get("Content-Length", 0))
        body = self.rfile.read(length)
        try:
            request = json.loads(body)
        except json.JSONDecodeError:
            self.send_response(400)
            self.end_headers()
            return

        cap = int(request.get("max_tokens", 1))
        time.sleep(0.2 + 0.01 * cap)

        payload = {
            "id": "mock-1",
            "object": "chat.completion",
            "created": int(time.time()),
            "choices": [{
                "index": 0,
                "message": {"role": "assistant", "content": "ok"},
                "finish_reason": "length",
            }],
            "usage": {
                "prompt_tokens": 12,
                "completion_tokens": cap,
                "total_tokens": 12 + cap,
            },
        }
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.end_headers()
        self.wfile.write(json.dumps(payload).encode())

    def log_message(self, fmt, *args):
        print("mock:", fmt % args)

if __name__ == "__main__":
    print("mock listening on :8010")
    HTTPServer(("127.0.0.1", 8010), Handler).serve_forever()

Run it in one terminal:

python mock_llm.py

Step 2: The probe

Now create budget_probe.py. It sends the same prompt at three token caps and records the response.

import json
import os
import time
import urllib.error
import urllib.request

BASE = os.environ.get("LLM_BASE", "http://127.0.0.1:8010/v1")
KEY = os.environ.get("LLM_KEY", "demo")
MODEL = os.environ.get("LLM_MODEL", "mock-llm")

def call(max_tokens):
    payload = {
        "model": MODEL,
        "messages": [
            {"role": "user", "content": "Return one short JSON object only."}
        ],
        "max_tokens": max_tokens,
        "temperature": 0,
    }
    data = json.dumps(payload).encode()
    url = f"{BASE.rstrip('/')}/chat/completions"
    request = urllib.request.Request(
        url,
        data=data,
        headers={
            "Content-Type": "application/json",
            "Authorization": f"Bearer {KEY}",
        },
        method="POST",
    )

    started = time.perf_counter()
    try:
        with urllib.request.urlopen(request, timeout=5) as response:
            raw = response.read()
    except urllib.error.URLError as exc:
        return {"error": str(exc), "elapsed_s": round(time.perf_counter() - started, 3)}

    elapsed = time.perf_counter() - started
    try:
        parsed = json.loads(raw)
        return {
            "elapsed_s": round(elapsed, 3),
            "prompt_tokens": parsed["usage"]["prompt_tokens"],
            "completion_tokens": parsed["usage"]["completion_tokens"],
            "total_tokens": parsed["usage"]["total_tokens"],
            "finish_reason": parsed["choices"][0]["finish_reason"],
        }
    except (KeyError, IndexError, json.JSONDecodeError) as exc:
        return {"error": f"bad payload: {exc}", "elapsed_s": round(elapsed, 3)}

for cap in [1, 64, 256]:
    print(cap, call(cap))

Run it in a second terminal:

export LLM_BASE=http://127.0.0.1:8010/v1
export LLM_KEY=demo
python budget_probe.py

Expected output:

1 {'elapsed_s': 0.202, 'prompt_tokens': 12, 'completion_tokens': 1, 'total_tokens': 13, 'finish_reason': 'length'}
64 {'elapsed_s': 0.844, 'prompt_tokens': 12, 'completion_tokens': 64, 'total_tokens': 76, 'finish_reason': 'length'}
256 {'elapsed_s': 2.803, 'prompt_tokens': 12, 'completion_tokens': 256, 'total_tokens': 268, 'finish_reason': 'length'}

The numbers are small on purpose. They show the relationship I need to see: higher token caps mean more latency.

Step 3: One error input

Change the loop in budget_probe.py to:

for cap in [1, 64, 1000]:
    print(cap, call(cap))

Run it again. The mock server sleeps for 0.2 + 0.01 * 1000 = 10.2 seconds, but the probe has a 5-second timeout. The third call fails.

1000 {'error': '...timed out...', 'elapsed_s': 5.001}

This is the failure mode I wanted to find before automating a real task. A free token allowance would not save this request.

What I should understand after this

• Auth success is not endpoint usability.
• Free tokens reduce cost, not waiting time.
• A large max_tokens cap can turn a useful call into a timeout.
• finish_reason: "length" means the output was cut off, not completed.
• I need to measure latency and token usage, not just read a status code.

Common mistakes

• Forgetting to set a timeout. Default network waits can hide a bad endpoint for a long time.
• Ignoring finish_reason. A truncated completion may still look useful.
• Comparing raw token price while ignoring latency and retries.
• Assuming a free tier behaves the same as a paid production endpoint.

Where a free endpoint fits this workflow

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The same probe can be pointed at any OpenAI-compatible endpoint. MonkeyCode is described by its operator as an open-source project with free model access and a free server option. The operator also advertises a 30 million token allowance at the time of writing. I treat that as an operator-supplied number, not a permanent guarantee. Limits and model names change, so verify the current project page before relying on them.

A practical test sequence for a free endpoint would be:

1. Point LLM_BASE and LLM_MODEL at the endpoint from the current docs.
2. Run the probe at 1, 64, and 256 tokens.
3. Record elapsed_s and finish_reason.
4. Compare usage from the response with your own request cost.
5. Break it with a larger cap to find the timeout wall.

This tells me whether the free tier is a usable lab resource or only a lightweight trial.

Who should not use this approach

• People who only need model quality and already have a paid endpoint with known SLOs.
• Production systems that require uptime, latency guarantees, or safety evaluation.
• Workflows that depend on a specific model identity. This probe tests the HTTP contract, not model behavior.

Extension exercise

Add retry logic and measure p50 and p95 latency over 20 runs. Then change the prompt length and watch prompt_tokens and elapsed time change. If you test the MonkeyCode free endpoint, share the finish_reason you see at 256 tokens. A controlled number is more useful than another screenshot of a chat window.