Learn Why Free Token Grants Still Queue by Building a Tiny Concurrency Probe

A 30,000,000 token grant is a budget, not a concurrency guarantee.

Last week's AI threads on DEV kept circling around watermarking and whether generated text can be detected. That is a useful debate, but I keep tripping on a more boring problem first: before I trust a free model API, I need to know what happens when two requests arrive at the same time.

Free tokens hide two failure modes:

• 429 rate-limit responses when capacity is full
• long queue delays when the shared server is busy

This article builds a tiny concurrency probe so you can see that failure before you wire a free tier into a real client.

What you will build

A single Python file that:

• runs a local mock API with a three-request concurrency limit
• sends eight identical requests at the same time
• prints status counts and per-request latency

You will see why a big credit balance can still return five failures from eight requests.

Prerequisites

• Python 3.10+ (I used 3.11)
• no third-party packages
• about fifteen minutes

The tiny probe

Copy this into concurrency_probe.py and run it.

import json
import threading
import time
from concurrent.futures import ThreadPoolExecutor, as_completed
from http.server import BaseHTTPRequestHandler, HTTPServer
from urllib import request, error


class Handler(BaseHTTPRequestHandler):
    lock = threading.Lock()
    active = 0

    def do_POST(self):
        length = int(self.headers.get('Content-Length', 0))
        self.rfile.read(length)

        with Handler.lock:
            Handler.active += 1
            current = Handler.active

        if current > 3:
            with Handler.lock:
                Handler.active -= 1
            self._send(429, {'error': 'queue_full'})
            return

        # Simulate a shared server doing real work.
        time.sleep(0.2)

        with Handler.lock:
            Handler.active -= 1

        self._send(200, {'ok': True})

    def _send(self, code, payload):
        body = json.dumps(payload).encode()
        self.send_response(code)
        self.send_header('Content-Type', 'application/json')
        self.send_header('Content-Length', str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, *args):
        pass


def start_mock():
    server = HTTPServer(('127.0.0.1', 9000), Handler)
    thread = threading.Thread(target=server.serve_forever, daemon=True)
    thread.start()
    return server


def probe():
    url = 'http://127.0.0.1:9000/v1/chat/completions'
    payload = json.dumps({'prompt': 'hi'}).encode()
    barrier = threading.Barrier(8)

    def call(n):
        barrier.wait()
        req = request.Request(url, data=payload, method='POST')
        started = time.perf_counter()

        try:
            with request.urlopen(req, timeout=2) as response:
                return n, response.status, round(time.perf_counter() - started, 2)
        except error.HTTPError as e:
            return n, e.code, round(time.perf_counter() - started, 2)
        except Exception as e:
            return n, type(e).__name__, round(time.perf_counter() - started, 2)

    results = []
    with ThreadPoolExecutor(max_workers=8) as pool:
        futures = [pool.submit(call, i) for i in range(8)]
        for future in as_completed(futures):
            results.append(future.result())

    statuses = [status for _, status, _ in results]
    print('status counts:', {code: statuses.count(code) for code in set(statuses)})
    for row in sorted(results):
        print(row)


if __name__ == '__main__':
    server = start_mock()
    time.sleep(0.1)
    probe()
    server.shutdown()

Expected output

The exact order can change, but the counts should be stable:

status counts: {200: 3, 429: 5}
(0, 200, 0.2)
(1, 200, 0.2)
(2, 200, 0.21)
(3, 429, 0.0)
(4, 429, 0.0)
(5, 429, 0.0)
(6, 429, 0.0)
(7, 429, 0.0)

Three requests pass. Five are rejected immediately, even though every caller had the same credentials and enough token budget.

What the output means

The probe separates two budgets that are easy to confuse:

• credit budget: how many tokens you may consume over time
• concurrency budget: how many requests may run at once

A free token grant can give you a very large credit budget while still enforcing a very small concurrency budget. The 429 is not proof that your key is invalid. It is the server saying that capacity is full right now.

Why this matters before you trust a free tier

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source project that currently advertises a free model API, a free server option, and a 30,000,000 token grant. I am treating that number and the free-server availability as operator-supplied; free-tier limits change, so I do not want to bake them into a client as if they were permanent.

The useful engineering habit is:

1. read the quota page for the credit budget
2. run a concurrency probe for the real request budget
3. keep those two numbers in separate places in your config

If the project is open source, inspect how the server path enforces limits. That tells you more than the marketing page.

How a free server option changes the bottleneck

A hosted free model tier usually limits concurrency on the provider side.

A free server option moves part of the queue onto infrastructure you control. The same 429 probe can then tell you whether the bottleneck is:

• the provider's upstream model limit
• your own server's worker count
• your network timeout

You do not need to know those details before coding. You just need to keep the probe parameterized so you can point it at either endpoint.

Common mistakes

• testing only one request and concluding the API is healthy
• using unbounded concurrency and blaming the provider for 429s
• treating a 429 as a permanent outage instead of a retry signal
• believing a large token balance means infinite parallel throughput

Who should skip this

Skip this if:

• you only send one prompt at a time
• you have a paid plan with a documented concurrency SLA
• you are benchmarking model quality rather than request capacity

This is not a production load test. It is a small mental model for the difference between credits and concurrency.

Extension exercise

Change the client to cap max_workers at 3 and add exponential backoff when the response is 429.

Expected result:

• no request should fail permanently
• total runtime increases because queued work retries after the active requests finish

Then try max_workers=6. Explain why the failure rate returns even though your code handles retries.

What you should understand after completing this

• a free token grant is a budget, not a concurrency guarantee
• 429 is a capacity signal, not proof of broken credentials
• bounded retry logic matters more than a large credit balance
• a local mock server is enough to learn provider-independent queue behaviour

If you have a trial token grant, run the local probe first. Then point it at the real endpoint and compare the two result sets before you build a client around it.