Jordan Huang

Posted on Sep 2

Rate-Limited or Truncated? A Five-Minute Free-Tier Triage

#llm #python #api #testing

Your LLM call stalls at 14 seconds.

You blame rate limits. You raise concurrency. Your quota burns faster.

That pattern is common. The diagnosis is often wrong. I have wasted free-tier quota the same way.

This post is a five-minute triage. It uses one small script. It separates rate limits from truncation.

The myth: every slow call is a throttle

Free tiers fail quietly. Paid tiers fail loudly. Silent failures hide the real cause.

Three counters get mixed into one blob:

Counter	Tracks	Typical status
Rate limit	requests per second	HTTP 429
Quota	tokens per window	HTTP 400 or 429
Generation	content length	HTTP 200, truncated body

Rate limits clamp frequency. Quotas cap spend. Generation stops when output is cut short. Label them apart before blaming one.

The triage script

Save this as triage_free.py. Pass the endpoint, model ID, and optional token as arguments.

#!/usr/bin/env python3
# triage_free.py - classify a free-tier LLM failure
import json
import sys
import time
import urllib.error
import urllib.request

URL = sys.argv[1]   # e.g. https://provider.example/v1/chat/completions
MODEL = sys.argv[2] # your model id
AUTH = 'Bearer ' + sys.argv[3] if len(sys.argv) > 3 else None

def call(payload):
    body = json.dumps(payload).encode()
    headers = {'Content-Type': 'application/json'}
    if AUTH:
        headers['Authorization'] = AUTH
    request = urllib.request.Request(URL, data=body, headers=headers, method='POST')
    start = time.monotonic()
    try:
        with urllib.request.urlopen(request, timeout=30) as response:
            data = response.read().decode()
            return response.status, data, time.monotonic() - start
    except urllib.error.HTTPError as error:
        return error.code, error.read().decode(), time.monotonic() - start

probe = {
    'model': MODEL,
    'messages': [{'role': 'user', 'content': 'Reply with the word pong.'}],
    'max_tokens': 8,
}

status, data, elapsed = call(probe)
print(f'probe 1 -> HTTP {status} in {elapsed:.2f}s')
print(data[:200])

probe['max_tokens'] = 1000000
status, data, elapsed = call(probe)
print(f'probe 2 -> HTTP {status} in {elapsed:.2f}s')
print(data[:200])

del probe['max_tokens']
status, data, elapsed = call(probe)
print(f'probe 3 -> HTTP {status} in {elapsed:.2f}s')
print(data[:200])

Run it once per symptom. Then compare the three lines.

Verify the probe locally first

Run the script against a stub before touching any vendor. A stub cannot prove vendor behavior. It can prove your client logic works.

# stub for local verification
def fake_provider(payload):
    if payload.get('max_tokens', 0) > 1000:
        return 400, '{"error":"max_tokens_limit"}', 0.05
    return 200, '{"choices":[{"finish_reason":"length"}]}', 0.05

The stub catches most client bugs in seconds. Vendor reality still needs a live probe.

Sample live run

This output is synthetic. Do not treat it as vendor behavior.

probe 1 -> HTTP 200 in 1.91s
usage: 9 tokens

probe 2 -> HTTP 400 in 0.18s
error: max_tokens_limit

probe 3 -> HTTP 200 in 22.80s
finish_reason: length

Probe 1 checks reachability. Probe 2 checks validation speed. Probe 3 checks generation behavior. Compare them. The story appears on the third line.

How to read the answers

A fast 400 means a parameter was rejected. That is not a throttle.

A slow 200 means generation is working too slowly or was cut off. Look for finish_reason.

Observation	Old reflex	Correct read	Next move
Fast HTTP 400	model is down	parameter rejected	fix the payload
Slow HTTP 200	heavy load	output truncated	raise limit or shorten prompt
Intermittent 429	code is broken	budget consumed	back off and wait
Single 5xx	panic	transient glitch	one retry, five seconds later

finish_reason: "stop" means the model finished cleanly. finish_reason: "length" means your limit ended the output. content_filter has its own payload. Read it before changing the client.

Read Retry-After before retrying

retry-after: 32

That header is an instruction. It says wait 32 seconds. Blind retries make the next 429 more likely. The body often names the quota field. The header names the wait. Read both fields.

The corrected mental model

Stop reading every failure as a server error. Separate capacity, cost, and content.

Capacity lives in HTTP status codes.
Cost lives in usage fields and headers.
Content lives in finish_reason and message length.

The three layers are independent. One layer can fail while the others stay healthy. A tiny probe tells you which layer is broken. You then spend effort on the right fix.

What the old reflex costs

The old reflex costs more than quota. It costs signal. Every blind retry rewrites your logs. Every re-request hides the first response. You lose the evidence you need later. That evidence is the only thing support can act on.

Where I run this workflow

This triage works on any HTTP endpoint. I practice it under the free model access and the free server option from MonkeyCode. Small probes run first. Real code runs after the probe says something useful.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Free tiers rarely promise stable quotas. Do not treat free access as an infrastructure guarantee. Treat it as an early signal. The final environment still needs its own measured baseline.

Who should not use this approach

This script expects HTTP POST and JSON responses. gRPC and streaming sockets need a different probe. Provider SDKs add an abstraction layer that blurs status codes. Client timeouts also hide server work. If your call path buffers a full response, only the final status will reach you.

Use this only for raw HTTP endpoints. Apply the same discipline anywhere else. The probe changes, the mental model stays.

Closing

Free models are rarely the problem. Guessing is the problem. Burning quota is the bill.

Next time an endpoint stalls: classify before retrying. Run the three probes. Then decide.

Your quota will thank you.

DEV Community