Everyone is arguing about whether model output can be detected and whether agents should be allowed to touch tools. While that argument raged, I spent an afternoon with a much smaller problem: a 30,000,000-token grant that told me nothing about what one request actually costs. The number looked generous. The missing part was the ledger. Without a ledger, a free grant is just a number with no unit of work attached.
MonkeyCode is an open-source project that currently lists free model access with a 30,000,000-token grant and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I wanted to test those claims at the only layer a student like me can reliably inspect: the API contract.
A token grant behaves like a water meter installed by the same company selling the water. I do not need the meter to be exact. I need to detect when the needle jumps between identical pipes. My question is narrow: can a free endpoint hide real cost by counting tokens differently than a simple local estimate, or by failing at the contract layer instead of the capacity layer? I built a tiny Python ledger to find out.
Prerequisites
You need Python 3.12 and a terminal. No third-party packages are required; the client uses urllib.request, and the fake server uses http.server. The whole experiment is two files.
Mock server: a free endpoint I control
Open one terminal and run this fake server first.
from http.server import BaseHTTPRequestHandler, HTTPServer
import json
class Handler(BaseHTTPRequestHandler):
def do_POST(self):
length = int(self.headers.get('Content-Length', 0))
body = self.rfile.read(length)
try:
payload = json.loads(body)
except Exception:
self.send_response(400)
self.end_headers()
return
messages = payload.get('messages', [])
prompt = messages[-1].get('content', '') if messages else ''
prompt_tokens = max(1, len(prompt) // 4)
if 'force_error' in prompt:
self.send_response(429)
self.send_header('Content-Type', 'application/json')
self.end_headers()
self.wfile.write(json.dumps({'error': {'message': 'rate limit'}}).encode())
return
self.send_response(200)
self.send_header('Content-Type', 'application/json')
self.end_headers()
response = {
'choices': [{'message': {'content': 'ok'}}],
'usage': {
'prompt_tokens': prompt_tokens + 3,
'completion_tokens': 5,
'total_tokens': prompt_tokens + 8
}
}
self.wfile.write(json.dumps(response).encode())
def log_message(self, *args):
pass
if __name__ == '__main__':
HTTPServer(('127.0.0.1', 8000), Handler).serve_forever()
The mock is deliberately boring. It estimates prompt tokens with a crude len(prompt) // 4 heuristic, then adds 3 tokens to the reported prompt count. That is not a faithful tokenizer. It is a fake provider whose contract I control, so I can see two failure modes a free endpoint often hides: over-reported usage and a 429 on a harmless prompt.
My local estimate is not a tokenizer. It is a yardstick. If a provider reports prompt tokens wildly higher than len(prompt) // 4 for short prompts, that does not prove cheating. It does tell me to stop treating the grant as an abstract number and start recording a baseline before I build any real evaluation.
Ledger probe: compare expected and reported usage
Open a second terminal and run the ledger.
import json
import os
import time
import urllib.error
import urllib.request
BASE = os.environ.get('BASE_URL', 'http://127.0.0.1:8000/v1/chat/completions')
PROMPTS = [
'Summarize one concept.',
'Summarize one concept with more words.',
'force_error: this request should fail.'
]
def expected_tokens(prompt):
return max(1, len(prompt) // 4)
def send_one(prompt):
body = json.dumps({'model': 'local-mock', 'messages': [{'role': 'user', 'content': prompt}]}).encode()
request = urllib.request.Request(BASE, data=body, headers={'Content-Type': 'application/json'})
started = time.perf_counter()
try:
with urllib.request.urlopen(request, timeout=5) as response:
raw = response.read()
data = json.loads(raw)
usage = data.get('usage', {})
return {
'ok': True,
'status': response.status,
'prompt': prompt,
'latency_s': round(time.perf_counter() - started, 4),
'expected': expected_tokens(prompt),
'reported_prompt': usage.get('prompt_tokens', 0),
'reported_completion': usage.get('completion_tokens', 0),
'reported_total': usage.get('total_tokens', 0),
'mismatch': usage.get('prompt_tokens', 0) - expected_tokens(prompt)
}
except urllib.error.HTTPError as error:
return {'ok': False, 'status': error.code, 'prompt': prompt, 'latency_s': round(time.perf_counter() - started, 4), 'expected': expected_tokens(prompt)}
except Exception as error:
return {'ok': False, 'status': None, 'prompt': prompt, 'latency_s': round(time.perf_counter() - started, 4), 'expected': expected_tokens(prompt)}
for prompt in PROMPTS:
print(send_one(prompt))
Expected output looks close to this; exact latency will vary.
{'ok': True, 'status': 200, 'prompt': 'Summarize one concept.', 'latency_s': 0.0011, 'expected': 5, 'reported_prompt': 8, 'reported_completion': 5, 'reported_total': 13, 'mismatch': 3}
{'ok': True, 'status': 200, 'prompt': 'Summarize one concept with more words.', 'latency_s': 0.0008, 'expected': 9, 'reported_prompt': 12, 'reported_completion': 5, 'reported_total': 17, 'mismatch': 3}
{'ok': False, 'status': 429, 'prompt': 'force_error: this request should fail.', 'latency_s': 0.0012, 'expected': 9}
Look at the third row. The request never reached a model; the contract failed first with 429. The first two rows returned 200, but each over-reported prompt tokens by 3. If I multiply that small mismatch across a real project, the grant shrinks faster than my local estimate predicted. A free grant is not a benchmark because the unit of work keeps moving.
Trusting the usage field is tempting because it arrives in the same JSON as the answer. But usage is itself part of the response. A buggy proxy, a caching layer, or a misconfigured server can alter it. My tiny ledger makes that field observable, not authoritative.
Trying it against MonkeyCode's free server
To try the same probe against MonkeyCode's free server, set BASE_URL to its chat-completions path and change the model value from local-mock to the name the project tells you to use. Keep the local estimate rough. It is a smoke test for obvious contract drift, not a token auditor. The free server option is exactly where this matters: a server can accept your request, return quickly, and still spend your grant faster than expected if reported usage is inflated or the error path is noisy. With a paid endpoint, a surprise cost eventually shows up on a bill. With a free grant, the surprise is silent until the grant stops accepting work, usually at the worst point in a semester project.
Thirty million tokens sounds like a lot. It is a lot if an average task is a few thousand tokens. It is not a benchmark for whether an agent pipeline can survive a week of tool retries. The only way to know how far it goes is to run small, repeatable requests and record the cost of each one, including the failures. After running this, you should understand that a free token grant is a spending allowance, not a measure of work. A ledger row needs at least three numbers: what you expected, what the endpoint reported, and whether the request failed before it got to a model. A 200 response can be expensive.
Common mistakes
Common mistakes include equating tokens with characters, ignoring reported usage because the endpoint returned 200, and treating a 429 as a capacity problem when it may be a contract problem. Another mistake is testing against a live free endpoint before you have a local mock that reproduces the failure. Start fake, then go real.
Extension exercise
Change the mock server to subtract 2 tokens from some requests and add a second error path that returns 200 with an empty usage object. Which failure is harder for the ledger to catch? If you adapt the script to MonkeyCode's free server, capture the same three rows and compare them with the fake ones.
If you try this against MonkeyCode's free server, I'd like to see the ledger rows where it breaks.
Top comments (0)