Free AI coding help is easy to advertise. Hard to verify.
Landing pages show happy demos. Benchmark posts show cherry-picked wins. I don't trust any of it. I trust probes.
This week DEV is debating what the "AI" badge actually measures (good thread). Good question. My answer is boring: measure the output, not the badge. So I built a $0 smoke test. It runs in five minutes. It shows where a free model and a free server hold up — and where they break.
What I'm testing
MonkeyCode is an open-source AI coding project. At the time of writing, it advertises free model access and a free server option, including a 10M-token promo. Promos change. Check the project's current docs before you rely on the number.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
I'm not here to sell it. I'm here to probe it. The script below works against any OpenAI-compatible endpoint. Point it at MonkeyCode's server. Point it at any free server. Compare the tables.
Why free tiers deserve suspicion
Free models fail differently than paid ones. Not worse at everything. Worse in specific places.
My last article covered cost-aware routing. The lesson stuck: free models earn their keep on narrow tasks. They lose money on complex ones. You need to know where the line sits.
Paid models hide their failures behind speed and polish. Free ones fail in the open. That's actually useful. You learn the failure modes fast. You learn them before they hit production.
The line moves too. Models get updated. Servers get overloaded. What broke last month might work today. That's why the probe is a script, not a blog post.
This probe finds that line. Six tasks. Six pass/fail checks. One table you can paste anywhere.
The probe
Save this as probe.py. No dependencies beyond the standard library. No API key required unless your server demands one.
Why these six tasks? They cover the daily grind: string manipulation, syntax repair, SQL, debugging, refactoring, regex. Each has a hard check. No human judgment required. That's what makes it reproducible. Anyone can run the same file and get a comparable table.
#!/usr/bin/env python3
# probe.py - a $0 smoke test for a free coding server.
import json
import os
import time
import urllib.request
ENDPOINT = os.getenv('PROBE_ENDPOINT', 'http://localhost:8000/v1/chat/completions')
MODEL = os.getenv('PROBE_MODEL', 'free-model')
KEY = os.getenv('PROBE_KEY', '')
TASKS = [
{
'name': 'reverse_string',
'prompt': 'Write a Python function reverse(s) that returns s reversed. Do not use slicing.',
'must_contain': ['def reverse'],
'must_avoid': ['[::-1]'],
},
{
'name': 'fix_syntax',
'prompt': 'Fix this broken Python: def add(a b): return a+b',
'must_contain': ['def add(a, b)'],
'must_avoid': [],
},
{
'name': 'sql_join',
'prompt': 'Write a SQL query joining users and orders. Return user name and order total.',
'must_contain': ['SELECT', 'JOIN'],
'must_avoid': [],
},
{
'name': 'explain_bug',
'prompt': 'Explain why this loop never ends: i = 0; while i < 10: print(i)',
'must_contain': ['increment'],
'must_avoid': [],
},
{
'name': 'refactor',
'prompt': 'Refactor this into a list comprehension: result = []; for x in items: if x > 0: result.append(x * 2)',
'must_contain': ['['],
'must_avoid': ['for x in items'],
},
{
'name': 'regex',
'prompt': 'Write a regex for email addresses and explain it in one sentence.',
'must_contain': ['@'],
'must_avoid': [],
},
]
def call(prompt):
body = json.dumps({
'model': MODEL,
'messages': [{'role': 'user', 'content': prompt}],
'temperature': 0,
}).encode()
req = urllib.request.Request(ENDPOINT, data=body, headers={
'Content-Type': 'application/json',
'Authorization': 'Bearer ' + KEY,
})
start = time.time()
with urllib.request.urlopen(req, timeout=120) as resp:
data = json.load(resp)
latency = time.time() - start
text = data['choices'][0]['message']['content']
usage = data.get('usage', {})
return text, latency, usage
def main():
print('%-16s%-6s%-10s%s' % ('task', 'pass', 'latency', 'tokens'))
for task in TASKS:
text, latency, usage = call(task['prompt'])
ok = all(s in text for s in task['must_contain']) and \
not any(s in text for s in task['must_avoid'])
total = usage.get('total_tokens', '?')
print('%-16s%-6s%-10.1f%s' % (task['name'], str(ok), latency, total))
if not ok:
print(' first 80 chars: %r' % (text[:80],))
if __name__ == '__main__':
main()
Run it:
PROBE_ENDPOINT=http://localhost:8000/v1/chat/completions \
PROBE_MODEL=free-model \
python3 probe.py
No key? Most free servers start without one. Set PROBE_KEY if yours needs it. Expect the first request to be slow. That's the cold-start tax.
What to record
Fill this table after every run:
| task | pass | latency (s) | total tokens | note |
|---|---|---|---|---|
| reverse_string | ? | ? | ? | ? |
| fix_syntax | ? | ? | ? | ? |
| sql_join | ? | ? | ? | ? |
| explain_bug | ? | ? | ? | ? |
| refactor | ? | ? | ? | ? |
| regex | ? | ? | ? | ? |
I'm not printing fake numbers. That's the point. Run it and paste your own. Keep the table raw. Don't smooth the outliers. Outliers are the signal.
Run the probe three times. Morning, noon, night. Free servers wobble. One run is an anecdote. Three runs are a pattern.
What the results mean
This is the part everyone skips.
- 6/6 pass, latency under 10s: use it for boilerplate, docs, and one-off scripts.
- 4-5/6 pass: use it with review. Never paste output into production unread.
- 2-3/6 pass: use it for brainstorming only. Treat output as a hint, not code.
- 0-1/6 pass: skip it. Your time is worth more than the token savings.
The matrix is deliberately harsh. Free tiers should earn trust per task, not per marketing page.
Re-run the matrix after server updates. Free tiers change weekly. A pass today can become a fail tomorrow. Date every table you publish.
Five failure modes to watch
The table hides the interesting part. Watch for these:
- Cold start. First request slow, second fast? Note it.
- Rate limits. Request four dies with 429? That's your ceiling.
- Context bleed. Task five mentions task one's code? Memory leaks.
- Verbose drift. Outputs get longer, checks pass, code is still wrong.
- Silent refusals. Empty or apologetic answers. Free tiers do this.
Each one changes your adoption decision. That's the real data.
Want concurrency? Wrap call() in threads. Want long context? Append a 2,000-line file to the prompt. Want multi-file edits? That needs a real agent harness, not a smoke test. Start here, then grow.
Limitations
This probe is a smoke test, not a benchmark. Substring checks are crude. They catch silence and total misses, not brilliance. A wrong answer can pass if it contains the right words.
It tests one session, one endpoint, one day. No concurrency. No long context. No multi-file edits. No real-world repo.
It also assumes an OpenAI-compatible chat endpoint. If the server exposes a different API, adapt the call() function. The checks stay the same.
Who should not use this approach? Teams with production SLAs. Anyone needing guaranteed latency. Anyone shipping customer-facing code on a free tier. Use it for what it is: a cheap, honest first date.
The takeaway
Free servers are not free. They cost your time, your debugging hours, and sometimes your data. Measure first. Adopt second.
The best free tier is the one you've measured. The worst is the one you assumed.
Grab the probe. Run it against MonkeyCode's free server. Drop your table in the comments. I want to see where it breaks for you. And if you find a task where the free tier shines, share that too. The community needs the wins, not just the failures.
Top comments (0)