I Tested 4 LLM APIs for Coding Work — All 20 Runs Passed, But Speed Varies 10x

I tested 3 free LLM API providers and one paid Claude plan on the same 5 real coding tasks. All 20 runs passed on first attempt.

The story isn't whether the free tiers work — they do. The story is that the free tier landscape in 2026 has shifted hard. What worked 12 months ago (Cerebras Llama 3.3, free OpenRouter Llama 3.3 70B) now requires a credit card or has been deprecated. What works today is different — and faster.

I started this on a Saturday because the conversation about Cerebras' new card requirement was getting more heat than the data. Felt like the right time to actually measure what works in 2026, not just argue about it. Here's the data, the methodology, and the things I didn't expect.

What I tested

I wanted to know three things:

1. Do the most-cited "free LLM API" providers actually work for real coding-agent work in 2026?
2. How does the quality compare to paid Claude?
3. Where do the rate limits and quotas actually bite?

To find out, I tried to test 7 free-tier providers. Of those 7, two required a credit card on file (Cerebras and the Z.ai free tier), two had model IDs that returned 404s on the day I tested, and one (ClinePass) turned out to be a paid aggregator, not a free tier. That left 3 genuinely free providers plus 1 paid control:

Provider	Model	Cost	Card required?
NVIDIA NIM	nvidia/nemotron-3-super-120b-a12b	Free	No
Groq	qwen/qwen3.6-27b	Free	Yes (free tier)
OpenRouter	minimax/minimax-m3:free	Free	No
Cline API	anthropic/claude-fable-5.1	Paid	Yes

The Cline API is a single-billed aggregator that routes to Anthropic, OpenAI, and Google behind one key. It's not free, but it's cheaper than calling Anthropic direct. I used it as the "what does paid Claude look like" baseline.

The 5 tasks

The task suite covers the four parallelism buckets from real coding-agent work:

• csv-header-infer — read a headerless CSV, infer column names from the data
• function-docstring — add a Google-style docstring to a Python function
• function-unit-test — add unit tests covering happy path + edge cases
• refactor-api-call — swap one API method for another, keep tests green
• cli-flag — add a --verbose flag to a CLI command

Each task runs in a fresh git worktree branched from main. I read the current file contents into the prompt, told the model "output each modified file in a file:path code block", parsed the response, wrote the new files, and ran the task's verifier (existing test suite plus a content check). Pass means the verifier exits 0 on first attempt — no human edit, no retry.

20 (provider, task) pairs. 1 trial each. ~5 minutes wall time per provider on the network.

The headline: all 4 passed, all 5 tasks

That's the result. No provider failed any task. The difference shows up in speed, not correctness.

Provider	Pass	Mean wall	Mean tok/s	Mean TTFT	Cost
OpenRouter MiniMax M3	5/5	2.9 s	178	1.4 s	Free
Cline API Claude	5/5	7.4 s	279	4.1 s	Paid
Groq Qwen 3.6 27B	5/5	8.8 s	511	5.8 s	Free
NVIDIA NIM Nemotron 120B	5/5	27.0 s	409	23.0 s	Free

Read the table right-to-left on speed and you get the story:

• OpenRouter is the surprise winner. 2.9-second mean wall time is fast enough for interactive coding-agent work. The underlying model (MiniMax M3) is a smaller open-weight model, but routed through OpenRouter's free tier it's the most responsive option here.
• Cline API Claude is the gold standard for paid. 7.4-second mean is 2.5x slower than OpenRouter, but the quality is Anthropic-grade. For a small team that can pay a bit, this is the cleanest answer.
• Groq has the highest sustained throughput. 511 tok/s during generation. The catch: a 5.8-second mean TTFT because of one long-thinking block. Groq is fast when it's flowing, slow when it isn't.
• NVIDIA NIM is the slowest free tier. 27-second mean wall time, 23-second time-to-first-token. The Nemotron 120B model uses heavy thinking tokens. For batch-async work, fine. For interactive use, brutal.

Per-task wall time (seconds)

Task	NVIDIA NIM	Groq	OpenRouter	Cline API
csv-header-infer	27.1	1.6	2.3	5.2
function-docstring	14.5	14.6	2.1	7.2
function-unit-test	16.9	4.0	4.8	10.4
refactor-api-call	14.6	2.7	2.5	6.5
cli-flag	61.7	21.1	2.7	7.9
Mean	27.0	8.8	2.9	7.4

The cli-flag task is the most striking. NVIDIA took 61.7 seconds (a long thinking block on a multi-file change); OpenRouter finished in 2.7. Same prompt, same task, 23x speed difference.

Things I didn't expect

The free tier landscape has changed hard in 12 months. I went in expecting to test 7 free providers. The 4 left standing are different from what every "free LLMs" listicle I read recommended 12 months ago. Llama 3.3 70B on Cerebras used to be the default recommendation; it now requires a card. The free OpenRouter :free pool rotates through different models and rate-limits unpredictably. If you copy-paste a 2024 "best free LLMs" list, half the entries no longer work the same way.

All 4 free providers passed all 5 tasks. I expected at least one failure. I did not get one. At this task complexity (real engineering work, not research-grade), the model capability gap has narrowed to style, not correctness.

The Cline API is the dark-horse recommendation. I had not heard of it before this test. I went in expecting to compare free tiers to Anthropic direct — I had not realized there was a single-billed aggregator routing to Anthropic, OpenAI, and Google behind one key, with prices below direct Anthropic. For a small team that wants a paid baseline without managing per-provider API keys, this is the cleanest answer. I'll be using it for the team benchmarks going forward.

The 1.8x speed difference matters more than the price difference. At a personal scale, the $0 vs ~$0.012/task price gap is rounding error. The 2.5x to 9x speed difference is the real signal. For an interactive coding agent, 3 seconds feels responsive and 27 seconds feels broken.

What this means for your coding agent

Situation	First pick	Fallback	Why
Personal coding, lowest latency	OpenRouter	Groq	2.5s P50, surprisingly good
Personal coding, best raw quality	Cline API	Groq	Claude is the gold standard; Groq is the closest free
No credit card on file	NVIDIA NIM	OpenRouter	Both work without card; NVIDIA is slow but high quality
5+ engineers, team coverage	Cline API	+ Groq fallback	Free tiers throttle under team load
Building on top of Claude Code	Cline API	—	Same Anthropic models, single key, lower price

For everyone else: the free tiers work today. Don't assume "free" means "broken" — the providers in this test all produced working code on the first try. The honest ceiling right now is "responsive enough for personal work, not for team load."

Build it yourself

The benchmark is in the public repo:

git clone github.com:Pitambarmahato/hardnumbers-experiments
cd hardnumbers-experiments/free-tier-llm-coders
python -m venv .venv
.venv/bin/pip install -r requirements.txt
echo "NVIDIA_NIM_API_KEY=nvapi-..." >> .env
echo "GROQ_API_KEY=gsk_..." >> .env
echo "OPENROUTER_API_KEY=sk-or-..." >> .env
echo "CLINEPASS_API_KEY=sk_..." >> .env
chmod 600 .env
.venv/bin/python src/benchmark.py --providers all

The full 20-run benchmark takes about 5 minutes. Results land in results/free_tier_<timestamp>.json with the wall time, tokens, rate-limit hits, and pass/fail for each pair.

What I didn't test

A few things I didn't measure and would change the recommendation:

• Tool use and function calling. The 5 tasks here are all text-only file rewrites. Real coding agents issue dozens of tool calls per session. A v2 benchmark should drive the OpenAI tool-call protocol end-to-end.
• Long context. Tasks here fit in 4K tokens of context. Real sessions routinely hit 50K-200K. Free providers vary in long-context support.
• Sustained multi-day load. 20 runs is a snapshot. To know whether free tiers sustain a team's workload, you'd need 200+ requests per day for a week.
• Multilingual work. All tasks are English. Several free providers (Qwen, MiniMax) are tuned for Chinese. Multilingual coding work might rank them differently.
• Claude Code end-to-end. I ran this through the OpenAI SDK pointed directly at the provider APIs, not through Claude Code's interactive REPL. The numbers here are about model quality, not the agent loop.

What's next

For personal work, OpenRouter is the surprise winner and I'll keep using it. For team work, the Cline API is the cleanest paid answer and I'll be testing it across more workflows.

The next benchmark in this series is going to be multi-agent throughput — 1 vs 2 vs 4 vs 8 parallel agents on the same task suite. The story there is whether parallelization gives you 4x speedup or whether tightly coupled work punishes you for adding agents. That article drops next week.

If you have API keys for free LLM providers I missed, open a PR on the experiments repo with the new provider — the benchmark is designed to be extended.

The full data, methodology, and what-didn't-test section is at hardnumbers.dev/articles/4-free-llm-apis-vs-claude-5-coding-tasks-real-data — that's the canonical version with the full breakdown, including rate-limit hit analysis and the Cline API vs direct Anthropic cost comparison.