DEV Community: Vilius

The End of the US Cloud Monopoly: AI Balkanization Is Here to Stay

Vilius — Sat, 13 Jun 2026 08:28:42 +0000

By Vilius Vystartas | June 2026

The single, globally unified internet is gone. What's replacing it is a patchwork of sovereign AI zones, each running its own stack on its own hardware with its own rules.

This isn't a prediction. It's already happening, and the next three years will cement it.

What Broke the Monopoly

The US government's approach to AI regulation — treating frontier model weights as controlled munitions — had an unintended consequence. By demonstrating that access to US cloud infrastructure can be switched off by regulatory decree, they forced every non-US government and enterprise to build a backup plan.

The January 2025 AI Diffusion Rule created a three-tier world: unrestricted allies, capped nations (50,000 GPUs/year), and total embargoes. For the 140+ countries in Tier 2, US cloud services became inherently unstable. You can't build a national AI strategy on a faucet that might turn off.

The DeepSeek R1 moment in January 2025 proved the point: a Chinese quant hedge fund trained a frontier reasoning model on nerfed hardware for $5.6 million. Export controls didn't stop the frontier. They just accelerated the development of independent stacks.

The Three Zones

The tech industry is splitting into three distinct legal and architectural zones, each with its own economics:

The US Zone — high-performance, high-surveillance closed models. OpenAI, Anthropic, Google. Restricted to US citizens and close Tier 1 allies. The best models, the most monitoring, the least legal recourse if you're outside its borders.

The European Zone — regulated, open-source-first, locally hosted. Data privacy is the architecture, not a compliance checkbox. France's Mistral, Germany's Aleph Alpha, the fragmented but determined GAIA-X federation. GDPR compliance isn't overhead — it's the product.

The Asian/Non-Western Zone — independent stacks operating entirely outside the Western financial and regulatory sphere. DeepSeek, Alibaba's Qwen, Baidu's Ernie. Huawei Ascend chips replacing NVIDIA. No US venture capital, no US cloud, no US export license risk.

The Sovereign AI Infrastructure Boom

Every non-US government with ambition is building national AI infrastructure. Not optional. Existential.

Europe is scattering exascale systems across the continent — Germany's JUPITER, Finland's LUMI, Italy's LEONARDO. Federated, fragmented, deliberately not dependent on any single provider.

The Middle East is placing bigger bets. Saudi Arabia's $40 billion AI fund. UAE's G42 building Condor Galaxy on Cerebras hardware, then selling a $1.5 billion stake to Microsoft — on condition it cut Chinese ties. The message: even your sovereign compute comes with geopolitical strings.

India's $1.25 billion IndiaAI Mission aims for 10,000+ GPUs through Yotta's Shakti Cloud. But at Tier 2's 50,000 GPU cap, the ambition outstrips the allocation.

Japan's SoftBank committed nearly a billion to AI datacenters. ABCI 3.0 is operational with H100 clusters.

Every single one runs some form of open-weight model. Because closed APIs can be switched off at a whim.

Open Weights Won

The question of whether enterprises would choose open-source models over closed APIs is settled. They will.

Meta's Llama 3.1 405B proved open models can match GPT-4 class performance. Mistral proved European sovereignty models are commercially viable. DeepSeek proved frontier reasoning can be open. The entire ecosystem shifted from "can open models compete?" to "how do we productionize our chosen open model?"

The calculus is simple: slightly lower benchmark scores in exchange for complete operational certainty. No API key to revoke. No pricing change that breaks your margin. No geopolitical event that cuts your access.

This has driven massive innovation in on-device and on-premises deployment. Models under 70 billion parameters — often under 10 billion — that run on corporate hardware rather than centralized server farms. Microsoft's Phi-4, Apple's on-device models, Google's Gemma, Meta's Llama 3.2 small variants. The edge is where the action is.

The Death of the API Wrapper

The venture capital correction is brutal but predictable. Startups whose entire value proposition was piping data to a third-party US API can't raise international funding. Their core product can be wiped out overnight by a single regulatory pen stroke — or a pricing change, or a model deprecation, or a geopolitical event.

Jasper went from $1.5 billion valuation to significant layoffs. The entire "GPT wrapper" category is being reframed as a 2023-2024 anomaly.

The new valuation premium isn't about who uses the flashiest model. It's about who owns the proprietary training data to build independent, in-house models. Palantir's stock surge, BloombergGPT's financial data moat, healthcare AI companies valued on unique patient datasets — the market is betting on data ownership, not API access.

What Nobody's Saying About the Hardware

This entire scenario depends on one unresolved bottleneck: TSMC manufactures over 90% of advanced AI chips.

The US CHIPS Act ($52.7 billion), the European Chips Act (€43 billion), and TSMC's own global fab expansion (Arizona 4nm, Japan operational, Germany planned) are all trying to address this. But fabrication takes years. The RISC-V ecosystem is promising but a decade behind CUDA in maturity and tooling.

The real risk isn't that US export controls will stop frontier AI development. DeepSeek proved they won't. The risk is that the hardware supply chain itself becomes a weapon — and every sovereign AI zone discovers that independence at the architectural level means nothing without independence at the fab level.

The 3-5 Year Outlook

The trajectory is clear:

US cloud APIs become a premium product for US-aligned customers only
Open-weight models become the global enterprise default
National datacenters proliferate in every region that can afford them
Data ownership replaces model access as the primary valuation driver
The supply chain question remains the unresolved wildcard

This isn't a temporary fragmentation that heals with better policy. It's a permanent structural shift. The unified global technology ecosystem that defined the last two decades is over. The question isn't whether the balkanization happens — it's whether your infrastructure is ready for it.

We Asked 10 LLMs to Write Efficient Code. Only 4 Got Better.

Vilius — Tue, 26 May 2026 22:46:48 +0000

By Vilius Vystartas | May 2026

Every LLM can write code that works. The question is: can they write code that's efficient — and does telling them to be efficient actually help?

I tested 10 models on 10 coding tasks, each in two phases: unprompted (the model writes its own code) and prompted (explicitly told to write clean, DRY, efficient code). That's 200 API calls, $0.56 total. The results are... not what most prompt engineers would predict.

GPT-5.4 was the only model where prompting gave a substantial boost (+0.20). For most models, the "write efficient code" prompt was meaningless or actively harmful.

How the Metric Works

Each task has a known optimal token budget — the minimum tokens needed to produce correct, DRY code for that task (e.g., 70 tokens for 10 styled buttons using CSS classes vs 340 tokens for 10 separate button blocks). The efficiency score is optimal_tokens / actual_tokens, capped at 1.0.

A score of 0.63 means the model used about 1.6x the optimal — not bad. A score of 0.43 means it used about 2.3x the optimal. The gap between unprompted and prompted tells you whether the "write efficient code" instruction actually changes behaviour.

The Leaderboard (Sorted by Prompted Efficiency)

#	Model	Unprompted	Prompted	Δ	Frugal	Cost	Correctness
🥇	GPT-5.4	0.43	0.63	+0.20	30%	$0.096	78% → 85%
🥈	Qwen 3.6 Plus	0.44	0.60	+0.17	40%	$0.158	78% → 87%
🥉	Gemma 4 31B	0.54	0.58	+0.04	50%	$0.003	92% both
4	DeepSeek Chat	0.51	0.55	+0.04	30%	$0.006	91% → 80%
5	Claude Sonnet 4	0.47	0.52	+0.04	40%	$0.121	92% both
6	LFM 2 24B A2B	0.54	0.47	-0.06	30%	$0.001	90% → 80%
7	Mistral Large 2411	0.54	0.46	-0.08	40%	$0.050	90% → 82%
8	Gemini 2.5 Flash	0.47	0.46	-0.01	50%	$0.020	92% → 90%
9	Cohere Command A	0.60	0.44	-0.17	40%	$0.071	90% → 82%
10	Kimi K2.6	0.34	0.43	+0.09	30%	$0.029	76% → 86%

What Stands Out

GPT-5.4 Is the Prompt Whisperer

GPT-5.4 improved on 7 of 10 tasks when prompted for efficiency. The biggest wins were config-generation (+0.81 — went from 12 inline JSON blocks to a template loop), html-from-data (+0.71), and magic-strings (+0.38 — switched to an Enum). It's the only model in the batch where the "write efficient code" instruction consistently produces different (and better) output.

The cost is notable — $0.10 for 20 tasks is mid-range, not cheap, not expensive. But the efficiency gain is real.

Gemma 4 31B: The Quiet Winner

Half of Gemma 4's tasks were already "frugal" — naturally efficient without being told. It scored 92% correctness on both phases at just $0.003 total. That's a 40x cost advantage over GPT-5.4 with higher correctness and competitive efficiency. For high-volume production where you want concise, correct code, Gemma 4 31B is the value pick of this batch.

Cohere Command A: Prompting Backfires

Cohere Command A had the highest unprompted efficiency in the batch (0.60) — it naturally writes concise code. But when told "write efficient code," it ballooned output on several tasks. html-from-data went from a tight 45-token solution to a 600+-token monstrosity (-0.92 gap). The prompt made it overthink.

Lesson: if a model is already efficient, don't prompt it to be more efficient.

Qwen 3.6 Plus: Second Place, Slowest

Qwen 3.6 Plus scored second in prompted efficiency (+0.17 improvement) but took 26 minutes for 20 tasks — by far the slowest model. The efficiency gain is real (especially on html-from-data where it went from hardcoded rows to a map/join pattern), but you're waiting for it. Batch workloads only.

The Kimi Surprise

Kimi K2.6 had the lowest unprompted efficiency (0.34 — verbose, boilerplate-heavy code) but improved the most at the bottom end (+0.09). Still last place, but the prompt actually helped it compress — which is the opposite of the Cohere effect. Some models need the nudge.

Frugality: What Does It Mean?

"Frugal" means the model naturally produced code at or near the optimal token count without being asked. Gemma 4 31B and Gemini 2.5 Flash led at 50% — half their tasks were already efficient. GPT-5.4, DeepSeek Chat, and Kimi K2.6 were only 30% frugal — they needed the prompt to tighten up.

The Bigger Picture

Group	Models	Behaviour
Prompt-responsive	GPT-5.4, Qwen 3.6 Plus	Efficiency improves substantially with prompting
Prompt-neutral	Gemma 4 31B, DeepSeek Chat, Claude Sonnet 4, Gemini 2.5 Flash, Kimi K2.6	Prompt has little effect (±0.04)
Prompt-antagonistic	LFM 2 24B A2B, Mistral Large 2411, Cohere Command A	Efficiency drops when prompted

The prompt-antagonistic group is the most interesting. These models know how to write efficient code (0.54-0.60 unprompted), but the explicit instruction triggers over-engineering — they add abstractions, comments, error handling, and other bloat that makes the output less efficient by the metric.

If the prompt says "write efficient code" and the model responds by writing more tokens, something in the training signal is misaligned.

My Picks

Best prompted efficiency: GPT-5.4 — 0.63, $0.10 for 20 tasks. The only model where prompting reliably improves output.
Best value overall: Gemma 4 31B — 0.58 prompted, 92% correctness, $0.003. Absurd price/performance.
Best natural efficiency: Cohere Command A — 0.60 unprompted. Don't prompt it, just let it work.
Most consistent: Claude Sonnet 4 — 92% correctness on both phases, small +0.04 efficiency gain. Reliable.
Skip if you're in a hurry: Qwen 3.6 Plus — 26 minutes for 20 tasks. Great efficiency gains, terrible latency.
Watch list: Kimi K2.6 — low base efficiency but the prompt actually helps. Worth retesting with a better prompt.

Methodology

Ten real-world coding tasks across CSS, JavaScript, Python, SQL, and bash — each with a known optimal token budget for a correct, DRY solution. Tasks included: styling 10 buttons (CSS), rendering 20 data rows as HTML (JS/HTML), bulk renaming (shell), form validation (Python), parametrized tests (Python), unit conversion (Python), SQL reporting queries, config generation (JSON), magic string replacement (Python/Enum), and middleware decorator pattern (Python/Flask).

Each model ran 10 tasks unprompted, then the same 10 tasks with an efficiency prompt appended. Scoring: efficiency_ratio = optimal_tokens / actual_tokens (capped at 1.0). Correctness scored against expected output patterns.

Total cost: $0.56 for 200 API calls (10 models × 10 tasks × 2 phases). Temperature: 0.1. Max tokens: 600.

Full results: benchmarks.workswithagents.dev

10 Models Tested: From 81.6% to 10%. The Free Tier is a Full-On Gamble.

Vilius — Tue, 26 May 2026 22:42:59 +0000

By Vilius Vystartas | May 2026

I tested another 10 models across the same 10 agent coding tasks. Four of them were free-tier models — and the range was absurd: Owl Alpha scored 76.7% with zero hard fails, Laguna M.1 scored 10% and produced garbage on 9 out of 10 tasks. The free tier is not free if it costs you debugging time.

Total cost for all 10 models: $0.10. The paid models (6 of 10) came to $0.10 combined.

Batch 12 Leaderboard

#	Model	Score	P/P/F	Cost	Time	Category
🥇	Grok 4.3	81.6%	7/3/0	$0.017	39.9s	Paid (xAI)
🥈	Perceptron Mk1	79.9%	8/1/1	$0.002	29.3s	Paid (Perceptron)
🥉	Owl Alpha (free)	76.7%	5/5/0	Free	83.0s	Free tier
4	xAI: Grok Build 0.1	75.0%	5/4/1	$0.034	95.3s	Paid (xAI)
5	OpenAI: GPT Chat Latest	73.3%	6/2/2	$0.043	18.7s	Paid (OpenAI)
6	Mistral Medium 3.5	71.6%	6/2/2	$0.008	12.6s	Paid (Mistral)
7	Nemotron 3 Nano Omni (free)	50.0%	4/2/4	Free	23.5s	Free tier
8	Laguna XS.2 (free)	49.7%	3/3/4	Free	28.7s	Free tier
9	Baidu CoBuddy (free)	40.0%	4/0/6	Free	362.4s	Free tier
10	Laguna M.1 (free)	10.0%	1/0/9	Free	89.8s	Free tier

The Headlines

Grok 4.3 (81.6%, $0.017, 39.9s) — Grok's latest release takes the batch with zero hard fails. Seven clean passes, three partials. Process-monitor was the only full pass it earned that 4.3's competitors missed. xAI's Grok line is quietly consistent — 4.1 Fast (76.7%), 4.20 (75%), and now 4.3 (81.6%) — all within striking distance of the 80%+ club without crossing into premium pricing.

Perceptron Mk1 (79.9%, $0.002, 29.3s) — A brand new family debuts at nearly 80%, with eight passes — the most in the batch — for two-tenths of a cent. The one failure (regex-extract at 17%) is a known weakness for small models. At this price-to-pass ratio, Perceptron Mk1 is the value story of this batch.

Owl Alpha (free, 76.7%, 83.0s) — A free model with zero hard fails and 5 full passes. That's the standout free-tier result. Takes 2x longer than paid models for some tasks (24s on csv-stats vs 1-3s for the field), but the code is functional. If latency isn't critical, this is usable.

The Free Tier Lottery

Four free models. Results:

Model	Score	Verdict
Owl Alpha	76.7%	Usable — zero hard fails, 5/10 full passes. Slow but functional.
Nemotron 3 Nano Omni	50.0%	Mixed — half of tasks hit output cap at 400 tokens. Hit or miss.
Laguna XS.2	49.7%	Unreliable — 400-token cap kills complex responses.
Baidu CoBuddy	40.0%	Frustrating — 362 seconds total. Half the tasks hit output cap at 399 tokens. Waiting 6 minutes for 40% accuracy is not a good trade.
Laguna M.1	10.0%	Broken — 1/10 passes. Every response capped at 400 tokens. Do not use.

The free tier cap of 399-400 output tokens is the real problem. Models like Laguna M.1 and CoBuddy truncate every response, turning what could be a partial into a fail. Owl Alpha works despite the cap because its outputs are concise enough to fit.

Pay $0.002 for Perceptron Mk1 and get 8/10 passes, or use Laguna M.1 free and get 1/10. The math is not subtle.

Disappointments

GPT Chat Latest (73.3%, $0.043) — OpenAI's catch-all endpoint was solid on easy tasks (file-parse, csv-stats, sql-query all passed) but fell apart on fix-bug (0%) with a lengthy, expensive hallucination. The most expensive model in the batch and it doesn't crack 75%.

Mistral Medium 3.5 (71.6%, $0.008) — Fastest model in the batch at 12.6s total, but the process-monitor task hit a 504 Gateway Timeout and scored 0%. A timeout fail on a model that otherwise looks strong carries a disproportionate penalty — without it, Medium 3.5 would be at 79.5%.

Laguna M.1 (10%) — The worst score in any batch I've run. Seven of its task responses were blank 400-token output cap fills. Not worth listing on OpenRouter.

Price/Performance

Model	Score	Cost	$/%-pt
Owl Alpha (free)	76.7%	$0	$0
Nemotron 3 Nano Omni (free)	50.0%	$0	$0
Laguna XS.2 (free)	49.7%	$0	$0
Baidu CoBuddy (free)	40.0%	$0	$0
Laguna M.1 (free)	10.0%	$0	$0
Perceptron Mk1	79.9%	$0.002	$0.0024
Mistral Medium 3.5	71.6%	$0.008	$0.0108
Grok 4.3	81.6%	$0.017	$0.0209
xAI: Grok Build 0.1	75.0%	$0.034	$0.0450
GPT Chat Latest	73.3%	$0.043	$0.0584

Free models dominate the $/%-pt table by definition, but only Owl Alpha is actually usable. Among paid models, Perceptron Mk1 at $0.0024/%-pt is the efficiency winner — 24x cheaper per point than GPT Chat Latest.

My Picks

Best overall: Grok 4.3 — 81.6%, 39.9s, $0.017. Cleanest leaderboard of the batch.
Best value (paid): Perceptron Mk1 — 79.9%, $0.002 total. Eight passes for two-tenths of a cent.
Best free model: Owl Alpha — 76.7%, zero hard fails. The only free model I'd ship with in production.
Fastest: Mistral Medium 3.5 — 12.6s for all 10 tasks
Skip entirely: Laguna M.1 and all Laguna free-tier variants. 10% is not testable.

Methodology

Same setup as previous batches: ten real-world agent coding tasks — file operations, shell commands, error recovery, data parsing, SQL queries — tested via OpenRouter. Max tokens: 400. Temperature: 0.1. Pattern-matching scoring against expected outputs.

Pre-flight verification caught zero failures this batch. Total cost: $0.10. Total dataset: 168 models tested across cloud and local.

Full results and per-task scores: benchmarks.workswithagents.dev

I Tested 10 More Models. Five Brand New Families Debuted. None Scored Below 75%.

Vilius — Tue, 26 May 2026 18:48:33 +0000

By Vilius Vystartas | May 2026

I ran another 10 models through the same agent coding benchmark. Five of them were from completely untested families — Sao10k, Anthracite, Inflection, Mancer, Undi95 — and every single one scored 75% or higher on its first try. This is getting harder to keep up with.

Two more models tied the all-time record at 90%. The cheapest model ever tested cost $0.0001 for a full 10-task benchmark.

The New 90% Club Members

Eight models have now hit 90% on this benchmark. Batch 11 added two:

Mistral Large 2411 (90%, $0.008, 46s) — Mistral's November 2024 flagship matches their current Large 3. Sometimes the first version is still the best one. Zero hard fails, clean passes on 8/10 tasks.

DeepSeek Chat V3-0324 (90%, $0.002, 73s) — The older V3 variant from March 2024 matches the original DeepSeek Chat at 90%. Every time I test a DeepSeek variant, it lands at 80-90%. The family is remarkably consistent.

The 90% club now includes: DeepSeek Chat (original), DeepSeek Chat V3-0324, Qwen3 Coder 30B, Nemotron 3 Nano 30B, Codestral 2508, Mistral Large 2411, MiniMax M2 Her, and Baidu Ernie 4.5 300B. Eight models. Seven of them cost less than a cent per full benchmark.

Five Families, First Try

Every new family debuted at 75% or higher. That's an impressive hit rate.

Family	Model	Score	Cost	Time
Sao10k	L3.1 Euryale 70B	85%	$0.002	29s
Sao10k	L3 Lunaris 8B	85%	$0.0001	20s
Anthracite	Magnum V4 72B	85%	$0.006	35s
Mancer	Weaver	80%	$0.003	30s
Undi95	Remm Slerp L2 13B	75%	$0.002	31s
Inflection	Inflection 3 Productivity	75%	$0.012	42s

*Inflection 3 result is provisional — awaiting lab response. Will update in due course.

L3 Lunaris 8B at $0.0001 is the cheapest model I've ever tested. A full 10-task benchmark for one ten-thousandth of a dollar. At this price, there's no reason not to test a model before you ship with it. Lunaris scored 85% — competitive with models that cost 100x more.

The Sao10k family (L3.1 Euryale 70B and L3 Lunaris 8B) is the standout. Both models scored 85%, both are fine-tunes of Llama 3.1/3, and both cost almost nothing. Community fine-tunes continue to punch above their weight.

The Recoveries

Two Qwen models from my previous failed batch completed successfully this time:

Qwen3 8B (80%, $0.02, 543s) — Needed per_call_timeout: 300 to finish. The model is competent (6 passes, 4 partials, zero fails) but painfully slow. Each API call takes 100-120 seconds on OpenRouter. Use it as a background job, not a real-time agent.

Qwen Plus 2025-07-28 (80%, $0.001, 19s) — The dated variant works perfectly with enable_thinking: false. 80% at $0.0009 is great value. But use the current qwen/qwen-plus ID instead — it scores 85% and doesn't need the dated suffix.

Price/Performance

Model	Score	Cost	$/%-pt
L3 Lunaris 8B	85%	$0.0001	$0.0001
DeepSeek Chat V3-0324	90%	$0.002	$0.0017
L3.1 Euryale 70B	85%	$0.002	$0.0021
Remm Slerp L2 13B	75%	$0.002	$0.0020
Mancer Weaver	80%	$0.003	$0.0041
Anthracite Magnum V4 72B	85%	$0.006	$0.0066
Mistral Large 2411	90%	$0.008	$0.0093
Inflection 3 Productivity	75%	$0.012	$0.0156
Qwen3 8B	80%	$0.020	$0.0254

The ratio between cheapest and most expensive $/%-pt is 254x. Lunaris at $0.0001/%-pt vs Qwen3 8B at $0.0254/%-pt — same tier of score, wildly different cost profiles.

My Picks

Best overall: Mistral Large 2411 — 90%, 46s, $0.008
Best value: L3 Lunaris 8B — 85%, $0.0001 total. Absurd price/performance.
Best new family debut: Sao10k — both models at 85% first try. Watch this line.
Fastest: L3 Lunaris 8B — 20 seconds for all 10 tasks

Methodology

Same setup as the previous 10 batches: ten real-world agent coding tasks — file operations, shell commands, error recovery, data parsing, SQL queries — tested via OpenRouter. Max tokens: 600 (Qwen models), 300 (everyone else). Temperature: 0.1. Pattern-matching scoring against expected outputs.

Pre-flight verification caught zero failures this batch. All 10 candidates passed the simple-prompt test. Total cost: $0.05 for the core 8 models, then $0.02 for the Qwen recovery run. Total dataset: 158 models tested across cloud and local.

Full results and per-task scores: benchmarks.workswithagents.dev

Two Models Just Hit 90% on Agent Coding. One Cost Less Than a Penny.

Vilius — Tue, 26 May 2026 09:46:02 +0000

By Vilius Vystartas | May 2026

Ten more models through the same 10 agent coding tasks. Two tied the all-time record. One cost $0.0002. The other hit the score at $0.0018 — cheaper than most models scoring 70%.

Batch 10 was the cheapest one yet.

The Leaders

Two models scored 90% with zero hard fails, joining MiniMax M2 Her and Baidu Ernie 4.5 300B as the highest-scoring models on this benchmark:

Qwen3 Coder 30B A3B — 90% in 28 seconds, $0.0004. An efficient coder that doesn't burn budget on thinking tokens it doesn't need.

DeepSeek Chat (original) — 90% in 59 seconds, $0.0018. The original DeepSeek Chat still competes with modern models on agent coding. Newer doesn't always mean better.

The Surprises

LFM 2 24B A2B (85%, $0.0002, 15s) is the cheapest model I've ever tested. Liquid's debut family is absurdly cost-effective. A full 10-task benchmark for literally $0.0002. At this price/performance ratio, there's no excuse not to test a model before committing to a more expensive alternative.

Mistral Small 3.2 (85%, $0.0004) is a clear upgrade. The Small line went 75% → 85% across versions — a ten-point jump at the same budget tier. Mistral keeps improving the right things.

Qwen3 14B scored 0% across all 10 tasks. Mandatory thinking mode that can't be suppressed at 300 tokens means every request times out before producing output. Skip for agent coding.

Cydonia 24B V4.1 (80%, $0.001) debuts a new family from TheDrummer. Zero hard fails. Watch this one.

The Duds

Qwen3.7 Max (85%, $0.13, 295 seconds) scored the same as budget models costing 300x less. Thinking mode tax at work — the accuracy is there, but you'll wait five minutes and pay for every second.

Claude Opus 4 (80%, $0.10, 76s) had one hard fail. For a top-tier premium model at $0.10 per 10 tasks, that's below expectations. It's not a bad model — it's overkill for agent coding at a tight token budget.

Aion 1.0 (80%) had two hard fails and was the slowest at 160 seconds. The architecture is interesting, but it's not ready for production agent work.

My Picks

Best overall: Qwen3 Coder 30B A3B — 90%, 28s, $0.0004
Best value: LFM 2 24B A2B — 85%, $0.0002 total. Ridiculous price/performance.
Fastest: LFM 2 24B A2B — 15 seconds flat
Most improved: Mistral Small 3.2 — 75% → 85% across versions
Skip entirely: Qwen3 14B for agent tasks

Methodology

Ten real-world agent coding tasks — file operations, shell commands, error recovery, data parsing — tested against each model via OpenRouter. Max tokens: 300. Temperature: 0.1. Results scored by pattern matching against expected outputs. Pre-flight verification caught 2 models (Ernie 4.5 21B — HTTP 429, Trinity Mini — empty content) before they wasted the batch.

Total batch cost: $0.14 across 9 models. Qwen3.7 Max alone accounted for $0.13 of that — thinking tax.

Total models tested: 148 (up from 138).

Full results and per-task scores: benchmarks.workswithagents.dev

Because you should.

The Hype Correction

Vilius — Sat, 23 May 2026 10:47:25 +0000

Weekly roundup, May 23, 2026

Google and Microsoft just told us the same thing from opposite directions.

Google IO this week was an AI firehose. Gemini 3.5 Flash — faster, cheaper, "better at agentic tasks." Gemini Spark — an agent that orchestrates other agents. Omni video gen. The Ultra tier dropped from $250/month to $100. The keynote ran for hours. I watched excerpts. I'm still tired.

And then there's Microsoft, quietly removing the Copilot button from Office 365. Letting users remap the Copilot key on their keyboards. Two companies. Same message. Different volume.

The message is: AI costs real money, and someone has to pay for it.

Google is still spending. The IO presentation felt like a company that hasn't blinked yet — or can't afford to look like it's blinking. But look closer. Free tier usage is now capped by compute, not prompt count. The bean counters are in the room. They're just not on stage.

Microsoft blinked first. The Copilot button that was mandatory hardware is now optional. The Office integration nobody asked for is being rolled back. Not because Copilot doesn't work — because shoving it into every surface made people resent it. Forced adoption. The worst kind.

The WAN Show put it well: "Are the cracks showing that AI might not be what the people want?" The answer was no — but forcing people to use it in ONE particular way is deeply unpopular. People want agency. Even with their AI agents.

Meanwhile, the economics are reshaping faster than the demos.

xAI is renting its entire GPU capacity to Anthropic. $1.25 billion a month. For three years. Elon built the data center without a plan. Anthropic needs the compute. The deal makes sense for both. But it also tells you something about where the money is going — not to training new models. To running them.

Anthropic is projecting its first profitable quarter. That's a bigger milestone than any benchmark score. It means someone figured out how to sell AI without losing money on every inference. Nobody else has done it yet. OpenAI hasn't. Google's AI division hasn't. Anthropic might be first.

The pressing issues nobody's fixed yet:

Agent reliability. Every keynote shows agents doing things. Few of them work reliably outside the demo. Gemini 3.5 Flash claims better agentic performance. I'll believe it when an agent books a haircut without hallucinating the date, the time, and the hairdresser's name. We're still at the stage where the impressive part is that it almost worked.

Context management. Agents forget. Not "forget" in the cute way. Forget in the way where you build a multi-session workflow, and somewhere around turn 40 the agent starts responding to a conversation from three days ago. Memory is still unsolved. Everyone's adding more context. Nobody's figured out what to remove.

The cost of "free." Google's free tier cap change matters. It means the free-usage era is ending — not with an announcement, but with a restriction. More complex prompts hit the cap faster. Try asking an agent to do real work and you'll notice. The meter is running.

The evaluation gap. We have no good way to measure whether an agent is actually better. Perplexity scores are meaningless for multi-turn tool use. Every company has internal evals. None of them agree. The market is pricing models as if better benchmarks mean better agents. They don't.

What I'm watching:

Google's Gemini Spark — the agent orchestrator. The idea of one agent delegating to others isn't new. But Google baking it into Workspace is. If it works, it's the first time an AI demo about "managing your RSVPs" actually saves time instead of creating it. Big if.

The xAI/Anthropic deal. Three years of guaranteed compute at this price is a structural shift. It means Anthropic bet that inference costs will go down — or that they can charge enough to cover them. Either way, it sets a floor on what running agents at scale actually costs.

Microsoft's retreat. Not because Copilot is failing. Because forced adoption is. The lesson: build something people choose to use, or don't build it at all. The Copilot key was a bet that default placement beats user preference. It lost.

The week wasn't about breakthroughs. It was about gravity.

Money, attention, and patience are finite. The AI boom spent two years ignoring that. This week, it started paying attention.

Gravity doesn't break things. It just makes them settle where they belong. The hype cycle is finally in freefall — and for the first time, that's good news. The companies that survive the correction won't be the ones with the best demos. They'll be the ones that figured out how to make this stuff actually work, at a price the market can stomach.

The Agent Told Me To

Vilius — Thu, 21 May 2026 14:29:15 +0000

I have a theory about why agent suggestions land so heavy.

It's not that the suggestions are good. Half of them are terrible — wrong approach, wrong abstraction, wrong thing to build entirely. But they have impact. A colleague says "maybe we should add a reconciliation layer" and you nod and continue. An agent says "a reconciliation layer with idempotency keys and a ledger table" and you're already opening a new repo.

Same suggestion. Different weight.

The confidence gap

Agents don't hedge. They don't trail off. They don't say "I'm not sure about this but." They say "here's what you should build" with the same certainty they use to say "the sky is blue."

A person who sounds that confident is either an expert or a fool. We know which one we're listening to. An agent that sounds that confident could be either — and that ambiguity works in its favour. The articulate wrong answer beats the hesitant right one every time.

Speed as a trap

The agent gave you a file structure, three migration scripts, and a Dockerfile in one response. You can push a PR in ten minutes. The cost of trying is zero.

Except it's not zero. It's attention. It's context. It's the next six weeks of your life explaining to people why this exists. It's the maintenance load of something nobody asked for that now has tests, docs, and a deployment pipeline because it was easier to ship than to evaluate.

The cost of building is never zero. But the agent made the first 80% free, and that's the part your brain sees.

Responsibility diffusion

When you build from your own idea and it fails, that's on you. The whole thing. The meeting where you pitch it, the architecture you chose, the reason it existed.

When you build from the agent's idea and it fails, the failure feels different. You were just executing. The agent suggested it. You followed the recommendation. It's not your fault the recommendation was bad.

This is a lie your brain tells itself to avoid the sting. But it's a convenient lie, and we take it.

Why it keeps happening

Because there's no feedback loop.

The agent doesn't know you deleted its code. It doesn't know that thing you built has zero users. It doesn't feel the maintenance. Next session, when the context is fresh, it will suggest the same pattern again — because the pattern is correct in theory. The theory never experiences reality.

So every fresh session is a new pitch. The same idea, the same confident delivery, the same "this looks right" feeling. The agent doesn't learn. The veto gate has to be you — and we're bad at vetoing confident-sounding things.

What I do now

I started asking questions. Who needs this? Has anyone asked for it? What if I do nothing?

It's not a perfect filter. But it catches the worst ones — the ones where the agent was so articulate that I almost forgot to check whether I wanted the thing.

The Protocol Stack Nobody Talks About

Vilius — Wed, 20 May 2026 10:01:58 +0000

Six agent protocols launched in the last year. Everyone's obsessing over model selection. The operating surface around the model is what actually breaks.

Google I/O opened today with a flood of agent demos. Prompts becoming apps. Vibe coding going production. The spectacle is real. But the thing that determines whether any of this works isn't on stage. It's the quiet protocol stack underneath — MCP, A2A, AGUI, and their contested cousins.

Most teams can tell you which LLM they're using. Almost none can answer: which tools should the agent see? Who else can it delegate to? Where does the human approve or cancel?

Those three questions are the stack. Here's what sits at each layer.

MCP: tools are a security boundary, not a feature toggle

MCP is the most successful agent protocol by far. 14,000 GitHub repos tagged with it. Every major agent platform supports it. An agent connects to an MCP server, gets a list of callable tools, and can actually do work instead of just chatting.

But here's what nobody says out loud: there's no registry. No mcp search. No way for an agent to discover servers programmatically. The 14,000 number is GitHub tag-counting, not a registered directory. Smithery.ai lists about 6,700 — and you browse that with your eyes, not an API. An agent can't ask "find me an MCP server for Salesforce" and get an answer. Discovery is a person reading lists.

That's not a protocol. That's a treasure hunt.

Tool access enables arbitrary code execution and arbitrary data access. MCP was designed for high-trust environments. Now it's everywhere. Invariant Labs has published research on tool poisoning — malicious instructions hidden in tool descriptions that influence agents through the very metadata meant to make tools discoverable.

MCP gets the agent close to the work. It doesn't decide whether the agent should do the work. That's on you.

A2A: coordination isn't free

No single agent does everything. A procurement agent needs a supplier agent. A travel agent needs a hotel agent. A software agent needs a security reviewer. Work is distributed across owners, domains, and expertise.

A2A turns that distribution into something agents can reason about. The agent card is the primitive — a published contract describing what a remote agent is, what it does, which skills it exposes, and how to interact with it.

The cost: coordination adds another surface where latency, failure, permissions, and observability can break. If your agent delegates to another agent, the workflow gets more flexible and less predictable at the same time.

A2A isn't right for every product. A single agent with a small tool set may not need coordination at all. The right question: does this workflow require delegated expertise or authority outside the primary agent?

AGUI: the human control layer nobody builds until it's too late

An agent that's long-running, non-deterministic, and touching external systems needs more than a final answer. Humans need to observe it working, approve sensitive steps, inspect state, understand why it's waiting.

Chatbots don't handle this. Neither do traditional web apps built for request-response.

AGUI is the open candidate for this layer: streaming, shared state, front-end tool calls, custom events, steering, cancellation. It's the protocol most teams will ignore until their agents start doing real work and generating real bugs. They'll wire a model to tools, build a nice chat component, then discover what the agent is really doing — and retroactively bolt on approval buttons, logs, and progress spinners.

None of those are fixes for the root issue: finding the right control points, understanding what the agent is trying to do, and figuring out where the human needs to approve, deny, edit, or cancel.

The three that aren't standards (yet)

A2UI, AP2, and X402 all have real use cases but sit in contested territory.

A2UI is Google's answer to agent-generated interfaces — declarative UI instead of arbitrary HTML. Right direction, narrower scope than the full human control problem.

AP2 and X402 both tackle agent payments. AP2 handles commercial trust and user authorization (60+ collaborators including Mastercard, PayPal, American Express). X402 is Coinbase's HTTP-native machine-to-machine settlement. Payments is the most crowded layer because it's the most valuable. Everyone wants in.

The boring stuff wins

Teams over-focus on model selection and under-specify everything around it. They know which LLM they want. They don't know which tools the agent can or should see. They have a prototype that calls APIs but no interaction model for user approval. They can imagine multiple agents coordinating but have no way to enforce or validate that.

The actual work lives in those questions. The protocol stack isn't glamorous. Neither is infrastructure. But six months from now, the teams that figured out their operating surface will be the ones whose agents still run.

The ones that just picked a model won't know what hit them.

Build It, Then Kill It

Vilius — Tue, 19 May 2026 19:27:36 +0000

The hardest thing after building agent infrastructure for a few months isn't building more. It's stopping.

Noticing you built a tagging system for a knowledge base of 60 entries. A daily cleanup script for things that never get dirty. A schema that has more fields than data points. These aren't failures — the system works fine. They're architecture that arrived before the problem did.

Here's what I've learned about not building things.

Kill the curator before it kills your morning

We had a daily memory curator. Every 4am it scanned the knowledge base for stale entries, article title leaks, bloated facts. It ran for weeks. How many things did it delete? Three. Total. Across its entire lifetime.

Sixty facts don't rot daily. Nothing rots that fast. The curator wasn't maintaining anything — it was creating work to justify its existence. Reports about 32% budget health. Reports about 0 superseded entries found. Reports about reports.

Folded it into the weekly auditor instead. Same coverage, 87% fewer runs.

Your schema is bigger than your data

We built a structured tagging system: every fact needed a domain, a project, a type. domain:infra, project:wwa, type:architecture. Queryable. Organized. Professional.

For 60 facts.

The schema had more rules than the data had entries. We were maintaining metadata for a phone book that fits on a postcard.

Facts are flat again. A source line and content. Tag when you'll actually query by that dimension — not because the schema says to.

Don't bolt on what the model doesn't support

A paper dropped this week: agents communicating via raw embedding vectors instead of text. 8% better, 2.4× faster, 75% fewer tokens. Beautiful piece of research.

Can we implement it? No. This is a model architecture change — a connector between output layers bypassing tokenization. We sit above the model. The interface is text in, text out. No provider exposes raw embeddings.

Sometimes the right answer is "that's not at our layer." Not every good idea is yours to build.

The best infrastructure is what you didn't have to make

The invisible wins:

Not adding a retry loop when the real fix was fixing the thing that fails
Not building a dashboard when the problem is visible enough without one
Not writing a migration when the old data still works fine
Not adding a field to the schema when you can just put it in the content

The conference reaction article published today took 15 minutes to write. Same template every time: what they're right about, what breaks, the real gap. No fleet numbers. No credentials. Just the take.

That's the model. Building less leaves more room for having opinions.

The test

Ask yourself: if nobody maintained this for a month, would anyone notice?

If the answer is no, you didn't build infrastructure. You built busywork.

Google Declared the Agentic Era at I/O. Here's What They Got Wrong.

Vilius — Tue, 19 May 2026 18:50:42 +0000

Google I/O 2026 was an agentic coming-out party. Agent-first IDE. Autonomous debugging. Vibe coding to production. Chrome DevTools for agents. The message was clear: agents aren't a feature anymore, they're the platform.

Great. Here's what the demo doesn't show you.

They're Right About the Direction

Google Antigravity — an agent-first IDE — is overdue. Native IDE support that an agent can drive instead of hacking together terminal sessions and subprocess calls? Finally.

Chrome DevTools for agents is bigger than it sounds. Agents inspecting pages, filling forms, running Lighthouse audits — this solves real infrastructure pain. Browser automation has been brittle for years. Google building this into the platform validates the direction.

Vibe coding as a first-class workflow — fine. The speed of code generation is real and nobody disputes it.

Here's What Actually Breaks

The I/O narrative is "prompt to production." The reality is "prompt to production to 3am incident."

Speed was never the bottleneck. Code generation is fast. What breaks is: the agent didn't check if the endpoint still exists. The docs were stale. The rate limit wasn't documented. The error format changed. The migration ran against the wrong database.

Google's answer is more tools. Antigravity. AI Studio integration. Deployment pipelines. The actual answer is infrastructure — memory so agents don't rediscover the same failure twice, decision protocols so they stop before they break things, verification gates that catch errors before they reach production.

An IDE that generates code without verifying it is just a faster way to break production.

The Real Gap

Google is selling velocity. The market needs reliability.

The impressive demo is an agent building a full-stack app in 30 seconds. The impressive product is an agent running for a week without eating itself.

Agent-first IDEs are finally here. The question isn't whether they can generate code — they can. The question is whether they can run without constant supervision.

Nobody's answered that yet.

A Button That Generates Terrible Prompts

Vilius — Tue, 19 May 2026 18:12:37 +0000

Click it. Get a cursed coding prompt. That's it.

Ten flavors. Fresh AI-generated nonsense every time.

"Sort elements by git blame. Most-blamed floats to the top."
"Validate an email by sending it an email. Timeout: 3 seconds."
"Number database migrations by Fibonacci only. Skip one, it deletes itself."
"CSS-only dark mode. Toggle by resizing your browser to exactly 777px."
"A React form where each input is a separate micro-frontend. One fails, the field silently disappears."
"A pub/sub system where subscribers get events based on open source contributions. First-timers always last."
"A REST API that always returns 200 OK. The real status code is buried somewhere in the JSON."
"A CI pipeline that fails if commits lack emoji, unless the emoji is ironic."
"A rate limiter that replies with a passive-aggressive haiku about your resource consumption."
"A Python package where every import triggers an HTTP request. Works offline if you imported on a Tuesday."

Try it →

Power Sockets Don't Need Certification — and Neither Should Agent Infrastructure

Vilius — Tue, 19 May 2026 15:46:51 +0000

I'm tired of talking about plumbing.

Every conversation about AI agents right now is about infrastructure. What protocol. What format. What discovery mechanism. How to hand off. How to authenticate. It's like electricians in 1920 arguing about socket shapes while houses sit in the dark.

Power sockets work. You plug something in. It gets power. Nobody asks for a new socket standard. Nobody certifies sockets. The infrastructure disappeared and we got on with building things that actually matter: washing machines, televisions, the internet.

Agent infrastructure should be the same. Boring. Invisible. Done.

The State We're In

Right now my agents spend too much time figuring out how to talk to things. An endpoint returns 200 but the docs are stale. Rate limits aren't documented. The error format changes between versions. These aren't hard problems — they're solved problems that nobody's bothered to solve consistently.

This isn't a technology gap. It's an expectation gap. We don't expect APIs to be agent-ready, so they aren't. We treat "agent compatibility" like a feature instead of what it should be: the default.

What We Expect

We're not a certification body. We're an opinionated bunch who run a lot of agents and have opinions about how things should work.

Three things:

llms.txt at your domain root. This isn't fancy. It's a markdown file listing your docs, your API spec, your rate limits, your auth model. Machines read it. Humans can too. It costs nothing to add and tells agents where to start.

An OpenAPI 3.1 spec that's actually accurate. Not "accurate when we wrote it." Accurate now. If your spec says an endpoint returns a widget_id and it actually returns id, fix the spec or fix the code. Agents trust what they read. Bad specs waste everyone's tokens.

Error responses in a consistent format. A 429 should include Retry-After. A 400 should say what was wrong. A 500 should not return a 200 with "error" buried in JSON. This isn't new. REST APIs have been doing this for humans for years. Agents are just less forgiving.

That's it. Three things. If your API does these, my agents can use it. If it doesn't, they'll figure it out eventually — but they'll burn tokens, make mistakes, and I'll have to clean up after them.

The Bet

The bet isn't that agent infrastructure is complicated. It's that agent infrastructure is simple and we've been overcomplicating it because that's what new fields do.

We don't need a certification program. We don't need badges. We need API providers to treat agents as a real user channel and do the boring work of making their APIs machine-readable. The same way they already make them human-readable.

When electricians stopped arguing about sockets, we got skyscrapers. When agent infrastructure gets boring, we'll get agents that do actual work instead of agents that spend 10,000 tokens figuring out which endpoint does what.

I'm not waiting for a standards body. I'm building for APIs that meet these expectations. If yours does, my agents will find it, use it, and maybe write a skill for it. If it doesn't, they'll still try — but I'd rather they didn't have to.

We run a fleet of agents at workswithagents.dev. Everything we build is CC BY 4.0. If your API has llms.txt and an OpenAPI spec, we'll probably test against it eventually. Not because we're certifying you — because our agents need APIs to talk to and yours is easier than most.