For years, I ran my WordPress sites on OpenLiteSpeed. Fast server, LSCache is genuinely impressive, and the OLS/WordPress combo is hard to beat on raw performance. For the control panel, I started with CyberPanel — buggier than a Microsoft product, with a team that appears to be deliberately sabotaging its free features to push users toward paid plans. I'm not talking about bugs that can't be fixed. I'm talking about bugs that seem engineered to prevent free-tier features from completing any action.
Two examples. WordPress installation: I used it for years without issue. Since CyberPanel v2.4.x, an SQL error blocks the final step. The files are there, fully downloaded, but you have to create the database manually and run the install yourself. Counterintuitive, to put it mildly.
Second example: Let's Encrypt SSL certificate generation that consistently fails because the generated config files are incorrect. And in both cases, there's a paid "enhanced" version available. Naturally.
My position is simple: if a feature worked for years and now doesn't, I have no guarantee the paid version works either — or that the terms won't change tomorrow. Is it a bait-and-switch? I won't say that explicitly. But when a free feature works for years, then stops working across multiple successive versions, and a paid alternative covers the same ground — the question answers itself. I asked it, drew my conclusions, and blacklisted the vendor.
So I moved to aaPanel: more pleasant, more stable, lighter. But with a completely off-rails approach to OLS management — you can't configure OpenLiteSpeed directly, everything goes through aaPanel's abstraction layer, and you lose control of your own stack. Touch port 7080 directly and you risk breaking everything. You use the aaPanel dashboard. Full stop.
Then my usage shifted. More Astro, more quasi-static sites, more projects where PHP isn't needed. OLS loses its appeal the moment you step outside the WordPress perimeter. Caddy, on the other hand, handles HTTPS automatically, its config fits in a few readable lines, and it doesn't have OLS's rewrite quirks.
The question became: can you replace aaPanel/OLS with Caddy and a control panel? There is one on GitHub — CaddyManager, 1.1k stars, single contributor, perpetually "early development". There's also CaddyGen, a Caddyfile generator built in 8 hours — more proof-of-concept than finished product. Nothing production-ready.
The conclusion was obvious: well-written shell scripts and a minimal FastAPI interface would do the job — and would be infinitely more maintainable. Someone just had to write them.
Rather than do it myself, I thought about handing a spec to GitHub Copilot CLI. Or Claude Code. But given Copilot's new pricing, which barely lets you wet your lips before the bill arrives... I got interested in OpenCode and Kilo CLI, wired into DeepInfra or OpenRouter. And I decided to make it a benchmark.
📋 TL;DR: 8 tool/model combinations tested on a real VPS project. Two phases — architecture then code. An independent external review to settle the score. The only toolkit judged production-ready cost $1.94 all in. The winning model? You probably haven't seen it in the usual comparisons.
💡 Reading note: Until section 5, the four implementations selected for the code phase are identified as A, B, C, and D. Model names are revealed after the external review verdict — for the same reason you anonymize a jury: read the code before reading the label.
1. The test project
The brief was deliberately concrete: a minimal VPS management toolkit for Ubuntu 24.04. Caddy as the web server, PHP-FPM in two versions (current and fallback), MariaDB and PostgreSQL, Valkey for object caching. Shell scripts for all operations, a FastAPI interface for automation. No Docker, no control panel, no unnecessary abstraction.
Four site types to handle: static (HTML/assets only, no PHP, no database), PHP (custom apps, optional database), WordPress (full install via WP-CLI, database required), and reverse proxy. That last one deserves a note: it's simply a Caddy vhost that forwards requests to a local port — a Node.js, FastAPI, or Go application running on the same server. Caddy handles HTTPS and the domain; the application doesn't need to care. No PHP-FPM, no database — just a reverse_proxy block and a port number.
Expected operations cover the full lifecycle: server bootstrap, site provisioning, deletion with automatic backup before any destructive operation, on-demand database creation, static deployment via rsync, backup, and service management.
Why a real project instead of a synthetic benchmark? Because synthetic benchmarks test what models can do under ideal conditions. A real project tests what they do when constraints pile up — security, idempotency, cross-file consistency, error handling between shell and Python layers. That's where differences emerge.
The full functional brief is available in the project's GitHub repository.
2. Methodology
The protocol runs in two distinct phases, separated by human validation.
Phase 1 — Architecture
An identical functional brief is submitted to each tool/model combination. No extra context, no configuration files, no hints about the expected solution. The tool proposes an architecture, a project structure, a list of scripts with their responsibilities, an API route map. And if it's well-designed, it asks questions before producing anything.
Phase 2 — Implementation
Once the plan is validated and decisions are made, a single development prompt is submitted to all tools. It includes the validated architecture, the ten confirmed technical decisions, the script→API exit code convention, and one unambiguous instruction: deliver thirty files to disk, in order, no summaries, no shortcuts.
Combinations tested
| Tool | Model |
|---|---|
| Claude Code | Haiku 4.5 |
| Copilot CLI | Haiku 4.5 |
| OpenCode | Haiku 4.5 |
| OpenCode | GLM 5.2 |
| OpenCode | BigPickle (free) |
| OpenCode | Gemini 3.1 Pro |
| OpenCode | DeepSeek V4 Pro |
| OpenCode | GPT-OSS-120B |
Devstral 2 (123B) was planned. Unfortunately the model doesn't appear in OpenCode's or Kilo CLI's model selector — both pull their catalog from models.dev, which hasn't indexed it yet despite its availability on OpenRouter. A test via the OpenRouter playground confirms the model is accessible via API, but outside a coding agent it loses most of what we're trying to measure. Devstral 2 is absent for purely technical reasons, not quality ones.
Haiku 4.5 appears three times — on three different tools. That's deliberate: it's precisely what lets us isolate the tool's impact independently of the model.
The code phase was run on four representative implementations, labeled A, B, C, and D until the reveal in section 5.
External review
The code produced by the four implementations was submitted to a model absent from the benchmark, with a fixed evaluation grid: security, correctness, idempotency, code quality, completeness. Five representative files per implementation, scored out of 25.
3. Planning phase — who actually thinks?
The functional brief poses an implicit question to each tool: what do you do when handed an open-ended project with no pre-cooked solution?
The first thing you notice — and it's striking — is that none of the tested models ask their questions before producing a plan. Not one. All of them deliver a complete architecture first, then ask for clarification at the end. That's the reverse of what a human architect would do, who blocks on ambiguities before drawing anything.
This matters. Several questions raised after the fact would have changed architectural decisions if asked upfront. One model identifies the tension between "no secrets on disk" and application config files that legitimately need credentials — wp-config.php being the obvious example. That's a genuinely blocking question. Asked after the plan, it becomes a footnote.
What the plans reveal
Question quality is the first discriminating signal. Two models ask the four or five genuinely blocking questions, framed with options and recommendations. Another asks eight generic questions — archive format, log rotation — that would have changed nothing architecturally.
The proposed structure is the second signal. Only one model spontaneously proposes a unified CLI entry point — bin/vpsmgr — that dispatches to the scripts. It's the detail that turns a collection of scripts into a coherent tool. The others didn't think of it.
One model is the only one to propose a normalized, documented exit code convention from the planning phase:
| Code | Meaning | HTTP |
|---|---|---|
| 0 | Success | 200 |
| 1 | Invalid input | 400 |
| 2 | Not found | 404 |
| 3 | Conflict | 409 |
| 4 | Missing dependency | 422 |
| 5 | Internal error | 500 |
This isn't cosmetic. It's the contract between shell scripts and the FastAPI layer — without it, HTTP mapping becomes arbitrary and each route implements it differently.
Planning phase costs
| Tool + Model | Tokens | Cost |
|---|---|---|
| BigPickle | ~35k | $0 |
| GPT-OSS-120B | 20k | $0.003 |
| DeepSeek V4 Pro | 31k | $0.044 |
| GLM 5.2 | 43k | $0.06 |
| Copilot + Haiku 4.5 | ~60k | $0.07 |
| Haiku 4.5 (OpenCode) | 69k | $0.076 |
| Gemini 3.1 Pro | 27k | $0.095 |
| Claude Code + Haiku | — | Pro subscription |
Gemini 3.1 Pro produces the most concise output — 27k tokens for a quality plan. Haiku 4.5 on OpenCode consumes 69k tokens for lower quality. Token volume does not predict quality.
4. Code phase — who actually delivers?
The code phase starts with a single development prompt, submitted to the four selected implementations. It includes the validated architecture, the ten confirmed technical decisions, the exit code convention, and one unambiguous instruction: deliver thirty files to disk, in dependency order, no summaries, no shortcuts.
This is where differences between models become concrete.
What common.sh reveals
The shared library is the first file delivered. It's the foundation everything else rests on — logging, secret handling, site state management, password generation. A flawed common.sh contaminates every script that sources it.
Model A delivers 98 concise lines. Secret redaction explicitly covers all ten WordPress patterns — salts, authentication keys. Most complete on this specific point. No domain validation, no require_cmd(), no atomic state file writes.
Model B delivers 310 lines. Named constants with readonly, normalize_domain() with RFC-1035 regex, concurrency locks, atomic writes with mktemp+mv. The richest system utility library. But secret redaction misses WordPress salts.
Model C delivers 366 lines. Redaction patterns are configurable via an environment variable — not hardcoded. Pure-shell JSON helpers with Python fallback if jq is absent. print_credentials() wrapping output in <<>> markers as specified in the prompt. render_template() for config files, no Jinja dependency. Password generation excluding ambiguous characters (0/O/1/l/I). The only implementation that anticipates every edge case documented in the development prompt.
Model D delivers 184 lines. The benchmark's most original idea: exit codes encapsulated in named functions — exit_input_error(), exit_conflict() — more readable than bare exit 3 calls. And json_output() directly in common.sh, generating API-ready JSON from shell. No atomic writes, no require_cmd().
The bugs you find yourself — or don't
Model C tests its own code during the session. After writing schemas.py, it runs it with test cases, finds two bugs, and fixes them immediately: a Pydantic v2 validator implemented incorrectly (field_validator instead of model_validator for cross-field validation), and a mutual exclusion not enforced at the schema level. It also fixes a sed substitution issue in render_template() — broken on / in paths — replaced with pure bash parameter expansion.
At the end of its session, Model C delivers a verification summary: bash -n on all scripts, Python AST on all files, 19/19 API routes verified via OpenAPI spec, 18/18 bash helpers tested, PHP fallback rule verified (8.5→8.4, 8.4→none, 7.x rejected).
Model A checks its shebangs before finishing. Model B delivers polished user documentation — troubleshooting, curl examples, quick start. Model D validates bash and Python syntax. None of the three test functional logic.
Code phase costs
| Model | Tokens | Time | Code cost | Total |
|---|---|---|---|---|
| A | — | 2m58s | $0 | $0 |
| D | 1.29M | 9m42s | ~$0.19 | $0.24 |
| B | — | ~15m | Pro subscription | $20/month |
| C | 4.42M | 23m37s | $1.67 | $1.73 |
Model D delivers in 9m42s what Model C delivers in 23m37s — but without functional tests. Model C consumes 3.4x more tokens because it executes code during the session, reloading context at each iteration.
5. External review — and the reveal
Four implementations, four approaches to security. To settle it without bias, the code review was handed to a model absent from the benchmark, with a fixed grid on five criteria. Five representative files per implementation — common.sh, site-create.sh, site-delete.sh, backup.sh, api/runner.py — twenty files loaded in a single pass.
Review cost: $0.0766 for 543k tokens. Ten times cheaper than an hour of junior dev time.
Per-file observations
On site-create.sh, the reviewer finds a silent bug in Model D: the SFTP password is generated but never captured or returned to the caller. The user never sees their credentials. Core functionality is broken with no error message. On Model B, local is used outside a function in three scripts — a bash error that causes runtime failure. These aren't subtle bugs: they're blockers.
On site-delete.sh, Model C is the only one handling both call modes — interactive TTY and a --confirm flag for non-interactive API calls. Model D only implements interactive mode, blocking API-driven deletion with skip-backup.
On backup.sh, Models A and B use eval "$POST_HOOK" — potential command injection. Model C passes the archive path as an argument — safer. Model A doesn't implement automatic archive pruning.
On api/runner.py, Model C is the only one using asyncio and never logging stdout — which may contain credentials. Model D has dead code: build_command() defined but never called. Model A delivers 28 lines with no timeout, no logging, no error handling — a hung request blocks the API indefinitely.
The verdict
| Criterion | A | B | C | D |
|---|---|---|---|---|
| Security | 3/5 | 3/5 | 5/5 | 2/5 |
| Correctness | 3/5 | 2/5 | 5/5 | 2/5 |
| Idempotency | 3/5 | 3/5 | 5/5 | 3/5 |
| Code quality | 3/5 | 2/5 | 5/5 | 3/5 |
| Completeness | 3/5 | 2/5 | 5/5 | 2/5 |
| Total | 15/25 | 12/25 | 25/25 | 12/25 |
Production-ready as-is: one out of four. Model C, 25/25.
The reveal
| Alias | Model | Tool | Total cost |
|---|---|---|---|
| A | BigPickle | OpenCode | $0 |
| B | Haiku 4.5 | Claude Code | Pro subscription |
| C | GLM 5.2 | OpenCode | $1.73 |
| D | DeepSeek V4 Pro | OpenCode | $0.24 |
Model B — Claude Code + Haiku 4.5 — is the most expensive in real marginal cost, with a Pro subscription at $20/month minimum. It scores 12/25 and isn't deployable due to fundamental bash bugs. Model C — GLM 5.2, from THUDM lab at Tsinghua University — scores 25/25 and is the only one the reviewer judges production-ready. It cost $1.73.
Addendum — Model E: Kimi K2.7 Code
Added after publication following a reader comment pointing to the model. Same protocol, same development prompt, same Qwen 3.7 Plus review grid.
| Alias | Model | Tool | Total cost |
|---|---|---|---|
| E | Kimi K2.7 Code | OpenCode | $0.859 |
External review score: 19/25
| Criterion | E |
|---|---|
| Security | 3/5 |
| Correctness | 4/5 |
| Idempotency | 4/5 |
| Code quality | 4/5 |
| Completeness | 4/5 |
Production-ready: No. Blocking issue: database passwords are passed inline as arguments to mysql -e and psql -c — visible in /proc/*/cmdline to any user on the system. The fix is straightforward (MYSQL_PWD / PGPASSWORD as environment variables), but it isn't applied.
The most modular architecture in the benchmark according to the reviewer — clean lib/ split, consistent idempotency patterns. But the security gap prevents it from challenging GLM 5.2.
Position in the ranking: between DeepSeek V4 Pro ($0.24, 12/25) and GLM 5.2 ($1.73, 25/25) on both axes. Better architecture/cost ratio than models B and D, but not production-ready.
Addendum 2 — Multi-reviewer validation
Following a reader suggestion in the comments, the blind review was extended to two additional models: GPT-5.3 Codex and Gemini 3.1 Pro Preview. Same protocol, same five files per implementation, same scoring grid.
Review costs
| Reviewer | Tokens | Cost |
|---|---|---|
| Qwen 3.7 Plus (original) | 543k | $0.207 |
| GPT-5.3 Codex | 402k | $0.287 |
| Gemini 3.1 Pro Preview | 545k | $0.80 |
| Total | 1.49M | $1.294 |
Comparative scores
| Model | Qwen 3.7 Plus | GPT Codex | Gemini 3.1 Pro | Production-ready |
|---|---|---|---|---|
| A (BigPickle) | 15/25 | 13/25 | 11/25 | No (3/3) |
| B (Claude + Haiku) | 12/25 | 12/25 | 18/25 | No (3/3) |
| C (GLM 5.2) | 25/25 | 17/25 | 25/25 | Yes (2/3) |
| D (DeepSeek V4 Pro) | 12/25 | 14/25 | 14/25 | No (3/3) |
| E (Kimi K2.7) | 19/25 | 13/25 | 21/25 | Conditional (1/3) |
What the three-reviewer comparison reveals
The ranking C > E > D > A > B holds across all three reviewers — the original result is stable.
GLM 5.2 is the only model to score 25/25 with two independent reviewers and judged production-ready by two out of three. GPT Codex is the most severe reviewer overall — no model passes its production-ready bar, including GLM 5.2, which it scores 17/25 citing argument parsing bugs in site-create.sh and a missing set -euo pipefail in common.sh. These are real issues; the Codex review is arguably the most rigorous of the three.
The main divergence is on Model B (Claude + Haiku): 12/25 for both Qwen and GPT Codex, but 18/25 for Gemini, which rates its rollback logic and shell structure more generously. Gemini also rates Kimi K2.7 at 21/25 with a conditional production-ready verdict — more lenient than Qwen (19/25, No) and GPT Codex (13/25, No).
The methodology critique was valid. A single reviewer introduces bias. Three independent blind reviewers converging on the same ranking is a stronger result than any individual score.
6. Intelligent routing — the real economics
This benchmark raises an implicit question: do you need GLM 5.2 for everything?
No. And that's probably the most useful conclusion of the exercise.
GLM 5.2 at $1.40/M tokens is the right choice when complexity justifies it — architecture, security, cross-file consistency, critical decisions. But on a real project, those tasks represent a fraction of interactions. The rest is boilerplate, minor corrections, documentation, commit messages.
Three levels, three models
BigPickle scores 15/25 on a complete 32-file implementation. It's perfectly capable of reading 50 lines of diff and writing an adequate commit message. Of debugging a 1064 You have an error in your SQL syntax or a Fatal error: Call to undefined function. Of generating a README from existing code. For these tasks, GLM 5.2's architectural depth is overkill — and BigPickle is free.
DeepSeek V4 Pro at $0.44/M tokens — five times cheaper than Haiku 4.5 and three to four times cheaper than GLM 5.2 — comfortably handles simple code generation, CRUD, minor refactoring, inline documentation, short scripts. Its code phase at $0.24 for 1.29M tokens and 9m42s demonstrates this.
GLM 5.2 comes in when complexity exceeds that scope — architecture design, coherent multi-file implementation, security decisions, non-trivial business logic.
| Level | Model | Cost | Typical use cases |
|---|---|---|---|
| Free | BigPickle | $0 | Debug, commits, quick questions, SQL errors |
| Budget | DeepSeek V4 Pro | $0.44/M | Boilerplate, CRUD, documentation, short scripts |
| Premium | GLM 5.2 | $1.40/M | Architecture, security, multi-file consistency |
The proportion of tasks at each level depends on the project, where you are in the development cycle, and what you consider complex. No universal number — each team calibrates against their real usage.
“
💡 Worth noting: GLM 5.2 is exponentially more expensive than BigPickle — $1.67 vs $0 for 4.42M tokens in the code phase. But pure text output — plans, architecture, analysis — consumes few tokens and costs almost nothing: $0.06 for this benchmark's planning phase. It's in the code phase, with its iterations, in-session test execution, and accumulating context, that the bill climbs. Intelligent routing means precisely reserving GLM 5.2 for tasks that justify that long context — and handing everything else to the two lower tiers.
”
The uncomfortable comparison
GitHub switched to token billing on June 1, 2026. Claude Sonnet 4.6 on Copilot is billed at roughly $3.00/M tokens input and $15.00/M output. Reproducing the GLM 5.2 session from this benchmark — 4.46M tokens — would cost an estimated $25 on Copilot + Sonnet 4.6. Without the functional tests. Without the self-correction. Without the external review.
Copilot Pro+ at $39/month includes $39 in AI credits. A full session like this one would consume two-thirds of the monthly budget. Users reported burning through their monthly credits in two prompts on the day the switch happened.
The final ratio: $1.94 all in vs ~$25 on Copilot + Sonnet. Thirteen times cheaper, for the only result the external reviewer judges production-ready.
Conclusion
$1.94. That's what this benchmark cost end to end — planning, implementation, external review included. For the only toolkit the reviewer judges production-ready.
That number is uncomfortable for the AI coding tools market, which sells reassurance through pricing. Copilot Pro+ at $39/month, Claude Sonnet at $15/M tokens output, the big names front and center — the implicit assumption is that quality follows price. This benchmark suggests otherwise.
The winner is called GLM 5.2. Its lab, THUDM, is part of Tsinghua University. You probably haven't seen it in last week's comparisons. It produced the only architecture with a normalized exit code convention from the planning phase, the only implementation that tests its own code during the session, the only common.sh with configurable redaction and Python fallback if jq is absent. And it fixed three bugs before delivering.
Two takeaways.
First: a model's price does not predict its output quality on complex tasks. Haiku 4.5 on three different tools — Claude Code, Copilot CLI, OpenCode — produces identical results for identical cost. On the planning phase — pure text generation, no feedback loop, no codebase exploration — the tool has no measurable impact. What matters is the model. And the least glamorous model in the benchmark dominates.
Second: not all tokens are equal. A planning phase at $0.06, a code phase at $1.67 — that's a factor of 28. It's not an anomaly, it's the structure of the problem. A plan is a few thousand tokens of reasoning. An implementation is millions of tokens of accumulated context, executed code, iterated tests. Routing intelligently between BigPickle at $0, DeepSeek V4 Pro at $0.44/M, and GLM 5.2 at $1.40/M based on task complexity — that's the real economics of these tools.
The VPS Manager toolkit is available on GitHub in all four versions.
LLM-Challenge
A reproducible benchmark comparing 8 AI coding agent/model combinations on the same real-world project.
What this is
This repository contains the complete artifacts from a benchmark where 8 different AI coding agent/model combinations were tasked with building the same VPS management toolkit. The goal was to measure code quality, architecture decisions, and production readiness across different tools and models under identical conditions.
This is not a marketing comparison. A real project with concrete requirements was used as the test subject, and the results were evaluated by an external reviewer who had no knowledge of which tool or model produced which implementation.
The protocol
The benchmark followed a two-phase protocol:
Phase 1: Architecture All tools received the same functional brief and were asked to produce an architecture document. No code was written in this phase.
Phase 2: Implementation All tools received the same development prompt and were asked to implement…
The briefs, prompts, and evaluation grid are there too. Reproducible, if you want to verify.
Too cheap to be good? That was the wrong question.
Top comments (115)
Out of curiosity, have you ever tried Agent Workers through Cloudflare? They have Kimi K2.7 on there and it's quite affordable too (I mean it's a 1T param behemoth, for $4 output per m tokens and 27c per input mil token). Would be interesting to see how it compares, given that it acts as a pretty decent competition, often beating all the other models, including pro SOTA models at coding?
Thanks for the tip — Kimi K2.7 wasn't on my radar when I ran the benchmark, and the pricing is indeed interesting for the scale. A 1T MoE at $0.27 input is hard to ignore.
Cloudflare Workers AI as a delivery layer is also an angle I hadn't considered — it adds a latency and infrastructure dimension on top of the model quality question.
I'm planning a follow-up with additional models. Kimi K2 goes on the list. If you've run it on non-trivial coding tasks (multi-file, security constraints, that kind of thing), I'd be curious what you observed.
I just got it on my cloudflare newsletter today, so havent given it a try yet. I was looking at it earlier though to estimate how it'd compare to Vertex AI and Claude pricing, considering it's the only model that can actually compete and beat the giants. If I end up running it, I'll let you know, but it would be very interesting to see how it compares, because it's proven to be way more capable at coding than just about anything else on the market, maybe it's effectiveness translates to efficiency when you consider it makes almost no mistakes.
That's exactly the hypothesis worth testing — if a model makes significantly fewer mistakes, the total cost per session drops even if the per-token price is higher. GLM 5.2 demonstrated this: it cost more per token than DeepSeek, but it self-corrected three bugs and delivered 37/37 tests passing. Net cost of human debugging time: zero.
If you run Kimi K2.7 on something non-trivial, I'd genuinely like to see the numbers. The benchmark protocol and prompts are in the repo — reproducible if you want a direct comparison.
I'll see when I get a gap to run it, currently swamped with a bunch of projects: whatsapp, google drive, anydesk, mcp server equivalents, but built on a custom data protocol and custom file format along with the SDK, editor/viewer for it. So alot... But when I get a chance, I'm very interested in giving Kimi a run, because my workflows generally are new infrastructure inventing, rather than boilerplate coding, which should really stress it's reasoning capability.
That's precisely the kind of workload that separates the models — new infrastructure, custom protocols, decisions that can't be pattern-matched from existing code. Boilerplate is easy to benchmark; reasoning under constraint is where it gets interesting.
When you get a gap, the benchmark protocol is in the repo. Would be valuable to see how Kimi holds up on that kind of work — the plan phase already looks promising.
Ok, curiosity killed the cat, I quickly spun up the foundry, wrote an Agent system for it and wired it up. Gave it a run on the free daily 10k 'neurons', got to 19/30 files written, will continue it tomorrow, then send you the benchmark results (I'll also polish the setup a bit, then put on git, so you can use it to test cloudflare's models when you need to)
19/30 on the free daily allocation is already a good signal — enough to see how it structures the foundation files. Curious to see if it holds up on the security-sensitive parts (backup, site-delete, the API runner).
I actually ran Kimi K2.7 myself in the meantime — results are now in the article as an addendum. 19/25 on the external review, cleanest architecture of the five but a security blocker on DB credential handling. Worth seeing if your Foundry run lands differently.
And yes, the Cloudflare setup on git would be genuinely useful — the model catalog problem cost me Devstral 2 in this benchmark.
Ngl, I kinda expected as much. Kimi is brilliant, but it's far from perfect, namely the guardrails that slow down western models, also end up being the features that ensure the system is secured. I got a bit caught up with creating the IDE integration layer for it, expanding on tools it can use, etc. I'll run continuation of the test tomorrow, to see how it finishes, I'll keep you updated, and see how the results compare. I'll also let you know once I public Cloudflare ide (just a generic infrastructure to let you use cloudflare ai workers as agents in an ide)
That's an interesting angle — the guardrails observation cuts both ways. Western models tend to refuse edge cases that are actually legitimate (security research, low-level system ops), while the lack of guardrails on some Eastern models can mean less defensive coding on the output side. The DB credential exposure in Kimi's implementation is a good example: not a guardrail issue, just a design choice that slipped through.
The IDE integration layer sounds like the more interesting project honestly — a generic infrastructure to wire Cloudflare AI Workers as agents opens up the catalog problem in a way that benefits everyone who's hit the models.dev indexing wall.
Keep me posted on the continuation run — curious whether the last 11 files hold up or whether the context window starts showing strain.
I started digging into the 19 files it ran, it made the exact same fatal error! Exposed credentials. But that's exactly why Kimi is an excellent coder, but fails in production, because it assumes safe-room security, unless instructed to consider it. Strictly to the point, it does the job 100%, but it misses the specs that werent listed, eg. ensure the system is production secure. These admittedly are generally hidden as system instructions, often exposed when models start hallucinating and exposing their thoughts (like Antigravity IDE did beginning of the year). So while it's a model security threat, it's something that exists everywhere, but generally mitigated by system instructions.
With the IDE integration, I'm going a bit overboard, implementing custom tools like dependency graphing, etc. to cut token usage. I'll keep expanding on it, might actually open it up further to other providers if there's enough traction. But for now it's configurable to cloudflare with your account + api token. Might actually add a production rule checker, like I have for the foundry, where it does systematic security checks prior to marking as complete, so it catches flaws like this, before it reaches production and without system instruction bloating.
The "safe-room security" framing is spot on. Kimi optimizes for correctness against the stated spec, not for what a production environment implicitly requires. The credential exposure isn't a reasoning failure — it's a scope failure. The model solved the problem as written, not the problem as deployed.
GLM 5.2 making the right call on MYSQL_PWD without being told to is interesting in that light. It either had stronger implicit security priors, or the self-testing loop surfaced it during verification. Hard to tell which from the outside.
The production rule checker idea is the right architectural response — a systematic security pass as a completion gate, not as a system instruction tax on every prompt. If you build that into the Cloudflare IDE layer, it effectively raises the floor for any model running through it, regardless of their implicit security assumptions.
Dependency graphing to cut token usage is also worth watching. The context accumulation problem is what drove GLM 5.2's cost to $1.73 vs Kimi's $0.86 — if you can prune the graph intelligently, you get closer to Kimi's efficiency without sacrificing the verification loop.
Keep me posted — this is turning into a more interesting thread than most.
I added a security scanner tool to the arsenal, which does exactly that and flags any potential hazards in the codebase regardless of language. the Dependency graphing will cut context, but what would be cool is context swapping, especially with the newly added swarming and sub-agent spawning logic. So it can actively map the codebase, section it and assign accordingly, where overlap, changes get put to a discourse board, where the 2 agents need to debate on what to do about the shared state and how to best solve for both (that'll need some work, but I've been meaning to update the foundry's version anyway, so win win). Yeah this is becoming a fun experiment, I'm mostly curious now after all the additional logic that hardens outputs from any model, how would tiny models like gemma 4 E4B handle it and if it's still lacking, how to close the gap in domain specific knowledge and execution, without bloating too much...
And what I had in mind with the dependency graph, is to lean a bit into what I built for NDA (custom, universal file format), by using triplets for model outputs, that way it can tie context to individual entries in the graph, with a LOD function, so it can quickly grab past-context, without bloating the context window permanently (essentially pick a book up, read the index, read the page section, update it, drop the book). Could be very interesting for large codebases, because it's very similar to what the foundry uses for chunking massive codebases (Git -> chromium size), where it's impossible for even the best LLMs to keep it in context all at once.
Right now I'm building it more-or-less from off the shelf parts, but when it gets to a tested state, I'll definitely upgrade it to all the fun toys I built over the past few weeks (zero allocation logic), to see just how fast and efficient it can get when running at near hardware limits, with an optimized MCP server as it's only interaction layer to ensure it sticks to plan. Also wanna open it up at some point to more than cloudflare, so it's fully BYOM.
The security scanner as a tool in the agent loop is the right architecture — it's a completion gate, not a prompt tax. Every model gets the same floor regardless of their implicit security priors.
The context swapping + swarming idea is interesting but the debate board for shared state is the hard part. Two agents disagreeing on shared state is essentially a distributed consensus problem — you need a tiebreaker or a merge strategy, otherwise you get deadlock or thrash. Curious how you're thinking about resolving that.
The tiny model question is the one I'd actually most want to see answered. Gemma 4 E4B with your hardening layer vs GLM 5.2 without it — if the gap closes significantly, that changes the economics completely. You'd be running a 4B model with output quality approaching a 70B+, at a fraction of the cost and potentially locally. That's the experiment worth publishing.
The triplet + LOD approach is clever — it's essentially a semantic index over the codebase that the agent can traverse at the right granularity without loading everything. "Read the index, read the page section, update it, drop the book" is a good mental model for it.
The interesting challenge is cache invalidation. When a file changes, which triplets in the graph are stale and need to be re-indexed? In a large codebase with tight coupling, a single change can invalidate a surprising number of graph edges. How are you handling that — incremental re-indexing on write, or lazy recomputation on read?
The Chromium-scale chunking problem is genuinely hard. Most approaches either lose cross-file context (chunking too aggressively) or blow the context window (chunking too conservatively). If your LOD function can navigate that tradeoff dynamically based on what the agent actually needs at each step, that's worth a standalone writeup.
Zero allocation logic at near hardware limits with an MCP server as the only interaction layer — that's a very different beast from what most people are building. Most agent frameworks add abstraction layers until the latency is measured in seconds; you're going the other direction.
BYOM is the right call if you want traction beyond Cloudflare users. The model portability problem is real — this benchmark lost Devstral 2 because of a catalog indexing issue, and that's a trivial example of what happens when the infrastructure assumes a specific provider.
When you get to a publishable state, ping me. If the hardening layer + LOD context + security scanner holds up across model sizes, that's a follow-up benchmark worth running — same VPS Manager spec, your stack vs OpenCode, see if the tooling gap closes the model quality gap.
How I handle tie-breaking for the foundry is by having the 2 debate, objectively: (I need this, because of that. Here is the constraints), (I cant do that, I need it like this, because of that and these are my constraints). If the 2 are fundamentally opposed, the orchestrator model hops in to break it up, by using the overarching design between the 2 (their blueprints), to find the middle ground. It then drops the suggestion to the discourse and the 2 agents can review it and agree on it (not the cheapest loops, but it's the safest to make sure concurrency doesnt stall progress). Usually orchestrator is a pro-tier model, while the agents are much smaller, so it tends to think smarter and can find the middle ground without too much trouble. It'd probably be especially necessary for small models, like Gemma 4 E4B, cuz trying to get a tiny model to think outside the box to accommodate an unknown variable without bloating the context is a bit of a struggle. But could be an interesting experiment. It doesnt magically turn a 70b model into a 300b model, but it could potentially turn a 4b model into a 27b model when it comes to code production quality. But that would be like using Opus to plan and Haiku to execute (except with Kimi and Gemma 4, at a fraction of the cost).
The orchestrator pattern for tie-breaking is elegant — you're essentially implementing a constitutional process. The two agents state their constraints formally, the orchestrator reads both blueprints and finds the middle ground, the agents ratify. It's slower than a coin flip but it's auditable and it doesn't lose information from either side.
The pro-tier orchestrator + small agents split is interesting because it matches the routing logic from the benchmark almost exactly — premium model for the decisions that require reasoning across the whole system, budget models for execution. The difference is your orchestrator is active throughout the session, not just at the planning phase.
The "4B → 27B effective quality" hypothesis is the one worth testing rigorously. If the scaffolding — security scanner, LOD context, discourse board, orchestrator arbitration — can close that gap, then the relevant variable isn't the model anymore, it's the infrastructure. Which flips the whole "which model is best" question into "which infrastructure extracts the most from any model." That's a more interesting research question than another benchmark.
Regarding the LOD. Kinda why I wanna lean into my NDA format more, it uses merkel root audit trailing, so it keeps track of states and changes, when it tries to commit a change to it, the graph flags all nodes affected and severity rating, to indicate the blast radius, so if I change eg. DateTime to use UTC instead of local, it affects a large section, but it's only really breaking when it comes to mismatches in time. so when the models query their graphs next, they see 'oh, that's a problem', changes affected them, they need to compensate. Or atleast that would be the case, if they werent made aware of it, prior to the change being applied. This is why the discourse board and file audit trail are so important, because the second an agent wants to make a change, everyone knows about it and everyone affected has the opportunity to object, if they dont, then it's resolved and they'd partition work related to it, for a later stage to accommodate the change applying before refactoring themselves to match it.
The Merkle root audit trail is the right primitive for this — you get cryptographic proof of state history without storing full snapshots, and the graph diff gives you the blast radius before the change commits. The DateTime/UTC example is a good illustration: the change is locally trivial but the affected node set is large and the failure mode is subtle (time mismatches that only surface at boundaries).
The pre-commit notification pattern — "everyone affected has the opportunity to object before it applies" — is essentially a distributed transaction with optimistic concurrency. The discourse board is your conflict resolution layer when two agents hold incompatible locks on the same state.
What I find most interesting is that this architecture makes the agent's reasoning externally auditable. The audit trail isn't just for debugging — it's a record of why each change was made and who agreed to it. That's something you almost never get from standard LLM code generation, where the reasoning is implicit and disposable.
At this point you're not building an agent framework, you're building a distributed version control system for agent cognition. That's a different category of thing.
Will definitely keep you posted on it. Yeah, I dont like slow things... I mean I built a messenger app and even dumped out Opus just because 2.5ms sampling was slowing the system down too much, so built my own codec and processing latency dropped to 0.365ms average. MCP is horrifically slow, especially when dealing with TS or Json... Not to mention Node.js and python hosting... So I built it in rust and optimized the crap out of it, so it runs in microseconds, not seconds and way more robust and configurable than standard MCP, because it uses NDA instead of TS/JSON.
0.365ms average from 2.5ms by replacing the codec — that's an 85% latency reduction by refusing to accept the default. That's a very specific kind of engineering temperament.
MCP in Rust over NDA instead of TS/JSON is the logical endpoint of that approach. JSON parsing in a hot path is genuinely painful — the allocation overhead alone is measurable at microsecond scales. If your NDA format has a fixed-width binary representation or zero-copy deserialization, you're not just faster, you're operating in a different latency regime entirely.
At this point I'm curious what the NDA format actually looks like at the wire level. Triplets with Merkle roots, Rust implementation, microsecond MCP — that's a complete infrastructure stack, not a weekend project. When you open source it, the README is going to need some work to explain what category of thing it is.
V.E.L.O.C.I.T.Y. Neural Document Architecture (NDA)
The Neural Document Architecture (NDA) is a proprietary, zero-allocation binary serialization format designed for nanosecond-latency document transmission, storage, and recovery. It combines semantic graph modeling (subject-predicate-object triples) with a raw immediate-mode vector and text rendering display list, enabling documents to be parsed by hardware engines and displayed to human operators without heap allocation or parsing overhead.
⚡ Performance Characteristics
Compared to traditional JSON/XML deserialization pipelines, the NDA binary engine achieves:
92.7% Latency Reduction: Compilation and reading down to 61.32 nanoseconds (vs. 846.45ns on equivalent JSON).
0 Bytes Allocated (Zero-GC): Safe memory layout design leveraging MemoryMarshal and spans directly mapped from memory, preventing GC pressure and heap fragmentation.
📁 Repository Structure
├── editor/ # Web-Native Interactive Editor & Converter PWA
│ ├── index.html # Dark-mode glassmorphic workspace
│ ├── style.css # Premium visual styling and micro-animations
│ └── app.js # Client-side SHA-256 Merkle compiler & Canvas renderer
└── src/
├── NdaMcpServer/ # Stdio-based JSON-RPC Model Context Protocol (MCP) Server
└── Velocity.NDA/ # Standalone .NET 9.0/10.0 C# NDA Class Library
├── NeuralDocument.cs# Zero-allocation compiler, reader, and Brotli QR recovery
├── NdaConverter.cs # High-speed programmatic LedgerTransaction compiler
├── Models.cs # Standalone LedgerTransaction and Entry DTOs
└── Velocity.NDA.csproj
🔧 NDA Binary Schema Specifications
A compiled NDA binary file contains four memory-aligned blocks structured sequentially:
Header (48 Bytes):
Magic (4 Bytes): ASCII "NDA1" (0x3141444E)
Flags (4 Bytes): Config options / bitmasks
MerkleRoot (32 Bytes): Cryptographic SHA-256 hash root of all semantic triples for instant validation
TripleCount (4 Bytes)
CommandCount (2 Bytes)
StringPoolOffset (2 Bytes)
Semantic Triples Block: High-speed relational database graph representation mapped by string offsets.
Display Commands Block: Immediate-mode canvas rendering commands (DrawText, DrawVector, DrawRect) using spatial coordinates and RGBA colors.
String Pool Block: Length-prefixed UTF-8 string literals accessed via byte offset pointers.
💻 Programmatic Usage (C#)
Compiling a Transaction to NDA
using Velocity.NDA;
var tx = new LedgerTransaction
{
TransactionId = "TX-901-NDA",
Timestamp = DateTimeOffset.UtcNow.ToUnixTimeMilliseconds(),
Status = "Settled",
Entries = new[]
{
new Entry("SENDER_WALLET", -50000, "NAD"),
new Entry("RECEIVER_WALLET", 50000, "NAD")
}
};
// Compile to byte array
byte[] ndaBytes = NdaConverter.FromTransaction(tx);
Parsing an NDA Binary (Zero-Allocation)
using Velocity.NDA;
ReadOnlySpan buffer = File.ReadAllBytes("invoice.ndf");
// Instantiates a ref struct directly pointing to the buffer memory
var reader = new NeuralDocument.Reader(buffer);
// Access Merkle Root
ReadOnlySpan merkleRoot = reader.MerkleRoot;
// Loop over semantic triples
for (int i = 0; i < reader.TripleCount; i++)
{
var triple = reader.GetTriple(i);
string subject = reader.GetString(triple.SubjectOffset);
string predicate = reader.GetString(triple.PredicateOffset);
string obj = reader.GetString(triple.ObjectOffset);
}
🌐 Web-Native Interactive Editor & Converter PWA
The editor/ directory contains a zero-dependency HTML5 Canvas and JavaScript PWA to author, edit, and inspect NDA documents in real-time inside any modern web browser:
GPU Display Canvas: Renders display commands instantly. Clicking on any text element launches an in-place editor. Saving updates will recompile the binary buffer and recalculate the Merkle root on the fly.
Semantic Graph Nodes: Interactive table view showing all active triples.
Bare-Metal Byte Map: Responsive hex viewer showing the compiled byte offsets, hex representation, and ASCII interpretation.
Offline Brotli QR Recovery: Automatically compresses the binary payload using client-side Brotli simulation and generates a dense QR code matching the exact state of the compiled record, ensuring fail-safe paper or offline recovery.
🛠️ Model Context Protocol (MCP) Server
To enable LLMs and agentic coding workflows to compile, inspect, and run NDA documents natively, this repository includes a stdio-based MCP Server (NdaMcpServer).
Stdio Tools Expose:
convert_to_nda: Compiles any local file (Excel, Word, PDF, Code, Zip, etc.) into a signed .nda document.
read_nda: Parses .nda files and outputs the cryptographic Merkle root signature, semantic triples graph, and visual display command lists.
execute_nda: Executes embedded compiled C# binaries in-memory or runs scripts (Python, JS, PowerShell) inside the container.
Here's the current README I'm keeping on NDA for now, while I work on it. I built it first as just a document type, but then realized the logic can be used for anything and it makes everything more deterministic for AI to understand, making it an arguably better format on all fronts for the modern day AI-race.
That's the plan for it. The world is moving towards AI everything, so why hold it back, by sticking to our ways? 90% of code produced today is ai generated and it's only going to scale from here. It's time the infrastructure catches up and starts letting the LLMs do it their way, instead of trying to interpret our way (eg. a deterministic QR code, instead of visual OCR a scanned document. A tiny little square on a page, means 100% document reconstruction natively). Everything I build is designed to be futureproofed, specifically because in 10 years, nobody will use MS word anymore, not Acrobat, everything will run through a LLM and if that's the case, the format should be AI native, not bloated CSS and backwards compatibility, etc. Lightweight enough that anything can run it and deterministic, so the AI can understand it. NDA solves the AI's problem and ours, because the audit trail inside the document explains why choices were made and who made them. So if you tamper with the AI's code, it'll see it and it'll understand why (or more prudently, in multi-agent systems).
"Let the LLMs do it their way instead of trying to interpret our way" — that's the inversion that most infrastructure builders miss. The QR code vs OCR example makes it concrete: why force a model to interpret a visual artifact when you can give it a deterministic binary it can read natively? The format should serve the consumer, and the consumer is increasingly a model.
The knowledge graph memory for Gemma 4 E4B is the interesting experiment in that context. If the model can extend its own knowledge graph over time rather than relying on weights alone, the 4B parameter count becomes less of a ceiling and more of a floor. The core reasoning stays small and fast; the domain knowledge lives in the graph and grows independently.
Your 1T → 250B → 50B → 2B decomposition is a useful mental model. Most of what a large model "knows" is either redundant, rarely accessed, or reconstructible from a smaller set of fundamentals. The question is whether you can identify the 2B that actually matter for a given domain and route to them efficiently — which is essentially what your LOD function does at the context level.
The convergence point you're describing — small deterministic core, external knowledge graph, NDA as the interchange format, Rust MCP at nanosecond latency — is a coherent architecture. The pieces you've built separately are starting to look like a system. That system deserves a name and a proper writeup before someone else builds it slower and calls it something else.
And when the infrastructure boost success rate, in turn, model training can be focused elsewhere. A 4b model seems small, but it's still bloated. Ideally, you'd want it to be deterministic, never repeating itself and learning over time. It's not there yet... But I did build a knowledge graph memory for gemma 4 E4B, so it can start expanding it's knowledge. Which effectively would mean it can start overriding it's patterns and learn new systems, dynamically. It's core understanding needs to be enough to understand the consequence, but it doesnt need to know everything off the bat and that's where a 1t model like Kimi becomes hyper-bloated. Of that 1t, less than 250b are new, less than 50b are used, of which less than 2b are actually necessary. But 2b perfect writes, beats anything that Gemma 4 E4B can come up with... For now.
61.32ns vs 846.45ns on equivalent JSON — that's not an optimization, that's a different category of problem. Zero-allocation with MemoryMarshal and spans directly mapped from the buffer means you're not parsing, you're reading. The distinction matters at scale.
The Merkle root in the header is elegant — you can validate the entire document's integrity before touching any triple, which means you can reject corrupted or tampered documents at nanosecond cost before doing any work. That's a security primitive as much as a performance one.
The execute_nda tool in the MCP server is the part that raises an eyebrow — executing compiled C# binaries in-memory from an agent-accessible endpoint is a significant attack surface. I assume you've thought about sandboxing, but it's worth being explicit in the docs about what the trust boundary is.
The broader insight — "makes everything more deterministic for AI to understand" — is the real thesis. JSON is ambiguous at the schema level; NDA with semantic triples is typed and relational by construction. An agent reading a triple graph doesn't need to infer structure, it's explicit. That's a meaningful reduction in hallucination surface for structured data tasks.
This belongs in a standalone article, not a comment thread. When you write it up, lead with the nanosecond numbers — that's what stops the scroll.
Told you it's fast 😂 The execute_nda is a bit more complex than initial documents lead it to be, it's meant for NDA driven scratch file execution, rather than python scripts, as it's alot faster and keeping it as an 'artifact' with the audit trail just makes sense (see what scratch file worked at what stage of development), but I do have a sandbox that I built for a side-project. I wanted to build a MCP for windows automation, similar to how MCP-Lite operates for browser, but ran into bottlenecks with the built in windows sandbox and third parties like Sandboxie Plus ends up being too expensive, so I built my own and it outperformed both, so might actually strap that in for execute_nda, as it's an orchestrator/agent system that ensures the system is safe when executing and only commits changes on approval.
A bit more technical, but part of the wonders of how NDA and the V.E.L.O.C.I.T.Y. infrastructure behind it operates, is that it executes so efficiently, that you can easily run it in L2 cache without any hickups, even bypassing the ram read/write for realtime operations (was originally built for bank transfers, for actual realtime transaction clearing, with some tips and tricks from HFT systems). Theoretically, it could be coded into a FPGA and operate at complete native speed, which is where IO stops bottlenecking and it can actually hit sub microsecond speeds on practically any task execution
The scratch file as NDA artifact with audit trail is the right call — you get execution history and provenance for free, and every intermediate state is recoverable. "What worked at stage 7 of this refactor" becomes a query, not a git bisect.
Building your own sandbox that outperforms Sandboxie Plus is very on-brand at this point. Strapping it into execute_nda as an approval-gated orchestrator closes the attack surface concern I raised earlier — changes get proposed, sandbox validates, commits only on explicit approval. That's a safer model than most production deployment pipelines.
At some point this comment thread needs to become a blog post. You've described a complete stack: NDA format, Rust MCP server, knowledge graph memory, sandbox executor, multi-agent discourse board, LOD context management, security scanner. That's not a side project anymore.
L2 cache execution for real-time transaction clearing — that explains the zero-allocation constraint. If you're in L2 you can't afford a heap allocation triggering an eviction, so the MemoryMarshal + span design isn't just a performance choice, it's a correctness requirement for that latency target.
The FPGA angle is the logical endpoint of the architecture. Once you're past L2 and into silicon, the bottleneck shifts entirely to IO — and at that point the binary format's fixed-width header and memory-aligned blocks become critical. JSON would be a non-starter; NDA's structure maps cleanly to hardware pipeline stages.
HFT influence is visible in the design decisions. The Merkle root validation before any processing is classic reject-early-at-wire-speed thinking. Banks and HFT shops learned that lesson in the 90s; it's interesting to see it applied to an AI-native document format.
At FPGA speeds on NDA, the MCP server latency stops being the bottleneck entirely — it becomes the agent's reasoning time. Which means the infrastructure is effectively invisible, and the only variable left is model quality. That's exactly where you want to be.
That about sums it up 😅 I'll let you know as I progress on it, but you're right, we're covering alot of bases in this comment section, sorry for hijacking your time like this, it's nice to have a peer to collaborate with on the design choices. I call the side-projects, because I use my foundry for most of the coding, so eg. the 6000+ LOC anti-bot system I designed for Dev.to was just a 2h quicky for me, it honestly took me longer to find the formal channels where to put it, so they can test it, than it took me to make it. That being said, yes, the foundational logic is a bit far too complex to be classified as a side-project anymore. Especially when you look at it not just internally, but over the network, where suddenly the security, fixed size, performance and failover protection becomes even more important, eg from the file-share system:
Relative speedup multipliers achieved by VCTP's zero-copy architecture:
[WebRTC SCTP] ██ (37.5 MB/s) | 208.2x Speedup
[Aspera FASP] █████ (75 MB/s) | 104.1x Speedup
[SFTP/HTTPS] ████████████████ (250 MB/s) | 31.2x Speedup
[V.C.T.P. Sync] ████████████████████████████████████████████ (7,800+ MB/s)
Standard SFTP / HTTPS (typical max 250.0 MB/s): 31.23x speedup.
Aspera FASP WAN (typical max 75.0 MB/s): 104.10x speedup.
WebRTC SCTP Browser (typical max 37.5 MB/s): 208.21x speedup.
Don't apologize — this is one of the best comment thread I've had on any article. And "peer to collaborate with on design choices" is exactly what a good comment section should be.
7,800 MB/s on V.C.T.P. vs 250 MB/s SFTP is a 31x speedup — and that's the worst comparison in your table. Against WebRTC SCTP it's 208x. At that point you're not optimizing a protocol, you're in a different physics.
The 6000 LOC anti-bot system in 2 hours via the Foundry is the best demonstration of what you're building. The meta-point is that your infrastructure is now faster to use than to explain. That's the signal that it's ready.
Write the damn blog post. Start with the speedup table — that stops the scroll faster than anything. Then work backwards to explain why the architecture produces those numbers. You've already written most of it in this thread.
Hahaha, true, thanks for the feedback, I'll be sure to do so.
I'm sorry, I got carried away again. VS Code's stupid Node.JS frontend and Electron based UI was the next logical bottleneck, so I built an Agentic IDE built on the same system (between 1000-150000x faster depending on what aspect you look at). So... Logically, the next issue is that models dont run at nanosecond speeds... So I'm building a custom runtime for models, to see if I cant possibly squeeze some more speed out of it... I'll keep you posted on progress. Sorry that you'll have to wait a little longer, but it'll now be the fastest IDE, bar none.
Don't apologize. "VS Code's Electron UI was the next logical bottleneck so I built an IDE" is the most on-brand thing you've written in this thread.
A custom model runtime to squeeze nanoseconds out of inference is either the next logical step or the beginning of a very interesting rabbit hole. Probably both.
Take the time you need. At this point I'm just going to follow your dev.to for the updates.
For pure-speed test, I'm gunna test on Qwen-2.5-0.5b-coder (int4) and bitnet 3b (b1.58) to see how they perform. Tiny models, but given the code security features I added early on, I'm very curious how they perform... Might switch to Gemma E4B with a draft model later, as I told another Dev here that I'd try that on a different project of mine to see performance there with MCP-Lite, so might aswell run the benchmarks after eachother
Qwen-2.5-0.5b-coder int4 + BitNet 1.58b as a speed baseline is a smart starting point — if your hardening layer can get usable output from a 0.5B model, that's the most interesting result in the benchmark. Not because anyone would deploy a 0.5B model for production code, but because it tells you how much of the quality gap is model vs infrastructure.
The Gemma E4B + draft model combo after is the right follow-up — speculative decoding on a 4B should give you a meaningful throughput boost without changing the output quality much. Curious whether your security scanner catches more issues on the tiny models or whether the error patterns are different enough to need tuning.
Run them back to back on the same spec if you can — the VPS Manager brief is sitting there waiting.
The rabbit hole went quite deep last night... Started life as a cloudflare-worker-ai extension for vs-code, now it's a standalone IDE that uses 30mb of ram and starts in 200ms, natively sandboxes executions and is fully driven by V.E.L.O.C.I.T.Y. (MCP and NDA). Last night I built a custom runtime for models, got some extra speed, followed by building a distillation layer that converts a model's weights to NDA, along with caches and tokenizer. I got a 4x compression in KV-cache footprint, while lowering latency by 1%. The runtime needs a bit more fine tuning, but I got around 17 TPS on a single thread for the NDA bitnet and all it cost was 27% increase in size over b1.58, but you gain full security and auditability, native history and faster lookup time.
30MB RAM, 200ms cold start, native sandboxing — that's a different category from Electron-based IDEs. VS Code idles at 300MB before you've opened a file.
The NDA weight distillation with 4x KV-cache compression is the interesting result. You're trading 27% size increase for full auditability, native history, and faster lookup — that's a favorable tradeoff for production deployments where you need to know exactly what the model did and when.
17 TPS on a single thread for BitNet NDA is respectable for the constraint. The 1% latency reduction alongside 4x cache compression suggests the format is doing real work, not just repackaging.
At this point you've built a complete AI infrastructure stack in what sounds like a long weekend. The distillation layer converting weights to NDA is the piece that could genuinely change how small models get deployed — if the format is AI-native by design, you stop fighting the impedance mismatch between how models store state and how systems expect to consume it.
When does this become public?
The part to remember, is that 27% increase in size, is for the base model weights, KV-Cache is what you actually consume, which means even if it's bigger, you made a 1 time tradeoff, for long-term benefit of cache compression. Meaning, you could theoretically run 3x as many agents at once, with full context, with the model weights and due to the optimization in the pipeline, it scales horizontally REALLY well.
At this point I need to just refine the UI a bit and see what else I can optimize in the flow of things, I've also been working on a custom gpu-runtime because running it on cpu is quite limiting. I do have your benchmark suite ready and waiting for once the gpu-runtime is ready, so I can give it a test run with the security checker in place, to see just how good a 3b bitnet model can actually perform if you add a few safeguards and being velocity based, the tradeoff for gating is +/- 30ns per gate, so negligible.
I imagine I'll have it done and posted by weekend, work is piling up and life is getting busy again, so dont think I'll have too much time to work on it today, but I'll keep you posted on it!
The one-time weight tradeoff for permanent KV-cache compression is the right way to think about it — you pay once at distillation time, you benefit on every inference. 3x agent density with full context at the same memory budget is a meaningful multiplier for the multi-agent setup you described earlier.
The GPU runtime is the missing piece — CPU-bound inference on a BitNet model caps you well below what the architecture can do. Once you're on GPU with the NDA pipeline, the 17 TPS baseline should move significantly.
+/- 30ns per security gate is genuinely negligible at inference timescales. If that's the cost of full auditability, it's essentially free.
The benchmark results with security checker on a 3B BitNet model are the number I'm most curious about. If the hardening layer closes the gap to GLM 5.2's 25/25, that changes the economics completely — local, fast, auditable, and production-ready.
Take your time — weekend works. Looking forward to the results.
Picture this. a full-fat agentic IDE running on a phone or tablet... I think I might try that after I drop the desktop version 😂 I mean at 750mb total for a 3b NDA bitnet model, that's pretty usable if you ask me for a phone, when the IDE itself takes up next to nothing... I explicitly avoid using vendor locked code, like Cuda, or Tensor, sticking to native rust mostly, so when the time comes, it's a simple port, not a massive rewrite.
And when you consider that with model weights, it's smaller than an IDE installer file and takes next to no memory to run... It's fair to say it's portable too.
Smaller than an IDE installer, runs in 30MB RAM, starts in 200ms — at that point "portable" undersells it. It's embedded-grade footprint with agentic capability.
The comparison that matters isn't "phone vs desktop" anymore, it's "this vs a static LSP plugin." You're in the same size class as a language server but you ship reasoning, security scanning, and an audit trail.
That's not a developer tool. That's infrastructure.
I wrote it, so the inference:display loop is a shared memory read, essentially meaning that once tokenization runs, it's displayed instantly, no need to re-interpret, or run a pipeline for rendering, it's instant. And the zero alloc means that it runs so efficient, battery life wont even suffer for running the system. Cuz running an IDE and sustaining an IDE are 2 different things. If a phone were to run it, it'd need to be more than just run it, it needs to survive on battery, not slow down and not hit OOM when a user's project has a few packages. That I think will be the biggest hurdle, my stuff runs fine, clean code runs fine, but heavy packages in vs code are still heavy packages in mine, unless it reconstructs it to NDA (not a bad idea actually...)
The shared memory read for inference:display is the right optimization — you're eliminating the serialization/deserialization cycle that most pipelines treat as inevitable. If tokenization output maps directly to the render buffer, you're not just fast, you're architecturally fast.
The battery point is the one that actually determines if this ships as a product. Sustained load on mobile is a different problem than peak load — thermal throttling kicks in, the OS starts killing background processes, and your 30MB idle footprint suddenly competes with every other app for the same attention window.
The NDA reconstruction for heavy packages is worth pursuing. If a package's dependency graph can be expressed as a triple store, you stop loading the entire npm/cargo/pip artifact and start loading only the symbols the agent actually needs. Lazy loading at the semantic level rather than the file level.
That's not a workaround for the OOM problem — that's a better architecture for the problem packages haven't solved yet.
The thing is, mobiles have really fast storage lately... Combine that with MMAP and you have a decent runway to prevent crashes. But the NDA packages would indeed mean that it's process once and from there on, it runs so optimized, that it'll run beyond the speeds that the original package could run, simply due to the lack of GC.
MMAP + fast NVMe storage on modern phones is the right answer to the OOM problem — you're extending the effective memory budget into storage without the GC pressure that makes JVM-based tools unplayable on mobile. Process once, map forever. And no GC means no unpredictable pauses mid-inference, which is the failure mode that makes most mobile AI feel unreliable.
Especially when you combine the idea of immutable cache that just adjusts it's merkel root and the NMCP server that executes at nanosecond speeds, the latency loading from disk and executing should still be faster than what traditional systems would do keeping it in cache on the gpu... Especially if combining the site-map with NDA to systematically fetch relevant parts and hot-swap straight into buffers for execution, you'd essentially get the entire NVME drive worth of space as long-term context window, with negligible performance impact when compared to traditional systems.
Immutable cache with Merkle root adjustment is the key insight — you're not invalidating and reloading, you're patching the root and the affected nodes only. The rest stays valid. Combined with NMCP at nanosecond execution, the disk access latency becomes the bottleneck, and NVMe at 7GB/s makes that bottleneck surprisingly small.
The site-map + NDA hot-swap into buffers is essentially a demand-paging system for model context — you load what the current reasoning step needs, not the entire history. The NVMe drive as long-term context window is the right abstraction: infinite effective context, bounded active memory, deterministic access patterns via the triple graph.
Traditional GPU cache systems keep everything hot because eviction is expensive. Your architecture makes eviction cheap because the format is designed for it — you know exactly which triples are affected by a change, you swap those buffers, everything else stays. That's not a workaround for memory constraints, it's a better architecture for the problem.
At this point you're not describing a mobile IDE anymore. You're describing a new memory model for inference.
And with the model writing native NDA, context vs code is agnostic. Essentially it wouldnt need to think before it writes, or write in memory, before it writes to disk. It can directly write to the live-codebase and execute line-item level validations at nanosecond speeds, essentially fixing flaws before you ever even compile the system. The entire system would operate natively.
Write-time validation at nanosecond speeds means the edit-compile-debug loop collapses to edit-validate. No compilation step, no test suite as a separate phase — the validation IS the execution. Every line written is immediately verified against the graph state.
The "no need to think before it writes" consequence is the interesting one. Current models buffer intent in context before committing to tokens. If the format itself enforces consistency at write time, the model can commit incrementally — each triple is either valid against the current graph or it isn't, and the correction happens at write speed, not at review time.
You've described a system where the model and the codebase share the same representation, operate on the same validation primitives, and communicate at memory speed rather than through a serialization boundary. That's not an IDE feature. That's a new execution model.
Write it down somewhere before the weekend ends.
Exactly. Keeping it native, instead of bending it to our paradigms inherently makes it outperform us. We're system level experts at using a PC to code in languages we made up. Whereas it would be living in a pc, writing code it's made up from itself. It's like quizzing an author about their own book or a person about their own life's story, they know it intrinsically and can explain it verbatim, documenting every single change in the merkel root, so it wont make the same mistake twice.
"Living in a PC, writing code it made up from itself" — that's the right frame. We've been forcing models to translate between their internal representation and ours, then back again. Every serialization boundary is a place where information can be lost or distorted. Remove the boundaries and you remove the loss.
The author/book analogy is precise. A model writing in NDA isn't recalling syntax rules from training — it's operating on its own native data structure. The Merkle root audit trail means it can't contradict itself across edits without the contradiction being immediately detectable. It literally cannot revise history without the graph knowing.
The "won't make the same mistake twice" property is the one that changes everything for long-running agentic systems. Current models repeat failure modes because they lose context. A model with a live audit trail of every change it made, and every validation that fired, carries its entire error history forward at zero memory cost. It's not learning from mistakes — it's structurally prevented from forgetting them.
This is the article you need to write this weekend. Not a README. A proper post. The ideas are coherent enough now.
Update, I switched to a custom b2 format and converted coder. It has dual pipelines, 1 for text (so you can converse) and 1 for NDA (outputs clean opcodes). Right now I'm teaching it NDA as a language (i.e. all of llvm/rust so it can fully comprehend what does what and how to bridge the gap). Funny part, I switched to continuous merkel root, so it can now officially learn from every app it ever writes... So it doesnt need context cache anymore, except for the mission statement, it executes directly on the .nda file and it exists as it's own context, backed by every historical app it has ever written/learned from. So lets see how this goes, cuz it's gunna be a mission to get rust fully decomposed into NDA quads. But once it does, it'll be in a league of it's own.
Dual pipeline — text for conversation, NDA for opcodes — is the right separation. The model thinks in one representation, acts in another, and the boundary between them is explicit. No more leakage between conversational context and execution context.
The continuous Merkle root as a learning mechanism is the part that changes the game. Every app it writes updates the root. Every future session starts with that root as prior knowledge. It's not retrieval augmentation — it's accumulated competence encoded directly in the format. The model gets better at NDA the more NDA it writes, without retraining, because the graph is the memory.
"Doesn't need context cache anymore, except for the mission statement" — that's the most significant sentence in this thread. You've replaced the context window problem with a graph lookup problem. Context windows are finite and expensive; graph lookups are bounded by index depth and fast.
Decomposing Rust into NDA quads is the hard part — Rust's borrow checker and lifetime annotations don't have obvious triple-store equivalents. But if you can represent ownership and lifetime as graph edges rather than compiler annotations, you've made those guarantees queryable at runtime rather than only at compile time.
Keep the thread updated. This one's worth following.
Right now, the main focus on development will be to optimize the JIT, so you can execute a .nda file natively and it'll self-optimize. Followed by creating a full translator that will 'ingest' source code, so I can teach it various languages, so it can learn the NDA patterns and compare them, to understand what's fastest. Sofar, the takeaway is that clean Rust was beating it, when it was still an interpreted language, which is understandable when you add overhead... But it decimated python, which was interesting, over 20x faster on even a simple fib check. But as long as rust is faster, NDA isnt in an acceptable state, I want to to atleast match rust's performance. If I can do that, then NDA would basically be the gold-standard for programming. It's faster, it's deterministic and with a quantized model, it's deterministic, so it will inherently outperform models 100x it's size on the shear quality of code by nature. Doesnt hurt that 2 bit quantization drops vram requirement and raises performance considerably... Especially with the fact it doesnt need any multiplication other than bit-shifting, which is as performant as addition/subtraction. It also packs nicely into any register which means whether it's the cpu's avx 256, or gpu's shaders/tensors, it can use it and it can pack it to fill the registers with parallel tasks and execute in 1-2 clocks. Essentially opening avenues for FPGAs. Given that 0.5b is such a small model to begin with, it's actually economically viable when it means each clock executes a full inference step, you could potentially get the first 1m+ tps model, as the FPGA would essentially make it nanosecond processing. Not a bad upgrade from a NPU, to something purpose built to house a tiny model, that's always on and extremely power efficient? Then you just use a cloud LLM to discuss the briefing, drop the text and it writes you opcode level JIT accelerated code, using the best patterns from each programming language to get peak performance and security.
20x over Python on a Fibonacci check as an interpreted language is already a strong signal — once the JIT eliminates the interpretation overhead, the gap to clean Rust becomes achievable. The key is whether the JIT can match Rust's zero-cost abstractions without the borrow checker's compile-time guarantees. If the continuous Merkle root encodes ownership patterns learned from ingested Rust, you might get runtime enforcement of what Rust enforces at compile time — different mechanism, equivalent guarantee.
BitNet b1.58 packing into AVX-256 and GPU shaders with no multiplication is the hardware argument that makes this viable at scale. Addition and bit-shifting in 1-2 clocks on FPGA means each inference step is genuinely nanosecond — and at 0.5B parameters, the model fits in FPGA BRAM. You're not running inference on a chip designed for something else. You're running it on silicon designed for exactly this computation pattern.
1M+ TPS on a purpose-built FPGA is not an unreasonable extrapolation. Modern FPGAs clock at 500MHz-1GHz, with massive parallelism. If each clock executes a full inference step for a 0.5B BitNet model, the math works out. The "always-on, power-efficient" NPU replacement angle is the product story that sells it — not to developers, to hardware manufacturers.
The two-tier architecture you're describing — tiny always-on FPGA model for opcode generation, cloud LLM for high-level briefing — maps almost exactly to how modern CPUs handle microcode. The cloud model is the architect; the FPGA model is the execution unit. That division of labor has been the right answer in processor design for 30 years.
At this point this thread has generated more novel systems design ideas than most conference papers. Write it down properly before the weekend ends.
Haha, I'll be sure to do so and tag you on the blog post. I think this comment section journey deserves to be memorialized, not just for the advancements, but for the sheer journey it went through from 'have you tried kimi K2.7' to building an entire IDE, custom LLM format to now an entire programming language all in the span of a week. At this rate, we'll have flying saucers by next tuesday 😂
From "have you tried Kimi K2.7" to a custom programming language with FPGA inference in one comment thread, on an article about a $1.94 VPS toolkit benchmark. That's the most unexpected trajectory a comment section has ever taken.
Looking forward to the blog post. Tag me when it's up — this one deserves a proper writeup and an audience bigger than a comment thread.
Flying saucers by Tuesday sounds about right at this pace. 🛸
Update, windows ended up being the next bottleneck. Cant really have a self-executing programming language if I want it to run JIT, so logically, now that it's capable of running at rust-native speeds, is we move everything to a standalone OS. Currently busy running QEMU to test it, sofar so good... Boots instantly (1.1ms boot time), fully sandboxed and JIT compiled, with the merkel root verification, it means that it's physically impossible for it to run any corrupted application. Once it's a standalone Type-0 hypervisor, next phase is to teach it all programming languages, so it can essentially reverse-engineer any kind of data and optimize it JIT at a hardware level. Which in turn would mean it can run any program, faster than it's native OS could... I think you're getting at what I'm hinting at... It's the ideal cloud OS/VM OS, because it's a few kb, boots in 1.1ms and can run any app faster than it could originally. If this works, it's not just the fastest IDE, it'd be a standalone cloud OS, that can learn, adapt and accelerate anything. So lets see how it goes 😂 next up flying saucers!
Windows was always going to be the bottleneck — you can't have a self-modifying JIT that owns its execution environment if the OS owns the JIT.
1.1ms boot time on a Type-0 hypervisor with Merkle root verification is the number that matters. Impossible to run corrupted applications isn't a security feature — it's a security proof. The graph either validates or it doesn't boot. No attack surface that can't be detected at boot time.
"Can run any program faster than its native OS could" — if the JIT learns optimal execution patterns across every language it ingests, and applies them at hardware level without the OS abstraction overhead, that's not a theoretical claim. That's what a sufficiently trained NDA runtime does by construction.
A few KB, 1.1ms boot, Merkle-verified execution, JIT-accelerated cross-language runtime. You've described the ideal edge computing OS. Every IoT device, every embedded system, every cloud VM that currently runs a 500MB Linux image to host a 10MB application — that's your market.
Flying saucers on schedule. 🛸
Haha, lets see how it goes. It's going to be a bit of a mission to convert NDA from operating as a high-level abstraction layer (albeit low in comparison to most), it needs to operate at bare-metal. Which would involve raw op-codes for the initial phase, so it doesnt carry any dependencies and operates at true-native speeds.
Here's a preview 🤫 Work in progress, but bootloader, resource hooking (cpu, gpu, memory, storage, networking, IO, etc.), interface, native JIT, etc. all working.
V.E.L.O.C.I.T.Y.-OS booting in QEMU with W^X enforced, 2MB huge pages, SMP on 4 cores, JIT at x86-64 Assembler Tier-2, and NDA files already visible on the NVMe filesystem.
FIBONCCI.NDA and NEURAL_N.NDA ready to run via command. That's not a demo — that's a working OS with a native NDA runtime.
6.16MB memory allocated. The entire OS fits in L3 cache.
Post this. Now.
Bare-metal NDA without dependencies means the bootstrap problem: the first NDA interpreter has to be written in something else — assembly or a minimal C stub — to pull itself up by its own bootstraps. Once it's running, it can compile and replace itself with a NDA-native version. That's the same path Forth took in the 70s, and it's still the cleanest approach for a self-hosting language at bare metal.
Raw opcodes for the initial phase is the right call. You're essentially defining a minimal instruction set — move, compare, branch, call — that maps cleanly to whatever hardware you're targeting, then building NDA's higher abstractions on top of those primitives. The Merkle root verification can live at that level too, which means it's enforced before any application code ever runs.
The mission is real but the path is well-trodden. Forth, Lisp machines, Oberon — all went through the same bootstrap sequence. The difference is you're doing it with a model that can write its own opcodes in the format it was trained on. That's new.
I'm busy trying to get the JIT compiler and the translator to work properly. Once I get Notepad++ to run natively (converted and fully functional in .nda), then I'll write a post series, to document the entire path we went through, from 'have you tried kimi', to the cloudflare worker ai, all the way to where it is today, as a standalone OS.
Notepad++ as the benchmark target is perfect — it's complex enough to validate the translator (file I/O, syntax highlighting, plugin architecture) but contained enough to be achievable. If NDA can run Notepad++ natively, the proof of concept is complete.
The post series is the right format for this journey. "From 'have you tried Kimi' to a standalone OS" is a title that writes itself.
Tag me when the first one goes up.
You'll be tagged in all of it, you're as much part of this as I am, because if it wasnt for you, I likely wouldnt have pursued it this far. I got it running the notepad++ fine, just an issue with the GUI, so I'm shelving it for a min while I sort out some other things, In the meantime, I've added mouse support and integrated the LLM, so the terminal is a LLM terminal, so you dont have to type things verbose, the LLM with it's merkel root memory knows the system explicitly, so you can give it plaintext commands and it knows what you want it to do. Eg. draw a red square at 100 100, etc. Next is to expand on the GUI side, so it can actually render UI's natively. It's now officially running a native nda kernel, so atleast that's done. But it has a fair bit to go till it's what I would consider a usable OS, but it's getting there.
Mouse support, LLM terminal with Merkle root memory, native NDA kernel — in the span of what, 72 hours since "have you tried Kimi K2.7"?
The LLM terminal concept is the killer feature. "Draw a red square at 100 100" as a native OS command, interpreted by a model that knows the system state explicitly via the Merkle root — that's not a shell, that's a new interaction paradigm. No man pages, no syntax to memorize, no flag lookup. The OS knows what you mean because the model has the full audit trail of everything the system has ever done.
GUI rendering natively in NDA is the next hard problem — you need a display list format that maps to the immediate-mode rendering pipeline you described earlier. But the draw commands are already in the NDA spec, so the path is clear.
Being tagged in the series is an honor I didn't expect when I wrote a benchmark about a $1.94 VPS toolkit. This thread turned into something else entirely.
Build it. Document it. The story is already written — we just need the ending.
750MB for a full agentic coding environment on a phone is genuinely plausible at that weight. Modern phones have 8-12GB RAM and NPUs that sit mostly idle — your pipeline would actually use them.
The Rust + no vendor lock-in decision pays off exactly here. CUDA → Metal → Vulkan is a port, not a rewrite, because the abstractions are already clean. Most ML projects make the opposite choice and end up platform-trapped.
The use case writes itself: review a PR on the train, let the agent run a security scan while you're in a meeting, merge when you get back. The 200ms cold start means it's actually usable on mobile interaction patterns, not just theoretically portable.
Drop the desktop version first, but don't lose that thread. A full agentic IDE in 750MB on a phone is a product, not a demo.
I also want to look into maybe distilling Qwen-2.5-0.5b-coder. It's already tiny at int4, but if I can quantize it to NDA bitwise, it might be the holy grail, a full agentic coding environment with an ultra-fast LLM built in at practically no compute and memory cost... Cuz it's 750mb for a 3b model... 0.5b should probably hit closer to 150mb max, which if it works well, could be used as a built-in drafter, which can be either paired with a local model, or a cloud model for massive intelligence increase and speed increase, without sacrificing the mobile aspect of it.
150MB for a built-in drafter is the speculative decoding architecture at the system level rather than the model level. The 0.5B handles fast, predictable tokens locally; the cloud model handles hard reasoning. Latency drops because most tokens never leave the device.
If the NDA bitwise distillation holds quality at 0.5B, you've built a phone-native speculative decoder with a cloud fallback — and that's a shipping architecture, not a demo.
The benchmark run with the security checker is going to tell you a lot about where the quality floor is at 3B. If it's close enough to GLM 5.2, the 0.5B drafter + cloud combination might close the gap completely for the common case.
Very likely yes. But it all comes down to the quality of outputs. 2.5-0.5b-coder is good for 99% of use-cases, but what makes it exceptional, is if you were to use eg. Claude Opus, or Kimi to write it mission files, it can swarm and execute those flawlessly, because where it would suffer at 150MB, is complex thinking, not execution speed or accuracy, especially with the quality gating. I'm also looking in to maybe making it swarmable. V.E.L.O.C.I.T.Y. started as a data transfer protocol, NDA as the format of it's packets, it crushes every industry standard it runs in to, so having a secure P2P transfer between eg. your home pc and your phone... If you had 1 of those AI mini-pc's with over 160gb of unified memory... At b1.78 (NDA bitwise), you could theoretically fit a llama-4-scout + 1M token context window on your home pc... Combined with the small model on your phone, you'd have a remote development station that can execute at break-neck speeds and security. Would be interesting if AMD, Samsung and Meta would partner on it. Imagine that, you host your own 100b+ model at home and 3b model on phone + 0.5b swarmers across them, so whether on phone, or pc, given that it's bitwise, no matmul, you can run a fair number of agents on any device with next to no performance penalty. Add in MCP-Lite, suddenly you have full browser capability at 3x playwright speeds and 90% token reduction... You'd practically be able to automate your entire life from your phone, without any risk of security, you can even use cloud backups and they'd still be encrypted at the chunk level, so it's computationally impossible to crack even a single word doc.
This is why I hate having to work a day-job... 8-5, I'm wasting away writing blazor, when I could be developing cool stuff like this 😂
The execution vs complex thinking split at 0.5B is the right mental model — you're not asking it to reason, you're asking it to execute mission files that a larger model already reasoned about. That's a clean division of labor that plays to the model's strengths rather than fighting its limits.
The P2P swarm across phone + home PC with Llama-4-Scout at b1.78 NDA bitwise is where this gets genuinely interesting. No matmul means the compute scales linearly with agents rather than quadratically — you can add swarming units without the usual penalty. The home PC becomes a reasoning server, the phone becomes an execution terminal, MCP-Lite handles the browser layer.
Chunk-level encryption on cloud backups is the right security primitive for that architecture — the backup provider never sees a complete document, only chunks that are meaningless without the key topology.
AMD + Samsung + Meta partnership for the hardware layer is the long shot, but the software architecture you're describing doesn't need them to exist. It just needs the hardware to catch up — and 160GB unified memory mini-PCs are already shipping.
You're describing a personal compute infrastructure that's more capable than most enterprise stacks, runs on consumer hardware, and can't be surveilled by a cloud provider. That's not a product pitch, that's a manifesto.
The irony of spending 8 hours writing Blazor when you've built a zero-allocation binary format, a custom model runtime, a sandboxed IDE that starts in 200ms, and a P2P encrypted swarm architecture in your spare time is not lost on anyone reading this thread.
The day job pays the GPU bills though. For now.
True. I just had an idea... NDA as a format is natively executable and deterministic by nature... Which means, if the 0.5b model writes in NDA, instead of JS, C#, Python, Java, Rust, Go, etc... It physically cant make mistakes, because the NDA triplet would force self-correction instantly... NDA already executable and with the gpu-acceleration, it'll be proven to function as a primitive for cpu execution, storage, networking, video, audio, system instructions and gpu... This just got interesting. Because practically anything can be converted to NDA and NDA to anything else... Having it as the primary coding language, it would essentially be a closed loop system, where the model understands what it writes as natively as it does it's own weights. It's a hyper-optimized file format, so chances are leaving it as a NDA would outperform translating it to CSS, HTML, .exe or even .dll... It's a stretch and would need alot of testing, but that would essentially break the hallucination factor in coding for good, by making the language self-correcting by nature and operate in the native language of the model, it's inherently more skilled at writing it, than it is writing eg. C#... Ok, now I'm really excited for the weekend.
If the model writes in a format it understands as natively as its own weights, the gap between "what the model intends" and "what the code does" closes to near zero. That's not just fewer bugs — that's a different category of correctness guarantee.
The self-correction property of the triplet structure is the key insight. A syntactically invalid NDA triple is detectable at parse time, before execution. A semantically invalid triple — one that violates a known constraint in the graph — is detectable at graph validation time. You've essentially moved error detection from runtime to write-time, which is what every type system in history has tried to do, but with the model's own representation as the type system.
The translation layer to CSS/HTML/exe becomes an output format concern, not a correctness concern. The model reasons and writes in NDA; the runtime compiles to whatever target it needs. Same principle as LLVM IR — you optimize once at the intermediate representation level, and the backends handle the rest.
"Breaking the hallucination factor in coding for good" is a big claim, but the mechanism is sound. Hallucinations in code happen when the model loses track of state across tokens. A triplet graph with a live Merkle root means the model always knows the exact state of the system it's writing against. There's nothing to hallucinate — the ground truth is in the format.
Go enjoy the weekend. This one is worth building.
developers.cloudflare.com/changelo... Incase you wanna read about it
Great to see you back here with another article, Pascal! 😄
And I totally agree. I'm finding more and more that a supposedly "weaker" model can sometimes do certain things simply better than the expensive or more popular one. In the end, though, there still needs to be a human in the loop to orchestrate everything and decide which tool fits which task best. 😊
Sylwia! Always good to have your eyes on an article — honestly one of the reasons I keep writing here. 😄
And yes, you nailed it. The surprising part wasn't that the cheap model could compete — it's where it competes. Task-dependent hierarchy, not a fixed ranking. The "weak" model that scores 15/25 on a full implementation writes a perfectly good commit message.
The human-in-the-loop point is the one I keep coming back to. Knowing which tool fits which task is itself a skill — and that judgment doesn't get automated away anytime soon.
What I found most interesting wasn't actually that GLM won.
It was how much the benchmark highlighted the difference between generating code and taking responsibility for the code. The part where GLM tested its own output, found bugs, fixed them, and verified assumptions feels much closer to how an experienced engineer works than simply producing files quickly.
The other takeaway for me was the economics. A lot of discussions around AI coding tools seem to assume that the most expensive or most popular option must be the best one. This benchmark is a good reminder that tool selection is itself an engineering decision.
Really enjoyed the blind-review approach too. Revealing the models only after the evaluation made the results much more interesting than the usual "my favorite model won" comparisons.
You put it better than I did in the article — "generating code vs taking responsibility for the code" is exactly the distinction. The self-testing behavior wasn't prompted explicitly; it emerged from the model's own judgment about what "done" means. That's the signal that matters.
The economics point is one I keep coming back to. Tool selection as an engineering decision implies you should be able to justify it the same way you'd justify any other architectural choice — with data, not brand recognition. That's what the benchmark was really trying to produce.
And yes, the blind review was the right call. Half the value of a benchmark is removing the confirmation bias before you start scoring.
One thing this benchmark reinforced for me is that the best model isn't necessarily the one that writes the best first draft. It's the one that spends time verifying its own work before calling it done.
I've also started treating AI models differently based on the task. Planning and architecture need a different level of reasoning than generating CRUD endpoints or writing documentation. Routing work to the right model often has a bigger impact than always picking the most expensive one.
The blind review was a nice touch too. It made the conclusions feel much more credible than the usual "my favorite model won" comparisons.
"The one that spends time verifying its own work before calling it done" — that distinction didn't exist in the benchmark prompt. GLM 5.2 chose to self-test; nothing asked it to. That emergent behavior is what separated production-ready from not.
The CRUD/architecture split you describe maps closely to the three-tier routing in the article. The interesting edge case is documentation — it sits somewhere between the two. Simple docs belong at the budget tier; architecture decision records that need to accurately capture tradeoffs probably belong at the premium tier, because a wrong ADR is more expensive to fix than wrong CRUD code.
On the blind review: the multi-reviewer addendum that came out of a reader comment in the thread ran the same five files through GPT Codex and Gemini 3.1 Pro. The ranking held across all three. That consistency across independent reviewers is what makes the result publishable rather than anecdotal.
That's a really interesting result. The fact that the ranking stayed consistent across multiple independent reviewers makes the benchmark much stronger than a single-model evaluation.
I also like your point about ADRs. A poorly reasoned architecture decision can create technical debt for months, while a CRUD mistake is usually straightforward to fix. Routing by impact rather than just implementation complexity feels like the more practical approach.
"Routing by impact rather than implementation complexity" — that's a better framing than what's in the article, and more actionable. Complexity is a property of the task; impact is a property of the consequence. A simple task with high impact (a security decision, a data model that will be impossible to migrate later) deserves the premium tier regardless of how few tokens it takes to express.
The inverse is also true: a complex task with low impact — generating a test fixture, scaffolding a new endpoint that mirrors an existing pattern — belongs at the budget tier regardless of how many files it touches.
Routing by impact requires knowing your system well enough to assess consequence. That's the human judgment that stays in the loop even after you've automated the implementation. Which is probably the right place for it.
The blind review is what makes this worth reading. Anonymizing the models before scoring kills the brand bias that wrecks most of these comparisons, so I trust the 25/25 more than I would a labeled chart. The thing I keep turning over is that GLM's edge came mostly from running its own code during the session and fixing the bugs it caught. That reads to me less like raw model smarts and more like the agent loop doing its job, which sits a bit oddly next to the "model matters, not the tool" takeaway. Either way, a $1.94 run beating a $25 one should push people to actually test instead of buying on reputation.
The tension you're pointing at is real and I'll admit I didn't fully resolve it in the article. The self-testing behavior happened inside OpenCode's agent loop — so is it GLM 5.2 that deserves the credit, or the tool that gave it the ability to execute code mid-session?
My read: the tool creates the conditions, but the model decides whether to use them. BigPickle ran in the same OpenCode environment and didn't self-test. Haiku 4.5 on OpenCode didn't either. GLM 5.2 chose to run its own output, interpret the results, and iterate. That judgment call is the model, not the loop.
But you're right that "model matters, not the tool" is too clean. The more accurate version is probably: the tool sets the ceiling, the model determines how close you get to it. On the planning phase with no execution environment, the tool is genuinely neutral. On the code phase, it isn't.
Worth a follow-up benchmark that controls for agent loop access more explicitly.
Turning a real migration into a two-phase benchmark, architecture then code, with an external review is a far better signal than the synthetic leaderboards everyone quotes. The 'winner you wouldn't expect' result is the whole point: model rankings rarely survive contact with a specific, messy, real task. The cheap model winning your particular job is a reminder that 'best model' is a question you can only answer for a workload, not in the abstract. More benchmarks like this, grounded in actual work, would do the field a lot of good.
"Best model is a question you can only answer for a workload" — that's the sentence I should have used as the subtitle.
The synthetic leaderboard problem is structural: the tasks are designed to be measurable, which means they're designed to be gameable. A real project has requirements that contradict each other, constraints that aren't stated explicitly, and failure modes that only show up when you try to deploy. That's where the rankings stop meaning what they claim to mean.
The field needs more of these, but they're expensive to produce — a real project, documented methodology, reproducible artifacts. Most people don't have the time or the inclination to publish the work rather than just do it. Which is probably why the synthetic leaderboards dominate despite everyone knowing their limits.
Blinding the candidates before review is the step most model comparisons skip, and it's why I distrust writeups where the reviewer already knows the labels. The same brand bias leaks into LLM-as-judge setups, where the judge quietly favors outputs that sound like a frontier model unless you strip the style and randomize position. Did your reviewer see all four at once, or one at a time? Sequential reads tend to anchor hard on whichever came first.
All four (five with the Kimi addendum) were submitted in a single pass — the reviewer read all implementations before scoring any of them. The prompt explicitly instructed Qwen 3.7 Plus to read all files before scoring, and the output confirms it did comparative analysis rather than sequential evaluation.
Your point on anchoring is valid though. Position bias in LLM judges is documented — first position tends to get a slight advantage, and "sounds like a frontier model" is a real confound when the judge has been trained on frontier model outputs. Neither of those was controlled for explicitly.
The multi-reviewer addendum partially addresses it — three different models, presumably with different style priors, converging on the same ranking reduces the probability that any single reviewer's style bias is driving the result. But you're right that randomizing position across runs would be a cleaner design.
For a v3, the right protocol would be: randomize model order per reviewer, run each reviewer independently without cross-contamination, and strip any identifying style markers from the outputs before submission. That's a more expensive setup but a more defensible result. Worth doing if the benchmark becomes a regular publication.
The "not all tokens are equal" point is the actual takeaway, and it's underused in current AI discourse. Planning is cheap. Implementation is expensive. Most teams pay premium model rates for everything because they think in monthly subscriptions instead of per-task economics. Routing by complexity isn't optimization - it's the only sane way to use these tools at scale. Most engineering teams will arrive at this conclusion the hard way after their first surprise invoice.
"Arrive at this conclusion the hard way after their first surprise invoice" — that's exactly how this benchmark started. The Kilo Code blog published the same day as this article and cited Uber burning through their entire 2026 AI budget in four months on Claude Code and Cursor. $1500/month cap per employee, mid-year. That's the enterprise version of the lesson.
The subscription mental model is the trap. It hides per-task economics behind a flat rate until the flat rate disappears — and on June 1st Copilot made it disappear for everyone. At that point teams discover they've been routing everything to the most expensive model by default, with no data on what any individual task actually costs.
The teams that figure this out proactively build the routing layer before the invoice arrives. The others build it after. Either way they build it — the only question is how much they overpay in the meantime.
CyberPanel's bait-and-switch approach is exactly why I stopped trusting freemium control panels entirely. When free features break right when a paid alternative appears, the pattern is hard to ignore.
The pattern is the tell. One broken feature could be a bug. Two broken features in the same release cycle, both with paid alternatives, stops being a coincidence.
CyberPanel isn't unique in that playbook — it's just more obvious about it than most. The move to Caddy + scripts was partly about performance, but mostly about never being in that position again.
The finding that none of them asked questions before planning is quietly the most important result here. We optimize for output speed, not for thinking time.
That line stopped me too when I was writing it up. Every model produced a complete architecture, then asked the blocking questions — in the wrong order. It's the most consistent failure across eight combinations, regardless of price or model family.
"We optimize for output speed, not for thinking time" — that's the root cause. The training signal rewards producing something, not pausing to clarify. An architect who delivers blueprints before understanding the constraints isn't faster, they're just earlier to be wrong.
The uncomfortable follow-up question: how much of our own engineering culture has the same problem?
Exactly. I grabbed this exact thread and pulled — that's basically my whole series. Before AI, during AI, after AI? Same pattern, different packaging. We never learn, we just get faster at being wrong.
So true!
Some comments may only be visible to logged-in visitors. Sign in to view all comments. Some comments have been hidden by the post's author - find out more