DEV Community: Werner Kasselman

What's new in llm-cli-gateway

Werner Kasselman — Tue, 19 May 2026 04:27:38 +0000

A few weeks ago I wrote Why CLI Wrapping Beats API Proxying for Multi-LLM Development, the case for spawning claude, codex, and gemini as child processes instead of proxying to their APIs. Three things have changed since I published that piece. Two of them fix real limitations I named at the time, and one of them is a new capability that I wish had been there from the start and I think it's worth a follow-up.

Codex sessions are now real, not bookkeeping

In the original post I said llm-cli-gateway uses real CLI continuity flags, "--continue and --resume, not bookkeeping". That was true for Claude and Gemini. For Codex it was, frankly, not quite there.

Codex did not have a documented resume mechanism at the time. So when you opened a Codex session through the gateway, the session record was real (UUID, created/lastUsed timestamps, the active-session-per-CLI invariant) but the codex process itself started fresh on every request. The gateway tagged subsequent requests as belonging to a session, you could see the session in session_list, but Codex did not know that.

Codex shipped exec resume <session-id> and exec resume --last, and the gateway now wires both. If you pass a real Codex session UUID (the kind that lives in ~/.codex/sessions/), codex_request invokes exec resume and you get genuine continuity, the same tool-use history, file context, and partial work the CLI itself preserves. resumeLatest: true pins to the most recent session without you having to look the UUID up.

Two caveats worth naming up front. First, only real Codex UUIDs are accepted, gateway-issued gw-* IDs are rejected on resume, because there is no Codex-side session for them to attach to. Second, --full-auto is dropped on resume, which is a Codex constraint and not something the gateway can paper over. The trade-off is reasonable, in that you keep the continuity, but need to restate the approval policy.

Codex now sits where Claude and Gemini sit. The bullet that said "Session continuity using real CLI flags, not bookkeeping" is now true for all of them.

Grok makes four, on purpose

xAI shipped an official Grok CLI (the grok-build TUI) and I added it as the fourth provider. The tools mirror the others one-for-one, grok_request and grok_request_async, sessions through --resume / --continue, model registry entries, self-update via grok update, the same circuit-breaker and approval-gate plumbing, the same flight recorder, the same metrics. Auth follows the same shape, a prior grok login (OAuth) or a GROK_CODE_XAI_API_KEY environment variable, with GROK_DEFAULT_MODEL, GROK_MODELS, and GROK_MODEL_ALIASES all honoured.

The interesting question is not whether to add Grok (the parity work is mechanical) but why. The case is consensus diversity.

Claude, Codex, and Gemini cover Anthropic, OpenAI, and Google. That lineup is well-suited for parallel review work, but it is three of the same kind of organisation, three model families that share a lot of training data lineage and a lot of post-training tendencies. When you ask all three to red-team the same change, the disagreements are real, but the agreements are sometimes less informative than they look, because you are sampling three points from a narrower distribution than the org names suggest.

Grok's training lineage sits outside the OpenAI/Anthropic/Google adjacent triangle. So when a four-way consensus check returns 4/4 agreement on a security finding, the signal is stronger than 3/3. And when Grok dissents alone, that is a data point worth reading, not a vote to discard. The value is not that Grok is better at reviews than the others (I do not believe that, and the workflows do not assume it). The value is independence.

Durable job results and auto-dedup

This is the change that came from running the gateway against real work for a few months and watching the same failure happen over and over.

The original architecture had a soft spot. Async jobs run long, sometimes longer than the orchestrating agent's polling window. The agent gives up, reissues the request, and the whole Codex or Claude invocation starts over. The CLI work you just paid 90 seconds for is thrown away and replaced with a second 90-second run that does exactly the same thing. I lost track of how much wall time this cost me before I sat down and fixed it properly.

The fix is two pieces, both wired into the existing flight recorder SQLite database at ~/.llm-cli-gateway/logs.db:

Every async job persists to a new jobs table on every state transition (start, throttled output flush, completion). llm_job_status and llm_job_result transparently fall back to the durable store when the in-memory job is gone, so a caller can collect a result regardless of how long ago the work finished. Retention defaults to 30 days, configurable via LLM_GATEWAY_JOB_RETENTION_DAYS. Jobs still "running" when the gateway stops are marked orphaned on next boot, and the partial output stays readable.
Identical requests within a dedup window short-circuit onto the existing running or completed job. The default window is 1 hour, configurable via LLM_GATEWAY_DEDUP_WINDOW_MS. The "polling timed out, reissue, run it all again" loop is structurally gone. For the case where the prior result is actually wrong and you want a fresh invocation rather than a re-attach, every request tool accepts forceRefresh: true.

The change moves the gateway closer to what I wanted it to be from the start, a durable result-collection layer for CLI agents rather than a thin process spawner that hopes the caller is still listening when the CLI finishes. 20 new tests cover persistence, dedup, restart-orphan, retention, and Grok parity, and the full suite passes at 322 tests.

What this changes about the original argument

Nothing, actually. The thesis from the first post still stands, that CLI wrapping gives you capabilities (real file access, real test execution, real session state) that API proxying fundamentally cannot. These three updates strengthen the same case rather than contradict it.

What they fix is the gap between the thesis and the implementation. Codex sessions now carry the same real-CLI continuity as Claude and Gemini. The consensus pattern now has a fourth, vendor-independent voice. And the long-running-job failure mode that always threatened to undercut the whole CLI-spawning approach is gone, because the result lives on disk regardless of who is or is not still polling for it.

If you are evaluating llm-cli-gateway against an API proxy, the comparison is slightly different now than it was in March, on three specific axes. That seemed worth writing down.

What's next?

Mistral shipped Mistral Vibe — their official open-source CLI coding agent, powered by Devstral 2. Will be adding it next for even more diversity!

llm-cli-gateway is MIT licensed. npm: llm-cli-gateway | GitHub: verivus-oss/llm-cli-gateway

Here's what stopped breaking, when you make LLM agents author in two formats

Werner Kasselman — Wed, 06 May 2026 04:39:27 +0000

LLM agents will happily produce a thousand lines of plausible Markdown describing work that doesn't compile, isn't tested, and contradicts a decision the same agent wrote down two files earlier. If you want to review their output without re-reading every paragraph, some of the work product has to be machine-checkable.

You also can't push everything into a schema. Intent, tradeoffs, the alternative you rejected: that material dies in JSON. The interesting question is the boundary. What belongs in prose, what belongs in structure, and what falls out when you draw the line in the wrong place.

I landed on this after running it for real. I introduced the runtime layer later, when I expanded this to multiple repos, and saw the flat files stopped scaling.

The split

Every unit of agent work produces three things:

Narrative. Markdown specs, designs, plans, notes. The human-readable record: intent, tradeoffs, what was rejected, context a future reader needs.
Structure. TOML files encoding the work itself: a dependency DAG, a traceability map (INT → FEAT → REQ → DEC → IMP → CODE → TEST → OUT), and a review-readiness bundle.
Evidence. Review artifacts that answer "is this actually reviewable, and does the claim match the proof?"

Markdown carries what structure can't. Intent and reasoning. Why the design has this shape. What was rejected. What the author worried about. Schema fields can't express ambivalence. Specs change during brainstorm and review, and prose is the right medium for that conversation; forcing every change through schema churn throttles thinking. Six months later, the reviewer needs narrative, not a graph.

TOML carries what prose can't reliably:

Machine-checkable invariants. blocks is the exact inverse of depends_on. Every ART: has exactly one producer. Every consumes matches a produces. These are enforced by validators, not by hoping a human noticed.
Graph queries. What's ready to start? What's the critical path? Which units conflict on files? Which REQ: has no downstream TEST:? These are queries over structure, not reading comprehension.
Stable identifiers. Prose drifts. U07a, REQ:auth-001, ART:schema-v2 don't.
Diff-readable state. A status transition is a one-line diff, not a paragraph to re-read.

Frame the split as narrative vs. structure, each in the medium that protects its own invariants. Calling it "docs vs. config" gets it wrong because both formats are doing real review-time work; one of them just gets to be checked by python -m.

Why TOML and not YAML or JSON

I picked TOML deliberately. YAML loses on parse ambiguity. The country: NO problem (Norway gets parsed as the boolean false under YAML 1.1) is real and gets worse when an LLM is generating the file under time pressure. JSON loses on the human-authoring axis: trailing commas explode, every string needs quotes, comments are forbidden. TOML parses unambiguously, reads cleanly enough to author and review by hand, and ships in the Python stdlib (tomllib since 3.11), so my validators stay dependency-light.

For agent-authored, human-reviewed structure, TOML is the boring choice. It wins because it's boring.

The three review pillars came from failure data

The review-readiness package didn't exist on day one. I added it after running an iteration-chain analysis across seven real review cycles and finding that almost every re-review came from one of three deficiencies, in the same order, over and over.

Missing prerequisite artifacts. Review blocked not on conceptual disagreement but on the absence of required planning docs, cross-links, prior diagrams, or test plans. The reviewer couldn't judge readiness because the artifact class wasn't actually complete.

Ambiguous contracts. Ordering rules, normalization, precedence, fallback, schema shape: reviewers had to infer semantics the author never wrote down. Every inference round added a re-review.

Overclaimed completeness. "Ready for implementation." "Production ready." "All findings resolved." Unbacked by proof, or backed by proof narrower than the claim. Each one cost another round.

Three failure modes, three artifacts. A readiness gate answers whether the artifact class is complete enough to review at all, and blocks opening a review until it passes. A contract declaration makes behavioral semantics explicit up front so reviewers never have to invent them. An evidence matrix binds every strong claim to a concrete proof artifact, a stated scope, and a list of known exclusions; a claim broader than its evidence fails validation.

The workflow is strict and intentionally rude. Fill the readiness gate first; if blocked, don't open review. Fill the contract second; vague statements get rejected. Fill the evidence matrix last; if a claim can't be backed by proof and bounded exclusions, downgrade the claim. Don't stretch the proof.

The validator's exit code is authoritative. No human override of a failed validation without updating the file to pass cleanly. I made this rule on purpose, because "it's close enough" was the phrase that caused most of the re-reviews I measured.

Where flat TOML stopped working

Flat TOML works great for authoring and validation. It stopped working the moment agents started mutating state during execution.

The hand-calculated [computed] sections were the first thing to rot. Critical path, conflict groups, progress percentages: all derived values, all authored by hand, all stale the moment a unit advanced. A human spots the inconsistency on re-read. An agent doesn't.

Editing status = "in_progress" in a text file leaves no record of when, by whom, from what prior state, against what evidence. For process control, "who moved this to done, and on what proof?" is not optional.

There was no programmatic query layer either. "Which tier-1 units are runnable right now?" required parsing TOML, walking the graph in Python, and rebuilding the same derivations every time.

And flat files don't compose across a fleet. Once more than one repo is under the same policy regime, per-repo TOML is the wrong shape for fleet-wide gating, policy packs, exception lifecycles, and release trains.

So I added a runtime layer, additively. The templates and validators didn't change.

A per-repository runtime imports a filled TOML file once. After that, an embedded SurrealDB is the source of truth. Status transitions go through a typed API with validation. Every change persists with timestamps and actor identity. Computed values become live queries instead of hand-edited fields. You can still export a TOML snapshot for human review, but it's a derived artifact, not the authority.

A fleet-wide control plane (FastAPI + Postgres) handles policy packs, signed snapshot intake, exception lifecycles, and release-train readiness across many repos. There's no flat-file counterpart; the multi-repo problem just isn't expressible in per-repo files.

The practical rule: TOML is the authoring medium and the interchange format. The database is the runtime authority. The TOML file you imported is stale from the first state transition onward. Treat it like a git tag — a snapshot in time, not live state.

What you actually get

Four things, none of which prose-only or structure-only would deliver alone.

Parallel agent execution without stepping on each other, because the DAG encodes depends_on, blocks, and files_modify conflict groups explicitly. Agents pick runnable units from the same layer and the system knows who may run concurrently.

Traceability from intent to test. Every requirement has a downstream realization path through implementation, code, and test. Unverified requirements and unmapped code surface as computed gaps in a query, not as gut feeling six weeks into review.

Reviews that fail at the right boundary. Readiness gates block un-reviewable work before a reviewer sees it. Explicit contracts stop the semantic-inference spiral. Evidence matrices stop overclaimed completeness from reaching review at all.

State that is queryable, auditable, versioned, and composable across repos. Single-repo: "what's ready now?" in one query. Fleet-wide: "is this release train green across every repo under policy?" — also one query, against the control plane.

Operating rules

Distilled from getting this wrong before:

Author narrative in Markdown. Author structure in TOML. Don't mix.
Validator exit code 0 is the only pass signal. No manual override.
Don't edit state fields by hand once they're in the runtime. Use the API.
Don't claim "complete," "production-ready," or "all findings resolved" without an evidence matrix. If the matrix is thin, the claim is wrong.
When behavior depends on ordering, fallback, normalization, precedence, or authority, write the contract before review, not during.
Computed fields belong to the runtime. Don't hand-calculate them.

Worked example: this article

I dogfood the same split. The Anti-AI-Tell style guide (mr-k-man/llm-tips on GitHub) is Markdown: rationale, evidence base, the prose rules humans read. The matching contract is TOML — 49 machine-checkable rules with regexes, density thresholds, and applicability tags. And the audit workflow is a 10-unit DAG, also in TOML, that orchestrates inventory, scan, triage, fix, and regression as discrete units that run in parallel where the dependency graph permits.

I ran the DAG on this article before publishing.

The pre-fix audit found two hits:

AIS:ST02 structural: tricolon-fraction 60% (3 of 5 single-token enumerations were three-item).
AIS:F03 formatting: inline-bold density 1.43 per 200 words (10 bolds in 1398 words; budget 7).

Weighted score: 1.0 + 0.25 = 1.25. The rewrite threshold is 3, so this routed to surgical-edit, not rewrite-from-scratch.

Three line-level edits:

Stripped four bullet-label ** markers in the "TOML carries..." list. The bullets already carry the structure; the bold was decoration.
Expanded a three-item prerequisite-artifacts list (docs, cross-links, test plans) to four by adding "prior diagrams".
Expanded a three-item adjective list (queryable, auditable, composable) to four by adding "versioned". The added word is true: the runtime persists history.

Regression scan: zero hits. Tricolon fraction 1 of 5 (20%, under the 30% threshold). Bold density 0.86 per 200 words (under 1.0). Linter exit 0.

You're reading the post-fix version. Everything is in mr-k-man/llm-tips on GitHub: the source guide at style_guide.md, the contract at tools/style_policy.toml, the linter at tools/lint_writing_style.py, and the audit DAG at tools/audit_dag.toml. MIT-licensed.

Takeaway

If you put LLM agents on real work, decide which invariants you want a validator to enforce and which you want a human reviewer to negotiate. Draw that line on purpose. Then accept that flat files have a ceiling: the moment your agents start mutating state, something has to own the audit trail and the live derivations, and a text file isn't it.

Narrative carries judgement; structure carries invariants. Force either of them to carry live state and you'll lose the audit trail inside a week.

the next software stack needs more than code generation

Werner Kasselman — Wed, 22 Apr 2026 04:55:17 +0000

Most people in software are staring at the wrong milestone. Models write API handlers, unit tests, and migrations fast enough that typing isn't the limiting factor anymore. In a world of high-concurrency agents, the act of writing code is no longer the bottleneck. That part of the problem is finished.

The real trouble starts the moment that code lands. Why was this change made? Which requirement forced it? And who actually checked the risky paths in the auth flow? You can still answer those questions today, but it takes a kind of technical archaeology—digging through PR threads, Slack messages, and documentation that was out of date the day it was written. That workflow held up while humans set the pace. It breaks the moment you stop being the bottleneck.

the velocity trap

Most teams run AI-assisted development through a loop of prompt, branch, code, review, and merge. At low volume, it holds up. Then usage increases. You start seeing changes that look fine but carry no clear origin story. A feature flag shows up in production with a name nobody recognizes. An environment variable gets added "just to make something work" and stays there for six months because nobody is sure what it’s gating.

Then we have a growing crowd of "psychosis coders" who think they are shipping masterpieces because they saw an agent move a cursor. They hit approve the second the diff looks plausible, never noticing the trail of empty TODO comments, shallow mocks, and tests that don't actually assert anything meaningful. They are shipping "passable" trash masquerading as velocity.

Maintaining real quality at agentic speeds requires a gauntlet. In my own work, I have to run Model B against Model A like a caffeine-fueled nitpicker for ten rounds just to reach consensus. Then Model C does the same dance. This cross-model review is mandatory to maintain velocity without the system collapsing into a pile of actual slop.

But even this gauntlet is a patch, not a solution. We are burning a mountain of tokens to force quality through a pipe that was never meant to handle it. This is "Approval Theater" as a survival strategy. No, your carefully crafted markdowns, prompt engineering nor harness stacking solves this.

why clean merges still fail

Agent A updates PricingEngine::price() to apply a discount based on User::join_date.

Agent B removes join_date from User and introduces a UserMetadata lookup that returns Option<NaiveDate>.

The pricing path now depends on a value that may not exist. In the failure case, the lookup returns None, and a later fallback resolves that missing value to Money::default(), producing 0.00.

Both changes compile. Both pass their unit tests. Because they don't touch the same lines of code, Git merges them without a single conflict.

In production, the pricing logic fails. Revenue doesn't drop to zero. That would be obvious. It becomes inconsistent instead. Some users are charged correctly. Others hit the missing metadata path and get a zero price. Support tickets appear first. Finance notices the reconciliation mismatch three weeks later.

You're left trying to unwind two changes that were never evaluated together. Each was correct in isolation; the failure only existed in the interaction. A human developer might have caught that by holding the context in their head, but that assumption doesn't scale when dozens of agents are moving at once.

the idempotency crisis

There is a deeper, uglier problem with agents and Git: retries. When a prompt fails or a network timeout hits, an agent often tries again. In a standard Git flow, this leads to double-commits, "dirty" working directories, or a messed-up HEAD state that requires a human to untangle. Then come additional worktrees and agents not checking if they're on the right branch in the right tree, or simply sticking to documentation paths you've specified instead of pollution the root with markdowns.

Git wasn't built for idempotent operations from a thousand concurrent workers. It was built for a human at a terminal who can see when a command failed. If the next stack doesn't have request-level idempotency built into the storage layer, you aren't building a system; you're building a race condition.

files are the wrong primitive now

Git shows you what changed in the text, but it doesn't show you why. You see two files modified, but you can’t see the requirement that triggered the edit. We review diffs and guess at intent.

Agents don't operate on files; they operate on relationships. A discount rule depends on a user attribute; a billing flow depends on an auth decision. When we take that rich graph of intent and flatten it into files, we lose the fidelity of the work. This mismatch leads to "clean" merges that are semantically murky, repeated edits to the same symbols, and retries that converge on something other than what we actually meant to build.

building the floor

I'm building a stack that treats intent as the primary object, not the diff. It's not one tool. It's a set of components doing work Git was never designed for.

aivcs is the version control core: a 9-crate Rust workspace. It uses blake3 for content-addressed hashing and groups changes around intent as an Episode instead of scattering them across commits. An Episode carries the requirement that triggered the change, the symbols actually touched, and the evidence (tests, benchmarks, profiles) attached when the work lands. It can import Git history as a baseline and export structured Episodes back into a branch, so teams don’t have to migrate all at once.

trstr is the parsing layer. It’s spec-grounded, not grammar-by-example. When an agent edits a symbol, the system knows what that symbol is, not just which bytes moved. Tree-sitter is built for editor features. This needs stricter guarantees.

sqry handles symbol-level indexing. It builds the graph from a rule like “apply a legacy discount” to every call site, call chain, and dependent type that touches it. That’s what lets an Episode carry semantic scope instead of a file list. It’s also how you catch the PricingEngine / UserMetadata class of failure before merge.

wsmux is the concurrency layer: a CRDT over the code graph. When dozens of agents edit the same repository, the merge surface isn’t text. It’s operations on symbols and relationships. wsmux makes those edits converge instead of producing two clean merges that disagree at runtime.

The storage layer is idempotent by construction. The same operation with the same content and intent resolves to the same Episode. Retries don’t duplicate work. A thousand workers hitting a flaky network stop being a race condition.

This doesn’t replace Git. It sits alongside it.

The goal is simple: when something changes, you can answer why without digging through history. Decisions travel with the change. Evidence is attached when the change is made, not reconstructed later.

The system remembers what changed. It should also remember why.

The bottleneck moved. The stack didn’t. That gap is where the risk lives.