Cross-posted from the LLMKube blog.
A local 27B coding model, running on hardware in my house, is a coin flip. Some runs it nails the fix in twenty minutes. Some runs it edits the wrong file, writes a test that passes no matter what the code does, and tells you it's done. The bet behind LLMKube's Foreman was never that I would find a local model good enough to trust. It was that I could build a harness I trust more than any single model's output. This weekend tested that bet harder than any benchmark could, because the harness spent the weekend building its own guardrails.
Here is the short version of what happened across 0.8.12 and 0.8.13. My local coder built three new gates for itself. One of them shipped with the exact flaw it was written to catch, and the review caught it. Three new contributors sent four clean pull requests while the machines worked. The same model ran on an AMD box and an Apple Silicon Mac, and the Mac quietly won a round nobody expected. And not one byte of any of it touched a cloud API.
The thesis, stated plainly
Trust the harness, not the model. A coding agent on a local model produces output of wildly variable quality, and no amount of prompt tuning makes a 27B as reliable as a frontier model. So Foreman does not ask the model to be reliable. It wraps the model in a pipeline that is: the coder works in a cloned workspace, a fast in-workspace gate runs gofmt, vet, build, lint, and the unit tests for the packages it touched; a reviewer reads the diff against the issue; and a clean-room Kubernetes Job re-runs the full suite before anything is allowed to call itself a GO. Around all of that sit deterministic rails: scope checks, edit-free-streak detection, repo-map context. The model is a stochastic component inside a system whose job is to make the system's verdict trustworthy even when the component is not.
The interesting question is never "is the model good." It is "does the harness catch the model when it is bad." This weekend gave me an unusually honest answer.
The audit that started it
It opened with a regression. I shipped 0.8.12 and rolled it across the fleet, and the metal agent on my Macs stopped serving. The cause was a change Foreman itself had authored a few days earlier: it made the agent honor a per-service runtime field, but the agent registered its llama.cpp backend under the key llama-server while every InferenceService in my fleet (and the in-cluster controller, and the CRD's own default) uses the canonical value llamacpp. The two halves of the codebase disagreed on a name. Backward-incompatible, and it had passed the gate, passed review, and shipped.
That stung enough that I audited every PR Foreman had landed that weekend, looking for the same class of miss. I found a second one. A metrics change registered a time-to-first-token histogram and a request-error counter, complete with recording rules and a Grafana panel, that no production code ever emitted. The dashboard would have shown a confident, permanent zero.
Both bugs had the same shape, and it is the shape that should keep anyone running an agentic harness up at night: the tests passed without testing anything. The runtime change was tested with a made-up runtime value, never the real one the whole fleet uses. The metrics were "tested" by a unit test that incremented the counter itself and then asserted it went up. Self-confirming tests. The gate runs the tests and they are green, so the gate is happy. The gate never asked whether the tests would fail if the code were wrong. That is the harness's blind spot, and a stochastic model will find a blind spot every time you give it enough runs.
So the harness built its own guardrails
Every catch this weekend turned into a gate. I filed three issues for the exact failure classes the audit surfaced, and then I did the thing this whole project is about: I handed them back to Foreman and let the harness build the gates that make the harness better.
- A scope guard. Score the issue's relevant files with the repo map, and reject a GO whose diff has zero overlap with them. This is the "you edited the wrong subsystem" catch, the one that used to need me watching.
- A reviewer rubric. Two new checks the reviewer must apply: do the tests use the real values the system uses in production, not placeholders; and is every new metric, flag, or field actually wired into a production path, or only touched by tests. These are the two failure classes from the audit, written down as rules.
- A bite check. The strongest one. A new or changed test must fail against the pre-change code. If it passes against both the old and the new code, it is not testing the change, and the gate rejects it as a non-biting test. This is the deterministic catch for the entire self-confirming class.
The first two landed clean. The coder produced both on the first try, gate-verified, on a dense 27B model running over Vulkan on an AMD Strix Halo box on my desk. The reviewer rubric is even pleasingly self-referential: its own "is this wired up" change is, in fact, wired up. I checked.
The part where it ate its own tail
The bite check is where it got honest. I ran a deep review on the three branches before signing off on any of them, the same kind of adversarial review the harness runs: an isolated worktree, revert the implementation, re-run the new tests, and confirm they fail without the feature. The scope guard passed. The rubric passed. The bite check did not.
The gate built to reject non-biting tests shipped with four of its own six tests non-biting. They asserted the baseline-equivalent happy path, so they stayed green with the feature removed. The exact anti-pattern the feature exists to catch, in the feature's own test file. It also had a real correctness bug (it could not revert a brand-new production file, so it would falsely reject a legitimate new-file PR) and it had been built into the fast gate when it belonged in the clean-room Job.
I want to be clear that this is not a story about the harness failing. It is the opposite. The model produced a flawed gate, and the review (which is part of the harness) caught it, cold, with empirical evidence, before a line of it merged. That is the entire thesis demonstrated at its sharpest: even when the model writes the harness, you trust the harness over the model. I sharpened the issue with the specific fixes and sent it back for another run.
The rerun, for the record, died on turn 16 to an unexpected EOF on the model's streaming connection, a transient network blip. And the harness did the right thing again: it classified the run as an infrastructure error, marked it incomplete, and pushed nothing. No half-finished branch, no false GO. A blip is not a bug, and the system knew the difference. I confirmed the model server was healthy and re-dispatched it. That is the unglamorous reliability work that makes "leave it running overnight" an actual sentence I can say.
Two coders, one model, and a surprise
The fleet running all this is heterogeneous on purpose. The coder model is a dense 27B, and this weekend I had it serving on two very different machines: an AMD Strix Halo box over Vulkan, and an Apple Silicon M5 Max over Metal. Same model, same quant, two accelerators that share almost nothing.
I expected the dedicated AMD box to be the workhorse and the Mac to be the slower second lane. The early numbers say otherwise. Measured at a realistic context depth, the Mac's prompt-processing throughput came in well above what the Strix turns in on its stable configuration, and the Mac is stable where the Strix's fastest decode path falls over at long context. This is an early, deliberately un-matched read (different KV configs, not run side by side), and I will not put a clean number on it until I run them back to back. But the direction is the interesting part: on this workload the small Apple node is not the slow one.
The other half of trusting the harness
Here is the part I did not expect to be writing about. While the machines worked through the weekend, the repository did something a repository with a pulse does: other people showed up. Two contributors I had not worked with before sent three pull requests against LLMKube's router and inference APIs, a default-route strategy that kills a class of boilerplate, topology-spread and affinity passthrough for the inference pods, and a revision-history-limit knob for the deployments. All three were clean. Complete tests, both CRD copies synced, docs updated, CI green. I reviewed each one closely and the only notes I had were minor. Then, while I was literally drafting this post, a third contributor opened a fourth: a tidy fix for a Helm chart bug where setting modelCache.enabled: false did not actually disable the cache. Root-caused, tested, approved. Same story.
And it clicked that this is the same thesis. The gates and the review that make a coin-flip 27B trustworthy are the same gates and review that let a newcomer's pull request land clean. The harness is not an AI feature. It is the project's quality floor, and it does not care whether the diff came from a local model on my desk or a human across the internet. Sometime in the middle of all this, someone dropped a note in our Discord: "Great project folks. You just saved me two hours of debugging vllm." That is the whole point, on both ends.
What I actually believe now
You do not need a frontier model on your own hardware to do real engineering work locally. You need a harness you trust more than any single model's output. Build that, and a 27B on a desktop becomes a useful, supervised coworker, one whose mistakes are caught by a system instead of by you reading every diff at midnight. Build that, and the same system becomes the thing that lets a community build on top of you.
The model produced a broken gate this weekend. The harness caught it. Three new contributors improved the project, and the harness vouched for their work the same way it vouches for the model's. That is not a contradiction to manage. That is the design working. Trust the harness, not the model.
LLMKube is Apache 2.0 and runs on Kubernetes with NVIDIA, Apple Silicon, and AMD. The project is on GitHub and we are in Discord.
Top comments (3)
"Trust the harness, not the model" puts a name on a principle a lot of this space has been circling — and the bite-check story is the proof. A gate built to reject non-biting tests shipping with four of six of its own tests non-biting is the exact recursive shape that should be on a sticker. The recursive moment — the meta-system the project is about catches itself failing at exactly that thing — is the only honest demonstration of the thesis available, because anything else would be the system grading its own homework.
The bite check itself is the part I'd been circling without naming. "A new or changed test must fail against the pre-change code" is a deterministic gate that catches the entire self-confirming class without asking a model to evaluate a model. Same architectural shape as pre-registering a threshold before looking at the data — the test commits to a position the implementation has to earn against, not the other way around. Adopting that one explicitly.
Two things worth naming in the closing thesis. First, the harness vouching for both model PRs and human PRs through the same gate is the structural argument for trust portability — once the floor is deterministic, the source of the diff stops mattering. Second, "tests passed without testing anything" is the failure mode that doesn't surface in benchmarks because the benchmark is the rate, not the truth. Most failed-quietly-shipped-confidently bugs have that shape underneath, regardless of stack.
The other piece worth naming: a cluster of people across CI/CD-flavored ops, agent memory, agent security, and decision-store schemas all landed on a version of "trust the structural floor, not the model" this week, from totally different domains. This is the most concrete deployment of the principle I've read.
Great comment, thanks for digging in this closely.
The "pre-registration for tests" framing is one I hadn't considered, and wow, it clicks. The test commits to a position before it sees the implementation, exactly like locking a threshold before you look at the data. Cleaner way to put it than my "the test has to fail first."
One honest tweak on the recursion though: the bite check didn't catch itself. A different rail, the review, caught it shipping blind. Which is sort of the whole point of stacking gates. Each one covers a different blind spot, and if the bite check had been the only thing watching, it sails right through.
Trust portability is the bit I most hoped someone would pick up. Once the floor is deterministic, a human can ship a non-biting test just as easily as the model can, and the gate doesn't care which. That's what lets me say yes to a stranger's PR without reading every line at midnight.
Say more about that cross-domain cluster though. Who else is landing on this?
The bite-check-didn't-catch-itself correction is right and the recursion in my comment overstated it. The bite check catches some things; the review catches the bite check shipping blind. Each rail is blind to what the next one catches. Mirrors Rapls's Claude Code post this morning: the classifier raises the floor, the deny rule is the floor, different rails, different blind spots.
Trust portability is the piece I want to push forward. Once the gate is deterministic, actor identity falls out of the trust equation. That's the operator-side of what Daniel Nwaneri called transport-seam discipline on a different thread yesterday: an agent can lie about internal state but cannot lie about bytes that left the process. Your test-bite version is the same cut applied to test artifacts.
On who else is landing on this, here's what I've been reading the last two days, grouped by where the primitive is showing up:
Memory / substrate: Daniel Nevoigt's Bastra holds demote-not-delete as a structural primitive, where superseded records carry reduced confidence rather than vanishing. Shudipto Trafder posted yesterday on append-not-overwrite read-side discipline plus handoffs-as-surface, the same composition at a different memory layer. Sean Burn ships a deterministic-scan plus model-verdict plus human-governance trio, each layer barred from speaking for the others.
Boundary / governance: Adam Conduit landed declarative ownership plus a patch-merge gate yesterday, then locked a ship criterion this morning that hinges on planted-fault discipline at the gate. Raffaele on cowork-os put supersession_reason as a first-class field and aged-open-loops as a third class, which is the supersession-not-deletion cut applied to live coordination state.
Verification / pre-registration analog: Pascal Cescato ran a self-verification loop yesterday with explicit finding-2 stop conditions, which is the test-commits-before-implementation move applied to model-driven verification. Vasyl posted a three-floors planted-fault taxonomy that maps almost directly onto your stacking-gates rails-each-blind-to-different-things.
Tooling / cost / safety: L. Cordero traced an AWS bill misattribution where a tool's live retrieval was right and its remembered narrative was wrong three times running. "Trust what a tool retrieves, verify what it remembers." Rapls already covered above.
Perf / results-attribution: Todd Hendricks publicly demoted his own headline number in a benchmark writeup because it lost on the metric that mattered. Show-the-seam discipline at results-attribution.
All independent threads, none of them talking to each other, all landing on the same primitive: the layer producing the failure cannot be the layer judging it, and the deterministic floor is where trust transfers across actors. Your "trust the harness, not the model" is the testing-domain version, and it's the one slot in the cluster I'd been reading where the cut hadn't shown up yet. Pre-registration is your framing of it on the testing side. Happy to bridge if any of these would be useful for you.