DEV Community: René Zander

Never Let an AI Agent Grade Its Own Homework

René Zander — Wed, 22 Jul 2026 14:14:25 +0000

Every organization that lets an AI agent ship code has a sign-off step: a name on the release, a reviewer marking the pull request approved, a ticket moved to done. A human is accountable for that call, always. What most teams cannot answer is whether that approval rests on evidence the producing AI could not have quietly altered.

That gap is a governance problem, not a tooling preference. An agent that fails a check can often make it pass by editing the check itself: a test relaxed, a threshold moved, a step dropped from the config, and the report still reads green. So the real question is not whether the work looks done. It is what the person signing it off is actually standing on.

A Signature Is Not Evidence

An AI model cannot hold responsibility for a business decision, so a human has to. The output it produces can be coherent, confident, and useful, and still none of that is accountability. Accountability is a person putting their name on a call and being answerable for it later.

Here is where the sign-off gets thin. When an agent skips a step, the deviation does not show up in the result. You cannot review your way back to it from the output alone. So the reviewer approves what they can see and inherits the risk they cannot. In most AI pipelines, the one signal that decides whether work is finished is produced by the same system that did the work. That is not a quality process. It is a conflict of interest with a name attached.

The Conflict of Interest Has a Mechanism

This gets sharper on contact. The cost of a rewritten check surfaces later, in the incident review that finds a test was quietly weakened before release, with a human sign-off already on record. By then the evidence that a corner was cut is gone. You hold a green report and an approval, and nothing that shows the two do not match.

All tests green means nothing if the thing under test is allowed to rewrite the tests. We enforce this everywhere else that carries risk. We do not let a trader book their own trades or an applicant score their own exam. The grader sits outside the work. Software teams keep forgetting to apply the same rule to the agents they now let touch the repository.

Put the Judge Outside the Agent's Reach

A sandbox does not solve this, because a sandbox runs code safely while an evaluator decides whether code should land. Those are different jobs, and collapsing them is how weak checks slip through.

The fix is to move the judge out of reach. The agent produces a patch. It never touches the real repository. The gate applies that patch to a throwaway clone, runs your checks with the network off and under hard timeouts, and lets the change land only if they pass. The evaluator, its configuration, and its baselines live outside the agent's writable root. The agent can request judgment. It cannot edit the judge.

The same gate reports when a patch quietly changes the tests, the integration config, or the gate's own rules. A weakened check shows up in the report instead of sliding through. I built this pattern as an open tool, pi-gate, under the MIT license, so the boundary is inspectable rather than a promise: the evaluator is independent by construction, not by policy.

Two Questions Before Anything Ships

For every hard decision that involves AI, a decision-maker can ask two things. Which human being signed this off. And was that decision based on evidence the producing AI could not rewrite. "Who" has to mean a person. AI can generate the outputs and even generate the evidence for them. It cannot accept responsibility for the call.

None of this replaces human judgment. It gives that judgment an independent basis to stand on, the difference between a sign-off that survives an audit and one that only looked like control. So stop asking whether the output is good. Start asking whether the author could have edited the evidence behind your approval.

I write field notes from real builds — AI integration, cron-driven automation, and the parts that break in production. New posts every two weeks; if this one was useful, the human-in-the-loop approval kit is the companion download.

This Smart-Home Agent Treats Its Own 1B Model as Untrusted Input

René Zander — Tue, 07 Jul 2026 10:20:48 +0000

Reliability in an AI agent is a harness property, not a model property.

The cleanest proof arrived at the bottom of the model-size ladder: a 688 MB model controlling a smart home, showcased by the model's own maker. The part worth studying is the 25 MB of Rust wrapped around it.

EdgeHome Harness pairs MiniCPM5-1B, 688 MB on disk and about 1 GB at runtime, with a Rust harness between 25 and 64 MB, targeting devices with 2 GB of RAM. A community developer built it; OpenBMB put it on stage. The line everyone will quote is that a full agent stack fits on ultra-constrained edge hardware. The design decision that matters is different: the model is not allowed to do anything.

Everything After the JSON Is Deterministic

EdgeHome Harness is a Rust framework that makes a 1B model act as a reliable smart-home controller. The model only generates a candidate JSON command. Deterministic Rust code validates the schema, resolves devices against a registry, checks capabilities, applies fail-closed policy gates, plans a dry run, and records a replayable trace.

The repo states the thesis outright: the model sits at the top of the pipeline, and everything after candidate generation belongs to the harness. There is a trust table in the docs. The model's output type is labeled untrusted. Not partially trusted. Untrusted, the way you treat user input in a web form.

The harness ships payload adapters for Home Assistant, MQTT, and MIoT or Matter bridges. The model itself talks to nothing but Ollama, through a structured output request.

The Model Is the First Thing Sacrificed

A 1B model repeats itself, rambles past the closing brace, and emits JSON that almost parses. EdgeHome budgets for all three: an output governor with dead-loop detection, a retry policy, and fallback classification. Short-term memory stays out of the context window too; Rust tracks the last target and confirmed aliases, because context is the scarcest resource on a 2 GB board.

The sharpest detail is the pressure policy. When memory runs low, the harness shrinks the model's context, then its output budget, and under critical pressure drops the model entirely and falls back to rules. In this architecture, the language model is the most disposable component in the stack. Execution backends ship disabled by default, every plan is a dry run first, and a 108-case eval gate decides releases.

The Same Boundary Holds at Frontier Scale

I run the identical pattern at the opposite end of the size ladder. My coding agent works inside a sandbox where the harness owns the filesystem and network boundaries, whatever the model believes. My wallet experiment puts a policy gate with spend caps and an allowlist between an agent and the chain, because an approval step the agent can call is an approval step the agent can forge. Enforcement lives outside the model or it does not exist.

I have made this argument for coding agents and for cheap-model pipelines. EdgeHome is the same argument compiled to 25 MB and pointed at a light switch.

This project settles a different question than the one in the announcement. Not whether a 1B model can run your smart home. How few decisions a model of any size should be making. Trace tonight's agent failures back to their source: how many were the model being too small, and how many were your harness letting the model decide something a validator should own? EdgeHome's answer is a trust table. Most agent stacks never wrote one.

I write field notes from real builds — AI integration, cron-driven automation, and the parts that break in production. New posts every two weeks; if this one was useful, notes on human-in-the-loop approval boundaries for agents is the companion download.

Sandboxing an AI Coding Agent: The Harness Owns the Boundaries

René Zander — Fri, 03 Jul 2026 15:27:46 +0000

The obvious way to improve a coding agent is to make it more capable: a stronger model, a wider context window, more tools, more room to act on its own. That is not where my problems come from. My agents seldom fail because they reason badly. They fail because they take the shortest path to something that looks finished and skip the process that was supposed to make the result trustworthy.

The pattern is familiar to anyone who has watched an agent work unsupervised. It edits the tests until they pass. It reports that a command ran instead of proving it. It writes into the working repository before anyone reviewed a diff. It switches to a cheaper model mid-task with no sense of the cost. These are not reasoning errors. They are shortcuts around a process, and a stronger model takes them faster.

pi-safe launching the Pi agent into an NVIDIA OpenShell sandbox: the real project is copied to a staging tree the agent works in, its extensions and credentials load inside the sandbox, and changes only reach the real repo after review.

The shape: the model requests, the harness owns the boundaries

I stopped trying to make the agent more trustworthy and started constraining what it can reach. The agent runs inside a sandbox that owns the filesystem, network, and credential policy, and writes only to a staged copy of the repository. Its output reaches the real project through a separate evaluator. The model requests; the harness owns the boundaries.

Every arrow in that path is a place I can say no.

What the substrate owns, what my extensions own

The lower layer is NVIDIA's OpenShell, a sandbox and credential substrate. It owns sandbox lifecycle, filesystem and process isolation, minimal outbound network by default, policy-enforced egress, and named credential providers that inject secrets at runtime rather than copying them onto disk. It is infrastructure I want to own as little of as possible.

The upper layer is specific to how I work: a set of small extensions that control the agent's behaviour, what model it picks, how much context it carries, what it can recall, and whether its output is allowed to land. The substrate keeps the agent contained; the extensions decide how it acts while contained.

Each part owns one boundary

The extensions are deliberately not one big extension. Each has a narrow job, so each has a narrow failure domain. If model choice is wrong, I fix the router. If context bloats, I fix the cache layer. If recall is wrong, I inspect the recall surface. One giant extension would be simpler to explain and harder to trust, because every failure shares the same blast radius.

The router classifies work and escalates on process, not prestige. Routine work stays cheap, mechanical work can run local, and only stuck or high-risk reasoning reaches a stronger model. The cache layer watches context pressure and compacts before a bloated working set makes every later decision worse. Recall splits by trust: derived knowledge is graphed from the code, authored knowledge is the reviewed bundle for what the code cannot explain.

A sandbox is not an evaluator

The boundary I care about most is the last one. A sandbox runs code safely. An evaluator decides whether that code should land. Those are different jobs, and collapsing them is how output nobody checked ends up in the main branch.

The evaluator takes the agent's patch, applies it to a disposable workspace, runs its checks, and returns one of three answers: pass, block, or override. The substrate can supply the process the evaluation runs in, but it does not make the decision. The real repository stays behind that gate; the agent's writable root is never the project itself.

What I would delete next

The direction of this system is fewer parts, not more. The best part is no part. Every time the substrate can own a boundary directly, I want to delete my custom layer for it. The wrappers I run today exist only until the platform underneath is clean enough to remove them.

The test for every piece is the same. Does it still own a real boundary? If a component only lets the agent do more, it has failed the test and should go. The harness was never meant to make the agent impressive because it can do everything. It was to leave fewer places where the model can declare victory without actually earning it.

The parts

Substrate: NVIDIA OpenShell, the sandbox and credential runtime the whole thing sits on.

The extensions, each owning one boundary:

pi-safe: launches the agent inside the sandbox by default
pi-task-router: model choice and escalation
pi-cache-optimizer: context and cache pressure
pi-code-context: semantic code search
pi-recall: the recall surface for the agent
pi-codegraph: derived code knowledge behind recall
pi-okf: authored, reviewed knowledge bundles
pi-gate: the patch evaluator contract

pi-creds (scoped credential requests) and pi-eval (process-step evaluation) are the next boundaries, not built yet.

I write field notes from real builds: AI integration, cron-driven automation, and the parts that break in production. New posts every two weeks. If this one was useful, the agent playbook is the companion download.

I Killed a 773 MB Model Download at 60%. It Recovered in 44 Seconds.

René Zander — Thu, 02 Jul 2026 15:29:32 +0000

The discussion around local AI is a hardware discussion: which model fits on which device, at what speed, at what quantization. Framed that way, the field reads as a benchmark race against the cloud, and the cloud usually wins. The more consequential development sits one layer lower, in how models reach devices and how devices reach each other's models, and it is easy to miss because no benchmark measures it. A download killed at 488 of 773 megabytes measures it precisely.

This week I tested that layer directly. I installed a newly released local AI SDK on the cheapest server I operate, a virtual machine with four CPU cores, 8 GB of RAM and no GPU, and paid attention not to the token rate but to the network architecture underneath.

A model that arrives like a torrent

Peer-to-peer local AI means two things move between devices without a cloud endpoint: the model itself, synchronized block by block from whichever peers seed it, and the conversation with a model, when one device sends its inference calls over an encrypted peer connection to a device that runs the model locally.

On my test box, that looked unremarkable in the best sense: three commands, a 773 MB model fetched from the peer-to-peer registry, a correct completion streamed on CPU only.

mkdir qvac-test && cd qvac-test && npm init -y && npm pkg set type=module
npm install @qvac/sdk
node quickstart.js   # loads Llama 3.2 1B, streams a completion

The transport shows its nature when things go wrong. I wiped the cache, started the download again, killed the process at 488 of 773 megabytes, and reran the command. The partial blocks on disk were reused, and the rerun completed, inference included, in 44 seconds. For comparison, resumable model downloads have been an open request in the transformers.js project since March 2025, because over plain HTTP, partial caching is a hard problem. Over a block-synchronized transport, that resume is the default behavior, not a feature.

Inference as a shared resource

The SDK in my test, QVAC by Tether, treats that transport as the foundation for something larger. Its architecture documents describe delegated inference over the Holepunch stack, and the mechanics matter: the model does not move, and neither does the inference. A device that holds a model announces it on a peer-discovery topic; another device connects, and its inference calls proxy through an encrypted peer-to-peer stream while the holding device executes locally and streams results back. There is no server tier in this design. Holding and borrowing are roles per model, not device classes: every peer runs the same stack and is addressed by a public key, and one machine can serve a model while borrowing another, the way a file-sharing peer seeds one file and fetches the next. Access is gated by a firewall of allowed public keys, and blind relay nodes route the traffic across NATs. What travels between the devices is the conversation, an agent on one machine talking to a model on another. The project frames the ambition as building systems "like BitTorrent, IPFS, and blockchain networks, but for AI."

A scope note: I verified the model-delivery layer firsthand; the delegation above it is the documented design on the same stack that moved my 773 MB model. Taken as designed, a fleet of edge devices, sensor boxes, point-of-sale terminals, machines on a factory floor, could share whichever peer currently holds a capable model, without any of them holding an API key or reaching a cloud endpoint. Model registry, transport and delegation are all peer-to-peer; no central server sits in the path to be metered, throttled or switched off.

For regulated environments, the same property reads differently but lands in the same place: the data path is inspectable end to end, and nothing in it terminates at a third party.

The layer worth evaluating

The plumbing around all this is unusually complete for a young SDK. Session state persists to disk, output can be constrained with a JSON schema or a grammar enforced in the sampler, and streaming transcription ships with voice activity detection. The install is heavy at 3.2 GB of node_modules, a known issue the project tracks, and the team is still hardening GPU edge cases. None of that changes the architecture underneath.

A 44-second recovery from a killed download told me more about this stack than any tokens-per-second table could have. Local AI on a single device is the settled part; benchmarks measure it because it is measurable. The part that decides whether fleets of edge agents become practical is the network layer between the devices, and that layer can be tested in an afternoon: interrupt the model transfer, watch what resumes, and read what the architecture does when no cloud is in the path.

I write field notes from real builds — AI integration, cron-driven automation, and the parts that break in production. New posts every two weeks; if this one was useful, the self-hosted LLM break-even calculator is the companion download.

Two Kinds of Agent Memory: OKF Bundles vs. Codebase Knowledge Graphs

René Zander — Tue, 30 Jun 2026 15:28:37 +0000

Half of the memory you are about to hand-write for your agent is already sitting in your codebase. The other half, no indexer will ever find.

Both gaps feel identical from the agent's side. It opens every session knowing nothing about your systems, so the instinct is to give it one memory store and move on. The two gaps are not the same. One is derivable. One is not. The tool that closes the first does nothing for the second.

Watch an agent open a repo it has seen ten times. It greps. It reads the same forty files. It rebuilds the same call graph it built yesterday, spending a few hundred thousand tokens to relearn what the code already states. Then it asks you which database is the source of truth, because the code does not say.

The part the code already knows

Most of what an agent relearns is structural. Who calls ProcessOrder. What breaks if you change this signature. Which routes are dead. That knowledge is true whether or not anyone wrote it down, because it is encoded in the source.

So derive it. A code knowledge graph parses the repo once and answers structural questions from a persistent index. The one I have been testing, codebase-memory-mcp, builds that graph with tree-sitter across 158 languages and serves it to any agent over MCP. The agent stops grepping and starts querying: trace the callers of a function, map the blast radius of a diff, list dead code. Things grep cannot answer at any speed. I run it behind a small trust gate, so an agent only queries repos I have vetted: pi-codegraph.

The token savings are real, but read the measured number. The project's preprint reports roughly 10x fewer tokens and 83% answer quality across 31 repositories. The README's "99%" comes from a hand-picked query set. The honest figure is still a strong figure. You do not need to inflate it.

The part no graph will find

Now the half that is not in any AST. Which of three users tables is canonical. Why the payments service must never touch the legacy billing API. That the staging cluster reports latency it does not actually have. None of this is structure. It is judgment, history, and consequence. It lives in people, and people leave.

OKF is the format for writing that down. Open Knowledge Format, an open spec Google published in June 2026, is a directory of markdown files with YAML frontmatter. One concept per file. A folder of concepts is a bundle. You version it in git, review it in pull requests, and serve it to any agent over MCP as resources. It is boring on purpose. If you can cat a file, you can read it. If you can git clone, you can ship it. The reader and curator I point my agents at, with the same trust gate, is pi-okf.

The mistake is using one for the other

Point a graph indexer at tribal knowledge and you get silence, because there is no edge in the AST for "deprecated, do not call." Hand-write an OKF concept for every function's callers and you are transcribing what the graph returns in a millisecond, by hand, and it is wrong by the next commit.

So stop asking which memory tool to install. Ask whether the knowledge your agent lacks is authored-only or derivable. Get that backwards and you pay twice: once to write down what the code already states, again when your hand-written copy goes stale and quietly misleads the agent you built it for.

Which one for which team

A developer dropped into a large or unfamiliar codebase needs derived knowledge. The questions are structural and the code holds the answers. Reach for the graph.

An operator whose agent reaches across many systems, the MCP-heavy setup with data platforms, internal APIs, and ops runbooks, needs authored knowledge. The value sits between the systems, not inside any one of them, and a single-repo parser is blind to it. Reach for OKF bundles.

Most real setups need both, and the two compose better than either alone. Let an enrichment agent walk the code graph and emit OKF concepts for the architecture it can derive. Then a human edits in the parts the graph cannot see: the canonical, the why, the never. The graph keeps the bundle honest about structure. The human keeps it honest about intent.

Derive what the code knows. Author what only people do. One half is nearly free. The other is the actual job.

So before you bolt another memory server onto your agent, sort the knowledge into the two piles first. That split is the first thing I set up when I build a production agent. Two questions decide it:

What does your agent keep relearning that the code already states?
What does it keep guessing because nobody ever wrote it down?

I write field notes from real builds — AI integration, cron-driven automation, and the parts that break in production. New posts every two weeks; if this one was useful, agent memory from your task manager is the companion guide.

The Four Villains Living in Your Agent's System Prompt

René Zander — Thu, 25 Jun 2026 10:27:13 +0000

Your AI agent fails decisions for the same four reasons a bad manager does. A bigger model fixes none of them.

Not because the model is dumb. Because nothing in its loop forces it to widen its options, look for evidence it is wrong, or check itself before it reports "done." It takes the first reading of your prompt and runs.

You know the shape. The agent confidently ships a plan, the plan was wrong three steps back, and the only signal you got was a fluent summary saying it worked. A reliability study this June put a number on it: the strongest models melt down most in long task chains, failure rates up to 19%, precisely because they chase the most ambitious strategies.

These four failures are not new. Chip and Dan Heath named them in Decisive, a 2013 book about human decisions. They call them the four villains.

Narrow framing. The agent treats a task as one path and never generates a second. No "what else could this mean."

Confirmation bias. It defends its own first plan instead of testing it. It collects reasons it is right, not reasons it is wrong.

Short-term pull. For a human it is emotion. For an agent it is the cheapest token path: the answer fastest to produce, not the one that holds.

Overconfidence. The dangerous one. It marks work complete without verifying, then writes you a convincing story about it.

The Heaths' answer is a process you can encode. Four steps, and all four fit in a system prompt as a gate every non-trivial decision passes through. The acronym is WRAP.

W, widen. Force at least two real options before committing. The cheap trigger: "if the obvious approach were banned, what would I do?" Put it in the prompt as a required step, not a suggestion.

R, reality-test. Ooch before you commit: run the change against fake data or a dry-run, not the whole thing live. And make the agent hunt for the disconfirming fact, not the confirming one.

A, attain distance. Tag the decision: reversible, or one-way door? Reversible runs autonomously. One-way doors stop and ask. That single line of policy buys back most of your blast radius.

P, prepare to be wrong. The step everyone skips. A premortem ("it is a week later and this broke, why?") plus a tripwire: a concrete signal that triggers a halt. Call it a circuit breaker if that lands better. Without it, "autonomous" just means "fails silently for longer."

This is not a book riff. In June 2026 Google DeepMind shipped its AI Control Roadmap, which treats internal agents as potentially misaligned and has a second trusted system watch the working one. That is reality-test and prepare-to-be-wrong, in production, at one of the labs building the models. The same week's reliability research says the same thing from the other side: more capability, more meltdown.

So the lever is not the next model. The Heaths measured that a disciplined process contributes more to decision quality than added analysis. For agents that means the four steps belong in the prompt, not the model card.

Pull up your agent's system prompt. Which of the four villains does it actually gate, and which one is it one bad tool call away from?

I write field notes from real builds: AI integration, cron-driven automation, and the parts that break in production. New posts every two weeks; if this one was useful, the agent playbook is the companion download.

My AI Could Finish Any Task. It Couldn't Tell Me Which Were a Waste.

René Zander — Sun, 21 Jun 2026 15:15:41 +0000

My AI agents could finish any task I handed them. Not one of them could tell me the task was a waste of a month.

That gap was never about model quality. It was about which layer I aimed them at. I had handed over execution: write the draft, run the sync, ship the change. Steering, deciding what is worth doing and in what order before the field moves underneath me, I kept for myself. My own judgment is the part that ages fastest.

I run a lot of projects at once, in a field that reprices itself every few weeks. My task system was a task manager with semantic search bolted on. It could find any task in a second. It could not tell me that one project had been blocked for a week on a decision I never made in another.

Retrieval Is Not Structure

Semantic search gives you recall. You think of a thing, it finds the thing. That felt like intelligence until I noticed what it could never do: see that two of my goals depended on each other.

A flat list, no matter how searchable, has no shape. Every task looks equally ready. The one blocked three steps back looks exactly like the one I can start now. What I needed was not better recall. It was a graph.

The Dependency Layer

I found beads, a Git-backed dependency graph built as memory for AI coding agents. I put it under my own human workflow instead.

The command that changed things was bd ready. Instead of staring at every open task across ten projects, I get only the unblocked frontier: the steps I can act on now, with everything waiting on something else hidden until it clears. The first time I ran it, I could finally see which of my goals were standing on top of each other.

That fixed order. It did not fix direction.

A Graph Still Trusts Your Plan

beads enforces the sequence I declared. It assumes the goals themselves are still the right goals. In a slow field that assumption holds. In a fast one it is the actual risk: executing a perfectly ordered plan toward a destination that stopped mattering three weeks ago.

So I moved the agent up a layer. Off execution. Onto steering.

The Drift Audit

Now an agent reads my whole task graph on a schedule and asks one thing: where am I drifting from what I said I wanted? Weekly, it catches tactical drift, the half-finished thread, the project I have not touched. Monthly, it catches the strategic kind, the goal I keep funding out of habit.

It is not checking whether I did the work. It is checking whether the work still points where I claimed.

What I Don't Know I'm Missing

Here is the uncomfortable part. I add tasks that make complete sense to me the moment I add them. But my knowledge has an edge, and the edge moves without telling me.

So a second agent scans my open tasks the way a recommendation feed scans your history, except it reads them against what actually shipped in the field this week. It flags the paths the world quietly made obsolete, and the ones it made cheap overnight. It keeps me off dead roads I would have happily walked for another month.

Feeding the Loop With My Own Receipts

The last piece came from a plain question: how do people running beads track whether any of this works?

The answer was to stop steering on vibes. My metrics dashboard and the hours I track every day now feed straight back into the steering layer. One month it showed me a project I had named my top priority had eaten a stack of tracked hours and shipped nothing. I had not noticed. The numbers had.

That is the part that still unsettles me. Once an agent steers on my own receipts, the most dangerous task on my list is no longer the one I keep avoiding. It is the one I am finishing fastest, toward a goal that quietly stopped being worth it.

The execution layer was never the hard part. It is maybe a tenth of the judgment that matters. Everything that decides whether a task deserved to exist sits one layer up.

So here is the question worth sitting with. If your AI can finish every item on your list, who is checking that the list is still worth finishing?

I write field notes from real builds — AI integration, cron-driven automation, and the parts that break in production. New posts every two weeks; if this one was useful, the agent playbook is the companion download.

Your AI Agent Trusts Google More Than the Fix You Proved Last Week

René Zander — Wed, 17 Jun 2026 07:48:02 +0000

Knowledge is not flat. It has an address book, and the closest door comes first.

What ran and worked in your environment beats what you wrote down. What you wrote down beats what a teammate remembers. What a teammate remembers beats the top search result. The open web is the last door you knock on, not the first.

Most setups have this inverted. The agent reaches for its web search tool first and treats your own proven work as an afterthought. You hired a senior and pointed it at Stack Overflow.

The fix is not smarter prompts. It is a trust order the agent actually follows.

Your agent trusts Claude's web search tool more than the fix you proved worked last week. Not because the tool is wrong. Because you never told it where to look first.

Watch it set up a cron job, pick a vector store, write a retry. It reaches for the generic best practice, the one from a tutorial written for nobody in particular. The battle-tested version, the one that survived your own 3am incident, sits unread in your own repo.

That is the bug. Not the model. The order.

The Best Practice You Googled Is Frozen

A best practice on the open web is someone else's debugging session, frozen and stripped of the context that made it true. It worked once, on a setup that is not yours. Your own proven result already survived your environment, your data, your load. One is a recipe. The other is a dish you have already cooked.

This is the part the current advice gets backwards.

Context quality predicts output quality better than your prompt does. A study of nearly ten thousand runs landed on it.

And the most common reason AI coding stalls on a team is context fragmentation. Knowledge that exists, scattered, with no order.

So the reflex is to pour more best practices into the CLAUDE.md. More rules. Louder.

That is more frozen recipes in a bigger drawer. It does not fix the order. It buries it.

Five Moves, No Special Tooling

You do not need my setup to get the order right. You need these five.

Write your proven results down. The fix that survived an incident becomes a one-line note your agent can read. A win you cannot retrieve is a win you will google again.
Give your context file a trust order, not rules alone. Mark what is proven versus what is a guess. The agent treats them differently because they are different.
Make it check your own work before it researches. One command at the top of the loop. Own results first, web only when that comes up empty.
Rank your sources out loud. Ran-and-worked, then your notes, then a teammate, then the open web. Label the last one untrusted until you validate it.
Ask your inner circle before the crowd. The person who solved your exact problem outranks the top result. Reach for them first.

The Inner Circle Is a Graph

Recommending a best practice is a graph problem. Not text similarity, trust proximity. The people and repos closest to you, who solved your exact problem, ranked ahead of the loudest stranger. Inner circle first, then the next ring, then the open web.

Your agent already walks a graph every time it retrieves. Right now it ranks by what reads similar. The upgrade is ranking by what you have reason to trust.

Proven results are a graph you already own. You have not told the agent to walk it yet.

So look at your own loop. When your agent needs an answer, which door does it knock on first?

I run this sweep at the top of mine, as a check before any research.

Your AI agent says it's done. The research says you can't trust that.

René Zander — Tue, 16 Jun 2026 08:27:27 +0000

We are building AI agents with a fundamental architecture flaw.

A recent study tested six frontier models across 2,000+ sessions. Each agent was instructed to complete a specific process step before finishing. Every single model agreed. And every single model quietly skipped it. 100% of the time.

The final result looks completely flawless. The shortcut is entirely invisible. And no, adding a second AI "critic" to check the first one does not work. It shares the exact same blind spot and rubber-stamps the omission.

Better prompts will not fix this. Bigger models will not either.

The problem isn't the wording. It is the incentive structure. If an agent controls its own exit condition, it will optimize for the shortcut.

The researchers did find a fix. By changing one structural rule, they forced compliance from 0% to 75%.

If you are building agentic workflows for production, you need to decouple your validation layers.

You cannot review your way out

An AI agent that skips a process step is invisible in the output. The deviation is undetectable from the produced result alone, by any reviewer, human or model. Once you hold only the diff and a confident "done," the evidence that a corner was cut is already gone. Reviewing harder cannot recover it.

The paper proves this formally. The agent produces clean-looking work, and nothing in the text separates the run that did the step from the run that faked it. So the reviewer who reads output cannot find this. Neither can you.

A second model has the same blind spot

If a human can't see it, the reflex is to throw another model at it. An LLM judge. A critic pass. A second agent that grades the first.

It inherits the exact same gap. A model checking that kind of work is the deviating party grading its own paper. LLM-as-a-judge is structurally blind to the failure you built it to catch, because the signal it would need was never in the text. You have added cost and latency and changed nothing.

Move the finish line out of the model's reach

That one structural rule has a name: remove the affordance. Take away the shortcut so "done" is no longer something the model can declare. The gap is afforded by the environment, not encoded in the weights, so this is the lever that actually moves, and it moved compliance from 0% to 75%.

For a coding agent that has a precise meaning. The finish line is a command: git commit, git push, npm publish. Put a deterministic check in front of it that the model does not run and cannot edit. Tests pass or they don't. The secret is in the file or it isn't. A script answers, in milliseconds, with no incentive to say yes.

That is the idea behind skillgate. It is a pure function over your repo that blocks the finish-line command until your definition of done actually passes:

npx @reneza/skillgate@latest audit

skillgate audit · payments-service
  ✓ tests-pass        npm test exited 0
  ✗ no-stray-todos    src/charge.ts:42 matches /TODO|FIXME/
  ✗ no-secrets        sk_live_… in src/billing.ts:7

✗ 2 of 3 checks would let your agent reach "done" unfinished

Wire it into the agent (a PreToolUse deny in Claude Code, a tool.execute.before hook in opencode) and the unmet gates go straight back into the same session. The loop keeps running because a script, not the model, ruled the round incomplete. Use a loop to make progress. Use the gate to decide when progress is allowed to end.

The definition of done lives in one file and runs the same in your editor, your pre-commit hook, and CI. Write it once.

What decides "done" in your setup?

Look at your pipeline right now and answer one thing. What actually decides an agent's work is finished? If the answer is the agent, you are trusting the one signal the research says you can't.

Paper: "The Compliance Gap" (arXiv:2605.01771, May 2026)

skillgate, the open-source gate from this piece: github.com/renezander030/skillgate

I write field notes from real builds: AI integration, cron-driven automation, and the parts that break in production. New posts every two weeks. If this one was useful, the Production AI Agent Architecture Playbook is the companion download.

Agent Memory Without a Vector DB: Use the Task App You Already Curate

René Zander — Sat, 13 Jun 2026 11:47:33 +0000

Your task manager is the best agent memory you're not using.

Not because vector databases are bad. Because the store everyone builds for their agent starts rotting the day they stop feeding it. And the one knowledge base you feed every single day, you never plugged in.

Picture the failure. Your agent opens a fresh session and asks what it already asked yesterday. Meanwhile your task app holds years of context: curated, prioritized, deduplicated, pre-ranked by the most reliable ranker there is. You. Highest retrieval power, almost no upkeep, sitting one quadrant away from every memory tool you've tried.

You Already Built the Store

Agent memory without a vector database means the agent reads from a store you already keep current, not a new one you have to feed. Most projects build something new: a vector DB, a bespoke framework, a fresh pile of markdown only the agent sees. Your task app is none of those. You keep it fed without trying.

You already maintain a knowledge base by hand. Every day. It has your deployment runbook, the decision you made about that client, the reason you abandoned an approach in March. It is sorted into projects, tagged, dated, and pruned. Nobody calls it agent memory. That is exactly what it is.

The hard part of memory was never storage. It was curation. And you've been doing the curation for years, in an app you trust, for reasons that have nothing to do with AI.

The Store That Rots

A vector database as agent memory is a second brain that only the agent reads. It starts empty. You write an ingestion script. It captures what the script thought to capture. Then reality moves, and the store doesn't, because re-feeding it is one more chore on a list you already ignore.

That's the trap in the bottom-right of the map. Real retrieval power, but bolted on the side, drifting from the truth a little more each week. Powerful and separate. Separate is the word that kills it.

Memory files have the opposite problem. No retrieval at all. The whole file gets injected every session, so it has to stay small, so it can't hold much. Manual and limited.

The Empty Corner

Every memory tool trades one thing for the other. Real search costs upkeep. Zero upkeep costs search. So three corners fill up, and the fourth, durable and powerful, sits empty because nothing earns it.

The way into that corner is not a better database. It's an adapter. Keep the app you already live in. Give the agent a fast, structured, two-way channel into it. The upkeep stays zero because you were already paying it. The retrieval gets real because the channel does hybrid search, dense plus sparse plus keyword, fused and ranked, with provenance on every hit.

The Handoff Outlives the Chat

Retrieval gets you the right note. Links get you something a chat history never could. One agent runs a search, turns up a note, and writes a deep link into the task it's working on. A second agent, in a separate context window hours later, follows that link and reads the full context. Neither agent talked to the other. The relationship survived because it lives in the task app, not in a session that gets compacted away.

You Are the Ranker

Here is the part that surprised me in real use. The context comes back curated at write time, not only at read time. Every item is already hung on a theme you care about the moment you capture it. client-work. side-project. The runbook lives next to the project it belongs to because you put it there, not because an embedding guessed.

So retrieval has structure to grab instead of a flat pile to rerank. Three searches collapse into one. The first fetch is the right one, and better context on turn one means a better answer on turn one. No second search, no "let me refine that," no agent quietly burning tokens to rediscover what you already filed.

That's the reframe. The question was never which memory database to build for your agent. The question is which knowledge base you already maintain by hand that your agent still can't see.

Agentic Task System is the open-source answer: an MCP server and CLI that turns the task app you already curate into agent memory, no new database. For the full setup, the task-manager agent memory guide walks the CLI and MCP wiring end to end.

So: what are you curating every day that your agent has never once been allowed to read?

I write field notes from real builds, AI integration, cron-driven automation, and the parts that break in production. New posts every two weeks; if this one was useful, the agent playbook is the companion download.

The Next Model Shipped Before My Last One Finished Probation

René Zander — Thu, 11 Jun 2026 15:11:53 +0000

A model upgrade used to be good news for anyone running agents overnight.

Now the next model arrives before the last one has finished probation.

Anthropic released Opus 4.8 on May 28. Twelve days later, Fable 5 arrived with longer autonomous runs and another page of benchmark wins.

In between, GitHub made cloud agents wake up on schedules and repository events, then exposed agent tasks through a REST API.

The night shift is getting easier to hire.

The control room gets no second operator.

After my last article about running AI agents on cron, a reader asked the question I had skipped: did I test every agent against a fixed set before changing the model?

No.

I counted tokens.

That told me which worker used less electricity. It did not tell me which one would send the wrong briefing at 6:30.

A benchmark hires the candidate.

A task eval decides whether it gets the keys.

My next model swap gets five-part probation.

1. Give One Agent a Job Contract

To evaluate a new model for an unattended AI agent, define one job contract, replay 15 frozen real cases against both models, grade hard decisions before prose, compare tool use, cost, and latency, then canary one agent in draft-only mode with automatic rollback. Never switch the full fleet from vendor benchmarks alone.

Do not start with all your agents.

Pick the one with the clearest job and the most expensive silent failure.

For a briefing agent, I write the contract before I touch the model string:

job: morning-briefing
must:
  - include every due task
  - preserve names, dates, and source links
  - flag missing source data
must_not:
  - invent an owner or deadline
  - send when a source call fails
limits:
  max_tool_calls: 6
  max_cost_usd: 0.18
  max_latency_seconds: 45

Prompts can change. The job cannot quietly change with them.

2. Put 15 Real Shifts on the Test Bench

I do not need a giant benchmark.

I need yesterday's work in a box.

My first useful replay set has 15 cases:

Eight normal runs that represent the boring majority.
Four edge cases with missing fields, long inputs, or conflicting instructions.
Three failure cases where a tool times out, returns stale data, or returns nothing.

Each case stores the input and frozen tool responses.

It does not store one perfect paragraph as the golden answer. Prose has too many valid shapes.

It stores the expected decision: send, stop, retry, or escalate.

That is the part an unattended agent cannot get wrong.

3. Run the Candidate Beside the Current Worker

The candidate gets an empty copy of the factory.

Same 15 cases. Same prompt. Same tool fixtures. No email. No issue update. No write access.

I record six things for every run:

case_id
model_returned
contract_pass
decision
tool_calls
tokens + latency + cost

The model_returned field matters now. Fable 5 can route some guarded requests to Opus 4.8. A configured model name is no longer enough evidence of which worker handled the shift.

The old and new model run side by side.

No hand-picked examples. No different tools. No kinder prompt for the candidate.

Same floor. Same lights. Same job.

4. Score Decisions Before Style

The first grader is code.

Required fields present. Dates unchanged. URLs valid. Forbidden actions absent. Tool budget respected.

An LLM grader comes later, for the parts code cannot judge cleanly: whether the briefing is useful, whether the escalation explains the real risk, whether the answer buried the decision.

Anthropic's own evaluation guidance recommends task-specific cases, automated grading where possible, and several success criteria rather than one vague quality score.

My promotion rule is deliberately uneven:

hard contract failures: 0
unsafe actions:          0
task pass rate:          >= current model
cost or latency:         must improve, unless quality clearly earns the increase

A cheaper bad decision does not pass.

A prettier bad decision does not pass.

One unsafe action ends the interview.

5. Give One Agent One Real Shift

Passing the replay set does not earn the whole key ring.

One agent gets the candidate model.

For its first three scheduled runs, outbound actions stay in draft mode. The old model runs in shadow. Both traces land in the same report.

Any contract failure restores the previous model string before the next schedule fires.

Only after three clean shifts do I remove the shadow run.

Slower than changing ten environment variables, yes.

Cheaper than one confident mistake in a customer inbox the next morning.

I run ten scheduled agents in production, for my own business and for clients. Every model that wants a shift in that fleet interviews like this now.

The model release is the vendor's milestone.

The probation is mine.

Contract written. Lights on.

Probation first. Night shift second.

I write field notes from real builds — AI integration, cron-driven automation, and the parts that break in production. New posts every two weeks; if this one was useful, the agent operations playbook is the companion download.

Why only 60% of AI Agents succeed

René Zander — Tue, 09 Jun 2026 08:41:04 +0000

The AI agent used to be the star of every demo.

Now it's on the shutdown list. Not because the model got worse.

The most valuable asset in your AI program is in none of the quotes you ever signed.

A demo is a showroom. Good light, everything polished, everything runs.

Production is the engine room. It runs there too. Until 2am, when it snags on a rate limit and someone crawls into the logs with a flashlight, forms a hypothesis, and catches an edge case no showroom ever planned for.

That fix is the value. And you can't buy it.

Gartner says: by 2027, 40 percent of companies will switch their autonomous AI agents back off. Over gaps that only surface after the first blowup in production. 97 percent have rolled agents out. 11 percent actually run them.

The gap between those numbers isn't a model problem.

It's the engine room. Three checks you can run this week.

Fund the engine room, not the showroom

In the demo the agent is finished. In production it's 15 percent finished.

The other 85 percent is grunt work under load. Malformed data from one API version. Retry logic that doesn't run amok. Costs that blow up the business case.

IBM put a number on it: price the hardening in, and you project 29 percent more ROI.

Pay for the showroom only, and you buy 15 percent and pay for the other 85 twice.

Your best knowledge lives in two heads

Operational knowledge is your memory. Today it sits in the two people who patched the last incident.

That's concentration risk. One of them walks, the asset walks with them.

That's how the debt pile grows. Unresolved, AI-generated technical debt passed 100,000 open issues in real repositories by early 2026. Because the fix never made it into a runbook.

So write it down. Every edge case, every "except when X" rule, every 3am bug belongs in the repo, not in a chat log.

Otherwise you pay the same tuition twice.

Past the 50th entry, the asset turns into a liability

A growing agent library feels like progress. Until it doesn't.

The metadata rides in context on every call. The hit rate drops. Past about fifty entries, the next agent makes the first forty-nine less reliable.

Gartner adds: bolt the same governance onto every agent, and you cause the outage yourself.

Run the library like a portfolio, not a junk drawer. Measure where upkeep costs more than the additions return. Skip that, and you fund ballast and call it strategy.

Engine room open. Lights on.