Kaizen Harness: patterns for making AI agents reliable

#opensource #devops #ai

Kaizen Harness is a set of patterns for the system around an AI model: trajectory logging, verification, self-healing, and multi-model council debates.

The idea is simple. When an agent makes a mistake, you don't rerun the prompt. You change the system so that class of mistake stops happening.

I kept hitting the same problem: AI agents fail silently, claim success without evidence, and repeat mistakes they already made an hour ago. The model wasn't the issue. The system around the model was.

The four patterns

Each pattern is a single bash script or TypeScript file with a README. No framework, no install, copy what you want.

Trajectory logger. Append-only JSONL of every action: task, model, tools, success/failure, failure type. Bash + python3, nothing else. This is the memory you need before you can fix anything. Without a log you're guessing.
Verification. Stop trusting the agent's "done." A script that checks exit codes, stdout patterns, and common error signatures (segfaults, connection refused, syntax errors) even when the exit code looks clean.
Self-healing. Reads a failure memory log, fingerprints each failure, auto-patches the classes it recognizes with a post-fix verification, and escalates the ones it doesn't to a council. Rate-limited by fingerprint so it can't get stuck in a patch loop.
Council debate. Fans a hard question out to 3-6 models in parallel, lets them disagree, then synthesizes the tradeoffs. Built around free models (OpenRouter free tier, Groq) and a local Ollama seat. Only the synthesis step touches a paid model, and only if you want it to.

There's a runnable demo in the README showing self-healing on its own: it reads its failure memory, patches and verifies three known failure classes, and kicks the unknown one to a council. No API keys, no network.

What we learned running it

The honest part is in the "What We Learned" sections in each pattern's README:

An early self-heal applied patches and assumed they worked. One bad patch made things worse. Now verification after every fix is mandatory, and a failed fix escalates instead of looping.
The council talked us out of adding Redis, microservices, and a message queue we didn't need at our scale.
Without a trajectory log, you're just guessing what went wrong. Structured logging turned "the agent broke" into "the agent broke at step 3 with error class X."

The patterns are framework-agnostic. You can copy one in without adopting the rest.