How to Keep AI Coding Agents from Hallucinating: A Guide to Harness Engineering
AI coding agents (like Claude Code, Devin, or open-source equivalents like OpenClaw) are incredibly powerful. They can navigate directories, write tests, refactor modules, and submit PRs.
Yet, if you drop them into a raw repository without boundaries, they suffer from context window pollution, agent amnesia, and scope drift. A simple bug-fix refactor can trigger a 6-hour loop where the agent rewires half the project, deletes unrelated tests, and gets stuck in "process theater."
To fix this, we need Harness Engineering.
What is Harness Engineering?
An Agent Harness is a structured, repository-local control layer designed to guide and verify the agent's work. Instead of feeding your LLM a monolithic prompt, you embed a lightweight system of record and physical feedback loops directly inside the workspace.
I have packaged the exact, battle-tested Markdown-based context rules I use to steer and constraint my local agents into a public repository: MasihMoafi/harnesses-I-use.
Rather than complex code, this repo shares raw configuration rule sheets:
-
AGENTS.md: The global router and primary entry point. Every AI agent reads this first to understand the Priority Kernel (reply constraints, English defaults, local-first behavior). -
CODEX_CODING_GUIDELINES.md: Surgical coding rules adapted from Andrej Karpathy's coding practices. Enforces minimal changes, style-matching, and goal-driven execution. -
TERMINAL_AND_GIT_RULES.md: Safe terminal navigation, SOCKS/HTTP proxy configuration, explicit git scoping (e.g., never usinggit add -A), and change safety (using Ubuntupkexecfor root commands instead of raw CLI password prompts). -
SESSION_HANDOFF_RULES.md: Guidelines for context awareness and continuity. Defines how state should be captured and transferred between sessions. -
ARTIFACT_RULES.md: Rules for generating verified visual and math outputs, including local HTML layout verification, MathML for formula rendering, and PDF inspection. -
abbn.md: A dictionary of standard abbreviations (e.g.,ctu= continue,fmy= familiarize,ver= verify) to save token count and maintain short, high-efficiency communication.
The Karpathy Connection: Simplicity & Rigor
This harness approach is inspired by Andrej Karpathy's open-source autonomous agentic research repository, autoresearch.
Karpathy's project shows how an AI agent can run a self-correcting research loop: proposing changes, running experiments, and either keeping them (if metrics improve) or discarding them. It focuses on a minimal, bounded loop directed by a program.md control file.
We applied that same philosophy to agent-driven code generation. The core rules of our harness require:
- Surgical Changes: Touch only the exact lines needed. No drive-by refactoring of adjacent code.
- Verifiable Outcomes: Before the agent claims a feature is complete, it must run a local script to verify the baseline, compile the code, and attach raw logs as physical evidence.
- Harness Simplification: Every rule is considered overhead. As models grow smarter, we run tests to prune redundant rules.
Case Study: Autonomously Generating ML Research Reports
To test this, I applied the harness methodology to design a structured academic seminar workflow. The agent was tasked with autonomously generating a graduate-level ML research report and pipeline, using Karpathy's repository as a reference for simplicity.
By restricting the agent to a single control manual and enforcing strict keep/discard criteria, the agent successfully completed the entire paper compilation and pipeline execution.
I've published the code, manuals, and report of this experiment to show how it is structured:
👉 MasihMoafi/seminar
Get the Harness
If you are building code with AI agents, stop writing 2,000-word system prompts. Start building repository harnesses.
Check out the templates and configurations:
👉 MasihMoafi/harnesses-I-use
For more of my work, experiments, and research, check out my website:
👉 masihmoafi.tech
Top comments (0)