Introduction
"2025 was the year of agents. 2026 is the year of agent harnesses."
This is article #136 in the Open Source Project of the Day series. Today's project is Harness Handbook — a tool that converts AI agent harness codebases into navigable behavior manuals, accompanying the arXiv:2607.13285 paper (July 2026).
First, a concept: a harness is the orchestration layer wrapped around a foundation model — constructing prompts, managing state, invoking tools, and coordinating execution. The hook system in Claude Code, the Harness module in Open Interpreter, the dispatch layer in any agent framework — all of these are harnesses.
Maintaining a harness is a continuous engineering problem. When requirements change, developers need to translate "I want to change this behavior" into "which specific locations in the codebase need to change." But production harnesses are large, tightly coupled, and behaviorally distributed — a requirement like "add secret masking to all capture paths" might require changing three non-adjacent locations: a log capture path, a pre-disk-write handler, and a cold-start fallback path. Keyword search won't find all of them.
Harness Handbook's approach: first automatically generate a handbook that maps each behavior to code evidence; then give an agent a navigation algorithm that progressively locates specific code positions from behavioral descriptions.
What You'll Learn
- What an agent harness is and why maintaining it is hard
- The handbook's three-level document structure (L1/L2/L3) and state-register view
- BGPD (Behavior-Guided Progressive Disclosure) — the four-step navigation algorithm
- Why scattered sites are the hardest category for AI code editing
- Resync: how the handbook stays current as code evolves
- Evaluation numbers from Codex (Rust, 2,267 files) and Terminus-2
Prerequisites
- Basic familiarity with AI agent framework concepts (tool calls, state management)
- Experience maintaining or using an LLM agent system
- Some awareness of code static analysis
Project Background
What an Agent Harness Is
One-sentence definition: a harness is the shell around a foundation model that turns an LLM into an agent that can do things.
User input
↓
Harness layer
├── Build prompts (inject context, tool descriptions, system prompt)
├── Manage state (conversation history, tool results, session variables)
├── Tool invocation (execute code, access files, call APIs)
└── Coordinate execution (multi-step planning, error retry, result synthesis)
↓
Foundation model (GPT, Claude, Gemini…)
↓
Output
A harness isn't a single component — it's logic distributed across the entire codebase. Prompt templates live in one file, tool registration in another module, state persistence in a different directory.
The Core Maintenance Problem
When a product requirement changes:
Requirement: "Add usage statistics to all tool call results"
Developer needs to find:
- All tool call execution paths (potentially 5-10 locations)
- The processing point before results are returned to the LLM
- Possible async paths (normal call + timeout retry + streaming)
- Where the statistics data gets stored
Finding all of these in a 2,000+ file Rust codebase with keyword search
will almost certainly miss some.
This is the problem Harness Handbook targets: edit localization — building a reliable mapping between behavioral descriptions and code positions.
Author / Team
- Author: Ruhan Wang
- Paper: arXiv:2607.13285 (July 14, 2026)
- License: Apache-2.0
- Language: Python, calls OpenAI-compatible API
Project Stats
- ⭐ GitHub Stars: 252
- 🍴 Forks: 25
- 📄 License: Apache-2.0
- 📝 arXiv: 2607.13285
Handbook Structure
Three-Level Document Tree (𝒟)
The handbook isn't flat documentation — it's a three-level hierarchical structure:
L1 — System Overview
Overall architecture, execution model, major stages, global data flow
("What are the core parts of this harness and how do they cooperate")
L2 — Stage Pages (per-stage)
Each stage's responsibilities, inputs, outputs, dependencies, local state
("What does this stage do, what does it accept, what does it produce, what does it depend on")
L3 — Source-Grounded Entries
Each behavior entry linked to exact file/function/code region locators
("Where in the codebase is this behavior implemented")
Two leaf modes:
-
Function-as-leaf: L3 entry = one function or contiguous regions; requires a pre-supplied skeleton (
skeleton.yaml); suitable for small codebases - File-as-leaf: L3 entry = one file; stage skeleton inferred automatically; used for Codex with ~2,267 files
State-Register View (𝒵)
This is one of the handbook's most critical designs, specifically addressing scattered-site problems.
For each shared state variable (register) that crosses stage boundaries, the view records:
- Every read location for that state (across all stages)
- Every write location for that state (across all stages)
Example: session_context register
Write locations:
- auth.rs: initialized in authenticate()
- session_manager.rs: updated in refresh_token()
Read locations:
- tool_executor.rs: injected before execute_tool() call
- response_formatter.rs: read in format_response() for user info
- audit_logger.rs: session_id recorded in log_event()
Top-down code reading doesn't surface these structural interdependencies — the locations are non-adjacent in the codebase but logically coupled. The state-register view makes these hidden dependencies explicit.
BGPD: Behavior-Guided Progressive Disclosure
After handbook generation, the second core contribution is the BGPD algorithm — guiding a code agent from a behavioral description progressively toward precise code positions.
Four steps:
Modification request: "Validate permissions before all tool executions"
Step 1: Stage selection
Read L1/L2 → find stages related to permission validation
Check state-register view → add stages coupled through shared state
(Discovers both tool_executor and auth stages are relevant)
↓
Step 2: Entry selection
Open relevant stage pages → select top L3 entries
"Entry body disclosed only when needed — limits unnecessary context"
(Expand only execute_tool, validate_permission, and related entries)
↓
Step 3: Call-relation expansion
Walk the function-call graph (or file-call graph)
Boundary nodes "may supply context but are never returned as edit sites"
(Finds call chain: request_handler → execute_tool → shell_runner)
↓
Step 4: Source verification
Verify candidate locators against the live repo
Retain only sites "that remain relevant" as verified evidence Ê_q
(Confirms three functions needing changes exist and are unmodified)
The key design: progressive disclosure rather than dumping everything at once. L3 entries expand "on demand" — before selection, the agent sees only summaries; after selection, full source links are revealed. This maintains token efficiency.
Resync: Keeping the Handbook Current
Code evolves continuously. A handbook can't expire after one use. The resync module handles incremental synchronization after code changes:
Code change (diff Δ) arrives
↓
Version alignment
Reparse repo, rebuild program graph
Match functions via "body fingerprints that ignore line numbers"
→ A moved function is identified as unchanged (not a new function)
↓
Scoped update
├── Stage skeleton still valid → refresh only affected L3 entries
└── Skeleton invalidated → rerun full algorithm on affected portion
↓
Conservative handling
Unresolvable locators → marked frozen and excluded
(Exclude rather than guess)
↓
Validation and packaging
New (ℛ′, ℋ′) pair becomes the start state for the next request
LLM calls during resync are limited to four types: classification, file assignment, within-stage organization, and description revision. The design minimizes LLM calls — anything static analysis can handle doesn't go to the LLM.
Evaluation Results
Tested on two real open-source harnesses:
- Terminus-2: Python, 6 files, small harness
- Codex (Open Interpreter's Rust version): Rust, 2,267 files, large harness
| Metric | Codex | Terminus-2 |
|---|---|---|
| Handbook win rate | 38.3% | 45.6% |
| Baseline win rate | 28.3% | 26.7% |
| Token reduction | 12.7% | 8.6% |
| Max F1 gain (symbol level, Opus ref) | +18.8 pts | +12.3 pts |
| Max Wrong↓ reduction | −25.9 pts | −13.3 pts |
Results hold across all three judge models (GPT-5.5, Opus 4.8, DeepSeek-V4-Pro), all three request types, and all three difficulty levels.
Three categories with the largest gains:
- Scattered Sites (SH): behavior spread across multiple non-adjacent locations
- Rarely Executed Paths: code branches that trigger infrequently
- Cross-File (CF) / Cross-Module Interactions: capabilities spanning multiple files or components
These three categories are exactly where keyword search fails most — the relevant code isn't in obvious places, it's scattered, or it's hiding in error handlers and fallback paths.
Scale facts (Phase I, deterministic, no LLM):
- Terminus-2 (Python, 6 files): 103 internal functions → 20 stages, 106 L3 entries, 10 state registers
- Codex (Rust, 2,267 files): 34,363 internal functions, 159,960 call edges → 140 stages, 2,267 L3 entries, 62 state registers
Quick Start
Installation
git clone https://github.com/Ruhan-Wang/Harness_Handbook.git
cd Harness_Handbook
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
Configure LLM API (OpenAI-compatible):
export OPENAI_API_KEY=sk-...
export OPENAI_BASE_URL=https://api.openai.com/v1 # or other compatible endpoint
export LLM_MODEL=gpt-4o
Generate a Handbook
Large codebase (no skeleton needed, auto-inferred):
cd handbook_generate_large
python run.py --repo /path/to/your/harness/
# Output to ./output/: overview.md, per-module pages, module_tree.json
Small codebase (requires skeleton.yaml):
cd handbook_generate_small
# Edit skeleton.yaml to define stage structure
python run.py --repo /path/to/your/harness/ --skeleton skeleton.yaml
Use as Agent Planner
cd handbook_as_helper
python planner.py \
--handbook /path/to/generated/handbook/ \
--request "Add rate limiting to all LLM API calls"
# Output: precise edit plan with files and functions to modify
Resync After Code Changes
cd handbook_as_helper
python resync.py \
--handbook /path/to/handbook/ \
--repo /path/to/repo/ \
--diff changes.diff
# Incrementally updates the handbook, processes only changed portions
Links and Resources
- 🌟 GitHub: Ruhan-Wang/Harness_Handbook
- 📄 Paper: arXiv:2607.13285
- 🌐 Project page: ruhan-wang.github.io/Harness-Handbook
- 💬 Hacker News discussion: Making Agent Harnesses Understandable, Auditable and Editable
Conclusion
Harness Handbook solves a precision problem in "AI editing AI code."
The biggest failure mode when using an AI agent to modify harness code isn't model capability — it's localization error. The agent changes three locations, misses two, the system behavior partially changes, and a bug hides in a corner. Bigger models and more tokens don't fix this, because the root cause is missing information: the agent doesn't know about "relevant locations scattered elsewhere."
The three-level document tree plus state-register view makes hidden dependencies explicit — gives the agent a map it didn't have before. BGPD's progressive disclosure lets the agent stop once it has enough information rather than stuffing the entire codebase into context. Resync keeps that map current so it doesn't expire with the next commit.
45.6% win rate vs 26.7% baseline, with 12.7% fewer tokens — quality improves while spending less. That combination is a good signal: the handbook makes agents more precise rather than more verbose.
Stars (252) are still low. But the importance of this problem scales with harness codebase size, and harnesses are getting larger. 2026 is the year of agent harnesses. Demand for tooling like this is only starting to grow.
Explore PrimeSkills — A marketplace for handpicked AI Agents and skills. Each is validated in real enterprise workflows, stripping away hype and keeping only what truly works.
Welcome to my Homepage for more useful insights and interesting products.
Top comments (0)