kination

Posted on Jul 7

Part 2 - Deconstructing the Harness

#ai #harness #development

When developers build their first AI coding assistant or autonomous worker, they often start with a single instruction document. They write a detailed Markdown file (a .cursorrules or system_prompt.md) containing the persona, files to read, rules for writing code, database restrictions, and instructions for running tests.

This monolithic approach is one kind of mistake you can face on first step.

Dumping every instruction into a single prompt fails because of how language models allocate attention. Recently LLM researchers from multiple institutions documented a limitation known as "Lost in the Middle". They showed that large language models are highly effective at retrieving information placed at the very beginning or end of a prompt, but their accuracy degrades significantly for information buried in the middle of the context window.

Put fifty rules in one Markdown document, and the model follows the first few and the last few. The forty in the middle drift out of focus. And this is still on-going known issue of LLM, regardless of 1M token models which recently released.

A monolithic prompt also prevents parallel execution. If one prompt controls the entire task, the agent must process every step sequentially, waiting for the model to think through file reading, code generation, and test execution in a single thread.

To build a reliable system, split the harness into Agent, Skill, and Orchestrator layer.

Agent Layer (Reasoning)

Agent is 'reasoning' kernel of system. It is language model configured with specific persona, targeted instructions, and strict input/output boundaries.

Agent does not execute actions. It decides actions based on received information.

To prevent attention loss, agent must operate on self-contained prompt without parent conversation history. Developers often pass full chat logs to sub-agents. Avoid this.

Context isolation guarantees three properties:

Focus Isolation: Agent is protected from noise of irrelevant conversation logs, keeping attention window focused on immediate task.
Cost Control: Excluding history saves input tokens during iterative loops.
Parallel Safety: Since agent does not share mutable state with other workers, multiple agents can run in parallel without race conditions.

Here is simple .md template for syntax-auditing agent:

# Agent: Syntax Auditor
Role: Audit Python code generated by the coder.
Input: File path and compiler error.
Output: JSON array of syntax errors.

## Instructions
1. Read the code file from the path.
2. Identify syntax errors using the compiler message.
3. Output JSON matching: {"errors": [{"line": int, "description": str}]}.

## Constraints
- Do NOT rewrite or generate code.
- Return ONLY valid JSON.

Good Agent should have

Narrow Scope: Good agent has single, well-defined job.
- Good Prompt: "You are Syntax Auditor. Review code in and list syntax errors. Output JSON list of line numbers and descriptions. Do not attempt to fix code."
Minimalist Context: Pass only information necessary for immediate decision.
- Good Prompt: "Review this specific file: app.js. Compiler error: Unexpected token."
Structured Outputs: Agent communicates via structured formats, like JSON schemas, allowing static code to parse decisions without fragile regex matching.
- Good Prompt: "Return ONLY JSON block: {"errors": [{"line": 12, "desc": "missing semicolon"}]}"

Bad Agent have

Do-it-all Agent: Single agent prompt attempting to plan project, write Python functions, execute tests, and format final report. This design guarantees attention dilution.
- Bad Prompt: "You are full-stack assistant. Plan project, write Python functions, execute tests, fix bugs, and write summary. Ask me if you need help."
Mixed Responsibilities: Agent prompt containing instructions for tools it does not use. For example, developer explains database backup utility to code reviewer.
- Bad Prompt: "Review code in app.py. Note that production database is hosted on AWS RDS and database backups run every midnight using cron job."
Unstructured Free-Text Output: Developer allows agent to respond in conversational prose, forcing downstream scripts to use fragile regular expressions to guess model intent.
- Bad Prompt: "Review code and write paragraph explaining what is wrong and how to fix it."

Skill Layer (Action)

Skill is atomic tool wrapper letting agent interact with outside world. Examples include reading file, searching web, executing shell command, or querying database. If agent is brain, skill is hand. In deconstructed harness, ability to spawn sub-agent (Agent tool) is itself treated as skill.

To enforce security and reliability, skills must be managed under principle of least privilege and defense-in-depth tool filtering:

Layered Restrictions: Filter tools based on agent role. Read-only exploration agent must have file-write and execution tools stripped at API layer, reinforced with negative prompts at model layer to prevent indirect command execution.
Interactive vs Async Safety: Background (asynchronous) agents must be restricted from tools requiring interactive prompts (like interactive user questions), allowing only safe, deterministic operations.

Here is simple .md template for file-reading skill:

# Skill: Read File
Goal: Read and return local file contents.
Parameters:
  - file_path (string)

## Rules
1. Verify the path starts with the workspace root.
2. If file is missing, return error: {"status": "error", "message": "File not found"}.
3. Read and return raw contents.

Good skill should have

Atomic and Pure: Good skill does one thing.
- Good Design: def read_file(file_path): return open(file_path).read()
Strict Schema Validation: Skill enforces input validation before passing parameters to system.
- Good Design: if not file_path.startswith(WORKSPACE_ROOT): raise ValueError("Access Denied")
Predictable Error Logs: When command fails, skill catches exception and returns structured error string to coordinator.
- Good Design: except FileNotFoundError: return {"status": "error", "message": "File not found"}

Bad skill have

Overly Broad Logic: Tool called manage_codebase reading files, editing lines, running tests, and pushing to GitHub in single execution step.
- Bad Design: def manage_codebase(file_path, edit_content, test_cmd): # does everything internally
Unvalidated Command Generation: Skill accepting raw shell command string generated by LLM and running it directly.
- Bad Design: def run_command(cmd_string): os.system(cmd_string)
Silent Failures: Tool catches critical exception, prints warning to console, and returns empty string. This forces agent to assume operation succeeded when it failed.
- Bad Design: except Exception as e: print(e); return ""

Orchestrator Layer (Control)

Orchestrator is coordinator of harness. It is deterministic state machine written in static code, like Python or TypeScript, rather than language model. It manages execution flow, tracks project state, decides which agent to call next, sequences skill execution, and handles errors.

Rather than executing tasks directly, orchestrator acts as central coordinator. It orchestrates flow through division of labor:

Lifecycle Management: Manage process lifecycles, ensuring background tasks are registered and terminated to prevent orphan or zombie processes.
State Tracking: Maintain clean, versioned workspace records, tracking task progress (pending, in-progress, completed) and routing logs.
Loop Control: Catch exceptions at boundaries and enforce strict iteration timeouts to prevent runaway API spending.

Here is Python example of deterministic orchestrator class managing code-compile-test loop:

class CodeGenerationOrchestrator:
    """
    A deterministic orchestrator managing the agent execution loop.
    Enforces iteration limits, executes skills, and routes traceback feedback.
    """
    def __init__(self, coder_agent, compile_skill, test_skill, max_attempts=5):
        self.coder = coder_agent
        self.compile_tool = compile_skill
        self.test_tool = test_skill
        self.max_attempts = max_attempts

    def execute_workflow(self, task_prompt):
        # Step 1: Initial generation
        code = self.coder.generate_code(task_prompt)

        # Step 2: Deterministic verification loop
        for attempt in range(self.max_attempts):

            compile_success, compile_error = self.compile_tool.run(code)
            if not compile_success:
                code = self.coder.fix_code(code, compile_error)
                continue

            # Run test suite assertions
            test_success, test_error = self.test_tool.run(code)
            if test_success:
                return code

            code = self.coder.fix_code(code, test_error)

        raise TimeoutError("Orchestrator: Max compilation/test retries exceeded.")

# Instantiating and executing the orchestrator workflow:
coder = CoderAgent()
compiler = PythonCompileSkill()
test_suite = PythonTestSkill(runner_path="test_suite.py")

# The orchestrator wires the components together
orchestrator = CodeGenerationOrchestrator(
    coder_agent=coder, 
    compile_skill=compiler, 
    test_skill=test_suite,
    max_attempts=5
)

# Run the deterministic loop
final_code = orchestrator.execute_workflow("Write a parse_and_sum function...")

Good orchestrator should be

Deterministic Control: Workflow is written in static code. Developer decides sequence.
- Good Control: run_agent(PLANNER); run_agent(CODER); run_skill(TEST_RUNNER)
Error Interception: Orchestrator catches failures at boundary, deciding whether to route traceback back to coder agent for correction or halt execution.
- Good Control: success, log = compile_code(); if not success: run_agent(CODER, feedback=log)
State Management: Orchestrator maintains clean, versioned record of environment, allowing rollbacks if task fails.
- Good Control: save_git_snapshot(); if tests_fail: rollback_to_snapshot()

Bad orchestrator has

Uncertain Routing: Developer asks LLM to choose next step. This approach frequently leads to infinite loops, deadlocks, and unpredictable execution paths.
- Bad Control: "Here are tools. Tell me which tool you want to run next to complete task."
State Clutter: System passes entire execution history of tool calls to agent on every turn, quickly filling context window with redundant information.
- Bad Control: history += f"\nRun: {run_id}, Output: {output}" # feeds 20 turns of logs into every prompt
No Timeout Limits: Loop runs without iteration limits. Agent burns API credits trying to fix same syntax error indefinitely.
- Bad Control: while True: run_agent_loop() # no iteration limit checks

Putting It Together

Three layers cooperate through division of labor. Orchestrator routes task to Coder Agent (reasoning), which requests write_file Skill (action) that orchestrator validates and runs. Orchestrator fires Test Skill, and on failure routes traceback back to agent for correction, looping until tests pass or attempt limit trips. No single component carries entire job: Coder Agent writes code, Test Skill runs assertions, orchestrator owns loop. Each prompt stays small, and instructions model sees sit at edges of context window, where attention holds.

Execution in Claude Code

Three-layer model is architecture, not Claude Code feature list. Map it onto Claude Code and pieces land in unexpected places. One component does not exist as theory defines.

Skill: built-in tools, MCP, and SKILL.md

System hand splits in two. Built-in tools (Read, Write, Bash, Grep) and MCP servers are actual tool wrappers touching files, shells, and external systems. Developer never registers read_file. It exists as Read tool, and external connections route through MCP.

What Claude Code calls "Skill" is different: .claude/skills//SKILL.md file. This is bundle of instructions, prompt, and optional supporting scripts. It pulls live data at load time through dynamic context injection. For example, line like !`git diff HEAD` runs shell command and inlines output before model reads file. Skill packages procedure, not raw capability. Custom slash commands use same mechanism, so .claude/commands/deploy.md and .claude/skills/deploy/SKILL.md both produce /deploy.

Agent: .claude/agents/.md

This layer maps cleanly. Save Markdown file in .claude/agents/:

---
name: syntax-auditor
description: Audits Python for syntax errors. Use after code changes.
tools: Read, Grep
model: haiku
---
You are Syntax Auditor. Read target file, list syntax errors,
and return JSON array. Do not fix code.

Only name and description are required. Body becomes subagent system prompt. Subagent receives prompt plus environment details, not parent conversation history. Each subagent runs in isolated context window with its own tool allowlist, returning only summary to caller. Lead model spawns subagent using Agent tool (previously named Task, which still works as alias), deciding when to delegate based on description. Setting model to Haiku here reduces audit costs compared to main model.

Orchestrator: SKILL.md playbook

Orchestrator lands in same place as skill: .claude/skills//SKILL.md file. Difference is content. Skill at this level is playbook, step-by-step instructions naming which agent to call, in what order, and what to do with each result. Invoke it with /audit and lead agent reads playbook and runs it, spawning agents through Agent tool and letting each reach for its own skills.

This is how Claude Code bundled skills work. /code-review and /debug are prompt-based. They hand model procedure and let it drive tools. Orchestrator skill is same shape.

One limitation. Lead agent, language model, executes this playbook. It is not deterministic state machine described earlier. That is trade-off Claude Code makes: orchestrator invoked with one command and shared as file, at cost of loop driven by model rather than guaranteed by code. Write iteration cap into playbook as instruction ("retry at most three times"), keep agent tools narrow, and model follows closely.

Flow of single request

Type /audit app.py:

Load: Claude Code reads .claude/skills/audit/SKILL.md, runs any !`cmd` injections, and hands playbook to lead agent.
Route: Following playbook, lead agent calls Agent tool with syntax-auditor type and self-contained prompt: "Audit app.py, return JSON errors."
Act: Subagent spawns in fresh context window containing only Read and Grep tools, reads file, and returns JSON summary. Only summary returns. File reads and logs stay in subagent context window.
Decide: Lead agent takes summary and picks next playbook step: report result, or route errors to coder agent for fix, then loop.

Context isolation keeps execution lean. Subagent noise never reaches main conversation, keeping lead agent context window small to maintain reasoning quality, core property whole architecture protects.

DEV Community