Oyedele Temitope for Hackmamba

Posted on Aug 10

How to Build a Production Agent Harness

#ai #agents #devops #agentskills

AI agents don't usually become unreliable all at once. They degrade quietly. One session the agent repeats a step it already completed. The next, it loses track of where it was mid-run and starts over from scratch. Eventually it produces output with nothing in place to verify whether it actually succeeded.

The problem is structural. Harness components get built as one-off scripts instead of versioned, reusable units. Your state file lives in one project, a slightly different version lives in another, and when you fix a bug in one place the fix stays local. Teams end up maintaining the same harness logic across multiple projects and debugging whichever version happens to be in front of them.

This tutorial shows how to fix that at the source. Instead of writing harness logic as inline scripts, you'll package it as versioned components in a shared Bit scope. Fix a bug once, tag and export, and Ripple CI (Bit's built-in component CI/CD pipeline) propagates the update to every project that depends on it automatically. By the end, your harness stops being something you rebuild per project and becomes something you install with a single command.

Before You Begin

Before you start, make sure you have the following in place so the setup steps run cleanly:

Node.js 18 or higher installed on your machine. Run node --version to check.
A bit.cloud account. You'll create a scope during setup. If you don't have one yet, head to bit.cloud/signup.
Basic familiarity with TypeScript. The components in this tutorial use TypeScript. You don't need to be an expert, but you should be comfortable reading typed function signatures.
A terminal and a code editor. The tutorial runs entirely from the command line with a few file edits along the way.

What You're Building

The three components are a state tracker, a context loader and a verifier. Each one targets a specific failure mode in the agent loop.

harness/state tracks what the agent has done, what it's currently working on and what comes next. It reads at session start and writes at session end.
harness/context pre-loads a structured map of your components and their relationships before the agent takes its first action.
harness/verifier evaluates agent output against a done condition before the loop continues, returning one of four verdicts: NO, YES, MAYBE or IFF.

Once all three are released to a shared scope, your entire harness installs into any agent project with a single command:

bit install @your-username/agent-harness.harness.state @your-username/agent-harness.harness.context @your-username/agent-harness.harness.verifier

Here's how the three components fit into the agent loop:

Set Up Your Bit Workspace

Create a new scope and give it any name you like. For this tutorial, the scope will be named agent-harness. You can also use an existing scope."

With your scope ready, install Bit's version manager by running this in your terminal. It handles Bit installations and keeps your version up to date across projects:

npx @teambit/bvm install

Next, initialize your workspace. This creates a workspace.jsonc configuration file at your project root and sets agent-harness as the default scope for every component you create in this workspace:

bit init --default-scope your-username.agent-harness

Replace your-username with your bit.cloud username. A successful run looks like this:

successfully initialized a bit workspace.

Open workspace.jsonc and uncomment the Node environment line. This tells Bit which runtime to use when building and compiling your components:

"bitdev.node/node-env": {}

Then run this to pull the Node environment and resolve its full dependency tree. This is a one-time step per workspace:

bit install

This will take a few minutes on first run. You'll see pnpm working through several hundred packages before it completes.

Before creating any components, run this to see the available templates for your environment:

bit templates

The template you want is module, listed under bitdev.node/node-env. Now run this to scaffold all three harness components at once:

bit create module harness/state harness/context harness/verifier

Each command generates a component folder with a TypeScript entry file, a test file and the necessary Bit configuration. You'll see output confirming all three components were created under your scope, each assigned the Node environment automatically:

One thing worth noting before you move on: harness/state, harness/context and harness/verifier are the component paths inside your workspace. When Bit publishes them to your scope, it generates a fully qualified package name by combining your scope and component path, for example @your-username/agent-harness.harness.state.

Your workspace is ready. Three component folders now exist under agent-harness/harness/, the Node environment is configured and you have a scope waiting to receive them once the implementations are complete.

Build the State Component

An agent without externalized state forgets everything between sessions. Every run starts blind: no record of what was completed, no awareness of what's in progress and no queue of what comes next. This is the failure mode that production systems are designed to prevent at the infrastructure level. The state component fixes that by reading a structured JSON file at session start and writing back to it before the session ends.

Open the agent-harness/harness/state/state.ts file generated by bit create module in the previous step and replace the generated content with this:

import { readFileSync, writeFileSync, existsSync } from 'fs';

import { resolve } from 'path';

export type HarnessState = {

  done: string[];

  inProgress: string[];

  next: string[];

};

const DEFAULT_STATE: HarnessState = {

  done: [],

  inProgress: [],

  next: [],

};

export function loadState(statePath = 'harness-state.json'): HarnessState {

  const abs = resolve(statePath);

  if (!existsSync(abs)) return { ...DEFAULT_STATE };

  const raw = readFileSync(abs, 'utf-8');

  return JSON.parse(raw) as HarnessState;

}

export function saveState(state: HarnessState, statePath = 'harness-state.json'): void {

  const abs = resolve(statePath);

  writeFileSync(abs, JSON.stringify(state, null, 2), 'utf-8');

}

HarnessState has three fields. done holds everything the agent has completed. inProgress holds whatever the agent is currently working on. next holds the queue of work still to come.

loadState reads the state file at the path you specify, defaulting to harness-state.json at the project root. If no file exists yet, it returns an empty default state rather than throwing an error. saveState writes the updated state back to the same path before the session ends. Between those two calls, your agent has a persistent, structured record of exactly where it is in the work, regardless of how many sessions it takes to get there.

Build the Context Component

Without pre-loaded context, your agent rediscovers the same dependency relationships from scratch on every run. It wastes the first part of every session figuring out what it already knew. The context component solves that by querying your Bit workspace for component dependencies and dependents before the agent takes its first action, handing it a structured map it can reason over immediately.

Open the agent-harness/harness/context/context.ts file generated by bit create module in the previous step, and replace the generated content with this:

import { exec } from 'child_process';
import { promisify } from 'util';

const execAsync = promisify(exec);
export type ComponentMeta = {

  id: string;

  description?: string;

  dependencies: string[];

  dependents: string[];

};

export type DependencyMap = Record<string, ComponentMeta>;
async function runBitShow(componentId: string): Promise<{ id: string; description?: string; dependencies: string[] }> {

  const { stdout } = await execAsyncbit show --json ${componentId}, { timeout: 30_000 });

  const data = JSON.parse(stdout);

  const deps: string[] = (data?.dependencies ?? []).map((d: { id: string }) => d.id);

  return {

    id: componentId,

    description: data?.description,

    dependencies: deps,

  };

}

async function runBitDependents(componentId: string): Promise<string[]> {

  try {

    const { stdout } = await execAsyncbit dependents ${componentId}, { timeout: 30_000 });

    return stdout

      .split('\n')

      .map((l) => l.trim())

      .filter((l) => l.length > 0 && !l.startsWith('┌') && !l.startsWith('└') && !l.startsWith('│') && !l.startsWith('─'));

  } catch {

    return [];

  }

}

export async function buildDependencyMap(componentIds: string[]): Promise<DependencyMap> {

  const map: DependencyMap = {};

  await Promise.all(

    componentIds.map(async (id) => {

      const [meta, dependents] = await Promise.all([runBitShow(id), runBitDependents(id)]);

      map[id] = { ...meta, dependents };

    })

  );

  return map;

}

There are two things worth understanding about how this works before you move on.

runBitShow shells out to bit show --json for a given component ID and parses the JSON output into a structured object containing the component's ID, description and direct dependencies. runBitDependents shells out to bit dependents and parses the plain-text output, filtering out the table-border characters Bit uses in its CLI output to leave you with a clean list of dependent component IDs.

buildDependencyMap takes an array of component IDs and runs both queries concurrently for each one using Promise.all. The result is a DependencyMap: a keyed record where each entry gives your agent a full picture of what a component depends on and what depends on it, all resolved before the first agent action fires.

Build the Verifier Component

Most agent loops use the same context to generate and judge output. That creates a real failure mode: the model can sound confident and still be wrong. A separate verifier keeps the check outside the generation loop and forces a second pass before the agent continues. The verifier component does that by sitting in a completely separate component, isolated from the generator, and evaluating output against a done condition you define before the loop starts.

Addy Osmani argues for keeping the maker away from the checker. The verifier is where that principle lives in your harness.

Open the agent-harness/harness/verifier/verifier.ts file generated by bit create module and replace the generated content with this:

export type DoneCondition = 'NO' | 'YES' | 'MAYBE' | 'IFF';

export type VerifyResult = {

  verdict: DoneCondition;

  reason: string;

  condition?: string;

};

export type VerifierOptions = {

  agentOutput: string;

  doneKeywords?: string[];

  failKeywords?: string[];

  iffPattern?: RegExp;

};

export function verify(opts: VerifierOptions): VerifyResult {

  const { agentOutput, doneKeywords = [], failKeywords = [], iffPattern } = opts;

  const normalized = agentOutput.toLowerCase();

  if (failKeywords.some((kw) => normalized.includes(kw.toLowerCase()))) {

    return {

      verdict: 'NO',

      reason: Output contains a failure signal.,

    };

  }

  if (doneKeywords.length > 0 && doneKeywords.every((kw) => normalized.includes(kw.toLowerCase()))) {

    return {

      verdict: 'YES',

      reason: All done keywords found in output.,

    };

  }

  if (iffPattern) {

    const match = agentOutput.match(iffPattern);

    if (match) {

      return {

        verdict: 'IFF',

        reason: Output satisfies pattern but requires conditional verification.,

        condition: match[0],

      };

    }

  }

  if (doneKeywords.some((kw) => normalized.includes(kw.toLowerCase()))) {

    return {

      verdict: 'MAYBE',

      reason: Some but not all done keywords found.,

    };

  }

  return {

    verdict: 'NO',

    reason: No done signals detected in output.,

  };

}

VerifierOptions takes four inputs:

agentOutput is the raw string output from the agent.
doneKeywords is a list of terms that must all appear in the output for it to pass.
failKeywords is a list of terms that immediately fail the output if any one of them appears.
iffPattern is a regular expression that triggers a conditional verdict when it matches.

The verify function evaluates in a fixed priority order. Failure signals are checked first: if any failKeyword appears in the output, the function returns NO immediately without evaluating anything else. If all doneKeywords are present, it returns YES. If iffPattern matches, it returns IFF along with the matched string as the condition the agent needs to resolve before continuing. If only some doneKeywords are present, it returns MAYBE, pausing the loop for human review. If none of those conditions are met, it returns NO.

The four verdicts map directly to actions in your agent loop. YES updates state and continues. NO retries the action. MAYBE pauses and flags for human review. IFF checks the named dependency before deciding either way.

Compose, Release, and Install

All three components are built. Next, you version them and release them to your scope so Ripple CI can pick up the export automatically.

Tag Your Components

Tag all three components with a single command. This versions the modified components selected for tagging in one command:

bit tag --message "initial implementation of harness components"

You'll see output confirming all three components were tagged at version 0.0.1:

Export to Your Scope

Now push all three components to your scope on bit.cloud:

bit export

You'll see Bit indexing your components and confirming a successful push:

You may see a warning about node-env not being loaded during export. Run bit install locally to clear it.

The moment bit export completes, Ripple CI picks up the push and kicks off a remote build job automatically. No configuration required. Head to the URL in the export output to watch compilation, tests and documentation generation run against all three components in the cloud.

Here's what a successful build looks like:

Ripple CI dashboard showing three harness components built successfully in 2 minutes 41 seconds

Install the Harness

With your components live on bit.cloud, install all three into any agent project with a single command:

bit install @your-username/agent-harness.harness.state @your-username/agent-harness.harness.context @your-username/agent-harness.harness.verifier

If you're working outside a Bit workspace, install via npm with Bit's registry instead:

npm install @your-username/agent-harness.harness.state \

            @your-username/agent-harness.harness.context \

            @your-username/agent-harness.harness.verifier \

            --registry https://node-registry.bit.cloud

Wire It All Together

With all three components installed, here's how they wire together at the boundaries of an agent session:

import { loadState, saveState } from '@your-username/agent-harness.harness.state';

import { buildDependencyMap } from '@your-username/agent-harness.harness.context';

import { verify } from '@your-username/agent-harness.harness.verifier';

// At session start

const state = loadState();

const context = await buildDependencyMap(['harness/state', 'harness/context', 'harness/verifier']);

console.log('Resuming from:', state);

console.log('Dependency map:', context);

// Agent does its work here

const agentOutput = 'scaffold auth module completed. tests passing.';

// Verify before continuing

const result = verify({

  agentOutput,

  doneKeywords: ['completed', 'tests passing'],

  failKeywords: ['error', 'failed'],

});

console.log('Verdict:', result.verdict);

// Update state based on verdict

if (result.verdict === 'YES') {

  state.done.push('scaffold auth module');

  state.inProgress = [];

  state.next = ['write tests for auth module'];

}

// At session end

saveState(state);

The context loads before the first action. The state picks up where the last session ended. The verifier evaluates output before the loop continues. When any of those three components changes in any project, tag and export from that project, and Ripple CI propagates the update to every downstream dependent automatically. That's the harness: three components, one install command, zero copy-paste.

The Harness Is Just the Start

You now have a production harness that lives outside your agent, versioned and shared across every project that needs it. When something breaks, you fix it in one place, tag it, export it and Ripple CI propagates the change automatically.

That's the difference between a harness you maintain and a harness that maintains itself.

The approach scales further than this tutorial goes. You can extend HarnessState to track token usage, session duration or retry counts. You can add a fourth component that handles context window management, trimming what gets loaded based on what the state says is already done. You can wire the verifier into a CI step so no agent output merges without passing a a done condition first.

Every extension is just another versioned component added to the same scope. Once the pattern is in place, growing the harness stops being a rewrite and starts being an addition.

DEV Community