Agent evaluation harnesses like SWE-bench, GAIA, and WebArena each implement their own execution sandboxes, tool boundaries, and result parsers. If you want to run the same agent across multiple benchmarks, you rewrite adapter code for each one. HarnessRouter provides a single interface across these harnesses, exposing what's actually different under the hood: how they isolate state, handle timeouts, pass context between steps, and decide when a task succeeds.
The Problem: Every Harness Reinvents Execution
Agent benchmarks fragment along three axes:
- Isolation model: SWE-bench spins up Docker containers per task. GAIA runs in a shared Python process with cleanup hooks. WebArena expects a persistent browser session.
- Tool calling convention: Some harnesses expect synchronous function calls with JSON schemas. Others stream partial results or accept free-form shell commands.
- Success criteria: One harness checks exit codes. Another parses structured logs. A third compares final DOM state against a reference snapshot.
You waste cycles adapting your agent to each harness's quirks instead of improving the agent itself. The router makes these architectural choices visible and comparable.
What HarnessRouter Actually Does
HarnessRouter implements the Unified Harness Protocol (UHP), an open standard that normalizes:
- Session lifecycle: Start, pause, resume, and cancel tasks with consistent semantics.
- File handling: Upload context files, retrieve intermediate artifacts, and stream logs without harness-specific paths.
- Streaming: Subscribe to partial results as they arrive, even when the underlying harness only supports polling.
- Failure modes: Distinguish between agent errors, harness timeouts, and infrastructure failures.
The router translates between UHP and each harness's native API. It does not change how the harness executes tasks. It surfaces what each harness does differently.
Architecture: Adapter Pattern with State Normalization
┌─────────────┐
│ Your Agent │
│ (LangGraph) │
└──────┬──────┘
│ UHP calls
▼
┌─────────────────┐
│ HarnessRouter │
│ ┌─────────────┐ │
│ │ Session Mgr │ │
│ └─────────────┘ │
│ ┌─────────────┐ │
│ │ Adapters │ │
│ └─────────────┘ │
└────┬───┬───┬────┘
│ │ │
▼ ▼ ▼
┌────────┐ ┌────────┐ ┌────────┐
│SWE-bench│ │ GAIA │ │WebArena│
└────────┘ └────────┘ └────────┘
Each adapter implements:
- Isolation translation: Map UHP session IDs to Docker container names, process IDs, or browser tabs.
- Tool call normalization: Convert UHP tool schemas to harness-specific formats (function calls, shell commands, API requests).
- Result parsing: Extract success/failure signals from exit codes, log patterns, or DOM snapshots.
The session manager tracks active tasks, handles cancellation, and enforces timeouts. It does not execute agent logic. It routes requests and normalizes responses.
Implementation: How Adapters Handle Isolation
| Harness | Isolation Model | State Cleanup | Timeout Mechanism |
|---|---|---|---|
| SWE-bench | Docker per task | Container teardown |
docker stop + SIGTERM |
| GAIA | Shared process + hooks | Python atexit hooks |
Thread interrupt |
| WebArena | Persistent browser tab | Page reload + cookies | Playwright timeout |
SWE-bench adapter:
- Starts a new container on session creation.
- Mounts task files as volumes.
- Streams stdout/stderr via Docker logs API.
- Parses exit code to determine success.
GAIA adapter:
- Registers cleanup hooks before task execution.
- Captures Python exceptions as agent errors.
- Uses threading to enforce timeouts (non-preemptive).
WebArena adapter:
- Reuses a single Playwright browser instance.
- Opens a new tab per session.
- Compares final DOM against reference using CSS selectors.
- Timeout triggers
page.close().
Tool Call Translation Example
Your agent calls a UHP tool:
{
"tool": "run_command",
"args": {
"command": "pytest tests/",
"timeout": 30
}
}
SWE-bench adapter translates to:
docker exec <container_id> bash -c "pytest tests/"
GAIA adapter translates to:
subprocess.run(["pytest", "tests/"], timeout=30, capture_output=True)
WebArena adapter translates to:
await page.evaluate(() => {
return window.runCommand("pytest tests/");
});
The router does not validate whether pytest is available. It delegates that to the harness. It only ensures the call shape matches what the harness expects.
Streaming and Partial Results
Some harnesses stream output. Others only return results after task completion. The router normalizes this:
- Streaming harnesses (SWE-bench Docker logs): Forward chunks as they arrive.
- Polling harnesses (GAIA): Poll every 500ms and emit deltas.
- Batch harnesses (WebArena): Buffer until task completes, then emit full result.
Clients subscribe to a UHP stream endpoint. The router handles the translation. Your agent code does not change.
Failure Handling and Observability
The router distinguishes three failure classes:
- Agent errors: The agent returned invalid tool calls or exceeded token limits.
- Harness timeouts: The task ran longer than the harness allows.
- Infrastructure failures: Docker daemon crashed, browser process died, network partition.
Each adapter maps harness-specific signals to UHP error codes:
{
"error": "harness_timeout",
"harness": "swe-bench",
"details": {
"container_id": "abc123",
"timeout_seconds": 300,
"last_output": "Running test 47 of 200..."
}
}
Observability hooks emit structured logs for:
- Session lifecycle events (start, pause, resume, cancel, complete).
- Tool call latency per harness.
- Adapter translation overhead.
- Failure rates by error class.
You can export these to OpenTelemetry or Prometheus.
Deployment Shape
HarnessRouter runs as a self-hosted service. You deploy it alongside your agent infrastructure:
- Single binary: Go binary with embedded adapters.
- Configuration: YAML file mapping harness names to connection strings (Docker socket, Python interpreter path, Playwright endpoint).
- API: REST or gRPC, depending on your orchestration layer.
For LangGraph users, the router exposes a tool that wraps UHP calls. Your agent graph invokes the tool. The router handles harness selection and translation.
from langgraph import Graph
from harnessrouter import UHPTool
graph = Graph()
graph.add_node("evaluate", UHPTool(harness="swe-bench"))
graph.add_edge("start", "evaluate")
Security Boundaries
The router does not sandbox agent code. It assumes the harness provides isolation. If the harness allows arbitrary code execution (like SWE-bench), the router inherits that risk.
Key boundaries:
- Session isolation: One agent cannot access another agent's session state.
- File access: Agents can only read/write files within their session directory.
- Network: Harnesses may allow or block outbound network calls. The router does not enforce this.
If you need stricter isolation, run the router inside a VM or use a harness that provides sandboxing (like SWE-bench with gVisor).
When to Use HarnessRouter
Use it when:
- You evaluate agents across multiple benchmarks and want consistent tooling.
- You need to compare how different harnesses handle the same task.
- You want observability into harness-level failures without rewriting adapter code.
Avoid it when:
- You only use one harness and can integrate directly.
- Your agent requires harness-specific optimizations (like SWE-bench's Docker layer caching).
- You need sub-100ms latency (the router adds 10-50ms per call for translation).
Technical Verdict
HarnessRouter is useful if you run agents across multiple evaluation harnesses and want to avoid writing custom adapters for each one. It does not improve agent performance. It reduces integration friction and makes harness differences explicit. The trade-off is translation overhead and the assumption that harnesses can be normalized (which breaks down for deeply custom benchmarks). If you control the harness or only use one, skip the router and integrate directly. If you evaluate across SWE-bench, GAIA, and WebArena, the router saves you from maintaining three adapter codebases.
Top comments (0)