Agent evaluation tooling is surprisingly thin. Most frameworks ship a way to run agents but not a rigorous way to tell whether they're getting better or worse across changes. You end up eyeballing outputs, or writing one-off scripts that don't compose, or skipping evals entirely because the setup cost is too high.
agentkit is my answer to this: a lightweight eval harness that works with any agent framework, records every run as a structured artifact, and lets you diff two runs to catch regressions automatically.
The core idea
bash
agentkit run --suite tasks/ --adapter my_agent.adapter:run --out baseline.json
change something
agentkit run --suite tasks/ --adapter my_agent.adapter:run --out new.json
agentkit diff baseline.json new.json
diff exits 0 if nothing regressed, exits 1 if anything did. That's a CI check. You don't need a hosted dashboard or a proprietary eval platform — just a JSON file and a diff command.
Design decisions worth explaining
No bundled agent. agentkit has no opinion about what model you use, what framework you use, or how your agent works. You supply an adapter — a single Python function that takes a Task and returns an AdapterOutput. That's the entire interface. This keeps the harness honest: it measures whatever you bring to it, not a strawman it controls.
Subprocess isolation per task. Each task runs in its own spawned subprocess with a timeout. A hung task gets killed; it doesn't hang the whole suite. This matters in practice — LLM calls timeout, tool calls block, network calls fail. An eval harness that can't survive a single bad task isn't useful.
Scoring is explicit, not implicit. Every task names a scorer — exact_match, contains, regex, or your own. The scorer sees the task's target field and the adapter's output and returns a Score. Nothing is inferred from output shape. This means you can score the same output multiple ways and the scores are always reproducible from the artifact alone.
The run artifact is the source of truth. agentkit run writes a run.json that contains every task, every output, every score, and every trace. agentkit view renders it. agentkit diff compares two of them. Nothing is held in a database or a server — the file is the record.
The diff output
suite: my-agent-suite
baseline: runs/baseline.json (2026-08-01)
new: runs/new.json (2026-08-02)
task baseline new delta
──────────────────────────────────────────────────
summarize-short pass pass —
summarize-long pass FAIL regressed
extract-entities fail fail —
classify-sentiment fail pass improved
1 regressed, 1 improved, 2 unchanged
exit 1
Exit code 1 means something regressed. Pipe it into CI and you catch regressions before they ship.
Two design problems I had to solve
Timeout that actually kills the process tree. subprocess.terminate() kills the adapter process but not its children — if the adapter spawned a tool call subprocess, that orphan keeps running. The fix: each adapter subprocess runs in its own process group (start_new_session=True), and on timeout the harness kills the whole group via os.killpg. This came up again when building xenarchos on top of agentkit — the layered timeout problem is real, not theoretical.
The task_ids=[] falsy check. agentkit's task filter is if task_ids: — an empty list is falsy, so passing task_ids=[] is treated as "no filter, run everything." This is the opposite of what you want when you've explicitly selected zero tasks (e.g., the user declined every step in a confirm gate). The fix is to route around run_suite entirely when the confirmed set is empty, building an empty Run directly. I found this while integrating xenarchos's confirm gate with agentkit's runner and the all-decline case silently ran the full suite.
What it doesn't do
No flakiness handling, no epoch tracking, no hosted dashboard, no distributed execution. These are all real problems in production eval pipelines — they're just not v0.1 problems. The scope is: run tasks, score outputs, diff runs, view traces. That scope is complete and useful on its own.
Result
44 tests passing. Five CLI verbs: run, score, diff, view, init. Works with any adapter you write. Run artifacts are plain JSON — readable by anything.
github.com/xenaarch-dev/agentkit
Top comments (0)