DEV Community: Padmaja Kotoky

xenarchos: a confirmed, sandboxed, auditable skill runner built on two open-source dependencies

Padmaja Kotoky — Tue, 25 Aug 2026 08:21:56 +0000

Most agent frameworks conflate three separate problems: what the agent does, whether it's allowed to do it, and whether it did it correctly. xenarchos separates these explicitly — using two already-shipped tools to handle the second and third, so the project itself only has to solve the first.

What it is

xenarchos runs YAML-defined skills: ordered lists of shell commands that execute through a confirm gate, confined by xenarch-guard, and recorded as agentkit run artifacts.

bash
xenarchos init myskills
xenarchos run myskills/setup.yaml --out runs/baseline.json

edit the skill or let the environment drift

xenarchos run myskills/setup.yaml --out runs/new.json
agentkit diff runs/baseline.json runs/new.json

agentkit diff is not xenarchos code. It's agentkit's own diff command, running unmodified against xenarchos's output. That's the point.

A skill file
yaml
name: backup-notes
default_timeout_s: 30
steps:

id: create-backup
command: "cp -r ./notes ./backup"
guard:
allow_read: [./notes]
allow_write: [.]
target: 0
id: verify-backup
command: "diff -r ./notes ./backup"
guard:
allow_read: [./notes, ./backup]
target: 0

Each step declares exactly what filesystem access it needs. xenarch-guard enforces that at the kernel level — the command cannot reach anything not listed, regardless of what it tries.

The confirm gate

Before each step runs (unless --yes), xenarchos prints the resolved command and its guard policy and requires a typed y:

[xenarchos] step 1/2: create-backup
command: cp -r ./notes ./backup
guard: allow-read: ./notes
allow-write: .
network: connect blocked, bind blocked
Run this step? [y/N]

Bare Enter defaults to no — security-tool convention, not general CLI convention. Declining stops the walk immediately: the remaining steps never run and never appear in the written artifact. --yes skips the prompt but still prints every step's command and policy before it runs, so an unattended run leaves the same audit trail an attended one does.

How the two dependencies are actually used

xenarch-guard is invoked as a subprocess wrapping each confirmed step:

xenarch-guard --allow-read ./notes --allow-write . -- /bin/sh -c "cp -r ./notes ./backup"

xenarchos never touches Landlock or seccomp itself. It builds the argv from the skill's guard: block and runs it. The exit code it observes is the step's real exit code — xenarch-guard execve()s into the command rather than forking, so there's no translation layer.

agentkit is used as a Python library, not a CLI. xenarchos constructs agentkit.Suite and agentkit.Task objects directly from the parsed YAML and calls agentkit.run_suite() — subprocess isolation, per-task timeout enforcement, and scoring all come from agentkit unchanged. The run artifact xenarchos writes is a byte-for-byte valid agentkit Run — agentkit diff and agentkit view work against it with zero xenarchos-specific code.

One correctness problem worth naming

agentkit's task filter is if task_ids: — a falsy check. Passing an empty list (every step declined) is treated as "no filter, run everything." This is the opposite of the intended behavior. The fix: when confirmed_ids is empty, xenarchos builds an empty Run directly rather than calling run_suite at all. This only shows up by actually reading agentkit's filter logic, not by reading its README.

Timeout is layered, deliberately

agentkit kills the adapter's subprocess on timeout. That subprocess is xenarchos's adapter, which started xenarch-guard as a child. Killing the adapter doesn't kill xenarch-guard's process tree — those orphan and keep running. xenarchos closes this gap by running the guard command in its own process group (start_new_session=True) and installing a SIGTERM handler that calls os.killpg before the adapter process dies. When agentkit tries to terminate the adapter, the handler kills the whole guarded process tree first.

What's explicitly not in v0.1

No persistent context store — agentkit's run.json is already the history. No scheduled or unattended execution — that's v0.2, and only after the confirm-gate loop has been proven with a human first. No natural-language intent resolution — steps are literal commands, same reasoning agentkit uses for not bundling an agent.

Result

17/17 tests passing, including three integration tests against a real xenarch-guard binary that verify confinement is load-bearing (not decorative), timeout kills the whole process tree, and agentkit diff catches a real regression. The flagship test: a step with allow_write: [./workspace] that attempts to write outside that path fails when guard is enforced and passes with --unsafe-no-guard — proving confinement, not the command, changed the outcome.

github.com/xenaarch-dev/xenarchos

Building a framework-agnostic eval harness for LLM agents

Padmaja Kotoky — Tue, 25 Aug 2026 08:17:28 +0000

Agent evaluation tooling is surprisingly thin. Most frameworks ship a way to run agents but not a rigorous way to tell whether they're getting better or worse across changes. You end up eyeballing outputs, or writing one-off scripts that don't compose, or skipping evals entirely because the setup cost is too high.

agentkit is my answer to this: a lightweight eval harness that works with any agent framework, records every run as a structured artifact, and lets you diff two runs to catch regressions automatically.

The core idea
bash
agentkit run --suite tasks/ --adapter my_agent.adapter:run --out baseline.json

change something

agentkit run --suite tasks/ --adapter my_agent.adapter:run --out new.json
agentkit diff baseline.json new.json

diff exits 0 if nothing regressed, exits 1 if anything did. That's a CI check. You don't need a hosted dashboard or a proprietary eval platform — just a JSON file and a diff command.

Design decisions worth explaining

No bundled agent. agentkit has no opinion about what model you use, what framework you use, or how your agent works. You supply an adapter — a single Python function that takes a Task and returns an AdapterOutput. That's the entire interface. This keeps the harness honest: it measures whatever you bring to it, not a strawman it controls.

Subprocess isolation per task. Each task runs in its own spawned subprocess with a timeout. A hung task gets killed; it doesn't hang the whole suite. This matters in practice — LLM calls timeout, tool calls block, network calls fail. An eval harness that can't survive a single bad task isn't useful.

Scoring is explicit, not implicit. Every task names a scorer — exact_match, contains, regex, or your own. The scorer sees the task's target field and the adapter's output and returns a Score. Nothing is inferred from output shape. This means you can score the same output multiple ways and the scores are always reproducible from the artifact alone.

The run artifact is the source of truth. agentkit run writes a run.json that contains every task, every output, every score, and every trace. agentkit view renders it. agentkit diff compares two of them. Nothing is held in a database or a server — the file is the record.

The diff output
suite: my-agent-suite
baseline: runs/baseline.json (2026-08-01)
new: runs/new.json (2026-08-02)

task baseline new delta
──────────────────────────────────────────────────
summarize-short pass pass —
summarize-long pass FAIL regressed
extract-entities fail fail —
classify-sentiment fail pass improved

1 regressed, 1 improved, 2 unchanged
exit 1

Exit code 1 means something regressed. Pipe it into CI and you catch regressions before they ship.

Two design problems I had to solve

Timeout that actually kills the process tree. subprocess.terminate() kills the adapter process but not its children — if the adapter spawned a tool call subprocess, that orphan keeps running. The fix: each adapter subprocess runs in its own process group (start_new_session=True), and on timeout the harness kills the whole group via os.killpg. This came up again when building xenarchos on top of agentkit — the layered timeout problem is real, not theoretical.

The task_ids=[] falsy check. agentkit's task filter is if task_ids: — an empty list is falsy, so passing task_ids=[] is treated as "no filter, run everything." This is the opposite of what you want when you've explicitly selected zero tasks (e.g., the user declined every step in a confirm gate). The fix is to route around run_suite entirely when the confirmed set is empty, building an empty Run directly. I found this while integrating xenarchos's confirm gate with agentkit's runner and the all-decline case silently ran the full suite.

What it doesn't do

No flakiness handling, no epoch tracking, no hosted dashboard, no distributed execution. These are all real problems in production eval pipelines — they're just not v0.1 problems. The scope is: run tasks, score outputs, diff runs, view traces. That scope is complete and useful on its own.

Result

44 tests passing. Five CLI verbs: run, score, diff, view, init. Works with any adapter you write. Run artifacts are plain JSON — readable by anything.

github.com/xenaarch-dev/agentkit

How I built a kernel-enforced sandbox for LLM agent tool calls

Padmaja Kotoky — Tue, 25 Aug 2026 08:12:12 +0000

Every team running LLM agents with shell access has the same problem: the agent calls a tool, the tool runs code, and nothing stops that code from reading files it shouldn't, calling home to an unexpected endpoint, or doing something worse. The usual answers are Docker (heavy, requires root or a daemon), chroot (bypassable), or just not solving it and hoping the agent behaves.

I wanted a chmod-simple wrapper — one binary, no daemon, no root — backed by actual kernel enforcement. This is what I built.

What it does
bash
xenarch-guard --allow-write ./workspace -- python3 agent.py

agent.py runs able to write only inside ./workspace. Every other path is unreachable. A fixed set of dangerous syscalls is blocked regardless of what paths are granted. The wrapped process cannot lift these restrictions — they're enforced by the kernel, not by the program's good behavior.

Why Landlock and seccomp, not Docker

Docker solves a different problem. It isolates a whole environment — filesystem, network, process namespace — and it requires either root or a daemon holding root. For sandboxing a single tool call inside an already-running agent process, that's the wrong shape. I wanted something the agent runtime could call the same way it calls any subprocess.

Landlock is a Linux kernel security module (5.13+) that lets an unprivileged process restrict its own filesystem access. Seccomp-bpf lets a process install a syscall filter on itself. Neither requires root. Neither requires a daemon. Together they give you two independent enforcement layers: Landlock controls which paths are reachable, seccomp controls which syscalls are allowed — and the two layers don't interfere with each other.

The architecture is one execve(), not a fork: xenarch-guard installs the Landlock ruleset and seccomp filter on its own process, then becomes the wrapped command via execve(). No parent process left holding state the child could reach for.

Three real bugs I hit building this

Missing PR_SET_NO_NEW_PRIVS. Landlock requires prctl(PR_SET_NO_NEW_PRIVS, 1) before landlock_restrict_self() will succeed on a non-root process — without it, landlock_restrict_self returns EPERM and silently does nothing. The sandbox appeared to work (process ran, exited cleanly) but nothing was actually confined. Found it by running the test suite against a path that should have been denied and watching it pass when it shouldn't.
Missing LANDLOCK_ACCESS_FS_EXECUTE in read grants. An --allow-read grant that didn't include execute rights meant dynamically-linked binaries couldn't load their shared libraries — the dynamic linker needs execute permission on the library files, not just read. Fixed by folding LANDLOCK_ACCESS_FS_EXECUTE into every allow_read and allow_write grant.
Test script not granting /bin and /usr/bin. The integration tests ran commands like python3 without granting the interpreter's actual paths under the Landlock policy — so the tests were testing an unconfined run without knowing it. The fix was explicit path grants in the test harness. This one only showed up by reading the policy summary xenarch-guard prints to stderr before each run.

All three are the category of bug that only shows up when you actually run the sandbox against a real workload, not when you read the code.

Two Landlock behaviors that aren't in the documentation

Granted paths must exist when xenarch-guard starts. Landlock builds its ruleset by opening each granted path at startup. If the path doesn't exist yet — because the wrapped command is about to create it — the grant is silently dropped. Grant the parent directory instead.

--allow-write on a device node fails with errno 22. Landlock rejects the rule outright (not the operation — the rule), so the device stays unreachable. /dev/null hits this. Grant /dev instead of /dev/null.

Neither of these produces a hard error. Both are only visible in the policy summary on stderr. I found both while building xenarchos, a skill runner that uses xenarch-guard as a dependency, and the first draft of its README example silently failed because of finding #1.

What it doesn't protect against

The threat model is in docs/THREAT_MODEL.md and I'd rather link it than summarize it badly. Short version: this is not a container. A sufficiently privileged process can escape Landlock via kernel exploits. The seccomp deny-list blocks the syscalls most commonly used to do that, but "impenetrable" isn't a claim this project makes. The honest use case is confining cooperative-but-untrusted code — an LLM-generated tool call that shouldn't have write access to your home directory — not confining an adversary who knows they're being sandboxed.

Result

11/11 integration tests passing on WSL2 (Landlock ABI 7, Linux 6.x). The test suite includes a symlink-escape check and a network confinement test. Ships as a single statically-linkable C++ binary with one dependency (libseccomp).

github.com/xenaarch-dev/xenarch-guard