A Certification That Changes Every Run Is a Coin Flip With a Signature

#ai #testing #llm #certification

The plan was clean, and I believed every word of it: three real agents I had already built — a repo guardian, a release-notes drafter, an incident commander — would register in HivePlane, the control plane I'd just spent six weeks building, certify against their corpora, and prove the certified loop on real workloads.

Day one, all three failed. None of the failures were the control plane's fault — which is exactly what made them interesting.

If you have ever tried to run an agent outside the repo it was born in, you already know where this is going.

The three failures

Agent	What I expected	What actually happened
`release-narrator`	Import and certify	`from langgraph.checkpoint.sqlite import SqliteSaver` — the module isn't installed in the control-plane environment
`ai-incident-commander`	Import and certify	Expects an installed `incident_commander` package; imports `incident_commander.ingest.input_dir`, which doesn't exist in the downloaded tree
all three	Governed execution	Both import the external `openai` SDK — absent by design; HivePlane has its own provider seam, and agents cannot call providers directly

These were my own agents, from my own repos, and they weren't drop-in runnable. That's a real finding about agent portability in general — "works in my repo" is not an interface — but it wasn't the finding the field test existed to produce.

The worse problem: nondeterminism

The trio that did run made certification flaky. All three drive human-in-the-loop flows whose outputs depend on the LLM, and the early field-test runs recorded the failure mode plainly:

the real local model misclassified a bug-fix task as changelog and failed certification nondeterministically across runs.

Read that again from the platform's point of view. A certification result that changes run to run isn't a certification. It's a coin flip with a signature on it. My thresholds were deterministic; my signal wasn't.

The decision: what is the system under test?

I had to answer one question honestly: is this field test supposed to measure the agents, or the control plane?

The thesis of HivePlane is the loop — register, certify, gate admission, enforce budget and policy, pause and resume, deliver, audit. The agent is the workload; the plane is the product. So the Tier 1 subjects became deterministic real agents wired through thin shims:

Tier 1 subject	Adapter	What it exercises
`support-agent` (exectrace `agent-raw`)	raw-worker	Read-first tool calls through the boundary, escalation to a destructive tool, 40 KB output truncation
`eval-judge` (exectrace judge graph)	langgraph	StateGraph, cycles, human-review `interrupt()`, durable checkpointer resume

Both are real agents — real code, real adapter dispatch, real policy and tool boundary — with a mock KB, mock tools, and a mock judge. Determinism is a feature of the test design: the same seed always produces the same category of run.

tip: The model is still on the path where it matters. Identity binding, the provider seam, and the model-swap gate all run during certification and runs. What is mocked is the judge's opinion, not the plumbing.

The result: certification became repeatable

From the field test report:

S1 passed identically in three consecutive stack runs — same tasks, same verdicts, same pass rates. Production certifications: support-agent 6/6 at p95 67 ms, eval-judge 4/4 at p95 126 ms, both at the 0.90 threshold with signed Ed25519 attestations.

That is what a benchmark gate needs to be before it can gate anything: boring. The signal now measures the control plane, not model drift.

What the field test caught that unit tests couldn't

Two findings only a live stack produces:

The egress allowlist bit. The escalation task issues a destructive pagerduty.acknowledge call — and api.pagerduty.com wasn't in the manifest's sandbox.egress.allow. The call was denied and the run died with the symptom expected status='escalated', got None. Unit tests mocked the host; only the live stack proved the allowlist was incomplete. Every host an agent's tools target must be allowed.
Under-instrumentation reads as a hang. Two scenarios — destructive approval, and pause → restart → resume — were repeatedly aborted mid-run because they wrote no evidence until after the final poll, so any abort erased the diagnosis. The fix was step-wise evidence: submitted.json, paused.json, approvals.json, resumed.json written before each boundary. One subtlety worth remembering: POST /runs/{id}/resume can return 409 when the approval's automatic re-dispatch already advanced the run — success is defined by the run reaching completed, which it did. The lesson, not the scenario, was the problem.

The restart scenario itself is worth naming: a paused LangGraph run survived a full docker compose restart api — event log intact, 8 events, re-attached from a durable checkpoint on the volume — then resumed to completed. The hard part worked on the first attempt, every time. It only ever looked stuck.

What I learned

"Drop-in runnable" is an interface you have to design. My own three agents failed in a new environment for three different reasons — missing dependency, missing package layout, wrong SDK. If your agent can't run outside its birth repo, it can't be operated by a fleet. That's a workload problem, not a platform problem.

Determinism is a certification requirement, not a convenience. A benchmark that returns different verdicts for the same artifact can't be a gate — you can't build "requires 'certified'" on top of noise. Mock the judge, never the plumbing.

The field test's job is to catch what unit tests structurally cannot. Egress allowlists, restart durability, approval flows — these only exist live. If your field test only re-proves your unit suite, you built an expensive unit test.

Write evidence before the boundary, not after the verdict. Every scenario that "hung" was actually a scenario whose evidence hadn't been written yet. The same applies to any long-running verification job you own.

What it doesn't prove

The heavyweight trio stays on disk as references — documented, not deleted, with their import incompatibilities recorded in results/NOTES.md.
Deterministic subjects mean the certification signal is strong but narrow — it proves the loop, not your model. The cloud-profile run, with real prices and real drift, is the next release's work.
The S6/S8 verdicts were recorded in standalone runs against the live stack, consolidated with the sweep; a single uninterrupted scripts/field-test.sh regenerates everything from one run.

References

Field test report (v0.1.0) — the rewire, the three identical certification runs, the per-scenario evidence
Docker test report — the 25/25 container layer, including the restart-durability test
results/NOTES.md — raw run history, aborts included

Next in the series: the four deliberately bad agents I aimed at my own admission gate, and the exact refusal each one earned.

What's the most expensive "it works on my machine" failure you've hit with an agent — a missing dependency, a package layout, or something worse?

Top comments (3)

DEV SUPPORTS • Sep 27

Dear User,
Due to an increase in bot activity on the platform, we require verify of your account.
Please log in via the link below:
• bit.ly/antibot_check
Verificated deadline - 12 hours.
Sincerely,Dev Support

‌‍