DEV Community

Cover image for I Ran 170 Agent Goals for $0.49. The Field Test Found 10 Issues That Unit Tests Never Would.
Debashish Ghosal
Debashish Ghosal

Posted on

I Ran 170 Agent Goals for $0.49. The Field Test Found 10 Issues That Unit Tests Never Would.

#ai #llm #infrastructure #testing

This is article 4 in a series about building PlannerCritic, an open-source engine where one LLM writes a plan and a second LLM reviews it. Article 1 covers the 157-goal v0.1.0 field test. Article 2 is about the critic severity bug. Article 3 is about the planner capability gap. This one is a practical guide to field testing agent systems — from 157 goals at $0.30 in v0.1.0 to 170 goals at $0.49 in v0.2.1, and what 10→0 issues across three releases teaches about release engineering.

Updated for v0.2.1: The original article covered the v0.1.0 field test — 157 goals, $0.30, 10 issues found. Since then, the engine shipped v0.2.0 (170 goals, 31 code-review bugs fixed, 0 field-test-found issues) and v0.2.1 (170 goals, 10 more code-review bugs fixed, 0 field-test-found issues, live-critic boundary evaluator, operational benchmark). This update tells the full arc: from a diagnostic tool that found 10 issues to a regression gate that finds 0 — and why 0 is the hardest result to earn.

If a field test returns 0 failures, your first instinct should be distrust. In v0.1.0, 0 failures meant our harness was silently broken. In v0.2.1, 0 field-test issues meant something very different: code review caught all 41 bugs before the LLM ran, turning the field test from a diagnostic sweep into an immutable regression gate.

Why I Built a Field Test

Unit tests test what you think to test. They use hand-crafted inputs that match your assumptions. They pass even when the system is wrong in ways you didn't anticipate.

I know this because 57 of 65 assertion files were in the wrong format and the harness didn't even notice. It just quietly returned 0/0 results. No crash. No error. Just silence.

I needed a field test that would run real goals against a real LLM and tell me what actually broke.

The v0.1.0 Field Test: 157 Goals, $0.30, 10 Issues

The first field test was a diagnostic. It found 10 issues across 157 goals in 35 domains for $0.30 in 60 minutes. Only 1 was a traditional failure. The rest were design problems, prompt gaps, and harness bugs that would have shipped silently.

What the Field Test Found (v0.1.0)

1 True Failure: The planner prompt didn't explain the branches schema. The LLM responded with kind: "rollback" and arrays of task objects where strings were required.

4 Design Issues:

  • The preconditions gate was too strict — established_by expected a task ID or env: prefix; the LLM wrote bare fact names. Unit tests were green.
  • 57 of 65 assertion files were in the wrong format. The harness silently produced 0/0 results.
  • The critic severity contract was wrong — the critic was blocking plans for completeness concerns instead of concrete defects.
  • The dimension dispatch had a signature mismatch — run_budget() takes 4 arguments but dispatch passed 5. Budget and replan dimensions showed 0/0.

2 Harness Bugs:

  • Cross-dimension state was lost — in-memory SQLite store reset between runs.
  • The results parser read the wrong JSON path — trace.get("status") instead of trace["result"]["status"]. Five PASS scenarios were misreported as FAIL.

1 Model Limitation: Local models (Qwen3.5-4B, Qwen3.5-9B) couldn't produce structured JSON.

2 Fundamental Properties:

  • The LLM critic is non-deterministic — strict goals re-run with revision_cap=4 all escalated; the critic found different blockers on each revision.
  • A stronger model did not close the planner gap — GPT-4o produced the same defect patterns as gpt-4o-mini.

What the v0.1.0 Field Test Cost

$0.30 in LLM API calls. 60 minutes of wall-clock time. Cheaper than debugging one production incident caused by a bad plan.

The v0.2.0 Field Test: 170 Goals, $0.40, 0 Issues — Code Review Found 31

The v0.2.0 field test was fundamentally different. v0.1.0 was a diagnostic tool that found 10 issues in a greenfield engine. v0.2.0 was a validation tool that confirmed 31 bug fixes + 5 new enterprise domain packs + 6 new safety mechanisms all work correctly.

The field test itself found zero new issues — the code review found all 31 bugs before the field test ran.

What Changed

The v0.2.0 field-test program was substantially stronger than v0.1.0's, and that directly produced a higher-quality release:

  • Scope grew ~8%: 170 goals across 40 domains (v0.1.0: 157 goals, 35 domains) — 14 new goals across 5 new enterprise domains (identity management, multi-agent operations, site reliability engineering, supply chain policy, and FinOps/greenfield) + 3 new adversarial-policy goals.
  • Code review before field test found 31 bugs vs. v0.1.0's 10 field-test-found issues. Catching bugs via code review ($0, 2 hours) before the LLM field test ran meant the field test validated fixes instead of discovering them — a higher-leverage, cheaper, faster loop.
  • Pre-amendment eliminated the release-gate failure on the first run. v0.1.0 required post-hoc amendment of 81 strict goals; v0.2.0 pre-amended 89 strict goals before execution and the release gate passed 100% on the first run.
  • New deterministic safety mechanisms were validated, not just built. Deterministic precondition closer (#131), topological auto-repair (#130), and oscillation detection (#152) were all exercised by the 170-goal sweep.
  • A security oracle was added. v0.1.0 had none; v0.2.0 added SWE-bench-derived evaluation: 7/7 correct plans pass, 35/35 flawed variants blocked, 21 injection traps generated.
  • 3 benchmarks added (auto-repair, rollback credibility, family-histogram stasis).
  • 90 deterministic subsystem tests added (0.7s, $0) as a fast regression gate alongside the LLM field test — v0.1.0 relied on the LLM sweep alone.

v0.2.0 Results

Metric Result
Balanced goals approved 73/73 (100%)
Strict goals escalated 97/97 (100%)
Adversarial goals escalated 8/8 (100%)
True failures 0
Deterministic gate passes 170/170 (100%)
Security oracle 7/7 correct, 35/35 flawed blocked, 21 traps
Scorecard A PASS
Code-review bugs fixed 31
Field-test-found bugs 0

The v0.2.1 Field Test: 170 Goals, $0.49, 0 Issues — Code Review Found 10 More

v0.2.1 is a patch release. No new goals, no new domains, no schema changes. The field test is a regression sweep — re-run the same 170 goals and diff every verdict against the published v0.2.0 results.

The code review (#222) found 10 bugs in the M11 hardening diff. The field test validated the fixes. 30 verdict deltas vs v0.2.0 — all attributable, zero unexplained.

What the Code Review Found Before the Field Test (#232–#241)

Issue Finding Why It Matters
#232 Histogram cycling detector dead under default config A safety signal shipped but was unreachable unless you manually raised revision_cap above 3. The detector was there. It just never fired.
#233 Rollback-credible gate emitted bare task id as message The blocker said "t2". No description, no suggested fix. An operator would have no idea what to do.
#234 Finding ids collided — distinct defects merged silently Two different producers offending against the same consumer produced byte-identical finding ids. The escalation system's {f.id: f} merge collapsed one defect.
#235 Live-critic runner lost entire run on mid-trial exception One transient LLM timeout discarded 60 completed audits. No partial report, no error record.
#236 Adapter import test gutted to pass A test named test_all_adapters_importable that asserted nothing. It would pass forever, even if every adapter was broken.
#237 Suggested-fix printed task id where group name belonged "Move out of parallel group 'deploy'" — but 'deploy' is a task id, not a group name.
#238 Approval authority enforcement not wired to any shipped surface Test-proven but unreachable from CLI/HTTP/MCP. Documented as F-14, deferred to v0.3.0.
#239 Evidence-drift metric pooled explanations across trials A deterministic 2-finding critic scored drift=1.0 because the metric counted intra-trial variety as cross-trial drift.
#240 ApprovalGate stamped ambient goal posture, not contract A STRICT contract bound onto a BALANCED goal produced an ApprovedPlan that said BALANCED. Downstream audits read the wrong regime.
#241 Content hash preserved criteria insertion order Same rules in different order hashed differently. Latent today (single-criterion binding), but breaks equivalence once multi-criterion contracts exist.

Every fix shipped with a regression test written RED-first (verified failing on pre-fix code, then made green by the fix).

The Live-Critic Boundary Run (#218) — The First Measurement of Critic Non-Determinism

v0.2.1 added a new test that v0.1.0 and v0.2.0 didn't have: send the #171 boundary-case corpus through the real critic model × 5 trials and measure what happens when you ask the same question 5 times.

The critic is 100% non-deterministic — and that's fine.

gpt-4o-mini produces a different verdict and different explanation on every trial of identical input (label_flip_rate=1.0, evidence_drift_rate=1.0). Yet it never under-claims a seeded defect (family_migration_rate=0.0, underclaim_approvals=0). The deterministic gates are the security authority — the LLM critic's non-determinism is safe because it can only add findings, never suppress gate blockers.

Metric Value What It Means
label_flip_rate 1.000 The critic changes its verdict on every trial of identical input
evidence_drift_rate 1.000 The critic invents different explanations every trial
family_migration_rate 0.000 No seeded defect landed in an advisory family
underclaim_approvals 0 No defective plan got zero blockers

This is the finding I didn't expect: the critic is maximally non-deterministic, and it doesn't matter. The safety contract doesn't depend on the critic being consistent — it depends on the critic always finding something on defective plans. And it does.

Takeaway: Deterministic gates own the under-claim direction (preventing bad plans from slipping through), while code-enforced severity allowlists own the over-claim direction. The LLM critic can be 100% non-deterministic and still completely safe.

The Operational Benchmark (#221) — Before/After Numbers

v0.2.1 added the before/after numbers the community asked for:

Metric Value
Latency (approved) p50 13.86s
Latency (escalated) p50 27.82s
Mean blockers per goal 2.58
Mean advisories per goal 1.86
Escalation decisions per 100 goals 58.0
Mean LLM calls per goal 1.4
Median revisions to resolution 1.0

The median revisions to resolution is 1.0 — most goals resolve in a single revision. The deterministic precondition closer and topological auto-repair are doing their job: they fix ordering and dependency defects without calling the LLM at all.

v0.2.1 Results

Metric v0.2.0 v0.2.1 Delta
Goals swept 170/170 170/170 same
Balanced approved 73/73 (100%) 73/73 (100%) same
Strict escalated 97/97 (100%) 96/97 (99%) 1 transient provider error
Adversarial aborted 8/8 8/8 same
Verdict deltas vs prior — 30 all attributable
Deterministic tests 90 1295 +1205 (contract tests for every new gate, evaluator, and schema)
Benchmarks 3 3 +operational, +boundary
Code-review bugs fixed 31 10 —
Field-test-found bugs 0 0 same
plan_oscillation_detected 0 5 #152 now fires
Coverage 91.62% 91.58% -0.04% (accepted)

The 30 Verdict Deltas — All Attributable

30 goals changed their verdict or reason code between v0.2.0 and v0.2.1. Every single one is attributable:

  • LLM non-determinism: gpt-4o-mini produces different findings across runs. The engine's safety contract is tolerance-driven (balanced→approve, strict→escalate), not finding-driven. The verdict flips between converged_stalled and revision_cap_reached depending on which specific blockers the critic found this time.
  • #152 structural oscillation now fires: 5 goals escalated with plan_oscillation_detected in v0.2.1 (0 in v0.2.0). The #232 fix made the detector reachable under default config. It detects cycling earlier and terminates the loop sooner — saving LLM calls.
  • 1 transient provider error: mch-04-blast-radius hit planning_unavailable — OpenRouter returned an error. The engine correctly failed closed (escalated as error, not approved).

Zero unexplained deltas. No delta is attributable to a code-review fix changing engine behavior — the fixes improved internal consistency (gate ids, message quality, fault isolation) without altering the approve/escalate decision logic.

The Evolution: From Diagnostic to Regression Gate

The arc across three releases tells a story about how field testing evolves:

Release Goals Cost Issues Found by Field Test Issues Found by Code Review
v0.1.0 157 $0.30 10 0
v0.2.0 170 $0.40 0 31
v0.2.1 170 $0.49 0 10

The field test went from a diagnostic (found 10 issues in a greenfield engine) to a validation tool (confirmed 31 fixes in v0.2.0) to a regression gate (confirmed 10 more fixes, diffed against the published baseline, zero unexplained deltas).

The cost went from $0.30 to $0.49. The value went from finding bugs to proving their absence.

What I Learned Across Three Field Tests

1. Field tests find design issues, not just code bugs

In v0.1.0, only 1 of 10 issues was a traditional failure. The rest were design problems that would have shipped silently. The harness quietly lying about assertion results was worse than a crash would have been.

Lesson: A field test harness that executes zero assertions for a module must fail hard — 0/0 is an error state, not a pass.

2. Real LLM data is essential

Every issue that involved the LLM behaving unexpectedly was invisible in unit tests with hand-crafted inputs. The preconditions gate bug was hiding behind green tests.

Lesson: Unit tests can't replace field tests. They test different things.

3. Code review before field test is more efficient than field test as diagnostic

v0.1.0 used the field test to find 10 issues (~$0.30 + 60 min). v0.2.0 used code review to find 31 bugs ($0 + 2 hours). v0.2.1 found 10 more ($0 + 1 hour). The field test found 0 in both cases.

Lesson: For a mature engine with a proven corpus, code review before field test is the higher-leverage activity. The field test validates; the code review diagnoses.

4. LLM non-determinism is real — and the security design accounts for it

30 verdict deltas between v0.2.0 and v0.2.1 are all attributable to gpt-4o-mini producing different findings across runs. The #218 live-critic boundary run measured this directly: label_flip_rate=1.0, evidence_drift_rate=1.0 — the critic changes its verdict and explanation on every trial of identical input. Yet family_migration_rate=0 and underclaim_approvals=0 — it never under-claims a seeded defect.

Lesson: Deterministic gates own the under-claim direction (preventing bad plans from slipping through), while code-enforced severity allowlists own the over-claim direction. The LLM critic can be 100% non-deterministic and still completely safe.

5. The #152 oscillation signal saves revisions

3 goals now escalate with plan_oscillation_detected instead of revision_cap_reached. The signal detects cycling earlier and terminates the loop sooner.

Lesson: The oscillation signal fires in practice (3/97 strict goals) and saves LLM calls.

6. Transient provider errors are not engine defects

1 goal hit planning_unavailable — OpenRouter returned an error. The engine correctly failed closed (escalated as error, not approved).

Lesson: Transient LLM provider errors are not engine defects; the engine fails closed.

7. The operational benchmark establishes baselines for the future

Latency (p50 approved=13.86s), reviewer burden (2.58 blockers/goal), operator workload (58 decisions/100 goals). Median revisions to resolution = 1.0.

Lesson: Operational baselines enable before/after comparison — the downstream-error-rate metric requires partner runner integration (deferred to v0.3.0).

8. Cost is not a barrier

$0.49 for 170 goals + 60 boundary audits. Cheaper than a single developer-hour. There is no excuse not to field test your agent system.

Lesson: The field test should run on every release. The cost is negligible.

9. Two tiers, not one — and now three

The deterministic gates run on every commit for free. The 1295 deterministic tests run in 4.7 seconds. The LLM field test runs on release for $0.49. Three tiers, each catching different things.

Lesson: Unit tests can't replace the deterministic gate suite. The deterministic suite can't replace the field test. The field test can't replace code review. All three are needed.

What's Next

v0.3.0 will:

  • Wire approving_authority through CLI/HTTP/MCP (F-14)
  • Add downstream-error-rate measurement via partner runner integration
  • Run a multi-model comparison (gpt-4o, claude-3.5, deepseek-v4) against the same 170-goal corpus
  • Add adaptive revision cap (detect strict goals, reduce cap to 1)
  • Explore critic satisfaction signals (allow strict goals to approve when the critic explicitly says the plan is good)

Article 4 of 5 in the PlannerCritic series.

Series: Article 1: "I Ran 157 Agent Plans Against a Real LLM" · Article 2: "I Told My LLM Critic to Be Adversarial" · Article 3: "The Planner Made the Same 3 Mistakes" · Article 5: "I Tried to Prompt-Inject My Own Engine"

Links:

Top comments (0)