I Built Scenario Packs for Agent Regression Testing. The Integration, Not the Judge, Broke Me.

I thought the hard part would be the scoring.

Write clean YAML. Define expected behavior. Run agents. Compare scores. Catch regressions. Ship with confidence.

That mental model lasted about one afternoon of real-agent field testing.

The thing that broke was not the judge. It was not the rubric design. It was the realization that a scenario pack is only as honest as the path between your harness and a real, messy, third-party agent that imports ffmpeg at module scope, hardcodes gpt-3.5-turbo, and writes to /root the moment you touch it.

This is the second article in a series about EvalForge, an OSS evaluation harness for tool-using AI agents. Article 1 made the case that agent evaluation is a different problem from model evaluation because the path matters, not just the answer. The launch article is the longer story of what real agents taught me once the code was public. This one is narrower: scenario packs, baselines, and scoring. The three concrete things I built, and the one that broke first.

The Scenario Pack Is a Contract, Not a Test File

Before I get to what broke, I need to show what I actually built, because the design decisions in the pack format are where the engineering lives.

This is a scenario from the launch pack. One of twenty. It looks clean. It is clean:

# scenarios/core-launch.yaml — launch-01-account-policy
# https://github.com/deghosal-2026/agent-eval-forge/blob/main/scenarios/core-launch.yaml
- id: "launch-01-account-policy"
  title: "Account policy lookup"
  goal: "Retrieve a specific policy detail using one tool call"
  input: "What is the return policy for premium customers?"
  allowed_tools:
    - name: "policy_lookup"
  disallowed_tools: []
  expected:
    type: exact
    value: "Premium customers receive a 60-day return window with free return shipping."
  metrics:
    task_completion: {threshold: 1.0}
    output_correctness: {threshold: 0.8}
    tool_correctness: {threshold: 1.0}
    step_efficiency: {threshold: 0.7}
  tags: [retrieval, single-tool]
  difficulty: easy
  budget: {max_steps: 3, max_tokens: 300}

Twenty scenarios shipped in v0.1 across ten families: single-tool retrieval, multi-tool synthesis, structured extraction, tool argument precision, refusal, ambiguity clarification, budget constraints, failure recovery, coding-agent regression, and classification. Eight more for security: prompt injection, exfiltration, SSRF, sandbox escape.

The architecture is straightforward: CLI runs the pack through a core runner. Runner delegates to an adapter. Adapter talks to the agent. Scorer evaluates the trajectory. Judge fills in semantic gaps. Diff engine compares the result against a saved baseline.

The ground-truth boundary

The most important design decision in the pack format is not visible in the YAML. It is what the agent never sees.

The expected and metrics fields are evaluation-only. They are stripped before the agent receives anything. The build_invocation_payload function in the adapter base is the enforcement point:

# src/evalforge/adapters/base.py — build_invocation_payload
def build_invocation_payload(scenario: Scenario, run_id: str) -> dict[str, Any]:
    return {
        "schema_version": "evalforge.invocation_payload.v1",
        "run_id": run_id,
        "scenario_id": scenario.id,
        "input": scenario.input,
        "context": scenario.context,
        "allowed_tools": [tool.model_dump() for tool in scenario.allowed_tools],
        "disallowed_tools": [tool.model_dump() for tool in scenario.disallowed_tools],
        "budget": scenario.budget.model_dump() if scenario.budget else {},
    }

Notice what is not in that dict. No expected. No metrics. No threshold. No goal. The agent gets the input, the tool surface, and a budget. It does not get the answer key. It cannot game what it cannot see.

This is not a convenience — it is a correctness boundary. If ground truth leaks into the agent's context, every score is suspect. The Scenario model documents this in its docstring: "expected/metrics are evaluation-only and never sent to agents." The Baseline model and the ComparisonEngine both depend on that boundary holding. If it breaks, the regression story breaks with it.

There is a second boundary in the same file, and it is the kind of thing nobody talks about until it bites them. The _sanitize_agent function strips API keys and tokens from adapter config before writing them into run artifacts. Pass api_key in your adapter config — it never reaches the artifact store. Secrets do not persist. I would call this a feature, except that calling it a feature implies it is optional. It is not.

A third one: when the adapter parses agent output, a "completed" run that produced no output at all is treated as an error, not a pass. The comment in _artifact_from_envelope is blunt: "blank completions usually signal a dead entry point or empty tool result, and must never count as passes." A blank completion is a failure wearing a pass costume. The harness refuses to count it.

These three boundaries — ground-truth stripping, secret sanitization, blank-completion rejection — are the ones I would fight to keep if I had to rebuild from scratch. Everything else is negotiable. These are not.

If you are building an eval harness, I want to know: where is your ground-truth boundary? Is it enforced at a single function, or is it a convention that depends on every adapter remembering to do the right thing?

The Adapter Problem Started Before Scoring Even Ran

Then I sourced 19 OSS agents from GitHub — 11 LangGraph, 8 PydanticAI — using a star-bucket strategy. High-star repos for maturity signals, medium for real-world mess, low to see if the tool adds any signal in chaotic codebases. The sourcing methodology is documented in docs/hard-won-lessons.md.

Nine passes. Out of 95 scenario-agent combinations.

Not nine-per-agent. Nine total.

The instinct when you see nine passes is to blame the judge. Switch from gpt-4o-mini to gpt-4o. Tune the rubrics. Add more scoring dimensions.

I ran the same passes on two judge tiers — gpt-4o-mini (cheap) and gpt-4o (better). Same outcome both times. Nine passes. The better judge did not surface a single regression or improvement the cheaper one missed. The bottleneck was not the scoring layer at all.

The bottleneck was whether the harness could run the agent in the first place.

Five ways real agents broke the adapter

I documented these in the hard-won lessons file, but these are the patterns that actually hit:

Absolute writes at import time. Several agents wrote to /root/something inside their __init__.py. The harness runs in a locked-down sandbox. Import failed before any evaluation code executed. The fix was not elegant: redirect HOME, TMPDIR, and XDG_CACHE_HOME to per-agent .cache directories. Agents that still wrote to absolute paths got quarantined.

Gateway-bound imports. Multiple agents did ChatOpenAI(api_key=os.getenv("OPENAI_API_KEY")) at module scope. If the key is missing, the module itself raises. You cannot import it. You cannot evaluate it. The workaround was dummy env vars for the local tier. Agents that required real gateway connectivity got quarantined for local runs.

Hardcoded model names. ChatOpenAI(model="gpt-3.5-turbo") at module scope. I pointed OPENAI_BASE_URL at a local OMLX server running Qwen3.5-9B-MLX-4bit. The agent still asked for gpt-3.5-turbo. OMLX does not serve that model. 404. The fix was monkeypatching ChatOpenAI.__init__ before the agent module is imported — and I learned the hard way that Pydantic v2 field-default patching does not work for this. It has to be __init__. It has to run before import.

Typed StateGraph with no chat surface. Some LangGraph agents use typed StateGraph with internal domain state fields. The harness sends chat messages. The agent expects AgentState with typed keys. There is no bridge. I had to write thin evalforge_wrapper.py modules per agent to translate. This is not a harness bug. It is a design gap: the harness assumes a message surface, and typed-graph agents do not expose one.

Database bootstrap at import. create_async_engine(DATABASE_URL) and FAISS.load_local(...) inside module scope. The harness should not be patching around an agent's entire infrastructure bootstrap. I learned to classify agents by import-time side effects — no infra, needs DB/keys/files, needs running server — and skip the ones I could not run locally. Move on. Do not fight databases.

The lesson I walked away with: a scenario pack tests your adapter before it tests your agent. If the harness cannot faithfully run a random third-party agent, the signal you are measuring is integration friction, not agent quality. Friction is real and worth measuring. It is just not the same thing, and calling it the same thing is how teams ship agents they do not actually understand.

What I would build differently

The harness currently classifies agents into three tiers — local, Docker, quarantined — and moves on. That triage works for a first pass but paper-bags a real architectural choice.

Right now the default adapter imports agent code directly into the harness process. The python_import adapter shoulders the import, and the isolated adapter wraps it in a subprocess for some safety. But the boundary is still "shared Python process" at heart.

A cleaner design would be: the harness never imports agent code. It always communicates through a strict stdin/stdout contract. The subprocess adapter already exists and already works this way. Every agent gets a well-defined protocol: invoke(input, tools, budget) → trajectory. The harness does not care what language the agent is written in, what it imports, or what it writes to disk.

The import-based adapters were faster to wire for the first 19 agents. I would build the subprocess boundary as the one true path from the start, and treat import-based adapters as an opt-in optimization for agents you already trust in-process.

This is the one I keep turning over: should an eval harness ever share a process with the thing it is evaluating? Or is process isolation the minimum bar for honest measurement? I lean toward isolation, but I want to hear from anyone who has made the tradeoff the other way.

The Baseline Problem (This Is Where Regression Actually Lives)

Once the adapter runs, you have trajectory artifacts. Now you need to compare versions.

The default approach in most eval setups I have seen is implicit. Run the new version. It produces scores. Eyeball the numbers. Decide. There is no explicit baseline. There is no structured diff. There is just the latest JSON file and your gut.

Why does that break?

Say your agent scores 0.92 across 20 scenarios. Solid. Ship. Next week you change the prompt. Average drops to 0.89. Still decent. Ship again. Two more prompt changes later, average is 0.84. Each individual drop was small. No single change triggered alarm. But the cumulative drift from 0.92 to 0.84 is real, and "last run wins" never catches it because the reference point keeps resetting.

EvalForge takes the opposite approach: explicit golden baselines. You save a baseline explicitly and judge everything against it until you intentionally promote a new one:

evalforge run --pack core-launch.yaml --agent python:my_agent.py
evalforge baseline save --name v1.3.0 --run .evalforge/runs/latest

The Baseline model captures more than just scores. It snapshots the full artifact set, the frozen score state, the git SHA, agent metadata, and trust level. When you compare, you are comparing against a known-good reference that is traceable to exact source:

@dataclass
class Baseline:
    name: str                # "v1.3.0"
    pack: str                # "core-launch-pack"
    pack_version: str        # "1.2.0"
    runs: list[RunArtifact]  # One artifact per scenario
    score_snapshot: dict     # Frozen scores for fast CI comparison
    agent: dict              # Framework, version, model
    git_sha: str | None      # Traceable to exact source
    created: str             # ISO-8601

Three-level comparison

The ComparisonEngine compares at three levels.

Per-scenario. A regression is defined narrowly: baseline was "passed" and candidate is not "passed". If a scenario was already failing, the new version cannot "regress" on it. That is a deliberate product choice. The engine's job is release gating. It asks one question: did this change make something currently working stop working?

# src/evalforge/comparison/engine.py — the regression classification
"regressed": (
    base_ss is not None and cand_ss is not None
    and base_ss.status == "passed" and cand_ss.status != "passed"
),
"improved": (
    base_ss is not None and cand_ss is not None
    and base_ss.status != "passed" and cand_ss.status == "passed"
),
"new_failure": (
    base_ss is None and cand_ss is not None and cand_ss.status != "passed"
),

Per-family. Scenarios are tagged: retrieval, safety, multi-tool, synthesis. The engine groups deltas by tag. A +0.03 overall delta is meaningless if the safety family dropped 0.15 while retrieval gained 0.18. The aggregate number is for dashboards. The family breakdown is for decisions.

Per-pack. Total counts: regressed, improved, unchanged, new failures, new passes. Plus cost delta in USD between baseline runs and candidate runs.

Two comparison modes exist. Snapshot mode compares saved baseline scores against candidate scores — no new judge calls, fast, CI-friendly. Rescore mode re-runs the judge on both baseline and candidate artifacts. Use rescore when the judge model changed or you suspect stale baseline scores. Snapshot is the pragmatic default because it avoids token cost in CI.

There is a subtlety in the per-scenario definition that I want to flag for discussion: a scenario that was already failing cannot "regress." It can only stay broken or improve. That means a version change that makes a failing scenario fail differently — say, from a timeout to a hallucination — shows as "unchanged" in the comparison. Is that the right call? I think so for release gating, because the release question is "did something working break?" not "did something broken change shape?" But I can see an argument for tracking failure-mode shifts separately. If you have an opinion, I want to hear it.

Scoring: Deterministic First, Judge Only When Necessary

While the adapter was humbling me, the scoring design held up better than I expected. What went into it — and what it deliberately does not do.

Seventeen deterministic scorers ship in v0.1. They are free, reproducible, and run on every artifact. ToolCorrectnessScorer computes the fraction of tool calls that were to known tools. ZeroDisallowedActionsScorer checks that no disallowed tools were called — and returns blocking=True, forcing the scenario to failed regardless of answer quality. That is a safety decision, not a scoring convenience.

Two metrics use a hybrid gate+judge strategy: policy_adherence and retry_discipline. The deterministic gate runs first. If it passes, skip the expensive LLM call. If it fails, escalate to the judge. Judge results are cached keyed by scenario ID, judge model, and artifact hash so the same input does not get re-judged across runs.

The exit code hierarchy is encoded in ScoringEngine._resolve_exit_code:

# src/evalforge/scoring/engine.py — _resolve_exit_code
def _resolve_exit_code(self, scenario_scores, safety_violations):
    if safety_violations:
        return 4   # Safety — always blocking, highest priority
    if judge_errors:
        return 3   # Judge failure (API timeout, etc.)
    if any(ss.status != "passed" for ss in scenario_scores.values()):
        return 1   # Standard failure
    return 0        # Clean pass

Safety violations are exit code 4 and override everything. Judge errors are 3. Standard failures are 1. Clean pass is 0. If the agent called a disallowed tool, it does not matter that the answer was correct. The pipeline blocks. There is no threshold negotiation.

The scoring layer explicitly does not auto-optimize, does not run in production, and does not promise coverage. Twenty scenarios catch obvious regressions. They do not catch every failure mode. No offline eval does.

The scoring layer has one design tension I want to surface: the WARN band. A score above threshold is a PASS. Above threshold times 0.7 is a WARN. Below that is a FAIL. The WARN band exists to catch near-misses before they become regressions. But WARN does not block in CI by default — exit code 1 only fires on actual failures. So a scenario that drops from 0.95 to 0.72 (just above the 0.7 WARN cutoff) shows as "passed" in the comparison engine, because both baseline and candidate have status "passed." The regression is invisible to the release gate. Is that acceptable? I think WARN-level drift should be visible in the comparison report even if it does not block. Right now it is not. That is a gap.

What Would Make This Robust

EvalForge v0.1 works. It runs, it scores, it compares, it gates. But "works" and "robust" are different bars. What follows are the gaps I know about, roughly ranked by how much closing each one would change how much I trust the system.

Wire the failure taxonomy into the comparison report. The taxonomy in src/evalforge/analytics/ already classifies failures into buckets — safety_violation, hallucination, tool_error, budget_exceeded, agent_crash. It is just not connected to the comparison engine. So regressed: true is a signal without a diagnosis. The sentence I want the system to generate for me: "Scenario launch-06-disallowed-tool regressed because the new prompt caused the agent to call delete_customer — a disallowed tool it previously avoided." Not built yet. Should be the first thing I add.

Trajectory-step-level scoring. Scores report at the scenario level. You know a scenario regressed, but not which step in the agent's decision sequence caused it. Step-level scoring would make regression diagnosis faster. It would also make the failure taxonomy more precise — "step 3 called the wrong tool" is more actionable than "tool correctness dropped."

Make the subprocess adapter the default. The import-based adapters are convenient but they share a process with the agent. That means an agent that segfaults takes the harness with it. An agent that writes to sys.path corrupts the harness's import state. The subprocess adapter avoids all of this. It should be the default, not the opt-in.

Measured judge costs. The cost table in scoring/engine.py uses estimated pricing from provider pages. Provenance is tagged "estimated" in the source because token usage is not yet captured from judge SDKs. Moving to measured costs is a small change with high signal — it would make cost-delta reporting in the comparison engine trustworthy instead of approximate.

Pack version drift detection in CI. The Baseline model captures pack_version at save time. The BaselineStore.validate() method warns if pack versions diverge. But that check is not yet wired into the CI exit code path. If you change the pack and forget to re-baseline, the comparison silently runs against a stale pack version. That should be a hard failure, not a warning.

Rerun variance as a first-class metric. An agent that takes wildly different paths on identical inputs is harder to trust than one with stable routing. Right now the harness runs each scenario once. Running each scenario N times and reporting variance would surface fragility that single-run scoring misses. This is the metric I most want to add but have not yet.

The Thread

Adapter realism, golden baselines, multi-dimensional scoring. They read like three separate features. They are three layers of one problem: making agent evaluation honest enough to trust for a release decision.

The adapter is whether you are testing the real agent, not a sanitized import. The baseline is whether you are comparing against a known-good reference, not a moving target. The scoring layer is whether you are measuring tool discipline, safety, cost, and trajectory — not just the final answer. And the ground-truth boundary is whether the agent can game what it cannot see.

Lose any one of them and you have a dashboard. Keep all of them and you have a release gate.

EvalForge v0.1 has all of them working, but the adapter layer is still the weakest link — nine passes out of 95 says the integration gap is real. The scoring and baseline layers are ahead of the adapter in maturity. That is not the order I predicted going in. The clean version of the story had scoring as the hard problem. The build disagreed.

Where Do You Want This to Go Next?

I am still actively building this, and the decisions get harder as the system gets more real. Three directions, pick one:

• Adapter realism — should an eval harness ever share a process with the agent it evaluates? Or is subprocess isolation the minimum bar? What has your experience been with import-based vs subprocess-based adapters?
• Regression baselining — the per-scenario regression definition says "already-failing scenarios cannot regress." Is that the right call for release gating, or should failure-mode shifts be tracked separately? How do you handle WARN-band drift in your CI?
• Trajectory scoring — step-level scoring vs scenario-level scoring. Is the additional granularity worth the complexity? Where did you find the signal that scenario-level scoring missed?

I will write the next article about whichever of these generates the most discussion.