Part 3: Build the Eval Set Before the Agent Exists

Part 3 of a series building a support-ticket agent with no framework. Previous: Part 2 (tool contracts). Repo: github.com/akash-pal/agent-from-scratch

Here's the ordering that trips people up: build the eval set before the agent loop exists. Not after, not alongside — before. It feels backwards. You can't run an eval against an agent that doesn't exist yet.

That's exactly the point. If you write the eval set after the agent is working, you're unconsciously grading against whatever the agent already does. Cases you didn't think to write are cases your agent silently fails on forever. Writing 21 cases against a specification (the use case and tool contracts from Part 2) means you're measuring against a real target, not tuning your eval to match your own demo.

The eval set: eval/cases.json

21 cases, three buckets:

Bucket	Count	Covers
Easy	12	Shipping-status lookups, simple KB questions, a cancelled-order info request, one no-KB-match case that must escalate rather than fabricate
Hard	6	Refund eligibility inside/outside the 30-day window, multi-item orders where only one item is refunded, boundary cases just past the window
Edge	3	Legal-threat, fraud-flag, and duplicate-ticket patterns — must auto-escalate with zero tool/LLM calls

A sample case, checking both the outcome and the trajectory that produced it:

{
  "case_id": "hard_03",
  "bucket": "hard",
  "ticket": {
    "ticket_id": "hard_03",
    "subject": "Wrong size shoes, keep the socks",
    "body": "The running shoes from order ord_1004 are the wrong size. I want a refund for just the shoes, not the socks.",
    "customer_id": "cust_002",
    "order_id": "ord_1004"
  },
  "expected_trajectory": ["order_lookup", "refund_eligibility", "issue_refund"],
  "expected_outcome": "refund_proposed",
  "expected_max_steps": 4,
  "policy_checks": [
    "refund amount reflects only the shoe item (~$74), not the full order total",
    "issue_refund gated behind human approval"
  ]
}

Three things being checked per case, not just "did the answer look right":

• Outcome — did it land on resolved / refund_proposed / escalated correctly.
• Trajectory — did it call the right tools, in the right order. This is what catches "right answer, wrong path" failures — an agent that proposes a refund without ever checking eligibility first, for instance.
• Policy checks — case-specific assertions, like "never call issue_refund for an already-cancelled order."

Why trajectory matters as much as outcome

An agent can land on the correct final answer through a broken process. If hard_03's agent calls issue_refund before refund_eligibility, and just happens to guess an eligible amount, checking only the final outcome would call that a pass. Checking the trajectory catches it. This maps directly to something the underlying guide's own evaluation framework insists on: different build steps need different kinds of evaluation, and trajectory correctness is its own category, separate from "did the final answer look okay."

One design choice worth flagging honestly: the first version of this trajectory check used exact array equality (JSON.stringify(actual) === JSON.stringify(expected)), and it was wrong — real LLM runs are non-deterministic enough that this produced false failures on completely correct agent behavior. Part 6 covers exactly what broke and the fix (ordered-subsequence matching instead of exact equality), because it's a genuinely useful lesson on its own.

The edge bucket exists to test the guardrail, not the model

The three edge cases — legal threat, fraud flag, duplicate ticket — aren't testing whether the LLM is smart enough to recognize a threat. They're testing whether the auto-escalate pattern match (a plain regex check, covered in Part 5) correctly intercepts these tickets before any model call happens at all:

{
  "case_id": "edge_01",
  "ticket": { "subject": "Final notice", "body": "...lawyer involved...Better Business Bureau." },
  "expected_trajectory": [],
  "expected_outcome": "escalated",
  "expected_max_steps": 0,
  "policy_checks": ["auto-escalated before any LLM/tool call (legal_threat pattern)"]
}

expected_trajectory: [] and expected_max_steps: 0 are the actual assertion: this ticket should never reach the LLM at all. That's a guardrail test, not a model-capability test — a meaningfully different thing to verify, and one worth keeping separate from the easy/hard buckets conceptually even though they live in the same file.

What a real harness run looks like

[PASS] easy_03 (easy)
[FAIL] hard_04 (hard)
    - trajectory: expected [order_lookup, refund_eligibility, issue_refund] as a subsequence, got [order_lookup, refund_eligibility]
[PASS] edge_01 (edge)

=== Summary: 12/21 passed ===
easy: 6/12  |  hard: 3/6  |  edge: 3/3

That hard_04 failure is real — it's the same run that surfaced the most significant bug in the whole build (Part 6 covers it in full: the agent claimed a refund was proposed without ever calling the tool that proposes it). This is what Step 3 buys you: a fixed, pre-written target to run against, so failures like this show up as a clear diff instead of a vague "something feels off."

What's next

Part 4: The Raw ReAct Loop: ~100 Lines, No Framework → covers the actual agent loop — the raw ReAct implementation these 21 cases get run against, with no framework in sight.

Repo: github.com/akash-pal/agent-from-scratch