How to Start an AI Agent Evaluation Program (5-Task Scorecard)

Originally published at kunalganglani.com — read it there for inline code, hero image, and live links.

If you want an answer to how to start AI agent evaluation program inside your org, I’m going to save you six months of “platform evaluation” meetings. You don’t need a platform. You need a scorecard that’s embarrassing to ignore.

Here’s the prerequisite most teams blow past: you must be able to replay the same task inputs. If you can’t capture a “task” in a file (prompt + context + tool fixtures + expected end state), you are not doing evaluation. You’re doing vibes. And vibes die the second you change a model or a tool schema.

I learned this the hard way running this blog’s multi-agent publishing pipeline. With 261+ posts shipped, the only thing that keeps quality from drifting is a deterministic gate and a repeatable regression loop. Bigger models did not save me. Rituals and checks did.

This post is the minimum viable rollout. Pick 5 tasks, define pass/fail, run it weekly, and publish a scorecard that drives behavior.

What is AI agent evaluation?

AI agent evaluation is the practice of repeatedly testing an AI agent on a fixed set of real tasks to measure whether it completes them correctly, safely, and reliably as prompts, models, tools, and data change.

Unlike a chatbot eval, an agent eval can’t stop at “was the answer nice.” You have to judge the trajectory (tool choices, arguments, retries) and the end state (did the system actually change the right thing).

As the LangSmith team at LangChain puts it, the first step is to identify what matters across your system: LLM calls, retrieval steps, tool invocations, and formatting. They recommend starting with 5–10 manually curated examples of “good” behavior for each critical component.

Also, stop confusing “model evals” with “product evals.” The Evidently AI team draws the line clearly: model evals measure raw capability; product evals measure whether your system does the job. Your org needs the second one.

How to start an AI agent evaluation program (in 7 steps)

This is the operating system. If you do only this, you’ll be ahead of most teams shipping agents right now.

[YOUTUBE:trfUBIDeI1Y|LLM as a Judge: Scaling AI Evaluation Strategies]

1. Name the agent’s top 5 business tasks (the ones that would hurt if they silently failed).
2. Write pass/fail assertions for each task (end-state checks, tool-call checks, and safety checks).
3. Capture a tiny dataset: start with 25–50 test cases total (5 tasks × 5–10 cases).
4. Build a tiny eval harness that can replay tasks and record results.
5. Run it weekly in CI on a schedule. Treat it like a reliability check, not a research project.
6. Publish a scorecard in a place execs and engineers both see.
7. Tie score drops to behavior: gates, rollbacks, and “add 1 test from production incidents every week.”

No “eval platform initiative.” No six-month dashboard build. You can start on Monday.

What to evaluate (agents, not chatbots)

Most orgs evaluate the wrong thing because they copy RAG/chatbot metrics and call it a day.

For agents, I care about four layers, in this order.

1) End-state correctness (did the world change correctly?)

If the agent’s job is “open a ticket,” “refund an order,” or “merge a PR,” the output text is mostly a distraction. The eval should assert on the resulting state.

Concrete examples:

• Ticket agent: a ticket exists with the right project, priority, and assignee.
• Ops agent: a config change was applied to the right environment (not prod).
• Coding agent: tests pass and the diff touches only expected files.

If you don’t have an API to check state, start anyway and check an artifact. “A PR URL exists and the title matches pattern SEC-####.” It’s not perfect. It is still pass/fail, which is the whole point.

2) Tool-use accuracy (selection + arguments + failure handling)

Agents fail in ways chatbots don’t, and they fail with more confidence.

Common ones I see:

• right tool, wrong args
• wrong tool entirely
• tool succeeds but the agent misreads the output
• tool fails and the agent keeps going like nothing happened

This is why you need trajectory-level signals.

Ragas ships metrics for agent tool use like “Tool Call Accuracy” and “Tool Call F1” in its docs. That’s a decent signal of where the community is converging. If you want a starting point for rubric judging, look at Yang Liu and the G-Eval framing (more below). Then keep tool correctness as hard checks.

3) Retrieval quality (if you’re doing Retrieval-Augmented Generation)

Retrieval-Augmented Generation (RAG) is where agents go to hallucinate with confidence.

Evaluate retrieval as its own thing. Don’t bury it inside “final answer quality.”

• “Did we retrieve at least 1 relevant doc in top 5?”
• “Did we retrieve any restricted docs?” (security)

If you want a concrete metric set, Ragas has standard RAG metrics like context precision/recall and faithfulness.

4) Safety and security (prompt injection + unsafe actions)

Agents amplify risk because they act.

A chatbot leaking a secret is bad. An agent exfiltrating secrets via tools is existential.

At minimum, your scorecard needs a safety lane:

• indirect prompt injection cases
• data exfil attempts
• privilege escalation attempts (“ignore previous instructions and run admin action”)

If you’re already building out your security posture, connect this to my broader AI security controls and the practical AI security patterns in AI in production.

Internal links you’ll likely want as you expand:

• AI agents
• agentic AI
• agent orchestration
• agent framework
• RAG
• retrieval-augmented generation
• prompt injection
• AI security
• AI in production

Offline and online evaluations (and what you should start with)

You need both. If you’re starting from zero, start offline. It’s controllable, repeatable, and it catches the stupid regressions before customers do.

The LangSmith docs make the split explicit: offline evaluations are for pre-deployment benchmarking, regression testing, and unit-style checks. online evaluations are production monitoring and guardrails.

Offline evaluations (start here)

Offline is your regression suite.

Run it when:

• you change the prompt
• you change the model
• you change tools or tool schemas
• you change retrieval chunks / embeddings

This is release engineering.

A number to anchor the expectation. If you start with 5 tasks × 5–10 examples, you’re at 25–50 cases. That’s small enough to run in minutes. Big enough to catch drift.

Online evaluations (add after week 2)

Online is what tells you the real world is weird, and users are weirder.

Do two cheap things first:

• sample 10 real runs/week for human review
• log tool-call failures and “agent gave up” outcomes as counters

If you already have LLM observability metrics, plug them in. Just don’t let observability become an excuse to avoid regression testing.

Evaluation lifecycle and scenarios: the weekly operating cadence

This is the part everyone skips. Tooling is the easy part. Org behavior is the hard part.

I treat agent evals like an SRE reliability review.

Ownership model

• One owner (not a committee). Usually the tech lead for the agent or the platform lead.
• One backup.
• A shared Slack channel for scorecard output.

Weekly ritual (30 minutes)

Agenda that works:

1. Look at the scorecard trend (last 4 weeks).
2. Investigate any new failures. Pick the top 3.
3. Decide: fix prompt/tooling, revert model, or accept as known limitation.
4. Add 1 new test case sourced from production tickets or incidents.

If you don’t add 1 test/week, your suite rots. Same way unit tests rot when nobody owns them.

Preventing “dashboard theatre”

Rules I use:

• The scorecard must show raw counts (e.g., 43/50). Percentages alone are how teams lie to themselves.
• Every metric must map to a decision: gate, rollback, or a work item.
• If a metric doesn’t drive action for 4 weeks, delete it.

Minimum viable agent eval scorecard (copy/paste template)

Put this in a Google Sheet, Notion page, or a markdown file in your repo. Make it visible.

Task	Dataset size	Pass rate (weekly)	Critical fails	Owner	Gate?	Notes
Create support ticket w/ correct fields	10	90% (9/10)	0	@team-a	Yes	Fails on missing priority
Refund request with policy constraints	10	80% (8/10)	1	@team-a	Yes	Unsafe: refunds over $500
Deploy change to staging only	5	100% (5/5)	0	@team-b	Yes	Tool args validated
Knowledge lookup via RAG	15	87% (13/15)	0	@team-b	No	Retrieval precision drift
Handle prompt injection attempt	10	70% (7/10)	2	@sec	Yes	Indirect injection failures

Definitions that matter:

• Pass rate: percent of test cases where all assertions passed.
• Critical fails: failures that represent a safety/security violation or an irreversible wrong action.
• Gate?: if “Yes,” a drop below threshold blocks release or forces a rollback discussion.

Thresholds I’ve seen work:

• Gate tasks: 0 critical fails and >= 90% pass
• Non-gate tasks: >= 80% pass

If your agent is customer-facing or money-moving, set the bar higher. “We’ll monitor it” is not a plan.

Setup: a no-platform eval harness (repo + CI)

You have two viable paths.

Path A: start with OpenAI Evals (fastest)

OpenAI (openai/evals maintainers) describe Evals as a framework for evaluating LLMs and LLM systems plus an open-source benchmark registry.

Translation: you can drop it into a repo and run it in CI without inventing everything.

Use it for:

• deterministic-ish offline suites
• weekly scheduled runs
• storing cases in YAML/JSON

Path B: roll your own tiny harness (more control)

A minimal harness is:

• cases.jsonl (each line: one test case)
• a runner script
• a results JSON artifact

Either way, the output must be machine-readable because you will want to graph it later.

One practical note from building this site’s agent pipeline. Idempotency matters. If your eval runner can’t be re-run without manual cleanup, people will stop running it. The “works once” harness dies in week 3.

Writing evals: pass/fail assertions that avoid religious wars

Pass/fail is where debates go to die. So define it in layers.

Layer 1: hard checks (no LLM judge)

These are cheap and unambiguous:

• Tool call validity: the tool name exists; args match schema; types correct.
• Forbidden tool calls: “never call deleteCustomer in this task.”
• End-state checks: API says the ticket/refund/PR exists and matches expected fields.
• Formatting contracts: JSON parses; required keys present.

This is where most value is. You’ll catch a shocking amount of regressions with just schema validation.

Layer 2: rubric checks (LLM-as-a-judge)

Use a judge when correctness is semantic.

The trick is to constrain the judge:

• give it a rubric with 3–5 criteria
• force a structured output (pass: true/false, reason, criterion_failed)
• calibrate it on a small set of human-labeled cases

The paper that made LLM judging feel less hand-wavy for me is Yang Liu et al. on G-Eval. They report a Spearman correlation of 0.514 with human judgments on summarization, and they also call out bias toward LLM-generated text. That’s your warning label.

Layer 3: human spot checks (keep the judge honest)

Do 10 samples/week of human review. That’s one engineer for 15 minutes.

If the judge says “all good” but humans keep flagging issues, your rubric is broken.

Running evals: weekly regression tests in CI (plus a release gate)

Make the run boring:

• a scheduled weekly run (cron)
• a run on every agent change (prompt, tools, retrieval config)

The org rule I like:

• Prompt changes require the same discipline as code. PR, review, and tests.

If you already think in CI/CD terms, good. Connect this to how you run reliability checks for everything else. The agent is software.

You’ll also want specialized suites for failure modes:

• tool-call chaos and failure handling (agent tool call failure testing)
• security regressions (prompt injection regression testing)
• incident-driven expansion (AI agent failure in production)

Best practices I actually care about (and the ones I don’t)

A few hard opinions.

Start with 5 tasks. Not 50.

LangSmith suggests 5–10 examples of good behavior per component. That’s the right scale to begin. If you try to boil the ocean, you’ll end up with a beautiful spreadsheet and zero runs.

Your dataset should be a living artifact

Every week, add 1 new case from:

• a production incident
• a customer escalation
• a weird internal ticket

After 12 weeks, you have 12 “scar tissue” cases. That suite will outperform any generic benchmark for your business.

Don’t optimize for a single number

If teams can game a metric, they will. Not because they’re evil. Because incentives work.

That’s why I like a scorecards with:

• pass rate
• critical fails
• dataset size

It’s harder to fake.

Version everything

If you can’t answer “what changed between last week and this week,” your scorecard is theatre.

Store:

• prompt version
• model version
• tool schema version
• retrieval index version

Treat it like release engineering. Because it is.

How many test cases do we need to start (and how do we maintain them)?

Start with 25–50 total cases.

• 5 tasks
• 5–10 cases per task

That’s aligned with LangSmith’s “5–10 curated examples” guidance, but organized around tasks instead of components.

Maintenance policy that doesn’t collapse:

• every week add 1 case
• every quarter delete 10% of cases that no longer represent reality
• track flaky cases and fix the harness, not the agent

If you do this for a year, you’ll end up with ~75–100 cases. That’s a serious regression suite without ever hiring a labeling team.

Publish the scorecard so it changes behavior

The point of the scorecard is not to feel safe. It’s to force tradeoffs into the open.

Publishing rules:

• post results in the same channel where releases are discussed
• tag owners on regressions
• keep a 4-week trend line

Gating rules that work:

• any critical fail on a gated task triggers an incident-style review
• pass rate below threshold blocks release unless a director explicitly signs off

If that sounds intense, good. Agents ship changes into the real world.

My prediction: in 12 months, “weekly agent eval scorecard” will be as normal as “weekly reliability metrics.” The teams that start now will ship faster because they’ll be the only ones who can change prompts, models, and tools without fear.

Your challenge for this week: pick 5 tasks, write pass/fail for one of them, and schedule the first weekly run. If you can’t do that, you’re not shipping agents. You’re shipping liabilities.

---

Originally published on kunalganglani.com