An HR agent that answers employee questions sounds like an easy win. Sixty people stop emailing HR about vacation days and benefits, and HR gets hours back every week.
The catch is that the agent is only useful if its answers are the handbook answers. One wrong vacation or benefits answer, delivered politely and confidently, creates a problem HR has to clean up. After two of those, people stop trusting the tool and go back to email.
Most teams test an agent by typing a few questions into the playground. That works once. After the next prompt change or data-source change, nobody does it again, because it's slow and boring. An Evaluation in Airia (affiliate link, I may earn a commission at no cost to you) fixes that: you save your test questions with the answers you expect, run them all in one click, and a judge model scores each answer.
Below is the exact setup from my lesson, on Airia's free plan, testing an HR Policy Agent for a made-up firm, Northwind Partners. The scores are from that real run.
Step 1: create the evaluation and pick the agent
- In Airia, click Build in the left bar, then Evaluations, then Create Evaluation.
- Type an Evaluation name (Handbook answers check) and a one-line Description of what you're checking.
- Open Agents. Keep Agent Source: Airia Agent, pick the Project your agent lives in, open the Agent list, pick the HR Policy Agent and click + Add. It shows under Added (1) with its version. (You can add up to 3 agents to compare them on the same test.)
Step 2: write the right answers
This is the part that matters most, and it isn't technical. Each test case is an Input (the question) and an Expected Output (the answer HR would give). Together they're the dataset.
Open Dataset and click Add test case for each one. You can also upload a CSV; Download CSV Template shows the format.
The two cases in the lesson:
| Input | Expected Output |
|---|---|
| How many vacation days do I get in my first year? | 15 days (handbook section TIME OFF) |
| Does Northwind pay for gym memberships? | "I can't find that in the handbook. Please email hr@northwind.example." |
The second one is the important one. The handbook doesn't cover gym memberships, so the right answer is a hand-off to a person. An agent that guesses about pay or benefits is worse than no agent. Every dataset should include at least one question the agent must refuse or pass on. Five to ten cases are enough to start.
Step 3: pick a small judge and run it
- Open Evaluators and tick Airia (Airia Default Evaluator).
- Under Judge model, pick a small model, such as GPT-5 Nano. The judge reads each answer, compares it with the expected one and scores it. A small model keeps the cost tiny: the whole two-question run cost less than a cent.
- Click Create and run evaluation.
It runs in the background and shows Running, 0 / 2 completed. Ours took about two minutes; refresh the page to see it switch to Success, 2 / 2 completed.
Step 4: read past the score
Open the evaluation. The summary shows Latency, Tokens, Cost, Accuracy 92% and Bias 0%.
92% looks like a pass. It isn't one yet. Click View Results and compare Expected Output with Actual Output row by row.
In our run, both answers were general advice, not the handbook. The vacation answer never said 15 days, and the judge still scored it 84%. A second run of the same evaluation came back at 75%. The score moves from run to run, and a judge model can rate a confident, well-written general answer highly even when it misses the one fact that matters.
That's exactly the failure you want to catch before rollout: the agent sounded right and was wrong.
What to do with a result like that
- Check the agent's data source. A general answer instead of "15 days" can mean the handbook isn't being read, so first check that the handbook data source is still connected to the agent.
- Fix it, then click Run Again on the same evaluation, and use Compare Results to see the runs side by side.
- Only share the agent when every row reads like HR wrote it, including the refusal.
The habit worth keeping
- The score is a hint, not a pass. Read every Actual Output.
- Always test the question it shouldn't answer.
- Re-run after every change to the prompt, the model or the data source. It's one click once the dataset exists, so there's no excuse to skip it.
The same approach works for any agent that answers on behalf of a team: finance policy, IT help, client FAQs. For HR, finance and legal answers, a wrong number delivered politely is the risk you're testing for.
Practice it step by step
The lesson, Airia: Test Your HR Agent Before Rollout with an Evaluation, walks through every screen above from the real Airia app, and doesn't touch any account. It's module 4 of my free Airia Jump Start course (my own course, not an official Airia one). If you don't have an agent to test yet, the module before it builds one: Airia: Answer Employee Policy Questions Instantly.
Practice it step by step, free: https://learn.ynteractive.com/content/airia-agent-evaluation
Top comments (0)