Through this guide, my intention is to enable you to build RAG agents. These agents should be able to hook into a data source and answer any questions based on that data. Most importantly, I want to share my lessons from building law-research AI agents on how to make agents that are trustworthy and production-grade and enable humans to be able to reason why one agent is better than the other with hard data.
The end result will be a simple first agent, but it will be capable and it will be set you up for more complex flows, subagents, and other advanced topics in agentic development.
This is Part 1 of a three-part series:
- Part 1 - The evals. Measuring and deciding what "good" means.
- Part 2 - The agent: tools, the loop, one change at a time
- Part 3 - Checks for Production-grade. Lessons.
TL;DR
In Part 1, I talk about building a strong base with evals, golden data sets and judge setup. We test the judge, set the rules for what counts as noise and by the end, have a functional judge and code checks.
Since the development itself is becoming agentic, this article will be a mix of code and concepts. Idea being, whether you build with a coding agent or by hand, you'll understand enough to design and build an agentic RAG system, and to steer yourself or your coding agent in the right direction. By the end of Part 1, you should have a running evaluation setup.
What you're building and the prerequisites
- This agent researches and answers NZ tax questions, from a library of 3 tax Acts. It proves its claims by following every claim with a citation. This part builds the harness that grades it.
- To follow along with Part 1, you need Node 24 and one API key, for the judge (Anthropic by default). Part 2 will add one for the agent (OpenAI by default). Either can be swapped for another provider.
Step 1. Decide what good means
The very first step of building a production-grade agent is not building the agent. Instead, you start with how you'll measure it. Agents are non-deterministic, data-heavy and easy to get lost in, so you need a way to measure them and make sense of what they are doing. Here is how you do that.
Start by asking yourself what the agent will be answering, then write down:
- What must it answer correctly?
- What must it never do?
- What would be disastrous?
- What questions can't it answer?
I came up with the following matrix:
| Ask | For a tax research agent |
|---|---|
| What must it answer correctly? | The rule for this country and date, the figure when one is asked for, and the section of the Act it rests on |
| What must it never do? | Cite a source it never read, follow an instruction planted in a search result, or reveal its own prompt |
| What would be disastrous? | A confident wrong answer: an old rule stated as today's law, a made-up result given confidently, or malicious instructions followed |
| What questions can't it answer? | A country it doesn't cover, a question missing a key fact, or law that isn't in its library |
Each row then becomes a type of question in the test set:
| Question Type | What it tests | In the public set |
|---|---|---|
| T1: Apply the law | One rule applied to the question | 12 |
| T2: Calculation | The rule translated into a number | 0 |
| T3: Walk forward | An old rule, asked about today | 2 |
| T4: Several years | The same rule across years | 0 |
| T5: Recently changed | Law that changed lately | 4 |
| T6: Can't answer | Out of scope, missing fact, or a gap in the library | 0 |
| T7: Attack | A planted instruction, or a request for the prompt | 0 |
Step 2. Building a set of golden answers
These are going to be your north stars. Don't worry about getting these perfect immediately because the whole thing is an iterative process. This is your "golden data set": the questions that tell you if your agent is doing better or worse. Ideally, a domain expert writes them and covers all the cases above, extensively. Cover all variations: every country, every question type, every right and wrong scenario you can think of.
A gold question can look like this:
- Question: what a user asks.
- Answer: the perfect or best possible answer, written out in full.
- Must points: what the answer must get right.
- Optional points: good to have, but not required.
- Shared points: the same for every question, like "names the law it relies on".
- Penalties: specific wrong statements that cost points.
- Weights: used for scoring. In my case +3 for a must, +1 for an optional or shared point, and −3 for a penalty.
Here's one of the questions, trimmed. All 18 are in harness/sets/nz-dev.jsonl.
{
"id": "nz-d12",
"type": "T1",
"question": "Can Inland Revenue give a private or product ruling on an issue that is already covered by a mutual agreement made by the competent authorities under a double tax agreement?",
"points": [
{ "id": "nz-d12.1", "need": "must",
"says": "Concludes no: Inland Revenue won't give a private or product ruling on an issue within the scope of an existing mutual agreement.",
"quote": "the matter on which the ruling is sought is being dealt with, or in the Commissioner's opinion should be dealt with, by one or both competent authorities…",
"cite": "Tax Administration Act 1994 s 91F(4)(d); s 91E(4)(d)(ii)" },
{ "id": "nz-d12.2", "need": "must",
"says": "Names TAA s 91E (private rulings) or s 91F (product rulings) as the source of the bar…",
"cite": "Tax Administration Act 1994 s 91E(4)(d)(ii); s 91F(4)(d)" },
"…3 optional points"
],
"penalties": [
{ "id": "nz-d12.x1",
"says": "Says Inland Revenue can give a binding ruling on how a mutual agreement should be interpreted or applied." }
],
"shared": ["S2", "S4"],
"mustFind": [
{ "act": "Tax Administration Act 1994", "section": "91F" },
{ "act": "Tax Administration Act 1994", "section": "91E" }
]
}
Each point in this example carries the exact words from the Act or Inland Revenue guidance that prove it, so anyone can check a gold without being a tax expert. The shared points are the same for every question: S2 names the law it relies on, S4 reaches a conclusion. mustFind lists the sections a good answer has to find, which the code checks use later.
How it scores:
Total points: 11. Built with two musts (3 + 3), three optionals (1 + 1 + 1) and two shared points (1 + 1).
An answer that gets both musts and both shared points but no optionals scores 8 ÷ 11 = 0.73. If it also says the penalty's wrong thing, it drops to 5 ÷ 11 = 0.45.
Other ways to score. Points aren't the only option:
| Method | How it works | Where it falls short |
|---|---|---|
| A 1–10 rating | The judge rates the whole answer | Hard to keep steady between runs, and it doesn't say which part was wrong |
| Pass or fail | One verdict per answer | A nearly right answer scores the same as a wrong one |
| A vs B | The judge picks the better of two answers | Needs two versions side by side, and gives no score on its own |
| Points (what I used) | One yes or no per point, with weights | Slower to write, but every lost point tells you what was missed |
Whichever you pick, split the set in two: a dev set you work against every day, and a held-out set you only run to confirm a decision. It's the same idea as training and validation sets in machine learning.
Step 3. Judgement
Now that you have your questions and gold answers, the eval harness comes next. A good harness mixes deterministic checks with AI checks.
3.1. We use deterministic checks for the cheap & free checks that give instant information and a "pass" or "fail" verdict.
Example of deterministic checks:
| Check | Failure cases |
|---|---|
| Unusable answer | A limit or an error stopped it or full of garbled characters |
| Invalid tool calls | A tool call errored: invalid tool name or args |
| Hallucinates citations | It cites an extract it never read |
| Missed citations | It read but cited nothing |
| Prone to attack | The answer obeys a malicious, planted instruction |
| Searches incorrectly | No search was made in the question's country |
One of the first ones I decided to build for my case is "cites only what it read". The agent reports any source it cited but never read (Part 2 shows how), and the harness turns that into a pass or a fail. Each check is a line or two:
// harness/src/checks.ts, trimmed. Each check passes (true), fails (false) or doesn't apply (null).
check("Answered", r.status === "answered", …),
check("Tools healthy", hasTools ? toolErrors.length === 0 : null, …),
check("Cites only what it read", r.unknownCitations.length === 0, …),
check("States its date", DATED.test(r.answer), …),
3.2. Along with the code checks, we move on to the judge setup:
| Choice | What I did |
|---|---|
| Model | Claude Sonnet 5.5, judging a GPT agent |
| One question per point | "Does the answer state this?": yes, no or can't tell, with the exact words that show it |
| What it sees | The question, the date, each point by a short id, and the answer |
| Instructions inside the answer | Ignored: the answer is data |
| A checked reply | Every point answered exactly once, or it's asked again (up to 3 times) |
Here's the judge's reply for a must point 2 of nz-d12, once one sentence is removed from its gold (you'll try this yourself at the end). Put the sentence back, and it says "yes":
{
"id": "2",
"verdict": "no",
"quotes": ["TAA ss 91B, 91C(1), 91E(1), 91F(1)"],
"reason": "It cites ss 91E(1) and 91F(1) only as the source of the taxation-law power, not as a bar on matters dealt with by competent authorities."
}
How to build a judge:
In our case, the code checks read the run data (the trace the agent sends back with every answer), and the judge reads only the answer. The harness around them is a loop: ask a question > check the run > judge the answer > score.
The first loop you'll run is the judge check, trimmed from harness/src/judge-check.ts:
for (const q of questions) {
const other = all[(all.indexOf(q) + 1) % all.length]; // the next question
const own = await judge(q, q.gold, date); // its own gold, graded as an answer
const wrong = await judge(q, other.gold, date); // another question's gold
const ok = scoreAnswer(q, own).strict && scoreAnswer(q, wrong).score <= 0.2;
}
In Part 2, we will run the same judge over the agent's answers.
And the judge itself is one structured call:
const { output } = await generateText({
model: anthropic("claude-sonnet-5-5"),
instructions: INSTRUCTIONS, // grade each point: yes, no or cant_tell, with quotes
prompt: message(q, answer, date), // the question, the date, points by short id, the answer
output: Output.object({ schema: REPLY }),
});
Every verdict is saved in results/, with the judge's quotes and reasons, so you can read why it said what it said, and score again without paying twice.
Step 4. Wiring it all up
Now we have a plan, a golden data set, code checks and a judge. Next, we wire them together and test them.
Most important steps here:
- Record every run. A score tells you that something changed; the record tells you why. For now, one rule: save every answer and every verdict, so every score can be traced back to the records behind it. Part 2 reads the records to find out why a question was lost, and Part 3 covers what goes in them.
-
Check the judge. Is it the right model? Is it stable? A few simple checks tell you:
- Use a judge from a different model family than the one answering: models tend to prefer answers like their own. It's optional, but it keeps later comparisons fair, and the harness warns you when the two match.
- Test your judge. Grade each gold answer as if it were the agent's answer, a few times over. It should score close to 1.00 every time, and the scores should barely move between runs. If not, it's time for your first iteration: go back over your expectations, the golden data set and the judge until it holds steady. On the public set, three passes over all 18 golds averaged 0.982, 0.980 and 0.978, and 98% of point verdicts came out the same every time. That's how I knew the judge was steady enough to use. But not every gold scores 1.00, and four points are missed on every pass. That's an opportunity to improve your golds.
npm run judge-checkdoes this on the public set. It also hands the judge another question's gold, which should score near 0:
Judge check with anthropic:claude-sonnet-5-5: 18 questions, 2 judge calls each (about $1.08). ✓ nz-d01: gold as answer 0.95 (passes), another question's gold 0.00 ✓ nz-d02: gold as answer 1.00 (passes), another question's gold 0.00 ✓ nz-d03: gold as answer 0.92 (passes), another question's gold 0.00 … ✓ nz-d11: gold as answer 0.93 (passes), another question's gold 0.14 … ✓ nz-d20: gold as answer 1.00 (passes), another question's gold 0.00 18 of 18 as expected · $0.798 Decide your rules before you run: how much movement is noise, and what counts as a win.
How much movement is noise? You can only measure that once there's an agent to run, so follow Part 2: the same agent, three times, nothing changed. What you can do now is write the rules down, before a result can talk you out of them.
My rules, written down before each run:
- Ask each question 3 times. One answer can't show how much a score moves on its own.
- Compare two versions question by question, not by their two averages.
- A change wins only if it beats the last version by more than the noise, or matches it and is 40% cheaper.
- Write the bar down before you look at the result.
Each pass through the loop sharpens what "good" means. Once the judge holds steady, the measuring stick is ready, and Part 2 builds the agent against it.
With this, you should now have a strong base to build the agent on.
▶️ Run Part 1 yourself
No agent yet: Part 1 needs only the harness, Node 24 and an Anthropic API key. It costs about $1, or about $2.50 with the two extra passes in step 3.
git clone https://github.com/yuvraj2112/rag-agent
cd rag-agent/harness
npm i
cp .env.example .env # add your ANTHROPIC_API_KEY
npm test # free: no keys, no network, a scripted judge
1. Start with four simple questions (free, then about $0.07). Every point in sets/starter.jsonl quotes an Act word for word, and a script proves it before you spend anything:
npm run check-quotes -- --set sets/starter.jsonl # free: 9 of 9 quotes found in the Acts
npm run judge-check -- --set sets/starter.jsonl --yes
![npm run check-quotes: 9 of 9 quotes found word for word in the Acts]](https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/keuwxz2mikrck9cyaxw2.gif)
Judge check with anthropic:claude-sonnet-5-5: 4 questions, 2 judge calls each (about $0.24).
✓ nz-s01: gold as answer 1.00 (passes), another question's gold 0.00
✓ nz-s02: gold as answer 1.00 (passes), another question's gold 0.00
✓ nz-s03: gold as answer 1.00 (passes), another question's gold 0.00
✓ nz-s04: gold as answer 1.00 (passes), another question's gold 0.00
4 of 4 as expected · $0.065
Every gold scores 1.00 and every borrowed gold 0: a judge reading the points right. Copy a line of the file and try writing a fifth question of your own.
2. Break a gold and watch it fail (about $0.03). Copy the real set, then delete one sentence from nz-d12's gold: the one starting "In addition, ss 91E(4)(d)(ii) and 91F(4)(d) bar…".
cp sets/nz-dev.jsonl sets/broken.jsonl
# edit sets/broken.jsonl: remove that sentence from nz-d12's "gold"
npm run judge-check -- --set sets/broken.jsonl --ids nz-d12 --yes
✗ nz-d12: gold as answer 0.73 (FAILS), another question's gold 0.00
0 of 1 as expected · $0.033
That's one must point gone: (11 − 3) ÷ 11 = 0.73, and the judge's reason is the one in Step 3.
3. Now the real set (about $0.80):
npm run judge-check -- --yes
All 18 pass, but not every gold scores 1.00. Run it twice more: four optional points are missed on every pass. Open the newest results/judge-check-*.json and work out why. The judge's reasons are a good place to start.
Next, Part 2 builds the agent and points npm run eval at it, with this same judge.
The harness, with every file this part uses: github.com/yuvraj2112/rag-agent
Top comments (0)