DEV Community: Sara Mo

What Are AI Evals, and Who Should Own Them?

Sara Mo — Tue, 11 Aug 2026 08:14:08 +0000

Your team shipped an AI feature. It went out on a Tuesday and it was good.

Six weeks later someone in sales says it has got worse. You ask engineering. Engineering says nothing changed on their side, and they are telling the truth. You ask for numbers and there are numbers, and none of them answer the question. You are now in a meeting about whether a thing is worse, with no way to settle it, and the meeting will happen again next month.

That meeting is what AI evals exist to prevent. And the reason most teams do not have them is not technical.

What an AI eval actually is

Strip the tooling away and an eval is three things.

A set of real examples of what people ask your product. A written description of what a good answer to each one looks like. And a repeatable way to check whether the answer you got matches the description.

That is it. Everything else is plumbing. There are good frameworks for the plumbing, deepeval among them, and picking one is genuinely the easy part of this.

The hard part is the middle item. Somebody has to write down what good means. And that is not an engineering document.

The four decisions inside an eval that are product decisions

What counts as correct. An agent answers a billing question accurately but coldly, to a customer who has already complained twice. Correct or not? There is no technical answer to that. There is a product answer, and if nobody gives it, the engineer writing the check will supply one by accident.

What a failure costs. A wrong tone and a wrong refund are not the same failure, and they should not sit behind the same passing bar. Somebody has to say what each kind of mistake costs the business. That person is not the person writing the test.

Which trade you are making. Every eval bar buys one thing and sells another. Push refusal rates down and you get more helpful answers and more wrong ones. Push them up and the opposite. This is a positioning decision wearing a percentage sign.

When it is good enough to ship. The one everybody agrees is a product call, right up until it arrives as a number in a dashboard, at which point it quietly becomes whoever built the dashboard.

Four decisions. All four get made whether or not anyone decides them. That is the actual risk: not that the team skips evals, but that the team builds evals and the product judgement inside them defaults to whoever had the file open.

So who owns them

Product owns the definition. Engineering owns the machinery.

In practice that means the PM writes what good looks like, in prose, before anything is built. Not a spec of the feature. A description of an answer, specific enough that two people reading it score the same output the same way. If two of your colleagues read your definition and disagree about whether an answer passes, the definition is not done, and no amount of engineering downstream will fix it.

Then engineering makes it run every night.

This split is unglamorous and it is the whole thing. The teams that hold quality over time are not the ones with the best framework. They are the ones where a named human owns the sentence that says what correct means, and that human sits on the product side.

The honest limit

Evals will not tell you your product is good. They tell you whether it changed, and which of two versions is better at the things you decided to measure. Everything you did not think to describe is invisible to them, permanently.

Which is an argument for writing the definitions badly and early rather than perfectly and never. The first twenty are worth more than the framework, and you can write them this week without asking engineering for anything.

The broader discipline that all of this sits inside is called harness engineering. But the part that decides whether it works for you is not in the tooling. It is one person, on the product side, willing to write down what good means and put their name on it.

How Do You Build an Evaluation Harness for AI Agents?

Sara Mo — Tue, 04 Aug 2026 11:19:24 +0000

You have an agent that works. Now someone asks how you know, and the honest answer is that you tried it about thirty times and it seemed fine.

That answer is fine for a prototype. It stops being fine the moment the thing is in front of people who did not build it.

An evaluation harness for AI agents is what replaces "it seemed fine". It is not a framework you install. It is four decisions, and you can make all four this week.

Decision one: where the cases come from

Not from your imagination. This is the one that quietly ruins most attempts.

When you invent test cases, you invent them out of the same mental model you used to write the prompt. You produce clear, well-formed, one-request-at-a-time inputs, because that is how you think about the problem. Real users send three questions in one message, leave out the order number, describe the thing without naming it, and switch language halfway through.

So take the last two hundred real interactions and read them. Pull twenty that made you uncomfortable. Not the twenty that failed, the twenty you were not sure about. Those are your first cases, and the set grows every week from the same source forever.

If you have no production traffic yet, the substitute is somebody else's messages. Support tickets, sales emails, forum questions. Anything written by a person who was not thinking about your agent when they wrote it.

Decision two: what a rubric has to survive

Write down what a good answer contains before you look at what the agent produced. This ordering is not a style preference. Read the output first and you will write a rubric that the output happens to pass.

Then run the only test that matters. Give the same output and the same rubric to a second person and ask them to score it. If you get a 4 and a 2, the rubric is not finished. Keep tightening it until two people who have not spoken land in the same place.

Most teams skip this and go straight to automating the scoring. You can automate a rubric two humans disagree about. You will get a number every night, and the number will mean nothing, and it will take you a quarter to notice.

Decision three: who judges

Three kinds, and the skill is knowing which question belongs to which.

Code judges anything checkable. Did it call the refund API. Is the total correct. Is the JSON valid. Did it stay inside the step budget. This is cheap, exact and boring, and it should cover more of your set than you expect.

Humans judge anything that is a judgement. Was that the right tone for a customer who is already angry. Was the escalation appropriate. You will not scale this, and you are not supposed to. It is the ground truth the other two are measured against.

A model judges only after it has agreed with your humans on cases you already scored by hand. Run it against fifty human-scored examples first. If it agrees, promote it. If it does not, you do not have an AI judge, you have a second opinion with no track record. Tools like openai/evals give you the plumbing for this; they do not give you the agreement, and the agreement is the part that matters.

Decision four: where the bar goes

Pick the passing bar from what a failure actually costs you, not from a number that sounds respectable.

Ninety-five percent is not a standard, it is a habit. If the failure mode is a slightly awkward sentence, eighty is generous. If the failure mode is refunding money that should not have been refunded, ninety-five is negligent and you need a human in the path regardless of the score.

Write the cost down next to the bar. Anyone who later argues the bar should move has to argue about the cost instead, which is a much better argument to have.

The honest limit

A harness does not tell you your agent is good. It tells you when your agent changed, and it tells you which of two versions is better on the things you decided to care about. Everything it cannot see is a case you did not put in the set.

That is still an enormous amount more than you had, and it is the difference between shipping and hoping. The wider discipline this belongs to is called harness engineering, and this is its first working piece.

Twenty cases. One rubric two people agree on. A code check where you can, a human where you cannot. One bar with a cost written beside it. That is an afternoon, and it is the afternoon that makes every week after it measurable.

Is Building in Public Backwards?

Sara Mo — Mon, 27 Jul 2026 02:17:42 +0000

I used to post the roadmap. The coming-soon screenshot, the "building something new" teaser, the thread narrating a feature I had not finished. It felt like momentum. Building in public, the advice everyone gives: share the journey, bring people along, market as you make.

I stopped. Not because visibility is bad. I publish more now than I ever did. I stopped because I had the sequence backwards, and the sequence is the whole thing.

Building in public, as most people practice it, means narrating intentions. You announce the thing before the thing exists. And announcing has two costs that nobody mentions in the "share your journey" advice.

What building in public actually rewards

The first cost is internal. Posting about the work lights up the same reward the work does. You write the announcement, the likes come in, and your brain files it under "made progress today." But you moved nothing. You can run this loop for weeks, feeling productive, shipping tweets instead of software. The narrating competes with the building for the same hours and the same dopamine, and narrating is easier, so narrating wins.

Attention is a loan; trust is a deposit

The second cost is external, and it is the one that actually matters. When you announce before you ship, you spend credibility on a promise. People give you attention for a thing that does not exist yet. That is a loan. You have to pay it back by shipping exactly what you described, on the timeline the excitement assumed. And unfinished work moves. The feature changes, the release slips, the idea turns out wrong. Hardware companies named the sharp version of this the Osborne effect: pre-announce the next thing and you can kill demand for the thing you already have. Now the version you shipped does not match the version people got excited about, and you have quietly spent trust to buy attention that already decayed.

Ship first and the transaction inverts. You build the thing, then you speak, and every word is backed by something a person can actually run. You are not asking for belief. You are reporting a fact. Attention spikes and fades no matter what you do, but trust earned against a real artifact compounds, because the next time you speak, people remember the last thing you said was true.

Build first, then speak

So the rule I follow now is boring and it works: let the work exist before the words about it. Not "never be visible." Visible constantly. But behind the work, not in front of it. Ship the small real thing, then write about the small real thing. The empty-database post went out after the tool was on PyPI and a reader could pip install it mid-sentence, not before.

This costs you the easy dopamine of the teaser and the pre-launch hype thread. What you get back is that you never owe anyone a thing you have not built, and every post you write is collateralized by something real. Over a year, the person who narrated intentions has a feed full of promises. The person who shipped first has a feed full of proof. One of those compounds.

Speak from behind the work, not in front of it.

Announce, and you borrow attention against work that does not exist. Ship, and you own it.

How Do You Contain an AI Agent Failure You Can't Prevent?

Sara Mo — Sun, 26 Jul 2026 03:16:32 +0000

Every part of this series has quietly agreed on one thing: the agent will be wrong sometimes. Part 1 set the bar at "acceptably wrong." Part 3 measured how often. So the last question is not how to stop it from ever failing. It is the one that actually decides whether you can ship: when it is wrong, what is the worst that can happen?

That worst case is not fixed. It is a design choice, and it is the one most teams never make on purpose.

Blast radius is something you choose

Two agents give the same wrong answer. One drafted an email for a human to send. The other sent it. One suggested a refund. The other issued it. Identical mistake, completely different consequence, because someone decided how much power the agent had when it was wrong. You set the blast radius by choosing what the agent is allowed to do, not by hoping it does the right thing.

Guardrails: match capability to proven trust

Give an agent the least authority the job allows. Let it read before it writes, propose before it executes. An action more dangerous than the agent's measured reliability has earned is a liability you chose. If Part 3 told you a step is right eighty percent of the time, that step does not get to move money unsupervised. Capability should track trust, and trust is a number you now have.

Put a human on the expensive failures, and only those

Human-in-the-loop is not "approve everything," which kills the speed that made an agent worth building. It is a gate on the small set of actions where a wrong one is irreversible or costly: the disqualifying failures you named in Part 1 (the known abuse modes are catalogued in the OWASP LLM Top 10). Everything reversible and cheap runs on its own. Everything that cannot be taken back waits for a person.

Make failures reversible and visible

Prefer actions you can undo, and log enough to undo them. A dry-run mode, a soft delete, a confirmation step: these turn an incident back into a mistake. And you cannot contain what you cannot see, so trace every action. A traced failure gets caught in minutes and becomes a new case in your eval set (Part 2). The loop closes: containment feeds the very thing that measures reliability.

That is the whole series. Define the bar, build the eval set that measures it, treat reliability as a distribution, budget the cost, and contain the failures you cannot prevent. None of it makes an agent perfect. All of it makes an agent you can actually ship.

You cannot build an agent that never fails. You can build one whose failures cannot hurt you.

What Does Your AI Agent Really Cost to Run?

Sara Mo — Fri, 24 Jul 2026 03:58:49 +0000

The agent works. It also takes eleven seconds to answer and costs more per task than the thing it replaced. In the demo nobody noticed. In the release review, finance runs the math to real volume and the project quietly dies. Accuracy got it into the room. Economics decided whether it shipped, and economics said no.

Part 1 called cost and latency a line on the bar. Part 4 is why that line is not a footnote: for an agent, the budget fights everything you did in Part 3 to make it reliable.

Cost is per successful task, not per call

A single model call is cheap. Priced per token, say three cents. But an agent does not make one call. It plans, calls a tool, reads the result, calls another, maybe retries. Eight calls a task is ordinary, so now you are at roughly twenty-four cents a task. At fifty thousand tasks a month that is twelve thousand dollars, for one feature.

Then it gets worse, because not every task succeeds. If the agent succeeds seventy percent of the time and you retry the failures, you pay for about 1.4 attempts per success, so your real number is closer to thirty-four cents per successful task. Cost per call flatters you. Cost per successful outcome is what you actually pay.

Latency compounds, and the tail is what people feel

Those eight calls are also eight round trips, plus tool time. The average task might land in eight seconds, but the distribution has a tail (Part 3), and users feel the tail, not the average. A p95 of twenty seconds means one in twenty people watches a spinner long enough to leave. Latency is not one number. It is a distribution with a cliff.

Reliability and economics pull against each other

Here is the tension nobody puts on the slide. Everything that made the agent more reliable in Part 3, running steps multiple times, adding retries, adding a verification pass, adds cost and latency. Everything that makes it cheaper and faster, fewer steps, a smaller model, no retry, tends to lower reliability. You do not get to maximize both. You design the trade on purpose, against your bar, or the trade designs you in production.

The levers are boring and they work: use a smaller model for the easy steps and the expensive one only where it earns its keep, cache what repeats, cap the number of steps so a task cannot loop forever, fail fast instead of retrying blindly, and set a hard ceiling on cost and latency per task so a runaway is caught by design.

Part 5 closes the series on the last thing between a working agent and a shipped one: what happens when it is wrong, and what stops the damage.

Budget is part of correctness: a right answer you cannot afford never ships.

How Do You Measure AI Agent Reliability?

Sara Mo — Fri, 24 Jul 2026 03:54:29 +0000

Your agent passed the eval, so you shipped. The next day a user sends almost the same input and it fails. Nothing changed. You just learned that "it passed" was one sample of a distribution, and you shipped on a coin flip that landed heads.

Part 1 defined the bar. Part 2 built the eval set that measures it. Part 3 is the uncomfortable part: an agent will not give the same answer twice, so a single green run tells you almost nothing about AI agent reliability.

One run measures luck, not reliability

A deterministic function passes a test or it does not, forever. An agent is a random variable. Run the same case ten times and you might see nine passes and one confident wrong answer. A single run showing the pass is not evidence the agent is good, it is evidence you got lucky once. Reliability is not a checkmark. It is a distribution, and you have to measure its shape.

Measure the distribution, not one draw

Run every case many times. Report a pass rate, not pass or fail. If a case passes 8 of 10 runs, that is an 80 percent case, and you should know before a user does. (This is the idea behind pass@k: score across repeated samples, not one.)
Care about the floor, not the average. A 95 percent average sounds fine until the 5 percent is the run that issues a refund or deletes a record. The average hides the worst case, and the worst case is what hurts you. Bound it explicitly.
Separate flakiness from failure. Some variance is harmless, like wording. Some is a real wrong answer. If you grade the property instead of the exact string (Part 2), harmless variance stops counting as failure and the failures that matter become visible.

Reduce variance on purpose

Once you can see the distribution, narrow it where it counts. Lower the temperature on calls that must be consistent. Constrain the output format so there are fewer ways to go wrong. Add a guardrail that catches the disqualifying failure before it reaches the user. None of this makes an agent deterministic, and chasing determinism is the wrong goal. The goal is a distribution tight enough, with a floor high enough, that you can trust it.

So the release gate is not "the eval passed." It is "the pass rate over many runs clears the bar, and the worst case is bounded." A green checkmark from a single run is not a decision, just an anecdote.

Part 4 turns to the other constraint that decides whether an agent ships: what all of this costs, in latency and dollars, at real volume.

Run it once and you measured luck. Reliability is the shape of the distribution, not a single pass.

Is Your AI Agent Eval Set Actually Testing Anything?

Sara Mo — Thu, 23 Jul 2026 01:32:04 +0000

Is your AI agent production-ready?

You shipped it with an eval set of five examples, all of them the demo you already knew worked. Two weeks later production is full of failures none of those five would catch. So you patch the prompt, the demo still passes, and you have no idea whether you fixed the class of problem or just that one screenshot.

That is the question this series opened with. In Part 1 we said production-ready is a bar you define before you build. The eval set is how you measure that bar, and it is worth more than the prompt it grades.

Why the eval set outlives everything else

Models change. You will swap the model, rewrite the system prompt, add a tool, switch providers to cut cost. Every one of those can silently break behavior that used to work. The only thing that tells you whether you regressed is the eval set. It survives every rewrite, because it encodes what "working" means independently of how you built it. A competitor can copy your prompt in an afternoon. They cannot copy two years of remembered failures.

Build it from real failures, not imagined ones

Most eval sets are weak because they are written from imagination at the start, when you know the least about how the agent fails. Invert it.

Every incident becomes a permanent case. When the agent fails in the wild, the fix is not done until that exact failure is in the eval set. A bug you fixed and did not test is a bug you will ship again.
Weight toward disqualifying failures. Part 1 asked you to name which failure modes are unacceptable. Your eval set should be heavy on those. That is where trust is won or lost.
Include the boring middle. If 80 percent of real traffic is mundane, an eval set that is all edge cases measures a job nobody does. Match the distribution of reality.

Grade behavior, not exact strings

An agent is non-deterministic, so a golden test demanding one exact output string will flake, and you will start ignoring it. Open eval frameworks let you assert on the property you care about: did it refuse the unsafe request, call the right tool, stay under budget, avoid inventing a policy. An eval you learn to ignore is worse than none, because it costs attention and returns false comfort.

If this sounds like curating good test data, that is exactly what it is. An eval set is test data for judgment, and the expensive part is the same as it is for any realistic test data: deciding what a good case looks like. Do that thinking once, capture it, and every future model swap pays you back.

Part 3 takes this into reliability: once you can measure the agent, how do you measure something that will not give the same answer twice.

Your prompt is disposable; your eval set is the asset.

Is Your AI Agent Production-Ready? Define the Bar First

Sara Mo — Wed, 22 Jul 2026 03:25:59 +0000

Every team shipping an agent has the same meeting. Someone asks "is it ready?" and the room splits. One person saw a great demo. Another watched it invent a refund policy an hour ago. The argument runs in circles because nobody agreed what "ready" means, so the loudest opinion wins and the agent ships on a vibe.

Making an AI agent production-ready is not a moment of confidence. It is a bar you write down before you build, then measure against. This post is about that bar: why agents need a different one than the services you already ship, and how to define it so "is it ready?" becomes a number instead of an argument.

Why "production-ready" breaks for agents

For a normal service, "production-ready" is settled. Correct output for valid input, handles errors, meets a latency target, has tests and a rollback. You know the shape of done. An agent breaks three of those assumptions at once:

It is non-deterministic. The same input can produce different output, so "correct" becomes "acceptably right, often enough."
Its failure surface is open-ended. A function fails in ways you enumerated; an agent fails in ways you never imagined, because it composes language, tools, and judgment on the fly.
Its worst case is not a 500 error. It is a confident wrong answer that looks right, which is far more expensive than a crash, because a crash at least tells you it failed.

So the honest question is not "is the agent correct." It is "is the agent acceptably wrong, safely, within budget, and repeatably enough to trust." That question has four parts, and each is a line on your bar.

The four lines of the bar

Write these down before you build. If you cannot fill them in, you do not have a spec, you have a wish.

Task success. On a fixed set of real tasks, not the happy-path demo, what fraction must the agent complete correctly? Pick the number. 85 percent means one in seven users gets a wrong answer. Acceptable for this job, or fireable? Decide on purpose.
Failure acceptability. Not all wrong is equal. An agent that says "I'm not sure, here's a human" failed gracefully. One that invents a policy and states it with confidence failed dangerously. Your bar names which failures are allowed and which are disqualifying.
Cost and latency ceiling. An agent that is right but takes forty seconds and three dollars a call is not ready for a chat box. The budget is part of correctness, not a footnote.
Containment. When it is wrong, and it will be, what stops the damage? Can it move money, delete data, or email a customer with no human in the loop? The bar defines the blast radius you accept.

Not one of these is "the demo worked." A demo is a single sample with a friendly operator. The bar is what you hold the agent to on inputs you did not hand-pick.

Define it first, on purpose

A bar written afterward is rigged. Once you have a working agent, every threshold quietly slides to whatever it already scores. Eighty-two percent becomes "eighty is fine." The scary failure mode becomes "edge case." You will rationalize the thing you built into the thing you meant to build. Set the numbers while you still have the nerve to fail them.

None of this tells you how to measure the bar yet. That is Part 2, where the eval set stops being an afterthought and becomes the product you are actually building. For now the win is smaller and larger at once: next time someone asks "is it ready?", you do not argue. You point at the bar.

A demo is an anecdote. A bar is a contract. You cannot ship what you refuse to define.

The empty-database problem: realistic test data in one command

Sara Mo — Tue, 21 Jul 2026 12:20:47 +0000

You finished the schema. The models are in, the first screen renders. Then you go to build the real thing and there is nothing there. The table is empty, and you cannot test a feature against zero rows.

So you improvise. A few hand-written fixtures. A prompt to a chatbot that returns five rows, two of them near duplicates, one quietly breaking your schema, and none of them the same when you ask again tomorrow. You came to build a feature, and instead you are maintaining fake data.

Here is the part most people miss: the expensive thing about test data is not the rows. It is deciding what a good row looks like. Do that thinking once, capture it, and you never pay for it again. That is the whole trick.

That is what a template is. You describe the data you want in a sentence, and you get a generator you can run forever:

pip install dugalaxy
dugalaxy gen customer-support

customer-support ships in the box. Describe your own in a sentence and you get your own template. Either way the output is varied, consistent, and reproducible: same seed, same data, every time. Tomorrow's run matches today's, which means your tests stop lying to you.

Three things that make it hold up:

It is an asset, not a batch. The template captures the judgment once and repays it on every run.

It is grounded, not improvised. Samples are generated against deterministic structure, so the fields line up and you can validate the shape. A chatbot re-rolls the dice on every call. A template does not.

It runs on your machine. No account, no upload, offline. Your data stays yours.

And the honest limit, because you should hear it from me first: it validates structure, not meaning. It checks that a sample fits the shape and rules you defined. It does not read prose and swear it is semantically perfect. A generated template is a strong starting point you still tune, not an oracle. The boring, repetitive part is handled, and your judgment goes where judgment actually matters.

A chatbot gives you five samples that drift. A template gives you data that survives.

If you have ever stared at an empty table thinking "I just need realistic data so I can build this," that is the entire reason this exists.

pip install dugalaxy. Fill the empty database, then tell me what breaks.

Repo: https://github.com/nugalaxy/dugalaxy · Built at https://nugalaxy.ai