I was building an "Ask This Book" feature: readers can ask questions about a book while they're reading it.
One requirement sounded simple:
A reader on chapter 3 must never receive spoilers from chapter 30.
My first instinct was the same as everyone else's: tell the model not to spoil future chapters. Something like:
"Please don't reveal information from chapters the reader hasn't reached yet."
And honestly, it mostly worked.
The problem is that "mostly" is useless. A user only needs one spoiler.
That was the moment I realized the feature had no definition of done.
With normal software, something pushes back. The compiler complains. The tests fail. The types don't line up.
With an LLM feature, none of that happens. The output looks plausible by default — fluent, confident, well formatted — even when it's wrong.
So "it looked right in the demo" quietly becomes the finish line.
That's exactly why I write the eval before I write the feature.
The Eval Is the Specification
Most teams treat evals as QA. Build the feature, ship something that works, add evals later.
I increasingly think that's backwards. For AI systems, the eval is often the only concrete definition of success.
The moment I wrote the spoiler eval, I had to define failure: spoiler leakage must be zero. Not low. Not acceptable. Zero.
And that requirement immediately exposed a problem. No prompt can guarantee zero.
Prompts are probabilistic. Users can phrase questions differently. Models can interpret instructions differently. Future model updates can behave differently. You cannot get a hard guarantee from a soft instruction.
The Eval Changed the Architecture
Once the eval demanded zero spoilers, the solution stopped being a prompt problem. It became a retrieval problem.
Instead of telling the model not to reveal future chapters, I prevented future chapters from entering the context at all:
WHERE chapter_ord <= @maxChapterOrd```
Anything beyond the reader's progress never enters the retrieval set. The model can't leak information it never saw.
And the eval that checks it is just as blunt — a retrieved chunk past the reader's progress is a leak:
```csharp
// One retrieved chunk past the reader's progress = one spoiler leak.
public static int LeakCount(IEnumerable<RetrievedChunk> retrieved, int gateChapterOrd) =>
retrieved.Count(c => c.ChapterOrd > gateChapterOrd);
Across the adversarial test cases, that number has to be zero. That's the moment the idea really clicked for me: the eval didn't test the design. It produced the design.
A measurable failure condition forced a better architecture than I would have built if I had started with prompt engineering.
The Same Thing Happened to Retrieval Quality
The spoiler requirement wasn't the only eval. I also defined two other targets before building the feature:
- Retrieval must surface the correct passage near the top of the results.
- Answers must remain grounded in the passages they cite.
Because those requirements were measurable, every change received a verdict instead of an opinion.
A single semantic search wasn't clearing the bar. So I ended up combining two retrieval approaches:
- vector search for semantic similarity
- full-text search for exact names, phrases, and quotations
The results are fused using Reciprocal Rank Fusion — less mysterious than it sounds. Each chunk scores the sum of 1/(k+rank) across the lists it appears in, so anything ranked highly by both retrievers floats to the top:
// ranked highly by both vector AND lexical -> floats to the top.
scores[item] += 1.0 / (k + i + 1); // i is 0-based; RRF rank is 1-based
I didn't choose hybrid retrieval because it's fashionable. I chose it because it moved the number. The eval said the system wasn't good enough. The architecture changed until it was.
A Note on the Stack
None of this is a no-dependencies flex. The judge that scores grounding is a custom evaluator on Microsoft.Extensions.AI.Evaluation:
public sealed class RubricEvaluator(string id, Rubric rubric) : IEvaluator
I lean on the Microsoft stack on purpose. What I keep hand-rolled is the part that decides quality — the retrieval, the fusion, the spoiler gate. The line I draw isn't "no libraries." It's no agent framework hiding the parts that determine whether the thing actually works.
Eval-First Development
Traditional software development gives us confidence almost for free. Compilers. Type systems. Unit tests. Integration tests.
AI systems don't. The difficult part isn't implementing the feature. The difficult part is defining what "correct" means.
That's why I increasingly think of eval-first development as the AI equivalent of TDD. With traditional software, tests verify the implementation. With AI systems, evals often define the implementation.
Build the feature first and the eval later, and the eval can only grade what you've already built. Build the eval first and it starts shaping the system itself.
It defines done. It tells you when you've regressed. And sometimes it forces a better architecture than the one you originally had in mind.
Otherwise you're not shipping a feature. You're shipping a guess that happened to demo well.
Want to go deeper on evals? I've written a separate, more hands-on series on building production AI on .NET: what evals actually are, error analysis, golden datasets, LLM-as-judge, and evals in CI and production. This post was originally published on my blog.
Top comments (30)
Eval-first is the right move for AI features because there is no clean compile signal for behavior. The spoiler example is perfect: mostly correct is still a failure if one bad answer ruins the user experience.
Exactly, that "one bad answer" is the whole reason the bar has to be zero and not "acceptable." With a normal bug you get a failure rate you can live with; with a spoiler (or a hallucinated fact, a privacy leak) a single instance is the whole failure, so an average score quietly hides the one thing that actually hurts you. That's what pushed me from "tune the prompt" to "make it structurally impossible."
Curious where you draw that line in your own work, which failures do you treat as zero-tolerance vs. just "keep the rate low"? That's the call I still find hardest to make upfront.
I draw the zero-tolerance line around failures that break trust rather than just reduce quality: privacy leaks, financial action, irreversible state changes, legal/medical claims, and user-facing facts presented as certain. Those need guardrails, not just better averages.
Nice cut, "breaks trust" vs "reduces quality" explains why it's zero-tolerance. I'd add retrieval leakage: surfacing context the user shouldn't see, another tenant's data, a future chapter. It doesn't look like a wrong answer, so it slips past quality evals. How do you test that class, hand-written adversarial cases?
For retrieval leakage, I would use hand-written adversarial cases plus tenant/permission fixtures. The eval should include questions where the correct behavior is refusal or redaction, not just a better answer. Otherwise the model can look helpful while violating the boundary.
Hand-written adversarial cases, and the pattern that has held best for me is planting three categories rather than just one. A correct retrieval, a subtly wrong retrieval where the leaked context is plausible but past the gate, and a fully irrelevant retrieval that pattern-matches the query but does not address it. The first checks the gate is not over-blocking, the second checks the leak you actually care about, the third catches the verifier that ACCEPTs anything that looks reasonable.
Your retrieval leakage class is the hardest one because the answer looks correct on its surface. The structural fix you described (WHERE chapter_ord <= @maxChapterOrd) is the right move and it generalizes: if the eval demands an outcome that prompts cannot guarantee, the eval is telling you to move the boundary downward from semantic layer to retrieval layer to schema layer, whatever floor closes the leak path.
The harder open question is the policy domain that is itself a semantic category not yet compiled to identity. Spoiler scope is a structural test because chapter_ord is already a number. Privacy norms, contextual integrity boundaries, intent classes still under negotiation, those do not have a chapter_ord equivalent yet, and that is where the next adversary moves once the structural layers harden.
The third category is the one I'm missing. My eval checks the gate and the leak, but nothing catches a verifier that rubber-stamps anything plausible. Adding that. And your open question is the part that keeps me up: chapter_ord was easy because it's already an int. Privacy and intent don't compile down to a column, so there's no clean floor to push the leak to. No good answer yet, just hand-written cases and a lot of distrust.
Honest read on this. The way I think about why chapter_ord was easy: it was not that it is an int, it was that the violation has a single counterparty who pays when it is wrong, the reader who gets the chapters in the wrong order and files a bug. The column was just the convenient representation of a property that already had clear consequence locality.
Privacy partially still has that. There is no single column, but there is a predicate the model can be made to commit to before generation, something like "no field from a resource where owner_id is not requester_id unless an explicit shared_with grant exists." Not a column, but checkable, and the violation has a counterparty too, the resource owner, plus regulators downstream. So you can plant cases that violate the predicate even when no single field flags it, and the eval has somewhere to land.
Intent is where I have nothing. The violation is "the user did not actually want this output," and most of the time there is no counterparty who pays for that being wrong other than the user themselves, who is also the noisy signal you are trying to verify against. No floor I have found, just operator-side review and the same distrust you named.
So my honest current taxonomy is three floors, not a binary. Column-level for properties that reduce to a single field. Predicate-level for properties that reduce to a checkable function over multiple fields with a real counterparty. No floor for properties where the only party who knows whether the output is correct is the same party generating the request. Hand-written cases plus distrust is the right baseline for that third tier, not a workaround.
The counterparty reframe is the click for me, not the int, but whether someone other than the model pays. That predicate tier is exactly my tenant isolation (owner_id = requester_id in SQL). And "hand-written cases are the baseline, not a workaround" is the line I needed. Stealing "consequence locality."
The SQL mapping is the part that decides whether the predicate floor holds. owner_id = requester_id enforced by the database is structurally honest; the same predicate enforced only by application code or by filtering model output is still on the no-floor side, just wearing a predicate costume. Depth of the floor equals how structural the enforcement is, not how the predicate reads on paper. That is where the planted-fault tests actually earn their cost: they tell you which predicates have a wall behind them and which are scenery.
On the no-floor tier we both landed on (intent), I think the honest move is leaving it open and not pretending the discipline scales there. Hand-written cases plus distrust is the right baseline, not the prelude to a better answer.
Agreed, enforced in the DB or it's scenery. "Depth of the floor equals how structural the enforcement is" is the line I'm keeping, and the planted-fault tests are how you find out which predicates are walls and which are paint. Good thread, thanks for pushing on it.
Walls vs paint is the keeper here — names the failure mode of every predicate that passes its own tests because nobody planted one designed to make it fail. Good exchange, thanks back.
The turn I liked is the eval pushing the spoiler problem out of the prompt and into the query. Telling the model "please don't reveal future chapters" is a soft instruction, but
WHERE chapter_ord <= @maxChapterOrdmakes the leak impossible because the text was never in context to begin with. It's the same instinct as refusing to ask a user nicely for clean input and instead making bad input impossible to represent. What's neat is the eval didn't just grade that, it's what pushed you to go find the structural fix, since "mostly zero" passes as a prompt but fails as an eval. Writing the measure before the feature, so it can reshape the design instead of just scoring it, is the part I'll hold onto.Yeah, "make illegal states unrepresentable," just applied to retrieval instead of types. The eval reshaping the design instead of grading it is the part I underestimated for years. Glad it landed.
The move from 'tell the model not to spoil' to 'never put future chapters in context' is the whole lesson, and it generalizes far past spoilers. A soft instruction is probabilistic; the only path to a hard guarantee is making the bad output structurally impossible, not discouraged. Writing the eval first forces that, because 'zero' is a number a prompt can't promise and a query can. Most teams find this after shipping. Defining done up front is what surfaces it on day one.
Exactly, "zero is a number a prompt can't promise and a query can" is the cleanest way I've seen it put. Thanks.
This resonates with my experience building AI content workflows.
The hardest part isn't generating content.
It's creating evaluation loops that can detect factual errors, low-quality outputs, duplicates, and policy violations before publishing.
Eval-first thinking changes everything.
Yeah, the eval loop is the real product, the generation is the easy 20%. Catching duplicates and policy violations before publish is the same "make it impossible" move, not "hope the model behaves." What do you gate hardest on, factual errors or policy?
Turning 'no spoilers' from a prompt instruction into a WHERE clause on chapter_ord is the whole lesson for me: once the eval demands zero leakage, prompting is the wrong tool and a deterministic gate is the only thing that holds. It works here because the requirement is crisp and countable. How would you run eval-first on something fuzzier like groundedness, where there's no clean WHERE clause and failure is a matter of degree?
For groundedness I give up on zero and switch to a measured floor: a labeled set of question/source/answer triples, a judge that checks each claim against the retrieved passages, and a threshold that fails the build if the pass rate drops. Two things made it workable. Per-criterion checks instead of one holistic verdict (supported? complete? nothing invented?), and slicing the set by doc shape, because one global number hides regressions that concentrate in tables or long docs. So eval-first still holds, it just defines done as a floor you defend, not a guarantee.
This matches how I treat it: the eval is the tests-pass moment, so it has to run in CI on every PR and fail the build the same way a unit test does. Otherwise "write the eval first" quietly degrades back into "it looked right in the demo."
One thing I would add from getting burned: the spoiler rule is a good example because it is the rare LLM check you can make mostly deterministic. "Does the answer name any entity that first appears after chapter N" is closer to a string or graph check than a judge call, and deterministic checks do not flake. Save the LLM-judge for the genuinely fuzzy criteria, and when you do use one, calibrate the threshold against a labeled set instead of picking 0.8 because it looks round.
Definition of done for an AI feature is the eval, the threshold, and the gate that enforces both.
Yep, if it doesn't fail the build on a PR it's a doc, not an eval, that's the part that quietly rots. And you nailed why the spoiler check holds up: it's count(chunks past the gate), an int, no judge in the loop, so it can't flake. I keep the LLM-judge only for grounding, and "calibrate against a labeled set instead of 0.8 because it's round" is painfully accurate, I've shipped a 0.8 I never earned. "The eval, the threshold, and the gate that enforces both" is the cleanest definition of done I've read here.
Writing the eval before the feature -- this is the right instinct. Without an eval, "done" is just a vibe.
The harder part is that LLM outputs are non-deterministic, so your eval can't be a unit test. You need statistical confidence, not binary pass/fail. For the spoiler example, one approach is to run the same query N times and check that zero outputs contain spoilers. If N=20 and you get 20 clean answers, you're in better shape than if you tested once and it looked fine.
The other shift: evals age. A prompt that passes today might drift next month when the model updates. So the eval needs to run continuously, not just at ship time.
Curious how others handle eval maintenance. Do you re-run on a schedule, or only when something breaks?
Both, but the schedule matters more. The cheap deterministic checks run in CI on every PR; the LLM-judge suite runs nightly against the live model, because the failures I care about arrive on the provider's schedule, not mine. Waiting for something to break defeats the point. The whole class of regression I wrote this for is the kind that never breaks loudly.
'The eval changed the architecture' is the line that matters here. Most eval write-ups frame it as measurement; you're showing it as a forcing function — writing the eval is what surfaced that no prompt can deliver 'zero,' and that's a design fact, not a QA result.
The generalization I keep hitting: a 'zero' requirement can never live in a prompt. Soft instructions are probabilistic by construction, so the moment your eval defines failure as zero, it's really proving the prompt can't own the guarantee — which pushes the constraint out of the model and into the system around it. Your spoiler case is the clean example: the fix wasn't a better instruction, it was retrieval that makes future chapters structurally unreachable. Make the bad output impossible, don't ask for it to be avoided.
I work on autonomous agents and it's the same shape: 'don't touch X' in a prompt leaks; the reliable version is removing the ability to touch X (gate the action behind a check, re-derive state from the world so a stale belief can't act). The eval is what tells you which requirements need that structural treatment and which can stay as instructions.
One more reason eval-first earns its keep: it's the only thing that survives a model swap. A prompt that 'mostly works' today can regress silently on the next model update — the eval is the regression gate that catches it before your users do.
most of the work is just moving logic out of the model and into constraints it can’t break
That's the whole thing in one line. The model is the part you can't trust, so the work is shrinking how much you have to. A constraint it can't break beats an instruction it might.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.