The last lesson was about validating what a model hands you. The story behind it: a set of prompts that had returned real, criteria-matched vendors for weeks came back in staging with placeholder junk, literally the words Vendor A, Vendor B, Vendor C. So I built the validation layer, and the last gate in it is a model checking a model.
Then FromZeroToShip asked three questions in the comments, and all three were about the gate rather than the model. That's the harder thing to look at, and I hadn't written all of it down. Here's the long version.
What was on the fail list that I hadn't already been burned by?
More than the question assumes, and not because I got clever about imagining failures.
The placeholder output changed what I do with a failure. I stopped fixing the instance and asked what class it belonged to, and that class is a lot wider than "the model emitted example data." It's a suggestion that looks fine and isn't usable. Two of those I had never hit went in on the back of it:
- A vendor that's wrong for the category.
- A vendor that's no longer in business.
Neither has anything to do with placeholder text, and both would sail through a schema check looking like a perfectly real answer. They also changed the prompt that produces the suggestions, not just the gate. Fixing only the failure I actually met would have left both of them live.
So the list isn't purely retrospective. It grows by generalizing from the one failure you hit to the class it sits in, and it keeps growing from what the running system actually throws at me rather than from what I remembered to imagine.
Is it foolproof? No. What's left is the case worth worrying about: results that read as real, pass the schema, satisfy every criterion I gave, and are still wrong. You can't validate the truth of a guess from inside the system. You can only lower the cost of it being wrong. That means a human in the loop at the stage where being wrong is expensive, the confidence surfaced so the answer is checkable, and a clear way for the user to push back and regenerate.
Has the gate ever failed on purpose?
A check nobody has watched reject anything isn't proven yet. Normally you'd prove one by breaking the thing it guards, watching it go red for the reason you expect, and then putting it back.
That runs into a wall here. A model's output is a guess. You can't predict when it'll be wrong, and you can't make it be wrong on demand. So the failure you built the gate for is exactly the one you can't go and produce.
So instead of provoking it, we need to simulate it. The mock model can be asked for a specific class of bad answer, and there are tests to do exactly that: produce the failure on purpose, then assert the gate rejects it for the expected reason. Same TDD loop, with the mock supplying the failure the real thing won't hand you on demand. Real bad responses at runtime don't get thrown away either. They go back in as cases, so what is being tested against keeps growing out of what actually happens.
The mocks earn their keep twice over, because the same setup drives the interface: what the user sees while a call is slow, when it comes back malformed, and when it doesn't come back at all.
Does a same-model gate count as a second opinion?
No, and the last lesson should have said that plainly instead of presenting the two options as equal.
The gate is a different task with a different prompt, so even on the same model it catches plenty: placeholders, filler, items that miss the stated criteria, instructions the first call didn't follow. What it can't catch is a shared blind spot. If the model believes something wrong, it'll believe it again when you ask it to check its own answer. So when the failure you're worried about is judgment rather than compliance, that's exactly when you want the gate on a different provider and model altogether.
And a long run of "nothing found" is not evidence that things are clean. At that point you're QA'ing the model API, not your system. Which is the second question again: the only way to tell a working gate from one that always passes is to make it fail on purpose.
What the thread was really about
Three questions, one thing underneath all of them. I'd built a layer to check the model and hadn't turned the same scrutiny on the layer.
A validation gate is code, so test it like code. Make it fail on purpose, confirm it fails for the reason you built it for, and don't read a long stretch of clean results as proof of anything.
The rest is the part you can't test away. Assume the data can be wrong, and design for that. What you do about it depends entirely on what being wrong costs you.
Thanks to @fromzerotoship for pushing on this. If you're building on top of AI workflows and running into the same questions, I'd be glad to compare notes.
Top comments (0)