The Ticket Router That Passed Every Test and Still Sent Refund Requests to Sales
A mid-sized SaaS client brought us in after their AI-powered support ticket router had been quietly misrouting a category of tickets for weeks. Not obviously, not consistently, just often enough that the sales team kept getting refund requests they had no way to process, and the support queue kept losing track of them in the shuffle.
The router had a respectable test suite. Every test case passed. Keyword coverage looked solid, intent categories were well-defined, and the team had even run semantic similarity checks comparing incoming tickets against a labeled reference set. On paper, this was a well-tested system. In production, it was misreading a specific, recurring pattern of language that none of those tests had been built to catch.
This case is worth walking through in detail, because the gap it exposed isn't rare. It's one of the most common blind spots I see in AI testing programs: teams confuse semantic similarity scoring, a single technique, with semantic validation, a broader testing discipline that covers how a system handles meaning, not just how closely its output resembles a reference answer.
What Was Actually Going Wrong
Once we pulled the misrouted tickets and looked at them side by side, a pattern emerged fast. Nearly every one of them used negation, indirect phrasing, or a paraphrase the model's training and test data hadn't anticipated.
A ticket that said "I was not charged what I expected, please look into this" was landing in a general billing-inquiry category instead of the refund category, because the surface language overlapped more with billing-inquiry examples in the training set than with the more direct refund tickets the classifier had been validated against. A ticket phrased as "can you reverse this transaction, it shouldn't have gone through" was being read as a fraud report rather than a refund request, again because the phrasing diverged from the direct "I want a refund" language the test set leaned on heavily.
None of this was a hallucination in the sense most teams test for. The system wasn't inventing facts. It was misunderstanding intent, consistently, on a specific and predictable class of input: negated statements, indirect requests, and paraphrases that shifted vocabulary without shifting meaning. That's a semantic validation gap, and it's structurally different from the output-correctness problems most AI testing programs are built to catch.
Why Semantic Similarity Testing Alone Missed This
The client's existing similarity testing compared new inputs against a labeled reference set using embedding distance, a reasonable and fairly standard technique. The problem is what that technique is actually built to measure: how close a new input sits to examples the system has already seen and been validated against. It tells you very little about whether the system correctly handles inputs that are meaningfully different in form but identical in meaning, which is exactly the case with negation and paraphrase.
Worse, negation is a specific, known weak point for many embedding-based approaches. "I was charged" and "I was not charged" often sit closer together in embedding space than intuition suggests, because the words are almost identical and negation is a small syntactic marker carrying a large semantic weight. A testing approach that leans entirely on similarity scoring will systematically underweight exactly the failure mode that caused this client's misrouting.
What Semantic Validation Actually Covers
Semantic validation, properly scoped, is a set of testing techniques aimed at whether a system correctly understands and preserves meaning, both on the input side (does it correctly interpret what a user meant) and the output side (does the generated response accurately reflect the meaning of the source material it was built from). It's broader than any single scoring method, and for this engagement, we ended up building test coverage across five specific categories.
Negation and polarity testing. Deliberately constructed test pairs where a small negation flips the correct classification or answer entirely, "I was charged twice" versus "I was not charged," "this feature works" versus "this feature does not work." If your test set doesn't include negated variants of your core scenarios, you have no evidence the system handles them correctly at all.
Paraphrase invariance testing. The same underlying request, expressed in multiple genuinely different phrasings, checked for whether the system produces the same correct classification or answer across all of them. This catches exactly the "reverse this transaction" versus "I want a refund" gap that caused the original incident.
Entailment and contradiction testing. Does the system correctly recognize when one statement logically follows from another, and when two statements conflict. This matters enormously for RAG-based systems specifically, where a generated answer needs to be checked not just for similarity to a reference, but for whether it actually follows from, or contradicts, the retrieved source material.
Ambiguity and underspecification handling. Test cases deliberately built to be ambiguous or missing key information, checking whether the system asks a clarifying question, makes a reasonable default assumption, or (the failure mode you're looking for) confidently picks one interpretation and proceeds without any signal that the input was underspecified.
Synonym and terminology drift testing. Domain-specific systems often get tested heavily against the exact terminology used during development, then encounter real users describing the same concept with different words entirely. A healthcare intake system tested against "chest pain" needs to also handle "tightness in my chest," and an enterprise IT support bot tested against "VPN" needs to handle "the remote access thing."
How This Differs From Output Quality Evaluation
I want to be precise about this distinction, because it's the one that gets blurred most often in practice. Output quality evaluation, semantic similarity scoring against reference answers, groundedness checks, LLM-as-judge scoring, is primarily concerned with whether a generated output is correct. Semantic validation, as I'm using the term here, is concerned with whether the system correctly handles meaning throughout the entire interaction, which includes how it interprets ambiguous or differently-phrased input before generation ever happens.
A system can score well on every output quality metric you throw at it and still have a semantic validation gap, exactly like this client's router did, because the failure happened at interpretation, not generation. Testing programs that only evaluate final outputs against reference answers will systematically miss input-side misunderstanding, because by the time you're scoring the output, the misinterpretation has already happened and the output is, in a narrow sense, a "correct" response to the wrong understanding of the request.
What We Actually Built for This Client
The fix wasn't a new model. It was a structured semantic test suite layered on top of what they already had, plus a change to how their intent categories were defined in the first place.
We built contrastive test pairs for every core ticket category, each direct example paired with a negated version, a paraphrased version, and an ambiguous version, and ran the classifier against all four systematically rather than just the direct phrasing the original test set leaned on. We added an explicit negation-detection pre-check as a lightweight rule-based layer ahead of the classifier, flagging inputs containing negation markers for closer review rather than trusting the classifier's raw output on them. And we restructured a handful of intent category definitions that had been drawn too narrowly around the specific phrasing the development team had used when building the original examples, rather than around the actual range of ways customers describe the same problem.
None of this required retraining the underlying model. It required testing for the right failure mode, which the team hadn't been doing, not because they were careless, but because their testing program had matured around output correctness and never explicitly built out an input-understanding layer.
When Semantic Validation Deserves Serious Investment, and When It Doesn't
Not every system needs the full five-category treatment. A low-stakes internal tool with a small, technically literate user base who phrase requests fairly consistently can often get by with lighter coverage, primarily paraphrase and negation testing on the highest-traffic intents, rather than the full suite.
Where I'd insist on it, without exception, is any system making routing, classification, or decision calls that affect real outcomes, refunds, account actions, compliance-relevant categorization, medical or legal intake triage. In those cases, an input misunderstood is functionally identical to an output that's wrong, and it deserves the same testing rigor. The cost of building negation and paraphrase test pairs is genuinely small relative to the cost of a misrouted refund request or a mishandled compliance-relevant ticket sitting unresolved for weeks.
Mistakes Worth Naming Directly
Treating semantic similarity scoring as complete coverage for meaning-related risk. It's one useful signal, not a substitute for deliberately constructed negation, paraphrase, and ambiguity test cases.
Building test data entirely from development-time phrasing. If your test set uses the same vocabulary your team used while building the system, it will systematically underrepresent how real users actually phrase the same requests.
Assuming a bigger or newer model resolves negation handling on its own. Negation sensitivity is a known, persistent weak spot across many embedding and language model approaches. It needs dedicated test coverage regardless of model choice, not an assumption that a model upgrade quietly fixes it.
No process for reviewing production misclassifications for a negation or paraphrase pattern. This client's incident sat unresolved for weeks because nobody was looking at misrouted tickets through a semantic lens, only through a "was this ticket eventually handled" lens. The pattern was visible the moment someone looked for it specifically.
Where This Leaves Enterprise Teams
The uncomfortable finding, for this client and for most teams I walk through this exercise with, is that a passing test suite and a genuinely well-tested system are not the same thing. Passing tests only tells you the system handles the inputs your test set anticipated. Semantic validation is what tells you whether it handles the actual range of ways people express the same meaning, which is a considerably harder and more interesting problem than matching phrasing to a reference set.
This layered approach to meaning-level testing is a core part of the AI Testing Framework we build out with enterprise clients at PrimeQA Solutions, because the incidents that cost the most are rarely the ones where a system got something factually wrong. They're the ones where it correctly answered a question nobody actually asked.
Top comments (0)