I found a bug in Scout that looked, at first, like a small classification mistake.
The assistant was being asked:
Is LeetCode documented as a gap?
The knowledge base clearly contained that relationship. The model had enough evidence to answer yes. The generated answer could even be semantically correct.
The system still treated the request like a different question:
Does Bradley know LeetCode?
That distinction is the entire story.
Scout did not fail because it lacked a fact. It failed because the fact was attached to the wrong proposition somewhere between the user’s sentence and the final response.
This is the next stage of Scout: not just retrieving information and asking a model to phrase it, but preserving what the user actually asked long enough for the model to answer that question.
The question looked simple because the failure was upstream
The knowledge graph contains a relationship equivalent to:
{
subject: "Bradley",
relation: "has_gap",
object: "LeetCode"
}
That relationship supports one answer very clearly:
Yes, LeetCode is documented as a gap.
It does not support this answer:
Yes, Bradley knows LeetCode.
Those sentences mention the same object, but they make different claims about it. One asks about a documented gap. The other asks about verified proficiency.
The correct semantic states are different:
| User question | What Scout must determine | Allowed answer |
|---|---|---|
| Does Bradley know LeetCode? | Is verified proficiency documented? | Unknown or qualified |
| Is LeetCode documented as a gap? | Is has_gap(Bradley, LeetCode) verified? |
Yes, affirmatively |
The first version of this path allowed the second question to fall into a generic negative-assessment route. That route made sense for unsupported skills. It did not make sense for a verified gap relationship.
The important change was not a better sentence in the prompt. It was carrying the requested proposition as structured state:
requested proposition: has_gap(Bradley, LeetCode)
direct answer: YES
fact state: TRUE
required stance: AFFIRM
Once that contract exists, the model has room to write naturally without having to guess what “documented as a gap” means.
The difference between an entity and a proposition
This distinction sounds obvious when it is written down. It is much easier to lose inside a live system.
Entity resolution answers questions like:
- Which project does “that app” refer to?
- Is “LeetCode” the same object as the item in the gap list?
- Is the user asking about Bradley, a project, a company, or a technology?
Proposition resolution answers a different question:
- What relationship is the user asking me to evaluate?
The entity can be correct while the proposition is wrong. Scout can resolve LeetCode perfectly and still answer the wrong question about LeetCode.
That is why a verified entity cannot be treated as blanket permission to make every claim involving it. Evidence has scope. A project’s React evidence does not automatically prove its deployment history. A gap record does not automatically prove proficiency. A source title does not automatically prove every sentence that contains a capitalized word from that title.
The contract has to carry the relationship, not only the nouns.
Conversation state is part of the answer
The proposition cannot always be read from the latest message alone.
Consider this conversation:
What roles fit me?
What about DevOps?
QA too.
Which is the strongest fit?
The last question does not name the comparison set. The assistant has to know that DevOps and QA were introduced as alternatives, not as properties of one project. It also has to avoid inventing a winner if one of the alternatives has incomplete evidence.
The same problem appears in shorter follow-ups:
| Follow-up | Likely interpretation |
|---|---|
| “What about DevOps?” | Add or switch to another role |
| “What about its deployment?” | Ask for a facet of the current project |
| “The other project” | Resolve an existing comparison member |
| “No, I meant QA” | Correct the current frame |
This is why Scout has a server-owned discourse frame. It tracks alternatives, corrections, ordinals, and facets instead of forcing the model to reconstruct the entire conversation from raw text on every request.
The model still writes the answer. It should not be responsible for inventing the conversation state that determines what the answer means.
Three kinds of evidence were being collapsed into one
Scout now keeps three separate axes:
- Conversation membership: did the user introduce this entity or alternative?
- Entity knowledge: what does the knowledge package document?
- Proposition support: does that evidence support this exact relationship?
An entity can be part of the conversation while a property is unknown. A project can be documented while its deployment history is not. A technology can appear in a gap list without being a verified skill.
This is an open-world rule. If Scout cannot find evidence that Bradley has used Rust, that does not automatically prove he has never used Rust. If Scout finds an explicit gap relationship, that is a supported statement about the gap, not an excuse to infer every other skill state.
For comparisons, the planner uses three-valued coverage:
-
FULL: comparable evidence exists for every member. -
PARTIAL: evidence exists for some members, but not enough for a complete comparison. -
UNKNOWN: the requested comparison is not supported by the available evidence.
That prevents the assistant from turning missing documentation into a ranking.
The evidence packet was teaching the model the wrong language
Scout also had a problem that was obvious in generated answers but easy to miss in code.
The model was receiving internal labels similar to:
FACT 2: LeetCode is documented as a learning gap
Sometimes it copied the label into the response. The validator rejected that correctly. A visitor should not receive a sentence that says “according to FACT 2.”
The deeper issue was architectural. The system was showing the model syntax that it did not want the model to imitate.
The evidence now has two representations. Internally, Scout keeps source identity, relationship ownership, and block boundaries for validation. The model receives clean factual text:
Evidence:
- LeetCode is documented as a learning gap.
- ProjectHub uses React and Node.js.
That separation is important for same-block validation. If one block says Atlas uses React and another says Orion uses Vue, the validator must not conclude that Atlas uses Vue because both facts appeared in one prompt.
The model should receive facts. The validator should retain ownership. Neither component should have to reverse-engineer the other’s representation from a string full of internal markers.
Why a fluent answer can still fail validation
Some failures were not model hallucinations at all.
An early relationship validator could see a project name, a relation verb, and a technology-looking word in the same sentence. It could then interpret the later word as a project technology even when the model had not actually made that claim.
That is a clause-boundary problem. The fix is not to disable relationship validation. The fix is to understand where one claim ends and another begins.
The same principle applies to entity grounding. A long source title can be a legitimate entity. A sentence-initial discourse word can be ordinary English. Expanding capitalization heuristics forever is not a substitute for structured provenance and better extraction rules.
Every remaining validation failure needs a cause, not just a label:
| Failure | Possible cause | Correct response |
|---|---|---|
entity_not_grounded |
hallucination, missing alias, or extractor false positive | reject, normalize, or fix extraction |
unsupported_relationship |
model claim, missing graph edge, or validator mistake | trace subject, relation, object, and evidence |
unsupported_tech_claim |
missing provenance or real hallucination | repair the evidence path, not a global whitelist |
overclaim |
claim exceeds tenant evidence or contract | enforce the claim ceiling |
This is why I do not want to weaken validators just to raise a score. A validator that rejects supported language is broken, but a validator that accepts unsupported claims is also broken.
Repair must preserve the same contract
Scout has a repair path for answers that fail validation. Repair can remove a scaffold leak, correct an unsupported technology, or fix an answer that takes the wrong stance.
But a second model call is not automatically recovery. It is recovery only if it receives enough of the original semantic state to answer the same question:
- original user question
- resolved question
- active entity
- requested proposition
- direct answer
- fact state
- required stance
- claim ceiling
- forbidden claims
- canonical evidence
- validation diagnostics as errors, not facts
If the first draft invents Kubernetes, the repair must remove Kubernetes. It must not repeat the word because it appeared in the rejected answer.
The repair packet needs separate namespaces:
FACTUAL EVIDENCE
Verified facts from the tenant knowledge package.
SEMANTIC CONTRACT
What proposition the user asked about and what stance is allowed.
VALIDATION DIAGNOSTICS
What the previous draft did wrong.
REJECTED DRAFT
Optional wording context. Not evidence.
This matters for future-capability questions too. “Could Bradley learn COBOL?” is not the same as “Is COBOL currently documented as one of his skills?” A repair that turns future potential into a current-state denial may avoid one overclaim while still answering the wrong question.
The numbers only matter when their scope is clear
The qualification work produced useful measurements:
| Check | Result |
|---|---|
| Local tests | 1,159 / 1,159 |
| Retrieval Recall@6 | 1.000 |
| Retrieval MRR@6 | 0.942 |
| Targeted live battery | 22–23 / 23 across runs |
| Larger conversation gate | 95 / 132 turns; 20 / 33 conversations |
Those are not one accuracy percentage. They measure different layers of the system.
- Unit tests ask whether a rule behaves as written.
- Retrieval tests ask whether the right evidence can be found.
- Live conversations ask whether meaning survives state, generation, deadlines, and validation.
- Deployment checks ask whether the intended runtime is actually serving requests.
The larger gate combines model generation, conversation state, retrieval, contract construction, validation, repair, provider behavior, and the evaluation harness. That makes it useful, but it also means a failed turn is only the beginning of the investigation.
The next useful report is a failure corpus: stance mismatches, entity-grounding errors, unsupported relationships, unsupported technologies, scaffold leakage, true prompt leakage, provider failures, deadline failures, repair attempts, and accepted repairs.
Scout is still generative
The surrounding system is becoming more deterministic because the model should not have to rediscover the boundaries of the question on every request.
The division is:
Code decides:
- what the user is asking
- which entities and relationships are involved
- what evidence is supported
- what stance is allowed
- what claims must be rejected
The model decides:
- how to express that supported meaning naturally
There are no benchmark-specific “if the question contains LeetCode, print this paragraph” answers in the intended design. The LeetCode fix works because has_gap is represented as a supported proposition. The same machinery should handle a warranty question, a project deployment question, or a future-role question without adding a new canned response.
What changed for me
Before this bug, I thought of retrieval as the grounding layer. Now I think of grounding as a chain:
- Resolve the conversation.
- Identify the exact proposition.
- Retrieve evidence for that proposition.
- Preserve the contract through generation.
- Validate the meaning of the generated claim.
- Repair the answer without changing the question.
If any link changes the proposition, the final answer can be wrong while every sentence still sounds professional.
That is more dangerous than an obvious hallucination because it can survive a quick read. The words look reasonable. The evidence is real. The answer is still answering something else.
The LeetCode case gave me a small, clear example of a larger design rule: reliable generative software is not only about giving a model better facts. It is about preserving the meaning of the request while the software around the model does its work.
That is what Scout is working on now.
Top comments (1)
Great article brotha! Keep em coming.