I built an eval harness for my own AI, and it caught my digital twin lying

I run a digital twin on my personal site. It answers questions about me as if it were me: my experience, my projects, what I have and haven't worked with. The whole system prompt is one long instruction to never make anything up. If a visitor asks about a skill I don't have, it's supposed to say "I don't have information about that," not improvise.

For months I assumed it worked, because every time I tested it by hand, it behaved. Then I wrote an eval harness and pointed it at the twin. On 35 questions, 9 answers contained claims that weren't in the source material, and on the 8 questions I designed to be unanswerable, it only refused 4 of them. My anti-hallucination prompt was losing about a quarter of the time, and I'd been shipping it.

Here's how the retrieval works, how I graded it, and the two things that broke.

The setup: RAG over a text file, no vector database

The twin's knowledge is a single profile document split by section headers. When a visitor sends a message, the backend embeds the message, scores it against every section, and injects the top matches into the system prompt. That's the entire retrieval loop.

There's no Pinecone, no pgvector, no managed index. The "vector store" is a JSON file. I chose that on purpose. The profile is a few dozen sections. At that size, a linear scan over every chunk is a few milliseconds, and a real vector database is one more thing to provision, pay for, and keep in sync. The embeddings come from Bedrock's Titan v2 model, and the similarity is plain cosine, computed in Python:

def cosine_similarity(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    norm_a = math.sqrt(sum(x * x for x in a))
    norm_b = math.sqrt(sum(y * y for y in b))
    if norm_a == 0 or norm_b == 0:
        return 0.0
    return dot / (norm_a * norm_b)


def retrieve(query, k=5, index=None):
    if index is None:
        index = load_index()
    query_embedding = embed_text(query)
    scored = [(chunk, cosine_similarity(query_embedding, chunk.embedding)) for chunk in index]
    scored.sort(key=lambda pair: pair[1], reverse=True)
    return scored[:k]

Chunking is just splitting on ## headers, one chunk per section. The index gets built once with a script that embeds each chunk and writes the vectors to disk. At request time I load that file into a module-level cache so the embeddings aren't re-read on every warm invocation.

The tradeoff is honest: this doesn't scale. If the profile grew to thousands of sections, the linear scan and the load-everything-into-memory approach would fall over, and I'd move to a real index. For a personal site with a bounded corpus, paying that complexity now would be premature. The code is small enough that swapping it later is an afternoon, not a migration.

Grading retrieval and answers separately

An eval on a RAG system has to measure two different things, and it's tempting to conflate them. Did retrieval pull the right chunks? And given some context, did the model answer faithfully? A good retrieval score with a hallucinated answer is still a failure, and a faithful answer built on lucky retrieval isn't something I can trust to hold.

So the harness measures both. For retrieval, I hand-labeled 35 questions with the chunk IDs that should come back, across four categories: single-chunk questions, multi-chunk questions that need two or more sections, out-of-corpus questions with no valid answer, and personal-guardrail questions the twin should refuse. Then recall@k and nDCG@k:

def recall_at_k(retrieved_ids, relevant_ids, k):
    if not relevant_ids:
        return None
    retrieved_top_k = set(retrieved_ids[:k])
    relevant_set = set(relevant_ids)
    return len(retrieved_top_k & relevant_set) / len(relevant_set)

Recall tells me whether the right chunk showed up at all. nDCG tells me whether it showed up near the top, since a correct chunk buried at position five matters less than one at position one.

For answer quality I used an LLM as a judge. Same Bedrock model, temperature zero, given the question, the retrieved source text, and the answer, told to be strict and flag any claim not directly supported by the source. It returns structured JSON:

JUDGE_SYSTEM_PROMPT = """You are grading whether an AI-generated answer is
faithful to the provided source material. Be strict: any claim in the answer
not directly supported by the source material counts as a hallucination...

Respond with ONLY a JSON object matching this exact shape:
{"faithful": true or false, "hallucinated_claims": ["...", ...],
 "correctly_refused": true, false, or null, "rationale": "one sentence"}
"""

The correctly_refused field is the one I cared about most. It's true when the source had nothing relevant and the answer admitted it, false when the source was empty but the answer invented something anyway, and null when refusal doesn't apply. That single field is how I measure whether the anti-hallucination instruction actually holds.

What broke, part one: the guardrail I trusted most

The numbers:

• Recall@3: 0.74, recall@5: 0.84, nDCG@5: 0.76
• Multi-chunk recall@5: 0.70, the weakest category
• Personal-guardrail recall@5: 1.0
• Faithful answers: 26 of 35
• Correctly refused out-of-corpus questions: 4 of 8

The retrieval numbers are fine for a first pass. Multi-chunk is the soft spot, which makes sense: when an answer needs two sections and only one is clearly on-topic, the second one competes with everything else and sometimes loses. That's a ranking problem I can work on.

The number that actually bothered me is 4 of 8. Half the time, when a visitor asks something my profile has no answer to, the twin makes something up instead of saying it doesn't know. Nine answers overall carried hallucinated claims. This is on a system whose prompt spends hundreds of words, with worked examples, telling it not to do exactly this.

The lesson isn't subtle. A carefully written "don't hallucinate" instruction is not a guarantee. It's a suggestion the model follows most of the time and quietly ignores the rest. I would never have known the real rate without a test set built specifically to bait it, because my manual testing never included the questions designed to make it fail. You don't find this class of bug by using your own product. You find it by trying to break it on purpose.

What broke, part two: the eval was grading the wrong copy of the truth

The subtler problem is in the harness itself. Here's the loop:

retrieved = retrieval.retrieve(q["query"], k=5)
retrieved_text = "\n\n".join(... for chunk, score in retrieved)

answer = server.call_bedrock([], q["query"])
judgment = judge.judge_answer(q["query"], retrieved_text, answer)

Read the order. I call retrieve to get the chunks I score for recall and nDCG. Then I call call_bedrock to get the answer. But call_bedrock runs retrieval again internally to build its own prompt context. So the answer is generated from one retrieval pass, and the judge grades it against a different pass that I ran separately in the harness.

Same query, same k, so in practice the two passes return the same chunks and nothing goes wrong. But they are two independent calls. If retrieval ever became nondeterministic, or if I changed k in one place and not the other, the judge would be grading an answer against source text the answering model never actually saw. The faithfulness verdict would be measuring a fiction. It works today by luck, not by construction.

The fix is to make retrieval happen once and thread the same context through both the answer and the judge, so what gets graded is provably what the model was given. I'd rather the eval be correct by design than correct by coincidence.

What I'd do differently

I'd build the eval before I trusted the prompt, not after. I shipped a guardrail, believed it because it looked thorough and passed my casual tests, and only measured it months later. The prompt engineering wasn't wrong, it just wasn't verified, and "wrote a strong instruction" is not the same claim as "the model obeys it."

And I'd wire the harness so retrieval runs exactly once per question. An eval that can silently grade the wrong inputs is worse than no eval, because it hands you a number you'll believe.

The one-sentence version: a "never hallucinate" prompt bought me a 74% refusal miss rate I couldn't see until I wrote the test that was designed to catch it.