I grade my AI twin on real traffic, not just an offline test set

Most eval setups have a quiet lie in them. You run a fixed question set through your agent, a judge model scores the answers, you get a number, and you feel good. But that number describes a sandbox. It says nothing about the question a stranger typed into your live app at 2am, or whether your anti-hallucination guardrail held for that specific input.

Twin is my personal AI digital twin. It answers questions about me over a RAG index of my profile, running on Bedrock with Claude Sonnet 4.5, FastAPI backend, Next.js frontend. I already had an offline eval harness. What I added this week is a second judge that grades production traffic as it happens, and building it forced me to fix an integrity bug I'd been ignoring in the offline one.

The problem with the offline harness

My offline harness in evals/ does the honest-looking thing: load queries.json, run retrieval, run inference, hand the result to a judge. The judge prompt is strict about faithfulness:

JUDGE_SYSTEM_PROMPT = """You are grading whether an AI-generated answer is
faithful to the provided source material. Be strict: any claim in the answer
not directly supported by the source material counts as a hallucination, even
if it sounds plausible.
...
- "correctly_refused" is true if the source material contains nothing relevant
  to the question and the answer clearly states the information isn't available;
  false if the source material was empty/irrelevant but the answer invented
  something anyway...
"""

The judge is good. The problem was structural: the offline harness ran its own retrieval pass, separate from the answer's retrieval pass. So the judge was grading the answer against a source set that wasn't necessarily the one the model actually saw. If retrieval is non-deterministic or the index shifts, the judge is checking the wrong evidence. You can pass an eval where the model hallucinated and fail one where it didn't. The number is measuring the wrong thing.

That gap is invisible on a static question set because retrieval usually returns the same chunks twice. It stops being invisible the moment you point it at live traffic, where you have no fixed inputs at all.

The approach: capture the exact source the model saw

The fix is boring and correct. Whatever chunks the answer was generated from, those exact chunks get judged. No second retrieval.

In the chat path, right after inference, I capture the request with the same retrieved_chunks object that produced the answer:

retrieved_chunks = retrieval.retrieve(request.message, k=5)
# ... inference runs against retrieved_chunks ...
capture_live_eval(request.message, retrieved_chunks, assistant_response)

capture_live_eval is fire-and-forget. It never raises, because an eval capture failing should never break a user's chat:

def capture_live_eval(query: str, retrieved_chunks: List, answer: str) -> None:
    """Fire-and-forget capture of a real chat exchange for async
    faithfulness judging. Never raises."""
    if not EVALS_BUCKET:
        return
    try:
        retrieved_text = "\n\n".join(
            f"## {chunk.section_title}\n{chunk.text}"
            for chunk, score in retrieved_chunks
        )
        key = f"live/raw/{datetime.now().isoformat()}-{uuid.uuid4()}.json"
        body = {
            "timestamp": datetime.now().isoformat(),
            "query": query,
            "retrieved_chunk_ids": [c.chunk_id for c, _ in retrieved_chunks],
            "retrieved_text": retrieved_text,
            "answer": answer,
        }
        evals_s3_client.put_object(
            Bucket=EVALS_BUCKET, Key=key,
            Body=json.dumps(body), ContentType="application/json",
        )
    except Exception as e:
        print(f"Live eval capture failed (non-fatal): {e}")

The important line is that retrieved_text is serialized from the same chunks the answer was built on. The judge later reads exactly this. There is no second retrieval anywhere in the live path.

The judging happens out of band

I did not want to block a chat response on a judge call. So the write to S3 under live/raw/ triggers a separate Lambda via an S3 event. That Lambda reads the raw record, judges it, and writes the result to live/judged/:

def process_record(bucket: str, key: str, s3_client) -> None:
    raw = json.loads(
        s3_client.get_object(Bucket=bucket, Key=key)["Body"].read().decode("utf-8")
    )
    judged_key = key.replace("live/raw/", "live/judged/", 1)

    output = dict(raw)
    try:
        judgment = judge.judge_answer(
            raw["query"], raw["retrieved_text"], raw["answer"]
        )
        output["judgment"] = judgment
    except Exception as e:
        output["judgment_error"] = str(e)

    s3_client.put_object(
        Bucket=bucket, Key=judged_key,
        Body=json.dumps(output), ContentType="application/json",
    )

The chat request writes one small JSON object and moves on. The user waits on nothing. The judge runs whenever the event fires. The same judge_answer function backs both the offline harness and the live judge, so I am grading production and my test set with identical criteria. That reuse is deliberate. It's the only reason the two numbers are comparable.

One detail I like: the judge runs on Bedrock with temperature: 0.0 and parses a strict JSON object with a brace-depth scan rather than a regex, because judge models love to wrap JSON in prose no matter how many times you tell them not to.

What broke, and what I'd do differently

The capture path writes retrieved_chunk_ids alongside the text. I don't use those IDs in the judge yet, but they exist for a reason: my next problem is retrieval drift. If I rebuild the profile index, the chunk IDs behind a past judgment change, and a stored judgment silently starts referring to source text that no longer exists. Right now the judged record freezes retrieved_text at capture time, which protects the judgment itself, but I have no way to diff "what the judge saw" against "what the index says today." Storing the IDs is the first half of that; I haven't built the second half.

The bigger honest gap: I only judge faithfulness. The judge can tell me the model stayed inside its source material. It cannot tell me the answer was useful, or that retrieval pulled the right chunks in the first place. A perfectly faithful answer to badly retrieved context still passes. Faithfulness is the cheap half of quality, and it's the half I automated because it's the half a single model call can check.

I'd also reconsider the fire-and-forget silence. capture_live_eval swallowing every exception means a broken eval bucket produces one print and nothing else. On Lambda that print lands in CloudWatch and I will not notice for a week. If I were doing this again I'd emit a metric on capture failure, because a silent eval pipeline is the same as no eval pipeline.

Takeaway: an eval number is only trustworthy if the judge grades the exact evidence the model actually used, and the fastest way to find out whether yours does is to point it at real traffic instead of a frozen question set.