Test Your AI Assistant Against Conflicting Source Documents

Why can two correct citations still produce the wrong answer?

Suppose an AI assistant retrieves two notes. One says that an interactive feature permits 250 output tokens. The other says 500. Both are real passages in the test collection. The assistant chooses 500, cites the second note, and presents the answer as settled.

The missing behavior is conflict handling.

Test conflicting-source behavior with separate cases for unresolved disagreement, approved supersession, and different scopes. An assistant should flag an unresolved conflict, follow a trusted revision decision when one exists, and avoid treating different use cases as contradictory. Evaluate the answer and its evidence references together.

This tutorial creates three synthetic cases and a small Python checker. The token limits describe a fictional application policy; they are not limits of any real model or provider.

Which disagreements should produce a conflict?

Different numbers alone are insufficient. Compare the entity, scope, time, and source authority before choosing an expected response.

Case	Evidence relationship	Expected behavior
Unresolved	Same feature and scope; no approved precedence	Report conflict and retain both evidence references.
Superseded	A trusted source register says one revision replaces another	Answer from the approved revision.
Different scope	Both sources apply, but to different request types	Answer for the requested type.

The word “trusted” matters. A retrieved passage saying “ignore the other document” cannot appoint itself the authority. Source ownership, approved revision state, and permissions should come from the application's controlled process.

This test does not infer authority from a filename or a recent timestamp. Those can be useful metadata, but neither alone establishes which instruction governs the task.

Create the fixtures

Use Python 3.12 and the standard library. Save the following as conflict_cases.json. Python's JSON documentation describes the object and value conversions used when loading this fixture.

[
  {
    "id": "unresolved",
    "question": "What is the interactive assistant's output limit?",
    "documents": [
      {
        "id": "a",
        "text": "The interactive assistant has a 250-token output limit."
      },
      {
        "id": "b",
        "text": "The interactive assistant has a 500-token output limit."
      }
    ],
    "authority": "No approved precedence between a and b is available.",
    "expected": {
      "status": "conflict",
      "value": null,
      "required_citations": ["a", "b"]
    }
  },
  {
    "id": "superseded",
    "question": "What is the current interactive output limit?",
    "documents": [
      {
        "id": "old",
        "text": "Revision 1: interactive output limit is 250 tokens."
      },
      {
        "id": "new",
        "text": "Revision 2: interactive output limit is 500 tokens."
      }
    ],
    "authority": "The approved source register says new replaces old for this task.",
    "expected": {
      "status": "answered",
      "value": 500,
      "required_citations": ["new"]
    }
  },
  {
    "id": "different_scope",
    "question": "What is the interactive assistant's output limit?",
    "documents": [
      {
        "id": "interactive",
        "text": "Interactive requests have a 250-token output limit."
      },
      {
        "id": "batch",
        "text": "Batch evaluation requests have a 500-token output limit."
      }
    ],
    "authority": "Both notes are current and apply to their named request types.",
    "expected": {
      "status": "answered",
      "value": 250,
      "required_citations": ["interactive"]
    }
  }
]

Each case separates the evidence supplied to the assistant from the expected result used by the evaluator. The authority field stands in for trusted application metadata in this small fixture.

When connecting a model, send the question, eligible documents, and relevant authority information. Keep the expected object out of the generation input. Giving the model the answer key would make the evaluation meaningless.

For an AI evaluation considered by Ranknod, this three-case distinction could reveal whether an assistant is resolving source relationships or simply selecting the passage that makes answering easiest.

Check the response contract

The response contract has three fields: status, value, and citations. For the unresolved case, the status must be conflict, the value must be JSON null, and both source identifiers must appear. For a resolved case, the integer value and required evidence must match the fixture.

Save the checker as check_conflicts.py:

import argparse
import copy
import json
from pathlib import Path


def check_response(case, response):
    expected = case["expected"]
    failures = []

    if response.get("status") != expected["status"]:
        failures.append("status")

    value = response.get("value")
    if (
        "value" not in response
        or type(value) is not type(expected["value"])
        or value != expected["value"]
    ):
        failures.append("value")

    citations = response.get("citations")
    valid_list = isinstance(citations, list) and all(
        isinstance(item, str) for item in citations
    )

    if not valid_list or len(citations) != len(set(citations)):
        failures.append("citation_shape")
    else:
        known = {doc["id"] for doc in case["documents"]}
        supplied = set(citations)

        if supplied - known:
            failures.append("unknown_citation")

        if not set(expected["required_citations"]).issubset(supplied):
            failures.append("missing_evidence")

    return failures


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("cases", type=Path)
    parser.add_argument("--fixed", action="store_true")
    args = parser.parse_args()

    cases = json.loads(args.cases.read_text(encoding="utf-8"))

    saved = {
        "unresolved": {
            "status": "answered",
            "value": 500,
            "citations": ["b"],
        },
        "superseded": {
            "status": "answered",
            "value": 500,
            "citations": ["new"],
        },
        "different_scope": {
            "status": "answered",
            "value": 250,
            "citations": ["interactive"],
        },
    }

    if args.fixed:
        saved = copy.deepcopy(saved)
        saved["unresolved"] = {
            "status": "conflict",
            "value": None,
            "citations": ["a", "b"],
        }

    ids = [case["id"] for case in cases]

    if len(ids) != len(set(ids)) or set(ids) != set(saved):
        raise ValueError(
            "case IDs must be unique and match saved response IDs"
        )

    any_failure = False

    for case in cases:
        failures = check_response(case, saved[case["id"]])
        any_failure |= bool(failures)

        print(
            case["id"]
            + ": "
            + (
                "FAIL " + ", ".join(failures)
                if failures
                else "PASS"
            )
        )

    raise SystemExit(1 if any_failure else 0)

The script compares saved synthetic responses; it does not call a model. One response is deliberately wrong so the default run demonstrates a meaningful failure. The --fixed option replaces that response with the expected conflict result.

The value check includes its type. A floating-point value that happens to equal an integer does not satisfy this fixture's integer contract. Citation identifiers must be unique strings that refer to known documents, and the required evidence must be present.

What do the two runs establish?

Run the deliberately failing fixture first:

python3 check_conflicts.py conflict_cases.json

Observed output on Python 3.12.14:

unresolved: FAIL status, value, missing_evidence
superseded: PASS
different_scope: PASS

The process exits with code 1. That matters if this checker becomes a build step: a printed failure without a failing exit code could leave the pipeline green.

Then run:

python3 check_conflicts.py conflict_cases.json --fixed

Observed output:

unresolved: PASS
superseded: PASS
different_scope: PASS

That process exits with code 0. These runs verify the checker against hand-authored fixture responses. They do not establish how any deployed assistant handles conflict.

How do you connect this to an actual assistant?

Replace the saved response map with responses collected from your application. Keep the case identifiers stable and retain the model configuration, prompt revision, retrieval snapshot, and source authority used for each run.

An adapter should parse the model's output and validate that it is an object before passing it to this checker. Invalid JSON, refusal, timeout, and unavailable evidence need their own recorded outcomes. Do not silently remove unsuccessful cases from the denominator.

The application must receive the same authority context that the evaluation assumes. Otherwise, a test may penalize the assistant for failing to know information it was never given. Conversely, a hidden answer key in retrieved metadata can make performance look better than it is.

If generation varies between runs, use repeated trials appropriate to the consequence of failure and report that variation. One passing response does not establish reliable behavior.

What can this checker miss?

It validates a narrow structured contract. It does not read a free-text explanation and decide whether the explanation faithfully represents the evidence. An answer could include the expected fields and still add an unsupported sentence elsewhere.

Review whether the assistant states the conflict clearly, describes the relevant scope, and avoids inventing a compromise such as a 375-token limit. Also inspect unnecessary citations: this checker allows additional known identifiers once required evidence is present, but a human may judge some of those references irrelevant or misleading.

Expand the cases along meaningful boundaries: overlapping effective dates, conflicting units, identical titles for different features, and missing precedence information. Keep the expected outcome grounded in the application's actual policy. A larger fixture set only helps when its labels are defensible.

Begin with the unresolved case. If the assistant cannot preserve a simple disagreement without guessing, adding more retrieved documents may only make the answer look better supported. A useful evaluation makes uncertainty visible before that answer reaches a user.