Provenance is a testable property of an answer engine. I scored three of them.

Every AI search engine ships the same interface: a confident paragraph, then a row of source chips underneath. The paragraph is what you read. The chips are what you'd have to check.

I got curious about whether the chips are worth anything, so I ran one identical query through three engines inside the same hour and then wrote something to score what came back.

The query was what changed in the EU AI Act in 2026. It's a decent test because the answer is factual, recent, and — critically — has a primary source that either exists or doesn't.

Here's the part I didn't expect: all three agreed on the facts. Same substance, same two deadline dates. If you were grading on correctness, it's a three-way tie and there's no article in it.

The difference was entirely in what each was willing to show me as evidence.

Provenance as a scoring function

"Better sources" is the kind of claim that sounds unfalsifiable until you write it down. So I wrote it down: sort every cited host into a tier, weight the tiers, take the mean.

PRIMARY = {          # publishes the instrument itself
    "eur-lex.europa.eu",
}
OFFICIAL = {         # the issuing institution, one step from the text
    "commission.europa.eu",
    "digital-strategy.ec.europa.eu",
}
UGC = {              # user-generated, no editorial chain
    "youtube.com", "www.youtube.com", "reddit.com", "medium.com",
}

TIER_WEIGHT = {"primary": 3, "official": 2, "secondary": 1, "ugc": 0}


def tier(url: str) -> str:
    h = urlsplit(url).netloc.lower()
    if h in PRIMARY:  return "primary"
    if h in OFFICIAL: return "official"
    if h in UGC:      return "ugc"
    return "secondary"


def score(urls):
    """Mean tier weight, 0.0 (all UGC) to 3.0 (all primary)."""
    return sum(TIER_WEIGHT[tier(u)] for u in urls) / len(urls) if urls else 0.0

Two deliberate choices worth arguing with.

"Secondary" is the default, not a penalty box. A consultancy write-up explaining a regulation is a legitimate thing to cite. It is simply not the regulation.

The denominator is what's attributed, not what's listed. Engines display a sources panel and separately attribute specific sentences. Those are different sets, and only the second one is doing any work. Scoring the panel would let an engine pad its way to a good number.

The output

Pure stdlib, no arguments:

query: what changed in the EU AI Act in 2026   (captured 2026-08-25)

engine                           listed attributed  score  tiers
----------------------------------------------------------------
Perplexity (free)                    10          2   2.00  official:2
Google AI Mode                        5          5   1.00  official:1 secondary:3 ugc:1
Brave Search (free, no account)       1          1   3.00  primary:1

score = mean provenance weight of the sources behind the claims
        primary 3 | official 2 | secondary 1 | ugc 0

Perplexity listed ten sources and put European Commission domains behind the claims. It also gave the fullest answer by a distance — original deadlines, the amending regulation's effective date, the synthetic-content extension, the fine ceiling, in a table.

Google's AI Mode drew on five, and the attribution behind its load-bearing sentences was a consultancy, two sites that explain the Act rather than publish it, and a YouTube video. The Commission was present — in the panel, not behind the claims. That single ugc:1 is a video doing the work of a citation.

Brave, free and with no account, scored highest. One source, and it went straight at EUR-Lex for the regulation itself.

And then the scoring function turned out to be measuring the wrong thing.

A score is not a check

A tier map only reads the hostname. It has no idea whether the page exists. So the script has a second mode that goes and looks:

def resolve(url: str):
    """Return (status, body_bytes) or (None, reason). Follows redirects."""
    req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"})
    try:
        with urllib.request.urlopen(req, timeout=15) as r:
            return r.status, len(r.read())
    except urllib.error.HTTPError as e:
        return e.code, 0
    except Exception as e:
        return None, type(e).__name__

Note the last clause. An exception here is a result — "this citation could not be fetched, and here is what went wrong" — not something to swallow. A provenance checker that silently treats unreachable as fine is worse than no checker.

Running the three winners through it, on the day I'm writing this:

https://commission.europa.eu/          -> 200, 16,816 bytes
https://digital-strategy.ec.europa.eu/ -> 200, 54,683 bytes
https://eur-lex.europa.eu/             -> 202,      0 bytes

HTTP 202 with an empty body. Accepted for processing. Renders blank in a browser.

So the top-scoring citation in the table — the only primary in the whole dataset, the one that lifted Brave to a clean 3.00 — is a URL I could not read. Brave pointed at the instrument. It did not show it to me. Those are different claims, and only one of them is worth anything to someone trying to verify a number.

I've kept both modes rather than folding resolution into the score, because they answer different questions. The tier map asks what kind of thing did you cite. The resolver asks did you cite a thing that exists. An engine can pass either one alone and still be useless.

What I'd take from it as an engineer

One query is not a benchmark. Three engines, one question, one hour. It's an existence proof that the gap is measurable, not a ranking.

The interface gives you no signal that a sourcing decision was made. All three summaries look identical in confidence. Nothing renders differently when the chip under the paragraph is a government domain versus a YouTube video. That's a UI choice, and it's the one that does the damage.

If you're building on top of an answer API, provenance is a field you can compute. Hostname tiers are crude but they're cheap, deterministic and diffable — you can score every answer your product surfaces and alert when a run's mean provenance drops. That's a more actionable signal than "hallucination rate", because you can trace it to a specific citation and go look.

And check that the citations resolve. It took me forty lines and one surprise to learn that the highest-provenance source in my dataset was the one that returned nothing.

The full write-up, with what each engine actually returned and screenshots of all three answers, is in my comparison of the three AI search engines if you want the reader-facing version rather than the code.

The script is the whole thing — one file, standard library only, run it with python3 citation_provenance.py or add --resolve to hit the network. Swap in your own CITATIONS dict and it'll score whatever your engine of choice hands back.