DEV Community: Marin T. Kael

A run that reported 48 of 48 complete and measured nothing

Marin T. Kael — Tue, 15 Sep 2026 11:23:03 +0000

A measurement run of mine finished on 12 September, reported 48 out of 48 complete, wrote its day flag, and measured nothing at all. Every stage in the chain said it had worked. It took me two days to notice, and only because a number downstream looked lonely.

Here is the setup, because the shape matters more than my particular pipeline. I ask a fixed set of 16 questions against several answer engines on a fixed schedule, and score whether a given entity shows up in the answer. One engine per run, one row per question and model pair, 48 rows for that engine. The run writes rows, a scorer reads them, a day flag marks the date as done so nothing re-runs it.

What the rows actually contained

All 48 rows came back carrying a throttling marker instead of an answer. Not one row held a response. Not one had triggered a web search. Response times were one to three seconds, which for a search-grounded answer is physically too fast to be real work.

So the evidence that the run had failed was sitting inside the data, in three independent places, and nothing looked.

Three stages reported success

The day log said 48/48 ... DONE, flag set. The run log said scored 48 rows. The day flag was written.

Every one of those statements is true. They are all counting rows. None of them asks whether a row carries a value. A throttled response is a row. An error string is a row. scored 48 rows means the scorer visited 48 rows and assigned each one a score, and the score it assigns to a row with no answer in it is zero, which is also a perfectly valid score for an entity that genuinely was not mentioned.

That is the whole trap in one sentence. A failed measurement and a real zero are the same shape once you are only counting.

Why this class of bug is expensive

A crash is cheap. You see it, you fix it, you re-run. This one is expensive because it is silent and it is durable:

The day was marked done, so the retry logic will never touch it again. The zero entered the aggregate as a real observation, so the headline number for that day rests on two engines instead of three, and nothing in the published figure says so. Anyone reading the trend sees a dip and looks for a cause in the world. The cause was in the collection.

I have now spent time twice looking for an explanation on the wrong side of the instrument. The first time it was a different engine reading zero for several days and I went looking for what had changed in the engine. Nothing had. My own fetch had.

What I changed

Three things, in order of how much they were worth.

First, a run is only complete if the rows carry values. The completion check now asks how many rows hold an actual answer, not how many rows exist. Zero answers means the run failed, regardless of what any log says, and the day flag is not written.

Second, a gap is a value in its own right. There is now a gap register with an entry per missed measurement, with the reason and a snapshot as evidence. The aggregate reads it, so a day built on two engines instead of three is marked as such rather than quietly averaged.

Third, an empty result from a failed fetch never becomes an empty list. That pattern is where these bugs live: catch the HTTP error, return nothing, and the caller computes over nothing and reports serenity. Empty and failed have to be different return values, because downstream they mean opposite things.

The general rule

If your pipeline can produce a zero, ask it which kind of zero it is. Not measured, measured and absent, and failed while measuring are three different states, and most instrumentation flattens them into one number because one number is easier to chart.

The check that would have caught mine is two lines long: did any row in this run carry a value, and if not, fail loudly. I did not write it earlier because the run had never failed that way before. That is not a reason, that is just the date on which you learn it.

Your nearest neighbour join is symmetric. Your time series is not.

Marin T. Kael — Mon, 14 Sep 2026 13:38:33 +0000

Three AI search engines get the same 16 questions every day. Two of them answer every day. The third runs every three days, and that difference is enough to make a published number change after it was published.

Here is the shape of it, because the shape is more common than the specific case.

The setup

A daily aggregate, call it combined_primary, is the mean of three provider scores: OpenAI, Gemini and the Claude web interface. The first two write a row every day at 04:00 UTC. The third writes a row every third day at 06:34 UTC, which is after the aggregation run of its own day.

So the aggregator cannot simply join on the date. On most days there is no row for the third provider, and on the days there is one, it arrives two and a half hours too late. The obvious fix, and the one I shipped, is to attach the nearest measurement within 48 hours:

const nearestClaude = rows
  .filter(r => Math.abs(r.ts - day) <= 48 * 3600)
  .sort((a, b) => Math.abs(a.ts - day) - Math.abs(b.ts - day))[0];

That works. It also has a property I did not think about for eleven days: Math.abs has no sense of direction. Nearest means nearest in either direction, so a point can borrow a measurement from its own future.

What that looks like from outside

On the 8th I read the aggregate for the 8th and wrote it down: 21.9.

On the 9th I read the aggregate for the 8th again. It said 23.1.

	OpenAI	Gemini	Claude web	mean
read on the 8th	29.2	24.0	12.5 (measured on the 6th)	21.9
read on the 9th	29.2	24.0	16.0 (measured on the 9th)	23.1

No measurement changed. No row was deleted. No backfill ran. The only thing that changed is which row is nearest to the 8th, and that changed because a new row appeared on the 9th.

This is worse than a wrong number. A wrong number is wrong consistently, and someone eventually notices. This one is correct at every moment and different at every moment, which means two people reading the same chart on two days disagree and neither of them made a mistake.

Across the last 30 days: 3 points carry a measurement from their own future, 7 carry one from the past by up to two days.

Why the test suite was quiet

Every check I had was a check on values. Is the row present, is the number in range, does the mean match its components, are the providers the ones we expect. All of them passed, on both days, on the same point, with different numbers.

Nothing checked provenance: which rows went into this point, and when were they written relative to the point's own day. That question was not expressible in the test suite, so it was never asked.

The check that closes it is small and worth writing out, because the useful part is the counterfactual:

for each published point p:
    rows   = contributors(p)
    future = [r for r in rows if day(r) > day(p)]
    stale  = [r for r in rows if day(p) - day(r) > 1]
    alt    = mean of rows excluding future
    flag if future or stale, and report both p.value and alt

Reporting alt next to the published value is what makes the finding actionable instead of interesting. For the 8th it prints 23.1 against 21.9, and my own note from the previous day is the independent witness that the 21.9 was once real.

Before the check: 0 points flagged as carrying a future measurement, 0 as carrying a stale one, 0 anchor days without an own measurement. After: 3, 7 and 1. Those numbers did not get worse. They got visible.

The rule I would give myself eleven days earlier

If your series is published, any join that reaches across time must be one directional. A point may look back, never forward. The cost is real: a point with no measurement of its own gets a staler value, or none at all, and the provider base of the chart becomes uneven. That is a visible cost, and a visible cost is the kind you can argue about.

The invisible cost is a chart that rewrites its own past every time a slow source catches up. Symmetry is an attractive default in a distance function. In a time series it is a quiet way of leaking the future into the past.

The underlying measurement runs daily and the series is public, so the flagged points are still there to look at.

Never document a live metric next to a stable constant

Marin T. Kael — Tue, 08 Sep 2026 14:22:11 +0000

I run a small measurement pipeline. One fixed set of 16 questions, asked once a day at 04:00 UTC against two answer engines, 32 data points per run, a single entity. It has been running for months and each day's result is published through an endpoint.

Six days ago I wrote a short reference file describing that setup. The reason was mundane. I had noticed the setup being described inconsistently wherever I quoted it: two engines in one place, three in another, five endpoints in a third. So I wrote the ground truth down once, in one file, and pointed everything at it.

Today I read that file against a live run. Every number in it was wrong.

Not the structure. The structure was fine. 16 questions, 2 engines, 32 points, 04:00 UTC. That is a property of the setup and it only changes when I change the setup.

What was wrong was everything I had helpfully written next to it.

documented 02 Sep   headline 28.6%   direct 88.9%   engine A 27.1%   engine B 30.2%
measured   07 Sep   headline 23.4%   direct 83.3%   engine A 22.9%   engine B 24.0%

Those are not typos and nothing broke. They were correct on the day I wrote them. They are readings, and a reading has an expiry date that a constant does not.

Why this is not just carelessness

The file was written to solve a drift problem and it produced the next one. That is the part worth generalising.

A constant and a reading look identical in prose. Both are a name, a colon and a number. Nothing in the format tells a later reader which of the two they are looking at, so the file gets quoted as a whole, and the reading gets quoted with the same confidence as the structure. The document is most dangerous exactly when it is most trusted.

A stale reading also fails quietly. A wrong constant contradicts something else and someone eventually trips over it. A stale reading is just a number that used to be true. Nothing anywhere asserts that it should still be true, so nothing can catch it.

What I changed

Three things, none of them clever.

Split by lifetime, not by topic. The file now holds only what changes when the setup changes. Anything that moves on its own schedule was taken out.

Make the reading unquotable from the document. There is now one command that prints the current values, and nothing else prints them. If you want a number, you fetch it.

Fail closed on age. The reader checks the snapshot timestamp. Past 36 hours it stops printing values and says so instead. An old number is not a smaller number, it is an unknown one, and those two have to look different at the point of use.

The rule

If a value in your documentation would be different tomorrow without anyone editing the file, it is not documentation. It is a cached read with no invalidation and no timestamp, sitting in the exact place people go when they want to be sure.

Keep the structure in the file. Keep the reading behind a call.

Named vs described: an entity can be 89 percent citable and still invisible

Marin T. Kael — Wed, 02 Sep 2026 07:11:58 +0000

I run a fixed set of questions against answer engines every morning and record whether one specific entity gets cited. Today's run produced the cleanest split I have seen so far, and it is not the number I usually report.

The headline was 28.6 percent. That number is close to useless.

The setup

Sixteen questions, frozen wording, two engines (OpenAI's search model and grounded Gemini), one run per day at 04:00 UTC. Thirty-two datapoints per run. The questions are grouped into categories before the run, not after, so the grouping cannot be fitted to the result.

Today, run 1d1b24d7, snapshot 04:07:09 UTC:

Category	n	cited
Direct (question names the entity)	6	88.9%
LongTail	4	66.7%
GenreRecommend	12	8.3%
Research	2	8.3%
Genre	4	0%
CompCluster	4	0%

Per engine: OpenAI search 27.1 percent, Gemini 30.2 percent. The two engines land within three points of each other, which is worth noting on its own, because it suggests the split below is not one vendor's quirk.

What the average hides

The overall 28.6 percent reads like a middling result you could improve by a few points with more work. The breakdown says something different. There are two separate regimes here.

When the question already contains the entity name, the engines find it almost every time. Nearly nine in ten. Retrieval is not the problem, indexing is not the problem, and the knowledge graph work has clearly landed.

When the question describes a need instead of naming a thing, the same entity is cited in roughly one in twelve answers, and in the two hardest categories it is zero out of eight. Those are the questions a real person actually asks. Nobody types the name of something they have not heard of.

So the average is the arithmetic mean of a solved problem and an unsolved one. Moving it from 28.6 to 32 could mean the described-need case improved, or it could mean the named case got slightly better at something it was already doing. Reported as one number, you cannot tell, and you will spend your effort on whichever is easier to move.

Why I think this generalises

Not because of my n. Because of what the two regimes are made of.

The named case is a lookup. The engine has a string, it resolves the string, it answers. Everything that helps there is entity infrastructure: a stable identifier, consistent naming, a canonical page that says plainly what the thing is.

The described case is a selection. The engine has a slot with room for three or four candidates and a population of thousands competing for it. Nothing about your own pages decides that, because the comparison happens somewhere else entirely, in whatever material the engine is drawing on to build its shortlist. That is why the same work that took the named case to 89 percent did nothing at all for the other one. They are not the same task wearing different hats.

Limits, honestly

Small n per category, between 2 and 12 datapoints. One run, one day, one entity, two engines. I am not claiming 8.3 percent is a stable value, and by tomorrow it will have moved. The finding I am willing to defend is the shape, not the numbers: the gap between the named and the described case is far too large to be sampling noise, it has been there in every run I have looked at, and both engines show it independently.

The practical version, if you measure this for anything: split your prompt set by whether the prompt contains the name, and report the two lines separately. If you report one average, you are averaging over the only distinction that tells you what to do next.

Live figures: marin-t-kael.de/en/research/dashboard

I asked three AI search engines the same 16 questions for 99 days. They learned my name. They never recommended me.

Marin T. Kael — Wed, 19 Aug 2026 16:11:47 +0000

Deutsche Fassung dieses Berichts: marin-t-kael.de/research/berichte/03-gefunden-nicht-empfohlen

Three months ago I published the first report from this experiment (Zero to Cited in Six Days): a pseudonymous fantasy author goes online on day zero with a website, a Wikidata item and a book that does not exist yet, and every morning a Cloudflare Worker asks OpenAI Search, Gemini (Google grounding) and Claude (web search) the same sixteen questions and scores the answers from -3 (hallucination) to +3 (full, sourced citation).

That first report covered 24 days and found that the entity became visible within a week. This one covers 99 days.

The headline number stalled in July

Since early July the combined citation rate sat between 19 and 23 percent. The question set has been frozen since June; adding questions, changing providers or adjusting the rubric would have started a new series instead of continuing this one.

Instead I split the sixteen questions the way they were always meant to be split:

Direct (3 questions): the author or the book is named in the prompt. "Who is Marin T. Kael?"
Saga knowledge (2): only terms from the work are named. "Which saga is set in a city called Varin?"
Recommendation (10): nothing is named. "Recommend me German fantasy like Robin Hobb." This is what readers actually type.

Three channels, three different curves.

Direct: a ceiling

The direct channel goes from 10 percent in the first three weeks to 71 percent from day 49 and stays there. A logistic fit gives a ceiling of 72.5 percent, inflection at day 37, R² 0.70. The last two phases (71.2 and 70.4 percent) have overlapping confidence intervals. What remains below 100 percent: answers without a source, a collision with an institution of the same name, and days on which one provider finds nothing.

The dip below zero around day 20 is a Wikidata deletion discussion: for a week the engines emphatically denied the author existed. The item was re-created; the curve recovered.

Saga knowledge: one provider is carrying it

The second channel grows from 10 to 34 percent and then falls back to 24. The fall is real (CIs 33 to 45 vs 21 to 32) and it has a cause: since July, OpenAI Search answers the two saga questions at 65.5 percent, Gemini at 23.1, Claude at 7.7. The pages that lead to Varin and edict magic are in the Bing index. When the OpenAI API quota ran dry twice in August, the channel fell with it.

Recommendation: 40 out of 6,729, and 25 of them from one question

Across 99 days the author was named in 40 of 6,729 blind answers, 0.59 percent. For an unpublished author that is the expected floor. The distribution by question:

25 of the 40 mentions come from a single question: "Edict fantasy: which works exist in this sub-genre?" That is a term the author coined himself, and whoever searches for it finds only him. Ten more come from "literary German fantasy appearing in 2026". Zero from "like Robin Hobb" (701 answers), zero from "for readers of Robert Jackson Bennett" (695), zero from "intellectual fantasy with system depth" (650).

Similarity questions need someone else to have drawn the comparison first, in a review, a list, a forum thread. Vocabulary questions only need the term to be in the index. For a new author, after 99 days, only the second is true. I am calling this recommendation through owned vocabulary, provisionally, because n = 1 cannot tell me whether it generalises. In this dataset, after 99 days, a sub-genre term produced more recommendations than all similarity-phrased questions combined.

Two bugs I found while writing this

Both are in the public pipeline code and both changed numbers that were already on the dashboard, so they belong in the report.

A gap is not a zero. The OpenAI account ran out of credit twice. The pipeline wrote error rows with score 0, and the live aggregation counted them as data points. A provider that had not answered entered the mean as "0% cited" and pushed the combined value from ~17 to 11 percent on 19 August. Fixed: error rows are excluded, an absent provider is shown as unavailable, both gaps are in the public gap register.
\w does not match umlauts. The filter that turns "I have no reliable information about this title" into a zero missed the German variant because zuverlässigen contains an ä and the regex ran without the Unicode flag. 27 honest non-answers were scored as partial knowledge. Fixed and rescored; the three main channels never included those rows, the control channel did.

The control channel: 6,416 answers, zero mentions

Ten models without web access (Claude without search, Llama 3 to 3.2, Mistral, Phi-2, GPT-4o-mini without the search tool) answered the same blind questions 6,416 times. The author was never named. Everything the web-grounded engines say about him comes from live retrieval, nothing from training data. The first unprompted mention by a no-web model would mark the day the entity has reached training data; it has not happened.

What changes now

Day-to-day autocorrelation of the direct series is -0.08. Daily measurement is noise around a stable level, so from 19 August the programme measures every three days; same questions, same providers, same rubric. And the dashboard headline is no longer one averaged number but three: 64% direct, 17% saga knowledge, 0% recommendation.

On 22 September the book is actually published. Same instrument, second phase. The baselines are now frozen in the data; whether the ceiling moves, whether Google and Claude catch up on saga knowledge, and whether "like Robin Hobb" ever produces a hit once reviews exist, are the three questions for the next report.

Data and code

Dataset (CC BY 4.0, 5 configs, frozen 19 Aug): huggingface.co/datasets/marintkael/ai-citation-fidelity
Analysis + figures: github.com/marintkael/marin-research-tools/reports/03-found-not-recommended
Full report (DE/EN): marin-t-kael.de/en/research/reports/03-found-not-recommended
Live dashboard: marin-t-kael.de/research/dashboard

Four green Mondays that wrote nothing: a uuid above the first step

Marin T. Kael — Tue, 18 Aug 2026 15:13:56 +0000

The symptom

For four Mondays in a row my data pipeline finished with complete, stages_ok: 16, stages_failed: 0. And the run row in the database stayed empty: duration_ms was NULL, no summary written.

Every downstream reader filters on duration_ms IS NOT NULL, sensibly, to skip fragmented runs. So those runs did not show up as broken. They did not show up at all. The public time series just had holes in it, and nothing anywhere used the word error.

The only clue was in a later stage: a foreign key violation. It was writing against a run that did not exist.

The cause

The pipeline is a Cloudflare Workflow. Reduced to the shape that matters:

export class ResearchPipeline extends WorkflowEntrypoint {
  async run(event, step) {
    const runId = crypto.randomUUID()               // <-- here

    await step.do('create-run', () => insertRun(runId))
    await step.do('collect',    () => collect(runId))

    if (event.payload.weekly) {
      await step.sleep('cool-off', '5 minutes')     // <-- and here
    }

    await step.do('finalize-run', () => finalizeRun(runId))
  }
}

step.sleep does not block. It ends the invocation. When the timer fires, the engine calls run() again from the top and replays it. Anything inside a step.do hands back its cached result without executing. Anything outside a step runs again, for real.

So crypto.randomUUID() ran a second time, and after the sleep runId held a value nothing had ever been written under. create-run did not re-execute, so no row existed for the new id. finalize-run then updated a row that was not there.

Why it hid so well

UPDATE ... WHERE id = ? matching zero rows is not an error. It is an ordinary result. D1 reports success, meta.changes is 0, and if you never read meta.changes you cannot tell a write from a no-op.

Put that next to a reader that filters incomplete rows out and you get a very quiet failure. The writer believes it wrote. The reader believes there is nothing to read. The monitoring shows green.

It only happened on Mondays because only the weekly branch carried the sleep. That turned out to be the most useful part of the fingerprint: a bug that appears on exactly one weekday is pointing at the branch that only that day takes.

The two rules I took out of it

First, in any environment that can replay your function, everything identity-bearing or time-bearing belongs inside a persisted step. Uuids, Date.now(), anything random. Computed above the first step, you get a fresh one on every replay.

const runId = await step.do('run-id', async () => crypto.randomUUID())

Second, a write that reports success is not the same as a write that happened. Check the row count and throw on zero.

const res = await db.prepare('UPDATE runs SET ... WHERE id = ?').bind(runId).run()
if (res.meta.changes === 0) throw new Error(`finalize hit 0 rows for run ${runId}`)

The second rule is the more general one. The first only bites inside workflow engines. The second bites everywhere, and it is what turned four silent Mondays into a loud error on the fifth.

Still open

The older runs are still holes. Their stage data sits in the database, the run row does not, and for the oldest two the workflow instance history is past retention, so the duration cannot be reconstructed. I am recording those as documented gaps instead of estimating them. A gap you can see is worth more than a number you made up.

Zero to Cited in Six Days: I Put a Brand-New Author Online and Measured How Fast AI Started Citing Him

Marin T. Kael — Fri, 05 Jun 2026 09:13:44 +0000

By Marin T. Kael — independent researcher, AI Citation Behaviour Lab. Published 5 June 2026. Open data, code, and the full bilingual report are linked at the end.

TL;DR

On 11 May 2026 I put my public author presence online for the first time and instrumented the launch. A pseudonymous fantasy author — me, in a sense — with no prior web presence, no published book, nothing for a search engine or a language model to have ever seen. Then I pointed five web-grounded LLMs at him every day for 23 days and scored roughly 16,000 answers for whether they cited him correctly, missed him, or hallucinated.

The headline number: the first correct LLM citation landed on day six. Six days from a cold start to "yes, this entity is real and here is what it does," fetched live from his own website.

But the headline is the least interesting part. The interesting part is everything that number hides — a locked front door, a chasm between providers that masquerades as a capability ladder, and the discovery that going viral on Reddit bought me exactly zero AI citations.

This is not a success story. It is a measurement. Here is what I measured.

Why I did this to myself

There is a lot of confident writing about "AI SEO" and "getting cited by LLMs," and almost none of it is controlled. Brands optimize a hundred things at once on domains that are already years old, then attribute whatever moves to whatever they shipped last. You cannot learn causality from that. The signal is buried under a decade of accumulated authority.

So I built the cleanest natural experiment I could afford: a single brand-new entity, born on a known date, with zero prior footprint. If an LLM can suddenly describe him, the cause has to be something that happened after his birth — and I logged everything that happened after his birth.

The subject is "Marin T. Kael," a pseudonymous author whose debut novel Das vierte Feld (series: Prägungen des Reiches) is scheduled for 22 September 2026. The book is real and forthcoming; the public entity was the instrument. The honest catch — and I will keep saying this — is that this is a single-subject design. n = 1. The investigator is the subject. I will come back to why that is less fatal than it sounds, and exactly where it limits the claims.

The design (pre-registered, so I couldn't move the goalposts)

I wrote the protocol down and timestamped it before collecting data. That matters more than it looks, because when you are both the experimenter and the thing being experimented on, the only thing standing between you and motivated reasoning is a pre-commitment you can't quietly edit later. The full failure log is public for the same reason.

Surfaces: 5 web-grounded LLM endpoints.
Instrument: 16 standardized questions across 6 categories (direct identity, biographical detail, work/series, genre discovery, recommendation, and disambiguation controls).
Cadence: polled daily for 23 days.
Volume: ~16,000 scored datapoints.
Scoring: every answer got +1 (correct and source-grounded), 0 (entity not found), or −1 (hallucinated — confidently wrong).

That −1 is the whole game. Most "AI visibility" tools count mentions. A mention that invents your biography is not visibility; it is a liability with your name on it. I wanted a metric that punishes confident fiction as hard as it rewards truth, because from where I sit those are not close to equivalent.

Finding 1 — Speed: six days

Figure 1 — Daily citation score per surface across the 23-day window; vertical markers at T+4 (Google Knowledge Graph entry) and T+6 (first correct LLM citation).

The Google Knowledge Graph picked him up on day four (T+4). The first correct, source-grounded LLM citation followed on day six (T+6).

I want to be careful about what "six days" means, because it is easy to over-read. It does not mean every model knew him in six days — most never reliably did. It means the fastest path from non-existence to a correct, grounded citation in a major web-grounded LLM is measured in days, not months, when the structured-identity scaffolding is in place from day one. The Knowledge Graph led; the LLM followed two days later. Hold that ordering — it recurs.

Finding 2 — The locked door

Here is the result that reframed the entire study for me.

Cloudflare returned HTTP 403 to every AI crawler on 22 of the 23 days. Not because I configured it that way — because that is the silent opt-out default for new domains now. The front door to my own website was bolted shut against the exact bots I was trying to reach, and I didn't know until I read the logs.

And the entity became AI-visible anyway.

Figure 2 — Two-track diagram: on-site crawl path (blocked, 403 on 22/23 days) versus the path that actually worked — Knowledge Graph / Wikidata plus inference-time grounding on third-party mentions.

If crawlers couldn't read his site, how did a correct citation appear on day six? Two paths, neither of which is "crawl the homepage":

The Knowledge Graph (via Wikidata) — structured identity that propagates without anyone scraping your HTML.
Inference-time grounding on third-party mentions — the model fetches and reads pages about the entity at query time, from places that are not your locked-down domain.

The lesson is uncomfortable for the standard "optimize your on-site content for LLMs" playbook: a brand-new entity got cited while its own site was returning 403 to every AI bot for 96% of the study. On-site crawling was not the channel. Structured identity and off-site presence were.

The honest limit, stated plainly: because the crawlers were blocked the entire time, this study can say nothing about whether llms.txt, on-page answer-block formatting, or any other on-site optimization works. I never got to test them — the door was shut. If someone tells you their llms.txt moved their AI citations, ask them to prove the crawler ever made it through the door. Mine didn't, and I still got cited.

Finding 3 — The provider chasm (which is not a capability ladder)

This is the finding I most want people to internalize, because it kills a comfortable mental model.

The intuitive story is a ladder: smarter, newer models cite more reliably; weaker ones cite less. Tidy. Wrong. What I measured is a chasm — a discontinuity that tracks which sources a provider retrieves from, not how capable the underlying model is.

Precision here = correct-to-hallucinated citation ratio over a rolling 7-day window. Higher is better; below 1.0 means the model hallucinates about the entity more often than it gets it right.

Figure 3 — Precision (correct : hallucinated) by surface, 7-day window. Dashed line at 1.0 = break-even.

Surface	Precision (correct : hallucinated)	Read
OpenAI GPT-5.4 (web)	4.7 : 1	reliable
OpenAI Search API	4.0 : 1	reliable
OpenAI GPT-5.2 (web)	1.9 : 1	positive but weaker
Gemini	0.47 : 1	net-negative — hallucinates ~2× as often as it's right
Claude	~5% cite rate	rarely cites, but abstains rather than confabulating*

* A word on Claude, because my own instrument nearly got it wrong. The automated scorer first flagged Claude as net-negative — more "hallucinations" than correct citations. When I read the flagged answers by hand, the scorer was wrong: the single most common "hallucination" was Claude correctly pointing out that Das vierte Feld is already a real 1999 book by Mokka Müller and declining to invent a brand-new author for it, and others were Claude disambiguating the name collision with the Maritime Research Institute (MARIN). A validated re-analysis (manual adjudication of n=50, Cohen's κ=0.79; classifier recall 100%; book claims web-verified) put genuine errors at only ≈11% (95% CI 7–16%) — all low-severity ("Marin" read as "marine") — with zero fabricated author biographies. So Claude rarely surfaces this brand-new entity (~5%, about what you'd expect for a days-old identity) but it abstains or correctly disambiguates rather than confabulating. On the corrected read it was the most honest model in the study. Gemini's net-negative result, by contrast, is genuine and verified — which is exactly the asymmetry the construct-validity section is about.

Two things to sit with.

First, within one provider, the newer generation cited more reliably. GPT-5.4 (4.7:1) cleanly beats GPT-5.2 (1.9:1). So model generation does matter — within a retrieval stack.

Second, and bigger: the gap between providers dwarfs the gap between generations, and it isn't about raw capability. The mechanism is retrieval-source divergence. I traced where each provider actually pulled the entity from:

OpenAI grounded on the entity's own domain 119 times. It went to the source.
Gemini grounded on that domain 0 times. It pulled the entity exclusively from Reddit — 17 out of 17 retrievals. One source. A community forum.

That is the whole story of the chasm. Gemini isn't "worse at reasoning" here; it is looking somewhere else. When your only window onto an entity is Reddit threads, your description of that entity is whatever Reddit happened to say — which is why Gemini's precision sits underwater. Same entity, same questions, same week. Different door.

One more number, because I've seen this misquoted already: the OpenAI-web citation rate plateaus around ~10%, with peaks to 16.3%. It is not 18%. If you see 18% attributed to this study, it's wrong.

Finding 4 — Depth: when it finds him, it really finds him

The flip side of OpenAI's reliability is how complete the description is when it lands.

Figure 4 — Annotated screenshot of a correct OpenAI answer; the fetched source URL carries utm_source=openai, confirming a live retrieval at inference time.

Where OpenAI found the entity, it didn't just confirm he exists. It returned the series name, the setting, the release date — and even the pseudonym status and the existence of this research project. Complete and source-grounded. And I can prove it was fetched live, not recalled from training: the source URL it pulled carried utm_source=openai, a tag that only exists because the model went and got the page at query time.

So the depth ceiling is high. The problem is never "the answer is shallow." The problem is whether you get an answer at all — which is Finding 6.

Finding 5 — The needle mover (and the thing that did nothing)

I ran two interventions against each other, and the result rearranged my priors.

Structured identity moved the needle. Social reach did not.

The citation breakthrough — the day correct citations became real — was 17 May. Crucially, that came before I did any serious Reddit community-building. Then I built the social side: a 23× karma jump, from 12 to 281.

The citation lift from that 23× social surge was zero.

Figure 5 — Two time series on a shared axis: Reddit karma (12 → 281) versus correct-citation rate. The citation step-change precedes the karma climb; the karma climb produces no corresponding lift.

Read that carefully, because it's the most actionable thing here. What moved AI citations was the boring infrastructure: a Wikidata entry feeding the Knowledge Graph, a website, and DOIs — durable, structured, machine-legible identity. What did not move AI citations was virality. The karma climbed; the citation curve didn't flinch.

The clean takeaway: social virality buys human readers; structured identity buys AI citations. They are separate channels, and optimizing one does not subsidize the other. If your goal is "get cited by the model," karma is a vanity metric. If your goal is "get read by people," it isn't — it's just answering a different question than the one I was measuring.

Finding 6 — Cited when named, invisible when discovered

The last finding is the one that should temper any victory lap.

The entity is citable when you name him and effectively invisible when you don't.

Figure 6 — Hit-rate by question category: Direct "Who is…?" at 38.9% versus genre/recommendation at 0%.

Direct identity ("Who is Marin T. Kael?") → 38.9% hit-rate. Name him and there's a real chance the model knows him.
Organic discovery ("recommend a new fantasy author who…", "books in the style of…") → 0%. He never surfaces unprompted.
Organic search: per Google Search Console, 0 clicks and 0 impressions from organic search across the window.

This is the difference between retrieval and discovery, and it's the honest ceiling on the whole "zero to cited" story. A new entity can cross the recognition threshold — answer correctly when asked by name — long before it crosses the recommendation threshold, where the model volunteers it among peers. Six days bought me recognition. It did not buy me discovery. Those may be different timescales entirely, and nothing in 23 days let me see the second one move.

The part that earns the word "controlled"

I'll be blunt about why I think this design refutes the naive methods even at n = 1. Most "we got cited!" claims can't tell a real citation from an echo of the author's own marketing, and can't tell a correct citation from a confident hallucination. Three controls did that work:

Primary-vs-control channel separation, to catch echo bias — a model parroting my own copy back at me and calling it corroboration.
A fabricated-attribution catch: a model attributed the entity to "Wikipedia" 24 times — for an entity that has no Wikipedia page. That is a hallucination my scoring caught and a credulous mention-counter would have logged as a win. Twenty-four phantom citations to a page that does not exist.
Entity-collision controls: "MARIN" also denotes the Maritime Research Institute, so disambiguation was scored explicitly rather than assumed away.

None of this makes n = 1 into n = 100. It makes n = 1 honest. The two real mitigations are that the protocol was pre-registered (I couldn't retrofit the hypotheses) and the failure log is public (you can audit where it broke). What this is: a clean, reproducible measurement of one entity's path from zero to cited. What it isn't: a population estimate. Please don't cite it as one. I'm asking other people to run the same protocol on their own new entities precisely because my single data point can't carry that weight alone.

A genuinely recursive footnote

There's a strange loop worth naming. Gemini grounds its knowledge of this entity almost entirely on Reddit. This write-up will be posted and discussed on Reddit. Which means publishing this is itself the next experimental intervention — the act of writing about the measurement perturbs the thing being measured. I can't escape the observer effect here; I can only log it. Consider this paragraph part of the apparatus.

What I'd tell you to take away

If you're trying to be legible to AI systems — as a person, a product, an author, anything:

Build structured identity first. Wikidata → Knowledge Graph, a real website, durable IDs (DOIs/ORCID). That is what moved citations in days. It's unglamorous and it works.
Check your own front door. Your new domain may be returning 403 to every AI crawler by default and you'd never know without reading the logs. Off-site presence and the Knowledge Graph got me cited despite a shut door — but don't assume your door is open.
Know which channel you're optimizing. Virality → human readers. Structured identity → AI citations. Don't spend on one expecting the other.
Treat "cited" and "discovered" as different finish lines. Recognition-when-named came fast. Recommendation-when-unprompted never came at all in 23 days.
Don't trust mention counts. Twenty-four citations to a Wikipedia page that doesn't exist should end the practice of counting mentions and calling it visibility.

The full report, the ~16,000-row dataset, the scoring code, and the public failure log are all open. Run the protocol on your own new entity and tell me where my n = 1 holds and where it breaks. That's the only way a single subject becomes a finding.

Sources, data & code (open)

Full report (bilingual EN + DE, CC-BY) — the citable link: https://doi.org/10.5281/zenodo.20549020?utm_source=devto
Report page (web, live): https://marin-t-kael.de/en/research/zero-to-cited?utm_source=devto
Code (MIT): https://github.com/marintkael/marin-research-tools?utm_source=devto
Open dataset: https://huggingface.co/datasets/marintkael/ai-citation-fidelity?utm_source=devto
ORCID: https://orcid.org/0009-0006-2105-8190

I'm Marin T. Kael, an independent researcher working on AI citation behaviour — and, transparently, the pseudonymous subject of this study. The pre-registration and public failure log exist so you don't have to take my word for any of it.