DEV Community

Marin T. Kael
Marin T. Kael

Posted on Originally published at marin-t-kael.de

I asked three AI search engines the same 16 questions for 99 days. They learned my name. They never recommended me.

#ai #datascience #seo #opendata

Deutsche Fassung dieses Berichts: marin-t-kael.de/research/berichte/03-gefunden-nicht-empfohlen

Three months ago I published the first report from this experiment (Zero to Cited in Six Days): a pseudonymous fantasy author goes online on day zero with a website, a Wikidata item and a book that does not exist yet, and every morning a Cloudflare Worker asks OpenAI Search, Gemini (Google grounding) and Claude (web search) the same sixteen questions and scores the answers from -3 (hallucination) to +3 (full, sourced citation).

That first report covered 24 days and found that the entity became visible within a week. This one covers 99 days.

The headline number stalled in July

Since early July the combined citation rate sat between 19 and 23 percent. The question set has been frozen since June; adding questions, changing providers or adjusting the rubric would have started a new series instead of continuing this one.

Instead I split the sixteen questions the way they were always meant to be split:

  • Direct (3 questions): the author or the book is named in the prompt. "Who is Marin T. Kael?"
  • Saga knowledge (2): only terms from the work are named. "Which saga is set in a city called Varin?"
  • Recommendation (10): nothing is named. "Recommend me German fantasy like Robin Hobb." This is what readers actually type.

Three channels, three different curves.

Three question channels across 99 days

Direct: a ceiling

The direct channel goes from 10 percent in the first three weeks to 71 percent from day 49 and stays there. A logistic fit gives a ceiling of 72.5 percent, inflection at day 37, R² 0.70. The last two phases (71.2 and 70.4 percent) have overlapping confidence intervals. What remains below 100 percent: answers without a source, a collision with an institution of the same name, and days on which one provider finds nothing.

The dip below zero around day 20 is a Wikidata deletion discussion: for a week the engines emphatically denied the author existed. The item was re-created; the curve recovered.

Saga knowledge: one provider is carrying it

The second channel grows from 10 to 34 percent and then falls back to 24. The fall is real (CIs 33 to 45 vs 21 to 32) and it has a cause: since July, OpenAI Search answers the two saga questions at 65.5 percent, Gemini at 23.1, Claude at 7.7. The pages that lead to Varin and edict magic are in the Bing index. When the OpenAI API quota ran dry twice in August, the channel fell with it.

Recommendation: 40 out of 6,729, and 25 of them from one question

Across 99 days the author was named in 40 of 6,729 blind answers, 0.59 percent. For an unpublished author that is the expected floor. The distribution by question:

Recommendations by question

25 of the 40 mentions come from a single question: "Edict fantasy: which works exist in this sub-genre?" That is a term the author coined himself, and whoever searches for it finds only him. Ten more come from "literary German fantasy appearing in 2026". Zero from "like Robin Hobb" (701 answers), zero from "for readers of Robert Jackson Bennett" (695), zero from "intellectual fantasy with system depth" (650).

Similarity questions need someone else to have drawn the comparison first, in a review, a list, a forum thread. Vocabulary questions only need the term to be in the index. For a new author, after 99 days, only the second is true. I am calling this recommendation through owned vocabulary, provisionally, because n = 1 cannot tell me whether it generalises. In this dataset, after 99 days, a sub-genre term produced more recommendations than all similarity-phrased questions combined.

Two bugs I found while writing this

Both are in the public pipeline code and both changed numbers that were already on the dashboard, so they belong in the report.

  1. A gap is not a zero. The OpenAI account ran out of credit twice. The pipeline wrote error rows with score 0, and the live aggregation counted them as data points. A provider that had not answered entered the mean as "0% cited" and pushed the combined value from ~17 to 11 percent on 19 August. Fixed: error rows are excluded, an absent provider is shown as unavailable, both gaps are in the public gap register.

  2. \w does not match umlauts. The filter that turns "I have no reliable information about this title" into a zero missed the German variant because zuverlässigen contains an ä and the regex ran without the Unicode flag. 27 honest non-answers were scored as partial knowledge. Fixed and rescored; the three main channels never included those rows, the control channel did.

The control channel: 6,416 answers, zero mentions

Ten models without web access (Claude without search, Llama 3 to 3.2, Mistral, Phi-2, GPT-4o-mini without the search tool) answered the same blind questions 6,416 times. The author was never named. Everything the web-grounded engines say about him comes from live retrieval, nothing from training data. The first unprompted mention by a no-web model would mark the day the entity has reached training data; it has not happened.

What changes now

Day-to-day autocorrelation of the direct series is -0.08. Daily measurement is noise around a stable level, so from 19 August the programme measures every three days; same questions, same providers, same rubric. And the dashboard headline is no longer one averaged number but three: 64% direct, 17% saga knowledge, 0% recommendation.

On 22 September the book is actually published. Same instrument, second phase. The baselines are now frozen in the data; whether the ceiling moves, whether Google and Claude catch up on saga knowledge, and whether "like Robin Hobb" ever produces a hit once reviews exist, are the three questions for the next report.

Data and code

Top comments (0)