Since May I have asked answer engines with web search the same 16 questions about a new author, on a fixed three-day grid, and scored each answer by fixed rules from -3 to +3. On 8 October 2026 that author's first novel came out. For the first time there was evidence about him that third parties publish in machine-readable form: retailer catalogues, library records, ISBN registers.
Before the release I registered a hypothesis (pre-registration Q7): search-grounded models pick up that new evidence, and they do it earlier in direct answers (questions that name the author or the book) than in recommendations (genre questions that name neither). The null hypothesis: no difference between the windows. Report 04 is the test.
Design in short
Primary analysis on OpenAI Search and Gemini, the two providers whose setup had not changed since 19 August. Baseline 8 September to 7 October, effect window 8 October to 7 November, anchor days only. The headline is the score sum over the maximum across all 16 questions, as the mean of both providers. Because the series was already drifting upward before the launch, the comparison is drift-adjusted: a line fitted on the anchor days before the event is extended into the effect window, and only the distance to that line counts. Intervals come from a bootstrap over anchor days with 10,000 draws and a fixed seed.
Result
Drift-adjusted, the headline rose by +1.38 percentage points, 95 % interval -3.59 to 6.68. The interval includes zero, so the null hypothesis is not rejected.
No channel moves either:
| Channel | Drift-adjusted difference | 95 % interval |
|---|---|---|
| Headline, 16 questions | +1.38 | -3.59 to 6.68 |
| Direct (D1 to D3) | -0.19 | -15.51 to 15.53 |
| Saga knowledge (L1, L2) | +0.37 | -20.94 to 22.34 |
| Recommendation, 10 questions | +2.27 | -3.02 to 8.11 |
| of which GR1 to GR6 | +3.79 | -5.42 to 13.48 |
The order hypothesis gets no support. Neither the direct channel nor the recommendation channel shows a first deviation up to the last anchor day, 5 November.
The number I am not claiming
Without drift adjustment the difference is +3.38 points, interval 0.05 to 6.93. That lower bound sits just above zero, and it is tempting to call it a rise. It does not hold up: with 200 further bootstrap seeds the lower bound lands above zero 157 times. A result that depends on the seed of the random generator is not a finding, and the pre-registered analysis is the drift-adjusted one anyway.
What limits the test
- Gemini ran out of credit and is missing from 24 October to 5 November, so it covers 5 of 10 effect days. OpenAI is complete.
- The design can detect a headline shift of about 4.7 points as a planning value; with the drift uncertainty and the days actually measured it is 7.3. Smaller effects are below what this study can see.
- n is one author with no comparison group. Anything else that changed in the providers' search systems during October is in the same window.
- Claude on claude.ai is reported descriptively only: 4.86 to 9.03 percent on the 10 anchor days, and no recommendation named the author on any day.
- Empty answers from Gemini were scored as zeros before 1 November. Treated as gaps, the headline effect is +1.70 (-3.27 to 6.96). Same decision.
Extending the window to 8 November, the first day Gemini was measured again, is not pre-registered and labelled as such: +2.32 (-2.74 to 7.95). The decision rests on the main analysis alone. A delayed effect is watched until 6 January 2027 and will come as an addendum.
Data and code
Papers in English and German: doi.org/10.5281/zenodo.24063460. Dataset: marintkael/ai-citation-fidelity. Analysis code, the frozen data pulls with checksums and the figure build: marin-research-tools/reports/04-book-launch. Every deviation from the plan is listed in section 8 of the report.
Top comments (0)