Last week I published a calibration study showing that my tool's GEO score does not predict whether Perplexity cites a page. The odds ratio was 0.9625, p = 0.7172. Nothing.
The study was pre-registered. I wrote the analysis plan before collecting the data, committed its hash to the repo, and ran the exact analysis I said I would run. No cherry-picking, no p-hacking, no "let me try a different model and see if it becomes significant."
This is how that worked and why I think every tool that sells a score should do the same.
What pre-registration means
Pre-registration is a commitment device. You write down what you are going to measure, how you are going to measure it, and what counts as a result, before you have any data. Then you collect the data and run the analysis you said you would. If the result is null, you publish a null.
In my case, the pre-registration was a markdown file in the repo. It specified:
- The sample: pages from sites that publish a sitemap, reachable, English-language
- The outcome: whether the page's domain appeared in Perplexity Sonar's sources for two question templates
- The primary analysis: logistic regression of domain citation on score/10, log rank, and log words
- The secondary analyses: each scoring category individually, with Holm correction
- The sensitivity analysis: excluding pages on sites that block AI bots
The file's SHA-256 hash is 441fcbb422964fdf2bf2b63eb02181ccbea9e7c5590d3e770399d567cd373ff8. You can verify it yourself:
sha256sum PREREGISTRATION.md
# should output: 441fcbb422964fdf2bf2b63eb02181ccbea9e7c5590d3e770399d567cd373ff8
The hash proves the plan existed before the data. I committed it to the repo on the same day I started collecting data. The analysis results were generated on October 17, 2026, on the exact rank values, to four decimal places.
Why I did not pivot
When the primary result came back null, I had options. I could have:
- Tried different outcome variables (page citation instead of domain citation)
- Tried different regression specifications
- Looked for subgroups where the score did predict something
- Run it on a different engine and hoped for a better result
- Not published it at all
I did run the secondary analyses, because they were pre-registered. None of them were significant either. Every Holm-corrected p-value was 1.0000.
I did run the sensitivity analysis, because it was pre-registered. Excluding blocked pages did not change anything (OR 1.0408, p = 0.7288).
The page-level analyses (whether the exact page, not just the domain, was cited) were also pre-registered as secondary outcomes. Those odds ratios were above 1.0 (1.0568 and 1.0478) but still not significant (p = 0.6434 and p = 0.6933).
None of these were the primary outcome. The primary outcome was domain citation on template A. It was null. That is the result.
Why publish a null
Three reasons.
1. The tool is open source. People can check the code. If I published a study that only showed positive results, someone would eventually run their own and find nothing. Better to publish the null myself and explain what it means than have someone else discover it and conclude I was hiding it.
2. Null results are informative. The null tells you that a 0-to-100 score based on technical readability does not predict citation. That is useful. It means the score measures what it measures (citability) and not what it does not measure (citation). Users of the tool should know that distinction. Agencies selling GEO services based on readability scores should know it too.
3. Every other tool should do this. There are at least 15 paid AI visibility tools on the market. Peec AI ($95/mo), Otterly ($29/mo), Scrunch ($250/mo), Profound (enterprise), Semrush AI Visibility ($99/mo), AthenaHQ ($295/mo), LLM Pulse (EUR 49/mo), and others. None of them have published a calibration study. None of them have tested whether their scores predict citations. They sell visibility scores and share-of-voice metrics without showing the validation data.
I am not saying their scores are wrong. I am saying I do not know if they are right, because they have not shown the work. The work is: pre-register the analysis, collect the data, publish the result, whatever it is.
What I learned from the process
The pre-registration forced me to think through the analysis before I had data. That is harder than it sounds. I had to decide what the primary outcome would be, what controls to use, what counts as a secondary analysis, and what sensitivity checks to run. All before seeing a single citation.
It also made the writing easier. When the result came back null, I did not have to decide how to frame it. The analysis was pre-committed. I ran it, reported it, and wrote up what it said. No decisions about what to include or exclude. The pre-registration made those decisions for me.
The limitation I acknowledge
The study has one engine (Perplexity Sonar), one date (October 17, 2026), and one model version. The question is built from each page's own headline, so it tests whether a page is cited when the question is literally about that page's topic. That is a generous test. A more realistic test would use generic queries about the topic, not the page's own headline.
The sample also favours reachable, English-language sites that publish a sitemap. That is not the full web. It is the part of the web that AI crawlers can easily find.
I plan to run the study again with ChatGPT and Google AI Overviews, and with generic queries instead of headline-based ones. If the score predicts citation in those settings, I will publish that. If it does not, I will publish that too.
The files
Everything is at github.com/Auriti-Labs/geo-optimizer-skill/tree/main/research/calibration-2026-10:
-
PREREGISTRATION.md: the analysis plan, with the SHA-256 to verify it existed before the data -
results.csv: 341 rows, one per page, no domain or headline, shuffled -
analysis-results.txt: the output of the pre-registered analysis
You can reproduce the primary result from the CSV. If you find something I missed, open an issue.
Top comments (0)