Originally published on Sitecheckly.
The survey found that 6.2% of sites with readable robots.txt files block an AI search crawler, while 15% block a training crawler. Across analyzed home pages, 16% publish llms.txt, and 13% run a detectable end-of-life or vulnerable component. That software figure is a lower bound because detection depends on exposed versions. These findings come from the Sitecheckly open dataset.
The October 2026 Sitecheckly survey used the Tranco top 10,000, list 7462X, covering 2026-10-02 to 2026-10-31. Home pages were fetched on 2026-11-01; 4,934 of 10,000 requested domains could be analyzed. The ranking came from Tranco. These are home-page benchmarks, not whole-site assessments.
Use the website security statistics below to identify checks worth running on your own pages. A common finding is not automatically your most urgent problem. Confirm what applies to your application before changing it.
What do the website security statistics reveal?
The average security score was 77.1 out of 100. The most common missing header was Cross-Origin-Opener-Policy, but the appropriate response depends on what the page does.
The following shares apply to the analyzed home pages:
| Missing header | Share | Owner action |
|---|---|---|
| Cross-Origin-Opener-Policy | 95% | Assess cross-origin isolation needs before choosing a policy. |
| Referrer-Policy | 73% | Choose a policy that limits unnecessary URL sharing. |
| Content-Security-Policy | 66% | Build and test a policy for the resources the page needs. |
| X-Content-Type-Options | 54% | Set the header to the nosniff value. |
Use the security headers guide for implementation details. A missing header is a configuration finding to assess, not proof of an exploitable vulnerability or a successful attack.
Two separate findings need their own review:
- Third-party scripts without Subresource Integrity: 61%. Review eligible static resources and add integrity hashes where appropriate.
- No CAA records: 78% of 4,141 sites checked. Review whether your DNS should specify authorized certificate authorities.
Do not put script integrity work in the same task as response-header configuration. Give each change an owner and test it against the page's actual dependencies.
To check your own page, run the free, passive security audit. It checks the URL entered, requires no sign-up, and does not audit every page on the site.
Which software needs maintenance?
The survey detected at least one end-of-life or known-vulnerable component on 13% of analyzed home pages. Only common components with exposed versions could be checked, so this is a lower-bound detection figure, not a complete software inventory.
Each row below uses its own exposed-version sample. End-of-life and known-vulnerability categories can overlap. Do not add the shares.
| Technology | Exposed-version sample | End of life | Known vulnerability | Action |
|---|---|---|---|---|
| jQuery | 661 sites | 30% | 44% | Upgrade and test dependent scripts. |
| WordPress | 201 sites | 0.0% | 3.5% | Update core and verify compatibility. |
| Font Awesome | 147 sites | 45% | 0.0% | Move to a supported release. |
| Nginx | 146 sites | 70% | 57% | Check vendor support and patch status. |
| Drupal | 135 sites | 7.4% | 0.0% | Update core and modules. |
| PHP | 110 sites | 42% | 42% | Upgrade the runtime and test the application. |
| Bootstrap | 95 sites | 52% | 41% | Upgrade and test layouts and scripts. |
An advisory match needs verification. A Linux distribution may backport a security fix without changing the exposed version string, which can make version-based risk estimates too high. Conversely, a component that exposes no version may escape these checks. The methodology explains these detection limits.
For your own maintenance queue, record the detected version, the installed package's patch status, and the team responsible for the upgrade. Do not treat a hidden version as evidence that software is current.
How ready are sites for AI search?
The average AI-search readiness score was 66.4 out of 100. The survey found gaps in structured data and in how pages presented answers.
| Finding | Share |
|---|---|
| No Organization schema | 72% |
| No concise answer-first paragraphs detected | 70% |
| No question-style subheadings | 68% |
| No Schema.org structured data found | 51% |
Treat these as review prompts, not a template every page must follow. Use accurate, relevant structured data. Put a direct answer near the start when the page addresses a question. Use question-style headings when they make the content easier to navigate, not simply to satisfy a check.
The free AEO audit checks the URL entered and can help identify these findings. Passing the checks does not guarantee an AI-search citation.
llms.txt adoption was 16%. That is an adoption measure, not a requirement for visibility. The file remains a proposal, and major engines have not committed to reading it. Google's guidance for AI features in Search says no new AI text files are needed to appear in those features. Keep that distinction in mind when reviewing AI crawler access and related files: publishing an optional file should not displace work on intended crawler access or useful page content.
Which AI crawlers do sites block?
Among the 4,546 sites with readable robots.txt, 6.2% blocked at least one AI search crawler, compared with 15% blocking at least one training crawler. In that same group, 9.2% blocked training crawlers while allowing AI search crawlers.
These AI crawler blocking statistics show why a blanket “AI bots allowed or blocked” label is not precise enough. Search access and training permissions need separate decisions.
Reported blocking shares for selected agents were:
| User agent | Share blocking it |
|---|---|
| CCBot | 12% |
| Bytespider | 11% |
| GPTBot | 11% |
| ClaudeBot | 10% |
| Google-Extended | 9.3% |
All shares in this table use the readable-robots.txt group, not every analyzed home page.
Check the purpose of each agent before changing policy:
- OAI-SearchBot supports ChatGPT search citations. GPTBot is for OpenAI model training. ChatGPT-User fetches content on a user's request. These are distinct roles in the OpenAI bot documentation.
- Claude-SearchBot is a search crawler, while ClaudeBot is a training crawler. PerplexityBot is Perplexity's crawler. The AI crawlers guide separates these user agents and their purposes.
- Google-Extended controls Gemini training, not Google Search or AI Overviews. Googlebot handles Search crawling. See Google's crawler overview.
Blocking a search crawler can affect that engine's ability to fetch content. Allowing it does not promise inclusion or citations.
Write down your intended policy, then test representative URLs against the actual robots.txt rules. Under RFC 9309, the longest matching rule wins. Do not assume that the first rule in the file decides the result.
What SEO and technology maintenance should owners prioritize?
The website SEO statistics point to several practical page-level checks:
| Finding | Share | Owner action |
|---|---|---|
| Images missing alt text | 61% | Write useful alternatives where needed. |
| Title length outside the recommended range | 49% | Review titles for clarity rather than length alone. |
| No h1 found | 34% | Add a descriptive main heading where appropriate. |
| No canonical URL | 31% | Review canonicalization and the intended preferred URL. |
Run the free SEO audit on the URL you want to review. Use the results to investigate the page, not to make automatic edits without context.
The survey also detected these CMS shares:
| CMS | Share of analyzed home pages |
|---|---|
| WordPress | 12% |
| Drupal | 3.6% |
| Adobe Experience Manager | 2.4% |
These are detection shares, not a complete CMS census. Maintain your own platform inventory and name a maintenance owner for each system. That inventory should be more authoritative than what an external home-page check can identify.
There is also a separate front-end dependency finding: 31% of 680 sites exposing a jQuery version ran jQuery 1.x or 2.x. This is a different version-exposing group from the software-health table above.
Use that finding as a prompt to inventory front-end dependencies, plan upgrades, and test compatibility. Keep the two jQuery samples separate when comparing results or reporting progress.
How was the survey conducted, and what are its limits?
The survey analyzed home-page responses fetched over HTTP or HTTPS, without rendering JavaScript. It also included checks beyond those responses, such as DNS CAA records. It respected robots.txt. Domains that could not be analyzed were mostly CDN, DNS and API hosts, sites asking crawlers to stay away, or sites blocking automated requests.
The scope and fetch dates are stated near the top of this report. The methodology describes the collection approach and its limits.
Read the benchmarks with these constraints in mind:
- Page scope: a home-page finding does not establish the condition of every page on a domain.
- Different denominators: robots.txt findings, CAA checks, and software-specific results use the groups stated beside them.
- Detection limits: exposed-version checks provide lower-bound detection shares, while advisory matches can overstate risk where fixes were backported.
- No individual-site claims: this report presents aggregates and names no individual sites.
- No proof of compromise: a configuration finding or version match is a reason to investigate, not evidence that an attack succeeded.
The reported figures and denominators are available in the open dataset, licensed under CC BY 4.0.
What to do next
- Choose an important URL and audit that page. Sitecheckly is free with no sign-up.
- Use the home page for SEO and AEO together, or choose a dedicated audit. Security is a separate audit.
- Verify software findings against your inventory and vendor patch information.
- Assign each confirmed fix to an owner, test the change, and rerun the relevant page audit.
Frequently asked questions
Does a missing security header mean a site is compromised?
No. A missing header is a configuration finding, not evidence that an attack has succeeded. Its importance depends on the application and the protection the header would provide.
Can a site block training crawlers without blocking AI search crawlers?
Yes. Operators can write separate robots.txt rules for different user agents, then check that the rules reflect their intended policy.
Does an exposed vulnerable version prove the installation is unpatched?
No. A distribution may backport a security fix without changing the version string used for detection. Check the vendor's patch information before assessing the risk.
Can Sitecheckly audit every page in a site at once?
No. Each run audits the single URL entered. The security audit is separate from the SEO and AEO audits.
Want to see how your own site does? Run a free audit at sitecheckly.com.
Figures in this article come from Sitecheckly's own audit data; see the methodology.
Top comments (0)