Over the last few weeks, I’ve been deep in the weeds of ‘Building AI-Accessible Infrastructure.’ It’s a fascinating architectural challenge: how do we shift from optimizing for eyeballs to optimizing for context-hungry agents?
As I was documenting the various layers of AI-accessible infrastructure — from llms.txt files to Model Context Protocol (MCP) endpoints — I kept hitting the same wall: I wished there was a quick, automated way to check all these signals at once. Not just ‘does this file exist,’ but ‘is it implemented in a way that an LLM agent can actually consume it?’
So, I built one.
Today, I’m launching GEO Score (Generative Engine Optimization Score), a free, no-login scanner at javatask.dev/geo/ that audits your site for AI readiness across 13 base technical signals — plus up to 3 bonus checks that can push your score above 100%.
How Scoring Works
GEO Score uses a fixed base denominator — the sum of all base check weights — so your baseline score reflects how well you’ve covered the fundamentals. Bonus checks add points on top of that denominator. Failing a bonus check never hurts your score; passing one rewards you.
Inspired by Mozilla Observatory’s approach, GEO Score now allows sites to exceed 100%. The grade scale mirrors what you’d expect, with one addition:
| Grade | Score |
|---|---|
| A+ | ≥ 100% |
| A | ≥ 90% |
| B | ≥ 75% |
| C | ≥ 60% |
| D | ≥ 40% |
| F | < 40% |
What GEO Score Measures
The scanner focuses on the ’technical plumbing’ required for an AI agent to discover, parse, and trust your content. It breaks these into three distinct layers, each representing a different stage of the agent-to-site interaction.
1. The Discovery Layer (Head Checks)
Before an agent can read your content, it needs to know where it lives — so the scanner performs rapid HEAD requests to verify the existence of these roadmap files:
- llms.txt (15 pts): The emerging standard for LLM-friendly documentation. It provides a human-readable and machine-parsable entry point for agents.
- llms-full.txt (5 pts): The expanded version for comprehensive context, often containing the full text of documents.
- sitemap.xml (10 pts): While built for search engines, modern AI agents use sitemaps to build their initial crawl frontier.
-
robots.txt AI Policy (8 pts): The base check tests whether 7 AI discovery agents are explicitly allowed (or at least not blocked):
OAI-SearchBot,ChatGPT-User,PerplexityBot,Claude-SearchBot,Claude-User,Gemini-Deep-Research,YouBot. These are the bots that power AI search citations and deep research across Perplexity, ChatGPT research mode, Gemini Deep Research, and Claude — blocking any of them quietly cuts you out of the AI discovery layer. Blocking training bots like GPTBot or Google-Extended is a separate concern handled in the bonus layer — more on that distinction below.
2. The HTML Layer (Homepage Parse)
Once your content is found, the next question is: can an agent extract semantic meaning and metadata from your raw HTML without relying on fragile scrapers?
- JSON-LD Schema (15 pts): We specifically look for Article or TechArticle schema. Structured data is the highest-signal way to tell an agent what your content is actually about.
- Meta Description (8 pts): We look for 50–160 characters of clean context. If it’s too short, it’s useless; too long, and it gets truncated.
- Meta Author (5 pts): Essential for attribution. Agents need to know who wrote the content to weigh its authority.
- Canonical Link (8 pts): Prevents duplicate-content confusion by telling the agent which URL is the source of truth.
- OpenGraph (OG) Tags (10 pts): Standardized titles, descriptions, and URLs that provide a fallback for agents when custom schema is missing.
- Performance (5 pts): TTFB (Time to First Byte) < 800ms. Agents have time budgets too; slow sites get skipped or deprioritized in real-time search.
- Payload Size (3 pts): Page size < 500 KB. Large payloads are expensive for LLMs to tokenize and process.
3. The MCP Layer (Conditional)
This is the next-generation approach to site-agent interaction, moving beyond static files to interactive capabilities.
-
.well-known/mcp.json (10 pts): The discovery endpoint for Model Context Protocol servers. The base check (10 pts) verifies that
/.well-known/mcp.jsonexists and returns valid JSON. If the manifest contains a valid MCP endpoint URL, the scanner runs a conditional live probe: it POSTs an MCPinitializerequest to the declared endpoint to verify the server is actually running. The probe result is reported separately as informational — it does not add or deduct points, but it tells you immediately if your MCP endpoint is down.
Bonus Checks: Scoring Beyond 100%
These three checks sit outside the base denominator. Passing them adds to your numerator; failing them changes nothing. The total is capped at 125%.
| Check | Bonus | Passes when |
|---|---|---|
| JSON-LD Richness | +7 pts | Your JSON-LD includes author, datePublished, and description fields |
| Training Bot Opt-out | +5 pts | GPTBot and/or Google-Extended are blocked in your robots.txt |
| Markdown Mirror | +8 pts | /index.md (or /<!-- raw HTML omitted -->/index.md) returns HTTP 200 with a plain-text or Markdown content type |
Note: Security Headers (base, 12 pts) and Strict Security Headers (bonus, +8 pts) were removed in March 2026. These checks duplicate what Mozilla Observatory already validates in depth. GEO Score stays focused on AI-agent discoverability signals.
A note on Markdown Mirror: it was originally a base check that penalized failure. It has since been promoted to a bonus — you’re rewarded for providing a clean Markdown version of your content, but not penalized if your CDN doesn’t serve .md files. The pass condition has also been relaxed to accept text/plain, since most CDNs don’t send text/markdown by default.
⚠️ Important: What GEO Score is NOT
Before we look at some results, let’s be clear about scope. GEO Score measures technical implementation signals only.
- A site like mozilla.org or wikipedia.org might score low on this tool because they haven’t implemented llms.txt yet — but they are still heavily cited by AI because of their massive domain authority and content quality.
- The tool does not measure domain authority, backlink profile, content quality, or citation history.
- It does not read your content to see if it’s ‘good.’ It only checks whether the delivery mechanism is ‘correct.’
Think of it this way: GEO Score tells you, ‘Have you implemented the plumbing?’ It does not guarantee, ‘Will AI cite you?’ Technical signals are necessary for long-term discoverability, but they aren’t a replacement for quality writing.
The Three-Tier Bot Taxonomy: Why Blanket Blocking Is a Mistake
When I first built the robots.txt check, I treated any bot blocking as a negative — you lose reach, so you lose points. That was wrong. The current scanner reflects a more sophisticated three-tier taxonomy underpinned by research into how modern AI crawlers work.
Tier 1 — Training crawlers (block these): GPTBot (OpenAI pre-training), ClaudeBot (Anthropic pre-training), Google-Extended (Gemini model training and Vertex AI grounding), CCBot (Common Crawl — feeds Llama, GPT, and many others). These bots ingest your content into model weights without referring traffic back. Blocking them is legitimate IP protection and has no effect on inference-time discoverability — agents don’t need to have trained on your content to cite it.
Tier 2 — AI search indexers (allow these): OAI-SearchBot (ChatGPT search results), Claude-SearchBot (Claude’s search index). These crawlers build real-time indices for conversational search interfaces. Being indexed by them is the AI equivalent of Google crawling your site.
Tier 3 — Deep research agents (allow these): ChatGPT-User (ChatGPT Deep Research mode), Claude-User (Claude real-time query fetcher), Gemini-Deep-Research (Google’s autonomous research agent), PerplexityBot (Perplexity AI search), YouBot (You.com AI search). These agents act on behalf of a specific user to complete a multi-step research task. Blocking them means a user asking Gemini or ChatGPT to research your topic won’t find you — even if you’re a top result in the underlying search.
The base check validates that all seven discovery/research agents (Tier 2 and Tier 3) are not blocked. Blocking training scrapers (Tier 1) while allowing discovery bots is now a separate bonus check (+5 pts). The original scanner penalized this configuration. It shouldn’t have.
Here is the robots.txt configuration that satisfies the base check and earns the training-bot opt-out bonus:
# Block training crawlers (protect your IP)
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
# Allow AI search indexers (Tier 2)
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
# Allow deep research agents (Tier 3)
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-User
Allow: /
User-agent: Gemini-Deep-Research
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: YouBot
Allow: /
Dogfooding: Scanning javatask.dev
Of course, the first thing I did after finishing the scanner was point it at my own site. I was confident I’d see a straight ‘A.’
After the upgrade, javatask.dev scores A+ (117.9%, bonus: +20.0 pts). Here’s what changed, including one check where my original thinking was backwards.
- llms.txt / llms-full.txt : Pass (20 pts) — My Hugo setup generates these automatically as part of the build process.
- sitemap / mcp.json : Pass (20 pts).
- robots.txt : Pass (8 pts) — I block GPTBot, ClaudeBot, Google-Extended, and CCBot to protect content from training ingestion. Under the 3-tier taxonomy, all 7 AI discovery and research agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, Claude-User, Gemini-Deep-Research, YouBot) are allowed, so the base check passes. Blocking training crawlers also earns the Training Bot Opt-out bonus (+5 pts). What was once treated as a penalty is now correctly recognized as a reward.
-
meta_author : Fail → Pass (0 pts → 5 pts) — I was shocked to find that while I had og:author in my OpenGraph tags, I was missing the standard
meta name="author"tag. This is a classic example of ‘shadow metadata’ — information that humans see (via the site UI) but machines miss. I fixed this immediately. - JSON-LD : Warn → Pass — When scanning the homepage, the tool found WebSite and Person schema, but not Article. I’ve since ensured individual post pages carry full TechArticle schema — which also satisfies the JSON-LD Richness bonus check (requiring author, datePublished, and description fields).
After addressing those gaps, all three bonus checks pass.
A Note on Scanning Homepages vs. Posts
One key insight from dogfooding: some Fail or Warn results are expected when you scan a site’s root domain rather than a specific article.
The scanner checks the specific URL you provide. If you scan javatask.dev/, the tool won’t find an Article schema because the homepage is a list of summaries, not a single post. Similarly, the Markdown Mirror bonus check is designed for individual post URLs — the homepage intentionally doesn’t serve a /index.md equivalent.
Pro-tip: To see your true score, scan a specific blog post URL. That’s where the Article schema and Markdown Mirror bonus check will actually shine.
Technical Brief: Rust, Axum, and Lambda
I wanted GEO Score to be fast — like, really fast. When a user enters a URL, they shouldn’t be staring at a spinner for 10 seconds. I built the backend in Rust using the Axum framework, deployed as an AWS Lambda function.
- Concurrency : The scanner doesn’t run checks sequentially. It uses a three-round parallel orchestrator. Round one handles HEAD requests for discovery files. Round two fetches the main HTML and parses it using the scraper crate. Round three probes any discovered MCP endpoints.
- Performance : The average scan takes ~200ms of actual compute time. The rest is network latency as we wait for the target site to respond. By using Rust on Graviton2 (ARM64) Lambdas, cold starts stay minimal and execution cost stays nearly zero.
-
Security : Building a tool that fetches arbitrary user-provided URLs is an SSRF (Server-Side Request Forgery) nightmare. The backend includes a robust
validator.rsmodule that resolves the hostname and rejects private IPv4/IPv6 ranges, loopbacks, and non-HTTPS schemes before a request is ever dispatched. - Rate Limiting : To prevent abuse while keeping the tool login-free, I implemented an in-memory rate limiter that tracks IP addresses, allowing 5 scans per hour. It’s a balance between preventing bot-on-bot violence and keeping the tool accessible.
Try it Yourself
javatask.dev scores 117.9%. That’s the current benchmark — run the scanner on your own site and see where you land.
It’s completely free, requires no login, and performs no tracking. If you score A+ (100%+), I’d like to know about it — reply to this post or tag me on LinkedIn. I’ll feature the highest-scoring community sites in a follow-up. If you score below 70%, the breakdown will show you exactly what’s missing and why it matters.
This also applies at the professional level. If you maintain documentation, a developer portal, or a marketing site for a product, your AI readiness score affects whether agents surface your content in their answers — not just whether humans find it in search. The infrastructure layer is the same regardless of whether you’re a solo blogger or an engineering team managing 10,000 pages.
What about licensing and intellectual property? The scanner focuses on technical discoverability signals. But there’s a related question this tool doesn’t answer: once AI agents can find and cite your content, what rights do you retain? Can you allow citation but prohibit training? Does a CC-BY-NC-4.0 license cover AI indexing? I’m working on a follow-up article on the legal and technical architecture for embedding IP rights signals in MCP tool responses, HTTP headers, and structured data — so your content stays citable without becoming training fodder. Stay tuned for Part 9 in the series.
Happy building, Andrii Melashchenko
Top comments (0)