DEV Community: Raphaël Aubry

Can AI crawlers read your site? robots.txt, llms.txt and JavaScript rendering, checked

Raphaël Aubry — Fri, 06 Nov 2026 04:41:37 +0000

Originally published on howseen.ai: howseen.ai/llms-txt

By Raphaël Aubry, founder of Howseen

An AI assistant can only cite a page it can read. Before anyone talks about "GEO" or content strategy, three boring things decide whether your site is even in the running:

Does your robots.txt let the AI crawlers in?
Is your content in the HTML the server sends, or only after JavaScript runs?
Do you have an llms.txt, and does it matter?

Here is how to check all three from a terminal, what the official documentation says, and a free tool that runs the same checks on up to 100 URLs at once.

Check 1: robots.txt, bot by bot

curl -s https://yoursite.com/robots.txt

The thing most people miss: AI companies run several crawlers with different jobs, and each one reads its own group in robots.txt. From each company's documentation:

Company	Token	Job
OpenAI	`GPTBot`	Collects content for model training
OpenAI	`OAI-SearchBot`	Surfaces sites in ChatGPT search
OpenAI	`ChatGPT-User`	Fetches pages for user actions in ChatGPT
Anthropic	`ClaudeBot`	Collects content for model training
Anthropic	`Claude-SearchBot`	Indexes content for Claude search results
Anthropic	`Claude-User`	Fetches pages when a Claude user asks
Perplexity	`PerplexityBot`	Surfaces and links sites in Perplexity search
Perplexity	`Perplexity-User`	Fetches pages when a user asks a question
Google	`Google-Extended`	Token for Gemini training and grounding, not a separate crawler

Three consequences:

Search bots and training bots are independent. You can block GPTBot (training) and keep OAI-SearchBot (ChatGPT search). OpenAI says sites that block OAI-SearchBot do not appear in ChatGPT search answers.
Google-Extended does not control AI Overviews. Those come from Google's search index, which Googlebot builds. Blocking Google-Extended keeps your content out of Gemini training, not out of AI Overviews.
The dangerous line is the generic one. User-agent: * followed by Disallow: / blocks every bot that has no group of its own, AI crawlers included. In my experience it is usually a leftover from an old scraper defense that nobody reopened.

Reading robots.txt by eye works until rules overlap. The standard (RFC 9309) says the most specific user-agent group wins, then the longest matching rule, and Allow wins a tie. Here is that logic in the checker I describe at the end of this post:

const specific = parsed.groups.filter((g) => g.agents.includes(token));
if (specific.length) {
    matchedGroup = token;
    rules = specific.flatMap((g) => g.rules);
} else {
    const star = parsed.groups.filter((g) => g.agents.includes('*'));
    if (star.length) {
        matchedGroup = '*';
        rules = star.flatMap((g) => g.rules);
    }
}
let best = null;
for (const r of rules) {
    if (!ruleMatches(r.path, path)) continue;
    const len = r.path.length;
    if (!best || len > best.path.length || (len === best.path.length && r.type === 'allow')) best = r;
}
const allowed = !best || best.type === 'allow';

Note the fallback: a bot with its own group ignores the * group entirely. Adding User-agent: GPTBot with a single Disallow: /private/ means GPTBot no longer sees anything you wrote under *.

One more layer sits before robots.txt: your firewall or CDN. A WAF rule that blocks unknown bots can reject AI crawlers before they read a single line. If your logs show no hits from these tokens at all, check there too.

Check 2: what a crawler sees without JavaScript

Most AI crawlers fetch the raw HTML and parse it. They do not run your JavaScript. If your page is a client-rendered shell, they get an empty <div id="root">.

Fetch the page with a crawler user agent and count the words that are actually in the HTML:

UA="Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot"

curl -s -A "$UA" https://yoursite.com/pricing \
  | perl -0pe 's/<script.*?<\/script>//gs; s/<style.*?<\/style>//gs; s/<[^>]+>/ /g' \
  | wc -w

It is crude (navigation and footer count too), but the order of magnitude tells the story. A few dozen words on a page that looks full in the browser means the content arrives through JavaScript. The same command also shows whether your server or CDN answers this user agent with a 200 at all: add -o /dev/null -w "%{http_code}\n".

The checker uses the same idea with explicit thresholds: under 100 words, the page scores 0 on this check, and if the HTML also contains a framework root, it is flagged as rendered in the browser.

const SPA_MARKERS = [
    ['id="root"', 'React'],
    ['id="__next"', 'Next.js'],
    ['id="app"', 'Vue or similar'],
    ['id="__nuxt"', 'Nuxt'],
    ['ng-version', 'Angular'],
    ['<app-root', 'Angular'],
    ['id="svelte"', 'Svelte'],
    ['data-reactroot', 'React'],
];
const likelyClientRendered = wordCount < 100 && (markers.length > 0 || /<script\b[^>]*src=/i.test(html));

The fix depends on the stack: server components in the Next.js App Router, SSR mode in Nuxt, pre-rendering or static generation for a React SPA. You can also do the manual test: View Page Source (not Inspect) and look for your headings.

Check 3: llms.txt, and what it does not do

llms.txt is a Markdown file at the root of your site that summarises it and links the key pages. Jeremy Howard proposed the format on September 3, 2024, at llmstxt.org. The only required part is an H1 with the site name.

curl -s -o /dev/null -w "%{http_code} %{content_type}\n" https://yoursite.com/llms.txt
# 200 text/plain; charset=utf-8   <- what you want

The classic failure is a site that answers every unknown path with its home page: you get a 200, but with text/html. The checker treats that as "no llms.txt".

Now the honest part. No major AI engine has confirmed using llms.txt to decide which brands to cite. Google says you don't need to create new machine readable files, AI text files, or markup to appear in AI Overviews and AI Mode. OpenAI points site owners to robots.txt and OAI-SearchBot for ChatGPT search. Where llms.txt is used is inference on documentation: OpenAI publishes its own docs index as an llms.txt file, and AI coding tools read these files to land on the right page.

File	What it does	Effect on AI visibility
`robots.txt`	Allows or blocks crawlers	Direct: a blocked crawler cannot cite you
`sitemap.xml`	Lists the URLs you want indexed	Indirect: helps pages get indexed
`llms.txt`	Summarises the site, links key pages	None confirmed by a major AI engine

So: write one if you have docs, an API or many pages (it takes minutes), and check robots.txt first.

Run all of it on many URLs

I packaged these checks as a free Apify Actor, AI Visibility Checker. Plain HTTP requests, no AI calls, no third party login. You paste up to 100 URLs per run and get a score out of 100 with one sentence per fix:

Check	Points
AI search and assistant crawlers in robots.txt	25
Text readable without JavaScript	20
Structured data (JSON-LD)	15
XML sitemap	7
AI training crawlers in robots.txt	5
Indexable, title, meta description, canonical, llms.txt	5 each
`<html lang>`	3

Training crawlers weigh little on purpose: blocking them is a valid choice. robots.txt, llms.txt and the sitemap are fetched once per domain, even if you check many pages of the same site.

{
  "urls": ["https://example.com", "https://example.com/pricing"],
  "timeoutSecs": 15
}

A useful pattern is to schedule it after each deploy, so a robots.txt change or a rendering regression shows up the same day instead of months later.

For the per-bot details (user agent strings, published IP ranges, how to spot a fake GPTBot in your logs), see the AI crawlers robots.txt guide.

See what AI says about your brand: howseen.ai

How to expose your SaaS data to Claude and ChatGPT with MCP (a real 44-tool server)

Raphaël Aubry — Fri, 06 Nov 2026 04:40:56 +0000

Originally published on howseen.ai: howseen.ai/mcp

By Raphaël Aubry, founder of Howseen

Our product measures how ChatGPT, Gemini, Perplexity and Google's AI answers talk about a brand. That data lived in a dashboard, while more of our users were already working inside Claude, ChatGPT or Cursor. So we brought the data to the assistant instead of asking people to open one more tab, and shipped an MCP server. It runs in production at https://app.howseen.ai/api/mcp, version 1.3.0, with 44 tools: 26 read tools and 18 write tools. This post walks through the design choices, with the real code, so you can do the same for your own SaaS.

Try it before reading

Tool discovery works without a key, so you can list the tools right now:

curl -s -X POST https://app.howseen.ai/api/mcp \
  -H 'Content-Type: application/json' \
  -H 'Accept: application/json, text/event-stream' \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' \
  | jq '.result.tools | length'
# 44

Count the read-only ones:

... | jq '[.result.tools[] | select(.annotations.readOnlyHint)] | length'
# 26

Now try to call a tool without a key:

curl -s -X POST https://app.howseen.ai/api/mcp \
  -H 'Content-Type: application/json' \
  -H 'Accept: application/json, text/event-stream' \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"list_brands","arguments":{}}}'

You get an HTTP 401 with WWW-Authenticate: Bearer realm="howseen" and a JSON-RPC error (code -32001) whose message tells you where to create a key. That split, open discovery and keyed calls, is the first design decision.

1. Transport: Streamable HTTP, stateless

The app is a Next.js route handler on a serverless runtime. Keeping MCP sessions in memory between requests does not fit that model, so the server is stateless: one server and one transport per POST, JSON responses, no session id.

const server = buildMcpServer(ctx);
const transport = new WebStandardStreamableHTTPServerTransport({
  sessionIdGenerator: undefined,
  enableJsonResponse: true,
});
await server.connect(transport);
return withCors(await transport.handleRequest(fwd));

sessionIdGenerator: undefined turns sessions off. enableJsonResponse: true returns a plain JSON body instead of opening an event stream. Since there is no stream to open, a GET asking for text/event-stream gets a 405, and a plain GET returns a small JSON description (name, version, tool names, auth rules) that doubles as a health check.

One detail to get right: we read the request body once to decide on auth, so we hand the transport a fresh Request with the same bytes. Otherwise the transport finds an already consumed body.

2. Discovery without a key

MCP directories and clients probe a remote server before the user has pasted anything: initialize, then tools/list. If those calls return 401, your server looks broken in every listing. These methods only return static metadata, so we let them through:

export const DISCOVERY_METHODS: ReadonlySet<string> = new Set([
  "initialize",
  "notifications/initialized",
  "ping",
  "tools/list",
  "prompts/list",
  "resources/list",
  "resources/templates/list",
]);

export function decideMcpAuth(keyPresent: boolean, body: unknown): McpAuthDecision {
  if (keyPresent) return "authenticate";
  return isDiscoveryOnly(body) ? "open" : "needs_key";
}

Two guards keep this safe. First, JSON-RPC allows batches, so isDiscoveryOnly requires every message in the batch to be a discovery method: one tools/call hidden in a batch makes the whole request need a key. Second, the keyless context passes a database handle that throws on any access, so even a coding mistake cannot leak data:

db: new Proxy({}, { get() { throw new Error("no data access without an API key"); } }) as SupabaseClient,

3. Auth: a Bearer key, hashed at rest

Users create a personal key in Settings. It starts with hs_live_, is shown once, and only its sha256 hash is stored. The server reads it from Authorization: Bearer hs_live_....

Some AI apps cannot send custom headers, so the key can also go in the server URL as ?key=. We document that this URL must be treated like a password and that a leaked key can be revoked in Settings.

When a key is missing or invalid, the 401 carries a JSON-RPC error body, not just a status code. MCP clients display that message to the user, so it is where you explain what to do next.

4. Tell the client which tools are safe

Every tool declares MCP annotations. Here is what tools/list returns for the first read tool:

{
  "name": "list_brands",
  "title": "List brands",
  "annotations": {
    "readOnlyHint": true,
    "destructiveHint": false,
    "openWorldHint": false
  }
}

Clients use readOnlyHint to decide what they can run without asking. Our 26 read tools cover the questions people actually ask: which buyer questions name a competitor and not their brand (get_prompt_gaps), which sites AI cites in those answers (get_easiest_wins), what moved since the last scan (get_changes), real answer excerpts with their citations (search_responses).

The read tools call the same data layer as our REST API, so the numbers in Claude match the dashboard. That matters more than it sounds: a figure in the chat that differs from the app would make users doubt both.

5. Write tools: preview first, confirm second

The 18 write tools (track a question, add a competitor, draft an article, publish it, request indexing...) all go through one gate, in a fixed order:

export function writeGate(o: { scopes: readonly string[] | null | undefined; confirm?: boolean | null; blocked?: string | null }): GateResult {
  if (!hasScope(o.scopes, "write")) return { kind: "forbidden", message: READ_ONLY_KEY_MESSAGE };
  if (o.blocked) return { kind: "blocked", message: o.blocked };
  if (o.confirm !== true) return { kind: "preview" };
  return { kind: "execute" };
}

Keys are read only by default. A key needs the write scope ("Read + actions" in Settings) to change anything.
Plan limits and quotas apply exactly as in the app. Over the limit, the tool refuses and says why.
Without confirm: true, nothing changes. The tool returns a preview: what will change, the quota impact, and the exact arguments to send back with confirm: true.
Only then does it execute, and the change is logged.

The server instructions tell the model to show the preview, wait for an explicit yes, and never chain several confirmed changes from one approval. A model can still ignore instructions, which is why the gate is in code and not only in the prompt.

6. Prompts and limits

The server also exposes three ready-made prompts (weekly_review, client_report, fix_my_site) that chain the tools for common jobs. Each key is limited to 120 requests per minute and 5,000 per day.

7. Get listed in the official registry

The server is listed in the official MCP registry as ai.howseen/howseen-mcp, with a remote entry like this:

"remotes": [{
  "type": "streamable-http",
  "url": "https://app.howseen.ai/api/mcp",
  "headers": [{ "name": "Authorization", "isRequired": true, "isSecret": true }]
}]

You can check any entry with curl "https://registry.modelcontextprotocol.io/v0/servers?search=howseen". Keyless discovery is what lets registries and directories read your tool list without credentials.

What I would keep if I started again

Stateless from day one. No session store to debug on serverless.
Open discovery, keyed everything else, with the batch rule.
One data layer for the app, the REST API and MCP.
Write tools that preview by default, enforced in code.
Error messages written for the human reading them in Claude, not for a log file.

The full tool list, setup per app (Claude, ChatGPT, Cursor, Claude Code, Codex) and the REST endpoints are on the Howseen MCP page, and ready-to-paste commands are in MCP use cases.

See what AI says about your brand: howseen.ai