DEV Community: Ray

How Much of Your Blog Does AI Search Actually Grab? Breaking Down Claude's WebSearch and WebFetch

Ray — Fri, 19 Jun 2026 14:23:46 +0000

A while back I wrote Is SEO Not Enough? Meet AEO — Getting Your Site Found by AI Search, and right after finishing it a question hit me: when AI does a web search, how much of my blog does it actually grab? The whole article verbatim? The first 500 characters? Or does it bail after seeing just the title? So I dug into it, and this post walks through Anthropic's official web_search and web_fetch tool specs, runs a quick test against my own blog, and ends with what all this concretely means for how you should write posts and copy.

"Search" and "fetch" are two different things

Before going further, the one thing worth being crystal clear on: when AI runs a query, "search" and "fetching the page body" are not the same operation. They're two separate stages.

Stage 1 (WebSearch): the AI takes your question and calls the WebSearch tool (which hits a search engine). What comes back is a list of search results — each entry has things like URL and title, but no page body.
Stage 2 (WebFetch): after looking at the search result list, the AI decides which entries are worth digging into, then fires a WebFetch request at each of those URLs, basically saying "give me the body of this page." That's when actual content gets pulled in.

Why doesn't it just grab the body during the search stage? Context window limits. If every search shoved 10 results' worth of full bodies in, your usable context would blow up fast — and then you'd start complaining the AI is dumb and forgets what you just asked it (because the context did overflow). So it's split into two stages: search first for a list, then decide which entries from that list to actually fetch.

Once that two-stage split makes sense, the rest of this post is about what each stage actually pulls in.

What does WebSearch pull in?

Going straight to Anthropic's official web_search tool docs — every search result entry has only four fields:

url: the page URL
title: the page title
page_age: when the page was last updated
encrypted_content: encrypted content, not for the AI to read the article — it's for multi-turn conversation citations

That's it. Four fields.

What the AI sees during the search stage is "URL, title, last updated" — three pieces of human-readable info. No body content at all.

What if the AI cites your content? There's a cap on that too:

Each web_search_result_location's cited_text is up to 150 characters of the cited content

In short: at most 150 characters of quoted text. And that's just the API-level spec.

Claude Code's built-in WebSearch shaves it down further. According to Mikhail Shilkov's breakdown of Claude Code's internal behavior, Claude Code even drops page_age and encrypted_content, keeping only title and url.

So basically — at the search stage, the AI sees nothing more than one title and one URL from your site. That's it.

What does WebFetch pull in?

Now for when the body actually gets pulled in — Stage 2, WebFetch.

Once the AI has the search results, if it decides to open up a few entries, it fires one WebFetch request per URL, and that's when the full body comes back. How much of it?

This needs to be split into two layers, because the API and Claude Code work differently.

Note
When I say "API" here, I mean the Anthropic API's web_fetch tool. "Claude Code" means the WebFetch feature built into Anthropic's own product. The two have different specs and flows.

API-level web_fetch

The Anthropic API's web_fetch tool has a parameter called max_content_tokens that developers can set themselves — though the official docs use 100,000 tokens in their examples.

The docs also give a reference conversion:

Content size	Estimated tokens
Average web page 10 KB	~2,500 tokens
Large doc page 100 KB	~25,000 tokens
Research paper PDF 500 KB	~125,000 tokens

So a medium-length blog post in plain text is usually 1–2,000 tokens, way below the 100K ceiling. Truncation basically isn't a concern unless you wrote a 50,000-character monster.

One thing to note: web_fetch's citation works differently from web_search. It uses start_char_index / end_char_index to pick out a specific position in the article (although the docs don't pin down a hard character limit).

Claude Code's built-in WebFetch

Claude Code's built-in WebFetch goes a different route.

Per Mikhail Shilkov's breakdown, the WebFetch flow is:

Convert HTML to Markdown using the Turndown package
Extract the first 100 KB of plain text
Pass that 100 KB to the Haiku 3.5 model
Haiku summarizes the answer based on your prompt and returns the summary to the main model

The real kicker is step 3. The main model — the Claude model you're actually using — never sees the page's original text. It only sees the version Haiku summarized. Which means what your writing turns into by the time it reaches the main model is decided by how Haiku reads it, not by how much you wrote.

The citation has a limit too. The rule Mikhail extracted from Claude Code's internal prompt is:

Enforce a strict 125-character maximum for quotes from any source document.

So quotes max out at 125 characters.

You're probably wondering — so how much is 100 KB of plain text? For Chinese, where each character is roughly 3 bytes, 100 KB fits 30,000+ characters; for English at 1 byte per character, that's well over 100,000 characters. No regular blog post is going to hit that ceiling. So the real concern should be "how do I get Haiku to extract the parts I want to be quoted," not "how much content am I feeding Haiku."

Running a real test against my own blog

Enough theory — let's actually test it.

I had Claude call the WebFetch tool against my blog's homepage (israynotarray.com) and asked it to grab the body.

To make this measurable, I leaned on Claude Code's Hook feature to log every stage's input and output. Here's the hook config — every WebSearch and WebFetch I/O gets logged:

{
  "hooks": {
    "PostToolUse": [
      {
        "matcher": "WebSearch|WebFetch",
        "hooks": [
          {
            "type": "command",
            "command": "/claude-tools/hooks/log-web-tool.sh"
          }
        ]
      }
    ],
    "SessionEnd": [
      {
        "hooks": [
          {
            "type": "command",
            "command": "/claude-tools/hooks/dump-transcript.sh"
          }
        ]
      }
    ]
  }
}

The prompt I used:

Please use the WebFetch tool to grab the content of israynotarray.com's homepage, then tell me what you extracted from that page.

The actual input passed to WebFetch:

{
  "url": "https://israynotarray.com",
  "prompt": "Tell me the main content of this homepage: site name, author bio, latest article list (titles and summaries), categories, plus any navbar or sidebar info."
}

After it ran, the WebFetch result — the summary that came back — looked roughly like this:

{
  "bytes": 1068,
  "code": 200,
  "codeText": "OK",
  "result": "# Page Content Analysis\n\nBased on the content snippet provided, I can share the following:\n\n## Site Structure\n- **Main category**: AI-related content, including Gemini and Gemini Omni products\n\n## Latest Article\n**Title**: \"Gemini Omni Video Generation: 30 Prompt Templates for Reels, Product Demos, and Logo Reveals\"\n\n**Summary**:\nThis post introduces Gemini Omni video generation, released by Google at I/O in May 2026, and explains how it replaces the earlier Veo 3.1. The article covers new features like '10-second clips, native synced audio, and Chinese text rendering,' and provides 30 practical prompt templates for use cases including Instagram Reels, product demos, logo animations, B-roll, transitions, quote cards, lifestyle, and food and travel.\n\n**Publish date**: 2026-06-06  \n**Reading time**: ~23 min\n\n## Limitations\nThe content snippet provided does not include author bio or full navbar info.",
  "durationMs": 4603,
  "url": "https://israynotarray.com"
}

See it? What the AI actually pulled from my blog is just this tiny under-1,000-character summary. A whole website obviously contains way more than this — the rest was never seen by the main model. The Haiku middle layer reads the page and only extracts what it judges relevant to my prompt. If I opened the same page in a browser, I'd see a full grid of article cards plus a sidebar — but Haiku doesn't ship the full grid back.

I also tried an older post with a deliberately broken URL path, and got this:

{
  "bytes": 0,
  "code": 404,
  "codeText": "Not Found",
  "result": "The server returned HTTP 404 Not Found.\n\nThe response body was not retrieved. If this URL requires authentication, use an authenticated tool (e.g. `gh` for GitHub, or an MCP-provided fetch tool) instead of WebFetch.",
  "durationMs": 588,
  "url": "https://israynotarray.com/dqwdqwdqwd"
}

Even the content of your 404 page is invisible to the AI — WebFetch just reports the 404 and the AI has no way to see what your 404 page says. Which means if your site has path issues, you've refactored URLs, or you only have frontend routing without real pages, the AI can't pull anything.

Side note — this lines up with a caveat in Claude's official docs:

The web fetch tool currently does not support websites dynamically rendered with JavaScript.

If your blog is a frontend SPA where content is entirely rendered by JavaScript at runtime, what the AI grabs might just be empty-shell HTML with no articles visible. Static generators (Hexo, Astro, Next.js in SSG mode) are relatively safe, since the build output is fully rendered HTML — the AI grabs and immediately sees content.

Don't forget the robots.txt layer

There's one more important piece — whether the AI can pull your site has a major prerequisite: robots.txt.

AI crawlers basically split into two types: search-style (cite and link back to your site) and training-style (eat content to feed the model, not necessarily linking back). The common mapping:

Crawler	Type	Behavior
Claude-SearchBot / Claude-User	Search	Real-time fetch when Claude answers, cites back
ClaudeBot	Training	Fetches content to feed Claude training
OAI-SearchBot / ChatGPT-User	Search	Real-time fetch when ChatGPT answers, cites back
GPTBot	Training	Fetches content to feed GPT training
PerplexityBot	Search	Used by Perplexity engine, cites back
Google-Extended	Training	For Gemini training
CCBot	Training	Common Crawl public dataset

If you want to be cited by AI but don't want your content used for training, the most common strategy is "allow search-style, block training-style."

Here's a robots.txt template you can copy-paste:

# Search-style AI crawlers: allow (they cite back)
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

# Training-style AI crawlers: block (consume data without citing)
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

# Content signal: searchable but not for training, not for direct AI input
Content-Signal: ai-train=no, search=yes, ai-input=no

For a full Agent Readiness setup to score 100, see From 3 to 100! How to Get Your Site to Pass isitagentready's AI Agent Readiness Check.

So what does this concretely mean for writing?

Once you understand all the constraints above, there are four things worth specifically working on.

Titles need to stand on their own

At the search stage, all the AI sees about your article is two fields — title and URL.

If your title needs a subtitle or context to make sense, when the AI lines it up against ten other results it'll get skipped.

A quick comparison:

Weaker: "Implementation Notes"
Stronger: "Complete Implementation Notes: Content Negotiation for HTML-to-Markdown in a Cloudflare Worker"

The stronger version packs in "topic, tool, what it does, and article type" — the AI doesn't even need to open the page to know whether it's worth fetching.

Lead with the answer

At the WebFetch stage, the Haiku middle layer reads top-down. The first 300–500 characters decide what it summarizes back. If your opening is "Before we get into X, let's recap a bit of history…", Haiku reads halfway through and discovers the intro is all background and no answer — so it just summarizes the background.

The right move is to make the first sentence of every H2 a direct conclusion, then add the context after. I covered this principle in Is SEO Not Enough? Meet AEO — Getting Your Site Found by AI Search too — worth reading alongside this.

Design single sentences that can be quoted standalone

cited_text is 150 characters on web_search and 125 characters on Claude Code's built-in WebFetch. That means when the AI quotes you, the slot it has is one short sentence that "makes sense without context."

Consciously design sentences like that. For example:

Weaker: "This is a bit different from the method mentioned earlier — the main difference is…" (makes no sense without context)
Stronger: "llms.txt was proposed by Answer.AI co-founder Jeremy Howard in 2024, with the goal of proactively telling AI what important content a site has." (stands on its own without surrounding text)

After writing a paragraph, pick one sentence and ask yourself: if someone who hadn't read the rest of the post saw just this sentence, would they get it? If yes, it has a shot at being quoted.

Use structure as Haiku's navigation markers

H2, H3, - lists, tables, `js code blocks — these Markdown structures are especially useful for the middleware summary layer. When Haiku reads the Markdown converted from your HTML, it treats headings as "what this section is about" indexes, lists as "main points" signals, and tables as "supporting data" units.

If your whole article is pure prose paragraphs, Haiku has no markers and has to grind through it semantically — what comes out is scattered. If you have clear structure, Haiku can summarize along the markers, and the result lines up with the points you actually want quoted.

Wrap-up

So how much of your blog does AI search actually pull?

The answer breaks down into three layers:

Search stage: title + URL only
Body fetch stage: the API default can fit your whole article, but Claude Code goes through Haiku summarization with a 100 KB cut-off
Citation stage: web_search is 150 characters, Claude Code WebFetch is 125 characters

Writing for AI search means targeting those three gates — it's not about getting the AI to memorize your entire post.

If your blog hasn't set up AI bot routing yet, copy the robots.txt template above to get the basics in place — the rest is just content over time.

Is AI Getting Quietly Dumber? A 24/7 Benchmark That Catches LLM Degradation

Ray — Thu, 18 Jun 2026 03:54:54 +0000

You've probably hit this before — yesterday the AI felt sharp, fixed your bug without you even asking, and threw in a few extra cleanups along the way. Then today, same kind of problem, and suddenly it refuses to touch anything you didn't explicitly point at, or starts going in circles. And you start wondering "wait, is AI getting quietly dumber?"

A lot of people have this feeling but it's hard to pin down. Is the provider messing with the model? Is today's problem genuinely harder? Or is it all in your head? Without a number to point at, it just stays an urban legend.

So this post is about a site called AIStupidLevel — the name pretty much spells it out. It runs round-the-clock tests against the major AI models and turns "is AI getting dumber" into an actual curve you can look at.

Why would an AI "get dumber" in the first place?

A reasonable pushback: doesn't the model get trained once and then frozen? A few things are actually going on behind the scenes:

Silent updates. If you've worked with the API, you've probably seen version strings like gpt-4.0-20240924 — that trailing date hints at a specific build. But when you pick a model inside Codex or similar tools, you don't see that level of detail. Some models don't even have versioned IDs, just a generic gpt-4. So you can't actually tell which build you're hitting, and a silent swap is really hard to notice.
Quantization. To handle global peak traffic and save compute, providers sometimes compress the model.
Throttled compute. Once usage crosses a threshold, providers may cap how much compute each user can pull, which makes outputs feel worse.
Compute migration. When a new model is about to launch, providers sometimes shift compute away from the older model. From my own time hitting the API, this is the one I see most — the same prompt suddenly degrades overnight, then a few days later a stronger version drops, and shortly after the old version goes back to normal.

I covered quantization more thoroughly in Want to Run an LLM Yourself? Understanding Model Parameters and Quantization So You Stop Picking the Wrong Model — short version, it's a compression technique that saves resources but loses some detail.

So an AI model isn't actually a frozen thing. Providers can silently update it, quantize it, throttle it, or migrate compute away from it — and any of those will change what you experience. The hard part is, you usually can't tell.

So what is AIStupidLevel exactly?

AIStupidLevel is a third-party benchmark platform (open source, MIT) that continuously monitors whether AI models are regressing. Think of it like a system health check, except it's checking the "health" of AI models. It runs 24/7 against 21 production models from 7 providers — OpenAI, Anthropic, Google, xAI, DeepSeek, Kimi, GLM — and turns each model's current performance into a score on a dashboard.

It's not run by any AI company, which matters here. You don't want the people grading the models to also be selling them.

How does it actually test?

The core idea is simple:

Fixed question bank, run repeatedly. It maintains a fixed set of tasks, throws them at each model on a schedule, and logs the scores.
Same task N times. Because model outputs are stochastic, it runs each task 5 times and takes the median, plus a 95% confidence interval.

Four suites on a rotation

It doesn't just run one kind of test — it has four suites taking turns, each watching for different things:

Test Suite	Frequency	What it tests
Speed (coding)	Every 4 hours	147 coding problems, overall coding ability
Deep reasoning	Daily	5–7 turn multi-turn dialogues, checks long-conversation logic
Tool calling	Daily	Spins up a real Docker sandbox so the AI can actually run multi-step `execute-command` / `read-file` / `write-file` flows
Drift detection (canary)	Hourly	12 lightweight quick checks, first line of defense

The hourly canary plays sentinel — if something starts looking off, it sounds the alarm. The daily deep reasoning and tool-calling runs are the heavier full-body checkup.

The tool-calling suite is the one I find especially interesting. It actually spins up a Docker sandbox (think of it as an isolated mini-computer) and has the AI run real commands inside it, instead of just "verbally" claiming it can use tools. The results end up much closer to what you actually feel when using AI to write code.

Scoring isn't just right vs wrong

A single coding task gets scored across 9 weighted dimensions:

Dimension	Weight
Correctness	40%
Complexity	20%
Code Quality	15%
Stability	10%
Efficiency	5%
Edge Cases	3%
Debugging	3%
Format	2%
Safety	2%

Correctness clearly dominates, but even if the answer runs, you'll still lose points if the code is a mess, misses edge cases, or spews garbage formatting.

How does it catch the moment a model starts getting dumber?

Just having a score isn't enough — the score naturally bounces around because AI is non-deterministic. The real question is: is this drop a real regression, or just noise?

This is where the project gets technically interesting. It uses an algorithm called CUSUM — short for Cumulative Sum Control Chart. CUSUM didn't come from AI research; it's an old quality-control method from manufacturing. The idea is to keep accumulating the gap between observed performance and the baseline. Once the accumulated gap crosses a threshold, you call it: this isn't noise, something actually changed.

On top of CUSUM, it also runs statistical significance testing (checking whether the difference is statistically meaningful, p-value below 0.05) as a second pass, to keep false alarms down.

The real win: with this statistical machinery, a degradation can be detected within hours of starting, rather than waiting until people are venting on social media that "AI got dumber." The system has 29 warning categories built in for different anomaly patterns.

Reading the Stupid Meter

Each model has a live 0–100 score next to it — higher is better. Next to the score is a status tag telling you which of four states the model is in:

STABLE — performing normally
VOLATILE — jittering
DEGRADED — already worse
RECOVERING — climbing back up

Beyond the current score, it lays out each model's historical curve, so you can compare scenarios like "this model was rock-solid last week, why is it jumping around this week."

In practice it feels more like a stock-trading dashboard — you're not looking at a static ranking, you're checking "right at this moment, which model is worth using." If you're still wrestling with which AI tool to pick in the first place, my earlier post Which AI Coding Tool Should You Pick in 2026? pairs well with this one.

Smart Router: route around degraded models automatically

Beyond monitoring, AIStupidLevel also ships a feature called Smart Router, which is a pretty interesting extension of the project.

It's an OpenAI-compatible API endpoint, meaning code you wrote against OpenAI barely needs to change. You drop each provider's API key into it (stored with AES-256 encryption), point your base URL at it, and it routes each request to whichever model is currently in the best shape based on the live monitoring.

It offers six routing strategies — just set the model field in your API call to one of these and it picks dynamically:

auto-best — overall pick, whichever model has the best combined score right now
auto-coding — best at coding right now
auto-reasoning — strongest at reasoning
auto-creative — leans toward creative output
auto-cheapest — cheapest model above the quality bar
auto-fastest — fastest model above the quality bar

If you just pass auto, it uses whichever strategy you've set as your default. And if you pass a specific model name (like claude-opus-4-7), it pins that model directly and skips the router. So when a model quietly starts slipping and Smart Router catches it, traffic auto-routes to a sibling that's still healthy. Pretty useful if you actually want to wire AI into a product.

Wrap-up

So what is AIStupidLevel?

An independent, open source (MIT) third-party benchmark site monitoring 21 production AI models across 7 providers (OpenAI, Anthropic, Google, xAI, DeepSeek, Kimi, GLM), 24 hours a day
Method: fixed question bank, run repeatedly — each task runs 5 times, median plus 95% confidence interval, across four test suites on a rotation
Uses CUSUM change-point detection plus statistical testing to catch a model quietly degrading within hours
Ships Smart Router that auto-routes API traffic to whichever model is in the best shape based on live monitoring

Next time you feel like the AI suddenly got dumber, don't jump straight to blaming yourself — pop the dashboard open and you might actually find the evidence.

Claude Code Auto Memory: Stop Re-Explaining Your Preferences Every Session

Ray — Wed, 17 Jun 2026 02:50:33 +0000

If you've spent any real time with Claude Code, you know this pain: you open a fresh session and it's forgotten everything you taught it yesterday. I'd tell it over and over not to dump a pile of emojis into my writing because it looks too AI — and the next morning, fresh session, emojis everywhere again.

Turns out there's a fix built for exactly this, called Auto Memory. I'd already written about CLAUDE.md for setting project rules by hand, but Claude Code's memory system is bigger than just CLAUDE.md. Auto Memory quietly records your corrections and preferences as you work, so next session you don't have to say them again.

Auto Memory needs Claude Code v2.1.59 or newer. Check with claude --version.

Auto Memory vs CLAUDE.md

A lot of people mix these two up, so let's get it straight first.

	CLAUDE.md	Auto Memory
Who writes it	You	Claude, automatically
What's in it	Instructions and rules	Learned lessons and patterns
Scope	Project / user / org	Per working tree
Loading	Fully loaded every session	First 200 lines / 25KB only
Best for	Code conventions, workflow, architecture	Build commands, debugging notes, preferences

In short: CLAUDE.md is the set of rules you write for Claude on purpose — "use 2-space indentation," "write commit messages in English." Auto Memory is what Claude picks up on its own while working with you. You don't write it; it does.

What Auto Memory actually does

Auto Memory is basically Claude taking its own notes. When you correct it, or tell it whether an approach worked, Claude decides whether that's worth remembering for later. If it is, it saves it.

So when I told it:

Don't use so many emojis, it looks too AI

it saved that as a memory file, and every new session loads it back. Now it remembers I don't like emoji soup.

It doesn't save everything, though — one-off discussions or things irrelevant to future work don't get written down.

Where it's stored (this is the confusing part)

Here's the thing that trips everyone up. Your project has a .claude/ folder, but Auto Memory files are not in there. The project .claude/ only holds settings (settings.json, settings.local.json) and rules.

The actual memory lives under your home directory:

/Users/ray/my-project/.claude/                              <- project settings, inside your repo
/Users/ray/.claude/projects/-Users-ray-my-project/memory/   <- Auto Memory, under your home dir

Both paths contain .claude, but one is in your project and one is in your home directory — don't mix them up. If you peek into your project's .claude/ and only find settings.local.json, that's normal. The memory was never there.

The memory path is derived from your git repo, so subdirectories and worktrees of the same repo share one memory. And it's machine-local — it does not sync to other machines or cloud environments.

If you want it somewhere else, set autoMemoryDirectory (Policy / Local / User level only — not project level, on purpose, so a shared repo can't redirect your memory somewhere shady):

{
  "autoMemoryDirectory": "~/my-custom-memory-dir"
}

The directory structure

~/.claude/projects/<project>/memory/
├── MEMORY.md          # index, loaded every session
├── debugging.md       # debugging notes
├── api-conventions.md # API design decisions
└── ...                # other topic files Claude creates

MEMORY.md is the index — Claude uses it to track which memories exist and which file each one lives in. Only the first 200 lines / 25KB of MEMORY.md loads at session start, so Claude keeps it lean and pushes detail into separate topic files. Those topic files aren't loaded on startup; Claude reads them on demand. So your memory can grow without bloating context every single time.

(That 200-line cap is only for MEMORY.md. CLAUDE.md always loads in full — though shorter still means better adherence.)

How do you know it's writing memory?

When you see "Writing memory" or "Recalled memory" in the Claude Code UI, that's it updating or reading memory files. You don't have to do anything; it handles it.

Managing it

/memory — the main entry point. See what's loaded, toggle Auto Memory on/off, open the memory folder, or open any file in your editor.
Just ask — "remember: always use pnpm, never npm" saves a memory. "forget the rule about pnpm" updates or deletes it.
Edit by hand — they're plain Markdown files under ~/.claude/projects/. Add, edit, or delete whatever you want.
Turn it off — toggle it in /memory, set "autoMemoryEnabled": false, or set CLAUDE_CODE_DISABLE_AUTO_MEMORY=1.

And in case you were wondering: /compact does not wipe it. The files are on disk, not in the conversation. Compacting only squeezes the chat history; MEMORY.md and CLAUDE.md get re-read from disk and injected back afterward.

That's the whole thing. It's a small feature, but once you stop repeating yourself every session it's hard to go back.