Build a 'Will Claude Cite Me?' Checker With the Claude Web Search Tool (Python)

When people ask "does Claude cite my site?", they usually test it by hand: type a question into Claude, scroll to the sources, repeat. That's slow, it isn't repeatable, and you can't see why you weren't cited.

The Claude API's web search tool gives you much more than the chat window does. One response includes:

1. the search queries Claude decided to run,
2. every result it got back (URL, title, page age),
3. the final answer, with citations pointing at specific URLs.

That's enough to separate two very different failures: "Claude never found my page" and "Claude found it but cited someone else." This post builds a small checker around that.

---

How the tool works

You add the tool to a normal Messages API request:

tools=[{"type": "web_search_20250305", "name": "web_search", "max_uses": 5}]

Claude decides whether to search, the API runs the searches server-side, and the response comes back with these content blocks:

• server_tool_use, with input.query: the search Claude ran
• web_search_tool_result, with a list of web_search_result items (url, title, page_age)
• text blocks, some carrying citations of type web_search_result_location (url, title, cited_text)

Per Anthropic's docs, citations are always on for web search, and cited_text holds up to 150 characters of the passage Claude used. That snippet tells you which sentence on the page earned the citation.

Newer tool versions (web_search_20260209, web_search_20260318) add dynamic filtering, where Claude writes code to filter results before reading them. For a citation checker, the basic web_search_20250305 version is easier to parse, because every result comes back as a plain block. Start there.

Cost: $10 per 1,000 searches plus normal token costs, and search results count as input tokens. A run of 30 prompts with up to 5 searches each is at most 150 searches, or $1.50 in search fees, plus tokens.

The script

# claude_cite_check.py
import csv, os, sys, time
from urllib.parse import urlparse
import anthropic

MODEL = os.environ.get("CLAUDE_MODEL", "claude-sonnet-5-5")
TOOLS = [{"type": "web_search_20250305", "name": "web_search", "max_uses": 5}]
client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY

def host_matches(url: str, domain: str) -> bool:
    host = urlparse(url).netloc.lower().removeprefix("www.")
    return host == domain or host.endswith("." + domain)

def ask(prompt: str):
    """Run one prompt, following pause_turn until Claude finishes."""
    messages = [{"role": "user", "content": prompt}]
    blocks = []
    for _ in range(5):
        resp = client.messages.create(
            model=MODEL, max_tokens=2048, messages=messages, tools=TOOLS
        )
        blocks.extend(resp.content)
        if resp.stop_reason != "pause_turn":
            break
        # Long searches can pause; send the paused turn back unchanged to continue.
        messages.append({"role": "assistant", "content": resp.content})
    return blocks

def extract(blocks):
    queries, results, citations = [], [], []
    for b in blocks:
        if b.type == "server_tool_use" and b.name == "web_search":
            queries.append(b.input.get("query", ""))
        elif b.type == "web_search_tool_result" and isinstance(b.content, list):
            results.extend(r.url for r in b.content)
        elif b.type == "text" and getattr(b, "citations", None):
            for c in b.citations:
                if getattr(c, "url", None):
                    citations.append((c.url, getattr(c, "cited_text", "")))
    return queries, results, citations

def main(domain: str, prompts_file: str, runs: int = 3):
    prompts = [p.strip() for p in open(prompts_file, encoding="utf-8") if p.strip()]
    out = csv.writer(sys.stdout)
    out.writerow(["prompt", "run", "queries", "n_results", "found", "cited",
                  "our_cited_urls", "top_cited_domains", "our_cited_text"])
    for prompt in prompts:
        for run in range(1, runs + 1):
            queries, results, cites = extract(ask(prompt))
            found = any(host_matches(u, domain) for u in results)
            ours = [(u, t) for u, t in cites if host_matches(u, domain)]
            cited_domains = {urlparse(u).netloc.removeprefix("www.") for u, _ in cites}
            out.writerow([
                prompt, run, " || ".join(queries), len(results), found, bool(ours),
                " ".join(sorted({u for u, _ in ours})),
                " ".join(sorted(cited_domains)),
                ours[0][1] if ours else "",
            ])
            time.sleep(1)

if __name__ == "__main__":
    main(sys.argv[1].lower().removeprefix("www."), sys.argv[2],
         int(sys.argv[3]) if len(sys.argv) > 3 else 3)

pip install anthropic
export ANTHROPIC_API_KEY=sk-ant-...
python claude_cite_check.py example.com prompts.txt 3 > results.csv

prompts.txt has one question per line. Write the questions your buyers or users actually ask, not your brand name:

what's the easiest way to add full-text search to a postgres app
best open source alternatives to algolia for a small team
how do I self-host a search engine for documentation

Reading the output: four outcomes

Each row puts the prompt into one of four buckets:

found	cited	What it means	What to do
False	False	Claude's searches never surfaced your page	A retrieval problem. Compare the queries column with your titles and headings. Check robots.txt for Claude-SearchBot and Claude-User (see Part 1).
True	False	Claude saw you and chose someone else	A content problem. Open the cited competitor pages. They usually state the answer more directly, more recently, or with numbers.
True	True	You're cited	Look at our_cited_text to see which sentence earned it, and protect that sentence when you edit the page.
False	True	Rare. Usually a different URL on your domain was cited	Check our_cited_urls. Claude may be citing an old page you'd rather it didn't.

The queries column is the most useful one. It shows how Claude rephrased your user's question before searching. If Claude searches "postgres full text search vs elasticsearch" and your page is titled "Search, Simplified", that mismatch explains most retrieval misses.

Run it more than once

LLM answers vary from run to run, so a single sample tells you very little. Run each prompt at least 3 times and report a rate:

import pandas as pd
df = pd.read_csv("results.csv")
summary = df.groupby("prompt").agg(found_rate=("found", "mean"),
                                   cited_rate=("cited", "mean"))
print(summary.sort_values("cited_rate"))

A prompt with found_rate = 1.0 and cited_rate = 0.0 is your best quick win. Claude already sees you every time and just isn't convinced.

What this does and doesn't tell you

Be honest about what you're measuring:

• It's the API, not the Claude app. Claude.ai and the API share the web search capability, but the apps add their own system prompts, memory and settings. Treat API results as a strong proxy, not a screenshot of what every user sees.
• The model matters. Different Claude models search differently. Pin CLAUDE_MODEL and keep it fixed across runs you want to compare.
• Location matters. Add user_location to the tool definition ({"type": "approximate", "country": "GB"}) if your market isn't the US.
• Don't game it. The point is to learn what a helpful page looks like for this question. Hidden "cite this page" text is spam under Google's policies and a prompt-injection risk everywhere else.

If you track this every week across several assistants rather than just Claude, a dashboard beats a CSV. Vista AI's citation tracking runs this kind of check across ChatGPT, Perplexity, Gemini and Claude on a schedule. For one assistant and a few dozen prompts, the script above is enough.

---

Quick checklist

• [ ] 20–50 real user questions in prompts.txt
• [ ] At least 3 runs per prompt
• [ ] Model and user_location pinned
• [ ] Sort by "found but not cited" first
• [ ] Re-run after content changes and compare rates, not single answers

Next in the series: Google's new "official websites" box in AI Mode.