A client asks whether ChatGPT can see their website. The question sounds simple, but it mixes several things: the policy the site publishes, whether a request can reach the server, whether content is indexed, and whether a particular answer cites it.
I would begin with the part we can inspect directly: the site's published crawler policy. That gives the technical team something concrete to review before anyone promises a change in AI visibility.
My AI Crawler Access Checker reads robots.txt, llms.txt, and llms-full.txt for public websites. It reports policy evidence for 16 named bots and keeps unavailable sources visible as unknown outcomes. The result is a review queue with source links, rather than a claim that the site will appear in an AI answer.
Begin with one website
The input can be small:
{
"websites": ["example.com"],
"maxConcurrency": 1
}
This demonstrates the input shape. No fresh request to example.com was made for this article. Replace it with a public domain you are responsible for reviewing, inspect the first result, and then add the rest of the portfolio. The documented input supports up to 100 requested sites per run.
The Actor reads conventional public policy paths. It does not execute JavaScript or run a complete technical SEO crawl. That narrow scope makes the output easier to interpret, as long as you keep the same boundary when discussing it with a client.
Look at individual bot verdicts
The following is a reduced, illustrative fragment from the documented example output. It is not a live observation of example.com:
{
"website": "https://example.com",
"robotsTxt": {
"exists": true,
"status": 200,
"state": "available"
},
"bots": [
{
"bot": "GPTBot",
"purpose": "training",
"verdict": "blocked"
},
{
"bot": "OAI-SearchBot",
"purpose": "search",
"verdict": "allowed"
}
],
"safeToAutomate": false
}
This is the distinction I would put in the review ticket. A policy can name training and search bots separately. A vendor-level statement such as “OpenAI is blocked” loses that detail and can send the team toward a change nobody intended.
Keep the individual verdicts, matching policy evidence, and observation time alongside any summary score. The full documented output also includes evidence URLs, confidence, gaps, and a recommended action. A score is convenient for sorting a portfolio; the underlying rules explain what someone might need to change.
Treat unknown as a useful result
An unavailable robots.txt endpoint should not become an all-clear. In the documented contract, an unavailable core policy produces unknown verdicts and null scores. That tells the team to investigate the source or retry after recovery.
A known absence is different. If the endpoint returns a recognized 404 or 410, the Actor can report that policy state and represent default access. A timeout and a confirmed missing file therefore lead to different interpretations. Keeping them separate is more useful than forcing both into a red or green badge.
Partial rules also deserve a careful reading. A bot may be allowed at the root while some paths are restricted. Review those paths against the content the client cares about. A categorical partial verdict is not a measurement that half of the site is accessible.
Turn observations into a small review queue
For each website, store the source URL, collection time, bot verdicts, restricted paths, and recommended action. Assign the ticket to the person who owns crawler policy. Ask them to compare the observed rules with the site's intended policy for training, search, and user-triggered access.
If the team approves a change, preserve the previous observation, deploy the change through the site's normal process, and collect another snapshot. Check the relevant bot and path again. That creates evidence of a policy change you can explain without suggesting that a visibility gain has already happened.
Keep policy publication under human control. The output's safeToAutomate: false is a useful boundary for an agent-driven workflow: collecting and prioritizing observations can be automated, while a live policy change needs context and approval.
Keep the measurement question separate
An allowed policy does not establish that a crawler obeyed it, fetched a particular page, indexed the content, or used it in an answer. The Actor also does not test every real bot IP against a site's firewall or inspect all page-level indexing controls.
Likewise, finding llms.txt confirms an observed document at a conventional path. It does not prove adoption by a vendor or a ranking improvement. If the business question is about citations, referrals, or purchases, collect those outcomes separately and state what each measurement actually observed.
Calculate the audit cost
In the November 10, 2026 Store snapshot, Free-plan event rates were $0.005 per Actor-start event and $0.02 per successfully delivered unique website audit. Start-event count depends on memory, with a minimum of one event. At one start event, ten billable audits have an event subtotal of $0.205; 100 have a subtotal of $2.005.
Unknown or failure outcome rows are free at the result level; the start fee remains separate. A known missing policy can be a completed, billable audit. These are dated event examples. Confirm current Store pricing and your Apify plan before committing a larger schedule.
Start with the AI Crawler Access Checker, inspect one domain, and write one evidence-backed review ticket. Expand to a portfolio once the team agrees on what the observations mean and who owns the next action.
Disclosure: I maintain this Actor. This draft was generated with AI assistance from product documentation and a dated Store snapshot; no new live collection was performed for this article.
Top comments (0)