People find information in both search engines and AI answers. I account for both when working on my site. For a developer, that starts with a narrower problem: can the relevant crawler reach a page, and is that page eligible to appear in search?
This walkthrough uses public HTTP endpoints, Python's standard library and Google Search Console. The example checks a real robots.txt; the later inspection steps need access to your own Search Console property. It does not test a model's choice of sources or guarantee an AI citation.
Start with the states you can observe
A URL moves through several separate stages. A search engine needs to discover it, retrieve its content and decide whether to index it. Google says a page must also be eligible for a search snippet to be considered as a supporting link in its AI Overviews and AI Mode. Even then, eligibility is not a promise that the page will be selected. The conditions are described in Google's documentation for AI features.
GEO, or generative engine optimization, and AEO, or answer engine optimization, are names used for work on visibility in generated answers. For Google's AI features, the documented starting point is ordinary SEO: accessible content, internal links and pages that meet the search technical requirements. There is no separate mandatory AI markup in that guidance.
My own project gives two different measurements: Google's index grew from zero to 117 pages, and a list of 167 search queries guides topic selection. The first is an index figure; the second is an input to planning. Neither measures enquiries or proves that an AI answer cited the site.
Check the crawler policy
OpenAI documents separate user agents for search and training. OAI-SearchBot is used to surface sites in ChatGPT search, while GPTBot crawls content that may be used to train foundation models. The permissions can be set independently. OpenAI's crawler reference also describes ChatGPT-User, which makes requests on behalf of users and is not the automatic search crawler.
The public robots.txt on my site explicitly allows OAI-SearchBot and GPTBot. You can check the rules for a URL with Python 3 and its standard library. Enter the full page URL when prompted. No package install or credentials are needed:
import urllib.parse
import urllib.robotparser
url = input("Page URL: ").strip()
robots_url = urllib.parse.urljoin(url, "/robots.txt")
robots = urllib.robotparser.RobotFileParser(robots_url)
robots.read()
for bot in ("OAI-SearchBot", "GPTBot"):
print(f"{bot}: {robots.can_fetch(bot, url)}")
For the live file checked while preparing this article, the output was:
OAI-SearchBot: True
GPTBot: True
This checks the parser's interpretation of the retrieved rules for that particular path. Change the URL to the page you want to examine: a policy may permit the homepage and restrict a subdirectory. A later change to robots.txt can also change the result. A True value does not prove that the crawler has visited the page, that the server served it successfully, or that the page was used in an answer.
The example reads a public site only. For production monitoring, add an explicit request timeout and record HTTP errors separately from a policy decision. A timeout, redirect problem or inaccessible robots.txt should be investigated; don't turn a network failure into a claim that a crawler is allowed or blocked.
Inspect the page as a separate request
Open the target page without a login and inspect the response as well as what the browser renders. A successful robots.txt check says nothing about whether a CDN, authentication wall or server error prevents a crawler from receiving the page. Check the final URL after redirects, and make sure the content that answers the query is available as text.
Look at both HTML meta robots directives and HTTP X-Robots-Tag headers. A noindex directive asks a compliant search engine not to index the page. Snippet restrictions such as nosnippet can affect whether it is eligible to appear with a snippet. Google's AI feature documentation lists the relevant controls; verify the actual policy for your site before changing them. A page intentionally excluded from search should not be opened up merely to improve an SEO check.
Use URL Inspection in Search Console for the exact URL. Compare the indexed state with the live test and the HTML Google received. They answer different questions: a live test can show what is reachable now, while the indexed version reflects what Google has already processed. If you change a directive or redirect, allow time for a recrawl and inspect again. A request for indexing is a request, not confirmation that indexing happened.
Check discovery without treating the sitemap as an index
My site's sitemap endpoint returns XML, and its address is listed in robots.txt. A sitemap provides discoverable URLs; it does not make every URL eligible for indexing. Check that a target URL is the preferred live address and that pages elsewhere on the site link to it. When a page moves, test the redirect from the old address and update internal links.
For a bilingual site, inspect the English and Russian URLs independently. A working page in one language does not establish that its counterpart is reachable, indexed or linked correctly. Keep the language and final URL in your notes so a change to one version is not mistaken for a result on the other.
Keep search and answer observations apart
In Search Console, track the page's impressions, clicks and queries. Google includes traffic from its AI search features in the overall Web search reporting, so do not assume the normal report gives you a separate count of citations in AI answers. Check enquiries separately if that is a business outcome you measure.
For a repeatable manual observation, save the exact question, date, AI product and whether a link to the page appeared. Repeat the same question later and record the result again. Responses can vary, and a missing link in one answer is not proof of a crawling fault. If the page is absent, first check access, indexing and whether its content answers the question, then revisit the observation.
The broader context for these checks is in my article on AI SEO, GEO and AEO. The useful output of a diagnostic run is a set of distinct states: crawler permission, page response, index status and observed search performance. Keep the unknowns visible instead of collapsing all of them into a single "AI ready" flag.
This article was created with the help of AI.
Top comments (0)