DEV Community: Edward Chapman

Robots.txt allows OAI-SearchBot. That does not prove ChatGPT can fetch your page.

Edward Chapman — Sat, 03 Oct 2026 07:37:20 +0000

A line in robots.txt that allows OAI-SearchBot grants permission to a compliant crawler. It does not tell you whether your CDN lets the request through, whether the response contains your content, or whether the page appears in ChatGPT answers. Each question needs its own evidence.

Two OpenAI user agents, two decisions

OpenAI documents OAI-SearchBot for ChatGPT search and GPTBot for training. Blocking GPTBot alone does not automatically block OAI-SearchBot. To allow search crawling and disallow training crawling, use separate groups:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

These rules state crawler permissions. They do not override firewall rules or guarantee training exclusion in every context. Check OpenAI's documentation for the current scope of each agent:
https://platform.openai.com/docs/bots

Check the rule that applies

Start with the hostname. The file at example.com/robots.txt does not govern www.example.com or shop.example.com.

Then check the exact path. Plain path rules match by prefix: Disallow: /pricing also matches /pricing-old and /pricing/eu.

Check the user-agent group too. Under standard matching, a crawler that finds a group naming it follows that group rather than falling back to User-agent: *. Review all groups for the named agent.

Permission to crawl is not indexing or citation. Robots.txt manages crawler traffic; it is not authentication or access control. Anyone can read the file and ignore it, so confidential data needs proper access controls.

Google explains the scope and limitations here:
https://developers.google.com/search/docs/crawling-indexing/robots/intro

The edge can refuse first

A CDN or firewall can block or challenge a permitted request before it reaches your server. Your origin logs may show nothing about that request.

Review edge security events and origin logs separately. Record the hostname, path, time, action or status, and the rule that matched. If a rule catches traffic you meant to allow, adjust that specific rule. Do not disable site-wide protection to troubleshoot one crawler.

Cloudflare's documentation, checked on 2 October 2026, separates Search, Agent and Training categories. An Allow setting in the AI bot policy does not remove other WAF controls. Published defaults do not tell you how an existing zone is configured. Check your own settings:
https://developers.cloudflare.com/bots/additional-configurations/block-ai-bots/

A copied user agent tests your request, not theirs

Running curl with OAI-SearchBot's user-agent string sends a request from your machine and IP address. If the edge uses IP ranges or verified-bot status, it may respond differently to the real crawler.

A 200 from your laptop does not prove the crawler gets a 200. A 403 does not prove it gets a 403. The User-Agent header comes from the client, so it does not verify identity.

Use the verification information OpenAI currently publishes and compare it with your edge request records. If no verified requests appear in the period you checked, report that they were not observed. That does not establish that they were blocked.

A 200 is not necessarily your content

A 200 response can contain a login page, a challenge interstitial, or a JavaScript shell with little text. Where response evidence is available for a verified crawler request, compare what was served with the expected public page.

Status and response size are clues, not proof that the intended text was delivered. Standard access logs usually do not contain response bodies. Even confirmed delivery does not prove the provider ingested the page.

Keep four findings separate

Declared permission: the matching robots.txt group and path rule. This does not show that a request arrived.

Observed access: edge or origin evidence of a verified crawler request and its response. This does not show that the content was indexed or used.

Provider indexing: signals the provider publishes, where available. This does not promise a citation.

Cited answer: an actual answer linking the page. This does not show that the citation will recur or appear for other queries.

A practical checklist

Read robots.txt on the exact hostname and find the group and path rule for OAI-SearchBot.

Check GPTBot separately against your training decision.

Review edge security events for that hostname and path, then origin logs.

Verify crawler identity using provider documentation, not the User-Agent string alone.

For verified requests, record the action, status, matching rule, and available evidence of the content served.

Report anything you could not confirm as unknown. Access evidence alone cannot establish that ChatGPT cited a page.

An optional check for the robots.txt part

I'm with Firm Beacon. Our free browser checker compares robots.txt permissions for search and training agents on a specific URL. You can enter a URL or paste robots.txt without creating an account or giving an email address:
https://www.firmbeacon.co.uk/tools/ai-crawler-check?utm_source=devto&utm_medium=article&utm_campaign=ai_crawler_access

It checks the declared rules, not your firewall. It does not verify indexing, citations or rankings. Use it for the first step, then check access evidence separately.

What to Check Before You Spend More Time on SEO

Edward Chapman — Sat, 03 Oct 2026 03:27:45 +0000

When a page does not perform in search, it is tempting to write more content or start building links. Those may be useful later, but first check whether the site is sending clear technical signals to crawlers and browsers.

A small audit will not explain every ranking change. It can show whether basic page information is present, whether important files are reachable, and whether the sample of pages you checked is consistent.

Start with the site-level files

Begin with HTTPS, robots.txt, and the XML sitemap.

HTTPS is the expected protocol for a public website. Check that the address you use is the HTTPS version and that it loads as a public page.

robots.txt contains crawler rules. A rule can block a path that you intended to make public, so read the file instead of assuming it is correct. The XML sitemap is another useful signal to inspect. It should be available at the address declared by the site and should represent the URLs you actually want discovered.

These files have different jobs. robots.txt controls crawler access to paths. A sitemap lists URLs for discovery. Neither one proves that a page is indexed or will rank.

Check the pages that matter

The homepage is a sensible starting point, but it is not enough. Include a product or service page, an important article, and a page that has changed recently. If the site is larger, use a sample rather than assuming that every template behaves the same way.

The free Firm Beacon Website Audit accepts a public website address and checks up to ten public HTML pages. It follows public links on the same site, so the result is a sample of what the audit can reach from the starting page.

Review the page signals

For each page, look at the title, meta description, and H1 heading. These elements should describe the page that the visitor sees. A missing value is a reason to inspect the template or CMS settings. It is not, by itself, proof of a ranking penalty.

Check the canonical URL as well. It should point to the version of the page that you want search engines to treat as canonical. Pay attention to the hostname and protocol. A page using https://www.example.com and a canonical pointing to a different version deserves a closer look.

The audit also reports the robots meta directive and the X-Robots-Tag response header. These can contain noindex instructions. If a page is meant to appear in search, confirm that an exclusion was intentional.

Language and mobile viewport tags are easy to overlook. The language value gives a page a declared language, while the viewport tag helps a browser size the page for a mobile screen. Open Graph tags control how a page is represented when someone shares it on a social platform. They are useful checks even though they do not establish search rankings.

Internal links matter for navigation and discovery. A small audit can show the links present in the public HTML it reads. Follow up on important pages that are hard to reach from the pages you checked.

Keep the result in context

A technical HTML audit has clear limits. It does not verify Google indexing, rankings, backlinks, Core Web Vitals, firewall access, or JavaScript-rendered content. A page can contain a title and canonical and still have a separate indexing problem. A page can also lack one optional signal without needing an urgent rewrite.

Treat each finding as a question:

Is the value missing on purpose?
Does it match the page's purpose and preferred URL?
Does the same issue appear in the other templates you sampled?
Can you confirm the result in the CMS, response headers, or Search Console?

This keeps a short audit from turning into a list of automatic fixes. The right change depends on the page and the site's publishing setup.

A short checklist

Before changing content or starting outreach, check:

The public address loads over HTTPS.
robots.txt does not block a path you need crawled.
The sitemap is reachable and contains intended public URLs.
Important pages have useful titles, descriptions, and H1 headings.
Canonical URLs use the intended hostname and protocol.
noindex directives are deliberate.
The pages declare language, viewport, and sharing information where needed.
Internal links reach the pages you want visitors and crawlers to find.

You can run this first pass with the free Website Audit. It requires no account or email and reports the signals found in up to ten public HTML pages. Use the result to choose what to investigate next, then confirm important changes with the tools that cover indexing, performance, or server access.

How to create an XML sitemap from a URL list

Edward Chapman — Fri, 02 Oct 2026 17:52:32 +0000

An XML sitemap is a list of public URLs that you want search engines to discover. It can help with discovery, but it does not guarantee crawling, indexing or rankings.

Start with the sitemap your site already has

If your site uses a CMS, check whether it already generates a sitemap. WordPress, Wix and other platforms can maintain one as pages are added or removed. Use that sitemap rather than creating a second file that will go stale.

A manual sitemap makes sense when you have a small site, a curated URL list, or a build process that produces the final URLs separately.

Choose the URLs

Include canonical, public pages that you want search engines to consider. Do not add:

login pages or other private URLs;
URLs that redirect somewhere else;
duplicate versions of the same page;
pages you deliberately exclude from search;
broken or temporary URLs.

Use absolute URLs, and keep the hostname consistent. For example, decide whether the site uses https://example.com or https://www.example.com and do not mix both versions in one sitemap.

Write the XML

The basic sitemap format is small. Each page goes inside a <url> element, and its address goes inside <loc>:

<?xml version=1.0 encoding=UTF-8?>
<urlset xmlns=http://www.sitemaps.org/schemas/sitemap/0.9>
  <url>
    <loc>https://example.com/</loc>
  </url>
  <url>
    <loc>https://example.com/about</loc>
  </url>
</urlset>

Escape characters that have a special meaning in XML. For example, an ampersand in a query string becomes &:

<loc>https://example.com/search?a=1&amp;b=2</loc>

You do not need to add priority or changefreq. Google ignores those fields. Add lastmod only when the date reflects a real, significant update to that URL. Invented dates make the file less useful, so it is better to leave the field out.

Check the limits

A sitemap can contain up to 50,000 URLs or 50 MB when uncompressed. Larger sites can split the URLs across several sitemap files and reference them from a sitemap index.

For a small, already curated list, a browser-local converter is enough. The free XML sitemap generator removes duplicate URLs and fragments, escapes XML characters and downloads sitemap.xml. It does not crawl the site or upload the pasted list.

Publish and submit it

Upload the file to the public location you choose, commonly:

https://example.com/sitemap.xml

Then check the URL in a browser or with an HTTP client. It should return the XML file, not an HTML error page or a login screen.

You can also declare the location in robots.txt:

Sitemap: https://example.com/sitemap.xml

For a site you manage, submit the sitemap URL in Google Search Console. Keep the sitemap current when public URLs are added, removed or replaced.

What a sitemap cannot tell you

A successful upload does not prove that Google has crawled or indexed every listed page. It also does not improve a weak page, override a canonical decision, or guarantee rankings. Review indexing evidence separately in Search Console and check the pages themselves for useful content, access, canonical URLs and appropriate robots directives.

The useful test is simple: include the right public URLs, publish valid XML at a stable address, and then verify what search engines actually did with it.