Getting a brand new site crawled: who actually shows up, and in what order

We put a static site online on 15 October, published nothing for four days, then published three articles in one evening. The access log was recording the whole time. Fifteen days later it holds 9,987 requests, and the order in which the internet noticed us is not the order anybody plans for.

No theory below. Every number is from that log.

The first visitor was an AI crawler, 65 minutes in

The first page went live at 13:14 UTC. The log of what happened next, in order of first appearance:

Crawler	First seen	Requests, week 1	Requests, 15 days
ClaudeBot	15.10, 65 min after launch	226	484
GPTBot	15.10, 70 min after launch	210	391
OAI-SearchBot	15.10, 70 min after launch	33	108
bingbot	16.10	70	164
Googlebot	19.10, 19:41	328	541
GoogleOther	19.10, 23:01	281	520
AhrefsBot	20.10	91	202
ChatGPT-User	21.10	4	39

Nothing linked to the site yet. No submission had been made anywhere. Anthropic's and OpenAI's crawlers found a brand new hostname on a small VPS within the hour, and started walking it.

That is the practical point: if you publish technical writing today, your first readers are retrieval pipelines, and they arrive before any search engine knows you exist. Decide your robots.txt policy before you publish, not after you notice.

Note what ClaudeBot did in its very first two requests, one second apart: /robots.txt, then /. It has asked for robots.txt 78 times in fifteen days, once per crawl session. GPTBot has requested it zero times in the same window, while fetching 391 URLs. Same two seconds of effort, opposite behaviour. We will come back to that in a separate piece, because it decides which blocking method is worth deploying.

Google ignored us entirely until one step was done

Zero Googlebot requests in the first four days. Not throttled, not slow: absent. The site was reachable, had a sitemap, had robots.txt, and was linked from an established domain we also run.

What changed it was verifying the domain in Search Console and submitting the sitemap. From the log, that evening:

• 19:35 - domain verified via a DNS TXT record.
• 19:41:31 - first Googlebot request in the site's history: /robots.txt and /sitemap.xml, in the same second.
• 19:42 to 19:46 - Googlebot walks every per-language sitemap in turn.

Six minutes. You can argue about whether Search Console is formally required - it is not - but the difference between four days of silence and six minutes is hard to ignore when you are starting from nothing.

Use a domain property rather than a URL-prefix property if you can add a DNS record: one TXT at the apex covers every subdomain you will ever add, and it survives an http/https or www change that would otherwise split your data across properties.

Then, three hours and twenty minutes after Googlebot, GoogleOther showed up and has since fetched almost as much as Googlebot itself: 520 requests against 541. It is a separate crawler using the same infrastructure for non-search product work. If you ever decide to restrict Google, know that it is two names, not one.

Bing came through IndexNow in days, with no account at all

IndexNow is a simple deal: you host a key file, then POST a list of URLs you changed, and participating engines (Bing, Yandex, Seznam, Naver) are told immediately instead of waiting for a crawl.

The whole protocol is one request:

curl -X POST https://api.indexnow.org/indexnow \
  -H "Content-Type: application/json; charset=utf-8" \
  -d '{
    "host": "example.com",
    "key": "YOUR-KEY",
    "keyLocation": "https://example.com/YOUR-KEY.txt",
    "urlList": ["https://example.com/new-post/"]
  }'

The key is a file at your domain root whose name is the key and whose content is the same string. That is the entire verification: if you can put a file there, you own the host.

A 200 means accepted, not indexed. But bingbot showed up on its own the day after our first submission, without an account, a console or a verification step - three days before Googlebot. For a new site that is a meaningful head start, and it costs one line in the deploy pipeline.

The index-file trap

Here is the part that nearly fooled us, and it is the most useful paragraph in this post.

Multilingual Hugo generates a sitemap index at /sitemap.xml - a list of other sitemaps, one per language - rather than a list of pages. We submitted that, and Search Console reported:

Status: Success. Discovered pages: 0.

Both halves are true and they look contradictory. The index was fetched and parsed correctly, so the status is a success; the index itself contains no page URLs, so the discovered count stays at zero until the children are fetched separately. A quick glance at "Success" and you walk away believing the job is done.

We submitted the child sitemap for the main language as well:

Status: Success. Discovered pages: 23.

So: if your sitemap is an index, submit the per-language children too, and treat the discovered-pages count, not the status word, as the thing that tells you it worked. The same applies to any generator that emits an index - large WordPress installs with Yoast do, and so does anything paginating its sitemaps past 50,000 URLs.

Day 12: the long tail arrives all at once

For eleven days the log had eight or nine distinct crawlers in it. Then, between 26 and 29 October, fourteen new ones appeared within four days:

ElevenlabsCrawler (448 requests), meta-externalagent (322), MJ12bot (319), AwarioBot (308), SofyaBot (232), SemrushBot (213), Amazonbot (96), PerplexityBot (63), DataForSeoBot (62), YandexBot (49), Applebot (30), archive.org_bot (26), Baiduspider (14), DuckDuckBot (8).

The pattern is clean enough to plan around. SEO backlink crawlers, brand-monitoring crawlers and the second tier of AI crawlers do not discover sites themselves - they read other people's indexes. Once Google and Bing have you, you are in a feed that resells discovery, and the rest follow within a fortnight whether you wanted them or not.

What that costs, measured: of 153.4 MB served in fifteen days, 88.8 MB went to user agents that identify themselves as crawlers. That is 58% of all bytes, against roughly 4,000 browser-like requests. On a static site served from a tiny VPS, nobody notices. On anything that renders pages per request, that is your capacity plan.

The order that actually works

For a new site, in this order, with the reason each step exists:

1. Be linked from somewhere that already gets crawled. One footer link from an existing domain is worth more than any submission form. Discovery is still mostly links.
2. Ship a sitemap and keep robots.txt honest. Point to the sitemap from robots.txt; make sure neither is left over from a staging config that disallows everything.
3. Send IndexNow on every publish, from the pipeline. Cheap, instant, no account. It covers Bing and friends; it does nothing for Google, which does not participate.
4. Verify the domain in Search Console and submit the sitemap - including the children. This is the step that turned four days of nothing into six minutes.
5. Then wait. Being crawled is not being indexed, and being indexed is not ranking. The first two are infrastructure and take minutes; the third is content and takes months.

What we would do differently

Submit the sitemap to Search Console on day one, before any content exists. Verification is a DNS record and takes five minutes; doing it early means the crawler already knows the site when the first article appears, instead of starting the clock after it. We spent four days publishing into a void we could have avoided.

The IndexNow ping we did fix: it now runs as a step in the deploy workflow, in the same place as the post-deploy smoke test, so a publish that reaches the server always reaches the engines too. Doing it by hand means forgetting it exactly once, on the article that mattered.

What this log cannot tell you

It is one site, one niche, one fifteen-day window, on a hostname with no history. Crawl budget for an established domain behaves differently. We also cannot separate "arrived because of Search Console" from "arrived because it was the fourth day" with a sample of one - the six-minute gap is suggestive, not a controlled experiment. And robots.txt fetch counts measure fetches, not obedience: a crawler that reads the file and ignores it looks identical to a polite one in this data.

---

Originally published at news.rozoom.app.