DEV Community

Daniel Pertu
Daniel Pertu

Posted on

Every page we serve advertises an RSS feed our own robots.txt forbids

#seo #typescript #webdev #nextjs

CogniPrep's robots.txt is generated by a 122-line app/robots.ts and arrives at the edge as 850 lines and 24,687 bytes. 48 of those lines are Allow and 796 are Disallow, because the whole policy is written twice: once for User-Agent: * and once for User-Agent: Bingbot, which crawls more predictably when it has an explicit group. The two groups hold 422 rules each and are identical, rule for rule.

A rules file that size is not something you read. It is something you query. So I wrote the matcher.

The matcher

Three rules are enough for the real thing:

  1. A pattern is a literal prefix, with * matching any run of characters and $ anchoring the end.
  2. Of all matching rules, the longest pattern wins.
  3. If an Allow and a Disallow tie on length, Allow wins. 
def decide(rules, path):
    best = None
    for kind, pattern, regex in rules:
        if regex.match(path):
            length = len(pattern.replace("*", "").replace("$", ""))
            if best is None or length > best[0] or (length == best[0] and kind == "allow"):
                best = (length, kind, pattern)
    return best or (0, "allow", "(default)")
Enter fullscreen mode Exit fullscreen mode

Then I fed it every URL in our live sitemap. All 530 are allowed, in both groups. That is the result you want and it is also the boring one, so I ran the inverse query: what does this file block that we also point at?

1. Our own feed 

Disallow: /feed.xml
Enter fullscreen mode Exit fullscreen mode

The root layout puts this in the <head> of every page:

<link rel="alternate" type="application/rss+xml" title="CogniPrep Blog RSS Feed" href="/feed.xml" />
Enter fullscreen mode Exit fullscreen mode

I sampled 25 of the 530 live URLs and all 25 carry it. The feed itself answers 200 and lists our articles. So every page on the site tells a crawler the feed exists, and the rules file in the same deployment tells that crawler not to fetch it.

A feed reader a person has subscribed in their own client is not affected. A search engine that discovers feeds by following rel="alternate" is: it finds the pointer and then declines to read the target. The Disallow was added with /manifest.ts in a block commented "Feed and manifest files", and nothing was pointing at the feed at the time.

2. A wildcard that reaches further than it looks 

Disallow: /*.json
Enter fullscreen mode Exit fullscreen mode

That is 6 characters of pattern after the wildcard is stripped, which beats Allow: / at 1. It covers the JSON our API returns, which is the intent, and it also covers /.well-known/assetlinks.json, the file Android reads to associate the installed app with the domain. Android does not consult robots.txt, so nothing breaks. It is still a rule with a wider blast radius than its comment suggests.

3. The 372 rules that are right

The longest block in the file is one Disallow per practice game, because each game route is auth-gated and a crawler that fetches one only sees a shell. Those lines are generated, not typed:

const GAME_ROUTES = GAME_LIBRARY.map((game) => game.route).sort();
Enter fullscreen mode Exit fullscreen mode

I checked the invariant the generation is supposed to buy, by diffing the 372 /games/ disallow lines in the live file against the 372 route values in the library. Both directions of comm came back empty. A game added tomorrow is disallowed the day it ships, which is exactly what the comment claims.

4. The claim right under it that is not true

The same comment block ends with a sentence about defence in depth:

The pages also send robots: noindex, nofollow and the public pages that link to them use rel="nofollow", so nothing depends on this file alone.

The first half is enforced in page metadata. The second half is a convention, and conventions are not checked by anything. So I checked it: I pulled every <a href> out of all 530 live pages, kept the ones pointing at a /games/<game-id> route, and read the tag.

13 pages carry 30 such links. Not one has a rel attribute of any kind.

Page Links into a disallowed route
/employers/spar 5
/employers/monuta 5
/employers/gvb 3
/employers/hema 3
/employers/cambio, /employers/foodora, /employers/pts, /employers/view-group 2 each
/blogs/phone-vs-laptop-for-online-assessments 2
three more pages 1 each

They are all hand-written <Link href={game.route}> calls in individual page files, most of them from our newest batch of employer pages. Nothing is leaking: the targets are auth-gated and carry noindex. What it costs is internal linking. A link into a disallowed path is a dead end for a crawler, so those 30 links do nothing for the pages that spend them, while the provider hub three paragraphs further down the same page is the page we actually want crawled.

The fix is not interesting (either rel="nofollow" or link the hub instead). The interesting part is that a generated block of 372 rules held perfectly and the sentence of prose next to it did not.

See it 

curl -s https://cogniprep.app/robots.txt | wc -l            # 850
curl -s https://cogniprep.app/robots.txt | grep -c '^Disallow'  # 796
curl -s https://cogniprep.app/robots.txt | grep -n 'feed.xml'
curl -sI https://cogniprep.app/feed.xml | head -1           # 200
Enter fullscreen mode Exit fullscreen mode

Then open the SPAR employer guide, view source and search for /games/gitp-. Five links, no rel on any of them. The page you want a crawler to reach from there is the GITP hub, which is the one link on the page that robots.txt allows.

Last thing worth copying from this: the matcher is 10 lines and it turned a 24 KB file into a question I can ask in a test. If your robots.txt is generated from application data, the sitemap is the input you should be feeding it.

Top comments (1)

Collapse
 
launchgatecheck profile image
Launch Gate •

The feed finding is my favorite kind of bug: the site disagrees with itself, so no knowledge of intent is needed to prove something is wrong - one of the two artifacts has to lose. That's the cheapest audit class there is, and it's available to any outside reader with curl, which is also why it's worth running inside before someone runs it outside. The second thing I'd wire up: the two 422-rule groups are identical today because you diffed them today. Two maintained copies of one policy always diverge eventually, and when they do, Bingbot gets a different site than everyone else and nobody notices for months. The comm check you ran once belongs in the deploy pipeline - "the groups match" should be a property the build enforces, not a coincidence you verified.