DEV Community

Hammad Shams Uddin
Hammad Shams Uddin

Posted on

My robots.txt check passed for six days on an attacker's robots.txt

#php #security #devops #webdev

Every morning a script checks my site's health and emails me. One of its lines, for six days running:

Fine. Sitemap index lists 54 child sitemaps holding 6,240 URLs in total,
robots.txt points at it, and every page earning impressions answers 200.
Enter fullscreen mode Exit fullscreen mode

For those six days, this was my robots.txt:

User-agent: *
Allow: /

Sitemap: https://utilorax.com/?sitemapindex.xml
Sitemap: https://utilorax.com/?sitemap249.xml
Sitemap: https://utilorax.com/?sitemap476.xml
Sitemap: https://utilorax.com/?sitemap531.xml
Sitemap: https://utilorax.com/?sitemap458.xml
Sitemap: https://utilorax.com/?sitemap522.xml
Sitemap: https://utilorax.com/goods.php?sitemap806.xml
Enter fullscreen mode Exit fullscreen mode

None of those are mine. goods.php is the giveaway: it is the shape of a Japanese shop-spam campaign, and the site had been hit by exactly that — a cloaker serving Googlebot a clothing store while people saw the real site. I cleaned that up on 5 September and wrote down, with some satisfaction, that the site was clean.

It was not. The cloaker had left a second thing behind, and every instrument I had built to catch the first one looked straight at it and said fine.

The check 

$robots = (string) @file_get_contents('https://utilorax.com/robots.txt');
if (!str_contains($robots, 'Sitemap:')) {
    $problems[] = 'robots.txt does not point at a sitemap';
}
Enter fullscreen mode Exit fullscreen mode

It asks whether robots.txt mentions a sitemap. The attacker's file mentions seven. It passed with a margin.

This is the same bug as a login check that verifies a password field is non-empty. The test was written against the failure I imagined — someone deleting the Sitemap line by accident — and not against the failure that happened, which is someone replacing it on purpose. A check for the presence of a field passes on anyone's field.

How a static file beat a dynamic one

My real robots.txt is not a file. index.php generates it, so the sitemap URL always matches the host:

if ($path === 'robots.txt' || /* sitemap*.xml */ ...) {
    Sitemap::handle($path);
    exit;
}
Enter fullscreen mode Exit fullscreen mode

and .htaccess sends a request there only when no real file exists:

RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^ index.php [L]
Enter fullscreen mode Exit fullscreen mode

So dropping a plain robots.txt into the web root does not fight the generator. It simply wins, silently, because that is what !-f means. There is no conflict to notice.

The file itself had been dressed for the job:

-r--r--r--  356 bytes   Jul 7 2024   robots.txt
Enter fullscreen mode Exit fullscreen mode

Read-only, so a careless cleanup would fail. And dated July 2024 — on a domain registered in July 2026. The only file on the server older than the project, and the only file outside vendor/ set to 444.

Three instruments, one blind spot each

What bothers me is not that one check was weak. It is that I had built three, after the hack, specifically so this could not happen again — and each one had a different reason to wave it through.

The health check tested for a string the attacker supplied.

The integrity monitor hashes every static file and reports anything that changes. I took its baseline after the cleanup — so the planted robots.txt went into the baseline as a known-good file. From then on, the monitor's job was to guarantee the attacker's file stayed exactly as they left it. It did that perfectly.

A baseline taken on a compromised system does not detect the compromise. It certifies it.

The off-server verifier compares live files against git and lists anything on the server that git does not know about. On 5 September it listed robots.txt. I read the list, saw a handful of expected runtime files — caches, a ratings database, a speed-test payload — and filed robots.txt with them, because a site having a robots.txt looked normal.

It was the only item on that list that could not legitimately exist. The real one is generated; a file by that name should never be on disk at all. The verifier was right and I explained it away.

What changed

The check now asks whose sitemap, and treats a file existing as the finding:

preg_match_all('~^\s*Sitemap:\s*(\S+)~mi', $robots, $sm);
$ours    = 'https://utilorax.com/sitemap.xml';
$foreign = array_filter($sm[1], fn($u) => rtrim($u) !== $ours);

if (!in_array($ours, $sm[1], true)) { $problems[] = 'robots.txt does not list our sitemap'; }
if ($foreign)                       { $problems[] = 'robots.txt lists sitemaps that are not ours'; }
if (is_file($webRoot . '/robots.txt')) {
    $problems[] = 'a static robots.txt exists — the real one is dynamic, so this is shadowing it';
}
Enter fullscreen mode Exit fullscreen mode

The verifier has a separate section for static files that shadow a generated route, so it cannot be mistaken for runtime clutter again. And a failed fetch is reported as not evaluated today, rather than as no sitemap — the old code turned an empty response into the same message as a real problem, which is part of how I found this: a false alarm made me open the actual file.

And then the fix was invisible

I deleted the file, fetched /robots.txt, and got my real one back. Then the new check ran and fired again:

*** robots.txt lists 7 sitemap(s) that are not ours ***
Enter fullscreen mode Exit fullscreen mode

I had tested with a cache-busting query string. The plain URL — the one crawlers request — was coming from the CDN:

cf-cache-status: HIT
Cache-Control: max-age=14400
Enter fullscreen mode Exit fullscreen mode

The origin was fixed. The edge was still serving the attacker's file, for up to four more hours, to every crawler that asked. If I had kept the old string check, it would have passed that too.

What I would take from it

Three things, and none of them are about robots.txt:

  • Test for the right value, not for a value. "Is there a sitemap line" and "is it our sitemap line" differ by exactly one attacker.
  • Never baseline a system you have just cleaned without first proving it against something the attacker could not touch. My proof was git, and I read its answer too quickly.
  • An "expected" list is where things hide. Every exception you wave through once becomes invisible forever. If a file cannot legitimately exist, it does not belong in a list of files that usually do.

And test your fix the way the crawler will see it, not the way you will.

I build Utilorax, a set of free browser-based tools. If you want to see what your own robots.txt actually tells a crawler — including which sitemaps it advertises — the robots.txt tester will show you.

Top comments (0)