Cloudflare's /security.txt 404s with Content-Type: text/plain

Quick answer

cloudflare.com/security.txt returns HTTP 404 with Content-Type: text/plain. So does github.com/security.txt.

If your "does this file exist" check is content_type.startswith("text/plain"), both of those are a present file. If it's status == 200, you'll miss the sites that serve a real security.txt from a soft-404 handler. You need both, plus a third check neither one gives you.

Three ways to be wrong about a text file 📄

Auditing well-known files — robots.txt, security.txt, ads.txt, humans.txt — sounds like the most boring HTTP work imaginable. Fetch a path, parse some lines. The difficulty is entirely in one question: is this file actually here?

Three failure modes, all live today:

1. The 404 that says text/plain. Cloudflare and GitHub both serve their security.txt from /.well-known/security.txt and return 404 for the legacy /security.txt path — but the 404 body is itself plain text, so the Content-Type header agrees with "yes, a text file." Google's legacy path 404s as text/html, so you can't even rely on the inconsistency being consistent.

2. The soft 404. A site with a custom branded "page not found" that answers HTTP 200 and serves you an HTML page where a robots.txt should be. Status says present. Body says otherwise.

3. The real file behind a fallback. RFC 9116 says security.txt lives at /.well-known/security.txt. Plenty of sites still serve the legacy root path. Check only one and you'll report "no security contact" for organisations that have published one for years.

The check that actually works

None of status, Content-Type, or path is sufficient alone. What's sufficient is the file's own syntax — does the body contain a line that only this format has?

_DIRECTIVE_LINE_RE = re.compile(
    r"^\s*(Contact|Expires|Policy|Encryption|Canonical|Preferred-Languages)\s*:",
    re.IGNORECASE | re.MULTILINE,
)

def is_soft_404(content_type, body):
    """Present iff Content-Type is text/plain OR an RFC 9116 field line matches."""
    if content_type and content_type.lower().startswith("text/plain"):
        return True
    return bool(_DIRECTIVE_LINE_RE.search(body)) if body else False

A branded HTML 404 page will never contain a line starting Contact: at the beginning of a line. A real security.txt always will. The format is its own proof of existence.

Same principle for each file: robots.txt is proven by User-agent: / Disallow:, ads.txt by its comma-separated four-field record shape. Ask the content what it is; don't ask the envelope.

And the fallback stays surgical — legacy is tried only on a genuine 404 from well-known, never on a timeout or a 403:

well_known = await fetch_text(session, url_for(host, ".well-known/security.txt"))
if well_known.status_code != HTTP_NOT_FOUND:
    return well_known, "well_known"
legacy = await fetch_text(session, url_for(host, "security.txt"))
return legacy, "legacy"

That distinction matters. A 403 on well-known means the file is probably there and you're being blocked; retrying a different path just gets you blocked twice and reports the wrong reason.

The parsing bug hiding in plain sight 🔍

Here's Cloudflare's actual security.txt, in full:

Contact: https://hackerone.com/cloudflare
# All abuse reports should be submitted to our Trust & Safety team through
# our dedicated page.
Contact: https://www.cloudflare.com/abuse/

Policy: https://www.cloudflare.com/disclosure/
Hiring: https://www.cloudflare.com/careers/jobs/

Preferred-Languages: en
Canonical: https://www.cloudflare.com/.well-known/security.txt

Two Contact: lines, separated by two comment lines and pointing at completely different teams. The first is the bug-bounty programme. The second is abuse reporting. They are not interchangeable — sending an abuse report to HackerOne gets it closed, and sending a vulnerability to the abuse form gets it triaged as spam.

RFC 9116 explicitly allows multiple Contact fields and defines them as ordered by preference. So the correct read is a list, in file order.

The bug is a one-character one:

_CONTACT_RE.search(body)      # -> the bug bounty URL, and nothing else
_CONTACT_RE.finditer(body)    # -> both, in order

search is the natural thing to type. It's also the thing that quietly discards half the answer on every site that publishes more than one contact — and the sites that publish more than one contact are precisely the large, well-resourced ones you most want the data for. Your audit looks complete and is systematically wrong on your most important rows.

We return every Contact: and every Policy: in file order, and parse Expires: with a real ISO-8601 parse so an expired security.txt is flagged rather than silently trusted. An unparseable Expires line returns the raw string and a null timestamp — never an exception, because one malformed date on one site is not a reason to lose the other 499 rows.

What the Actor gives you

The Robots.txt & Security.txt Compliance Auditor fetches four well-known files per site and returns one structured audit row:

• Named AI-crawler posture — is GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot or Bytespider blocked, allowed, or simply never mentioned? "Never mentioned" is its own answer and the most common one.
• Security contacts and expiry — every Contact: and Policy:, plus whether Expires: is in the past.
• ads.txt record counts for ad-tech supply-chain vetting.
• Per-file fault isolation — one file missing or erroring never sabotages the other three. You get a full row with clear markers, not a failed run.

Built for SEO and compliance teams auditing their own estate, ad-ops teams vetting supply chains, and security researchers mapping disclosure coverage across a sector.

The honest limitations 🚧

• We report what the file says, not whether the site honours it. A Disallow in robots.txt is a request, not an enforcement mechanism.
• Expires: is parsed best-effort — a genuinely malformed date is returned raw with a null timestamp rather than guessed at.
• No PGP signature verification on signed security.txt files; the Encryption field is detected, not validated.

Pricing

$0.20 per run plus $0.003 per audited site — about $3.20 per 1,000 results. A run that finds nothing costs the start fee and nothing else.

→ Robots.txt & Security.txt Compliance Auditor on Apify

---

Built by Devil Scrapes. We handle the soft 404s, the legacy path fallbacks, and the second contact line everyone's parser throws away.