DEV Community: Rehanfaisal

I scanned the top 1,000 websites for security headers. Google scored an E.

Rehanfaisal — Wed, 02 Sep 2026 09:14:53 +0000

Security headers are the cheapest defence a website has. A few lines of server config, no code changes, no performance cost, and they shut down whole categories of attack.

So I wondered how many of the world's most-visited sites actually bother. I have a scanner — I built one for a site I run — so instead of guessing, I pointed it at the Tranco top 1,000 and graded every one.

The results were worse than I expected, and the worst offenders were not who I expected either.

The headline

Of the 648 domains that returned a scannable page:


55.7%	send no Content-Security-Policy at all
52	median score out of 100 — a grade C
19.1%	score an outright F
8	scored a perfect 100

Eight. Out of 648.

Google scores an E

This is the bit I keep coming back to.

google.com        E   28
microsoft.com     E   32
office.com        F   12
wikipedia.org     E   32

youtube.com       A   92
github.com        A   95
amazon.com        A   92

google.com and youtube.com are the same company. One scores 28, the other 92.

office.com — Microsoft's productivity suite, where a very large number of people keep their working lives — scores 12 out of 100. So does cloud.microsoft. So does office365.com.

I want to be fair about this, because it would be easy to be cheap. Google's homepage is an unusual artefact: it is one of the most performance-tuned pages ever built, it has been iterated on for twenty-five years, and a strict CSP on it would be a genuinely hard retrofit. These companies employ superb security teams.

That is sort of the point. If organisations with that much security expertise haven't retrofitted headers onto their older properties, the odds your side project has them are not good.

Which headers people actually use

Header	Correct	Partial	Absent
Content-Security-Policy	23.3%	21.0%	55.7%
Strict-Transport-Security	56.9%	6.2%	36.9%
Clickjacking protection	58.6%	0.8%	40.6%
X-Content-Type-Options	52.0%	—	48.0%

There's a clear pattern: the headers that are one line and have no side effects get adopted. X-Content-Type-Options: nosniff is a single value with no configuration and roughly half the web sends it.

CSP is the one that requires you to actually understand your own page — every script source, every style, every frame — and it is the one most people skip. 155 sites send a correct HSTS header but no CSP at all. They did the easy one and stopped.

Having a CSP is not the same as being protected

This was the finding I didn't anticipate.

Of the 287 sites that send any Content-Security-Policy, 102 of them — 35.5% — still allow unsafe-inline for scripts.

That directive re-opens the exact hole CSP exists to close:

Content-Security-Policy: script-src 'self' 'unsafe-inline'

An attacker who can inject a <script> tag into your page runs their code. The policy is present, it shows up green in a lot of checking tools, and it is not stopping the attack it was deployed to stop.

It's usually not negligence — it's the pragmatic outcome of retrofitting CSP onto a site with inline handlers everywhere. unsafe-inline makes the page work again. But it should be understood as a migration step, not a destination. Nonces and hashes are the way out.

Being popular doesn't mean being secure

Tranco rank	Sites	Mean score
1–100	70	59.4
101–250	94	52.8
251–500	148	52.7
501–1000	336	50.0

Under ten points of spread across the entire top thousand. Sixteen of the 35 scannable sites in the top 50 score a D or worse.

I expected a much steeper curve — that the biggest sites would be dramatically better resourced and it would show. It barely does.

What I'd actually do with this

If you run anything, the useful takeaway isn't the statistics, it's that this is unusually cheap to fix relative to how much it buys you.

Start with the two that have no side effects and cannot break your site:

X-Content-Type-Options: nosniff
Referrer-Policy: strict-origin-when-cross-origin

Then HSTS, once you're confident every subdomain is HTTPS-only:

Strict-Transport-Security: max-age=31536000; includeSubDomains

Then CSP, which is the real work. Deploy it in report-only mode first:

Content-Security-Policy-Report-Only: default-src 'self'; report-uri /csp-report

Collect violations for a week, see what actually breaks, then enforce. And if you land on unsafe-inline to get it shipping — fine, but write yourself a ticket. You are in the 35.5%.

Method, and the data

Because a statistic you can't check is worthless:

Source list: Tranco top 1,000, retrieved 30 August 2026. Tranco averages several ranking providers over 30 days, which makes it far more stable than any single list.
One HTTPS GET per domain, homepage only, following redirects. No crawling.
648 of 1,000 scanned. 259 were excluded because they have no A record at the apex — Tranco ranks by DNS query volume, so CDN and resolver infrastructure like akamai.net and cloudfront.net floods the list without serving any website. The rest were TLS failures, timeouts and redirect loops.
Scanned from a single vantage point, so sites that geo-block that region are under-represented. A cluster of Russian and Chinese domains timed out.
Headers are one dimension of security. A site can score an A here and be insecure in ways this measures nothing about.

One thing worth mentioning because it nearly wrecked the results: 161 domains initially failed with certificate has expired, including Wikipedia and OpenAI. Their certificates were fine — my machine's certificate store carried an expired root, and OpenSSL was building chains that terminated in it. Had I not checked, I'd have published a study that silently dropped Wikipedia. If you ever do measurement work like this, verify your failures, not just your successes.

Full write-up and the complete dataset (CSV, CC BY 4.0)

The scanner is open source, so this is reproducible rather than something you have to take my word for. You can also run it against your own site — which, given the numbers above, is probably worth five minutes.

I'd genuinely like to know what your own site scores. Post it below, good or bad.

Your URL fetcher is probably an SSRF vulnerability

Rehanfaisal — Fri, 28 Aug 2026 13:08:00 +0000

I spent last week building a site that lets anyone type in a URL and have my server go and look at it. Security header scanner, link expander, that sort of thing.

About an hour in I realised I had built a machine for attacking my own server, and had been quite pleased with myself about it.

Here is the line of code responsible. It appears in a great many codebases:

requests.get(user_supplied_url)

That looks like fetching a URL. What it actually is, is letting a stranger make HTTP requests from inside your network, using your server's credentials and its position behind your firewall.

The address that ruins your day

Try imagining what happens when the user types this:

http://169.254.169.254/latest/meta-data/

On AWS, GCP and Azure, that address is the instance metadata service. It's link-local, so it isn't routable from the internet — only from the machine itself. Which is exactly why your server can reach it and the attacker can't.

Until your code fetches it for them and hands back the response.

On an EC2 instance with IMDSv1 enabled, /latest/meta-data/iam/security-credentials/ returns temporary AWS credentials for whatever role the instance has. Not a hint of them. The actual keys.

In fairness: AWS's IMDSv2 requires a PUT to get a session token first, which defeats a naive GET like this, and it's the default on newer instances. But IMDSv1 is still switched on in an enormous number of accounts, GCP and Azure have their own metadata quirks, and "the cloud provider probably fixed it" is a poor foundation for your security model.

And metadata is only the most dramatic target. The same line reaches:

Target	What the attacker gets
`http://127.0.0.1:6379`	Your Redis, usually unauthenticated
`http://127.0.0.1:9200`	Elasticsearch, often wide open
`http://192.168.1.1`	The router, and everything else on the LAN
`http://localhost:8080/admin`	The internal admin panel with no auth because "it's internal"
`file:///etc/passwd`	Whatever the library will read

This is Server-Side Request Forgery, it's in the OWASP Top 10, and it is depressingly easy to introduce by accident. If your app does link previews, webhook testing, avatar imports by URL, RSS fetching, PDF generation from a URL, or image proxying — you have this code somewhere.

The naive fix, and why it fails

Most people reach for something like this:

from urllib.parse import urlparse
import ipaddress, socket

def is_safe(url):
    host = urlparse(url).hostname
    ip = ipaddress.ip_address(socket.gethostbyname(host))
    return not (ip.is_private or ip.is_loopback)

Better than nothing. Still broken, in three separate ways.

1. It only checks the first address

socket.gethostbyname returns one address. A hostname can resolve to several, and an attacker who controls DNS can return one public address and one private one. You validate the public one; the connection picks whichever it likes.

You need getaddrinfo and you need to check every address it returns.

2. It forgets most of the address space

is_private and is_loopback miss link-local — which is where 169.254.169.254 lives. It's the single most valuable SSRF target and the naive check sails straight past it.

The full set worth blocking:

ip.is_private or ip.is_loopback or ip.is_link_local
    or ip.is_multicast or ip.is_reserved or ip.is_unspecified

Python's ipaddress module already classifies all of these. Use it rather than hand-writing CIDR lists that go stale.

3. It only checks the URL the user typed

This is the one that gets people, and it's worth dwelling on.

You validate https://totally-innocent.com. It's public, it passes, you connect. The server replies:

HTTP/1.1 302 Found
Location: http://169.254.169.254/latest/meta-data/

Your HTTP library helpfully follows the redirect. Your check ran once, on a URL that was never the target.

Every hop needs re-validating, not just the first. That means turning off automatic redirects and following them yourself.

What a working version looks like

Three things have to be true: check every resolved address, block the full set of internal ranges, and re-validate on every redirect.

import ipaddress, socket, urllib.request, urllib.error
from urllib.parse import urljoin, urlparse, urlunparse

class SafeFetchError(Exception):
    pass

def is_blocked_ip(ip_str):
    try:
        ip = ipaddress.ip_address(ip_str)
    except ValueError:
        return True                      # unparseable: refuse, don't guess

    # ::ffff:127.0.0.1 must be judged on the IPv4 address it represents
    if isinstance(ip, ipaddress.IPv6Address) and ip.ipv4_mapped:
        ip = ip.ipv4_mapped

    return (ip.is_private or ip.is_loopback or ip.is_link_local
            or ip.is_multicast or ip.is_reserved or ip.is_unspecified)

def resolve_and_check(hostname):
    try:
        infos = socket.getaddrinfo(hostname, None)
    except socket.gaierror:
        raise SafeFetchError("Could not resolve %r." % hostname)

    for info in infos:                    # EVERY address, not just the first
        addr = info[4][0]
        if is_blocked_ip(addr):
            raise SafeFetchError(
                "%r resolves to %s, which is internal." % (hostname, addr))

Then suppress automatic redirects so you can inspect each one:

class _NoRedirect(urllib.request.HTTPRedirectHandler):
    def redirect_request(self, req, fp, code, msg, headers, newurl):
        return None                       # handle them ourselves

def fetch(url, timeout=8, max_redirects=5, max_body=65536):
    opener = urllib.request.build_opener(_NoRedirect)
    current = normalise_url(url)

    for _ in range(max_redirects + 1):
        parsed = urlparse(current)
        resolve_and_check(parsed.hostname)          # <- runs on every hop

        req = urllib.request.Request(current, method="GET")
        try:
            resp = opener.open(req, timeout=timeout)
            return {"url": current,
                    "status": resp.status,
                    "headers": {k.lower(): v for k, v in resp.headers.items()},
                    "body": resp.read(max_body)}    # cap it
        except urllib.error.HTTPError as e:
            if e.code in (301, 302, 303, 307, 308):
                location = e.headers.get("Location")
                if not location:
                    raise SafeFetchError("Redirect with no destination.")
                current = normalise_url(urljoin(current, location))
                continue
            raise

    raise SafeFetchError("Too many redirects.")

Two more things belong in normalise_url:

if parsed.scheme not in ("http", "https"):
    raise SafeFetchError("Only http and https.")

file://, gopher:// and dict:// are standard SSRF escalation paths. Allow-list the two schemes you want rather than trying to enumerate the bad ones.

And rebuild the authority from host and port only, dropping anything else:

netloc = host if not parsed.port else "%s:%d" % (host, parsed.port)
return urlunparse((parsed.scheme, netloc, parsed.path or "/", "", "", ""))

That quietly strips embedded credentials, so a URL like https://user:pass@example.com/ doesn't forward someone's password to the target.

Test it like an attacker

Assertions beat intentions. Every one of these must be refused:

BLOCKED = [
    "http://127.0.0.1/", "http://localhost/", "http://[::1]/",
    "http://169.254.169.254/latest/meta-data/",   # cloud metadata
    "http://10.0.0.1/", "http://192.168.1.1/", "http://172.16.0.1/",
    "http://0.0.0.0/",
    "file:///etc/passwd", "gopher://x/", "dict://x:11/",
]

for url in BLOCKED:
    try:
        fetch(url)
        raise AssertionError("SHOULD HAVE BEEN BLOCKED: " + url)
    except SafeFetchError:
        pass                              # correct

Then the redirect case, which is the one worth being paranoid about — stand up a server that 302s to 169.254.169.254 and confirm you don't follow it.

What this does not fix

I'd rather say this plainly than let you think you're done.

DNS rebinding. There is a gap between your check and your connection. An attacker running a DNS server with a one-second TTL can answer with a public address for the check and a private one for the connection. Closing it properly means resolving once, connecting to that pinned IP, and passing the hostname separately for TLS SNI and the Host header. That's awkward with urllib, so the code above doesn't. For most apps this check is proportionate; if your input is genuinely hostile, pin the address.

Your own public IP. If your server has a public address and also runs something private on it, none of this helps — the address isn't in a private range. Use a firewall too.

Being a proxy. If you return the fetched body verbatim to the user, you've built an open proxy regardless. Return only what you need.

Rate limiting. This stops you reaching inside. It doesn't stop someone using your server to hammer a third party.

Take it

I pulled this out of my own project into a single dependency-free file, MIT licensed:

github.com/Rehanfaisal/ssrf-safe-fetch

safe_fetch.py is one file with no dependencies — copy it into your project. The test suite covers every blocked range, IPv4-mapped IPv6, scheme rejection and credential stripping.

It came out of IsSiteSafe, where the whole premise is that strangers type in URLs and my server goes and looks at them. Nothing sharpens your interest in SSRF like shipping the exact feature that causes it.

If your app fetches user-supplied URLs, go and try http://169.254.169.254/latest/meta-data/ against your own staging environment right now. Finding out it works is a much better afternoon than finding out from someone else.