DEV Community

Juanjo
Juanjo

Posted on

How to Safely Fetch Metadata from User-Submitted URLs (Without Getting Hit by SSRF or DNS Rebinding)

#security #python #api #webdev

If your app lets a user paste in a URL — a link preview, a "import from website" button, an AI agent that browses the web on someone's behalf — you have a Server-Side Request Forgery (SSRF) problem whether you've thought about it or not. This post walks through exactly why naive URL fetching is dangerous, why the obvious fixes don't work, and how to actually close the hole.

The naive approach (and why it's broken)

Most "fetch a URL and extract data from it" code looks like this:

import requests

def fetch(url: str):
    response = requests.get(url, timeout=5)
    return response.text
Enter fullscreen mode Exit fullscreen mode

The first instinct to secure this is a blocklist check on the hostname or IP before fetching:

import socket
import ipaddress

def is_safe(url: str) -> bool:
    host = extract_host(url)
    ip = socket.gethostbyname(host)
    addr = ipaddress.ip_address(ip)
    return not (addr.is_private or addr.is_loopback or addr.is_link_local)
Enter fullscreen mode Exit fullscreen mode

This looks reasonable and will pass casual testing. It is still exploitable.

DNS rebinding: the gap the blocklist misses

The check above resolves the hostname, validates the IP, and then — separately — requests.get() resolves the hostname again when it actually opens the connection. Those are two different DNS lookups, at two different points in time, and nothing guarantees they return the same IP.

An attacker controlling the DNS for evil.example.com can configure a very short TTL and serve a legitimate public IP on the first lookup (the one your safety check sees) and then flip the DNS record to 169.254.169.254 (the AWS/GCP/Azure cloud metadata endpoint) or 127.0.0.1 for the second lookup (the one that actually happens when the HTTP client connects). Your check passes. Your fetch hits internal infrastructure anyway.

This is DNS rebinding, and it's not a theoretical attack — it's been used in real SSRF exploits against exactly this pattern: "validate URL, then fetch URL" as two separate steps.

Redirects are the second gap

Even if you solve DNS rebinding, a URL that resolves cleanly to a public IP can still respond with an HTTP redirect to http://169.254.169.254/latest/meta-data/. If your HTTP client follows redirects automatically (most do, by default) and you only validated the original URL, the redirect target never gets checked at all.

The actual fix: resolve once, pin the connection, recheck every hop

The only approach that closes both gaps is to make the IP validation and the connection use the same resolved address, and to repeat that validation on every redirect hop, not just the first request:

  1. Resolve the hostname's DNS once.
  2. Validate that specific IP against private/loopback/link-local/cloud-metadata ranges.
  3. Open the HTTP connection directly to that IP (not the hostname) — while still sending the correct Host header and TLS SNI so the target server routes and certificate-validates correctly.
  4. If the response is a redirect, repeat steps 1–3 for the new location before following it. Never let a redirect chain sneak past validation.

In Python, this means using requests' HTTPAdapter with a custom connection pool (or httpx's transport hooks) to force the socket connection to the pre-validated IP instead of letting the library re-resolve the hostname at connect time. It's more code than a hostname blocklist, and it's the only version that's actually correct.

# Simplified sketch — the real implementation needs to handle
# IPv6, connection pooling, and TLS SNI carefully.
import socket, ipaddress
import httpx

BLOCKED_RANGES = [
    "127.0.0.0/8", "10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16",
    "169.254.0.0/16",  # link-local + cloud metadata (169.254.169.254)
]

def resolve_and_validate(host: str) -> str:
    ip = socket.gethostbyname(host)
    addr = ipaddress.ip_address(ip)
    if any(addr in ipaddress.ip_network(r) for r in BLOCKED_RANGES):
        raise ValueError(f"Blocked IP range: {ip}")
    return ip

def safe_fetch(url: str, max_redirects: int = 5):
    for _ in range(max_redirects):
        host = extract_host(url)
        ip = resolve_and_validate(host)  # re-validated on every hop
        response = httpx.get(url, headers={"Host": host}, extensions={"sni_hostname": host})
        # pin the socket to `ip` via a custom transport — omitted here for brevity
        if response.is_redirect:
            url = response.headers["location"]
            continue
        return response
    raise ValueError("Too many redirects")
Enter fullscreen mode Exit fullscreen mode

When this matters most: AI agents fetching arbitrary URLs

This whole class of bug gets a lot more dangerous the moment an LLM agent is the one deciding what URL to fetch — from a prompt, from a tool call, from a webpage it just scraped. A user (or a malicious webpage the agent visited earlier) can steer the agent toward fetching http://169.254.169.254/latest/meta-data/iam/security-credentials/ and exfiltrate your cloud provider's IAM credentials through the agent's own output. If you're building a RAG pipeline, a browsing tool, or any MCP server that accepts a URL parameter from a model, this isn't optional hardening — treat every URL as attacker-controlled input.

If you'd rather not build this yourself

This exact resolve-once/pin-connection/recheck-every-hop pattern is what powers the URL-fetching layer of the Web Metadata & Contact Extractor API — it's a small REST API (and MCP server) that takes any URL and returns metadata, contacts, tech stack, and clean Markdown, with this SSRF/DNS-rebinding protection built in by default. The code is MIT-licensed on GitHub if you want to read the real implementation or self-host it — there's also a live demo you can try with no signup.

Either way — build it yourself with the pattern above, or use something that already has it — the important part is not shipping the naive version.

Top comments (0)