The contract for our API proxy has no target-URL parameter. A widget sends a connector handle and a relative path. The server loads the destination registered for that connector, then checks whether it is still safe to call.
That choice rules out the usual request saying, in effect, "fetch this internal address for me". It leaves several ways for a registered destination to become unsafe, including DNS changes and redirects. Those are the cases the rest of this article covers.
Who gets to choose the destination?
We built this proxy for GoodBarber, where generated widgets need to call third-party APIs without receiving the private keys themselves. The AI extension feature is still a prototype in a small pilot. The proxy holds the credentials and injects them into outbound requests, which makes its destination checks part of the secret's protection.
The OWASP cheat sheet on SSRF distinguishes services that can restrict calls to known destinations from those that must fetch arbitrary URLs. We chose the first model so that an authenticated owner could approve a destination before a widget used it.
The decision: a handle, not a URL
The handle is a random identifier. It selects a server-side record containing the scheme, host, base path and permitted methods. The caller supplies the relative path, query and body:
What the widget sends What the server already holds
----------------------------- ---------------------------------------
handle: k3J…x9 (opaque) base URL: https://api.example.com/v2
path: /forecast?city=Paris allowed hosts, allowed methods
body: { … } the secret, encrypted
When the owner approves an API and supplies its key, the base URL goes through the same validation function used for subsequent requests. Approval records the intended destination; it cannot guarantee what that hostname will resolve to later.
What is left to defend
Each outbound request still has to satisfy these checks:
| Vector | What we do |
|---|---|
The host is an IP in disguise: 2130706433, 0x7f000001, 127.1, octal forms |
rejected before any resolution: a destination must be a real hostname, and a name whose labels are all numeric or hexadecimal is refused |
| The name resolves to an internal address | blocked on the resolved IP, not on the string: loopback, private ranges, link-local and its cloud metadata address, carrier-grade NAT, and their IPv6 equivalents |
| One public A record and one private one | every address in the answer is validated, and a single forbidden address rejects the whole answer |
| DNS rebinding between the check and the connection | one resolution, then the connection goes to the validated IP |
| An IPv4 address wrapped in IPv6 | IPv4-mapped and NAT64 addresses are unwrapped and the embedded IPv4 is validated too |
| A redirect to somewhere else | redirects are never followed, the 30x status goes back to the caller as it is |
../ to escape the registered base path |
the path is normalised first, then compared with the base path |
| Headers that change the meaning of the request | three headers are relayed from the client: Content-Type, Accept, Accept-Language. Host, forwarding headers and method overrides are set or dropped by the server |
| A proxy configured in the host's environment | the HTTP client ignores environment proxies |
Lookalike hosts such as api.example.com.evil.com
|
host matching is exact equality, never a suffix or a pattern |
The NAT64 case came from an adversarial review before the first commit. In the well-known prefix defined by RFC 6052, 64:ff9b::7f00:1 embeds 127.0.0.1. An IPv6 classification alone can miss the address a translating network would reach. The embedded IPv4 address needs validation too.
Cloud metadata is one reason that distinction matters. AWS added IMDSv2 in 2019 as defense in depth against SSRF and related paths to instance metadata. Our proxy rejects those destinations rather than relying on the target to resist the request.
Connect to the address you validated
Checking a name and then handing the same name to the HTTP client is two resolutions, and the second one can answer differently. The fix is to resolve once and pin.
def resolve_and_pin(host: str, port: int) -> str:
ips = resolve_ips(host, port) # one resolution, A and AAAA
for ip in map(ipaddress.ip_address, ips):
if is_blocked_ip(ip): # one bad address rejects the whole answer
raise SsrfBlocked(host)
return ips[0] # the address we will connect to
The request then goes to that IP, while TLS still verifies the real hostname:
url = httpx.URL(scheme="https", host=pinned_ip, path=path)
headers["Host"] = hostname # the real name, set by the server
request = client.build_request(method, url, headers=headers, content=body)
request.extensions = {**request.extensions, "sni_hostname": hostname} # SNI and certificate check too
The Host header and the SNI carry the real name, so certificate verification is as strict as on a normal call, and it is never disabled. The socket layer has nothing left to resolve. The client is created with follow_redirects=False and trust_env=False.
The read timeout that never fires
An upstream can keep a worker occupied by sending one byte every twenty seconds. That never trips a 25-second httpx read timeout, because the timeout limits the wait for each chunk. We therefore put a wall-clock deadline around the exchange, including body streaming:
async def forward(...):
# httpx's read timeout restarts at every chunk. This deadline does not.
return await asyncio.wait_for(_execute(...), timeout=TOTAL_DEADLINE) # 30 s, for a 25 s read timeout
The response size is capped the same way, by counting the bytes actually streamed, not by trusting a Content-Length header.
The key arrives last
Decryption happens after the destination checks. A request rejected for its method, path, host or resolved address therefore does not load the plaintext credential. Once admitted, the request receives the key for the approved upstream.
What the decision costs
Adding another API requires another approved connector. A widget cannot discover an arbitrary endpoint and ask our server to call it. That restriction suits a platform where an app owner approves which services receive their credentials.
The egress code landed on July 7 with 105 tests, including 18 for SSRF cases such as numeric hosts, mixed DNS answers, NAT64 and rebinding. Its specification contains 19 egress requirements. The tests cover the checks; the request contract limits what those checks have to accept.
In a review of another proxy, I would start where the outbound URL is constructed. Trace each component back to its source, then follow the chosen host through resolution and connection. A validator can be correct while the HTTP client quietly makes a different request.
Top comments (3)
The handle-not-URL decision plus resolve-once-and-pin with SNI intact is the cleanest version of this I've seen written up. Two places I'd poke at, both from things that bit us:
Is the response cap counting wire bytes or decoded bytes? httpx decodes gzip/br/deflate transparently in aiter_bytes(), so if the cap counts aiter_raw() (or trusts the upstream's framing) while you relay or buffer the decoded body, a small compressed response can expand well past it. We had the inverse of this on an upload path: a 20 MB .xlsx held about two million rows, because a repeated row compresses to ~11 bytes. Counting decoded bytes, or relaying the raw encoded body untouched, closes it.
Does path normalisation run before or after percent-decoding? /v2/%2e%2e/admin and /v2/..%2fadmin normalise as clean under a literal comparison, and some upstreams decode them back into a traversal. Decoding first (or rejecting encoded ., / and \ in the relative path) keeps the base-path check honest.
The "key arrives last" ordering is a nice touch. A rejected request never touching the plaintext is easy to skip and hard to retrofit.
Decoded bytes. The counter sits on aiter_bytes(), after httpx has undone gzip or deflate, and the relay drops the upstream’s Content-Encoding and Content-Length, so the proxy regenerates the framing for the body it actually holds. Your .xlsx case is closed on that path.
Your question sent me one layer down, though, and that layer was not closed. The counter only sees a chunk once it has been decoded, and httpx 0.28 decompresses each network read with no output limit. I measured it today: one 64 KiB read of gzip comes out at about 64 MiB, and with Content-Encoding: gzip, gzip a 3,476-byte response decodes into 2 GiB in a single chunk. The cap bounded what we relay, not what a worker allocates on the way. The fix is to refuse encodings we did not ask for and to decompress with a bounded output, so the count happens before the memory is spent.
On the path: neither, at first. The server decodes the path once, and whatever is still percent-encoded goes out untouched, to be decoded by the upstream. /customers/%2e%2e/account/keys, sent double-encoded, passed the /customers prefix and came out as /account/keys on the other side. Our eighth security review caught it, and the fix landed on September 22. We took your second option: any escape that decodes to ., /, \, % or a control byte is rejected before the prefix check, and so is a literal backslash. We don’t know which decoder sits on the other side, so we refuse the ambiguity instead of guessing. Harmless escapes such as %20 or UTF-8 still pass. The residue we accepted is overlong UTF-8 like %c0%ae, which only a broken decoder reads as a dot.
Thanks for both pokes.
That gzip, gzip number is a great find. 3.5 KB to 2 GiB in one chunk is exactly the layer nobody looks at, because the cap reads as if it covers it.
On the bounded decompress, two things from doing the same for archives: zlib’s decompressobj().decompress(data, max_length) gives you that cleanly, but the leftover sits in unconsumed_tail and has to be fed back in a loop, or the limit silently becomes “first chunk only”. And if brotli or zstandard is installed, httpx will offer and decode those too. Their streaming APIs don’t all take an output limit the same way, so “refuse what we didn’t ask for” is probably the safer half of the fix. Sending Accept-Encoding: identity (or one encoding you can bound) and rejecting anything else makes it explicit.
One more on the path, in the same “we don’t know the decoder on the other side” spirit: ..; with no encoding at all. /customers/..;/account/keys is not a .. segment to Python’s normalisation, so it passes the prefix check, but Tomcat and some Spring setups strip ;params from each segment and then resolve it as ... If any upstream could be Java, rejecting ; in path segments (or at least ..;) closes it.