DEV Community

Cover image for Protocol Verification Playbook for HTTP Proxies, SOCKS5, and PROXY Protocol
gabriele wayner
gabriele wayner

Posted on

Protocol Verification Playbook for HTTP Proxies, SOCKS5, and PROXY Protocol

#proxy #networking #devops #troubleshooting

If an incident is "users can't log in" or "upstream sees the wrong IP," the fastest way to lose two days is to "fix" the wrong layer first. This playbook is about confirming what is actually happening on the wire before you touch configs, rotate pools, or blame TLS. It's an executable companion to the main hub article, where proxy IPs matter, written for real incidents where you need evidence fast.

Run these steps in order. Each one gives you a command and the expected signals. If a signal doesn't match, stop and fix that layer before moving on.

The fastest way to waste two days is to debug the wrong layer

Before you run anything, write down your expected path:

  • Client → Proxy (HTTP or SOCKS5) → Origin (or LB) → Backend
  • DNS should resolve at the client or at the proxy
  • Client identity should be established at the edge, not guessed in the app
  • PROXY protocol should be enabled or disabled per listener, not "somewhere"

Then verify those expectations with traffic, not assumptions.

Step 1 Verify HTTP proxy behavior with CONNECT tunneling

You're proving three things:

  1. The client is actually using the proxy
  2. HTTPS is tunneled via CONNECT
  3. The TLS handshake happens after the tunnel is established

For a quick capability map and terminology reference, keep this open: HTTP Proxies.

Command: curl with verbose proxy output 

curl -v -x http://PROXY_HOST:PROXY_PORT https://example.com/ -o /dev/null
Enter fullscreen mode Exit fullscreen mode

Expected signals

In -v, the order matters:

* Connected to PROXY_HOST (IP) port PROXY_PORT
> CONNECT example.com:443 HTTP/1.1
< HTTP/1.1 200 Connection established (or similar)
* TLSv1.3 ...
Enter fullscreen mode Exit fullscreen mode

If you see TLS negotiation before CONNECT, you're not tunneling via the HTTP proxy you think you are.

Command: force a failure to prove the proxy is in-path 

# Wrong proxy port should fail fast
curl -v -x http://PROXY_HOST:1 https://example.com/ -o /dev/null
Enter fullscreen mode Exit fullscreen mode

Expected signals

  • Immediate connect failure to PROXY_HOST:1
  • If it still succeeds, you're bypassing the proxy (env vars, PAC, transparent proxying, or the client isn't honoring flags)

Optional: tcpdump the proxy port to see plaintext CONNECT 

sudo tcpdump -i any -s 0 -A 'tcp port PROXY_PORT and host PROXY_HOST'
Enter fullscreen mode Exit fullscreen mode

Expected: you can see CONNECT example.com:443 in plaintext. If you can't, you're on a different path than your mental model.

Step 2 Verify SOCKS5 and force DNS through the proxy path

SOCKS5 is where teams accidentally leak DNS. Verification here is about proving where name resolution happens.

A concise reference on how SOCKS5 behaves in real routing stacks: SOCKS5 Proxies.

Command: compare SOCKS5 local DNS vs remote DNS 

# DNS resolved locally (potential leak)
curl -v --socks5 SOCKS_HOST:1080 https://example.com/ -o /dev/null

# DNS resolved by the SOCKS proxy (preferred when you want remote DNS)
curl -v --socks5-hostname SOCKS_HOST:1080 https://example.com/ -o /dev/null
Enter fullscreen mode Exit fullscreen mode

Expected signals

  • With --socks5-hostname, you should NOT see local DNS traffic leaving the client
  • With --socks5, you may see local resolution and then a connect to the resolved IP

Command: tcpdump to detect DNS leaks from the client 

sudo tcpdump -i any -n 'port 53'
Enter fullscreen mode Exit fullscreen mode

Run the two curl commands again and watch:

  • DNS packets during --socks5-hostname indicates a leak (client library behavior, OS resolver, or misrouting)
  • A clean run: no port 53 traffic from the client during the request

Tip: write down whether you expect "DNS at client" or "DNS at proxy." The correct answer depends on your routing intent, not preference.

Step 3 Verify client identity headers with a real trust boundary

Headers like X-Forwarded-For and Forwarded are claims, not facts. Verification here means proving you have a boundary that strips untrusted claims and injects a canonical identity at the edge.

If you want a compact map of proxy-layer protocols and where they sit, keep this reference handy: Proxy Protocols.

Command: attempt header spoofing from the client 

curl -sS -D- https://your-edge.example/ \
  -H 'X-Forwarded-For: 1.2.3.4' \
  -H 'Forwarded: for=1.2.3.4;proto=https;by=evil' \
  -o /dev/null
Enter fullscreen mode Exit fullscreen mode

Expected signals

On the server side (edge logs or application logs), confirm:

  • The spoofed values do NOT appear as the authoritative client identity
  • The value you trust comes only from your edge component (LB, gateway, ingress), not from the public client

If you can't see this easily, add temporary structured logging for:

  • Remote socket peer IP (what the TCP connection says)
  • The canonical header you trust (your sanitized X-Forwarded-For or Forwarded)
  • Whether the request came from a trusted hop (CIDR allowlist and/or mTLS identity)

Command: chain sanity check 

curl -sS https://your-edge.example/debug/ip
Enter fullscreen mode Exit fullscreen mode

Expected signals

  • Stable chain format (same number of hops under normal conditions)
  • No sudden extra hops (often indicates an unexpected proxy layer or misconfigured internal forwarder)

Step 4 Verify PROXY protocol on TCP listeners without breaking first bytes

PROXY protocol prepends metadata to the TCP stream so backends can learn the original client address. The failure mode is consistent: if PROXY bytes hit an HTTP or TLS parser, you'll see "mystery 400s" or handshake failures because the first bytes aren't what the backend expects.

If you need a production-focused configuration and verification guide to cross-check, use this: proxy protocols configure verify production.

What you're verifying

  • Does this backend listener expect PROXY protocol
  • If yes, is it v1 (text) or v2 (binary)
  • Is the component in front (LB/proxy) actually sending it

Command: tcpdump the first bytes on the backend port 

# Keep this simple and reliable under pressure
sudo tcpdump -i any -s 96 -X 'tcp port BACKEND_PORT'
Enter fullscreen mode Exit fullscreen mode

Generate a single request through the LB/proxy, then inspect the first payload bytes.

Expected signals

  • PROXY v1 starts with ASCII PROXY
  • PROXY v2 starts with a binary signature (often shown as): 
0d 0a 0d 0a 00 0d 0a 51 55 49 54 0a
Enter fullscreen mode Exit fullscreen mode
  • If you see GET / first, PROXY protocol is not being sent to that listener
  • If you see a TLS ClientHello first (often 16 03 01 or 16 03 03), PROXY protocol is not being sent to that listener

Quick first-bytes heuristic

  • Backend expects HTTP but receives PROXY → HTTP parser errors / 400s
  • Backend expects TLS but receives PROXY → TLS handshake failures
  • Backend expects PROXY but receives HTTP/TLS → client IP missing symptoms

Your fix is aligning sender and receiver expectations on that listener, not "retrying harder."

Common failure patterns and the first fix that usually works

  • CONNECT returns 407 auth required

    • First fix: verify proxy credentials and whether the client is sending Proxy-Authorization. Re-run curl -v and confirm the 407 is from the proxy, not the origin.

  • HTTPS works without proxy flags but fails with them

    • First fix: you're pointing at the wrong proxy type/port. Use tcpdump on the proxy port and confirm you can see plaintext CONNECT.

  • SOCKS appears to work but DNS-based routing is wrong

    • First fix: switch to --socks5-hostname and confirm no local port 53 traffic via tcpdump.

  • App logs show spoofed X-Forwarded-For values

    • First fix: strip inbound X-Forwarded-For / Forwarded at the edge and inject your own canonical header. Re-test spoofing.

  • Sudden wave of 400s after enabling PROXY protocol

    • First fix: backend listener isn't expecting PROXY bytes. Disable PROXY on that hop or enable PROXY parsing on the backend, then confirm with first-bytes tcpdump.

End-of-run checklist you can paste into your incident notes

  • [ ] I proved the client is using the intended proxy path (curl -v shows proxy connect).
  • [ ] For HTTP proxy plus HTTPS, I saw CONNECT → 200 established → TLS handshake (in that order).
  • [ ] For SOCKS5, I verified where DNS happens and confirmed no unexpected local port 53 traffic when remote DNS is required.
  • [ ] I tested header spoofing (X-Forwarded-For, Forwarded) and confirmed the edge enforces a trust boundary.
  • [ ] I confirmed what the backend receives as first bytes (HTTP, TLS, PROXY v1, or PROXY v2 signature).
  • [ ] I fixed mismatches by aligning sender/receiver expectations on the listener.
  • [ ] I captured one known-good command and its expected signals for future incidents.

When you're done, tie your findings back to routing intent in the hub article: where proxy IPs matter.

Top comments (0)