Most client IP preservation failures do not come from missing features. They come from teams trusting the wrong identity signal at the wrong hop. Edge logs show one address, the app reads another, and rate limiting quietly keys off something else. If you want the broader framework behind these choices, start with Proxy Protocols vs PROXY protocol in production and how to preserve client IP correctly.
The dangerous part is that these rollouts often look healthy at first. Requests succeed. Dashboards stay green. But once incident response, abuse controls, or geo-based policy depends on the preserved client identity, the mismatch becomes expensive.
This post is not a generic setup guide. It is a validation workflow for operators who need proof on the wire, proof at the listener, and proof in downstream consumers before rollout.
Verify the sender, the receiver, and the first bytes on the wire
Before touching a listener, verify three things.
First, identify exactly which hop emits the client identity signal. That may be a reverse proxy writing HTTP forwarding headers, a load balancer sending transport metadata, or an ingress tier normalizing the result for the application. The label matters less than the actual sender. Many teams blur the line between header-based identity and transport-level identity even though they fail differently in production. That distinction becomes much easier to reason about once you separate the broader family of Proxy Protocols from the specific PROXY protocol handoff.
Second, confirm what the receiver expects on the exact port under test. A listener is a contract, not a guess. In HAProxy’s official documentation, PROXY protocol works only when both the sender and receiver support it and have it enabled, which is exactly why sender and receiver validation must be paired instead of checked in isolation. See HAProxy’s client IP preservation guide for PROXY protocol
Third, inspect the first bytes on the wire. This is the fastest way to cut through configuration assumptions. If the connection starts with an HTTP request line, you are not looking at a PROXY preamble. If it starts with a PROXY header, the receiver must be ready to consume it before anything else. If TLS bytes arrive first, TLS ordering controls the rest. Tools like Wireshark are built for exactly this kind of packet-level inspection.
Run a practical validation workflow before rollout
Start by drawing the path in one line. Write down each hop and the single identity signal that leaves it. Keep it simple: client, edge, load balancer, ingress, application, logs, policy engine. If one hop emits forwarded headers and another emits transport metadata, write both down explicitly.
Then verify listener contracts one port at a time. For each receiving port, answer four questions. Does it expect TCP or TLS first? Does it expect PROXY protocol first? Does it trust forwarded headers from that source? Do health checks use the same path as real traffic?
Next, capture the first bytes at the receiving hop:
sudo tcpdump -A -s 128 'tcp port 443'
You do not need a long packet trace. You are checking ordering. A readable PROXY TCP4 line strongly suggests PROXY protocol v1. A binary preface suggests v2. TLS handshake bytes first tell you TLS begins before any HTTP semantics. That single observation often explains the failure faster than reading pages of configuration.
After that, send one controlled request with a simple correlation marker:
curl -H 'X-Debug-Trace: ip-lab-01' https://service.example.com/health
That marker is not the identity signal. It only helps you line up wire capture, intermediary logs, and application logs without guessing which request you are reading.
Now compare every consumer of client identity. Check the edge access log, the intermediate proxy log, the normalized application field, the rate-limit key, the access-control source field, and any audit trail that records request origin. They should all converge on the same normalized client address.
This matters even more when you validate mixed estates built around HTTP Proxies, where header-based preservation can look correct in an app log while policy engines still act on a different field. The formal model for forwarding identity in HTTP is defined in RFC 7239, which exists precisely because proxy chains can lose or rewrite origin information unless the semantics are made explicit.
Spot the failure pattern from the signal it leaks
Some failures are loud. Others are quietly wrong.
If the sender emits PROXY protocol but the listener does not expect it, the connection usually fails early. You may see broken parsing, handshake errors, or abrupt closes. If the listener expects PROXY protocol but receives plain traffic instead, you get the reverse symptom: invalid preamble, bad signature, or a request that never parses cleanly.
If forwarded headers are trusted from the wrong hop, the service may appear functional while remaining spoofable. This is often more dangerous than a hard failure because the logs still look believable. The system works until somebody relies on those fields for enforcement.
TLS ordering mistakes are another classic problem. If the receiver expects to parse PROXY metadata before TLS but the sender starts TLS immediately, the failure appears during connection setup. If TLS terminates earlier in the chain and identity is normalized later, your validation point moves with that trust boundary.
Do not stop at the application. Trigger a rate-limit rule. Test an allowlist or denylist path. Confirm that policy uses the same normalized identity visible in logs. This becomes especially important in estates where SOCKS5 Proxies or header-forwarding paths coexist with L4 identity handoff in neighboring services.
Use a production-safe rollout checklist
Validate one hop at a time instead of flipping the whole chain at once.
Confirm sender and receiver contracts for every listener on the request path.
Capture first bytes on the wire before changing trust rules.
Correlate one request across packet capture, proxy logs, application logs, and policy decisions.
Verify that rate limiting, access control, and audit logs all consume the same normalized client identity.
Test health checks separately. Many green status pages hide the fact that health probes and real user traffic do not traverse the same listener path.
Roll out behind a narrow slice first. This is especially useful when traffic characteristics vary under load, because identity bugs often hide behind otherwise successful requests. Teams validating layered paths with Rotating Proxies or other multi-hop patterns usually learn more from a controlled slice than from a big-bang deployment.
Close only when the wire, the listener, and policy agree
Client IP preservation is real only when the first bytes on the wire match the listener contract and every downstream consumer uses the same normalized identity. If logs look right but rate limiting and access controls still trust something else, the rollout is not finished.
That is the standard worth holding before production. Successful requests are not enough. What matters is whether the same client identity survives capture, parsing, normalization, logging, and enforcement without ambiguity.
Top comments (0)