DEV Community

Ionut-Robert Sandu
Ionut-Robert Sandu

Posted on

Your App Works Everywhere Except the Corporate Network

You ship something. It works on your machine, in CI, in staging, and for every user who tries it. Then one customer opens a ticket: it doesn't work for them. Same version, same config, same everything. It just hangs, or throws a certificate error, or fails in some way your error handling never anticipated.

They're on a corporate network. Somewhere between their machine and your server, a device is opening every TLS connection, reading it, and building a new one.

This is normal. Large organisations are often legally required to inspect traffic leaving their network, and they've been doing it for twenty years. The problem isn't that it happens. The problem is that from where you're standing, it's nearly invisible — and the failures it causes look like bugs in your code.

Here's what's actually happening, why five different things break in five different ways, and how to work out which one you're looking at without access to the customer's network team.

What interception actually does

A normal TLS connection is between your client and your server. The server presents a certificate, the client checks it chains to a certificate authority it trusts, and the two negotiate keys that nobody in the middle can derive. That's the entire point.

An inspection proxy breaks this into two connections. It terminates the client's TLS session itself, then opens a separate one to the real server. In the middle, it holds plaintext.

For the client to accept this, the proxy has to present a certificate for the site the client asked for. It generates one on the fly, signed by a CA whose root certificate the organisation has installed on every managed machine. On a managed laptop, this works transparently. The browser sees a valid chain to a root it trusts, and shows a padlock.

One consequence is worth stating plainly, because a lot of developers get it backwards: this isn't a vulnerability being exploited. It works because an administrator deliberately installed a root certificate. Browsers make a specific exception for locally-installed roots — Certificate Transparency requirements and most key-pinning checks are enforced for publicly-trusted CAs but relaxed for locally-added ones. If they weren't, corporate inspection would break the web for a large fraction of enterprise users, so the exception is deliberate.

That exception is also the reason the failures are so confusing. Interception works fine for the browser, which is why the customer insists their internet is "working" — and breaks for everything else.

Five mechanisms, five different failure signatures

1. Your language runtime doesn't use the system trust store

This is the most common one and the least known.

The corporate root CA gets installed into the operating system trust store. The browser picks it up. Your application might not, because several runtimes ship their own bundled list of certificate authorities and ignore the OS entirely:

  • Node.js bundles its own root store. It does not read the system store by default. The escape hatch is NODE_EXTRA_CA_CERTS, pointing at a PEM file.
  • Python, when using requests or anything else built on certifi, uses the certifi bundle rather than the system store. REQUESTS_CA_BUNDLE and SSL_CERT_FILE override it.
  • Java uses its own cacerts keystore, managed with keytool, entirely separate from the OS.
  • Go does read system roots on Linux and macOS, but any code that sets a custom tls.Config with its own RootCAs pool has opted out.

Signature: the site loads in the browser on the same machine, but your application throws a certificate verification error. That specific combination — browser fine, code broken — almost always means a trust store mismatch rather than anything wrong with the network.

Firefox is worth a special mention. It maintains its own trust store independently of the OS, so an environment where Chrome works and Firefox doesn't is the same problem wearing a different hat.

2. Certificate pinning

If your application pins a specific certificate or public key — common in mobile apps and in anything handling payments — you've explicitly said you will only accept one identity, regardless of what the trust store says. An inspection proxy cannot satisfy that. It doesn't have the private key.

Signature: fails on every corporate network, works everywhere else, and no amount of installing certificates fixes it. Unlike the trust store problem, this one is unfixable from the client side. That's the point of pinning.

If you pin, you need a documented way for administrators to allow-list your domain from inspection, and your error message should say so. Most apps that pin fail with a generic network error, which turns a five-minute fix into a week of support tickets.

3. Mutual TLS

If your server requires a client certificate, the proxy has to present one on the client's behalf. It doesn't have the client's private key, so it can't. Some proxies detect this and pass the connection through untouched; others don't, and the handshake dies.

Signature: the handshake fails after the server requests a certificate, often with an unhelpful message about a missing or bad certificate. The tell is that it fails at a specific handshake stage, not at verification.

4. Protocol upgrades and long-lived connections

WebSockets, server-sent events, gRPC streams and anything else that holds a connection open or upgrades it mid-flight depend on the middlebox handling that correctly. Many do. Some strip the Upgrade header, some apply an idle timeout far shorter than your keepalive interval, and some buffer responses that you intended to stream.

Signature: the initial request succeeds and then the connection dies, or the upgrade silently doesn't happen and you fall back to polling. Reconnect loops with no error are typical.

This one is nasty because it is intermittent and timing-dependent, which makes it look like a bug in your reconnection logic. If your connections die at a suspiciously round interval — sixty seconds, five minutes — that's an idle timeout somewhere in the path, not your code.

5. QUIC and HTTP/3

QUIC encrypts most of its transport metadata, which defeats the inspection techniques built for TLS over TCP. The usual enterprise response is to block UDP/443 outright and force clients back to TCP.

Browsers handle this gracefully — they try QUIC, fail, and fall back. Custom clients often don't.

Signature: a delay of several seconds before every connection, or a hang, on one network only. If your HTTP client prefers HTTP/3, test it with QUIC disabled before blaming anything else.

Triage from the client side

You usually can't get access to the customer's network. You can get them to run two commands.

The first question is always: is the connection being intercepted at all?

openssl s_client -connect example.com:443 -servername example.com
Enter fullscreen mode Exit fullscreen mode

Look at the issuer line. On an unintercepted connection you'll see a public CA. On an intercepted one you'll see something internal — the organisation's name, or a firewall vendor's. That single line answers the question.

To see the whole chain, including what the proxy is signing with:

openssl s_client -connect example.com:443 -servername example.com -showcerts
Enter fullscreen mode Exit fullscreen mode

Then compare what your application sees against what the system sees:

curl -v https://example.com
Enter fullscreen mode Exit fullscreen mode

If curl succeeds and your application fails on the same machine, you're in mechanism 1 — a trust store mismatch. If curl fails too, the problem is at the network layer and affects everything.

To confirm a certificate issue specifically rather than a network one, try without verification:

curl -vk https://example.com
Enter fullscreen mode Exit fullscreen mode

If it works with -k and fails without, it's trust. If it fails both ways, it isn't. Never leave -k in anything but a diagnostic.

For a suspected HTTP/3 problem, there is a trap worth knowing about. curl does not negotiate HTTP/3 by default — that needs a build with QUIC support and an explicit flag. So check what you actually got before drawing conclusions:

curl -sS -o /dev/null -w "%{http_version}\n" https://example.com
Enter fullscreen mode Exit fullscreen mode

If that prints 2, curl never attempted QUIC, and comparing it against --http1.1 proves nothing at all. Force the attempt instead:

curl -v --http3 https://example.com
Enter fullscreen mode Exit fullscreen mode

If the explicit HTTP/3 attempt hangs or fails while the ordinary request succeeds, UDP/443 is being blocked somewhere in the path. Browsers conceal this, because they try QUIC, fail, and fall back silently — which is precisely why testing in a browser will tell you everything is fine.

What to build so this is less painful

You will not stop corporate networks from inspecting traffic. What you can do is stop your application from being the hardest part of the diagnosis.

Say what actually failed. "Network error" is worthless. "Certificate verification failed: issuer not trusted" points the user at their administrator immediately. Surface the issuer name if you have it.

Document the domains you need. Every serious enterprise vendor publishes a list of hostnames to allow-list, and states plainly if a domain must be exempt from inspection. If you pin certificates, this documentation is not optional.

Make the trust store configurable. Respect NODE_EXTRA_CA_CERTS, SSL_CERT_FILE, or whatever your runtime's equivalent is, and say so in your docs. Administrators know what to do with that.

Never disable verification as a workaround. The number of production systems running with verification switched off because someone hit this once and needed it working by Friday is genuinely alarming. It converts a support ticket into a permanent vulnerability, and nobody ever turns it back on.

Test against an interception proxy. mitmproxy in front of your integration tests, with its CA installed, reproduces most of this in an afternoon. It's a better use of time than debugging it live against a customer who can't share their network configuration.

The short version

If a customer reports that your application fails only on their network, the question isn't whether your code is wrong. The question is which of five things a middlebox is doing to your connection — and the first openssl s_client tells you most of it.

Top comments (1)

Collapse
 
p_o_26e854a54d851cd606f08 profile image
P O

this is usually a transparent proxy or ssl inspection thing. i start with curl -v to the same host from inside and outside, then compare the cert chain. if the leaf issuer changes on the corporate path, youre looking at mitm inspection not an app bug.