DEV Community

Pinnadash
Pinnadash

Posted on

When curl prints 000: building an egress check that cannot lie

The alarm

Our daily ops job has one job: decide whether this machine can reach the public internet. It ran a plain curl against an overseas endpoint, got 000, and marked the outbound link down. The publish queue stopped. Nothing else in the stack complained.

Then the real egress check ran — the one that goes through the proxy the same way production traffic does — and reported up (HTTP 204, TLS verified).

Both tools were "measuring the network". They disagreed, so at least one of them was lying. It was the curl.

000 is not a status code. It is the absence of one.

curl -w '%{http_code}' prints three digits because the format string is three digits wide, not because it has an HTTP status to give you. 000 means "no HTTP response was received". DNS failure, connection refused, TLS handshake failure, timeout — all of them collapse into the same 000.

Which makes it a falsy integer that looks like data, and exactly the value that an if code != 200 check will silently convert into "the world is down".

The cause in our case: curl reads http_proxy / https_proxy from the environment, but it does not read the OS proxy settings — not the Windows WinINET/WinHTTP configuration, not an application's own "system proxy" toggle. The proxy was configured in a GUI. The environment variable was never exported into that subprocess. So curl honestly reported "I was told to connect directly, and direct is blackholed." The process was right. The inference was wrong.

Node's built-in fetch (undici) has the same trap: no system proxy, no HTTP_PROXY pickup. Without an explicit ProxyAgent the request goes direct and dies with an ECONNREFUSED that reads exactly like an outage.

Rule 1 — probe the path production actually uses

If your application's traffic goes through a proxy, the health check must go through the proxy. "Can this box reach the internet directly?" is a different question, and usually the answer doesn't matter.

Our probe became two stages, each with its own verdict:

def probe(proxy):
    # Stage 1: is the proxy even listening?
    if not tcp_alive(proxy, timeout=1.0):
        return {"ok": False, "stage": "port", "detail": "proxy port refused"}
    # Stage 2: can we get an HTTP response through it?
    try:
        with proxy_opener(proxy).open("https://www.google.com/generate_204", timeout=6) as r:
            return {"ok": True, "stage": "tls", "detail": "HTTP %s" % r.getcode()}
    except HTTPError as e:          # 4xx/5xx: TLS path worked, the remote answered
        return {"ok": e.code < 500, "stage": "tls", "detail": "HTTP %s" % e.code}
    except URLError as e:
        return {"ok": False, "stage": "tls", "detail": repr(e.reason)}
Enter fullscreen mode Exit fullscreen mode

Two things matter more than the code itself:

  • The stage is named in the output. "Proxy not running" and "proxy up, node dead" produce the same red light but completely different phone calls.
  • Any HTTP answer below 500 counts as a live path. A 403 from a remote host proves DNS, CONNECT and TLS all worked. The remote refusing you is not the network being down. Only "no response at all" is.

Rule 2 — give "unknown" its own state, and its own exit code

The original script had two outcomes: up and down. Anything that wasn't a 200 became "down" — including the script crashing. A typo in the check reported the network as broken, and a human spent an hour on proxy settings because of a NameError.

So: three codes, plus a rule that a bug may never look like an outage.

code meaning
0 path verified up
1 path verified down (port refused, or no response at all)
2 state could not be persisted / internal error — the reading is untrustworthy

The second half of the same idea lives in the dashboard. The state file stores ok plus last_ok_at, and the renderer maps ok: null to "not probed yet" — never to "down". The most valuable line in the whole file is this one:

"I didn't measure" is not "there was no data."

Uptime dashboards get this wrong constantly, and then a panel shows 0% for a host that was simply never scraped. 0% gets escalated; "no data" gets investigated by whoever owns the scrape. Different problems, different words.

The corollary for red days: keep last_ok_at and never overwrite it on failure. down; last success = 2026-10-24 09:12 tells you how long you've been out. A bare down restarts the clock every run, and you can no longer tell a five-minute blip from a three-day outage.

Rule 3 — exactly one function writes the human sentence

The probe writes the status string; the dashboard, the daily report and the alert channel all call that same function. As soon as a consumer composes its own phrasing, they drift — and on the day they drift you get a green panel beside a red alarm with two different timestamps, and nobody believes either.

Ours has four branches: not probed / up / down with history / down without history. Every branch is a sentence a human can act on, and "whose hand fixes this" is written into the string — because the fix for a dead proxy is a person opening a GUI on their desktop, and that is not something a monitoring job can do to itself.

Rule 4 — a 200 can still mean "gone"

While we were in there: a vendor's submission page answered 200 with a 39-byte empty document. Both of our tools called it "available". It wasn't — the page was a skeleton and the form had been removed. Two others in the same list answered 404, one refused to connect, and sorting by status code would have ranked them exactly backwards.

A pass state has to be producible only by the thing you actually care about. For "is the outbound path alive?" the only acceptable green is a real HTTP response through the real proxy. For "does the submission form still exist?" the answer is the form in the DOM, not a status line.

Why a false red is worse than no signal

A missed alert costs you one incident. A false alarm costs you every future alert: people learn that the red light means "go look at the box, it's probably nothing", and the real outage then queues up behind it. Our 000 didn't just stall one publish — it taught everyone a little more distrust of the check.

So the check must refuse to say "down" until it can prove that the path it tested is the path that matters. Everything above is one question asked four times: what else could produce this same result?

Write your probe so that its green is impossible to fake and its red is impossible to misread.

If you build dev tools, come get a key

We make small desktop utilities — batch image processing, a local-first disk manager, and a CLI — at pinnadash.com. Early users get 3 free months of membership: no card, no trial hoops, just a redeem key. Comment below or DM me here on dev.to and I'll send you one.

And if you've been burned by a lying health check, tell me which one — I'm collecting them.

Top comments (0)