I Blamed the Firewall for 48 Hours. localhost Resolved to IPv6 First

I started with a boring health check, the kind you write when a remote box should just answer on a port. Locally the worker looked fine, and curl printed ok so fast I almost shipped the script. Then the same probe on the remote Linux box failed with connection refused, and I spent two days arguing with a firewall that was never the villain. Have you ever trusted localhost so hard that you forgot it is a name, not an address?

This is a 48-hour field notebook, not a victory lap. I will show what I tried, what actually broke, and the small reproduction I wish I had run in the first hour. If you strip every product name out of this post, the checklist still stands on its own.

Hour 0: a tiny worker, a free model, a free server

I wanted a disposable Python worker I could restart without ceremony, plus a one-line probe I could run from cron. I asked a coding assistant for a stdlib-only HTTP handler, because I did not want a framework arguing with me on a scratch box. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used MonkeyCode's free model access to draft the first worker, then ran that draft on the free server option so the failure would show up on a real remote filesystem.

The generated shape was honest enough. A ThreadingHTTPServer, a /healthz path, and a bind host that looked like every tutorial you have already copied. I did not treat the draft as production code. I treated it as a hypothesis I could break on purpose.

# health_worker.py — first draft, labeled as generated starting point
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path != "/healthz":
            self.send_error(404)
            return
        body = b"ok\n"
        self.send_response(200)
        self.send_header("Content-Type", "text/plain; charset=utf-8")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, fmt, *args):
        return

def main():
    server = ThreadingHTTPServer(("localhost", 8080), Handler)
    server.serve_forever()

if __name__ == "__main__":
    main()

Does that look harmless to you? It looked harmless to me, which is how these notes always start.

Hours 1–8: the wrong suspects

I did what I always do when a remote probe dies. I blamed the network, then the process, then the person who wrote the probe. The field notes from that stretch are ugly because they are familiar.

• Confirmed the process was alive with ps and ss, then assumed a security group was dropping packets.
• Opened the same port with python -m http.server and watched it fail in a slightly different way.
• Rewrote the probe in requests, then in urllib, then in raw socket.create_connection.
• Added retries, which only made a refused connection look like a flaky connection.

The commands I actually ran looked like this, and none of them named the real bug.

python3 health_worker.py &
ss -ltnp | grep 8080
curl -v http://localhost:8080/healthz
curl -v http://127.0.0.1:8080/healthz
python3 -c "import socket; socket.create_connection(('localhost', 8080), 2)"

On my laptop, both curl lines returned ok. On the remote box, localhost failed and 127.0.0.1 sometimes failed too, depending on which process I had left running. Why would two names that I have treated as synonyms for a decade disagree now?

Hours 9–24: print the thing localhost actually means

I finally stopped curling and asked Python what it believed. That is the moment these notes stop being folklore and become a reproduction. socket.getaddrinfo is the API your HTTP client is already using, even when you never import it yourself.

# resolve_localhost.py — run this on the same interpreter as the worker
import socket
import sys

HOSTS = ("localhost", "127.0.0.1", "::1")
PORT = 8080

print("python", sys.version.split()[0])
print("has_ipv6", socket.has_ipv6)
print("getdefaulttimeout", socket.getdefaulttimeout())
print()

for host in HOSTS:
    print(f"== {host!r} ==")
    try:
        infos = socket.getaddrinfo(
            host,
            PORT,
            type=socket.SOCK_STREAM,
        )
    except socket.gaierror as exc:
        print("  gaierror:", exc)
        continue
    for family, socktype, proto, canon, sockaddr in infos:
        fam = {socket.AF_INET: "AF_INET", socket.AF_INET6: "AF_INET6"}.get(
            family, family
        )
        print(f"  {fam:8} {sockaddr}")

On the remote box, localhost came back as ::1 first. The worker had bound an IPv4 socket to 127.0.0.1 because ThreadingHTTPServer(("localhost", 8080)) followed a different path than my probe. One process listened on IPv4 loopback. The other happily connected to IPv6 loopback and found nobody home. Is that a firewall? No. That is two loopbacks pretending to be one hometown.

I also printed the listening sockets instead of trusting ss output I had skimmed too quickly.

ss -ltnH '( sport = :8080 )'
python3 - <<'PY'
import socket
from contextlib import closing

def try_bind(host):
    sock = socket.socket(socket.AF_INET if ":" not in host else socket.AF_INET6,
                         socket.SOCK_STREAM)
    sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    try:
        sock.bind((host, 8080) if host != "::1" else ("::1", 8080, 0, 0))
        sock.listen(1)
        print(f"bind ok  {host!r:12} -> {sock.getsockname()}")
    except OSError as exc:
        print(f"bind fail {host!r:12} -> {exc}")
    finally:
        sock.close()

for host in ("127.0.0.1", "::1", "localhost", "0.0.0.0"):
    try_bind(host)
PY

The bind test made the split obvious. localhost was not a stable bind target on that host. 127.0.0.1 and ::1 were different living rooms with the same furniture.

Hours 25–40: make the worker and the probe share an address family

I stopped letting either side choose. The worker now takes --host and --port, and the probe takes the same pair. No more romantic localhost in a remote health check. If I want dual stack, I say so in the code instead of hoping glibc feels generous.

# health_worker.py — explicit bind, still stdlib only
import argparse
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path != "/healthz":
            self.send_error(404)
            return
        body = b"ok\n"
        self.send_response(200)
        self.send_header("Content-Type", "text/plain; charset=utf-8")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, fmt, *args):
        return

def parse_args():
    parser = argparse.ArgumentParser()
    parser.add_argument("--host", default="127.0.0.1")
    parser.add_argument("--port", type=int, default=8080)
    return parser.parse_args()

if __name__ == "__main__":
    args = parse_args()
    ThreadingHTTPServer((args.host, args.port), Handler).serve_forever()

The probe became equally boring, which is what I wanted after a day of poetry.

# probe.py — same host string the worker used, no DNS surprise
import argparse
import socket
import sys
import urllib.error
import urllib.request

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--host", default="127.0.0.1")
    parser.add_argument("--port", type=int, default=8080)
    args = parser.parse_args()
    url = f"http://{args.host}:{args.port}/healthz"
    try:
        with urllib.request.urlopen(url, timeout=2) as resp:
            body = resp.read()
    except (urllib.error.URLError, socket.timeout) as exc:
        print(f"FAIL {url} {exc}")
        return 1
    if body != b"ok\n":
        print(f"FAIL {url} body={body!r}")
        return 1
    print(f"PASS {url}")
    return 0

if __name__ == "__main__":
    sys.exit(main())

Would I still use localhost in a README? Maybe, if the README also prints getaddrinfo. I would not use it as the only contract between a worker and a probe on a machine I did not provision myself.

A decision table I should have drawn at hour one

Bind host	Probe host	What I observed	Keep it?
localhost	localhost	Family order depends on the host resolver	No
localhost	127.0.0.1	Worker may be on ::1 while probe uses IPv4	No
127.0.0.1	127.0.0.1	Stable on dual-stack boxes I tested	Yes, for local-only
::1	::1	Stable if IPv6 loopback exists	Yes, if you mean IPv6
0.0.0.0	127.0.0.1	Listens on all IPv4 interfaces	Only if that is intended
0.0.0.0	localhost	Probe may still walk to ::1 first	No

I keep this table next to the script now. It is uglier than a slogan, and it would have saved me a day of ufw folklore.

The 48-hour test plan I will repeat

This is the sequence I will run before I believe any generated server on a remote box. It is short on purpose, because I will not remember a twelve-step ritual at 2 a.m.

1. Print sys.version, socket.has_ipv6, and getaddrinfo("localhost", port) on the same interpreter.
2. Bind with an explicit numeric host, then confirm ss shows that exact tuple.
3. Probe with the same numeric host, not a nickname, and fail on any other body than ok\n.
4. Repeat the probe over IPv4 and IPv6 only when I have chosen dual stack in the bind.
5. Keep the first generated draft, then diff it against the bind change, so I know what the assistant actually assumed.

What broke for me was not the HTTP handler. The handler was fine. The assistant, and then I, treated localhost as a portability layer. It is a resolver policy, and remote boxes do not share your laptop's policy.

What I would repeat is the cheap remote run. A free server is useful here because the mismatch does not always reproduce on a developer laptop that happily maps localhost both ways. I still review every bind line by hand. I do not ask a model to certify a socket family.

Limitations, and who should not copy this

This workflow is for a throwaway worker and a loopback probe. It is not a production ingress design, and it does not replace TLS, authentication, or a real process manager. ThreadingHTTPServer is a teaching socket, not an application platform, and I am not publishing latency numbers I never measured.

Do not put secrets, customer data, or privileged credentials on a free shared server. Do not bind 0.0.0.0 just to make a probe pass, especially if the box is reachable beyond your laptop. If you need a guaranteed runtime, a pinned Python version, or an SLA, this scratch-server path is the wrong tool and I will not pretend otherwise.

The assistant can scaffold the handler. It cannot see getaddrinfo order on a machine it is not running on, unless you paste that output back into the conversation. I wasted hours because I pasted curl failures instead of resolver rows. Would I do that again? Only if I enjoy arguing with IPv6 ghosts.

If you run resolve_localhost.py on a remote box this week, I want the getaddrinfo listing that surprised you, not another screenshot of a red firewall rule.