48-Hour Field Notes: localhost Resolved Fine. The Listener Never Saw the Packet.

Have you ever watched a local health probe succeed on a laptop and then fail on Linux without changing a single line of application code? I burned forty-eight hours on that loop, staring at a listener that looked healthy while every client still reported connection refused. The process table claimed the HTTP server was bound, and curl on the same host still bounced, which felt almost rude. So what was actually answering when I typed localhost, and why did that answer disagree with the bind address I had chosen?

The first night of wrong theories

I started where most of us start, which means I blamed the firewall, the port, and a stale process that I could not see. I ran lsof, ss, and netstat until the terminal looked like a crime scene, and every tool agreed that port 8000 was occupied. I killed the process, rebound the socket, and probed again, because repeating a ritual sometimes feels like progress even when it is not. Nothing changed, which is the moment a field note should stop guessing and start printing the addresses the resolver actually returns.

Night one, the cargo-cult checklist

I wrote the failed checks down so I would not run them twice without a new question attached.

1. Confirm the Python process still owns port 8000 after the restart, and capture the exact command line that launched it.
2. Probe with curl against http://127.0.0.1:8000/health and against http://localhost:8000/health, then save both exit codes.
3. Restart the listener with an explicit bind host instead of a default, then repeat the two probes before touching application logic.
4. Dump /etc/hosts and the glibc resolver order, because localhost is a name, not a promise about address family.

The laptop made both probes look identical, which is how I talked myself into ignoring the hostname for half a day. On the Linux box the numeric address worked and the name failed, and that split should have ended the mystery immediately. Did I notice? No. I kept reading framework docs as if a web stack could hide a TCP bind family behind a health JSON payload.

What actually broke

The server had been created with a bind host of 127.0.0.1, which is IPv4 only, and that choice is silent until the client arrives on another family. localhost on that Linux image resolved to ::1 first, so the client opened an IPv6 socket toward a port where nothing was listening. Connection refused was the honest answer, not a flaky framework, not a missing route, and not a health-check race. Why did the laptop hide it? Because its resolver order, Happy Eyeballs behavior, and dual-stack listener defaults were simply not the same shape as the Linux box.

I needed a Linux shell that was not my laptop, because the failure refused to show up on macOS in a reliable way. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used MonkeyCode's free server option as a throwaway Linux shell so I could compare Linux getaddrinfo with my laptop. I also used free model access to draft the first dump script, then I edited the families and error handling by hand. That pairing did not diagnose the bug for me, because the printed sockaddr tuples were already enough.

The dump I should have run in hour one

Label this as a reproduction I actually ran on both machines, not as a benchmark and not as a claim about anyone else's network. Save it as dump_localhost.py and run it with the same Python you use to boot the listener.

#!/usr/bin/env python3
import socket
import sys

HOST = sys.argv[1] if len(sys.argv) > 1 else "localhost"
PORT = int(sys.argv[2]) if len(sys.argv) > 2 else 8000

FAMILY_NAMES = {
    socket.AF_INET: "AF_INET",
    socket.AF_INET6: "AF_INET6",
}

print(f"has_dualstack_ipv6={socket.has_ipv6 and socket.has_dualstack_ipv6()}")
print(f"getaddrinfo({HOST!r}, {PORT})")

for family, socktype, proto, canon, sockaddr in socket.getaddrinfo(
    HOST, PORT, type=socket.SOCK_STREAM
):
    print(
        FAMILY_NAMES.get(family, family),
        socktype,
        proto,
        canon or "-",
        sockaddr,
    )

Then bind an IPv4-only stdlib server so the mismatch cannot hide inside a framework default.

#!/usr/bin/env python3
from http.server import BaseHTTPRequestHandler, HTTPServer

class Health(BaseHTTPRequestHandler):
    def do_GET(self):
        body = b"ok\n"
        self.send_response(200)
        self.send_header("Content-Type", "text/plain")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, format, *args):
        return

if __name__ == "__main__":
    # IPv4 only on purpose. This is the bug, not a production bind policy.
    server = HTTPServer(("127.0.0.1", 8000), Health)
    print("listening on 127.0.0.1:8000", flush=True)
    server.serve_forever()

The probe that finally made the split visible is boring, which is the point of keeping it outside a test runner.

python3 dump_localhost.py localhost 8000
python3 dump_localhost.py 127.0.0.1 8000
python3 dump_localhost.py ::1 8000

# Terminal A
python3 bind_ipv4.py

# Terminal B
curl -sS -m 2 http://127.0.0.1:8000/health ; echo ipv4_exit:$?
curl -sS -m 2 http://localhost:8000/health ; echo name_exit:$?
curl -g -sS -m 2 http://[::1]:8000/health ; echo ipv6_exit:$?
ss -ltnp | grep 8000 || netstat -ltnp | grep 8000

On the Linux shell, getaddrinfo("localhost", 8000) listed ::1 first and 127.0.0.1 second, which matches glibc preferring IPv6 when both records exist. curl http://localhost:8000/health then followed that first result and died, while 127.0.0.1 returned ok. On the laptop both names often landed on IPv4, so the same source tree looked green until I left my own disk.

A decision table I now keep next to bind code

I got tired of rediscovering this in comments, so I turned the forty-eight hours into a table I can reread before changing a host string.

Client target	Listener bind	Typical result	What I do next
127.0.0.1	127.0.0.1	Works on IPv4 only	Keep for local-only IPv4 services
localhost	127.0.0.1	Fails when ::1 is first	Dump getaddrinfo before blaming the app
::1	127.0.0.1	Always refused here	Do not mix families and call it a timeout
localhost	:: or ::1	Works if IPv6 is actually enabled	Confirm ss shows tcp6
0.0.0.0	0.0.0.0	IPv4 on all interfaces	Still not IPv6, despite the all-zeros look
localhost	127.0.0.1 plus [::1]	Works if both sockets exist	Two binds, or one dual-stack socket

Would I trust this table on Windows without a dump? No, because Winsock order and IPv6 policy can disagree with Linux in ways this note does not cover. Would I disable IPv6 on the host to make the table prettier? Also no, because that hides the next dual-stack bug instead of making the bind explicit.

The workflow I would repeat

If the name works on one machine and the numeric address works on another, I now refuse to open the application log first. I run the dump, I bind with an explicit family, and I probe three targets before I touch retries, timeouts, or health-check intervals. That order would have saved both nights, and it still fits in a single shell scrollback. The steps below are the repeatable part of this note, not a claim that every refused connection is IPv6.

1. Print socket.getaddrinfo for localhost, 127.0.0.1, and ::1 using the same interpreter that runs the server.
2. Ask ss or netstat whether the listener is 127.0.0.1, 0.0.0.0, ::1, or * on tcp6, and refuse to guess from a PID alone.
3. Probe the three targets with a two-second curl budget so a refused IPv6 socket cannot look like a hung application.
4. Change one thing: bind host, client host, or /etc/hosts order, then rerun the dump instead of stacking config changes.
5. Only after the families match do I look at frameworks, reverse proxies, or container network plugins.

A bind I now prefer for a local-only Python smoke server is explicit and loud, even if it is uglier than localhost in a README. Dual-stack needs a real decision, not a default that happened to work on a laptop.

import socket
from http.server import BaseHTTPRequestHandler, HTTPServer

def bind_dual_or_fail(host, port, handler):
    # Proposal for local smoke tests only. Labelled because production bind
    # policy belongs in the service mesh or unit file, not in a gist.
    httpd = HTTPServer((host, port), handler, bind_and_activate=False)
    httpd.socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    if host in {"::", "::1"}:
        try:
            httpd.socket.setsockopt(socket.IPPROTO_IPV6, socket.IPV6_V6ONLY, 0)
        except OSError:
            pass
    httpd.server_bind()
    httpd.server_activate()
    return httpd

I still would not drop that helper into production unchanged, because IPV6_V6ONLY behavior depends on the kernel, the container runtime, and whether IPv6 is even routed. The artifact I trust is the dump plus the three probes, not a clever socket option I copied from a gist at two in the morning.

Limitations, and who should skip this

This note is a bind-family checklist, not a dual-stack design guide, and it will mislead you if you treat it as one. It assumes a POSIX shell, a Python 3 stdlib HTTP server, and a host where localhost is defined in /etc/hosts for both families. It does not measure latency, it does not compare products, and it does not claim that IPv6-first resolution is a bug in glibc.

Skip this approach if you are debugging a public load balancer, a Kubernetes Service that already pins ipFamilyPolicy, or a client that never uses hostnames. Skip it if your failure is TLS, HTTP/2 settings, or an idle timeout, because those can mimic connection refused once you start guessing from logs alone. Skip it if you cannot run getaddrinfo on the same kernel that runs the listener, because copying a dump from a laptop is how I wasted the first night.

I also would not use a free remote shell as a production stand-in, and I would not paste secrets into any throwaway environment just to reproduce a bind. The Linux box only had to answer one question: which sockaddr does localhost return first, and is anything listening there?

What I keep from the forty-eight hours

The useful scar is small enough to fit on a sticky note, which is how I know the rest was noise. localhost is a name with an ordered address list, and a bind host is a single family unless I make it otherwise. When those two disagree, the logs will sound like an application failure, and they will keep sounding like one until I print the tuples. Next time the probe is green on my laptop and red on Linux, I will run dump_localhost.py before I reread a single framework page.