48-Hour Field Notes: localhost Resolved. The Listener Never Saw ::1.

Why did a plain health check against localhost succeed on my laptop and hang on a clean Linux box? I burned forty-eight hours on that mismatch, and the process table never showed a crash or a bind error. The port looked open, the application logs stayed quiet, and localhost still resolved to something that looked trustworthy. Have you ever trusted that hostname because every tutorial uses it, then watched only one machine honor the assumption?

What broke in the first eight hours

I started from the wrong story, which is almost always how these notes begin for me. The worker process stayed up, so I treated the failure as a slow boot and stretched every timeout I could find. Health checks moved from two seconds to ten, then to thirty, and the only result was a slower red dashboard. Does that sound familiar, or is it just my default panic move when a loopback URL misbehaves?

Here is what I actually changed before I understood the socket:

1. Increased the HTTP client timeout from two seconds to thirty seconds.
2. Added a sleep in the entrypoint because I assumed migrations were racing the bind.
3. Restarted the process manager until the logs repeated the same listening line.
4. Opened the port with a manual curl from the same host, still using localhost.

None of those steps were useless in every future incident, but they were useless for this particular bug. The application had bound successfully, the client still had a route, and the name service answered without an error. I was debugging latency that did not exist, which is a humbling way to spend a night.

What I tried next, and why it still lied

Between hour eight and hour twenty-four I blamed DNS, then a container proxy, then an overzealous host firewall. I printed /etc/hosts on both machines and felt clever when both files mentioned localhost in the usual way. I compared ss -lnt output and saw 127.0.0.1:8000, which I cheerfully read as proof that the port was fine. Have you noticed how easy it is to stop reading after the port number and miss the address family?

The laptop curl was IPv4 without me asking for IPv4, because that machine's resolver order happened to agree with my bind. The clean box preferred IPv6 for the same hostname, and my listener had never joined that family at all. ss was telling the truth in a dialect I refused to hear until a SYN on ::1 went unanswered. I captured packets for an hour, then finally admitted the hang was not a slow handler.

Commands I ran, in the order that slowly got less wrong:

ss -lnt | grep 8000
getent ahosts localhost
python3 probe_localhost.py localhost 8000
curl -v --max-time 3 http://localhost:8000/health
curl -v --max-time 3 http://127.0.0.1:8000/health
curl -g -v --max-time 3 "http://[::1]:8000/health"

The localhost curl hung on the clean host while the process still sat in a listen state. The 127.0.0.1 curl returned 200 immediately, which should have ended the timeout theory on the spot. The [::1] curl failed fast, and that fast failure was the first honest signal of the night. Why did I need a failing IPv6 probe before I believed an IPv4 listen line?

Hour twenty-four: Python agreed with getaddrinfo, not with my story

I stopped arguing with curl and asked CPython what it would do with the same hostname and port. This is the snippet I wish I had run on hour one, because a probe you will paste beats a framework you will postpone. It prints every socket.getaddrinfo result in order, which is the order a typical client will try. Run it on every machine; one laptop printout is how this bug survives.

# probe_localhost.py — run this on every machine
import socket
import sys

def inspect(host: str, port: int) -> None:
    print(f"python={sys.version.split()[0]} host={host!r} port={port}")
    infos = socket.getaddrinfo(host, port, type=socket.SOCK_STREAM)
    if not infos:
        print("  (no results)")
        return
    for family, socktype, proto, canon, sockaddr in infos:
        names = {
            socket.AF_INET: "AF_INET",
            socket.AF_INET6: "AF_INET6",
        }
        family_name = names.get(family, str(family))
        print(
            f"  {family_name:8} socktype={socktype} "
            f"proto={proto} sockaddr={sockaddr}"
        )

if __name__ == "__main__":
    host = sys.argv[1] if len(sys.argv) > 1 else "localhost"
    port = int(sys.argv[2]) if len(sys.argv) > 2 else 8000
    inspect(host, port)

On the laptop, IPv4 often appeared first, or IPv6 was missing because the stack was configured that way. On the clean Linux box, AF_INET6 with ::1 came first, and my server socket was AF_INET bound to 127.0.0.1. Two truthful answers can still produce one broken handshake, which is the part that feels unfair. Have you been burned by happy-path order in getaddrinfo before, or was this one new for you too?

I reproduced the bind with the same mistake many tutorials still teach without comment:

# ipv4_only_listen.py — labeled example, not production code
import socket

server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
server.bind(("127.0.0.1", 8000))
server.listen(1)
print("listening on 127.0.0.1:8000 only")
conn, addr = server.accept()
print("accepted", addr)
conn.close()
server.close()

If a client honors IPv6 first, that accept never runs and the process still looks healthy. That pattern is the whole incident, and it hides behind every log line that only says listening. I wish the listening line had printed the family, not just the port.

Hour thirty-six: a clean server, not another tweak on my laptop

I needed a machine that did not inherit my laptop's resolver order, extra loopback aliases, or desktop IPv6 toggles. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used MonkeyCode's free server option as that clean Linux environment, and I used its free model access to turn the messy notes into the probe script above. I am not claiming a model name, a quota, a hardware profile, or a duration, because those details are not the lesson here.

The lesson is that the second environment has to be actually second, not a container that still shares your desktop's network personality. I pasted the same probe_localhost.py onto that box and compared the printed families side by side without editing the binary. The model pass helped expand the decision table, and it did not invent a root cause I had not measured on both hosts. If you strip that product out of this writeup, the probe and the curl trio still stand on their own.

Would I skip the clean box next time, now that I know the pattern by heart? Probably not, because laptops lie with confidence, and resolver order is a local setting pretending to be a universal law. A second printout is cheaper than another night of timeout archaeology. I will keep that box in the test plan even when I feel sure.

Decision table I will keep next to the probe

What you observe	Likely family mismatch	What to try first
localhost hangs, 127.0.0.1 returns 200	Client used ::1, server bound IPv4 only	Bind dual-stack, or connect to 127.0.0.1
localhost hangs, [::1] returns 200	Client used IPv4, server bound IPv6 only	Bind :: with IPV6_V6ONLY=0, or listen on both
Both numeric forms work, the hostname fails	Resolver, search domain, or nsswitch issue	Inspect getent ahosts and /etc/nsswitch.conf
Laptop works, clean host fails the same binary	Local IPv6 policy or /etc/hosts differs	Run the probe on both; do not compare curl alone
ss shows 127.0.0.1:port, clients still time out	You stopped reading after the port number	Check AF_INET versus AF_INET6 on the listener

Fixes I would actually repeat, in order of least surprise:

• Make the client use 127.0.0.1 when you truly want the IPv4 loopback.
• Make the server listen on IPv4 and IPv6, or on :: with dual-stack enabled if the kernel allows it.
• Keep health checks on a numeric address in production configs, then keep the hostname for humans.
• Print getaddrinfo output in the boot log when a loopback service starts, before the first client arrives.

A dual-stack listen sketch, labeled as an example you must test on your kernel:

# dual_stack_listen.py — example, verify IPV6_V6ONLY on your platform
import socket

server = socket.socket(socket.AF_INET6, socket.SOCK_STREAM)
server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
server.setsockopt(socket.IPPROTO_IPV6, socket.IPV6_V6ONLY, 0)
server.bind(("::", 8000))
server.listen(1)
print("listening on ::8000 with dual-stack enabled")

Some platforms ignore IPV6_V6ONLY=0, and some containers disable IPv6 entirely by policy. That is why the probe exists, and why I will not treat the sketch as a portable guarantee. Test the families, then keep the table, then change the bind.

A forty-eight hour test plan I can reuse

I want a checklist that does not depend on memory of this incident. Run it whenever a local health check works on one machine and times out on another. Label the results with hostname, numeric IPv4, and numeric IPv6, then stop guessing.

1. Start the server and record the exact ss -lnt line, including the address, not only the port.
2. Run probe_localhost.py on the client host and save the family order.
3. Curl 127.0.0.1, then [::1], then localhost, each with a three-second cap.
4. If only the hostname hangs, fix bind or client address; do not increase timeouts yet.
5. Repeat the same four steps on a second environment that does not share the laptop resolver.

That fifth step is the one I skipped until hour thirty-six, and skipping it cost me the first day. The plan is boring on purpose, because boring probes survive better than clever theories. Would I still packet-capture later? Yes, but only after the table has a row.

What I would repeat, and what I would not

I would run the probe before I touch timeouts, because timeouts hide family mismatches as a slow network. I would curl three addresses, not one, and I would paste all three transcripts into the ticket without summarizing them. I would refuse to trust a laptop ss listing as a universal truth about loopback. I would keep the clean-box check, even when the first machine looks identical on paper.

I would not start with packet captures again, at least not before getaddrinfo prints something I can read. I would not bump retries until a numeric address fails too, because retries amplify the wrong family. I would not let a model invent a root cause before both environments printed families in order. The model helped me write a table, and the sockets told me the story.

Limitations, and who should not copy this

This workflow is for loopback and local health checks, not for public dual-stack load balancers you have not measured yourself. It will not explain TLS failures, HTTP/2 settings, or application deadlocks that only look like connect timeouts from the client. It also will not save you if the process never bound any port, which is a different field note and a different probe.

Skip the dual-stack bind if you are in a network policy that forbids IPv6, or if your runtime already wraps sockets for you. Skip a remote server if you cannot execute the probe on it, because reproducing family order requires running code. Do not treat my laptop-versus-clean-box split as a benchmark or a ranking; it is one incident shape, not a performance claim.

If you want a second Linux environment for the same probe, MonkeyCode's free server option is one place I used, and the script does not depend on staying there. Copy the probe first, then argue with timeouts, then argue with me in the comments if your family order still disagrees. I will probably ask whether you curled 127.0.0.1 before you curled localhost.