48-Hour Field Notes: connect() Hung Because getaddrinfo Handed Me IPv6 First

Have you ever stared at a Python client that hangs on connect(), while curl to the same host answers immediately? I have, and I still walk into the same trap whenever I am tired enough to skip the dump. These are the 48-hour field notes I now keep in a text file beside the terminal. They are a lab log on purpose, not a war story dressed up with invented dashboards.

The symptom always looks like a firewall, a security group, or a flaky load balancer. Would you actually check address families first, or would you raise the timeout and hope? I raised the timeout, then the retry count, and that was the wrong hour-one move. The rest of this log is the cheaper path I should have taken on hour one.

Hour 0–2: the lie I told myself

I reproduced a tiny service on loopback so the notes would stay honest and local. The server listened on IPv4 only, which is a common default people forget they chose. The client used a hostname, and that hostname had more than one DNS-shaped answer. Nothing in this lab needed a public load balancer sitting in the way.

What did the process look like from the cheap seats, before any packet capture? The HTTP server was up, the port was open, and ss agreed with that story. The client still sat in connect() like the network had vanished. Would you trust ss here, or would you ask which address family that listener actually owns?

I blamed the firewall for a long stretch, because that story is easy to tell. I also blamed Docker networking, then nginx, then a supposedly slow Python HTTP library. None of those tools were the first villain in the trace. The first villain was the tuple I kept from getaddrinfo.

The IPv4-only lab server

Here is the lab server I actually ran, and it binds IPv4 only on purpose:

# ipv4_only_server.py
# Lab setup: listen on 127.0.0.1 only, never on ::
from http.server import ThreadingHTTPServer, SimpleHTTPRequestHandler


def main() -> None:
    server = ThreadingHTTPServer(("127.0.0.1", 8080), SimpleHTTPRequestHandler)
    print("listening on 127.0.0.1:8080", flush=True)
    server.serve_forever()


if __name__ == "__main__":
    main()

The client that keeps infos[0]

And here is the naive client I have written more than once, under deadline pressure:

# naive_client.py
# Label: this is the bug, not a recommendation.
import socket


def fetch(host: str, port: int) -> None:
    infos = socket.getaddrinfo(host, port, type=socket.SOCK_STREAM)
    family, socktype, proto, _, sockaddr = infos[0]  # first answer wins
    with socket.socket(family, socktype, proto) as sock:
        sock.settimeout(30)
        sock.connect(sockaddr)
        sock.sendall(b"GET / HTTP/1.0\r\nHost: lab\r\n\r\n")
        print(sock.recv(256))


if __name__ == "__main__":
    fetch("localhost", 8080)

Do you see the landmine sitting in that first assignment? infos[0] is not the reachable address, and it never promised to be. It is whatever getaddrinfo decided to hand back first on this libc, this /etc/hosts, and this interface set. On a dual-stack laptop that first row is often ::1.

Hour 3–8: commands that did not help

I did the ritual that feels like progress and rarely is, because it changes knobs instead of questions. I raised timeouts. I added retries. I restarted a reverse proxy even though no reverse proxy was in the path. Those hours produced a pile of noise, so I am listing it here.

1. Doubled sock.settimeout from 30 to 60, then to 120, and learned nothing new.
2. Wrapped the call in a retry loop that retried the same first address forever.
3. Ran curl http://127.0.0.1:8080/ and then declared the service fully healthy.
4. Compared that success with curl http://localhost:8080/ and still missed the family split.
5. Blamed Python, then blamed curl, then blamed "the network" as if it were one object.

Why did curl sometimes work when my script failed in the same shell? curl often walks addresses, or I had passed it a literal 127.0.0.1 without noticing. My script asked for localhost and kept the first tuple like a souvenir. That is not the same experiment, and it is not the same client. Are we even measuring one code path when the host strings differ?

The artifact: print every address, then time each connect

I finally wrote the dump I should have written at hour one, before any retry logic existed. It does not fix production, and it does not pretend to. It makes the lie visible as rows, elapsed time, and an exception name.

# addrinfo_field_notes.py
"""Reproducible getaddrinfo / connect field notes. Lab use only."""
from __future__ import annotations

import socket
import time
from typing import List, Tuple


def dump_addrinfo(host: str, port: int) -> List[Tuple]:
    rows = socket.getaddrinfo(host, port, type=socket.SOCK_STREAM)
    print(f"getaddrinfo({host!r}, {port}) -> {len(rows)} rows")
    names = {socket.AF_INET: "AF_INET", socket.AF_INET6: "AF_INET6"}
    for i, (family, socktype, proto, canon, sockaddr) in enumerate(rows):
        fam = names.get(family, family)
        print(f"  [{i}] family={fam} socktype={socktype} proto={proto}")
        print(f"      sockaddr={sockaddr} canon={canon!r}")
    return rows


def timed_connect(family, socktype, proto, sockaddr, timeout: float) -> str:
    start = time.monotonic()
    sock = socket.socket(family, socktype, proto)
    sock.settimeout(timeout)
    try:
        sock.connect(sockaddr)
        elapsed = time.monotonic() - start
        return f"OK in {elapsed:.3f}s"
    except OSError as exc:
        elapsed = time.monotonic() - start
        return f"FAIL in {elapsed:.3f}s ({type(exc).__name__}: {exc})"
    finally:
        sock.close()


def probe(host: str, port: int, timeout: float = 3.0) -> None:
    rows = dump_addrinfo(host, port)
    print(f"\nconnect timeout={timeout}s per address")
    for i, (family, socktype, proto, _, sockaddr) in enumerate(rows):
        result = timed_connect(family, socktype, proto, sockaddr, timeout)
        print(f"  [{i}] {sockaddr} -> {result}")


if __name__ == "__main__":
    import sys

    host = sys.argv[1] if len(sys.argv) > 1 else "localhost"
    port = int(sys.argv[2]) if len(sys.argv) > 2 else 8080
    probe(host, port)

Commands I actually run next to the script

Run the server, then run the probe against the hostname and against the literal address. The notes should look like rows in a terminal, not like a monitoring screenshot from some other layer.

python3 ipv4_only_server.py
python3 addrinfo_field_notes.py localhost 8080
python3 addrinfo_field_notes.py 127.0.0.1 8080
getent ahostsv4 localhost || true
getent ahostsv6 localhost || true
python3 -c "import socket; print('dualstack', socket.has_dualstack_ipv6())"

On a dual-stack laptop, localhost often yields ::1 first and 127.0.0.1 second, in that order. The IPv4-only server accepts the second row and never sees the first one. The first row is Connection refused, or it is a hang if something drops IPv6 silently instead of sending RST.

Want the hang version without touching a shared firewall policy you do not own? Probe a documentation-prefix address that nothing legitimate should answer on your machine.

python3 addrinfo_field_notes.py 2001:db8::1 443

2001:db8::/32 is reserved for documentation, so it should not be a production dependency in anyone's config. The timed connect() shows the full timeout instead of an instant reset, which is the shape of an IPv6 black hole. Sequential clients wait through that timeout. Happy Eyeballs clients race the families and move on.

What actually broke

Three separate mistakes stacked, which is why the forty-eight hours felt cursed and personal. I treated a hostname as if it were already a single socket address. I copied infos[0] from a snippet that assumed IPv4-only networks forever. I compared a failing hostname client with a succeeding literal-IPv4 client and called that a fair test.

socket.create_connection((host, port)) already walks the list, and I had bypassed it for a "simple" socket tutorial. asyncio.open_connection can take happy_eyeballs_delay when you opt into that race. The stdlib socket.create_connection still tries addresses one after another in order. If the first address black-holes, you sit there for the whole timeout before IPv4 gets a chance at all.

Does your HTTP library document which of those behaviors you actually paid for? I also checked /etc/hosts, because glibc search order is part of the plot and not a footnote.

grep -n localhost /etc/hosts
# common layout on Linux, not a promise:
# 127.0.0.1 localhost
# ::1       localhost ip6-localhost ip6-loopback

NSS plugins, getaddrinfo, and the AI_ADDRCONFIG flag can hide families when an interface is down or a VPN stole the default route. Docker Desktop, broken ip6tables policies, and a laptop that "mostly" has IPv6 will change the same dump. The script does not care about the story you prefer. It prints rows, then it times connect().

Hours 24–48: what I would repeat

If I am doing this again, I will not start with vendor dashboards or a timeout lottery. I will start with a notes file, three host strings, and the probe script above. That sounds slower than flipping a sysctl. It is not slower than two days of retries against ::1.

Hour-one checklist

1. Write the hostname, the literal IPv4, and the literal IPv6 as three separate probes.
2. Print getaddrinfo rows before changing timeouts, retries, pool sizes, or library versions.
3. Time connect() per sockaddr with a short timeout, not a five-minute confession.
4. Compare curl -4, curl -6, and default curl against the same host and port.
5. Only then ask whether a proxy, TLS handshake, or HTTP library is involved at all.

curl -4 --connect-timeout 3 -sv http://localhost:8080/ -o /dev/null
curl -6 --connect-timeout 3 -sv http://localhost:8080/ -o /dev/null
curl --connect-timeout 3 -sv http://localhost:8080/ -o /dev/null

Would I disable IPv6 on the machine to "fix" a Python client I control? No, because that hides the bug for one laptop and ships it to the next dual-stack host. Prefer walking every address, racing them on user-facing paths, or binding the server to :: when IPv6 is supposed to work. Cargo-cult sysctls are how this bug becomes folklore.

A walker that is still not Happy Eyeballs

A patched client looks boring, which is the entire point of the notes:

# walking_client.py
import socket


def fetch(host: str, port: int, timeout: float = 3.0) -> None:
    last_error: Exception | None = None
    for family, socktype, proto, _, sockaddr in socket.getaddrinfo(
        host, port, type=socket.SOCK_STREAM
    ):
        sock = socket.socket(family, socktype, proto)
        sock.settimeout(timeout)
        try:
            sock.connect(sockaddr)
            sock.sendall(b"GET / HTTP/1.0\r\nHost: lab\r\n\r\n")
            print("connected via", sockaddr, sock.recv(256))
            return
        except OSError as exc:
            last_error = exc
            print("skip", sockaddr, "->", exc)
        finally:
            sock.close()
    raise OSError(f"all addresses failed for {host}:{port}: {last_error}")

That is still sequential, and I am saying that out loud so nobody copies it as magic. It is enough for Connection refused on ::1 when 127.0.0.1 is healthy. It is not enough for a silent IPv6 drop unless the per-address timeout stays small and boring. For a black hole you want a race, asyncio happy eyeballs, or an HTTP library that already implemented RFC 8305. I am not going to pretend my ten-line walker is that protocol.

A spare environment, used once

I wanted a second machine whose /etc/hosts and IPv6 state I had not already contaminated with earlier experiments. A local laptop lies after you have been toggling sysctls, Docker settings, and VPN clients for hours. Did I need a GPU for that second opinion? No. I needed a clean python3, a shell, and permission to run the probe against loopback.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I used MonkeyCode there as a free-server scratch space with free model access, only to review the dump and rewrite the walker. The model did not discover infos[0] by magic, and I would not claim it did. I pasted the probe output, asked why localhost failed when 127.0.0.1 worked, and kept the script that runs without the product. If you already have a throwaway VM, skip that step and keep the artifact.

Who should not copy this approach

This lab is a connect() microscope. It is not a personality test for your network team, and it is not a reason to disable IPv6 on a shared host.

• Do not cargo-cult sysctl -w net.ipv6.conf.all.disable_ipv6=1 on machines other people use.
• Do not apply a thirty-second per-address timeout and call sequential walking good enough for users.
• Do not trust this lab if your real bug is TLS, HTTP/2, DNS search domains, or HTTP_PROXY.
• Do not paste production hostnames into a shared scratch server you do not control.

The probe lies when getaddrinfo is intercepted by a sidecar, a corporate DNS filter, or AI_ADDRCONFIG on a host with no global IPv6 route. It also lies if you probe a different name than the library uses after a redirect. Would you still trust one dump after a 301 to another hostname? I would not.

Limitations I am not going to paper over

These notes do not measure HTTP performance, TLS cost, or application throughput under load. They measure whether connect() ever ran, against which sockaddr, and how long that attempt sat there. They do not replace tcpdump. They do not tell you whether urllib3, httpx, or your RPC stub already races address families for you.

I did not collect production timings for this write-up, and I will not invent any to make the log look more serious. socket.create_connection behavior, libc NSS plugins, and Docker IPv6 defaults are all moving targets across distros. Re-run the dump on the OS you actually ship, including Alpine musl versus glibc, because the rows can disagree.

What would I repeat tomorrow without waiting for another wasted evening? The probe, the three-way curl, and the refusal to raise a timeout before I print sockaddr tuples. What would I not repeat, even if ss looks green? Blaming the firewall because a listener existed on a different address family than the client used.

If this log saves you the hours I wasted on infos[0], copy the script into your notes file and throw the rest of my opinions away.

A free server option is enough to reproduce the setup.