I Chased a Hung Fetcher for 48 Hours. getaddrinfo Waited on AAAA.

Have you ever watched a tiny HTTP client sit there doing nothing, while the same file on your laptop returned 200 in a blink? I just spent forty-eight hours in that gap, and the source tree never changed between the two machines. Was it TLS, DNS TTL, or a User-Agent the model invented on my behalf? None of those guesses survived; the clean server tried IPv6 first, and connect() waited on an address that would never answer.

This write-up is a field notebook from that chase, not a claim that I outsmarted the network stack. I wanted a boring downloader for a public health endpoint, so I asked a free coding model for a first draft. Then I moved the file onto a free remote server so I would not keep testing against my laptop's dual-stack lie. What follows is what I tried, what broke in my face, and the small artifact I will keep using.

Why the laptop was a dishonest test harness

My workstation speaks IPv6 for real, and the browser hides failures with Happy Eyeballs so a broken AAAA rarely feels like a hang. Python's stdlib is less kind to that fantasy, because socket.create_connection() still walks getaddrinfo() in list order. urllib.request sits on that sequential loop, so the first tuple can own your entire timeout window. If that tuple is IPv6 and the route is a black hole, the process looks busy while it is only waiting.

Does your daily laptop have a working 2001: path to the public internet? Mine did, which made every local run a liar. The clean box had an IPv6 address on the interface, a default route that looked plausible in ip -6 route, and no actual path to the destination. That combination is worse than having no IPv6 stack at all, because the kernel will not fail fast with ENETUNREACH.

Hour 0–8: I trusted the generated client

I asked for a small Python 3 script that would resolve a host, GET a URL, print the status, and exit non-zero on failure. The draft looked like every snippet you have already pasted into a terminal at midnight. I ran it locally, saw 200, and treated that blink as evidence.

# generated_fetch.py — first draft I actually ran
from urllib.request import urlopen, Request
import sys

url = sys.argv[1]
req = Request(url, headers={"User-Agent": "field-notes/0.1"})
with urlopen(req, timeout=30) as resp:
    print(resp.status)
    print(resp.read(200))

On the laptop, python generated_fetch.py https://example.com printed 200 immediately and I almost committed that feeling. Have you noticed how often a single successful run becomes the entire test plan for generated networking code? I copied the same file to a clean remote interpreter with the same Python major, the same URL, and the same thirty-second timeout. It sat there until the timer expired, then raised URLError: timed out.

Retrying did not change the shape of the failure at all. It just burned another half minute of wall clock each time I pretended the internet had recovered.

Hour 8–16: the wrong suspects

I did what I always do when I want the bug to live in someone else's service. I rotated through comfortable villains instead of printing the address list.

1. I blamed DNS cache, even though dig A and dig AAAA both returned records and the remote box had nothing to flush.
2. I blamed TLS, until openssl s_client -connect example.com:443 -brief completed from the server when I forced IPv4.
3. I blamed the generated User-Agent header, deleted it, and watched the hang remain identical.
4. I blamed timeout math, raised timeout=60, and only succeeded at making the hang twice as long.

Commands I actually kept in the scrollback, because they later became the real test plan:

python -c "import socket; print(socket.getaddrinfo('example.com', 443, type=socket.SOCK_STREAM))"
curl -4 -I --max-time 8 https://example.com
curl -6 -I --max-time 8 https://example.com
ip -6 route
getent ahosts example.com
ss -tlnp | head

curl -4 was fast enough to feel boring. curl -6 waited until --max-time and then died without a useful HTTP status. getaddrinfo listed the AAAA tuple first on the clean box. That is the whole mystery once you stop arguing with the HTTP layer and look at the sockaddr.

Hour 16–30: strace, then the address list

strace -e trace=network -f python generated_fetch.py https://example.com showed connect() to a sin6_addr and then silence for the rest of the timeout. No EHOSTUNREACH. No quick RST from a neighbor. Just a black hole, which is why a thirty-second timeout feels like an outage instead of a bad first hop.

So the generated client was not wrong in the unit-test sense you would write after the fact. It used the stdlib the way the docs describe, and the environment was dual-stack in name only. Would a local pytest that monkeypatches urlopen have caught this? Not on my laptop, because the mock would never have seen the AAAA black hole.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used MonkeyCode's free model access to draft the first fetcher, then its free server option as the clean box that did not inherit my laptop's IPv6 path. I am not claiming a model name, a quota, a hardware profile, or any benchmark here, because those details are not what failed. The useful split is generate on one side, then reproduce on a machine that does not share your home route table.

The artifact: print the tuples, then race the connect

I want a helper that fails loudly with the address list, plus a connect path that does not worship the first tuple. This is a field tool, not a replacement for httpx or aiohttp, and it should stay small enough to paste into a notes file.

# addrcheck.py — reproducible field helper
from __future__ import annotations

import socket
import sys
import time
from concurrent.futures import ThreadPoolExecutor, as_completed
from typing import List, Tuple

Target = Tuple[socket.AddressFamily, tuple]


def list_targets(host: str, port: int) -> List[Target]:
    infos = socket.getaddrinfo(host, port, type=socket.SOCK_STREAM)
    targets: List[Target] = []
    seen = set()
    for family, socktype, proto, _canon, sockaddr in infos:
        key = (family, sockaddr)
        if key in seen:
            continue
        seen.add(key)
        targets.append((family, sockaddr))
    return targets


def try_connect(family: socket.AddressFamily, sockaddr, timeout: float) -> float:
    sock = socket.socket(family, socket.SOCK_STREAM)
    sock.settimeout(timeout)
    start = time.monotonic()
    try:
        sock.connect(sockaddr)
        return time.monotonic() - start
    finally:
        sock.close()


def happy_enough(host: str, port: int, per_attempt: float = 0.25) -> Target:
    targets = list_targets(host, port)
    if not targets:
        raise OSError(f"no addresses for {host}:{port}")

    print("getaddrinfo order:")
    for family, sockaddr in targets:
        print(f"  {family.name:6} {sockaddr}")

    with ThreadPoolExecutor(max_workers=len(targets)) as pool:
        futs = {
            pool.submit(try_connect, family, sockaddr, per_attempt): (family, sockaddr)
            for family, sockaddr in targets
        }
        for fut in as_completed(futs):
            family, sockaddr = futs[fut]
            try:
                dt = fut.result()
            except OSError as exc:
                print(f"fail  {family.name:6} {sockaddr} ({exc})")
                continue
            print(f"ok    {family.name:6} {sockaddr} in {dt:.3f}s")
            return family, sockaddr

    raise TimeoutError(f"no address answered within {per_attempt}s each")


if __name__ == "__main__":
    host = sys.argv[1]
    port = int(sys.argv[2]) if len(sys.argv) > 2 else 443
    print(happy_enough(host, port))

Run it next to the original client so the two stories share one timestamp:

python -u addrcheck.py example.com 443
python -u generated_fetch.py https://example.com

If addrcheck.py prints a fast IPv4 ok and a failed IPv6 line, you have the bug in writing. If both families fail, stop blaming address order and go back to firewalls, SNI, or the name itself.

A tiny test plan I will actually rerun

I do not keep a public IPv6 black hole inside CI, so I treat this as a manual matrix plus one local negative sketch. The remote interpreter remains the honest fixture for the hang, because only that route table lied in the same way as the outage.

1. On the laptop, run addrcheck.py and record which families answer, including the printed order.
2. On the clean server, run the same command against the same host and save that order beside the laptop output.
3. Keep a curl pair (-4 / -6) in the same notes so Python libraries cannot monopolize the story.
4. Only after the tuple list is visible, decide whether the HTTP client needs a race, an explicit family, or no change at all.

Local negative sketch, labeled as a lab setup rather than production, because it does not simulate a black hole:

# lab_only: listen on ::1 and refuse IPv4 for a dummy port
import socket

s = socket.socket(socket.AF_INET6, socket.SOCK_STREAM)
s.setsockopt(socket.IPPROTO_IPV6, socket.IPV6_V6ONLY, 1)
s.bind(("::1", 9477))
s.listen(1)
print("listening on [::1]:9477")
input("hold the process open")

That snippet only proves "family present and bindable on loopback," which is a different bug class. For the black-hole case, you still need a machine whose IPv6 default route looks real and then goes nowhere.

Decision table I wish I had at hour two

Observation	Likely layer	Do not do this	Do this instead
Laptop 200, server timeout, curl -6 also hangs	Address family / route	Raise the HTTP timeout and hope	Print getaddrinfo, then test -4 and -6
Both curl families fail	Upstream, firewall, DNS	Rewrite the Python client first	Check resolver config, security groups, SNI
IPv6 connect fails fast with ENETUNREACH	Stack correctly has no route	Force AF_INET through every library	Let getaddrinfo skip; you are already fine
IPv6 connect hangs until the full timeout	Broken IPv6, not absent IPv6	Set timeout=None	Cap each address, or use a Happy Eyeballs race
Only asyncio code feels snappy	Stdlib sequential connect	Assume "Python is just slow"	Remember happy_eyeballs_delay on asyncio

Yes, asyncio already grew a delay parameter on open_connection, and that is easy to miss when a prompt asked for a small script. The generated file used urllib, so it never received that gift. Why do assistants default to the sync stdlib in this shape? Because "tiny script" still maps to urlopen in most training data, and tiny scripts still get copied onto servers.

What broke while I was being clever

The first fix I accepted from the model was timeout=None, which turned a thirty-second hang into an immortal process that I had to kill by hand. That change felt confident in the chat transcript and was actively worse on the clean box. If a helper cannot fail, you cannot debug it either.

The second fix was socket.AF_INET hard-coded through every call site I could find with grep. That will strand you on an IPv6-only network, which is a real deployment shape and not a conference thought experiment. Pinning v4 is a field workaround for one broken route, not an architecture you should ship.

The third break was believing a free remote interpreter equals production, which it does not, even when it saved me from the laptop lie. It only equals "not my workstation." Routing, /etc/gai.conf, Docker --sysctl net.ipv6.conf.all.disable_ipv6=1, and the next VPC can still diverge from the box your users actually hit.

Buffered logging also wasted a quiet hour I will not get back. Without PYTHONUNBUFFERED=1 or python -u, debug prints sat in a pipe until the timeout killed the process. Then it looked like the script printed nothing, which is a different ghost with the same empty terminal. I already knew that lesson and still skipped -u. Why do we do that under fatigue?

What I would repeat in the next forty-eight hours

• Reproduce generated networking code on a machine that does not share the laptop route table, before I argue with certificates.
• Print getaddrinfo() order as the first line of any fetch helper, not as a last-ditch debug after HTTP blame.
• Pair curl -4 and curl -6 with the Python run so library folklore cannot own the incident notes.
• Reject timeout=None from any assistant output the way I already reject chmod 777 in a paste.
• Keep the connect race small and visible, because I do not need a full RFC 8305 implementation to stop lying to myself.

Would I still ask a free model for the first draft after this mess? Yes, because the draft was a fine sketch and I did not want to hand-write Request boilerplate again. I would not ask that sketch to certify itself on a clean server, because that is a runtime question, and runtime questions need a runtime.

If you want the same split I used for these notes — a model for the sketch, plus a clean interpreter that is not your laptop — MonkeyCode's free server option is how I ran the remote half.

Limitations, and who should not copy this

This helper is not a production HTTP client, and I will not pretend the thread race is ready for a payment path. It does not honor Retry-After, cookies, proxies, redirects that change host, or HTTP/2. Short-lived sockets can leak under load, and winning a 250ms race does not make your financial-grade client dual-stack correct.

Skip this approach if you already have a battle-tested library with Happy Eyeballs, or if your environment is IPv4-only and you truly control that invariant. Skip it if you cannot run commands on the remote box, because guessing from laptop curl is how I lost the first eight hours. Skip it if you need a vendor SLA, a capacity plan, or a compliance boundary from the remote interpreter.

A free remote box is an honesty check against your workstation. It is not proof that production IPv6 is healthy, and it is not a substitute for staging that shares the real route table. The generated file was never the villain in this incident. The villain was a test harness that shared my working IPv6 path, and once the clean server showed the AAAA tuple first, the hang stopped being mystical and became connect() waiting on an address that would never pick up.