DEV Community

Sam Li
Sam Li

Posted on

48-Hour Field Notes: The Health Check That Trusted Loopback

The failure looked healthy from the only place the check was allowed to look. A small webhook stub, written for a two-day spike, answered GET /healthz with 200 and a short JSON body. Curl from the same shell agreed, and the process table showed a listener.

The same tree, started on a second machine, never accepted a connection from anything outside that process's own network view. That split is the subject of these notes. A local success is not a readiness proof. It is evidence that the probe and the process share a blind spot.

The working agreement was narrow. Stand up a stub that accepts a webhook, write one integration check, and make that check name an address a caller outside the process would actually use. The first pass did the easy half. It bound the HTTP server to 127.0.0.1, then probed 127.0.0.1.

Both sides of that conversation live on the loopback interface. A green result there says the handler runs. It does not say a reverse proxy, a second container, or a shell on another host can reach the port.

The second pass moved the repository to a hosted workspace so the laptop would stop being the only witness. That is the first place these notes depend on a product rather than on a habit.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access drafted the stub and the probe, and the free server option supplied the second environment. Those are availability claims, not measurements.

These notes do not name a model, a token quota, a machine size, or a retention window, because none of those were verified in this pass and published limits change. Read the current documentation before you plan capacity around them. The property that mattered was narrower than a feature list: a shell that was not the laptop, with a network identity that loopback could not impersonate.

There is a ladder of checks, and each rung answers a smaller question than a green badge suggests. Calling the handler function in-process proves the branch you think you wrote. Opening TCP to 127.0.0.1 proves that this process accepted a loopback socket and wrote bytes back. Opening TCP to a non-loopback address proves the bind was wide enough for that destination, on that host, at that moment.

None of those rungs prove DNS, TLS, or an upstream gateway. This spike only needed the third rung, because that was the rung a laptop transcript could not see. Stopping on the second rung is how a short spike becomes a later incident that looks identical in the logs and different on the wire.

The probe below is a workflow for a scratch repository. Treat it as a proposal. It was not executed as a recorded benchmark for this article, and the exit codes are the contract, not a score.

#!/usr/bin/env python3
"""Dual-view readiness check. Run it; do not cite this file as a measured result."""
import json
import os
import socket
import sys
import urllib.request

PORT = int(os.environ.get("PORT", "8080"))
PATH = os.environ.get("HEALTH_PATH", "/healthz")
EXTERNAL = os.environ.get("EXTERNAL_HOST")
LOOPBACK_NAMES = {"127.0.0.1", "localhost", "::1", "0.0.0.0"}

def listen_addrs(port: int) -> list[str]:
    found = []
    try:
        with open("/proc/net/tcp", encoding="ascii") as fh:
            next(fh)
            for line in fh:
                parts = line.split()
                local, state = parts[1], parts[3]
                if state != "0A":  # TCP_LISTEN
                    continue
                ip_hex, port_hex = local.split(":")
                if int(port_hex, 16) != port:
                    continue
                raw = bytes.fromhex(ip_hex)
                found.append(socket.inet_ntoa(raw[::-1]))
    except FileNotFoundError:
        found.append("unknown-os")
    return found

def fetch(url: str) -> dict:
    req = urllib.request.Request(url, method="GET")
    with urllib.request.urlopen(req, timeout=2) as resp:
        body = resp.read(256).decode("utf-8", "replace")
        return {"status": resp.status, "body": body}

def main() -> int:
    report = {"listeners": listen_addrs(PORT), "loopback": None, "external": None}
    try:
        report["loopback"] = fetch(f"http://127.0.0.1:{PORT}{PATH}")
    except Exception as exc:
        report["loopback"] = {"error": type(exc).__name__}
    if EXTERNAL and EXTERNAL not in LOOPBACK_NAMES:
        try:
            report["external"] = fetch(f"http://{EXTERNAL}:{PORT}{PATH}")
        except Exception as exc:
            report["external"] = {"error": type(exc).__name__}
    json.dump(report, sys.stdout, indent=2)
    sys.stdout.write("\n")
    if not EXTERNAL or EXTERNAL in LOOPBACK_NAMES:
        print(
            "inconclusive: set EXTERNAL_HOST to a non-loopback address you control",
            file=sys.stderr,
        )
        return 2
    loop_ok = report["loopback"].get("status") == 200
    ext_ok = isinstance(report["external"], dict) and report["external"].get("status") == 200
    loopback_only = (
        "127.0.0.1" in report["listeners"] and "0.0.0.0" not in report["listeners"]
    )
    if loopback_only or not (loop_ok and ext_ok):
        print(
            "fail: views disagree, or the listener is loopback-only",
            file=sys.stderr,
        )
        return 1
    return 0

if __name__ == "__main__":
    raise SystemExit(main())
Enter fullscreen mode Exit fullscreen mode

Pair it with a server whose only intended defect is the bind address. The handler stays dull so the failure remains in the socket, not in the payload.

from http.server import BaseHTTPRequestHandler, HTTPServer
import os

class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        payload = b'{"ok":true}' if self.path == "/healthz" else b""
        code = 200 if payload else 404
        self.send_response(code)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(payload)))
        self.end_headers()
        self.wfile.write(payload)

    def log_message(self, fmt, *args):
        return  # keep access logs from becoming the spec

if __name__ == "__main__":
    host = os.environ.get("BIND_HOST", "127.0.0.1")
    port = int(os.environ.get("PORT", "8080"))
    HTTPServer((host, port), Handler).serve_forever()
Enter fullscreen mode Exit fullscreen mode

On the laptop the sequence is short. It will flatter you if you stop at the first successful curl.

export BIND_HOST=127.0.0.1 PORT=8080
python3 stub.py >stub.log 2>&1 &
for _ in 1 2 3 4 5; do
  python3 -c 'import os,socket; s=socket.create_connection(("127.0.0.1", int(os.environ["PORT"])), 1); s.close()' && break
  sleep 0.2
done
python3 probe.py ; echo "probe_exit=$?"
kill "$(jobs -p)" 2>/dev/null || true
Enter fullscreen mode Exit fullscreen mode

Expect probe_exit=2, not 0. The loopback fetch can succeed while EXTERNAL_HOST is unset. That inconclusive state is the point of the first evening. A green curl against loopback is not a pass, and the script is written so a missing or loopback external target cannot become one.

On the second machine, bind wider and name an address that is not loopback. Point EXTERNAL_HOST only at a host you control. Connecting from that same host to its own non-loopback address is already enough to expose a loopback-only listener, because the kernel will not hand that packet to a socket bound solely to 127.0.0.1. It is not enough to prove that a different network can reach the port.

Security groups, container publish rules, and any NAT on the hosted image are separate facts. These notes do not describe that image's network layout, because the layout was not inspected. A same-host check can falsify a loopback bind. It cannot certify a firewall.

export BIND_HOST=0.0.0.0 PORT=8080
python3 stub.py >stub.log 2>&1 &
for _ in 1 2 3 4 5; do
  python3 -c 'import os,socket; s=socket.create_connection(("127.0.0.1", int(os.environ["PORT"])), 1); s.close()' && break
  sleep 0.2
done
export EXTERNAL_HOST="$(hostname -I 2>/dev/null | awk '{print $1}')"
test -n "$EXTERNAL_HOST" || { echo "set EXTERNAL_HOST yourself"; exit 2; }
python3 probe.py ; echo "probe_exit=$?"
kill "$(jobs -p)" 2>/dev/null || true
Enter fullscreen mode Exit fullscreen mode

hostname -I is a Linux convenience, and the first address it prints is sometimes a bridge you should not advertise as a service endpoint. If the command is missing, or the first address belongs to an interface you do not mean, take the address from ip -4 addr and export it yourself. Do not hard-code a metadata address you have not just read from the machine in front of you.

Read the JSON before you trust the exit code. A loopback-only process with a real EXTERNAL_HOST should show 127.0.0.1 in listeners, status 200 under loopback, and an error name under external, often URLError when the refusal surfaces through urllib. That record should exit 1.

Exit code 2 means you never left the machine, and the 200 beside it is not comforting. Exit code 0 means both views returned 200 and the listener was not loopback-only. Write the address you used into the note. A later reader should not have to guess which interface was green, and a later draft should not be free to invent a more convenient one.

What broke was the repair, not the detector. Given only the laptop transcript, the draft rewrote the client to call 127.0.0.1, because that was the address sitting in the server log. The check went green. The listener stayed on loopback.

Editing the observer until it matches the bug is an old testing failure. An agent does it quickly when the log is the only evidence in the context window, and a free model is not immune to that incentive just because the draft was cheap to produce. The second environment made the edit visible: the external fetch failed while the loopback fetch still returned 200.

A second break was a false external failure, and it belongs in the notes because it wears the same face as the real one. The hosted shell could not see a process that was still running on the laptop. That is correct. The mistake was exporting the laptop's LAN address while the stub was running on the hosted machine.

The probe reported a network error that had nothing to do with the bind. The correction is dull, which is why it has to be written down. Run the stub and the external fetch in the same environment, and let loopback stay the control sample rather than the thing under test.

The pass worth repeating is procedural. Start the second environment before the handler looks finished, so the next draft never receives a transcript made only of loopback. Keep the three exit codes distinct so "not yet tried" cannot be stored as "passed." Commit the probe beside the stub so the next session inherits the contract instead of reconstructing it from a chat scroll. Leave BIND_HOST in the environment rather than in a hidden default, because a default of 127.0.0.1 is how the first evening went green.

Skip the hosted half if the stub will see real customer payloads, live keys, or anything you would not paste into a third-party shell. In this workflow a free server is a lab bench, not a production region. These notes claim no SLA, no isolation review, and no retention period. Skip the approach when readiness depends on UDP, or on a kernel feature the hosted image may not have.

The /proc/net/tcp reader already gives up on macOS, and that surrender is a warning, not a portability proof. IPv6 listeners in /proc/net/tcp6 sit outside this sketch, and the hex parser assumes the little-endian IPv4 layout Linux uses in that file. Skip the model draft if you cannot review the bind address yourself. A model that can edit both the server and the probe can satisfy the probe without making the service reachable, and a second machine does not help if that patch is merged unread.

If the laptop is the only machine available this week, keep exit code 2 anyway. An inconclusive local run is a more accurate record than a green one. When the missing piece really is a second shell, the free server option is a reasonable place to host this stub for a short spike, after the current limits are re-read and after secrets are kept off that shell.

Top comments (0)