48-Hour Field Notes: HTTPS Passed on My Laptop. The Clean Runtime Rejected the Chain.

I spent the better part of forty-eight hours convinced a staging HTTPS API was randomly rejecting my client. The laptop made every request look healthy, and that is usually how these stories start, right? A clean runtime told a different story almost immediately, which still feels slightly unfair. This is the field note I wish I had written on hour three instead of hour forty-seven.

Hour 0–8: the symptom that looked like “the API is flaky”

The client was a small Python script using urllib.request, not a framework and not a service mesh. Locally it printed status 200 and a short JSON body, so I assumed everyone else had fat-fingered the URL. Have you ever watched a TLS complaint turn into a referendum on retries, timeouts, and the vendor status page? I ran the comforting local checks first, because that is what muscle memory does under pressure.

What I actually ran, in order:

1. Hit the same URL with curl -v and watched the handshake complete without protest.
2. Printed the interpreter’s OpenSSL string and shrugged at a perfectly boring version line.
3. Wrapped the call in retries, because retries are the comfort food of networked scripts.
4. Blamed DNS anyway, then confirmed the hostname resolved to the address I expected.
5. Added a louder User-Agent, which is not debugging, just superstition with headers.

None of that was wasted, exactly. It also did not explain why a teammate’s fresh VM failed with ssl.SSLCertVerificationError while my machine stayed green. When the only passing runtime is the one you live inside, you are not testing the API. You are testing your laptop’s autobiography.

Hour 8–24: I asked a model, then almost shipped verify=False

I wanted a second pair of eyes that was not another tab of half-remembered TLS folklore. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used MonkeyCode’s free model access to shrink the traceback into a reproduction, then I ran that reproduction on MonkeyCode’s free server option so I was not debugging inside my own snow globe. The point was not “ask a chatbot to invent networking.” The point was to get a draft probe, then execute it somewhere that did not inherit my Keychain, my Homebrew certs, or my forgotten extra CA.

The model did something useful and something dangerous in the same breath. It tightened the reproduction into a short stdlib script, which I genuinely needed. It also suggested disabling certificate verification “just to confirm the API is reachable,” which is the kind of suggestion that survives review when everyone is tired. Would you have caught that if the only green box you had was your own machine? I did not ship verify=False. I almost pasted it into a branch named fix-tls-flakiness, which is not a name I am proud of.

The artifact: a trust-store probe you can run twice

Do not debate TLS in the abstract. Print the trust material each runtime can see, then keep verification on. The script below is the probe I wish I had run first; treat it as a checklist with teeth, not as production client code. Run the same file on the laptop and on a clean server. Compare the JSON, and refuse to “fix” a delta by turning verification off.

#!/usr/bin/env python3
"""ssl_trust_probe.py — show what THIS runtime trusts, then try one GET.

Usage:
  python ssl_trust_probe.py https://staging.example.invalid/health

Do not pass secrets. Do not disable verification.
"""
from __future__ import annotations

import json
import os
import ssl
import sys
import urllib.error
import urllib.request
from datetime import datetime, timezone

WATCHED_ENV = (
    "SSL_CERT_FILE",
    "SSL_CERT_DIR",
    "REQUESTS_CA_BUNDLE",
    "CURL_CA_BUNDLE",
    "HTTPS_PROXY",
    "HTTP_PROXY",
    "NO_PROXY",
)


def snapshot() -> dict:
    paths = ssl.get_default_verify_paths()
    return {
        "utc": datetime.now(timezone.utc).isoformat(),
        "executable": sys.executable,
        "platform": sys.platform,
        "openssl": ssl.OPENSSL_VERSION,
        "env": {k: os.environ.get(k) for k in WATCHED_ENV},
        "verify_paths": {
            "cafile": paths.cafile,
            "capath": paths.capath,
            "openssl_cafile_env": paths.openssl_cafile_env,
            "openssl_cafile": paths.openssl_cafile,
            "openssl_capath_env": paths.openssl_capath_env,
            "openssl_capath": paths.openssl_capath,
        },
    }


def probe(url: str) -> dict:
    ctx = ssl.create_default_context()
    # Keep verify_mode at the default. The whole point is to fail closed.
    req = urllib.request.Request(url, method="GET")
    try:
        with urllib.request.urlopen(req, context=ctx, timeout=15) as resp:
            return {
                "ok": True,
                "status": resp.status,
                "tls_version": resp.fp.raw._sslobj.version()  # type: ignore[attr-defined]
                if hasattr(resp.fp, "raw") else None,
            }
    except urllib.error.URLError as exc:
        reason = exc.reason
        return {
            "ok": False,
            "error_type": type(reason).__name__ if reason else type(exc).__name__,
            "error": str(reason or exc),
        }


def main() -> int:
    if len(sys.argv) != 2:
        print("usage: ssl_trust_probe.py URL", file=sys.stderr)
        return 2
    url = sys.argv[1]
    report = snapshot()
    report["url"] = url
    report["probe"] = probe(url)
    print(json.dumps(report, indent=2, default=str))
    return 0 if report["probe"].get("ok") else 1


if __name__ == "__main__":
    raise SystemExit(main())

Labeled as a field probe, not a load test: I did not collect latency percentiles, and I would not trust a single GET as capacity evidence. Save laptop output as local.json and clean-runtime output as clean.json. Then diff the verify_paths and env objects before you touch application code. If those objects disagree, your client is not flaky. Your trust roots are.

A tiny helper I keep next to the probe, because I will otherwise eyeball JSON and lie to myself:

python ssl_trust_probe.py "$URL" | tee local.json
# run the same command on the clean runtime, save clean.json
python - <<'PY'
import json
from pathlib import Path

def load(name):
    return json.loads(Path(name).read_text())

a, b = load("local.json"), load("clean.json")
for key in ("openssl", "verify_paths", "env"):
    if a.get(key) != b.get(key):
        print("DELTA", key)
        print(" local:", json.dumps(a.get(key), indent=2, default=str))
        print(" clean:", json.dumps(b.get(key), indent=2, default=str))
PY

Hour 24–40: what the two runtimes actually disagreed on

The laptop report was embarrassingly specific once I stopped staring at HTTP status codes. SSL_CERT_FILE was unset, which had made me feel virtuous. ssl.get_default_verify_paths() still pointed at a CA bundle that included a private staging root I had installed during an old mkcert experiment. curl was even less honest on macOS, because it can succeed through the system trust store while CPython is using OpenSSL’s files. Two green tools are not two independent tests when they do not share a CA pile. Are we still calling that “reproduced locally”?

The clean runtime had a boring public bundle and no private root. Same URL string, same SNI hostname, same Python stdlib call, different answer. That is the whole incident. The staging certificate was never “randomly invalid.” It was invalid for any process that had not been quietly enrolled into my laptop’s history. Retries could not enroll a CA. Timeouts could not enroll a CA. A louder User-Agent definitely could not enroll a CA.

I also caught a second lie I had told myself about requests. A notebook cell using requests.get can pass through certifi’s bundle while a job script using urllib.request uses ssl.create_default_context(). If you mix those clients during debugging, you are comparing two products and calling it one repro. The probe uses stdlib on purpose so the second runtime cannot “helpfully” swap bundles on you.

Decision table: local green, clean red (and the reverse)

Local probe	Clean probe	Do this next	Do not do this
ok: true, extra CA in cafile	CERTIFICATE_VERIFY_FAILED	Install the intended CA only on runtimes that should trust staging, or use a publicly trusted cert	verify=False, or copy your entire laptop trust store “to make it work”
ok: true, HTTPS_PROXY set	connection error or different cert	Debug the proxy / MITM path as its own system	Assume the origin API is down
ok: false on both	ok: false on both	Fix the server chain; openssl s_client -showcerts on a machine you control	Blame Python
ok: false locally, ok: true clean	public trust is fine	Search local SSL_CERT_*, corporate roots, and leftover REQUESTS_CA_BUNDLE	“Upgrade OpenSSL” as a ritual
Status 200, body looks truncated	same	that is an application bug; leave TLS alone	rotate certificates for luck

If you cannot name which row you are in, you are still in hour eight. Write the JSON files before you write the postmortem.

Hour 40–48: what broke, and what I would repeat

What broke was not the handshake library. What broke was my habit of treating the developer laptop as a neutral instrument. The model’s first instinct was reachability, so it offered the fastest way to get a 200. A clean server has no interest in my 200. It only has the bundle someone actually shipped. That social difference matters more than prompt wording. I would still ask a model to draft a probe. I would not ask it whether verification is optional.

What I would repeat, as a boring checklist:

• Capture the exact exception type, not a screenshot of “SSL error.”
• Run one stdlib probe on two runtimes before changing application retries.
• Diff verify_paths and the watched environment variables, not just the HTTP status.
• Keep private CA material out of chat logs, tickets, and anyone else’s server.
• If staging needs a private root, install that root with the service, not with folklore.

What I would not repeat: editing sitecustomize.py to force a CA path “temporarily.” Temporary CA hacks become the next person’s forty-eight hours. They also make the next clean runtime look broken when it is the only honest machine in the room.

Limitations, and who should not use this

This workflow assumes you are allowed to hit the URL from a runtime that is not your laptop. Internal-only APIs, air-gapped networks, and environments that require a client certificate will not become clearer on a generic free server; they will become unreachable. A clean public trust store is the right control for something that claims to be publicly trusted. It is the wrong control for a service whose whole contract is a company root.

Do not upload cookies, bearer tokens, .pem keys, or netrc files to a shared or free machine in order to “make the probe realistic.” The probe above needs a URL and a default context. If your bug only appears with production credentials, reproduce the TLS layer with a non-secret health endpoint, then handle auth on a runtime you actually own. I also did not benchmark model quality, server hardware, or session length here, because I do not have numbers I can defend. The useful claim is smaller: a second runtime that does not share your laptop’s CA history will puncture a fake green faster than another local retry loop.

If you need that second runtime and you want a model to draft the probe instead of the workaround, MonkeyCode’s free model access and free server option are what I used as the clean room for this comparison. Your trust store is still your problem. That part does not get outsourced.