48-Hour Field Notes: curl Looked Instant. Python Paid a Handshake on Every Call.

I spent forty-eight hours arguing with an API that was not actually slow on the wire. My Python probe kept printing extra delay, while a single curl call still looked almost instant. Have you ever trusted a client-side timer and then shipped a performance scare that never existed upstream?

The disagreement showed up during a quiet check, not during an incident bridge or a customer ticket. I had a tiny loop hitting the same health URL and printing elapsed time after every response. curl made the endpoint look cheap, and Python made the same URL look heavy once iteration started. Why would two otherwise honest clients disagree so loudly about one boring health endpoint?

What I thought was broken

I started where most of us start, which is blaming the network I do not control. DNS felt like a reasonable villain, and a distant origin felt even better. TLS inspection on a laptop also felt plausible, because corporate machines lie in creative ways.

Here is the curl line I kept rerunning, as if repetition would finally make the two clients agree:

curl -s -o /dev/null -w "namelookup=%{time_namelookup} connect=%{time_connect} appconnect=%{time_appconnect} total=%{time_total}\n" https://127.0.0.1:8443/health

One process stayed cheap, and that made Python look even more guilty by comparison. Was urllib retrying in silence? Was a proxy header rewriting keep-alive? Was the laptop hopping between networks in the middle of my loop?

I wrapped the Python side in time.perf_counter() instead of time.time(), because wall clocks are a separate class of lie. The timer was honest. The experiment was not. Can a probe fail while every status code is still 200? Yes, and this one did.

What I tried during the first night

I treated a measurement bug like a production mystery, and that was the expensive move. The checklist felt responsible, which is how it hid the real mistake for hours.

1. I enabled debug logging for redirects, retries, and status codes, then stared at a calm stream of 200s.
2. I pinned the host to an IP address, hoping DNS would confess, and the gap remained.
3. I cleared HTTP_PROXY and HTTPS_PROXY in the shell, because proxy env vars have ruined evenings before.
4. I compared plain HTTP against TLS on a local stub, and HTTPS made the gap much louder.

None of that knowledge was useless later, but none of it was the bug either. The Python loop looked too boring to be guilty, which is usually a warning:

# example probe — labeled because this is the broken measurement, not a recipe
import ssl
import time
import urllib.request

url = "https://127.0.0.1:8443/health"
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE  # throwaway loopback cert only

for i in range(50):
    t0 = time.perf_counter()
    with urllib.request.urlopen(url, context=ctx, timeout=5) as resp:
        resp.read()
    print(f"call={i} seconds={time.perf_counter() - t0:.4f}")

Every iteration opened a fresh TCP connection and a fresh TLS session, then tore both down. curl printed one handshake because I ran one process. Python printed fifty handshakes because I asked it to. I was not timing the handler. I was timing socket setup and calling it origin latency.

What actually broke

The handler was a local TLS wrapper around http.server, sitting on loopback, doing almost no application work. Connection reuse was the entire plot, hiding in the client lifecycle. urllib.request.urlopen(), and requests.get() without a Session, will pay for a new handshake on every call unless something else keeps the socket warm.

Is that a Python language problem? Not really. It is a client lifetime problem that measurement scripts conceal in plain sight. A reused http.client.HTTPSConnection told a different story after the first call, because TCP and TLS dropped out of the hot path. That does not mean the API became faster overnight. It means I stopped billing handshake cost to the handler.

requests.Session is the same fork in the road, just with a friendlier name. One-shot helpers look clean in a gist. They also recreate transport state that production servers try hard to avoid.

The reproducible harness

Do not point the first draft at a public URL. Stand up a local TLS server, generate a throwaway certificate, and compare cold sockets against a reused connection. Absolute milliseconds will differ on every machine, and they should. Watch the ratio between cold calls and reused calls, not a trophy number you cannot reproduce tomorrow.

1. Throwaway certificate

openssl req -x509 -newkey rsa:2048 -keyout /tmp/loopback-key.pem \
  -out /tmp/loopback-cert.pem -days 1 -nodes -subj "/CN=localhost"

That certificate is for loopback experiments only. Do not ship it, and do not disable verification against a real hostname on the public internet.

2. Tiny TLS server

# local_tls_server.py — example harness, not a production server
import http.server
import ssl

class Handler(http.server.BaseHTTPRequestHandler):
    def do_GET(self):
        body = b'{"ok": true}\n'
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.send_header("Connection", "keep-alive")
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, format, *args):
        return

server = http.server.HTTPServer(("127.0.0.1", 8443), Handler)
ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
ctx.load_cert_chain("/tmp/loopback-cert.pem", "/tmp/loopback-key.pem")
server.socket = ctx.wrap_socket(server.socket, server_side=True)
print("listening on https://127.0.0.1:8443/health")
server.serve_forever()

3. Cold path versus reused path

# measure_handshake.py — example harness
import http.client
import ssl
import time

def make_context():
    ctx = ssl.create_default_context()
    ctx.check_hostname = False
    ctx.verify_mode = ssl.CERT_NONE
    return ctx

def cold_calls(n=30):
    times = []
    ctx = make_context()
    for _ in range(n):
        t0 = time.perf_counter()
        conn = http.client.HTTPSConnection("127.0.0.1", 8443, context=ctx, timeout=5)
        conn.request("GET", "/health")
        resp = conn.getresponse()
        resp.read()
        conn.close()
        times.append(time.perf_counter() - t0)
    return times

def reused_calls(n=30):
    times = []
    ctx = make_context()
    conn = http.client.HTTPSConnection("127.0.0.1", 8443, context=ctx, timeout=5)
    for _ in range(n):
        t0 = time.perf_counter()
        conn.request("GET", "/health")
        resp = conn.getresponse()
        resp.read()
        times.append(time.perf_counter() - t0)
    conn.close()
    return times

def summarize(label, times):
    rest = sorted(times[1:]) or times
    p50 = rest[len(rest) // 2]
    print(
        f"{label} n={len(times)} first={times[0]:.4f}s "
        f"p50_after_first={p50:.4f}s min={min(times):.4f}s"
    )

if __name__ == "__main__":
    summarize("cold_new_socket", cold_calls())
    summarize("reused_connection", reused_calls())

Run the server in one terminal and the probe in another, then read those two summary lines together. If reused p50_after_first is not clearly cheaper than the cold path on loopback TLS, the harness is still paying setup cost. Are you reconstructing HTTPSConnection inside the loop? Is the server sending Connection: close? Did a proxy eat keep-alive and quietly restore the cold path?

Print the first call separately on purpose. The first reused call still contains the handshake, and blending it into a single average recreates the original lie. I wanted one number. The honest version is two named experiments.

Decision table I wish I had on hour one

What you are trying to learn	Reuse the connection?	Why it matters
Handler time after a session already exists	Yes	Setup cost is not the handler
TLS config, cipher, or certificate changes	No	The handshake is the subject
Load balancer spreading across backends	No	Stickiness hides imbalance
Default behavior of a client library	Both	Compare one-shot helpers against a session
Public SLO or multi-region latency	Neither, from loopback	Local TLS cannot answer that question

If you need handler time, reuse the client and discard the first call on purpose. If you need user-facing first-byte time, keep the cold path and stop calling the result an API regression. Those are different questions, and one blended average should not pretend to answer both.

The noisy laptop in the middle of day two

My laptop fans were part of the noise, which I did not want to admit while hunting origin latency. Thermal throttling makes a handshake study look like a flaky service, especially when the probe is single-threaded and short. I needed a second place to run the identical file, not a new theory about the upstream.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I used MonkeyCode's free model access to review the harness for accidental close() calls, and the free server option to rerun the same script away from a throttling laptop. That did not invent a profiler, and it did not prove the product owns the measurement. It gave me a quieter shell for the same two p50 lines, which is all I actually needed by hour thirty.

Would I skip the local run and jump straight to another machine? No. If the cold/reuse ratio does not appear on loopback, you do not have a hosting problem. You have a script problem, and moving computers will only relocate it.

What I would repeat

• Start with one curl timing line and one Python loop, then explain the gap in client-lifecycle terms before touching dashboards.
• Measure cold sockets and reused sockets as two named experiments, never as one blended average that hides the first call.
• Keep TLS verification disabled only for the throwaway loopback certificate, never for the real endpoint later.
• Record whether keep-alive survived, because a proxy can turn a reused client back into a cold client.
• Label DNS, TCP, TLS, and handler time separately, even if the first pass only has a cheap split between first and later calls.

I would not repeat the hour where I drafted a status update about origin latency. The origin was fine. My probe was invoicing TCP and TLS to the handler, then asking why the handler looked expensive.

Limitations, and who should not use this

This workflow is a client-lifecycle check, not a capacity test and not an SLO certificate. Loopback TLS inflates the handshake share, and a real WAN path may hide that share behind genuine origin work. Reused connections can also mask server bugs that appear only when a new TCP session is created, including per-connection memory leaks and unbalanced load-balancer membership.

Do not use this approach when you are certifying a public latency SLO, because loopback will flatter you. Do not use it to tune a load balancer, because reuse fights the distribution you want to observe. Do not disable certificate verification outside a throwaway local cert. Do not treat a remote shell as a substitute for a load generator that controls concurrency, warmup, and backlog.

If your traffic is HTTP/2 or HTTP/3, one connection multiplexes streams, and this HTTP/1.1 harness will not tell that story. If the client is a browser, connection coalescing follows different rules than http.client. If a platform injects Connection: close, reuse never happens, and the cold path is the only honest path you have.

What I am keeping next to the script

The useful question is not whether the API is slow. The useful question is which part of the client you are actually timing. Once I named the cold path and the reused path, the forty-eight hour argument ended in a few minutes of boring, repeatable output.

If you run the harness and the reused path is not cheaper, inspect Connection headers before you inspect the origin. That is the check I wanted on hour one, and it is the check I will run before I blame the network again. If your curl/Python gap survives that check, I want to hear which header lied.