The Model Answered in 300ms. The Request Still Took 2.4 Seconds.

I spent an afternoon convinced my model was slow. The API was returning responses in under 300 milliseconds, yet my end-to-end request was taking 2.4 seconds, and I could not figure out where the missing two seconds were hiding. The model was not the problem, my code was not the problem, and the free server was not the problem; the network path between them was.

I was running a small summarization service on MonkeyCode's free server tier, which gives you a reported 10M-token allowance plus a free server for experiments. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The free tier is genuinely useful for this kind of diagnosis because you can hammer your own endpoint without worrying about a bill, but the network behavior you discover will apply to any remote API.

The First Mistake: Timing the Wrong Thing

My initial benchmark was embarrassingly simple. I wrapped the whole request in a timer and blamed whatever came out:

import time

start = time.perf_counter()
response = client.chat.completions.create(
    model=MODEL,
    messages=[{"role": "user", "content": "Summarize this article."}],
)
elapsed = time.perf_counter() - start
print(f"total: {elapsed:.2f}s")

That number told me the request was slow, but it told me nothing about why. The model's own latency, which I could see in the API response metadata, was consistently under 300ms. The gap between 300ms and 2400ms was the real mystery, and a single timer could not see inside it.

Breaking Down the Request Into Stages

I needed to know where the time was going, so I instrumented each stage of the network journey separately. The stages are always the same: DNS resolution, TCP connection, TLS handshake, request upload, waiting for the server, and response download.

import socket
import ssl
import time
import urllib.request

HOST = "api.example.com"
PATH = "/v1/chat/completions"

# Stage 1: DNS resolution
t0 = time.perf_counter()
ip = socket.getaddrinfo(HOST, 443, type=socket.SOCK_STREAM)[0][4][0]
t1 = time.perf_counter()
print(f"DNS: {t1 - t0:.3f}s -> {ip}")

# Stage 2: TCP connection
sock = socket.create_connection((ip, 443), timeout=10)
t2 = time.perf_counter()
print(f"TCP connect: {t2 - t1:.3f}s")

# Stage 3: TLS handshake
ctx = ssl.create_default_context()
ssock = ctx.wrap_socket(sock, server_hostname=HOST)
t3 = time.perf_counter()
print(f"TLS handshake: {t3 - t2:.3f}s")

# Stage 4: Send request + wait for response
request_body = b'{"model": "test", "messages": []}'
ssock.sendall(b"POST " + PATH.encode() + b" HTTP/1.1\r\nHost: " + HOST.encode() + b"\r\nContent-Type: application/json\r\nContent-Length: " + str(len(request_body)).encode() + b"\r\nConnection: close\r\n\r\n" + request_body)

t4 = time.perf_counter()
response = b""
while True:
    chunk = ssock.recv(4096)
    if not chunk:
        break
    response += chunk
t5 = time.perf_counter()
print(f"Request + wait + response: {t5 - t4:.3f}s")

ssock.close()

The output from my free server looked like this:

DNS: 0.482s -> 172.67.x.x
TCP connect: 0.231s
TLS handshake: 0.312s
Request + wait + response: 1.375s

DNS was eating nearly half a second. That was the first surprise. The second surprise was that the request phase, which includes waiting for the model to generate, took 1.375 seconds even though the model itself reported 300ms of inference time.

Why DNS Was So Slow

A 482ms DNS lookup is not normal, and the cause turned out to be embarrassingly mundane. The free server's /etc/resolv.conf pointed at a single nameserver that was slow to respond, and every new connection triggered a fresh lookup because the OS resolver does not cache aggressively by default.

I verified this with a simple loop:

for i in $(seq 1 10); do
  time getent hosts api.example.com
done

Every lookup took between 400ms and 600ms. The fix was to stop resolving the hostname on every request and instead pin the IP address with a fallback resolver.

import socket

# Resolve once at startup, cache forever
_IP_CACHE = {}

def resolve(host: str) -> str:
    if host not in _IP_CACHE:
        _IP_CACHE[host] = socket.getaddrinfo(host, 443, type=socket.SOCK_STREAM)[0][4][0]
    return _IP_CACHE[host]

That single change cut 400ms off every request. The lesson is boring but important: DNS is not free, and on a free server with a bad resolver configuration, it can dominate your entire latency budget.

The Hidden Cost of Connection Setup

TCP connect plus TLS handshake added another 543ms. That is the price of a cold connection, and it is paid on every request if you create a new connection each time. The fix is connection reuse, which the official client libraries already do for you, but only if you let them.

from openai import OpenAI

client = OpenAI(
    api_key=API_KEY,
    base_url=BASE_URL,
    max_retries=2,
    timeout=30.0,
)
# The underlying httpx client keeps connections alive automatically.
# Do NOT create a new OpenAI client per request.

If you are building your own HTTP layer, use a persistent connection pool instead of opening a socket per request. The urllib3 or httpx pools will keep the TLS session alive and skip the handshake on subsequent requests.

The 1.375 Seconds That Was Not Model Time

The request phase included the model's 300ms of inference, so where did the other second go? I added timestamps to the server logs and compared them with client-side timestamps, and the gap turned out to be request serialization and response buffering on the server side. The free server was spending time parsing my JSON, validating the request, and then buffering the full response before sending it back.

I could not change the server's behavior, but I could change what I sent. A smaller request body and a simpler schema meant less parsing work. I also switched from sending a full conversation history to sending only the last two messages, which cut the payload size by 60%.

The Reproducible Test Plan

Here is the test plan I used to verify each fix, so you can reproduce the diagnosis on your own setup:

1. Baseline: Run 20 requests with the naive single-timer benchmark. Record the total time and the model's reported inference time.
2. DNS test: Run getent hosts <api-host> 10 times. If the average is above 100ms, implement IP caching and re-run the baseline.
3. Connection test: Check whether your client reuses connections. Run 20 requests and count how many TCP handshakes happen using ss -tn or netstat. If every request opens a new connection, switch to a persistent pool.
4. Payload test: Measure your request body size. Cut it down by removing unnecessary history or context, then re-run the baseline.
5. Final comparison: Compare the before and after distributions, not just the averages. The median matters more than the mean when you have network jitter.

I ran this exact plan and the numbers went from a 2.4s median to a 0.9s median, with the model's inference time staying flat at 300ms the whole time. The model was never the bottleneck.

Limitations: When This Approach Does Not Help

Network optimization has a floor, and you should know where it is before you start. If your model call is genuinely slow because the model is doing heavy reasoning, no amount of DNS caching will help. If your free server is geographically far from the API endpoint, you will see a fixed latency floor that no client-side trick can remove. And if the API itself rate-limits you, connection reuse will not save you from a 429.

You should probably not use this approach if you are building a single-user script that runs once a day; saving 1.5 seconds per request does not matter when you make one request. Use it when you are building a service that makes many requests per minute, because the per-request savings compound quickly.

The Takeaway

The model answered in 300ms, but my request took 2.4 seconds because I never looked at the network path between me and the model. DNS resolution, TCP handshakes, TLS negotiation, and payload serialization are all part of your real latency, and they are all fixable with boring, well-understood techniques. If you want to reproduce this diagnosis on a zero-budget stack, MonkeyCode's free server is a reasonable place to run the experiment, mostly because the only thing you waste while debugging is time.

MonkeyCode provides free models that can run this workflow.