DEV Community

Harvey He
Harvey He

Posted on

My gateway looked fine for 3 days. Then one request took 92 seconds.

#ai #monitoring #llm #devops

I run an OpenAI-compatible gateway. Every day I fire the same request at it three times and write down how long each one took. Tiny request, max_tokens=10, nothing clever.

Here's the last four days:

When #1 #2 #3
Aug 31 morning 2.42s 11.22s 92s (timeout)
Aug 31 evening 2.07s 2.30s 1.77s
Sep 1 1.64s 2.53s 8.40s
Sep 2 2.75s 2.12s 4.17s
Sep 3 2.46s 1.73s 3.00s

Fourteen requests. Thirteen of them under 12 seconds. One took 92 and then gave up.

If I'd only tracked the median, I would have concluded "about 2.3 seconds, very stable" and moved on with my life. The median is doing exactly what a bad metric does: it hides the one number that would actually cost me a customer.

Splitting the measurement in half

A slow response can come from three places: the upstream provider, my box, or the network between them. So I ran the same call straight at the upstream, skipping my gateway entirely.

Upstream direct:  20.77s | 1.47s | 4.52s   (3/3 completed)
My gateway:        2.42s | 11.22s | 92s     (2/3 completed)
Enter fullscreen mode Exit fullscreen mode

The upstream was slow and jumpy too — that 20.77s is ugly. But it never hung. My gateway did.

Then I looked at the box itself:

load average: 0.00, 0.00, 0.00
Mem: 205M / 1975M used
Disk: 7%
gateway process: up, port 3000 healthy
Enter fullscreen mode Exit fullscreen mode

Load 0.00 with 205 MB of memory used. The server had nothing to do. It wasn't struggling — it was waiting.

That's the whole diagnosis, and it took about four minutes: the jitters came from upstream, but the 92-second hang was mine. My gateway had no request timeout configured. When upstream stalled, my client just sat there holding the connection, for as long as upstream felt like taking.

Upstream's bad minute became my customer's bad minute, with interest.

The script I use

Two functions, same payload, two endpoints. Run them back to back and the difference tells you where to look.

import os, time, urllib.request, json

PAYLOAD = {
    "model": "deepseek-chat",
    "messages": [{"role": "user", "content": "say hi"}],
    "max_tokens": 10,
}

def call(base_url, token, timeout=30):
    req = urllib.request.Request(
        f"{base_url}/chat/completions",
        data=json.dumps(PAYLOAD).encode(),
        headers={"Authorization": f"Bearer {token}",
                 "Content-Type": "application/json"},
    )
    t = time.time()
    try:
        with urllib.request.urlopen(req, timeout=timeout) as r:
            r.read()
            return round(time.time() - t, 2), "ok"
    except Exception as e:
        return round(time.time() - t, 2), type(e).__name__

for label, url, key in [
    ("gateway",  "https://your-gateway/v1", os.environ["GATEWAY_KEY"]),
    ("upstream", "https://api.upstream.com/v1", os.environ["UPSTREAM_KEY"]),
]:
    print(label, [call(url, key) for _ in range(3)])
Enter fullscreen mode Exit fullscreen mode

Two things this buys you that a dashboard won't:

  1. A timeout is part of the measurement. If I'd used the default (no timeout), the 92-second call would just have hung and I'd have nothing to compare.
  2. You get three numbers, not one. One sample tells you nothing about variance. Three already shows you whether you have a latency problem or a tails problem.

What I'd tell anyone running their own gateway

Set a request timeout. Not next week — this is the single highest-value line of config you're probably missing, and the failure mode is silent: everything looks fine right up until a customer is staring at a spinner.

Then decide what happens on timeout. Retrying immediately into a struggling upstream just adds load. Falling over to a second provider costs you money but keeps the request alive. I picked the second one, because a slow answer beats no answer for most of what people build.

And log the max, not just the average. I now keep both:

p50 2.4s  |  max 92s  |  timeouts 1/14
Enter fullscreen mode Exit fullscreen mode

That one line is the reason I found this at all.

I'm writing this while the timeout config is still on my own to-do list, so treat it as a field note rather than a success story. The second-upstream fallback is live and tested; the timeout is the piece I haven't shipped yet.

If you run a gateway and you've never watched your own tail latency, go run those six calls. It takes two minutes and it's the cheapest insurance you'll buy this week.

Top comments (0)