DEV Community

sichi chen
sichi chen

Posted on

Stop trusting a 99% uptime claim: the LLM API relay failures we hit, and how we test stability

#api #devops #llm #monitoring

*Originally published on the Folkbench blog: https://folkbench.com/en/blog/ai-api-relay-stability-guide?utm_source=luntan&utm_campaign=dev *

A relay site may promise "99.9% fast and stable" and a "dedicated high-concurrency channel". Once you connect, things look different. This is how we went from buying on instinct to running automated checks before we pay for or integrate a relay.

1. The failures that forced the method

Fake 200 OK and a quiet truncation

Our early check was a Python script that POSTed to the relay every 5 minutes. HTTP 200 with some text = healthy.

Less than two days into production, users reported the model stopping mid-sentence. The logs showed that some relays still returned 200 OK when the upstream pool was exhausted (429 / 401), when the gateway timed out (504), or when a node crashed. Mid-stream they would quietly insert data: [DONE], or put {"error": "rate limit exceeded"} inside the JSON as if it were the model's answer. A status-code monitor was fully fooled.

The nighttime stall

Daytime average response stayed near 2 seconds. At the night peak, timeouts and abandonment climbed. The worst case still finished the whole answer in about 4 seconds, but the first token took 12 seconds. Users stared at an empty box and left.

Lesson: one ping, or two casual requests, is not a test of a streaming AI API.

2. Three versions of the test system

  • 1.0 – blind trials: a few prompts in a web UI or 10 hand-written requests. Tiny, noisy sample; rare disconnects only show up in production.
  • 2.0 – concurrency + status codes: Postman / Locust, status-code mix, average latency, success rate. Sees the outer gateway, misses broken streams, forged errors and stalls between tokens.
  • 3.0 – a stream state machine: treat the relay like a long-connection network middlebox (think realtime audio/video or market-data feeds, not web requests). Multi-region probes split connection setup, TLS handshake, first-token arrival, token throughput and SSE parsing.

3. The three dimensions that decide production experience

Latency – separate waiting from generating

  • TTFT (time to first token): request → first SSE packet. Our production bar is TTFT < 1.5 s.
  • TPS: generated tokens / duration after the first token.
  • P95 / P99: ignore the flattering average.

Reliability – look through the fake success

  • Parse success: the official SDK must parse the payload, JSON must be complete, no relay-injected error inside the answer.
  • Retry-to-success: with exponential backoff, does a 429/5xx recover within 3 retries?

Stream quality – catch the silent failure

  • Stream interruption: share of calls that drop or go silent before data: [DONE].
  • Inter-token jitter: standard deviation of gaps between SSE packets. High jitter = unstable path or stacked proxy buffers.

4. References, not invented in a closed room

  • Google SRE four golden signals: latency, traffic, errors, saturation — the relay is treated as an untrusted external service.
  • OpenTelemetry + W3C Server-Sent Events: record TCP handshake, TLS, headers, first chunk and [DONE].
  • Wilson score interval: 10/10 (100%) is not automatically more trustworthy than 9,900/10,000 (99%). We rank by the lower bound: 
Score = (p̂ + z²/(2n) − z·√(p̂(1−p̂)/n + z²/(4n²))) / (1 + z²/n)
Enter fullscreen mode Exit fullscreen mode

p̂ = observed success rate, n = sample size, z = normal quantile for 95% confidence. Small samples get penalized.

5. How the probe is built

  • Multi-region probes (AWS US West, GCP East Asia – Hong Kong/Tokyo, Europe) separate a stuck relay from an upstream swing.
  • Socket state machine: CONNECTING (t0) → FIRST_BYTE (TTFT = t1 − t0) → STREAMING (gaps, TPS, jitter) → COMPLETED ([DONE] present, valid JSON).
  • Automatic attribution:
    • 502/504 → the relay's own gateway node failed.
    • 429/401 → concurrency control failed or upstream key balance ran out.
    • 200 + mid-stream drop or parse failure → the relay's billing/filter middleware overflowed.

6. Three rules before you pick a relay

  1. Don't trust one test. Sample day and night for at least 3–7 days; watch P95 and success rate in the 21:00–01:00 peak.
  2. Test streaming. Non-streaming hides first-token delay, buffering and mid-stream deaths.
  3. A price far too low is not a gift. It usually means a shared account pool, a poor proxy, or a cheap open model presented as a premium one.

Folkbench publishes these measurements (P95 latency, availability and per-token price per relay) with no sign-up needed: https://folkbench.com/?utm_source=luntan&utm_campaign=dev

Top comments (0)