DEV Community

Jordan Huang
Jordan Huang

Posted on

Shared Lane or Shared Outage? A Free-Tier Probe and FAQ

You see one slow request. The page hangs for nine seconds.

Then it recovers. Your p50 looks fine. Your users still felt the lie.

Free-tier latency is not a flat number. It's a queue. The queue is shared. Model and server tend to share the same lane. When the lane jams, both choke.

In this post you'll get a probe script, a coupling test, a decision table, and a five-question FAQ. Run them before you rewrite your stack.

The two clocks in every HTTP call

Every request has two visible times:

  • Time to first byte (TTFB)
  • Time to last byte (TTLB)

TTFB captures queueing. TTLB captures transfer plus server work.

If TTFB balloons while TTLB stays flat, your code is waiting for a slot.
If both balloon, the service itself is slow.

This distinction is your first weapon.

A probe that exposes the queue

Save this as probe_queue.py.

import concurrent.futures as cf
import urllib.request
import time
import statistics

URL = 'https://your-endpoint.example/ping'
N = 20
CONCURRENCY = 5

def probe(i):
    t0 = time.perf_counter()
    try:
        with urllib.request.urlopen(URL, timeout=30) as r:
            r.read()
        return time.perf_counter() - t0
    except Exception:
        return time.perf_counter() - t0

with cf.ThreadPoolExecutor(max_workers=CONCURRENCY) as ex:
    results = list(ex.map(probe, range(N)))

latencies = sorted(results)
print(f'p50: {statistics.median(latencies):.3f}s')
print(f'p90: {latencies[int(N * 0.9) - 1]:.3f}s')
print(f'p99: {latencies[int(N * 0.99) - 1]:.3f}s')
print('sorted:', ' '.join(f'{s:.2f}' for s in latencies))
Enter fullscreen mode Exit fullscreen mode

Run it with:

python probe_queue.py
Enter fullscreen mode Exit fullscreen mode

Run it three times. Different days. Different hours.
Record p99. Watch it jump.

How to read the output

Here is a synthetic output from a crowded shared queue:

p50: 0.42s
p90: 3.10s
p99: 14.80s
sorted: 0.21 0.33 0.40 0.44 0.51 0.62 0.89 1.70 2.90 4.80 14.80 15.10
Enter fullscreen mode Exit fullscreen mode

See the gap between p50 and p99. That gap is a queue.
A dedicated system keeps p99 close to p50. A shared system rarely does.

Rule of thumb: if p99 is more than 3x p50, expect a queue.

Want one number? Compute the coefficient of variation:

mean = statistics.mean(latencies)
stdev = statistics.stdev(latencies)
print(f'CV: {stdev / mean:.2f}')
Enter fullscreen mode Exit fullscreen mode

CV above 1.0 means heavy-tailed traffic. That's a queue signature.

The coupling test: model and server together

Now bring in a second endpoint. The myth says model and server are independent.

Let's probe them simultaneously.

# probe_both.py
import concurrent.futures as cf
import urllib.request
import time
import statistics

MODEL_URL = 'https://model-endpoint.example/ping'
SERVER_URL = 'https://server-endpoint.example/ping'
SAMPLES = 10

def measure(url):
    t0 = time.perf_counter()
    try:
        with urllib.request.urlopen(url, timeout=30) as r:
            r.read()
        return time.perf_counter() - t0
    except Exception:
        return float('inf')

def one_sample(_):
    return measure(MODEL_URL), measure(SERVER_URL)

with cf.ThreadPoolExecutor(max_workers=2) as ex:
    pairs = list(ex.map(one_sample, range(SAMPLES)))

model = [p[0] for p in pairs]
server = [p[1] for p in pairs]

print('model:', ' '.join(f'{x:.2f}' for x in model))
print('server:', ' '.join(f'{x:.2f}' for x in server))
Enter fullscreen mode Exit fullscreen mode

Run it during peak hours. Look at the pairs.
If they spike together, you've found shared admission control.

The five-question myth-busting FAQ

Q1: Is my free server down when it's slow?

No. It's queued.
A short wait usually restores it. A restart helps less than patience.

Q2: Should I blame my code first?

No. Run the probe first.
If TTFB stretches across many requests, your code is innocent.

Q3: Should I increase retries?

Only with exponential backoff and jitter.
Raw retries multiply queue pressure. Sleep 1s, 2s, 4s, add randomness.

Q4: Is a 95% success rate enough?

It depends on your users.
A chat app feels the tail. A nightly batch job doesn't.

Q5: Are free model and free server truly independent?

Usually not.
They often share the same lane. That's the myth this post is meant to kill.

A mental model that works

Stop picturing two separate products.

Think: one shared lane, two toll booths.

Lane empty? Fast lane.
Lane crowded? Both suffer.

Design for a crowded lane.

Decision table: should you use free tier?

Use case Free tier OK? Why
Local experiments ✅ Latency doesn't matter
Low-volume internal tools ⚠️ Add timeouts and caching
User-facing chat ❌ Tail latency kills UX
Payments or auth ❌ Correctness beats savings
Scheduled offline batcher ✅ Queue-friendly and retryable

If you see ❌, buy a dedicated plan or self-host.

One workflow that fits

Build batch jobs, not real-time calls.

Outline:

  1. Push tasks to a Redis queue.
  2. Worker picks one task.
  3. Worker calls the free model endpoint once.
  4. On timeout, worker sleeps 5–20s, then retries.
  5. Success writes to DB. Failure writes to dead-letter queue.

This pattern absorbs spikes. I use it for side projects.

MonkeyCode offers free model access and a free server option. That pair maps nicely to steps 3 and 2.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Who should not use this approach

Skip this pattern if you:

  • Have a strict response-time SLA
  • Process payments or auth
  • Demo live to a client
  • Face a customer support queue

Free tier is a lab, not a luxury box.

Final thought

Measure first. Blame second.

Probe the queue. Respect the tail.
Then decide if free tier fits your workload.

That's all. Go run probe_queue.py.

Top comments (0)