DEV Community

Odd_Background_328
Odd_Background_328

Posted on

Reject Paid-to-Free Failover Before First-Byte Wait Beats Slack

#ai #devops #monitoring #sre

Your pager fires at 02:14 UTC during batch. Token spend sits near zero on the cost panel. Job p99 sits at forty-seven seconds against fifteen.

Free failover queue age reads thirty-eight seconds already. Remaining deadline slack is only nine seconds now. Which operational action follows from that evidence?

You do not enlarge the free pool tonight. You reject paid-to-free failover for that class.

Zero spend is not a recovery signal

A green cost panel can hide a dead SLO. Zero token spend often means jobs waited on free capacity. You paid with wall-clock time, not with tokens.

Deadline slack died while finance still looked healthy. Keep spend and latency on two separate ledgers. Never let a cheap dashboard veto a red latency page.

Topology for the local failover drill

Treat this as a labeled lab, not folklore. You need one admission process and two backends.

client -> admission (slack gate)
            |-- paid_backend   (reserved, metered)
            |-- free_backend   (best-effort FIFO)
            |-- reject_429     (fail closed)
Enter fullscreen mode Exit fullscreen mode

Keep both backends on separate local ports. Do not hide them behind one reverse proxy.

You must record which path each job took. Log backend, queue_age_ms, first_byte_ms, and remaining_slack_ms together.

Declared workload

Write the conditions down before any fault injection.

  • Job class: 200-word summary, one shot, no tools
  • Arrival: eight jobs per second for ninety seconds
  • Deadline: fifteen seconds from enqueue timestamp
  • Paid concurrency: two workers, 1200ms p95 first-byte
  • Free concurrency: one worker after the FIFO wait
  • Free first-byte p95 after queue: 4000ms
  • Free queue: FIFO, no preemption, no reservation
  • Retry: none on timeout; reject the job instead
  • Success: first byte before slack falls under 2000ms

These figures are lab defaults only, not production. They are not product benchmarks or fleet observations.

Change them if your serving path differs. Keep every number declared in the runbook header.

When free capacity is the wrong bet

Paid-to-free failover looks smart during a burst. Overflow onto a free queue looks fine on invoices. All three fail when wait age exceeds remaining slack.

You are not buying inference in that window. You are buying a lottery ticket on queue delay.

Timeouts make the free-capacity bet worse fast. A timed-out job retries into the same cold FIFO.

Token spend stays near zero on every retry. Deadline miss rate climbs with each requeue.

That is still a cost incident for the SLO. The invoice is not the only bill you pay.

Artifact: slack-gated paid-to-free failover

Label this helper as a proposed local gate. It is a sketch, not a vendor client.

# slack_gate.py — proposed local admission helper
import json
import time
from dataclasses import dataclass

DEADLINE_MS = 15_000
RESERVE_MS = 2_000
PAID_P95_FIRST_BYTE_MS = 1_200
FREE_P95_FIRST_BYTE_MS = 4_000
FREE_QUEUE_AGE_CUT_MS = 3_000

@dataclass
class Snapshot:
    paid_inflight: int
    paid_limit: int
    free_queue_age_ms: int
    free_inflight: int
    enqueued_at_ms: int

def remaining_slack_ms(now_ms: int, snap: Snapshot) -> int:
    waited = now_ms - snap.enqueued_at_ms
    return DEADLINE_MS - waited - RESERVE_MS

def admit(now_ms: int, snap: Snapshot) -> str:
    slack = remaining_slack_ms(now_ms, snap)
    if slack <= 0:
        return "reject_deadline"
    paid_ok = snap.paid_inflight < snap.paid_limit
    if paid_ok and slack > PAID_P95_FIRST_BYTE_MS:
        return "paid"
    free_need = snap.free_queue_age_ms + FREE_P95_FIRST_BYTE_MS
    age_ok = snap.free_queue_age_ms < FREE_QUEUE_AGE_CUT_MS
    if free_need < slack and age_ok:
        return "free"
    return "reject_no_slack"

if __name__ == "__main__":
    now = int(time.time() * 1000)
    snap = Snapshot(
        paid_inflight=2,
        paid_limit=2,
        free_queue_age_ms=8_000,
        free_inflight=1,
        enqueued_at_ms=now - 4_000,
    )
    decision = admit(now, snap)
    print(json.dumps({
        "decision": decision,
        "slack_ms": remaining_slack_ms(now, snap),
        "free_need_ms": snap.free_queue_age_ms + FREE_P95_FIRST_BYTE_MS,
    }))
Enter fullscreen mode Exit fullscreen mode

Run the helper once with that snapshot. Expected output for this labeled snapshot follows.

{"decision": "reject_no_slack", "slack_ms": 9000, "free_need_ms": 12000}
Enter fullscreen mode Exit fullscreen mode

Paid workers sit full in this snapshot. Free path needs twelve seconds of wait plus infer.

You only hold nine seconds of remaining slack. You reject the failover instead of waiting cheaper.

Runbook table for the on-call

paid inflight free queue age remaining slack action
below limit any greater than paid p95 plus reserve send paid
at limit under 3s greater than free p95 plus age send free
at limit 3s or more any reject 429
any any at or below zero reject 429

Pin the three-second free queue age cut. Rationale: age already spent the cheap part of slack.

Free worker utilization can still read twenty percent. One blocked first-byte hides behind that low CPU.

Queue age and slack beat utilization here. Do not add hosts because CPU looks idle.

Failure injection on the free backend

Do not wait for production to teach failover cost. Inject three faults against the local free stub.

  1. Sleep six seconds before the first response byte.
  2. Hold one extra FIFO job so age passes three seconds.
  3. Return HTTP 200 with an empty body after eight seconds.

Local stub

Proposed stub for fault one:

python3 - <<'PY'
from http.server import BaseHTTPRequestHandler, HTTPServer
import time

class H(BaseHTTPRequestHandler):
    def do_POST(self):
        time.sleep(6)
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.end_headers()
        self.wfile.write(b'{"ok":true,"bytes":0}')
    def log_message(self, *args):
        pass

HTTPServer(("127.0.0.1", 8088), H).serve_forever()
PY
Enter fullscreen mode Exit fullscreen mode

Drive the gate after the stub is up:

python3 slack_gate.py
# keep arrival at eight jobs per second for ninety seconds
Enter fullscreen mode Exit fullscreen mode

What you should see

Watch these fields instead of the token counter:

  • decision as paid, free, or reject_*
  • queue_age_ms on the free path only
  • first_byte_ms by path
  • remaining_slack_ms at admit time
  • deadline_miss after the handler returns
  • token_spend as a secondary ledger only

Expected observation under faults one and two follows. Reject count rises while deadline misses stay flat.

Token spend stays near zero on both paths. The useful page is reject_no_slack, not spend.

If misses rise with rejects, raise RESERVE_MS. Do not reopen unbounded paid-to-free failover tonight.

Telemetry fields to emit

Ship counters and histograms without blending them together.

job_decision{path="paid|free|reject"}
job_first_byte_ms{path="paid|free"}
job_queue_age_ms{path="free"}
job_remaining_slack_ms{stage="admit"}
job_deadline_miss{path="paid|free|reject"}
job_token_spend{path="paid|free"}
Enter fullscreen mode Exit fullscreen mode

Alert on this contradiction and nothing else. Token spend falls while deadline misses rise together.

That pattern is zero-spend failover eating slack. Page on-call with queue age and slack samples. Leave the cost panel off the paging policy.

Use free access for the drill, not the SLO

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option. Use that pair to stage this failover drill.

Burn retries and reject paths there without an invoice. Keep the same slack gate on every request.

If free queue age plus first-byte exceeds slack, reject. Do not promote the free path onto SLO traffic.

Free capacity is a lab and an overflow probe. It is the wrong bet for hard deadline work.

Rollback and cleanup

Keep the gate behind an environment flag.

export ADMIT_FREE=0
# or force reject by cutting age to zero
export FREE_QUEUE_AGE_CUT_MS=0
systemctl reload admission
Enter fullscreen mode Exit fullscreen mode

Drain free inflight before you touch paid workers. Do not SIGKILL paid workers during the drain.

curl -s localhost:9090/metrics | grep job_inflight
kill $(lsof -t -i:8088)
Enter fullscreen mode Exit fullscreen mode

Revert lab p95 constants to paid measurements only. Delete the delay stub from staging DNS and unit files.

Record reject ratio in the incident note. Record that spend was not the recovery signal.

Who should not use this gate

Skip this gate for chat with no deadline. Skip it if you cannot measure first-byte wait. Skip it when a late answer beats a hard reject.

Batch research jobs can wait on free capacity. User-facing summaries with a fifteen-second SLO cannot.

If your only backend is free, you cannot fail over. Buy a reserved paid path before you add failover. Then place the slack gate in front of it.

Limits of this note

The p95 constants are declared lab values. They are not observations from a named fleet.

First-byte wait is not full generation time. A long completion can still miss after a fast first byte. Add a completion budget if the SLO covers the full answer.

This note ignores auth, prompt size, and cache hits. Those belong in a later operational drill instead. Do not copy the three-second cut without your own age histogram.

Threshold question for the next review

Which signal trips reject in your runbook today? Is it queue age, utilization, or remaining slack?

Utilization lies when one worker blocks on first byte. Queue age tells you wait already spent. Remaining slack tells you wait you can still afford.

Trip on slack versus queue age plus p95 first-byte. Write that comparison in the runbook before the next burst.

Top comments (0)