Reject Orphan Streams Before Connection Age Beats Token Value

The pager fires at 02:11 with a cost alert. Active streams still look modest on the panel. Useful completions per minute barely moved at all.

Which operational action follows from that evidence? You should not scale replicas on utilization alone. You should inspect connection age against token value.

The contradiction on the board

Queue depth can stay inside the admission band. Orphan streams still burn tokens after clients leave. That split is a cost incident, not capacity.

You serve streaming chat behind a reverse proxy. The worker keeps generating after TCP close. Billing still counts output tokens on that worker.

Free capacity makes this failure easy to miss. Idle spend looks like a quiet night first. Deadline slack then collapses on the next burst.

Topology you can reproduce locally

Run one proxy, one worker, and one collector. Keep the whole drill on loopback interfaces. Do not point this test at shared production pools.

Proposed layout, not a measured production fleet:

• edge: reverse proxy with a 60s read timeout
• admit: small admission process on port 8090
• worker: one local generator with a single slot
• collector: scrapes /metrics every fifteen seconds

client -> edge:8080 -> admit:8090 -> worker:8100
                         |
                         +-> collector:9090/metrics

Declare the worker as a single-slot generator. One in-flight stream. No batching in this drill.

Declared workload

Label these as drill conditions, not production facts.

• Twenty clients open streaming completions on loopback
• Each prompt is fixed at about 400 input tokens
• Target output stops at 256 tokens, hard cap
• Eight clients abort at 1.5 seconds on purpose
• Useful-answer deadline slack is eight seconds
• Admission rejects streams older than six seconds

You are measuring waste, not model quality. Do not change sampling settings during the run. Pin max tokens so output cannot run away.

Telemetry fields you must export

If a field is missing, the drill is incomplete.

• stream_age_seconds: time since the request started
• client_connected: one while the client socket lives
• tokens_out_total: output tokens the worker actually billed
• tokens_useful_total: tokens delivered to a live client
• queue_age_seconds: wait at the head of the line
• slot_utilization: busy slots divided by available slots
• deadline_slack_seconds: remaining time before the deadline

Expected relationship, labeled as a hypothesis only. tokens_out_total minus tokens_useful_total is orphan spend. If that gap grows while utilization stays flat, reject.

When free capacity is the wrong bet

Free model access looks attractive for soak tests. A free server looks attractive for overnight evals. Orphan streams turn both into a silent leak.

You should refuse free capacity when these three hold:

1. Clients can disconnect without sending a cancel frame
2. The worker does not bind generation to socket life
3. Deadline slack is tighter than median stream age

Queue age, utilization, and slack will disagree here. Utilization can look fine with one zombie stream. Queue age then grows because the slot never frees.

Ask for a threshold with a written rationale. Do not pick a round number without a clock. Proposed threshold for this labeled drill:

• Reject when stream_age_seconds is greater than six
• Rationale: useful answers still need eight seconds slack
• Six seconds leaves two seconds to drain and enqueue

Prefer queue age over utilization for this failure. Prefer slack over raw token rate for the page. Time and retries are secondary once the slot is stuck.

A local drill can use free model access for the worker. Some teams also park that worker on a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. If you already stage soak traffic there, run this drain loop on that path before production.

Artifact: admit on stream age

Treat the next process as unexecuted example code. It rejects orphans before the worker spends more tokens. It also closes upstream when the client socket dies.

# unexecuted example: stream-age admission on loopback
import time
import threading
from http.server import BaseHTTPRequestHandler, HTTPServer

MAX_STREAM_AGE_S = 6.0
SLOT = {"busy": False, "started": 0.0, "client_live": False}
LOCK = threading.Lock()
METRICS = {
    "tokens_out_total": 0,
    "tokens_useful_total": 0,
    "rejected_age": 0,
    "cancelled_orphan": 0,
}

def snapshot():
    now = time.monotonic()
    with LOCK:
        age = (now - SLOT["started"]) if SLOT["busy"] else 0.0
        util = 1.0 if SLOT["busy"] else 0.0
        slack = 8.0 - age if SLOT["busy"] else 8.0
        return {
            "stream_age_seconds": round(age, 3),
            "client_connected": 1 if SLOT["client_live"] else 0,
            "slot_utilization": util,
            "deadline_slack_seconds": round(slack, 3),
            "queue_age_seconds": round(age if SLOT["busy"] else 0.0, 3),
            **METRICS,
        }

class Admit(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path != "/metrics":
            self.send_error(404)
            return
        body = "\n".join(f"{k} {v}" for k, v in snapshot().items()) + "\n"
        self.send_response(200)
        self.send_header("Content-Type", "text/plain")
        self.end_headers()
        self.wfile.write(body.encode())

    def do_POST(self):
        if self.path != "/v1/stream":
            self.send_error(404)
            return
        now = time.monotonic()
        with LOCK:
            if SLOT["busy"]:
                age = now - SLOT["started"]
                if age > MAX_STREAM_AGE_S:
                    METRICS["rejected_age"] += 1
                    busy = True
                else:
                    busy = True
            else:
                busy = False
            if busy and (now - SLOT["started"] <= MAX_STREAM_AGE_S):
                self.send_response(429)
                self.end_headers()
                self.wfile.write(b"slot_busy\n")
                return
            if busy:
                SLOT["busy"] = False
            SLOT["busy"] = True
            SLOT["started"] = now
            SLOT["client_live"] = True
        self.send_response(200)
        self.send_header("Content-Type", "text/event-stream")
        self.end_headers()
        try:
            for i in range(256):
                time.sleep(0.02)  # labeled fake generation, not a benchmark
                with LOCK:
                    METRICS["tokens_out_total"] += 1
                    live = SLOT["client_live"]
                    age = time.monotonic() - SLOT["started"]
                    if live:
                        METRICS["tokens_useful_total"] += 1
                if age > MAX_STREAM_AGE_S:
                    with LOCK:
                        METRICS["rejected_age"] += 1
                    break
                try:
                    self.wfile.write(f"data: {i}\n\n".encode())
                    self.wfile.flush()
                except BrokenPipeError:
                    with LOCK:
                        SLOT["client_live"] = False
                        METRICS["cancelled_orphan"] += 1
                    break
        finally:
            with LOCK:
                SLOT["busy"] = False
                SLOT["client_live"] = False

    def log_message(self, fmt, *args):
        return

if __name__ == "__main__":
    HTTPServer(("127.0.0.1", 8090), Admit).serve_forever()

Start it in a dedicated terminal before any clients connect.

python3 admit_orphan_stream.py

Failure injection you can run tonight

Use two terminals. Leave metrics scraping in the third.

# terminal B: a client that stays until 256 tokens
curl -N -X POST http://127.0.0.1:8090/v1/stream

# terminal C: eight aborting clients (labeled load, not a benchmark)
for i in $(seq 1 8); do
  curl -N -m 1.5 -X POST http://127.0.0.1:8090/v1/stream &
done
wait

# terminal D: scrape the same fields the pager should use
curl -s http://127.0.0.1:8090/metrics

Kill the long client with Ctrl-C during generation. Watch client_connected drop to zero. tokens_out_total must stop within one loop tick.

If output tokens keep climbing after the abort, you failed. The worker is ignoring socket death. That is the production leak in miniature.

Labeled expected output

These numbers are expected drill output, not production measurements.

tokens_out_total 64
tokens_useful_total 48
cancelled_orphan 8
rejected_age 1
stream_age_seconds 0.0
client_connected 0
slot_utilization 0.0
queue_age_seconds 0.0
deadline_slack_seconds 8.0

Orphan gap equals billed tokens minus useful tokens. Here that gap is sixteen tokens. Eight aborts plus one age reject should not hold the slot.

If slot_utilization stays at 1.0 after aborts, rollback. If queue_age_seconds exceeds six with slack under two, page. Do not wait for utilization to look red.

Failure handling during the drill

Follow this order. Do not skip to replica scaling.

1. Freeze new admissions when stream_age_seconds exceeds six
2. Send a cancel to the worker, then close the upstream
3. Mark client_connected=0 before counting more useful tokens
4. Drain the single slot until queue_age_seconds returns to zero
5. Only then reopen admissions for the next labeled client

Retries belong after the slot is free. Retrying against a zombie stream doubles token spend. Time spent retrying is not deadline slack.

Cleanup and rollback

Stop the admission process and any leftover curl jobs. Confirm nothing still listens on 8090 or 8100. Restore the previous proxy timeout if you changed it.

pkill -f admit_orphan_stream.py || true
pkill -f "curl -N -X POST http://127.0.0.1:8090/v1/stream" || true
ss -ltnp | grep -E '8090|8100' || echo "loopback ports clear"

Rollback rule for a real edge proxy is equally small. Restore the last known timeout and cancel mapping. Revert admission to "slot busy means 429" only. Do not leave the six-second age cut in production unreviewed.

Write the revert in the same change window. Leave a note on the threshold rationale. The next on-call should see slack, not folklore.

Decision table: queue age, utilization, slack

Use this table during the page, not after it.

• High utilization, low queue age, healthy slack: keep serving
• Low utilization, high queue age, shrinking slack: reject orphans
• Low utilization, low queue age, high token gap: cancel zombies
• High utilization, high queue age, slack under two seconds: shed load
• Free capacity plus any zombie gap: stop the soak, do not scale

Free capacity is the wrong bet for interactive deadlines. It is also wrong when cancel frames never reach the worker. Pay for a slot you can kill, or do not offer streaming.

Limitations and who should not use this

This drill assumes one slot and a hard output cap. It does not model multi-tenant fairness. It does not prove quality, cache hit rate, or safety.

Do not use this admission cut on long-running batch jobs. Batch work needs queue age against a batch deadline. Six seconds will false-reject those jobs immediately.

Do not use it as a billing auditor. Token fields here are local counters. They are not invoices from any provider.

Skip this approach if your proxy already ties generation to connection lifetime. Also skip it if clients cannot stream and only poll. Polling needs a different cancel path and different metrics.

The example sleeps for fake tokens on purpose. Replace that loop with your real worker only in a lab. Keep production traffic off this process.

What you do after the metrics agree

You now have a reject path before token value goes negative. Connection age is the first lever, not replica count. Queue age tells you the slot is trapped.

Run the drain drill on a throwaway worker before the next cost page. Keep the threshold tied to deadline slack in writing. If slack and queue age disagree, trust slack and drain.