Reject Orphan Retries Before Token Waste Beats Slack

You open the 02:17 pager with two contradictory graphs. Queue depth looks calm under the admission cap. Token spend still climbs after clients already cancelled.

Which operational action follows from that split evidence? You do not scale the free overflow pool yet. You hunt orphan in-flight work before adding capacity.

Topology you can reproduce

You run mixed inference behind a single admission queue. Paid workers hold leases only for hard deadline work. A free overflow pool takes quiet-hour batch overflow only.

Name every hop in the runbook before the drill. Missing hops are how cancel events disappear.

• The gateway records client cancel, deadline, and retry-id.
• The admission queue exposes age, depth, and in-flight count.
• Each worker heartbeats a lease keyed by task_id.
• The token ledger accumulates cost per task_id only.
• Overflow workers stay isolated from the paid lease pool.

That overflow pool is optional capacity, not an SLO. Treat it as a scratch replica during this rehearsal.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project you can use for that isolated replica. It provides free model access and a free server option.

Park the drill there, then tear the replica down. Do not attach production SLOs to that pool.

Declared workload

You must freeze the workload before you inject cancels. Changing payload size mid-drill invalidates the token ledger.

Write these knobs on the incident whiteboard first. Keep them unchanged until the drill ends.

1. Eight synthetic tasks share one twelve-second deadline each.
2. Each prompt is a fixed eight-hundred-token fixture file.
3. One client cancel arrives at three-and-a-half seconds.
4. Retry policy allows one retry with the same retry-id.
5. Overflow workers heartbeat their leases every four seconds.
6. Paid workers stay capped at two concurrent leases.

Do not treat these numbers as production benchmarks. They are drill knobs, not capacity claims.

Failure mode: orphan in-flight work

Clients cancel while the gateway still shows healthy depth. Workers keep decoding because leases never saw the cancel. Tokens still accrue against task_ids you already abandoned.

That is orphan in-flight work, not a capacity shortage. Free overflow makes the miss cheap to ignore at first. It becomes expensive when retries spawn a second decoder.

Watch three signals together, never one graph alone. One green graph is how this page stays open.

• queue_age_ms can look healthy while decoders still run.
• in_flight_by_retry_id should drop to zero after cancel.
• tokens_by_task_id must freeze within one cancel grace.

If the third signal keeps moving, you have an orphan. Scaling workers will multiply the waste, not clear it.

Artifact: cancel-and-shed probe

You can reproduce the miss with a local probe. The script below is a rehearsal harness, not production code.

Save it as cancel_shed_probe.py beside a fixture prompt.

#!/usr/bin/env python3
"""Local cancel-and-shed rehearsal. Not a production worker."""
from __future__ import annotations

import argparse
import json
import threading
import time
from dataclasses import dataclass
from typing import Dict, Optional

DEADLINE_S = 12.0
CANCEL_AT_S = 3.5
CANCEL_GRACE_S = 1.0
DECODE_TOKENS_PER_S = 40  # drill knob only
PROMPT_TOKENS = 800


@dataclass
class Task:
    task_id: str
    retry_id: str
    deadline_mono: float
    canceled: bool = False
    in_flight: int = 0
    shed_reason: Optional[str] = None
    tokens_at_cancel: int = 0
    tokens_at_end: int = 0


class Ledger:
    def __init__(self) -> None:
        self._by_task: Dict[str, int] = {}
        self._lock = threading.Lock()

    def add(self, task_id: str, n: int) -> None:
        with self._lock:
            self._by_task[task_id] = self._by_task.get(task_id, 0) + n

    def get(self, task_id: str) -> int:
        with self._lock:
            return self._by_task.get(task_id, 0)


class Worker(threading.Thread):
    def __init__(
        self,
        task: Task,
        ledger: Ledger,
        cancel_event: threading.Event,
        orphan_mode: bool,
    ) -> None:
        super().__init__(daemon=True)
        self.task = task
        self.ledger = ledger
        self.cancel_event = cancel_event
        self.orphan_mode = orphan_mode

    def run(self) -> None:
        self.task.in_flight += 1
        self.ledger.add(self.task.task_id, PROMPT_TOKENS)
        start = time.monotonic()
        while True:
            now = time.monotonic()
            slack = self.task.deadline_mono - now
            if not self.orphan_mode:
                if self.cancel_event.is_set() or self.task.canceled:
                    self.task.shed_reason = 'client_cancel'
                    break
                if slack < CANCEL_GRACE_S:
                    self.task.shed_reason = 'deadline_slack'
                    break
            if now - start > DEADLINE_S * 1.5:
                self.task.shed_reason = 'drill_bound'
                break
            time.sleep(0.25)
            self.ledger.add(self.task.task_id, int(DECODE_TOKENS_PER_S * 0.25))
        self.task.in_flight -= 1
        self.task.tokens_at_end = self.ledger.get(self.task.task_id)


def run_one(mode: str, idx: int) -> dict:
    orphan_mode = mode == 'orphan'
    task = Task(
        task_id=f't-{idx}',
        retry_id=f'r-{idx}',
        deadline_mono=time.monotonic() + DEADLINE_S,
    )
    ledger = Ledger()
    cancel_event = threading.Event()
    worker = Worker(task, ledger, cancel_event, orphan_mode)
    worker.start()
    time.sleep(CANCEL_AT_S)
    task.canceled = True
    cancel_event.set()
    task.tokens_at_cancel = ledger.get(task.task_id)
    time.sleep(0.3)
    in_flight_after = task.in_flight
    worker.join(timeout=DEADLINE_S * 2)
    return {
        'mode': mode,
        'task': task.task_id,
        'retry_id': task.retry_id,
        'cancel_at_s': CANCEL_AT_S,
        'deadline_s': DEADLINE_S,
        'in_flight_after_cancel': in_flight_after,
        'tokens_at_cancel': task.tokens_at_cancel,
        'tokens_at_deadline': task.tokens_at_end,
        'shed_reason': task.shed_reason,
    }


def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument('--mode', choices=['orphan', 'shed'], required=True)
    parser.add_argument('--tasks', type=int, default=1)
    args = parser.parse_args()
    rows = [run_one(args.mode, i) for i in range(args.tasks)]
    print(json.dumps(rows, indent=2))
    leaked = [r for r in rows if r['tokens_at_deadline'] > r['tokens_at_cancel'] + 20]
    if args.mode == 'orphan':
        print('VERDICT: orphan decode after cancel' if leaked else 'VERDICT: unexpected shed')
    else:
        print('VERDICT: shed within grace' if not leaked else 'VERDICT: cancel missed')


if __name__ == '__main__':
    main()

How you run it

Create a venv so the drill stays off the host Python. Install nothing beyond the standard library for this probe.

python3 -m venv .drill
source .drill/bin/activate
python3 cancel_shed_probe.py --mode orphan --tasks 8
python3 cancel_shed_probe.py --mode shed --tasks 8

Compare the two modes on the same fixture. Do not mix timestamps across the two runs.

Expected output (labeled, not observed production)

Orphan mode should look roughly like this example.

mode=orphan task=t-0 retry_id=r-0
cancel_at_s=3.50 deadline_s=12.00
in_flight_after_cancel=1
tokens_at_cancel=140
tokens_at_deadline=480
shed_reason=drill_bound
VERDICT: orphan decode after cancel

Shed mode should freeze tokens near the cancel point.

mode=shed task=t-0 retry_id=r-0
cancel_at_s=3.50 deadline_s=12.00
in_flight_after_cancel=0
tokens_at_cancel=140
tokens_at_deadline=152
shed_reason=client_cancel
VERDICT: shed within grace

Those figures are fixture math, not vendor benchmarks. Recalculate them if you change the decode knob.

Inject the cancel miss

You now break the cancel path on purpose. That is the drill, not a production experiment.

Run these faults one at a time. Combining faults hides which hop dropped cancel.

1. Drop the cancel event on overflow workers only.
2. Heartbeat leases slower than the remaining deadline slack.
3. Retry without forwarding the original retry-id header field.
4. Attribute tokens to worker_id instead of task_id.
5. Leave a second decoder running after the client left.

After each fault, capture these four telemetry fields.

• deadline_slack_ms
• lease_age_ms
• orphan_token_delta
• duplicate_in_flight

You promote a fault only when orphan_token_delta stays zero. Utilization is the last signal you should trust here.

Decision table: shed versus retry

Ask yourself this operational question before you ship. Do you shed on queue age, utilization, or deadline slack?

Write the rule in the runbook with a rationale.

Signal	Trust it for	Do not use it for
queue age	Backup shed when depth lies	Free-pool idle that hides orphans
utilization	Paid pool saturation only	Overflow CPU while decoding abandoned tokens
deadline slack	Primary cancel and shed trigger	Long batch jobs without a client deadline
orphan token delta	Proof the cancel path worked	Capacity planning or vendor comparison

The primary trigger is deadline slack, not utilization. Queue age is the backup when depth and spend disagree. Orphan token delta is the proof, not the actuator.

Proposed shed rule for this rehearsal only:

• Shed if lease_age + cancel_grace > deadline_slack.
• Shed if in_flight_by_retry_id > 1 after one retry.
• Reject new overflow admission if orphan_token_delta > 0.
• Never retry onto the free pool without the same retry-id.

Label that rule as a proposal until you measure it. Then store the measured grace on the service page.

When free overflow is the wrong bet

Free capacity is the wrong bet under several conditions. You should fail closed to paid workers, or reject the work.

Reject overflow admission when any line below is true. One true line is enough to fail closed.

• Cancel propagation time exceeds the remaining deadline slack.
• The worker has no cancel channel and no lease kill.
• Retries lack a stable retry-id for duplicate detection.
• The ledger keys cost by host instead of task.
• Heartbeat period is larger than one third of the deadline.
• A second decoder can start before the first sheds.

Time, tokens, retries, and queueing all stack here. A free slot that cannot shed still burns wall-clock slack. It also burns tokens the client will never read.

That is a cost incident, not a savings story. Paid path is cheaper when slack is already thin. Reject the work before the queue looks busy.

Waiting for utilization to spike is too late. Scale only after the orphan counter returns to zero.

Cleanup and rollback

You end the drill with an explicit teardown. Do not leave replica workers holding open leases.

pkill -f cancel_shed_probe.py || true
rm -f /tmp/orphan-ledger.json
deactivate

Rollback the admission rule if the shed was too aggressive. Restore the previous retry limit and heartbeat period. Record the change in the same incident ticket.

If production ever copied the fault, follow this order. Keep the steps in this exact rollback order.

1. Stop overflow admission at the gateway.
2. Cancel all leases older than one grace period.
3. Freeze the token ledger for affected task_ids.
4. Drain in-flight decoders before reopening overflow.
5. Reopen only after orphan_token_delta stays at zero.

Who should skip this drill

Skip this approach if you have no client deadlines. Batch jobs without slack cannot use shed-on-slack. Use a calendar drain window instead, with a human gate.

Skip it if you cannot key work by retry-id. Without that key you cannot detect duplicate decoders. Fix the identity model before you tune overflow.

Skip it if legal or safety work cannot be cancelled. Those tasks need a paid, interrupt-safe worker pool. Do not park them on optional free capacity.

Limitations

This probe does not speak any vendor model protocol. It fakes decode cost with a constant token knob. Real streaming cancel behavior will differ per gateway.

It also ignores prompt-cache effects and tokenizer drift. Treat the ledger as an ops counter, not a billing replica. Do not publish these fixture numbers as capacity results.

The topology assumes one retry and one overflow pool. Multi-region failovers need a different lease story. Leave that design work outside this runbook.

What you write in the runbook

Put one threshold on the service page tonight. State the signal, the number, and the rationale.

Example line you can adapt after the drill:

Shed overflow when lease_age + 1s > deadline_slack; utilization on the free pool is not an SLO.

Then ask the oncall the same question you opened with. Queue looks calm, tokens still move, clients already left. Do you scale the pool, retry the task, or shed?

You already know the answer from the split graphs. Shed first, then scale only after orphan_token_delta is zero.