Reject Free-Server Preempts Before Restart Tokens Beat Slack

You get paged at 02:14 local time.
GPU utilization sits at only four percent now.
Queue age already reads eleven full minutes.
Deadline slack on the batch is eight minutes.
Which operational action now follows from that evidence?

Do not add workers from the idle reading.
Idle GPU often means a recent preempt event.
The free shared host reclaimed your decode slot.
Your generation died without a usable checkpoint.
Restart tokens will now consume remaining slack.

Why idle GPU lies during free-server drain

Free shared servers do not promise reserved time.
Your process can vanish between two decoder steps.
The collector still reports low utilization after that.
That number describes an empty seat, not spare capacity.
You must watch queue age against deadline slack.

Restart cost is not a simple HTTP retry.
It is a full prompt replay plus partial output.
That replay burns tokens you already spent once.
It also burns wall clock you no longer have left.
Reject new work before that replay loop starts.

A four percent GPU graph can look healthy.
The queue behind it is already past halfway.
That contradiction is your admission signal tonight.
Scale-out copies the same preempt onto new hosts.
Drain and reject instead of packing more tasks.

Topology you can run on a laptop

Keep the drill local and fully declared.
Use one admission process and one fake worker.
Inject preempts with a timer inside the lab.

Declared topology:

• admit.py handles admission and the token ledger
• the worker is a fake decoder with coarse checkpoints
• SIMULATE_PREEMPT=1 kills progress after two ticks
• one laptop process, with no remote model host

Declared workload:

• one task with 800 prompt tokens attached
• 200 output tokens as the completion target
• 120 second deadline for the whole task
• preempt after two decode ticks in this lab
• no production traffic and no shared cluster

This is a labeled lab, not a production claim.
Expected JSON below comes from the script only.
Do not read those numbers as a billed invoice.

Admission rule before you enqueue anything

You need one threshold with a written rationale.
Use restart-token waste versus remaining deadline slack.
Do not use GPU utilization as an admission gate.

Rationale sits in one operational sentence here.
If a restart cannot finish inside slack, reject it.

Concrete threshold for this declared drill:

• Reject enqueue when queue age exceeds half the deadline
• Drain the worker when restart tokens exceed 20 percent
• Abandon free-server mode after two recorded preempts
• Fail the task instead of starting a third full replay

Queue age beats utilization for this decision.
Slack is the scarce resource, not GPU percent.
Two preempts already imply an unstable slot.
Write that comparison beside the alert in git.

Artifact: admission plus a preempt loop

Save the next block as admit.py locally.
It is a simulator, not a vendor client SDK.

#!/usr/bin/env python3
"""Local admission drill for free-server preempt waste."""
from __future__ import annotations

import json
import os
import time
from dataclasses import dataclass, field
from pathlib import Path

LEDGER = Path(os.environ.get("LEDGER_PATH", "/tmp/preempt_ledger.jsonl"))
DEADLINE_MS = int(os.environ.get("DEADLINE_MS", "120000"))
TASK_BUDGET = int(os.environ.get("TASK_BUDGET", "1000"))
QUEUE_AGE_FRAC = float(os.environ.get("QUEUE_AGE_FRAC", "0.5"))
RESTART_FRAC = float(os.environ.get("RESTART_FRAC", "0.2"))
MAX_PREEMPTS = int(os.environ.get("MAX_PREEMPTS", "2"))


@dataclass
class Task:
    task_id: str
    prompt_tokens: int
    output_tokens: int = 0
    restart_tokens: int = 0
    preempts: int = 0
    enqueued_ms: int = field(default_factory=lambda: int(time.time() * 1000))
    checkpoint_tokens: int = 0


def now_ms() -> int:
    return int(time.time() * 1000)


def queue_age_ms(task: Task) -> int:
    return now_ms() - task.enqueued_ms


def deadline_slack_ms(task: Task) -> int:
    return DEADLINE_MS - queue_age_ms(task)


def emit(event: dict) -> None:
    event["ts_ms"] = now_ms()
    LEDGER.parent.mkdir(parents=True, exist_ok=True)
    with LEDGER.open("a", encoding="utf-8") as handle:
        handle.write(json.dumps(event) + "\n")
    print(json.dumps(event), flush=True)


def admit(task: Task) -> str:
    slack = deadline_slack_ms(task)
    age = queue_age_ms(task)
    if slack <= 0:
        emit({
            "event": "reject",
            "reason": "slack_zero",
            "task_id": task.task_id,
            "queue_age_ms": age,
            "deadline_slack_ms": slack,
        })
        return "reject_slack_zero"
    if age > QUEUE_AGE_FRAC * DEADLINE_MS:
        emit({
            "event": "reject",
            "reason": "queue_age",
            "task_id": task.task_id,
            "queue_age_ms": age,
            "deadline_slack_ms": slack,
        })
        return "reject_queue_age"
    if task.preempts >= MAX_PREEMPTS:
        emit({
            "event": "reject",
            "reason": "preempt_budget",
            "task_id": task.task_id,
            "preempts": task.preempts,
            "restart_tokens": task.restart_tokens,
        })
        return "reject_preempt_budget"
    if task.restart_tokens > RESTART_FRAC * TASK_BUDGET:
        emit({
            "event": "reject",
            "reason": "restart_tokens",
            "task_id": task.task_id,
            "restart_tokens": task.restart_tokens,
            "task_budget": TASK_BUDGET,
        })
        return "reject_restart_tokens"
    emit({
        "event": "admit",
        "task_id": task.task_id,
        "queue_age_ms": age,
        "deadline_slack_ms": slack,
        "preempts": task.preempts,
    })
    return "admit"


def on_preempt(task: Task) -> None:
    wasted = task.prompt_tokens + task.output_tokens - task.checkpoint_tokens
    task.restart_tokens += max(wasted, 0)
    task.preempts += 1
    task.output_tokens = task.checkpoint_tokens
    emit({
        "event": "preempt",
        "task_id": task.task_id,
        "wasted_tokens": wasted,
        "restart_tokens": task.restart_tokens,
        "preempts": task.preempts,
        "deadline_slack_ms": deadline_slack_ms(task),
        "gpu_util_lie": 0.04,
    })


def decode_tick(task: Task) -> None:
    task.output_tokens += 20
    if task.output_tokens % 100 == 0:
        task.checkpoint_tokens = task.output_tokens
        emit({
            "event": "checkpoint",
            "task_id": task.task_id,
            "checkpoint_tokens": task.checkpoint_tokens,
        })


def main() -> None:
    task = Task(task_id="batch-0214", prompt_tokens=800)
    decision = admit(task)
    if not decision.startswith("admit"):
        return
    ticks = 0
    while task.output_tokens < 200:
        if os.environ.get("SIMULATE_PREEMPT") == "1" and ticks == 2:
            on_preempt(task)
            decision = admit(task)
            if not decision.startswith("admit"):
                emit({
                    "event": "drain",
                    "task_id": task.task_id,
                    "action": "fail_task_not_replay",
                })
                return
        decode_tick(task)
        ticks += 1
        time.sleep(0.05)
    emit({
        "event": "complete",
        "task_id": task.task_id,
        "output_tokens": task.output_tokens,
        "restart_tokens": task.restart_tokens,
    })


if __name__ == "__main__":
    main()

Run the happy path first without a preempt.

export LEDGER_PATH=/tmp/preempt_ledger.jsonl
rm -f "$LEDGER_PATH"
DEADLINE_MS=120000 TASK_BUDGET=1000 python3 admit.py

Expected output, labeled as lab output only:

{"event": "admit", "task_id": "batch-0214", "queue_age_ms": 0, "deadline_slack_ms": 120000, "preempts": 0}
{"event": "checkpoint", "task_id": "batch-0214", "checkpoint_tokens": 100}
{"event": "complete", "task_id": "batch-0214", "output_tokens": 200, "restart_tokens": 0}

Now inject the preempt with the environment flag.

SIMULATE_PREEMPT=1 DEADLINE_MS=120000 TASK_BUDGET=1000 python3 admit.py

Expected preempt path from the same script:

{"event": "preempt", "task_id": "batch-0214", "wasted_tokens": 840, "restart_tokens": 840, "preempts": 1, "gpu_util_lie": 0.04}
{"event": "reject", "reason": "restart_tokens", "task_id": "batch-0214", "restart_tokens": 840, "task_budget": 1000}
{"event": "drain", "task_id": "batch-0214", "action": "fail_task_not_replay"}

Read the ledger after the injected run completes.

python3 - <<'PY'
import json
from pathlib import Path
rows = [json.loads(line) for line in Path("/tmp/preempt_ledger.jsonl").read_text().splitlines()]
for row in rows:
    print(
        f"{row['event']:12} "
        f"slack={row.get('deadline_slack_ms')} "
        f"restarts={row.get('restart_tokens')}"
    )
PY

Compute slack from the ledger, not dashboards

GPU graphs lag behind the preempt by minutes.
The ledger is the source for the reject call.
Parse the last event for each task_id value.

# parse_slack.py — lab helper, declared expected fields only
import json
from pathlib import Path

last = {}
for line in Path("/tmp/preempt_ledger.jsonl").read_text().splitlines():
    row = json.loads(line)
    last[row["task_id"]] = row

for task_id, row in last.items():
    action = "reject" if row["event"] in {"reject", "drain"} else "hold"
    print(task_id, action, row.get("reason"), row.get("restart_tokens"))

If restart_tokens already exceeds 20 percent, stop.
A third replay cannot buy back lost slack.
Fail that task and keep the remaining queue moving.

Failure injection beyond the environment flag

Kill a worker only on hosts you already operate.
Do not point this drill at unknown remote machines.

# Lab only: freeze then kill a local worker pid.
WORKER_PID=$(pgrep -f admit.py | head -n 1)
kill -STOP "$WORKER_PID"
sleep 2
kill -KILL "$WORKER_PID"

Watch these telemetry fields during the reject call:

• queue_age_ms as the wait you cannot hide
• deadline_slack_ms as the budget that pages you
• restart_tokens as waste from replayed prompts
• preempts as the stability vote for that slot
• checkpoint_tokens as the only reusable output
• gpu_util_lie as the metric you must ignore

Ignore host CPU when making the reject call.
CPU and GPU idle recover after an eviction.
Deadline slack will not recover after that.

Checkpoint cadence versus restart waste

Checkpoint every 100 output tokens in this lab.
That cadence is a declared test condition only.
A longer gap raises wasted tokens on each preempt.
A shorter gap raises fsync cost and log volume.

You pick the gap from slack, not from comfort.
If slack is thirty seconds, checkpoint more often.
If slack is fifteen minutes, checkpoint less often.
Measure wasted tokens after a forced worker kill.

Prompt-heavy tasks lose more on every restart.
An 800 token prompt plus 40 decoded tokens wastes 840.
That already exceeds a 20 percent restart budget.
Fail the task. Do not replay the same prompt.

Alerting that names the next action

Wire the reject to an alert that names the action.
A graph without an action will not drain the queue.

# lab_alerts.yaml — local threshold, not a vendor rule
groups:
  - name: free_server_preempt
    rules:
      - alert: RestartTokensBeatSlack
        expr: restart_tokens > 0.2 * task_budget
        for: 0s
        labels:
          action: drain_and_reject
        annotations:
          summary: reject replay; slack cannot absorb another preempt
      - alert: QueueAgeHalfDeadline
        expr: queue_age_ms > 0.5 * deadline_ms
        labels:
          action: reject_enqueue

The alert must tell you to reject, not to scale.
Scaling onto another free host copies the preempt.
You then own two unstable queues, not one.

When free capacity is the wrong bet

Free model access looks cheap on the invoice line.
Free servers look idle on the utilization graph.
Both can still lose a deadline with one preempt.

Do not bet free capacity in these cases:

1. The page has a sub-minute latency SLO attached.
2. The task cannot checkpoint any partial output.
3. Prompt tokens dwarf the remaining output tokens.
4. You already recorded two preempts on that host.
5. Queue age already consumed half of the deadline.

Paid burst can beat a third full prompt replay.
Time is the budget that actually pages on-call.
Tokens are the budget that silently compounds overnight.
Replays without checkpoints multiply both budgets together here.

You may need a scratch host for this drain drill.
MonkeyCode offers free model access and a free server option.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
That option can host the same admission loop.
You still avoid opening a paid burst for the lab.
Keep the ledger either way during the run.
Free slots still vanish under a preempt.

Rollback and cleanup after the reject

Fail the in-flight task. Do not replay it.
Flip admission to reject_preempt_budget immediately.
Drain the queue with a short local script.

# Stop enqueue. Drain. Then tear the lab down.
export ADMIT_MODE=reject_preempt_budget
sleep 5
rm -f /tmp/preempt_ledger.jsonl
unset SIMULATE_PREEMPT ADMIT_MODE DEADLINE_MS TASK_BUDGET

Rollback order matters under page pressure:

1. Reject new work before touching any worker.
2. Checkpoint or fail in-flight work second.
3. Kill workers only after the drain flag.
4. Restore the old threshold from version control.

Do not scale out onto another free host yet.
That copies the preempt onto a second waiting queue.

Who should not use this approach

Skip this drill if you lack a per-task ledger.
Skip it if your model host is already reserved.
Skip it for interactive chat with tight p95 limits.
Skip it if policy forbids local token ledger files.

This method will not size a production cluster.
It will not prove a vendor capacity SLA.
It only answers one operational question tonight.
Can this task finish after one more preempt?

Limits you should write on the runbook

The simulator does not talk to a live GPU.
Token counts are declared, not billed invoices.
SIGKILL is not the only real failure mode.
Disk full and NIC loss need separate later drills.
Do not treat expected JSON as a capacity benchmark.

Revisit the half-deadline queue-age fraction quarterly.
Tight SLOs need a smaller age fraction written down.
Long batch windows can tolerate more queue age.
Write the rationale next to the threshold in git.

You now have a reject path, not a hope path.
Idle GPU is not permission to enqueue more work.
Restart tokens are the signal that pages you later.

If you repeat this drain, send ledger pairs.
Paste queue_age_ms versus restart_tokens from the ledger.
Do not paste a screenshot of idle GPU.