Reject Shared-Host Burst Packing Before Swap-In Beats Token Budget

At 04:12 UTC your pager fires on inference p99.
Queue age still looks green on the dashboard.
Host CPU idle still sits near forty percent.

You now hold a contradiction, not spare capacity.
Which operational action follows from that evidence?
You do not add replicas from that idle chart.

The scene you actually walked into

A batch packer stuffed eight long contexts onto one box.
That box was a shared free host with no isolation.
The model process stayed runnable while pages left RAM.

p99 jumped while utilization stayed flattering and quiet.
Tokens still completed, just after the deadline window.
Cost looked free. Tail latency was not free.

You are operating deployed inference, not designing raft.
Measure reclaim, then admit work, then spend tokens.
Free capacity is a bet with a hard stop condition.

Topology under test

Keep the drill on one host you can destroy.
Do not point this at a customer-facing endpoint.
Label every number below as a lab expectation.

Proposed lab topology:

• One shared host running a single inference worker
• One Redis list as the admission and drain queue
• One sidecar that samples PSI, swap, and queue age
• One packer that can emit oversized context jobs
• cgroup memory max set below the unpacked working set

[packer] -> redis:jobs -> [admitd] -> [worker]
                              |
                              +-> [sidecar: psi, si, queue_age_ms]
worker cgroup: memory.max = 2G, memory.swap.max = 0

Declared workload, not a marketing benchmark:

• Job type: offline summarization with a hard deadline
• Context: 8 files, about 24k tokens of prompt text
• Concurrency: 8 packed jobs on one worker process
• Deadline: 8 seconds from enqueue to first useful token
• Host: 4 vCPU, 2G memory.max, swap disabled in cgroup

If your box differs, rewrite the thresholds, not the story.
Do not copy these numbers onto a production runbook.

Signals that beat a green CPU chart

CPU idle cannot see anonymous pages leaving the working set.
Queue age cannot see a runnable worker waiting on reclaim.
You need host pressure fields beside the job counters.

Collect at least these telemetry fields every second:

• queue_age_ms from Redis LINDEX plus enqueue time
• deadline_slack_ms as deadline minus now minus age
• psi_mem_some_avg10 from /proc/pressure/memory
• si from vmstat 1 as swap-in pages per second
• cg_mem_current from memory.current in the cgroup
• tokens_out and retry_count from the worker log

# labeled lab commands, not production scrape config
while true; do
  date --iso-8601=seconds
  awk '/some/ {print "psi_mem_some", $0}' /proc/pressure/memory
  grep -E '^(si|so)' <(vmstat 1 2 | tail -1 | awk '{print "si", $7, "so", $8}')
  cat /sys/fs/cgroup/infer/memory.current
  redis-cli LPUSH_SENTINEL >/dev/null 2>&1 || redis-cli LLEN jobs
  sleep 1
done

Expected lab pattern, clearly labeled, not observed prod:

• CPU idle stays above thirty percent during the pack
• psi_mem_some_avg10 climbs above 15.00 within 20s
• queue_age_ms stays under 500 while p99 crosses 8s
• tokens_out still increases, so the queue looks healthy
• deadline_slack_ms goes negative before retries start

That is the contradiction you page on.
Idle CPU plus shrinking slack means reclaim, not underload.

Local fault injection before any shared free host

Inject packing on localhost first.
Do not discover swap-in during a customer batch window.
Keep memory.swap.max at zero so the failure is loud.

# create a tight cgroup; rollback later with cgdelete
sudo mkdir -p /sys/fs/cgroup/infer
echo "+memory +cpu" | sudo tee /sys/fs/cgroup/cgroup.subtree_control
echo 2147483648 | sudo tee /sys/fs/cgroup/infer/memory.max
echo 0 | sudo tee /sys/fs/cgroup/infer/memory.swap.max

# start a fake worker that touches a 3G anonymous map
echo $$ | sudo tee /sys/fs/cgroup/infer/cgroup.procs
python3 - <<'PY'
import time, mmap, os
print("pid", os.getpid(), flush=True)
buf = mmap.mmap(-1, 3 * 1024 * 1024 * 1024)
for off in range(0, len(buf), 4096):
    buf[off] = 1
print("packed", flush=True)
time.sleep(30)
PY

Watch the sidecar while that map fills.
You should see PSI rise before the process is killed.
If the kernel OOM-kills the worker, admission was already late.

Failure handling you want in the worker, not in chat:

1. Read deadline_slack_ms before loading the next context.
2. Reject the job when slack is under 1500 ms.
3. Reject the job when psi_mem_some_avg10 exceeds 10.00.
4. Nack with reason=host_pressure, never a blind retry.
5. Cap in-flight packed tokens, not just in-flight jobs.

# proposed admitd gate; unexecuted example, not a benchmark
PSI_LIMIT = 10.0
SLACK_LIMIT_MS = 1500
PACKED_TOKEN_LIMIT = 32000

def admit(job, psi_mem_some, slack_ms, packed_tokens):
    if psi_mem_some > PSI_LIMIT:
        return "reject_host_pressure"
    if slack_ms < SLACK_LIMIT_MS:
        return "reject_deadline_slack"
    if packed_tokens + job.prompt_tokens > PACKED_TOKEN_LIMIT:
        return "reject_packed_tokens"
    return "admit"

Retries belong on a different host class.
A nack into the same packed worker repeats the reclaim.
That is how free capacity burns wall clock twice.

When free capacity is the wrong bet

Ask for one operational threshold, with a rationale.
Do not mix queue age, utilization, and slack as equals.
They fail in different orders on a shared host.

Signal	Cheap on a free shared host?	Reject when	Why that beats CPU idle
CPU idle	Often yes	Never use alone	Misses reclaim waits
Queue age	Until packing starts	Age > 2s with slack < 1.5s	Age can lag runnable stalls
PSI memory some	No, shared noisy neighbors	avg10 > 10.00	Shows stall time, not idle
Swap-in si	No	si > 0 with swap disabled	Means the cgroup leaked
Packed prompt tokens	No	In-flight > 32k on 2G	Working set, not job count
Token price	Yes, until retries	Retry_count > 0 after nack	Retries double wall clock

Use queue age to see waiting work.
Use PSI and packed tokens to see host steal.
Use deadline slack to decide reject versus run.

Free hosts win for rehearsal, soak, and dead-letter replay.
They lose when p99 is the product, not the invoice.
They also lose when neighbors can allocate anonymous maps.

You should refuse the free-host bet when:

• The job has a user-facing first-token deadline
• The prompt working set can exceed memory.max
• You cannot disable swap inside the worker cgroup
• Retries reenter the same packed process
• You lack PSI and si on the same scrape interval

Token, time, and retry math you can audit

Do not price a packed job by input tokens alone.
Wall clock during reclaim still holds the worker slot.
A retry after a late nack spends tokens without slack.

cost_proxy = prompt_tokens
            + completion_tokens
            + retry_count * prompt_tokens
            + stall_ms * workers_held / 1000

reject if deadline_slack_ms < stall_ms_p99 + 500

stall_ms_p99 comes from PSI, not from the tokenizer.
If you cannot measure stall, you cannot spend free capacity.
That is an ops gate, not a model-quality debate.

Run a one-hour drain math check before any cutover:

# unexecuted checklist; fill from your sidecar, not guesses
redis-cli LLEN jobs
redis-cli LRANGE jobs 0 0   # inspect enqueue_ts on the head
journalctl -u admitd --since "-10 min" | grep reject_

Expected output shape from admitd during a pack storm:

reject_packed_tokens job=7 packed=36012 limit=32000
reject_host_pressure job=8 psi=12.40
reject_deadline_slack job=9 slack_ms=420

If you instead see retry worker_idle=1, the gate failed.
Idle workers can still be stuck in reclaim.
Believe PSI over worker idle every time.

Rollback and cleanup

Leave the host quieter than you found it.
A forgotten 3G map is the next person's incident.
Roll back admission to deny-all if slack stays negative.

# drain, then unpin, then delete the cgroup
redis-cli PAUSE jobs || redis-cli RENAME jobs jobs_drain
sudo kill -TERM "$(pgrep -f packed)" || true
sleep 2
echo 0 | sudo tee /sys/fs/cgroup/infer/cgroup.procs || true
sudo rmdir /sys/fs/cgroup/infer
redis-cli LLEN jobs_drain

Rollback path for the serving path:

1. Flip admitd to reject_all with reason host_pressure.
2. Stop the packer. Do not stop Redis until drain completes.
3. Move the drain list to a dedicated host class.
4. Re-enable admit only after PSI avg10 stays under 1.00.
5. Keep swap disabled until the next declared drill.

If rmdir fails, a task is still in the cgroup.
Find it with cat cgroup.procs before you walk away.
Do not reboot to hide a packing bug.

Limitations, and who should skip this

This drill assumes you can set cgroup memory.max.
It assumes you can read PSI on the same host.
It assumes jobs carry a real deadline, not best effort.

Skip this approach when:

• You run fully managed inference with no host scrapes
• Your batch has no deadline and no token budget
• You cannot fail closed without losing required work
• You need multi-region consensus design, not host ops

Do not treat lab PSI numbers as a vendor benchmark.
Do not keep swap disabled if your platform requires it.
Do not run the 3G map on a laptop you cannot lock.

The useful rule is narrow.
Reject packed work before reclaim steals the token budget.
Green CPU on a shared free host is not an SLO.

A rehearsal slot, not a production bet

You can rehearse admitd against a scratch model endpoint.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
MonkeyCode offers free model access and a free server option, which is enough to run this packing drill off the production path.

Keep customer deadlines off that shared box.
Use it to prove reject reasons, then throw the cgroup away.
If the sidecar cannot see PSI, do not promote the worker.