Reject Interactive Leases Before Free-Server Hold Time Beats Slack

02:14, two metrics disagree

Picture this on-call scene before you touch replicas.
Your pager fires during an otherwise quiet batch window.
The oldest summarizer job has waited 740 seconds already.

Your service objective still demands completion within 480 seconds.
The free pool reports only 19 percent CPU busy.
You almost add replicas to chase that age number.

That scale-out would hide the fault and waste money.
Do not scale on this metric contradiction just yet.
Ask which operational action the evidence actually supports.

Low utilization with rising age means stolen worker leases.
The scarce resource is hold time, not processor cycles.
Treat CPU as a contradiction check, not the page.

What you are actually running

You share one serving pool across two work classes.
Batch summarization carries a hard completion deadline tonight.
Interactive debug sessions carry no deadline of their own.

Use this proposed topology for the local drill.
Keep it on one box so blast radius stays tiny.

• The gateway handles admission decisions and lease tracking.
• The free-server process runs shared model serving locally.
• The batch-worker submits deadline-bound jobs with an 8-minute SLO.
• The debug-client holds notebooks, curl loops, and open streams.

Declare these test conditions before any fault injection.
Copy them into the drill script header before you run.

• Run one serving replica on the shared local pool.
• Fix maximum concurrent leases at two worker slots.
• Keep batch compute near 25 seconds for each job.
• Fix the batch deadline at 480 seconds wall clock.
• Let the interactive session hold one slot for 180 seconds.
• Keep interactive CPU load near five percent on purpose.
• Allow three retries with a 10-second backoff budget.
• Arrive six batch jobs within a 40-second window.

You will measure hold time, not CPU, as the scarce resource.
Utilization will lie while a debug client holds the slot.

Why free capacity is the wrong bet

Free model access looks cheap on a quiet dashboard.
A free server often looks idle under debug sessions.
Idle CPU does not mean idle workers on that box.

A lease occupies a slot until the client hangs up.
Interactive sessions hold connections far beyond useful compute.
Streaming responses stay open while you reread the prompt.

Retries clone the same hold without adding progress.
Queue age grows while the processor mostly sleeps.
You should treat hold time as the scarce resource now.

Queue age versus deadline slack beats utilization here.
Reject interactive work when slack cannot absorb another hold.
Do not wait for CPU to look busy first.

Time, tokens, retries, and queueing move together under a hold.
A cheap pool still spends finite tokens on duplicated retries.
Free capacity is the wrong bet when hold time stays unbounded.

Telemetry you need on the box

Export these fields from the gateway and server.
Without them you will page on vanity CPU again.

• lease_hold_seconds tracks how long a slot stays checked out.
• lease_class labels the holder as batch or interactive.
• queue_age_seconds records the oldest waiting batch job.
• deadline_slack_seconds equals deadline minus now minus remaining compute.
• retry_count captures debug storms and batch copies together.
• token_in and token_out stay finite even when unmarked free.
• cpu_busy_ratio exists only for contradiction checks on-call.

Keep these PromQL sketches labeled as unexecuted examples.
Do not paste them into production without a recording rule.

deadline_slack_seconds
  = batch_deadline_seconds
  - queue_age_seconds
  - expected_compute_seconds

reject_interactive
  = deadline_slack_seconds < 90
    and lease_hold_seconds{class="interactive"} > 60

If slack falls under 90 seconds, reject new interactive leases.
You should not wait for CPU saturation as confirmation.

Local fault-injection drill

Run this drill on your laptop before touching production.
The script occupies one lease with almost no CPU.
It then watches batch age while the slot stays held.

# lease_hold_drill.py — proposed local drill, not production telemetry
import json
import os
from collections import deque
from dataclasses import dataclass, field

BATCH_DEADLINE_S = 480
MAX_LEASES = 2
INTERACTIVE_HOLD_S = 180
BATCH_COMPUTE_S = 25
SLACK_REJECT_S = 90


@dataclass
class Lease:
    cls: str
    t0: float
    job_id: str


@dataclass
class Pool:
    leases: list = field(default_factory=list)
    batch_q: deque = field(default_factory=deque)
    log: list = field(default_factory=list)

    def slack(self, now):
        if not self.batch_q:
            return BATCH_DEADLINE_S
        age = now - self.batch_q[0][1]
        return BATCH_DEADLINE_S - age - BATCH_COMPUTE_S

    def admit(self, cls, job_id, now, reject_interactive):
        slack = self.slack(now)
        if cls == 'interactive' and reject_interactive and slack < SLACK_REJECT_S:
            self.log.append({
                't': now, 'event': 'reject_interactive',
                'job': job_id, 'slack': round(slack, 1),
            })
            return False
        if len(self.leases) >= MAX_LEASES:
            if cls == 'batch':
                self.batch_q.append((job_id, now))
                self.log.append({
                    't': now, 'event': 'enqueue_batch',
                    'job': job_id, 'q': len(self.batch_q),
                })
            else:
                self.log.append({
                    't': now, 'event': 'reject_interactive_full',
                    'job': job_id,
                })
            return False
        self.leases.append(Lease(cls, now, job_id))
        self.log.append({
            't': now, 'event': 'lease', 'cls': cls,
            'job': job_id, 'hold_slots': len(self.leases),
        })
        return True

    def release(self, job_id, now):
        self.leases = [lease for lease in self.leases if lease.job_id != job_id]
        while self.batch_q and len(self.leases) < MAX_LEASES:
            jid, enq = self.batch_q.popleft()
            age = now - enq
            self.leases.append(Lease('batch', now, jid))
            self.log.append({
                't': now, 'event': 'dequeue_batch', 'job': jid,
                'age': round(age, 1),
                'slack': round(BATCH_DEADLINE_S - age - BATCH_COMPUTE_S, 1),
            })


def run_drill(reject_interactive=True):
    pool = Pool()
    pending_batch = []
    arrivals = {5, 10, 15, 22, 30, 40}
    for t in range(0, 220):
        now = float(t)
        if t == 0:
            pool.admit('interactive', 'dbg-1', now, reject_interactive)
        if t == 8 and not reject_interactive:
            pool.admit('interactive', 'dbg-2', now, reject_interactive)
        if t in arrivals:
            ok = pool.admit('batch', f'b-{t}', now, reject_interactive)
            if ok:
                pending_batch.append((f'b-{t}', now + BATCH_COMPUTE_S))
        finished = [job for job, end in pending_batch if end <= now]
        pending_batch = [(job, end) for job, end in pending_batch if end > now]
        for job in finished:
            pool.release(job, now)
        if t == INTERACTIVE_HOLD_S:
            pool.release('dbg-1', now)
        if t == INTERACTIVE_HOLD_S + 5 and not reject_interactive:
            pool.release('dbg-2', now)
    return pool.log


if __name__ == '__main__':
    mode = os.environ.get('DRILL_MODE', 'reject')
    log = run_drill(reject_interactive=(mode == 'reject'))
    print(json.dumps(log, indent=2))

Run both admission modes and compare queue age.
Save the logs so you can diff hold time later.

python3 lease_hold_drill.py > /tmp/reject.json
DRILL_MODE=admit python3 lease_hold_drill.py > /tmp/admit.json

python3 - <<'PY'
import json
for name in ('reject', 'admit'):
    rows = json.load(open('/tmp/%s.json' % name))
    ages = [row['age'] for row in rows if row.get('event') == 'dequeue_batch']
    rejects = sum(1 for row in rows if row['event'].startswith('reject'))
    print(name, 'dequeued', len(ages), 'max_age', max(ages, default=0),
          'rejects', rejects)
PY

Expected output, clearly labeled unexecuted

On a quiet laptop the shape should look like this.
Treat the block as a target shape, not a benchmark claim.

reject dequeued 6 max_age 25.0 rejects 1
admit  dequeued 6 max_age 180.0 rejects 0

Your exact numbers will move with scheduler timing.
The shape should not move if the leak is real.
A 180-second hold on two slots starves batch slack.

Slack-based admission keeps max age near compute time.
The admit mode spends hold time, retries, and tokens together.
That is the cost story hiding under a calm CPU chart.

Decision table: keep free capacity or reject

Use this table during the 02:14 argument.
Pick one primary signal before you debate replicas.

Signal	Threshold	Action	Rationale
Interactive lease while slack is under 90s	90s slack	Reject debug	Deadline work owns the slots
CPU under 0.3 and queue age over 60s	60s age	Do not scale	This is a hold leak
Interactive retry count over 2	2 retries	Shed the client	Retries clone hold time
Tokens grow with no batch progress	Any growth	Kill the lease	Free tokens are still finite
Slack over 180s and a slot free	180s slack	Admit debug	Spare slack is the only window

Ask which threshold you would defend at 02:14.
Choose among queue age, utilization, and deadline slack.
Pick deadline slack as the primary reject signal.

Utilization lags and flatters a quietly held worker.
Queue age without remaining compute still lies to you.
Slack folds remaining compute into the same reject line.

Where a free isolated box still helps

You still need a place to run the drill.
Production serving is the wrong blast radius tonight.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.
MonkeyCode provides free model access and a free server option.
Use that isolation for the lease drill described above.

Do not park deadline-bound production batch on that pool.
A free shared pool is a lab, not spare production capacity.
Treat it as a blast-radius box you can throw away.

Inject holds, watch slack, then destroy the box.
If you already isolate drills on a free server, keep both signals.
Place deadline_slack_seconds beside lease_hold_seconds on that board.

Failure handling

Inject these faults on the laptop only.
Stop if any step escapes the local process namespace.

1. Hold one streaming client open for 180 seconds.
2. Retry the same debug prompt three times on timeout.
3. Start six batch jobs during that interactive hold.
4. Watch deadline slack cross the 90-second line.
5. Confirm new interactive leases return 429 with reason.

Use this proposed gateway body for the reject path.
Keep the reason field stable so clients can fail closed.

{
  "error": "interactive_lease_rejected",
  "reason": "deadline_slack_seconds below 90",
  "queue_age_seconds": 410,
  "retry_after_seconds": 120
}

Do not convert that 429 into a client retry loop.
Debug clients must fail closed on admission rejects.
Batch clients may retry with jitter and a budget.

Rollback and cleanup

Tear the drill down before you walk away.
Leftover leases are how tomorrow's 02:14 page is born.

kill %1 2>/dev/null || true
curl -sf -X POST localhost:8080/admin/drain || true
rm -f /tmp/reject.json /tmp/admit.json

Rollback if admission shipped wider than you intended.
Keep metrics hot even while the reject flag is off.

1. Flip the reject_interactive feature flag to off.
2. Keep emitting slack metrics so you are not blind.
3. Cap interactive leases at one until you re-enable.
4. Page on slack, never on CPU busy ratio alone.

Leave the pool empty after the last batch drain.
Do not leave the notebook holding a production-like slot.

Who should not use this approach

Skip this pattern if you have dedicated debug hardware.
Skip it when every job already has a hard lease timeout.
Skip it if you do not hold a batch deadline at all.

Do not mix paid production traffic onto a free shared server.
Noisy neighbors and preemptible capacity will break the story.
Free capacity is the wrong bet under unbounded hold time.

Limitations

The script is a discrete-time toy, not a load test.
It does not model tokenizer stalls or network jitter.
It does not name models, quotas, or hardware SKUs.

Your real threshold will differ from ninety seconds.
Treat 90 seconds as a starting argument, not law.
Measure expected_compute_seconds from traces, never from hope.

Token counters still matter when the invoice reads zero.
Retries spend both tokens and hold time together.
Count both before you call the pool cheap.

What you do at 02:14

You do not scale the quiet pool under this alert.
You kill the debug lease that stole the slot.
You reject new interactive work until slack recovers.

File hold time as the cause, not low CPU.
Rerun the local drill after the reject flag stays on.
Admit debug sessions only when slack is honestly large.

Which page threshold will you defend tomorrow morning?
Will it be slack, age, or utilization under lease contention?