Test Free-Path Dwell Locally Before the Batch Cutoff

The page hits during a quiet overnight batch window. Your free worker still shows spare memory and idle cores. Queue age has already climbed to forty-one seconds.

The cutoff sits ninety seconds from the original enqueue. Two unused retries remain inside the job policy. Raising concurrency looks calm, and that reading is wrong.

Spare memory is not a promise of service time. That slot is shared, and nobody reserved it for you. Wait, service, and one retry can still miss the cutoff.

Read the contradiction first

Host utilization said the machine still had room. Queue age said the batch was already slipping. You need one figure that can reconcile both views.

Call that figure free-path dwell for this batch. Add queue age, expected service, and a retry tax. Compare the sum with remaining deadline slack first.

Admit on the free path only when dwell still fits. If dwell exceeds slack, free capacity is the wrong bet. Defer the job, or hand it to a reserved worker.

Do not buy another retry on a slot you do not own. A cheap token price cannot repay a missed cutoff. Wall-clock slack is the budget that actually expires.

Declare the drill before you touch a host

Write the conditions down before you start the process. This drill runs in a local process, not your cluster. Lines marked expected are illustrations, not your metrics.

Use these declared inputs for the first pass. Change them if your cutoff differs, and record the change. Do not tune them after you see a passing printout.

• Batch size is forty jobs, one attempt each at first.
• Each job estimates twelve hundred tokens of completion.
• Expected service on the free path is eight seconds.
• Deadline slack at enqueue starts at ninety seconds.
• Retry tax is one extra full service interval.
• Max queue age you will tolerate is thirty seconds.

The token count is your estimate, not a vendor quota. Do not treat it as a measured bill from any provider. You are testing admission math, not model quality.

Keep the topology on one page

Keep the pieces small so the failure stays local. One gate sits between enqueue and every worker choice. It may admit, defer, or shift to reserved.

enqueue -> dwell_gate -> free_path
                 |-> defer_queue
                 |-> reserved_path

The gate reads four fields and writes one decision. Log those fields on every attempt, including the rejects.

• queue_age_s records seconds since the job was enqueued.
• expected_service_s stores the service time you already declared.
• retry_tax_s prices one unused attempt in wall seconds.
• deadline_slack_s is cutoff time minus elapsed wall time.
• The decision field is admit_free, defer, or reserved.

Utilization stays on the dashboard, not inside the gate. It explains a hot host, not a late queue. Slack is the threshold, because the cutoff is absolute.

Queue age gets a veto, not the whole decision. A young queue can still lose if retry tax is fat. That split is the point of pricing dwell at all.

Run the gate on a laptop

Save the script below as dwell_gate.py on a laptop. Run it with Python 3, and keep it off the cluster. It does not call a model and does not open a port.

#!/usr/bin/env python3
# Local admission drill. Expected lines are comments, not your metrics.

from dataclasses import dataclass

@dataclass
class Job:
    queue_age_s: float
    expected_service_s: float
    retry_tax_s: float
    deadline_slack_s: float
    max_queue_age_s: float = 30.0

def dwell(job: Job) -> float:
    return job.queue_age_s + job.expected_service_s + job.retry_tax_s

def decide(job: Job) -> str:
    if job.queue_age_s > job.max_queue_age_s:
        return 'defer'
    if dwell(job) > job.deadline_slack_s:
        return 'reserved'
    return 'admit_free'

def decide_safe(job: Job) -> str:
    if job.expected_service_s <= 0 or job.deadline_slack_s <= 0:
        return 'defer'
    return decide(job)

samples = [
    Job(12, 8, 8, 90),   # expected admit_free
    Job(41, 8, 8, 90),   # expected defer, age cap
    Job(20, 8, 8, 30),   # expected reserved, dwell over slack
    Job(20, 8, 0, 30),   # expected admit_free, retry tax removed
    Job(10, 0, 8, 90),   # expected defer, non-positive service
    Job(10, 8, 8, -1),   # expected defer, negative slack
]

if __name__ == '__main__':
    for i, job in enumerate(samples, start=1):
        choice = decide_safe(job)
        print(
            'job=%s dwell=%.0f slack=%.0f decision=%s'
            % (i, dwell(job), job.deadline_slack_s, choice)
        )

Run the file with one local Python command. Stay in the directory where you saved the script. A missing file is a drill error, not a capacity signal.

python3 dwell_gate.py

Expected lines

Treat the next lines as expected output, not a measurement.

job=1 dwell=28 slack=90 decision=admit_free
job=2 dwell=57 slack=90 decision=defer
job=3 dwell=36 slack=30 decision=reserved
job=4 dwell=28 slack=30 decision=admit_free
job=5 dwell=18 slack=90 decision=defer
job=6 dwell=26 slack=-1 decision=defer

Job one fits, so the free path is an acceptable bet. Job two is young in slack but old in queue age. You defer it instead of hoping the shared slot clears.

Job three still has queue age under the cap. Dwell overruns slack, so you leave the free path. Job four drops the retry tax and fits again.

That last case is the trap inside a retry policy. A free retry looks cheaper than a reserved attempt. It is only cheaper when the extra seconds still fit.

Job five has a zero service guess, so the guard defers it. Job six has negative slack, so you do not admit it. Both guards fail closed, which is the point of the drill.

Use this table when the printout and the page disagree. The why column is the operational action, not a host guess.

queue_age_s	dwell_s	slack_s	decision	why
12	28	90	admit_free	dwell fits and age is young
41	57	90	defer	age cap fired before slack
20	36	30	reserved	age is fine, dwell is not
20	28	30	admit_free	retry tax removed, dwell fits
10	18	90	defer	service guess is not positive
10	26	-1	defer	cutoff already passed

Inject one boring failure

Add a stuck attempt before you trust the printed gate. Set queue age to fifty and slack to forty. Force retry tax to a full service interval of eight.

Dwell becomes sixty-six against forty seconds of slack. The age cap fires first, so the printed decision is defer. If the script still prints admit_free, stop the rollout.

A second injection zeros expected service by mistake. Dwell then under-counts, and the gate admits too early. Reject a non-positive service guess before you compare.

Negative slack means the cutoff has already passed. Defer, alert, and do not hide that miss inside a retry. Cleanup is deleting the local script and stopping the process.

rm -f ./dwell_gate.py

If a unit or cron copied the drill, disable that unit first. Then confirm the process table no longer shows the script. Leave production flags untouched until the canary section below.

Price tokens by the seconds they occupy

A free completion still spends wall time you cannot refund. Multiply estimated tokens by dwell seconds for a pressure gauge. That product is not an invoice and not a benchmark.

Twelve hundred tokens over sixty-six seconds is a heavy wait. The same tokens on an eight-second reserved turn weigh less. You are buying clock time, not only the completion text.

Count retries as extra token copies, not as free do-overs. Two attempts can double the token load and the dwell. Your ledger should store both, or the cost story splits.

If the model hop has its own queue, add that age too. A zero price does not make the second queue instant. Another wait can erase the slack you just protected.

Where a free path is allowed to help

Disclosure: This article was prepared as part of MonkeyCode's product outreach. Operator notes for this draft mention two availability claims. One is free model access, and one is a free server option.

Neither claim states a quota, a chip, or an expiry here. Confirm those facts in current product docs before you plan. A free path can still be the wrong bet for this batch.

Use it only when the dwell gate returns admit_free. A free server is useful for this local class of work. Use it to replay the gate, not to absorb a late queue.

If the option disappears tomorrow, your SLO must still hold. Do not place the fallback on that same free host. Reserved means a worker you can drain and roll back.

Shared free capacity cannot be that rollback target. Keep any longer trial on jobs you can defer safely. Point only slack-rich jobs at the free model path.

Leave cutoff work on the reserved path you already trust. Free access can host the drill, not the promise. Your cutoff still belongs to a worker you control.

Thresholds worth arguing about in review

Pick deadline slack as the primary admit threshold. The cutoff is a wall-clock fact, not a utilization gauge. Queue age is the leading indicator you watch beside it.

Utilization is third, and it should not cast a vote. A cold GPU can still sit behind someone else's queue. A hot GPU can still serve you if your dwell fits.

Start with these caps for the declared workload above. Treat them as review numbers, not as universal law.

• Defer the job when queue age exceeds thirty seconds.
• Leave the free path when dwell exceeds remaining slack.
• Price every unused retry as one full service interval.
• Refuse a service guess that is zero or negative.

Revisit the thirty-second cap after one real week of logs. If deferrals cluster near twenty seconds, tighten that cap. If reserved shifts dominate, your service guess is too timid.

Alert on the miss, not on the idle host

Page when an admit_free decision later records a slack miss. Do not page only because memory looks full or empty. The miss is the user-facing failure you set out to avoid.

Use the sketch below, and do not treat it as a live alert. Names must match your metrics before you paste anything.

decision=admit_free AND slack_miss=true

Route that alert to the same channel as queue-age pages. Attach dwell, slack, and retry count to the incident note. Without those three fields, the on-call will chase utilization again.

If misses exceed one percent of free admits, freeze the flag. That one percent is a starting bar for this drill only. Replace it with your error budget before production use.

Roll the gate back without draining the wrong queue

Ship the gate behind a flag, defaulting to off. Admit a canary slice of five percent of batch jobs only. Watch slack misses, defer rate, and reserved shift rate.

If the miss rate rises, turn the feature flag off. Drain jobs already admitted on the previous stable path. Do not flush the defer queue into the free worker.

Paste these steps into the runbook before the canary starts. Do not start the slice until the steps are in the ticket.

1. Disable the feature flag that is named dwell_gate_enabled.
2. Confirm new jobs skip the gate within one minute.
3. Let in-flight reserved jobs finish, then stop canary hosts.
4. Delete local drill files so a cron cannot revive them.

Rollback is complete when new decisions stop appearing in logs. Keep the reserved pool warm until that log line goes quiet. A half-disabled gate is worse than no gate at all.

Who should skip this approach

Skip it when the work cannot wait and cannot move. A pager path with no reserved fallback needs another design. This gate only chooses among admit, defer, and reserved.

Skip it for interactive leases that need sub-second slack. That failure mode needs a different hold-time control. Do not mix that control into this batch cutoff drill.

Skip it if you need a benchmark of model quality. The script never sends a prompt and never scores answers. It will not tell you which model is cheaper per token.

Skip it when service time is still an unknown distribution. A single guessed interval will lie under a long tail. Measure a p95 service time before you automate the shift.

What you should log before the next review

Keep one row per decision, not one row per host. Store queue age, dwell, slack, decision, and retry count. Drop raw prompts from this log so the cost file stays small.

Review three ratios when the week of logs ends. Keep the window fixed so the ratios stay comparable.

• Count free admits that later missed their slack budget.
• Count defers that would have fit on a reserved worker.
• Count reserved shifts whose dwell would have fit free.

The first ratio means the service guess was too kind. The second means you deferred work a reserved turn could save. The third means the age cap is stricter than the cutoff.

Adjust one cap at a time, then rerun the local samples. Changing age and slack rules together hides the cause. Write the new cap beside the old expected output lines.

Free capacity stays a tool, not a standing reservation. Use the free model path when dwell fits your slack. When it does not, pay for a worker you can actually drain.

If those options are available in your account, replay the samples. Run that replay on the free server before any batch attach. Check current docs first, because availability is not an SLO.