Shed Stale Compaction Before Queue Age Misses the Turn

You catch the latency page at 02:14 UTC.
Chat p95 sits past the four-second turn budget.
The serving GPU is idle, so saturation is not the cause.

A background compaction job still occupies the free path.
Its transcript version is already behind the live turn.
The caller moved on six seconds before this page.

Free-path queue age reads eleven seconds in your collector.
Remaining slack on that turn is four seconds.
Another summary cannot land before the next token is due.

Read the contradiction first

Do not open this incident with a model swap.
The idle GPU and the late queue disagree.
Idle compute does not cancel a queued generation.

You pay tokens for work the turn will never read.
Separate what you measured from what you infer.
You measured queue age, slack, and version skew.

You infer that a zero price hid a deadline miss.
That inference is not a metric.
Keep it out of the alert name.

Name the topology

Keep this drill on one laptop you control.
You need four boxes and one shared clock.

• An API process owns the turn deadline.
• A compaction worker reads only a versioned transcript.
• A local queue can age on purpose.
• Two adapters exist: a free path and a paid path.

MonkeyCode states free model access and a free server option.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
Treat both as availability notes, not as latency promises.

Do not hard-code a model name from memory.
Read the live console before you bind a path.
Neither availability claim is a latency SLO you can page on.

Grants, hold times, and model lists change without your deploy.
A number copied from an old post is not a control.
If the console is silent, leave the free path dark.

Declare the workload

This is a local drill, not a production trace.
You inject queue age instead of sampling production.
Label every number below as declared drill input.

Use these conditions, and do not widen them mid-run.

• One compaction job per turn, never a fan-out.
• Versions increase only when the user sends text.
• Estimated generation time is a constant you pass in.
• Token estimate is characters divided by four, rounded up.
• Retry count starts at zero and stops at two.
• The token ceiling is a budget you choose.

Do not paste a vendor grant into this file.
A grant is not your ceiling unless you measured spend.
Your ceiling should be smaller than the turn can afford.

Gate the job

Admit compaction only when all three checks pass.
Reject stale work before you touch the queue.
Reject late work before you touch the free path.
Reject over-budget work before a retry multiplies tokens.

from dataclasses import dataclass

@dataclass(frozen=True)
class TurnBudget:
    slack_ms: int
    queue_age_ms: int
    est_gen_ms: int
    est_tokens: int
    retry_count: int
    token_ceiling: int
    transcript_version: int
    latest_version: int

def admit_compaction(b: TurnBudget) -> str:
    if b.transcript_version != b.latest_version:
        return "reject_stale"
    projected_ms = b.queue_age_ms + b.est_gen_ms
    if projected_ms >= b.slack_ms:
        return "reject_slack"
    spend = b.est_tokens * (b.retry_count + 1)
    if spend > b.token_ceiling:
        return "reject_tokens"
    return "admit"

The first check is a fact about versions.
The second check compares queue wait with slack.
The third check enforces a ceiling you set.

A free price does not skip any of these checks.
The paid path uses the same three checks.
Price changes the bill, not the deadline math.

Decision table

Signal you hold	Comparison	Action
Version skew	job version != latest	reject_stale
Time	queue age + est gen >= slack	reject_slack
Tokens	est tokens × (retries + 1) > ceiling	reject_tokens
All three clear	current, inside slack, under ceiling	admit

Queue age beats utilization as the shed signal.
A cool GPU can still sit behind an old queue.
Deadline slack is the budget that actually expires.

Run the drill

Save that function as admit.py in a scratch directory.
Then run the three cases declared in this section.

python - <<'PY'
from admit import TurnBudget, admit_compaction

cases = [
    TurnBudget(4000, 11000, 1500, 800, 0, 5000, 7, 9),
    TurnBudget(9000, 1200, 1500, 800, 2, 2000, 9, 9),
    TurnBudget(9000, 1200, 1500, 800, 0, 5000, 9, 9),
]
for case in cases:
    print(admit_compaction(case))
PY

The expected lines below come from this function.
They are not a vendor benchmark and not production.

reject_stale
reject_tokens
admit

Case one is stale, so age never matters.
Case two is current, but retries blow the ceiling.
Case three is current, inside slack, and under budget.

If your printout differs, stop and diff the function.
Do not tune thresholds until the labels match.
A mismatched label means the gate is not the one you reviewed.

What each input stands for

Case one mirrors the page you opened.
Slack is 4000 ms and queue age is 11000 ms.
Version 7 is behind version 9, so you shed first.

Case two is on time and on the current version.
Two retries turn 800 estimated tokens into 2400.
Your ceiling is 2000, so you stop before submit.

Case three is the only admit in this drill.
Queue age plus estimate is 2700 ms.
Slack is 9000 ms, and spend stays at 800.

Change one field and rerun before you trust a new threshold.
Write the new expected line next to the change.
If you cannot predict the label, you do not understand the gate.

When the free path is the wrong bet

Use the free path only for work that can expire.
Compaction can expire cleanly.
A user-visible answer cannot.

Shed the job when any line below is true.

• Queue age plus estimated generation meets slack.
• The transcript version moved after enqueue.
• Retries would push estimated tokens over your ceiling.
• You cannot cancel the generation after submit.
• The result must return inside this same turn.
• You have no fresh console reading for the free path.

A zero price is not a reason to wait.
Waiting spends wall time the caller already lost.
Retries then spend tokens on a summary nobody reads.

That is the cost failure, not a model quality debate.
You lost slack first.
You burned tokens second.

Do not failover an interactive turn onto the free server.
The free server option is still a queue with unknown age.
Unknown age is a reason to shed, not a reason to hope.

If hold time is unpublished, do not invent one.
If the token grant is unpublished, do not invent one.
Schedule only against numbers you just read.

Failure handling

On reject_stale, drop the job and ack the queue.
Do not requeue it under a new idempotency key.
A new key hides a duplicate spend in your bill.

On reject_slack, leave the last good summary in place.
Serve the current turn with the uncompacted window.
Page only if this reject rate crosses your own threshold.

Pick that threshold from missed turns, not from pride.
A starting page rule is five slack rejects in five minutes.
Replace it once you have a week of your own series.

On reject_tokens, stop retries for that version.
Emit compaction_rejected_total with the reason label attached.

Suggested fields, not a required vendor schema:

• queue_age_ms
• slack_ms
• est_tokens
• retry_count
• transcript_version
• reason

If you cannot cancel after submit, skip that path.
Uncancelable free work becomes silent token debt later.
Debt without a cancel handle is an ops bug.

A tiny collector check

After a reject, your log line should carry the reason.
Grep the scratch log before you call the drill done.

python - <<'PY'
print("reason=reject_slack queue_age_ms=11000 slack_ms=4000")
PY
grep -E 'reason=reject_(stale|slack|tokens)' /tmp/compaction-drill.log || true

The print is a format sample, not a caught incident.
The grep proves your eye can find the label.
If the label is missing, the gate is not operable.

Cleanup and rollback

The drill queue is local.
Drain it before you leave the shell.

# scratch only — do not point this at production
rm -f /tmp/compaction-drill.queue /tmp/compaction-drill.log
python -c "from admit import admit_compaction; print('rollback-ok')"

Roll the gate back with one config flag.

compaction:
  enabled: false
  free_path: disabled
  token_ceiling: 0
  max_retries: 0

Disabling the worker is the rollback, not a rewrite.
The chat path must keep serving without summaries.
If chat fails when compaction is off, fix that coupling first.

Re-enable only after three clean drill labels.
Then watch reject rate, queue age, and token spend together.
A green GPU graph alone is not a reopen signal.

Who should not use this gate

Skip this pattern if you lack a turn deadline.
Skip it when transcript versions are not monotonic.
Skip it if free capacity is your only generation path.

This gate sheds work.
It does not create capacity.
Teams that need every summary will see more full windows.

That is a product choice, not a bug in the check.
Do not use this drill as a capacity certificate.
One laptop queue does not prove a hosted server.

Do not publish these case numbers as latency results.
Do not treat a free server as a standby SLO.
Do not let a retry loop compact the same version twice.

What you should do next

Read queue age beside slack on the next late page.
If age already exceeds slack, reject the compaction job.
Confirm the live free-path limits in your console first.

If those limits are missing, keep the free path dark.
A missing hold time is not a zero hold time.
Point this gate at a MonkeyCode free path only after a fresh console check.