Reject Tool-Call Fanout Before Token Burn Beats Slack

You get paged at 02:14 UTC tonight.
Queue age already sits at 47 seconds.
Interactive CPU still looks almost idle.

You open the token meter next.
Spend is climbing at 3x the planned rate.
Deadline slack on the batch window is 90 seconds.

Which operational action follows that evidence?
Scale the free worker, or reject the fanout class?
This note argues for reject, then prove it locally.

The contradiction you should not ignore

Queue age is high. Utilization is low.
That pairing is not a capacity shortage.
It is usually a work-shape problem under metering.

Tool-calling jobs hide nested HTTP work.
Each nested call can enqueue sibling calls.
Retries then multiply tokens without moving the head job.

Free capacity makes that shape look cheap.
Wall-clock slack still expires on the original deadline.
Tokens still leave the shared pool either way.

Topology for the local drill

Keep the lab topology tiny and explicit.

• admitd: admission proxy with class rules
• workq: FIFO queue with age gauges
• worker: one process, one in-flight job
• meter: token and retry counters
• deadline: remaining slack per job class

Do not share this worker with interactive traffic.
Label every number below as a lab expectation.
Do not treat it as a production observation.

Proposed config, saved as admitd.yaml:

classes:
  tool_fanout:
    max_depth: 1
    max_siblings: 2
    reject_when:
      queue_age_ms: 15000
      slack_ms_below: 60000
      tokens_per_success_est: 8000
  interactive:
    max_depth: 0
    reserved: true
queue:
  visibility_timeout_s: 30
  max_inflight: 1
meter:
  window_s: 60
rollback:
  drain_timeout_s: 45
  fail_closed: true

Declared lab workload:

1. 20 parent jobs, each with two tool calls.
2. Each tool call may spawn one retry on 429.
3. Parent deadline is 120 seconds after enqueue.
4. Worker concurrency stays at one.
5. No other publishers touch workq.

Start from the alert, not the model card

You should bind alerts to evidence fields.
Do not page on “tokens used” alone.
Page on slack versus age versus burn together.

Proposed telemetry fields:

• queue_age_ms
• deadline_slack_ms
• tokens_in_window
• retries_in_window
• tool_depth
• class
• admit_decision

Expected lab log line, clearly labeled expected:

# expected lab output, not a live incident
ts=02:14:07 class=tool_fanout queue_age_ms=47012
deadline_slack_ms=90000 tokens_in_window=24110
retries_in_window=14 tool_depth=2 admit_decision=reject
reason=token_burn_vs_slack

If CPU is idle while that line prints, you do not scale.
You reject the class that fans out.
Then you drain what already entered.

Artifact: a fail-closed admission check

The checker below is a local, unexecuted example.
Run it against recorded metrics, not guesswork.
Keep it fail-closed when slack is unknown.

from dataclasses import dataclass

@dataclass
class Sample:
    queue_age_ms: int
    deadline_slack_ms: int
    tokens_in_window: int
    successes_in_window: int
    tool_depth: int
    retries_in_window: int

class Reject(Exception):
    pass


def tokens_per_success(sample: Sample) -> float:
    if sample.successes_in_window <= 0:
        return float("inf")
    return sample.tokens_in_window / sample.successes_in_window


def admit_tool_fanout(sample: Sample, cfg: dict) -> str:
    rules = cfg["classes"]["tool_fanout"]["reject_when"]
    if sample.deadline_slack_ms < rules["slack_ms_below"]:
        raise Reject("slack_below_floor")
    if sample.queue_age_ms > rules["queue_age_ms"]:
        raise Reject("queue_age_over_limit")
    if sample.tool_depth > cfg["classes"]["tool_fanout"]["max_depth"]:
        raise Reject("tool_depth_over_limit")
    est = tokens_per_success(sample)
    if est > rules["tokens_per_success_est"]:
        raise Reject("token_burn_vs_success")
    if sample.retries_in_window >= 10 and sample.successes_in_window == 0:
        raise Reject("retry_amplification")
    return "admit"

Wire a one-shot probe. Do not loop this in production yet.

python3 - <<'PY'
from admit_check import Sample, admit_tool_fanout, Reject
import yaml
cfg = yaml.safe_load(open("admitd.yaml"))
sample = Sample(
    queue_age_ms=47012,
    deadline_slack_ms=90000,
    tokens_in_window=24110,
    successes_in_window=1,
    tool_depth=2,
    retries_in_window=14,
)
try:
    print(admit_tool_fanout(sample, cfg))
except Reject as e:
    print("reject", e)
PY

Expected lab output:

reject tool_depth_over_limit

You now have a binary operational action.
Reject beats adding another free worker here.
The extra worker would only clone the fanout.

When free capacity is the wrong bet

Free model access can still be useful.
A free server option can isolate the drill.
Neither one repairs a bad work shape.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.
If you need an isolated box for the same admission drill, MonkeyCode's free model access and free server option can host the worker without mixing it into paid interactive traffic.

Use this decision table before you bet on free capacity.

Evidence	Free capacity bet	Operational action
Queue age high, CPU idle, depth=1	Maybe	Keep one worker, watch slack
Depth >= 2, retries rising	Wrong	Reject class, fail closed
Tokens/success unknown	Wrong	Reject until meter exists
Slack < 60s, age > 15s	Wrong	Drain, do not enqueue more
Interactive class shares the pool	Wrong	Split queues before any scale
Successes move, retries stay flat	Acceptable	Keep free worker, cap depth

Ask for one threshold and one rationale.
Prefer deadline slack over utilization.
Utilization lies when nested calls wait on I/O.

Rationale you can defend in a review:

• Queue age over 15s with one inflight job means head-of-line delay.
• Slack under 60s leaves no drain window.
• Tokens per success over 8k, in this lab, means nested waste.
• Retry count without successes means amplification, not load.

Change those numbers only after a declared rerun.
Do not copy them into production blindly.

Failure injection you can run locally

Inject one fault at a time.
Record age, slack, tokens, and retries after each.
Stop if slack crosses the floor.

# 1) nested fanout
python3 inject_fanout.py --parents 20 --depth 2 --siblings 2

# 2) retry storm on 429
python3 inject_fanout.py --parents 20 --depth 1 --force-429 8

# 3) slow tool, no extra tokens
python3 inject_fanout.py --parents 5 --depth 1 --tool-sleep-ms 4000

# 4) meter blackout
python3 inject_fanout.py --parents 5 --drop-meter

Expected lab pattern, labeled expected:

• Case 1: reject on tool_depth_over_limit.
• Case 2: reject on retry_amplification.
• Case 3: admit if age stays under 15s.
• Case 4: reject because tokens per success is infinite.

Case 3 is the trap.
Slow tools raise age without burning tokens.
You still reject if age beats slack, not if tokens look fine.

Drain, rollback, cleanup

Rejection without drain still burns the in-flight parent.
Practice the rollback on the lab queue.

# stop publishers for tool_fanout only
curl -X POST localhost:8080/classes/tool_fanout/reject

# wait for inflight, then snapshot meters
sleep 45
curl -s localhost:8080/metrics > /tmp/meter.after-reject

# fail closed if drain exceeds timeout
if [ "$(curl -s localhost:8080/queue/age_ms)" -gt 15000 ]; then
  curl -X POST localhost:8080/queue/purge?class=tool_fanout
fi

# restore only after slack recovers
curl -X POST localhost:8080/classes/tool_fanout/enable \
  --data 'max_depth=1&max_siblings=2'

Cleanup the lab files after the drill.

rm -f /tmp/meter.after-reject
rm -rf ./workq-data ./meter-wal

If you pointed the worker at a free server, tear that worker down too.
Do not leave a preemptible box holding visibility locks.
Stale locks look like queue age the next morning.

What this drill does not claim

This is an admission workflow, not a model benchmark.
It does not name models, quotas, or hardware SKUs.
It does not claim free capacity is faster or slower.

It also does not replace tracing of each tool HTTP call.
You still need span ids on nested calls.
Without spans, depth is only an inference.

Separate observed fields from architectural guesses.
queue_age_ms is observed.
“The model is slow” is not observed here.

Who should not use this approach

Skip this reject path if you lack a token meter.
Skip it if parents have unknown tool graphs.
Skip it for user-facing streaming with sub-second SLOs.

Do not use a shared free worker for mixed classes.
Interactive work will inherit batch fanout delays.
Paid isolation is cheaper than a missed deadline then.

Do not tune thresholds during an active incident.
Run the injectors on a laptop first.
Promote the rule only after two clean lab passes.

The action that follows the 02:14 evidence

You had high age, idle CPU, and rising tokens.
That is reject-and-drain, not scale-on-free.
Free capacity is the wrong bet once fanout starts.

Keep depth at one on cheap workers.
Keep a slack floor of 60 seconds in this lab.
Keep retries from counting as progress.

Then rerun the four injectors after every parser change.
Tool schemas change more often than queue code.
Your reject rule must track the schema, not the slogan.