Reject Model-Backed Probes Before Token Spend Beats Ready Time

You open Grafana at 09:12 after a quiet night.
Token counters climbed while request logs stayed empty.
Your cluster still looks healthy on the default probes.

Which operational action follows from that evidence?
Stop blaming the model and inspect the probe target first.

This is probe spend, not user demand

Kubernetes did its usual job with boring regularity.
A readiness probe hit your generate route all night.
Each check spent tokens and occupied a queue slot.

Free capacity makes this failure easy to miss.
The server looks idle on CPU and replica count.
The token ledger and queue age tell another story.

Lab topology you can run locally

Run one adapter process and a probe loop together.
Keep the model call behind an explicit complete route.
Point health checks at a cheap /readyz handler only.

• adapter: HTTP process, in-memory queue, token ledger
• probe: curl loop that mimics a kube httpGet
• ledger: JSON lines on disk for later jq diffs
• worker: fake completion with a hard timeout

Use this proposed Compose file for the isolated drill:

services:
  adapter:
    build: .
    ports: ["8080:8080"]
    environment:
      HOURLY_TOKEN_CAP: "50000"
      MAX_QUEUE_AGE_MS: "2000"
      COMPLETE_TIMEOUT_S: "30"
      ADMIT_PROBE_COMPLETES: "0"
      LEDGER_PATH: "/var/lib/adapter/ledger.jsonl"
    volumes:
      - ./ledger:/var/lib/adapter

Do not attach this stack to production DNS.
Do not share the queue with real user traffic.
Tear it down when the drill ends.

Where a free server still fits

Disclosure: This article was prepared as part of MonkeyCode's product outreach.
MonkeyCode is an open-source project with free model access.
A free server option is useful for this probe drill.

It is the wrong bet for SLO-bound interactive serving.
Separate probe tokens from user tokens on day one.
If those counters share one bucket, rollback stays blind.

Cost ops starts with attribution, not with extra quota.
Write the route split before you invite any users.

Declared workload

Treat the numbers below as lab conditions only.
They are not production benchmarks or vendor claims.

1. Idle sixty seconds with probes on /readyz only.
2. Switch the probe URL to /v1/complete for fifteen minutes.
3. Restore /readyz and drain non-user jobs first.
4. Diff token counters, queue age, and ready latency.

Declare these knobs before you start the loop:

• Probe interval: 10 seconds
• Probe timeout: 1 second
• Completion timeout: 30 seconds
• Packed prompt: 128 estimated tokens
• Concurrent user traffic: none
• Ready SLO for the drill: 200 milliseconds
• Hourly token cap in lab: 50000 (local knob only)

Cheap ready versus generate

Your ready check should answer three questions only.
Keep it inside the adapter process on purpose.
Remote dependency checks turn ready into generate.

• Is the HTTP process up and serving locally?
• Is queue age under a quarter of deadline slack?
• Is the hourly token ledger under its cap?

It must not forward a prompt into the model.
It must not wait on remote batching or GPUs.
It must not retry, because retries multiply probe waste.

Proposed adapter code for the local drill:

# lab-only adapter. Not a production service.
import json, os, time, threading
from http.server import BaseHTTPRequestHandler, HTTPServer

LEDGER = os.environ.get("LEDGER_PATH", "./ledger.jsonl")
CAP = int(os.environ.get("HOURLY_TOKEN_CAP", "50000"))
MAX_AGE = int(os.environ.get("MAX_QUEUE_AGE_MS", "2000"))
ADMIT_PROBES = os.environ.get("ADMIT_PROBE_COMPLETES", "0") == "1"
EST_TOKENS = 128

lock = threading.Lock()
queue = []  # {enqueued_ms, tokens, reason}
spent_hour = 0
hour_start = time.time()

def emit(route, **fields):
    rec = {"ts": time.time(), "route": route, **fields}
    with open(LEDGER, "a") as f:
        f.write(json.dumps(rec) + "\n")

def queue_age_ms():
    if not queue:
        return 0
    return int((time.time() * 1000) - queue[0]["enqueued_ms"])

def ready_ok():
    global spent_hour, hour_start
    now = time.time()
    if now - hour_start >= 3600:
        spent_hour = 0
        hour_start = now
    if queue_age_ms() > MAX_AGE:
        return False, "queue_age"
    if spent_hour >= CAP:
        return False, "token_cap"
    return True, "ok"

class Handler(BaseHTTPRequestHandler):
    def _send(self, code, body):
        raw = json.dumps(body).encode()
        self.send_response(code)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(raw)))
        self.end_headers()
        self.wfile.write(raw)

    def do_GET(self):
        t0 = time.time()
        if self.path.startswith("/readyz"):
            ok, reason = ready_ok()
            ready_ms = int((time.time() - t0) * 1000)
            slack = max(0, MAX_AGE - queue_age_ms())
            emit("readyz", queue_age_ms=queue_age_ms(),
                 queue_depth=len(queue), ready_ms=ready_ms,
                 deadline_slack_ms=slack, reason=reason,
                 est_tokens=0, actual_tokens=0)
            self._send(200 if ok else 503, {"ok": ok, "reason": reason})
            return
        # GET-on-generate is a common kube probe footgun.
        if self.path.startswith("/v1/complete"):
            self._complete(reason_header=True)
            return
        self._send(404, {"error": "not_found"})

    def do_POST(self):
        if not self.path.startswith("/v1/complete"):
            self._send(404, {"error": "not_found"})
            return
        self._complete(reason_header=True)

    def _complete(self, reason_header=False):
        global spent_hour
        reason = self.headers.get("X-Traffic-Reason", "user") if reason_header else "user"
        if reason == "probe" and not ADMIT_PROBES:
            emit("complete_reject", reason="probe_rejected",
                 queue_age_ms=queue_age_ms(), queue_depth=len(queue),
                 est_tokens=EST_TOKENS, actual_tokens=0,
                 ready_ms=0, deadline_slack_ms=max(0, MAX_AGE - queue_age_ms()))
            self._send(429, {"error": "probe_rejected"})
            return
        ok, why = ready_ok()
        if not ok:
            emit("complete_reject", reason=why, queue_age_ms=queue_age_ms(),
                 queue_depth=len(queue), est_tokens=EST_TOKENS, actual_tokens=0,
                 ready_ms=0, deadline_slack_ms=0)
            self._send(503, {"error": why})
            return
        job = {
            "enqueued_ms": time.time() * 1000,
            "tokens": EST_TOKENS,
            "reason": reason,
        }
        with lock:
            queue.append(job)
            spent_hour += EST_TOKENS
        # Fake a completion. Label this as unexecuted model I/O.
        time.sleep(0.05)
        with lock:
            if queue and queue[0] is job:
                queue.pop(0)
            elif job in queue:
                queue.remove(job)
        emit("complete", reason=reason, queue_age_ms=queue_age_ms(),
             queue_depth=len(queue), est_tokens=EST_TOKENS,
             actual_tokens=EST_TOKENS, ready_ms=0,
             deadline_slack_ms=max(0, MAX_AGE - queue_age_ms()))
        self._send(200, {"ok": True, "tokens": EST_TOKENS})

    def log_message(self, fmt, *args):
        return

if __name__ == "__main__":
    HTTPServer(("0.0.0.0", 8080), Handler).serve_forever()

Reject probe completes in the handler, not later in dashboards.
A dashboard alert cannot refund the tokens already spent.

Kubernetes probes: wrong versus right

Health checks are GET by default in Kubernetes.
Teams still burn tokens in two common ways.
They map generate onto GET, or they wrap curl in exec.

Wrong probe looks simple and still burns tokens.
Copy this only as a failure fixture in lab.

# expected failure config. Do not ship this.
readinessProbe:
  httpGet:
    path: /v1/complete
    port: 8080
    httpHeaders:
      - name: X-Traffic-Reason
        value: probe
  periodSeconds: 10
  timeoutSeconds: 1
  failureThreshold: 3
livenessProbe:
  exec:
    command: ["curl", "-sS", "-X", "POST", "http://127.0.0.1:8080/v1/complete"]
  periodSeconds: 10

Right probe checks admission health and nothing else.
Ship this shape if you must run under Kubernetes.

readinessProbe:
  httpGet:
    path: /readyz
    port: 8080
  periodSeconds: 10
  timeoutSeconds: 1
  failureThreshold: 3
livenessProbe:
  httpGet:
    path: /readyz
    port: 8080
  periodSeconds: 10
startupProbe:
  httpGet:
    path: /readyz
    port: 8080
  periodSeconds: 5
  failureThreshold: 12

Never let a liveness probe call generate.
A slow completion becomes a kube restart loop.
Restarts cold-start the queue and waste more tokens.

Telemetry fields you actually debug

Emit one JSON line per ready check and complete.
Name the fields the same in every process replica.
Mismatched keys make the jq drill lie to you.

• ts
• route
• queue_age_ms
• queue_depth
• est_tokens
• actual_tokens
• ready_ms
• deadline_slack_ms
• reason

Debug with queue age, not with CPU graphs.
CPU stays quiet because those probes stay sparse.
Queue age and token counters will move first.

# proposed local queries after the drill
jq 'select(.reason=="probe") | .actual_tokens' ledger.jsonl \
  | awk '{s+=$1} END {print s}'
jq 'select(.route=="readyz") | .ready_ms' ledger.jsonl \
  | awk '{s+=$1; n++} END {print s/n}'
jq 'select(.route=="complete_reject")' ledger.jsonl | wc -l

Expected lab output on the bad path

This block is labeled expected output, not a measurement.
You should reproduce the shape, not chase a vendor number.

{"route":"complete","reason":"probe","queue_age_ms":0,"est_tokens":128,"actual_tokens":128}
{"route":"complete","reason":"probe","queue_age_ms":40,"est_tokens":128,"actual_tokens":128}

Fifteen minutes at a ten second period is ninety probes.
Ninety times 128 estimated tokens is 11520 tokens spent.
Ready latency can still sit under 200 milliseconds.

That metric contradiction is the page you want.
Tokens moved while users and CPU stayed flat.
Free capacity looked idle while the ledger drained.

Call the next paragraph architectural inference, not observation.
Probe jobs steal the same admission slots as users.
When real traffic arrives, deadline slack is already thin.

Fault injection you can run today

Keep this drill on localhost and do not probe production.
Record every curl in the ledger before you change flags.

# proposed drill. Label results as lab-only.
python adapter.py &
ADAPTER_PID=$!
sleep 1

# Stage A: cheap ready for 60s
for i in $(seq 1 6); do
  curl -sS -o /dev/null -w "%{http_code}\n" http://127.0.0.1:8080/readyz
  sleep 10
done

# Stage B: restart with probe completes admitted
kill "$ADAPTER_PID"
ADMIT_PROBE_COMPLETES=1 python adapter.py &
ADAPTER_PID=$!
sleep 1

for i in $(seq 1 90); do
  curl -sS -H "X-Traffic-Reason: probe" \
    -X POST http://127.0.0.1:8080/v1/complete
  sleep 10
done

Inject a hung worker as a second failure mode.
Leave one job on the queue past COMPLETE_TIMEOUT_S.

# proposed fault: never dequeue the probe job
time.sleep(int(os.environ.get("COMPLETE_TIMEOUT_S", "30")))

Watch queue_age_ms climb while CPU stays boring.
Page on that contradiction, not on replica count.
Utilization is the wrong primary signal for this.

Thresholds and why queue age wins

Pick one operational threshold and write the rationale.
Write it in the runbook before the first page.

Signal	Threshold	Action
queue_age_ms	greater than 0.25 times deadline	reject new complete work
probe actual_tokens per hour	greater than 5 percent of cap	move probes to /readyz
ready_ms	greater than 200	page the adapter, not the model
deadline_slack_ms	less than 2000	fail closed and drain
CPU utilization	high at QPS about 0.1	suspect probes or retries

Queue age beats utilization for this failure mode.
Utilization stays low because the probes are infrequent.
Deadline slack shrinks because each probe is real work.

Token cap percent catches a silent overnight drain.
Ready time catches adapter stalls without model I/O.
You need all three signals because one graph will lie.

When free capacity is the wrong bet

Use a decision, not a hope, before you keep the free path.
Free serving still has a queue, a timeout, and retries.

Free capacity is the wrong bet when:

• The work is user-facing and SLO-bound
• Probes share the complete queue with users
• You cannot attribute tokens by reason
• Retry or abort policy is still unbounded
• Ready time must stay under 200 milliseconds
• A hung complete can restart the process

Use free capacity when:

• You are running an isolated local drill
• Batch jobs have explicit deadlines you can miss
• You can roll back without user impact
• Token accounting is the thing under test

Time, tokens, retries, and queueing always stack together.
Probes add a fifth term: periodic guaranteed load.
Ignore that term and unit cost inverts at night.

Rollback and cleanup

Do these steps in order and do not scale first.

1. Point kube probes at /readyz immediately.
2. Set ADMIT_PROBE_COMPLETES=0 on the adapter.
3. Drain queue entries with reason!=user first.
4. Snapshot the ledger. Then drop probe rows only.
5. Delete the compose stack and the volume.

kubectl patch deploy adapter --type=json -p='[
  {"op":"replace","path":"/spec/template/spec/containers/0/readinessProbe/httpGet/path","value":"/readyz"},
  {"op":"replace","path":"/spec/template/spec/containers/0/livenessProbe/httpGet/path","value":"/readyz"}
]'

kubectl set env deploy/adapter ADMIT_PROBE_COMPLETES=0

# local cleanup
kill "$ADAPTER_PID"
rm -f ledger.jsonl
docker compose down -v

If ready_ms stays high after the probe patch, roll the image.
Do not add replicas to hide probe spend.
More replicas often multiply probe frequency through the mesh.

Limitations and who should skip this

This drill does not measure model quality at all.
It does not prove a quota, SLA, or hardware profile.
It does not replace a user-shaped load test.

Do not use this approach when the constraints below apply.

• Your platform already exposes a dedicated health port
• You cannot change probe paths in the service mesh
• Completions are synchronous and have no queue
• You still lack a token ledger split by route

Skip it if you only run a laptop script without probes.
Skip it if a free server is your production SLO path.
Skip it if legal retention forbids dropping probe logs.

Monday checklist

Wire readyz before you expose the complete route.
Attribute tokens by route before the first deploy.
Reject probe work before queue age eats deadline slack.

Record the contradiction in the runbook tonight itself.
Tokens up, users flat, CPU flat means probes.
Act on that evidence and do not buy more capacity first.