Free Tokens Are Not an SLO: An Ops Cost Drill for AI Batch Queues

This week, two numbers trended: a harness at 100%, a model at 30%. For platform teams, a better pair is queue age and deadline slack. This article is a cost drill for the simplest AI batch path: free tokens, free server, non-negotiable deadline.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode offers free model access and a free server option. That capacity is real. It is not an SLO. The tokens cost nothing. The queue is patient. Your deadline is not.

The missing variable

Token cost is easy to measure. Operations cost is easy to ignore. A free endpoint converts a per-token bill into a per-hour bill. The bill becomes your time, your retries, and your queue age.

This drill keeps the ledger honest. It answers one question: what does a completed request cost when the token price is zero?

Topology

# worker.py (minimal, single-threaded)
import queue
import time
import csv

work = queue.Queue()
for i in range(1000):
    work.put({"id": i, "prompt_tokens": 512, "max_tokens": 256})

def call_model(payload):
    # replace with your free model endpoint
    return {"ok": True, "in_tokens": 512, "out_tokens": 180}

completed = 0
retries = 0
started_at = time.time()

while not work.empty():
    item = work.get()
    attempt = 0
    while attempt < 4:
        try:
            call_model(item)
            completed += 1
            break
        except Exception:
            retries += 1
            attempt += 1
            time.sleep(2 ** attempt)

The worker is deliberately single-threaded. Free capacity often serializes. Serialization turns a token problem into a time problem.

Declared test conditions

• 1,000 requests.
• One worker process.
• One free model endpoint.
• No client-side rate limiting.
• Deadline: 30 minutes.
• Ledger: one CSV row per request.

Ledger and report

# cost_ledger.py
import csv
import time

HOURLY_OPS_COST = 50.0  # loaded engineering rate, adjust

def record(item, elapsed, retries):
    with open("ledger.csv", "a", newline="") as f:
        csv.writer(f).writerow([item["id"], round(elapsed, 3), retries])

def report(completed, retries, elapsed_s, deadline_s):
    ops_cost = (elapsed_s / 3600.0) * HOURLY_OPS_COST
    retry_ratio = retries / max(1, completed)
    slack = deadline_s - elapsed_s
    print(f"completed={completed}")
    print(f"retries={retries}")
    print(f"wall_clock_s={elapsed_s:.1f}")
    print(f"ops_cost_usd={ops_cost:.2f}")
    print(f"retry_ratio={retry_ratio:.3f}")
    print(f"deadline_slack_s={slack:.1f}")
    return retry_ratio, slack

The token spend is zero. The ledger rows still carry a cost.

Reading the ledger

A row looks like this:

id,elapsed_s,retries
0,1.234,0
1,3.456,2

High retries on early rows mean throttling, not a crash. Growing elapsed times mean the queue is the bottleneck. Both are signals for one control decision.

Observed output (labeled expected)

Normal free-tier conditions:

Metric	Expected value
Completed	987 / 1000
Retries	214
retry_ratio	0.217
wall_clock_s	1742 (29:02)
ops_cost_usd	24.19
deadline_slack_s	58

That is the good case. 58 seconds of slack. 24 dollars of human time on a "free" job.

Failure injection

Cut the network to the endpoint for three minutes.

tc qdisc add dev eth0 root netem loss 100%

The worker retries with backoff. The queue grows. The ledger fills.

After the fault:

Metric	After injection
Completed	801 / 1000
Retries	1034
retry_ratio	1.291
wall_clock_s	1900
ops_cost_usd	26.39
deadline_slack_s	-20

Negative slack. Deadline gone. Tokens still free.

Decision rule

Read the ledger as a control loop. Two thresholds matter more than token spend:

1. retry_ratio > 0.10
2. deadline_slack_s < 0

When either fires, stop the worker and re-route.

kill $(pgrep -f worker.py)
export MODEL_ENDPOINT="https://paid.example/v1"
./worker.py --resume ledger.csv

The ledger turns a crash into a resume. You know which items completed.

Cleanup

tc qdisc del dev eth0 root
kill %1
rm -f worker.py
# keep ledger.csv if you want a cost trend

Limitations

This drill assumes you control the client. It does not measure shared CPU noise, hidden rate limits, or model quality. Those need separate experiments.

Do not use this pattern for regulated data, payments, or any job with a hard SLO. The paid path exists for a reason.

Who should skip this

• Teams with strict latency SLOs.
• Teams with no retry budget.
• Teams with no one awake when the free endpoint degrades.

Free capacity is a bet. The ledger is the odds table. Run the drill, set the thresholds, and stop before the deadline sign turns negative.