Reject Job 1 Before It Eats Job 2,000's Token Budget

Illustrative scenario, not a measurement. You queue 2,000 batch jobs against one free pool. The pool holds 10M tokens by your own accounting. That averages 5,000 tokens per job.

Averages do not admit work. Averages hide the tail that actually drains the pool.

Here is the arithmetic. Suppose the heaviest job class needs 60,000 tokens. Ten million divided by 60,000 is about 166 jobs. If those 166 arrive first, jobs 167 through 2,000 fail at the wall, not at the deadline.

The problem is accounting, not capacity

A global balance is one shared counter. Every job can spend it. Nothing reserves it.

That gives you three failure modes you will meet in order:

1. Head-of-line drain. Early heavy jobs consume the pool before the tail is admitted.
2. Retry amplification. A failed job re-spends its prompt while its first hold is still open.
3. Silent overspend. A job that exceeds its estimate has no ceiling, so it borrows from the future.

All three are bookkeeping bugs. None require more capacity to fix.

Topology: admit, hold, settle, reclaim

You need four moving parts, not one queue:

• An admission step that reserves a per-job envelope before dispatch.
• A ledger that records holds, so the balance reflects committed spend.
• A settle step that replaces the hold with actual usage once per job.
• A reclaimer that releases holds whose worker disappeared.

A free server option is a reasonable place to run this control plane. A free model endpoint is a reasonable place to run the jobs. Both are operator-supplied availability claims, and the 10M-token pool figure came from the operator as well. Treat every quota and free-tier term as time-sensitive and check current official terms before you size anything.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The rest of this article works with any pool that exposes a usage export you can reconcile against.

Reserve at admission, atomically

The reserve step must be a single conditional update. Read-then-write lets two workers hold the same tokens.

-- ledger.sqlite3, illustrative schema
CREATE TABLE IF NOT EXISTS pool (
  name    TEXT PRIMARY KEY,
  balance INTEGER NOT NULL          -- remaining tokens
);
CREATE TABLE IF NOT EXISTS reservation (
  id               TEXT PRIMARY KEY,
  pool             TEXT NOT NULL,
  job              TEXT NOT NULL,
  amount           INTEGER NOT NULL,  -- envelope held
  state            TEXT NOT NULL,     -- held | settled | released
  lease_expires_at REAL NOT NULL,
  settled_amount   INTEGER
);

# Illustrative. Not executed here. Python 3.11 + stdlib sqlite3.
import sqlite3, time, uuid

def reserve(conn, pool, job, amount, lease_s=900):
    rid = str(uuid.uuid4())
    with conn:  # BEGIN ... COMMIT
        cur = conn.execute(
            "UPDATE pool SET balance = balance - ? "
            "WHERE name = ? AND balance >= ?",
            (amount, pool, amount))
        if cur.rowcount != 1:
            return None            # refuse early, do not dispatch
        conn.execute(
            "INSERT INTO reservation "
            "(id, pool, job, amount, state, lease_expires_at) "
            "VALUES (?, ?, ?, ?, 'held', ?)",
            (rid, pool, job, amount, time.time() + lease_s))
    return rid

Two properties matter here. The WHERE balance >= ? clause makes the check and the debit one transaction. A refused reservation returns None, and the job goes back to the queue undispatchable.

That refusal is the point. You rejected work at admission instead of failing it mid-flight.

Settle once, clamp the overspend

Actual usage rarely equals the envelope. Settle must be idempotent, and it must not create balance out of nothing.

# Illustrative.
def settle(conn, rid, actual):
    with conn:
        row = conn.execute(
            "SELECT pool, amount, state FROM reservation WHERE id = ?",
            (rid,)).fetchone()
        if row is None or row[2] != "held":
            return 0                     # already settled or reclaimed
        pool, held, _ = row
        refund = held - actual
        if refund < 0:
            refund = 0                   # clamp: never mint tokens
            # emit overspend alert with rid, held, actual
        conn.execute(
            "UPDATE pool SET balance = balance + ? WHERE name = ?",
            (refund, pool))
        conn.execute(
            "UPDATE reservation SET state = 'settled', settled_amount = ? "
            "WHERE id = ?", (actual, rid))
    return refund

Clamping is a deliberate trade. The local balance stays conservative. The overspend surfaces as an alert, not as a negative pool that silently blocks every later job.

Reclaim stale holds

Workers crash. Leases expire. A hold with no owner is stranded budget.

# Illustrative.
def reclaim(conn, now=None):
    now = now or time.time()
    with conn:
        rows = conn.execute(
            "SELECT id, pool, amount FROM reservation "
            "WHERE state = 'held' AND lease_expires_at < ?", (now,)).fetchall()
        for rid, pool, amount in rows:
            conn.execute(
                "UPDATE pool SET balance = balance + ? WHERE name = ?",
                (amount, pool))
            conn.execute(
                "UPDATE reservation SET state = 'released' WHERE id = ?", (rid,))
    return len(rows)

Set the lease longer than your p99 job runtime plus one retry. A short lease double-spends, because the provider may still bill the work you just released.

Derive the envelope from the tail

Do not size the envelope on the mean. Size it on p95 input, allowed attempts, and fixed overhead.

envelope = ceil(p95_input * fanout) + p95_output * attempts_allowed + overhead

Job class	p95 input	Attempts	Envelope	Jobs the 10M pool admits
summarize	6,000	2	14,000	~714
extract-json	18,000	2	40,000	250
agent-loop	55,000	3	180,000	~55

Numbers in that table are illustrative arithmetic, not measurements. Recompute them against your own usage export.

Then cap concurrency, not just balance: max_in_flight = floor(balance / envelope). That ceiling is a safety valve, not a schedule. Pair it with deadline slack so long jobs do not sit in the queue for hours holding reservations.

Trip conditions and why

Watch four signals. Each maps to one action.

• Reservation refusal rate above 5% over 10 minutes. The pool is undersized for the admitted mix. Action: shrink envelopes or lower concurrency.
• Unspent reserve ratio above 40%. Envelopes are too fat, and you are refusing work you could run. Action: re-derive from p95.
• Overspend count above zero. The envelope is wrong for one class. Action: alert per job class, fix that class only.
• Queue age above half the deadline slack. Admission is not the bottleneck anymore. Action: stop admitting and shed load.

Thresholds above are starting points, not universal truth. Pick a refusal rate you can defend to the team that owns the deadline.

Where this ledger lies

A local ledger is a proxy for spend, not the spend itself.

• Reclaim is a guess. Released tokens may already have been billed upstream.
• Refunds require a usage export. Without one, actual is your own estimate.
• Ephemeral disks lose the ledger. If your free server recycles its filesystem, keep the SQLite file on a persistent volume, or accept that the pool resets.
• One pool, no fairness. Two teams sharing one counter will starve each other. Add per-tenant budgets before you share it.

If you need hard provider-side spend guarantees, or you cannot export usage, this design is the wrong tool. Use provider-side budget limits instead.

When free capacity is the wrong bet

Situation	Free pool?	Reason
Nightly batch with hours of slack	Yes	Refusal is cheap, requeue is free
Interactive requests under 2s	No	Retry and queueing cost exceeds the savings
Jobs with heavy retries	Only with envelopes	Retries re-spend the same prompt
Deadline inside one hour	No	Head-of-line drain is unrecoverable
Long-running stateful agents	No	Lease expiry double-spends

The pattern is consistent. Free capacity is a bet on slack. If your deadline slack is thin, the savings are borrowed from your incident budget.

Rollout, cleanup, rollback

1. Seed the pool row with a fraction of the real quota, such as 10%.
2. Replay a recorded job list in dry-run mode, writing holds but never dispatching.
3. Reconcile sum(settled_amount) against your usage export. Investigate any gap before dispatch.
4. Raise the seeded balance in steps and watch the refusal rate.

Rollback is one statement: stop dispatch, call reclaim until it returns zero, then delete the reservations with state = 'held'. Never delete settled rows mid-incident, because they are your audit trail.

Cleanup after the test run is equally short. Drop the test database, keep one export of the reconciliation diff, and record the envelope table in your runbook.

Start with the arithmetic

Before you point anything at production traffic, write down three numbers. Your pool size, your p95 envelope, and your deadline slack.

If those three do not leave room for a refused job to be retried, the pool is not free. It is a delayed bill against your on-call rotation.

When you do want a cheap place to run the admission drill, a free server plus a free model endpoint is enough to exercise reserve, settle, reclaim, and rollback end to end. Just verify the current quota terms yourself before you size the pool, and keep the ledger on storage that survives a host restart.