Weight Your Queue by Tokens, Not Jobs

A queue that counts jobs lies about capacity. Two jobs can differ tenfold in token cost. Admit by count, and one fat job starves the lane.

Free model access gives you a finite token allowance. A free server option gives you a worker you do not pay for. One resource is scarce in tokens. The other is scarce in wall-clock time. They do not convert into each other.

That mismatch is the whole architecture problem. Read this as an architecture review: constraints, data flow, failure domains, and the changes I would queue next.

Start from the constraint

Your budget is measured in tokens. Your throughput is measured in seconds. A sleeping worker burns neither, but it also drains nothing. A waking worker can drain faster than your allowance can absorb.

So the admission decision must be expressed in the scarce unit. That unit is tokens, not jobs. Everything else is scheduling detail.

Data flow in one pass

Intake records the job. An estimator assigns a token weight per stage. Admission compares that weight to what remains. The worker takes a lease, not a job. Settlement writes actual usage back to the ledger.

Keep the estimator and the admission gate loosely coupled. Estimators drift. The gate must survive a wrong number without poisoning the ledger. That seam is where most of my review comments landed.

Here is the sketch. Treat it as unexecuted pseudocode until you have run it yourself.

# lane.py — token-weighted admission for one slow, free worker (sketch)
import sqlite3, time, uuid

DB = sqlite3.connect("lane.db", isolation_level=None)
DB.executescript("""
create table if not exists job(
  id text primary key, stage text, est int, actual int default 0,
  state text, lease_until real default 0);
create table if not exists spend(lane text primary key, spent int);
insert or ignore into spend values ('free', 0), ('paid', 0);
""")

FREE_CEILING  = 8_000_000   # headroom under the stated allowance, your call
BURST_WEIGHT  = 40_000      # max weight admitted per tick, not job count
TAIL_CAP      = 25          # leased jobs allowed to exist at once
LEASE_SECONDS = 90


def remaining(lane="free"):
    spent = DB.execute("select spent from spend where lane=?", (lane,)).fetchone()[0]
    return FREE_CEILING - spent


def submit(stage, est):
    jid = uuid.uuid4().hex[:12]
    DB.execute("insert into job(id, stage, est, state) values (?,?,?,'new')",
               (jid, stage, est))
    return jid


def tick(now=None):
    now = now or time.time()
    budget = remaining()
    leased = DB.execute("select count(*) from job where state='leased'").fetchone()[0]
    weight = 0
    rows = DB.execute(
        "select id, est from job where state='new' order by est asc").fetchall()
    for jid, est in rows:
        if leased >= TAIL_CAP:
            break
        if weight + est > BURST_WEIGHT or est > budget:
            continue          # small jobs first: the slow worker stays busy
        DB.execute("update job set state='leased', lease_until=? where id=?",
                   (now + LEASE_SECONDS, jid))
        weight += est
        leased += 1
    return weight or "tail"    # zero weight means: admit nothing this tick


def settle(jid, actual):
    cur = DB.execute(
        "update job set state='done', actual=? where id=? and state='leased'",
        (actual, jid))
    if cur.rowcount:          # idempotent on job id plus state
        DB.execute("update spend set spent = spent + ? where lane='free'", (actual,))

Two choices deserve a note. Ordering by smallest weight first keeps the slow worker fed without touching the ceiling. Settlement checks state, so a restart cannot bill you twice for one job.

The soak test that tells you if it holds

You cannot review this on a diagram. Push noisy weights at it and watch the tail.

for i in $(seq 1 200); do
  python lane.py submit --stage draft --est $((RANDOM % 9000 + 500))
done
python lane.py report

Then read drift, not totals. Compare estimate to actual per stage.

select stage,
       count(*)                as jobs,
       sum(est)                as est_tokens,
       sum(actual)             as actual_tokens,
       round(1.0*sum(actual)/nullif(sum(est),0), 2) as drift
from job where state='done'
group by stage;

A drift near 1.0 means your gate is roughly honest. Drift above 1.5 means the lane is overspending on every promise it makes. Fix the estimator before you touch the ceiling.

Four failure domains to review

Estimator drift is the first. The lane believes it has room it does not have. Cap the damage with a hard stop well under the ceiling.

Cold drainer is the second. The free worker wakes late, the tail fills, and leases expire before settlement. A preflight ping before leasing is cheaper than a retry storm.

Tail leak is the third. Parked jobs never page anyone, and nobody notices for a week. Give every parked job a deadline and a visible counter.

Double settlement is the fourth. A crash between settle and commit replays usage. Make the write idempotent, or your budget disappears quietly.

What I would change next

Version the estimator. Store est_v2 beside est so drift is attributable, not mysterious.

Separate the tail queue from the new queue. One table is simple, but parked jobs contaminate your admission scan.

Add a stop at eighty percent. The last fifth of any free allowance should be spent on finishing, not on starting.

Keep one paid escape hatch. Reviews that assume the free lane always answers are reviews of a different system.

Who should not use this

If your jobs are uniform, counting jobs is fine and simpler. If you need sub-second latency, do not route through a worker that sleeps. If a hard stop would break your product, keep a metered path behind the gate.

If you want a lane to practice on, MonkeyCode provides free model access and a free server option. The operator advertises a 10M-token allowance. Point one stage at it, cap that stage at a small share of the allowance, and see whether your estimator survives a week.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.