Designing for a 10M-Token Ceiling: A Token-Budget Scheduler for Background LLM Jobs

A fixed token allowance is an architectural constraint, not a billing footnote, and treating it that way changes how you design background jobs. Once you see a 10M-token ceiling as a finite resource that needs admission control, your pipeline stops dying at the worst possible moment. The scheduler I built around that idea reserves tokens before every call, settles the ledger after the response, and defers low-priority work instead of dropping it on the floor.

The Failure Mode Nobody Designs For

Most LLM pipelines are written as if the meter never runs out, and the loop looks correct until the allowance hits zero mid-week. You pull a message from a queue, send it to the model, parse the response, and move on, and nothing in that loop knows how much budget remains. The trigger is rarely the steady-state workload; it is the re-ingestion script, the backfill job, or the retry storm that replays thousands of records in a single afternoon.

Monitoring tells you the quota is gone, but it does not tell you how to survive the next twelve hours. What happens to the queue when every call returns a quota error, and how much of your remaining allowance is spent on work that could wait until tomorrow? I wanted a mechanism that refuses to spend the last tokens on low-value work, and that mechanism turned out to be a small scheduler with three rules.

Rule 1: Estimate Before You Send

You cannot budget what you cannot predict, so the first rule is a conservative token estimator that runs before the API call. A rough heuristic — four characters per token for English, plus a fixed overhead for the system prompt — is accurate enough for admission control, even if it is wrong by twenty percent.

def estimate_tokens(text: str, system_prompt: str = "") -> int:
    # Conservative heuristic: ~4 chars per token for English text.
    return len(text) // 4 + len(system_prompt) // 4 + 64

The estimator does not need to be perfect, because the ledger corrects itself after the real call. What it needs is to be consistently conservative, so a burst of long documents cannot silently overrun the ceiling before anyone notices.

Rule 2: Reserve, Then Settle

The core idea is a reservation model borrowed from transaction systems: check the budget, hold the estimated tokens, make the call, then settle the difference with the actual usage from the response. This prevents two concurrent workers from both seeing enough budget and spending the same tokens twice, which is exactly the race that blows through a fixed allowance.

import json
import time
from pathlib import Path

class TokenBudget:
    def __init__(self, daily_cap: int, state_path: Path):
        self.daily_cap = daily_cap
        self.state_path = Path(state_path)
        self.state = self._load()

    def _load(self) -> dict:
        if self.state_path.exists():
            return json.loads(self.state_path.read_text())
        return {"day": time.strftime("%Y-%m-%d"), "used": 0}

    def _persist(self) -> None:
        self.state_path.write_text(json.dumps(self.state))

    def remaining(self) -> int:
        today = time.strftime("%Y-%m-%d")
        if self.state["day"] != today:
            self.state = {"day": today, "used": 0}
            self._persist()
        return self.daily_cap - self.state["used"]

    def reserve(self, estimate: int) -> bool:
        if self.remaining() < estimate:
            return False
        self.state["used"] += estimate
        self._persist()
        return True

    def settle(self, estimate: int, actual: int) -> None:
        self.state["used"] += actual - estimate
        self._persist()

The persistence matters more than it looks, because a crash mid-batch should not reset the ledger to zero. A JSON file is enough for a single process, while a database row is the right choice for a distributed worker pool.

Rule 3: Defer Instead of Drop

The final rule is the one that changes behavior: when the remaining budget falls below a threshold, low-priority tasks wait for the next day instead of failing. The queue becomes a two-level priority structure, and the scheduler simply stops draining the low-priority level when the ceiling is close.

from collections import deque

class BudgetAwareQueue:
    def __init__(self, budget: TokenBudget, low_priority_floor: float = 0.2):
        self.budget = budget
        self.floor = low_priority_floor
        self.high_priority = deque()
        self.low_priority = deque()

    def push(self, task, priority: str = "normal") -> None:
        target = self.high_priority if priority == "high" else self.low_priority
        target.append(task)

    def pop(self):
        if self.budget.remaining() < self.floor * self.budget.daily_cap:
            if self.high_priority:
                return self.high_priority.popleft()
            return None  # preserve the budget; the task waits
        source = self.high_priority or self.low_priority
        return source.popleft() if source else None

This is the difference between a hard failure and a graceful one, and the distinction is what makes the system operable. A hard failure wakes somebody up at 3 AM, while a deferred task simply shows up in tomorrow's queue with a few hours of latency on a report nobody reads until Friday.

Putting the Three Rules Together

The workflow looks like this when the pieces are wired into a background job:

1. Estimate the token cost of the next task before touching the API.
2. Call budget.reserve(estimate); if it returns False, push the task back and stop the loop.
3. Make the API call, read the usage field from the response, and call budget.settle(estimate, actual).
4. On any retry, re-reserve the tokens, because a retry storm is exactly the burst that blows through a ceiling.

The loop is deliberately small, since the discipline lives in the order of operations rather than in the size of the code. Reserve before you call, settle after you respond, and defer before you fail.

Why a Fixed Allowance Changes Your Architecture

A metered API charges you for overuse, which means a mistake costs money but does not stop the system. A fixed allowance is different: the ceiling is absolute, and the only question is whether your code respects it or crashes into it. That is why I ran this scheduler against MonkeyCode's free tier, which currently includes a 10M-token allowance and a free server option for running this kind of orchestration. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The free tier matters here because it makes the constraint visible: there is no credit card to absorb a burst, so the scheduler is not an optimization but the difference between a job that survives the month and one that dies on day nine. On a paid API the same code still saves you money, while on a free allowance it saves you the entire pipeline.

Limitations and Who Should Not Use This

A token budget scheduler is the wrong tool for interactive workloads, where a user is waiting and deferring is not an option. It is also wrong when task sizes are wildly unpredictable and the estimator is too naive, because the settlement correction can lag the reservation by a large margin.

The free tier's exact limits, reset period, and server capacity are things you should verify against the current published terms before relying on them, since availability details change faster than code. And if your workload is tiny, the whole scheduler is overkill; a single if remaining < estimate: sleep line inside your loop gives you eighty percent of the benefit.

The Takeaway

A fixed token allowance is a deadline, and deadlines need admission control rather than hope. Estimate before you send, reserve before you call, settle after you respond, and defer the work that can wait until tomorrow. The scheduler above is small enough to read in one sitting and robust enough to run unattended, and a free tier is a practical place to watch it work.

MonkeyCode provides free models that can run this workflow.