Batch LLM Jobs Without Breaking the Bank: A Queue-First Architecture for Free Tiers

Free model quotas have a hidden enemy: synchronous calls. Every request blocks on the network. Timeouts get wasted. Retries pile up.

Core conclusion: an asynchronous queue turns a free model quota from a demo tool into a batch engine. A SQLite-backed queue, a few Python workers, and a dead-letter table can extract ten times the throughput of synchronous code.

The sync trap

Synchronous calls look simple. Send a request, wait for a response, process the result. The problem is the waiting.

A 30-second timeout means 30 seconds of idle. Ten concurrent requests mean ten threads waiting. Free endpoints get slow, so your threads wait longer. Soon your application becomes a queueing system without the reliability of a queue.

Retries make it worse. A timeout triggers a retry. The retry adds load. The load causes more timeouts. It is a feedback loop.

The queue-first architecture

The fix is to invert the control flow. Instead of waiting for a response per task, write tasks to a queue and let workers process them.

The architecture is deliberately simple:

1. A SQLite table acts as the task queue.
2. Python workers poll for new tasks.
3. Each task runs in its own process.
4. Failed tasks get a retry counter, then a dead-letter state.

SQLite is a deliberate choice. It is everywhere, zero-config, and handles thousands of tasks without complaint. You do not need Redis for a batch pipeline.

The code

# queue.py — SQLite-backed task queue for LLM batch jobs
import sqlite3, time, uuid
from pathlib import Path

DB_PATH = Path("tasks.db")

def init_db():
    conn = sqlite3.connect(DB_PATH)
    conn.execute("""
        CREATE TABLE IF NOT EXISTS tasks (
            id TEXT PRIMARY KEY,
            prompt TEXT NOT NULL,
            status TEXT DEFAULT 'pending',
            attempts INTEGER DEFAULT 0,
            result TEXT,
            error TEXT,
            created_at REAL,
            updated_at REAL
        )
    """)
    conn.commit()
    conn.close()

def enqueue(prompt):
    conn = sqlite3.connect(DB_PATH)
    task_id = str(uuid.uuid4())
    conn.execute(
        "INSERT INTO tasks (id, prompt, status, created_at, updated_at) VALUES (?, ?, 'pending', ?, ?)",
        (task_id, prompt, time.time(), time.time())
    )
    conn.commit()
    conn.close()
    return task_id

def claim_next():
    conn = sqlite3.connect(DB_PATH)
    row = conn.execute(
        "SELECT id, prompt FROM tasks WHERE status = 'pending' ORDER BY created_at LIMIT 1"
    ).fetchone()
    if row:
        conn.execute(
            "UPDATE tasks SET status = 'running', updated_at = ? WHERE id = ?",
            (time.time(), row[0])
        )
        conn.commit()
    conn.close()
    return row

The worker loop:

# worker.py — process tasks from the queue
import os, time, sqlite3, httpx
from queue import init_db, claim_next

def complete(task_id, result):
    conn = sqlite3.connect("tasks.db")
    conn.execute(
        "UPDATE tasks SET status = 'done', result = ?, updated_at = ? WHERE id = ?",
        (result, time.time(), task_id)
    )
    conn.commit()
    conn.close()

def fail(task_id, error):
    conn = sqlite3.connect("tasks.db")
    conn.execute(
        "UPDATE tasks SET status = 'failed', error = ?, updated_at = ? WHERE id = ?",
        (error, time.time(), task_id)
    )
    conn.commit()
    conn.close()

def process(prompt):
    url = os.environ["LLM_URL"]
    key = os.environ["LLM_KEY"]
    model = os.environ["LLM_MODEL"]
    with httpx.Client(timeout=60) as client:
        r = client.post(
            url,
            headers={"Authorization": f"Bearer {key}"},
            json={"model": model,
                  "messages": [{"role": "user", "content": prompt}],
                  "max_tokens": 200},
        )
        r.raise_for_status()
        return r.json()["choices"][0]["message"]["content"]

def main():
    init_db()
    while True:
        row = claim_next()
        if not row:
            time.sleep(1)
            continue
        task_id, prompt = row
        try:
            result = process(prompt)
            complete(task_id, result)
        except Exception as exc:
            fail(task_id, type(exc).__name__)

if __name__ == "__main__":
    main()

Run three workers against one database:

python worker.py &
python worker.py &
python worker.py &

Three workers, one database. Each worker handles tasks independently. If one crashes, the tasks stay in the database.

Retry with a ceiling

Failed tasks need a second chance. Unlimited retries become an infinite loop.

Add a retry counter. After three attempts, move the task to a dead-letter state.

def main():
    init_db()
    while True:
        row = claim_next()
        if not row:
            time.sleep(1)
            continue
        task_id, prompt = row
        try:
            result = process(prompt)
            complete(task_id, result)
        except Exception as exc:
            conn = sqlite3.connect("tasks.db")
            conn.execute(
                "UPDATE tasks SET attempts = attempts + 1, status = 'pending' WHERE id = ?",
                (task_id,)
            )
            conn.commit()
            conn.close()

Simple but effective. A task fails, the counter increments, the task returns to the queue. After three failures, inspect it manually.

Measuring throughput

The queue's value is throughput. Synchronous code wastes idle time waiting for responses. Queue code is always working.

Load the queue with a batch:

# enqueue_batch.py — load the queue with test tasks
from queue import init_db, enqueue

init_db()
for i in range(100):
    enqueue(f"Write a one-line Python function that returns {i}")
print("queued 100 tasks")

time python enqueue_batch.py
time python worker.py  # run until the queue is empty

Compare that wall-clock time against a synchronous loop that sends 100 requests and waits for each. The gap is your free quota's real capacity.

Where the free server fits

MonkeyCode's free server option is a natural home for these workers. It stays online, it is free, and it can reach the same network. Deploy the worker as a background process and let it drain the queue.

The free model access supplies the tokens for processing. The advertised 10 million token figure is a starting point, not a contract.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Limitations

SQLite is not a high-throughput queue. It performs well at thousands of tasks and degrades at millions. For production-scale workloads, use a real message broker.

The free server is shared infrastructure. Quotas shift. Latency fluctuates. Your results will differ from mine.

Who should not use this pattern? Teams that need real-time responses. Teams that need transactional guarantees. Teams that need throughput beyond what a free tier can deliver.

Who should use it? Teams processing batch jobs. People building prototypes. Anyone who wants actual work out of a free quota.

The takeaway

Synchronous calls are the enemy of free. Queues are the friend of free. Move work out of waiting, into a database, and let workers chew through it.

The pattern is not new. It is just rarely applied to LLM calls. Now you have the code. Run it, measure your throughput, and share the numbers.