Don't Retry Model Calls. Queue Them.

Every developer who has built on a free model tier has felt the same panic: a batch job that worked yesterday fails today, and the error message says "timeout" without telling you why. You bump the timeout, add retries, and make things worse. I've been there more times than I want to count, and the pattern that finally fixed it wasn't a better client or a smarter retry loop. It was a queue.

Running on MonkeyCode's free model access and a free server tier, I needed a way to process a few hundred text items without babysitting the process. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The queue pattern below is what made it work. It's not glamorous, but it turned a flaky, unpredictable batch job into something I could start and walk away from.

Why Retries Make the Problem Worse

When a model call fails, the instinct is to retry it immediately. That's exactly the wrong move. A burst of retries hits the API at the same moment, creating a thundering herd that the free tier is ill-equipped to handle. Worse, each retry holds a connection open longer, which can starve other requests in the same process.

The alternative is to decouple the act of "wanting a result" from "getting the result." A queue lets you accept work immediately, process it at a controlled rate, and handle failures without blocking your main flow.

The Pattern: A Local Queue with a Worker

The idea is simple: instead of calling the model API directly from your main thread, you push a job into a local queue. A worker thread (or process) pulls jobs one at a time, calls the model, and stores the result. If a call fails, the worker retries it with backoff, up to a limit, then marks it as dead.

On a free server, this pattern is especially valuable because it keeps memory and connection usage flat, regardless of how many items you throw at it.

Step 1: Define the Job Table

I use SQLite for this because it's everywhere, even on the smallest free server. No extra services to install, no Redis to keep alive.

import sqlite3
import json
import time
import threading
from datetime import datetime, timezone

DB_PATH = "jobs.db"

def init_db():
    conn = sqlite3.connect(DB_PATH)
    conn.execute("""
        CREATE TABLE IF NOT EXISTS jobs (
            id INTEGER PRIMARY KEY AUTOINCREMENT,
            payload TEXT NOT NULL,
            status TEXT NOT NULL DEFAULT 'pending',
            attempts INTEGER NOT NULL DEFAULT 0,
            result TEXT,
            error TEXT,
            next_attempt_at REAL NOT NULL,
            created_at REAL NOT NULL
        )
    """)
    conn.commit()
    conn.close()

The next_attempt_at column is what makes backoff possible. When a job fails, we set it to now + delay, and the worker skips jobs until that time arrives.

Step 2: Add Jobs to the Queue

Adding a job is just an INSERT. You can do this from anywhere in your app, and it returns immediately.

def enqueue(payload: dict):
    conn = sqlite3.connect(DB_PATH)
    conn.execute(
        "INSERT INTO jobs (payload, status, next_attempt_at, created_at) VALUES (?, ?, ?, ?)",
        (json.dumps(payload), "pending", time.time(), time.time()),
    )
    conn.commit()
    conn.close()

Step 3: The Worker Loop

The worker runs in its own thread. It polls the database for the next pending job whose next_attempt_at has passed, claims it, and processes it.

def claim_next_job(conn):
    conn.execute("BEGIN IMMEDIATE")
    row = conn.execute("""
        SELECT id, payload FROM jobs
        WHERE status = 'pending' AND next_attempt_at <= ?
        ORDER BY created_at LIMIT 1
    """, (time.time(),)).fetchone()
    if row:
        conn.execute("UPDATE jobs SET status = 'processing' WHERE id = ?", (row[0],))
    conn.commit()
    return row

The BEGIN IMMEDIATE transaction prevents two workers from claiming the same job, which matters if you ever scale to multiple threads.

The processing function is where you call the model. I'll use a placeholder here; replace it with your actual API call.

def call_model(payload: dict) -> str:
    # Replace this with your real model call.
    # Example: response = requests.post("https://api.example.com/v1/complete", json=payload)
    # return response.json()["text"]
    time.sleep(0.5)  # simulate latency
    return f"processed: {payload['text'][:20]}"

The worker loop itself is straightforward:

def worker_loop():
    while True:
        conn = sqlite3.connect(DB_PATH, timeout=5)
        job = claim_next_job(conn)
        if job is None:
            conn.close()
            time.sleep(1)
            continue

        job_id, payload_str = job
        payload = json.loads(payload_str)
        try:
            result = call_model(payload)
            conn.execute(
                "UPDATE jobs SET status = 'done', result = ? WHERE id = ?",
                (result, job_id),
            )
            conn.commit()
        except Exception as exc:
            attempts = conn.execute(
                "SELECT attempts FROM jobs WHERE id = ?", (job_id,)
            ).fetchone()[0] + 1
            delay = min(2 ** attempts, 60)  # exponential backoff, capped at 60s
            conn.execute("""
                UPDATE jobs SET status = 'pending', attempts = ?, error = ?, next_attempt_at = ?
                WHERE id = ?
            """, (attempts, str(exc), time.time() + delay, job_id))
            conn.commit()
        conn.close()

Notice that on failure, the job goes back to pending with a next_attempt_at in the future. The worker will pick it up again, but only after the backoff delay. After a few attempts, you can decide to mark it as dead instead of retrying forever.

Step 4: Start the Worker and Enqueue Your Batch

if __name__ == "__main__":
    init_db()
    threading.Thread(target=worker_loop, daemon=True).start()

    for i in range(100):
        enqueue({"text": f"article number {i}"})

    # Keep the main thread alive
    while True:
        time.sleep(10)

That's the whole pattern. You can run this on a free server with a single process, and it will chew through the batch at a rate determined by your model's latency, not by your patience.

Why This Works on a Free Server

Free server tiers usually have tight memory limits and aggressive CPU throttling. A queue keeps your resource usage flat: you're never holding more than one model call in memory at a time, and you're never blasting the API with a hundred concurrent requests. That's the opposite of the connection-pool problems I've hit before, where a burst of parallel calls would exhaust local resources and time out before reaching the network.

The queue also gives you a natural place to add observability. You can query the jobs table anytime to see how many are pending, how many failed, and what the errors were. That alone is worth the switch.

Limitations and When Not to Use This

This pattern is not a silver bullet. If your model API is down for an hour, the queue will simply accumulate jobs and process them late. That's fine for batch work, but useless for real-time features like chat.

It also assumes a single worker process. If you need to scale to multiple machines, you'll need a real message broker like Redis or RabbitMQ. SQLite works beautifully for one process, but it's not a distributed queue.

Finally, if you're only making a handful of calls per day, the queue is overkill. Use it when you have a batch of, say, a hundred items or more, or when you're tired of babysitting a script that keeps dying.

The Takeaway

The next time a model call fails, resist the urge to retry it immediately. Push it to a queue, let a worker handle it with backoff, and move on. This pattern turned my free-tier batch jobs from a source of anxiety into something I genuinely trust.

If you've been fighting flaky model calls, try the queue pattern before you reach for another retry library. It's boring, it's simple, and it works.