Build an Async Job Queue So Your Free-Tier LLM Never Blocks Your App

First, the problem. You call a language model from a web request. The model takes ten seconds. Your user stares at a spinner. Then the model returns a 429 because the free tier is momentarily busy. Your request fails, and the user sees an error. This is the classic synchronous trap.

You can do better. Treat LLM calls like network requests to a rate-limited API. Move them off the critical path. Put them in a queue.

MonkeyCode is an open-source LLM gateway whose free tier includes free models and a free server you can deploy. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The exact model list and limits change, so check the project README before you commit. The pattern below works with any LLM provider, including the free models MonkeyCode exposes.

Why a queue? Because free-tier endpoints are not guaranteed to answer in under a second. They are shared, they have rate limits, and they occasionally go down. A queue smooths out bursts. It retries failed calls. It lets your app continue while the model catches up.

Here is a minimal queue in Python, using only the standard library. This version stores tasks in memory, which is fine for a single process.

import asyncio
import random
from dataclasses import dataclass

@dataclass
class Task:
    prompt: str
    max_retries: int = 3

class RateLimitError(Exception):
    pass

class LLMQueue:
    def __init__(self, call_llm):
        self.queue = asyncio.Queue()
        self.call_llm = call_llm
        self.results = {}

    async def worker(self, worker_id):
        while True:
            task = await self.queue.get()
            for attempt in range(task.max_retries):
                try:
                    result = await self.call_llm(task.prompt)
                    self.results[task.prompt] = result
                    break
                except RateLimitError:
                    # exponential backoff with jitter
                    wait = min(2 ** attempt, 30) + random.random()
                    await asyncio.sleep(wait)
            else:
                self.results[task.prompt] = "failed"
            self.queue.task_done()

You need a call_llm async function that raises RateLimitError on HTTP 429. The worker retries up to three times, doubling the wait after each failure. Jitter prevents a thundering herd.

Status codes other than 429 should not be retried. A prompt that triggers a content filter or produces a malformed response will not improve by retrying. Handle those in a separate flow.

The decision table below tells you when a queue is the right tool.

Situation	Strategy
User waits for a chat reply	Synchronous, with a timeout
Background summarization	Queue with 3 retries
Bulk classification of 10k items	Queue with batching
Any call that may hit rate limits	Queue, always

A queue only helps if you can observe it. Log every state change. Here is a compact log line.

ts=2026-09-02T10:15:03Z task=summarize-42 attempt=1 state=queued
ts=2026-09-02T10:15:03Z task=summarize-42 attempt=1 state=started
ts=2026-09-02T10:15:04Z task=summarize-42 attempt=1 state=retry reason=429
ts=2026-09-02T10:15:06Z task=summarize-42 attempt=2 state=started
ts=2026-09-02T10:15:07Z task=summarize-42 attempt=2 state=done tokens=123

That log lets you answer: How many calls got rate limited? How long did each retry take? Did any task exhaust its retries?

Run the queue on MonkeyCode's free server. It is a lightweight Python process. One port, one endpoint, no database required for in-memory mode.

python -m venv .venv
source .venv/bin/activate
pip install fastapi 'uvicorn[standard]' httpx
uvicorn main:app --host 0.0.0.0 --port 8080

Then push tasks via a POST endpoint. Your web app returns a 202 immediately. The queue works in the background.

Memory queues lose everything on a crash. If you need durability, add a SQLite table and mark tasks as queued, running, done, or failed. A file-backed queue is fifty lines and gives you crash recovery. Do that before you reach for Redis.

You also want more than one worker. Start two or three asyncio tasks on the same loop. They share the queue and consume tasks in parallel. This raises throughput without making the code more complex.

Do not use free models for medically relevant decisions, financial computations, or security-critical paths. Free tiers do not offer SLAs. If your code crashes, the in-memory queue loses tasks. Use Redis or Postgres if you need durability.

This pattern is for prototypes, internal tools, and cost-sensitive products. It turns an unreliable free endpoint into a dependable background worker. The price is latency: a task may run minutes after submission. If your product needs sub-second LLM answers, a free tier is not the right foundation.

Try this queue with MonkeyCode's free models. You will learn more about retry semantics and rate limits than any tutorial can teach you. That experience carries over to paid models too.