How I’d Build a Production Image Generation Queue with One API

Generating one AI image is easy.

Generating a few thousand of them reliably is where things get interesting.

Once image generation becomes part of a real product, the model call itself stops being the hard part. You start dealing with queues, retries, rate limits, storage, failures, model selection, and cost.

That’s why I wouldn’t let the application call an image model directly.

I’d treat every generation as a job.

A simple job might look like this:

json
{
  "id": "product-4821",
  "type": "product",
  "prompt": "Minimal studio product photo...",
  "model": "auto"
}

That job goes into a queue, and a worker decides when and how to process it.

This gives you a clean separation between your product logic and whichever model or provider happens to be behind the API.

The next thing I’d add is routing.
Not every image needs the most expensive model. A premium ad creative and a batch of internal thumbnails probably shouldn’t go through the same path.
Something simple is often enough:
def choose_model(job):
    if job["type"] == "premium_ad":
        return "high_quality_model"

    if job["type"] == "bulk_content":
        return "fast_model"

    return "default_model"

The specific models can change later. The useful part is that model choice becomes configuration instead of being baked into the application.
Concurrency is another thing that matters much earlier than people expect.

If 500 jobs arrive at once, firing all 500 requests immediately is usually a bad idea.
I’d use a worker pool and keep the number of active generations under control:
MAX_CONCURRENCY = 8

he exact number depends on the provider and workload, but the principle is the same.
A concurrency limit helps with:
- rate limits
- memory usage
- network pressure
- 429 errors
- unexpected spend
Retries need a little more thought too.

Temporary failures are worth retrying. Invalid requests usually aren’t.
I’d retry things like network errors, 429s, and transient 5xx responses, ideally with exponential backoff.
I would not keep retrying bad authentication, malformed parameters, or unsupported model names.
That sounds obvious, but I’ve seen batch systems waste a surprising amount of time and money retrying failures that were never going to recover.

The output should also be normalized before the rest of the application sees it.
Different APIs may return a hosted URL, base64 data, or some other response shape.
Your downstream code shouldn’t care.
I’d convert everything into one internal result format, for example:
{
  "job_id": "product-4821",
  "status": "completed",
  "path": "/generated/product-4821.webp",
  "model": "..."
}

At that point, the rest of the product only needs to understand your own format.
Logging is the last piece I’d consider non-negotiable.

For each job, I’d want at least:
- job ID
- selected model
- latency
- retry count
- success or failure
- estimated cost
- final output location
You can get away without this when you generate 20 images.

You really can’t when you generate 20,000.
The main reason I like using a unified API for this kind of system is not that writing another HTTP request is difficult.
It’s that every new provider usually brings another credential, another SDK, another retry path, another billing surface, and another adapter.

I’ve been testing this kind of setup through CometAPI because it lets the queue and worker layer stay mostly unchanged while the model behind it changes.
That makes it much easier to compare image models without rebuilding the rest of the pipeline.
If I were building this today, the architecture would be simple:

app
-> job queue
-> worker pool
-> model router
-> image API
-> storage
-> result database

The model can change next month.
The pipeline shouldn’t have to.
Disclosure: This post is adapted from research originally published by the CometAPI team.