Sync vs async image generation APIs: how to poll jobs without hammering the server

AI-assisted draft. I wrote and ran every snippet below locally before publishing.

Image generation is slow by web standards. A single render can take a few seconds or close to a minute, depending on the model, the size, and how busy the provider is. That is why most image APIs give you two shapes:

• Sync: POST /generate holds the connection open and returns the image when it is done.
• Async: POST /generate/async returns right away with a job id, and you check GET /jobs/{id} until the job finishes.

Sync is lovely in a notebook. In production it tends to break in boring ways: load balancer timeouts, serverless functions that cap at 10 to 30 seconds, mobile clients that drop the connection, and a retry that quietly starts (and bills) a second render.

This post builds a tiny async job API and a client that polls it properly. The pattern is the same whichever image provider you use.

When sync is fine

Use the sync endpoint when all of these are true:

• The call runs in a backend job or script, not in a user request with a short timeout.
• Typical renders finish well under your HTTP timeout.
• A dropped connection costs you little (you can just try again).

Otherwise, go async.

The async contract

A good async job API gives you four things:

1. 202 Accepted on submit, with a job id and a Location header pointing at the job.
2. A status field with a small, fixed set of values, for example queued, running, succeeded, failed, canceled.
3. A Retry-After hint while the job is still in progress.
4. Support for an Idempotency-Key header, so retrying a submit returns the same job instead of creating a new one.

Not every provider does all four. Read their docs, and code defensively for the parts they skip.

1. A tiny job server to practice against

uv venv && source .venv/bin/activate
uv pip install fastapi uvicorn httpx

server.py:

import asyncio
import random
import uuid

from fastapi import FastAPI, Header, Response
from pydantic import BaseModel

app = FastAPI()
JOBS: dict[str, dict] = {}
IDEMPOTENCY: dict[str, str] = {}


class GenerateRequest(BaseModel):
    prompt: str


async def fake_render(job_id: str) -> None:
    JOBS[job_id]["status"] = "running"
    await asyncio.sleep(random.uniform(3, 8))  # pretend the model is busy
    if random.random() < 0.1:
        JOBS[job_id].update(status="failed", error="model_timeout")
    else:
        JOBS[job_id].update(status="succeeded", result_url=f"https://cdn.example.com/{job_id}.png")


@app.post("/generate/async", status_code=202)
async def generate_async(
    body: GenerateRequest,
    response: Response,
    idempotency_key: str | None = Header(default=None),
):
    # Same key, same job: a retried POST does not start (or bill) a second render
    if idempotency_key and idempotency_key in IDEMPOTENCY:
        job_id = IDEMPOTENCY[idempotency_key]
    else:
        job_id = uuid.uuid4().hex
        JOBS[job_id] = {"id": job_id, "status": "queued", "prompt": body.prompt}
        if idempotency_key:
            IDEMPOTENCY[idempotency_key] = job_id
        asyncio.create_task(fake_render(job_id))
    response.headers["Location"] = f"/jobs/{job_id}"
    response.headers["Retry-After"] = "2"
    return JOBS[job_id]


@app.get("/jobs/{job_id}")
async def get_job(job_id: str, response: Response):
    job = JOBS.get(job_id)
    if job is None:
        response.status_code = 404
        return {"error": "not_found"}
    if job["status"] in ("queued", "running"):
        response.headers["Retry-After"] = "2"
    return job

Run it with uvicorn server:app --port 8000. Renders take 3 to 8 seconds and about 1 in 10 fail on purpose, so you can see the failure path.

2. Submit once, safely

import random
import time
import uuid

import httpx

BASE = "http://127.0.0.1:8000"
TERMINAL = {"succeeded", "failed", "canceled"}


class JobFailed(Exception):
    pass


def submit(client: httpx.Client, prompt: str) -> str:
    key = str(uuid.uuid4())  # one key per logical request, reused on retries
    for attempt in range(3):
        try:
            r = client.post(
                "/generate/async",
                json={"prompt": prompt},
                headers={"Idempotency-Key": key},
            )
            r.raise_for_status()
            return r.json()["id"]
        except (httpx.TransportError, httpx.HTTPStatusError) as exc:
            retryable = isinstance(exc, httpx.TransportError) or exc.response.status_code >= 500
            if not retryable or attempt == 2:
                raise
            time.sleep(2**attempt)
    raise RuntimeError("unreachable")

Two details matter here:

• The idempotency key is created once per logical request, outside the retry loop. If you make a new key on each attempt, the server cannot tell a retry from a new job.
• Only retry what is worth retrying. A network error or a 5xx can be retried. A 400 or 422 means the request itself is wrong, and sending it again will not help.

3. Poll with backoff, jitter, and a deadline

def wait_for(client: httpx.Client, job_id: str, deadline_s: float = 120) -> dict:
    start = time.monotonic()
    delay = 1.0
    while True:
        r = client.get(f"/jobs/{job_id}")
        r.raise_for_status()
        job = r.json()
        if job["status"] in TERMINAL:
            if job["status"] != "succeeded":
                raise JobFailed(f"{job_id}: {job['status']} ({job.get('error')})")
            return job

        elapsed = time.monotonic() - start
        if elapsed > deadline_s:
            raise TimeoutError(f"{job_id} still {job['status']} after {deadline_s}s")

        # Respect the server's hint, else back off with jitter
        hint = r.headers.get("Retry-After")
        wait = float(hint) if hint else delay
        wait = min(wait, deadline_s - elapsed)
        time.sleep(random.uniform(wait * 0.5, wait))
        delay = min(delay * 2, 15)

What each piece is doing:

• Terminal states end the loop. Treat failed and canceled as answers, not as reasons to keep polling.
• A deadline, not a retry count. "Give up after 120 seconds" is easier to reason about than "give up after 40 tries" when the delay keeps changing.
• Respect Retry-After. The server knows its queue better than you do. Note that Retry-After can also be an HTTP date; this demo only handles the seconds form.
• Exponential backoff with a cap. 1s, 2s, 4s, 8s, then flat at 15s. Without the cap, a slow job ends up waiting minutes between checks.
• Jitter. If you submit 200 jobs in a batch and they all poll on the same schedule, they hit the server in waves. Randomizing each wait spreads them out.

4. Put it together

if __name__ == "__main__":
    with httpx.Client(base_url=BASE, timeout=10) as client:
        job_id = submit(client, "flat illustration of a coffee cup, brand colors")
        print("submitted", job_id)
        try:
            job = wait_for(client, job_id)
            print("done:", job["result_url"])
        except JobFailed as e:
            print("failed:", e)

Run it a few times:

submitted 81afa3ff92ad4fae9c3344cf55926950
done: https://cdn.example.com/81afa3ff92ad4fae9c3344cf55926950.png

And check that idempotency actually works:

import httpx

with httpx.Client(base_url="http://127.0.0.1:8000") as c:
    a = c.post("/generate/async", json={"prompt": "x"}, headers={"Idempotency-Key": "k1"})
    b = c.post("/generate/async", json={"prompt": "x"}, headers={"Idempotency-Key": "k1"})
    print(a.status_code, a.json()["id"] == b.json()["id"])  # 202 True

Polling vs webhooks

If the provider offers webhooks (a callback URL it calls when the job finishes), use them for high volume. You stop paying for status checks that return "still running", and you get the result as soon as it exists.

Keep polling anyway, as a fallback:

• Webhooks get lost: your endpoint was down, a deploy was in progress, a firewall rule changed.
• A slow sweep (say, every few minutes for jobs older than your deadline) catches the ones that never called back.
• Verify webhook signatures, and make your handler idempotent, because providers usually retry deliveries.

Common mistakes

1. Polling in a tight loop. while True: get() with no sleep is how you get rate limited.
2. A new idempotency key per retry. That defeats the point and can double your bill.
3. No upper bound. A job stuck in queued should not keep a worker busy forever.
4. Treating 404 on a fresh job as fatal. Some providers write the job record a moment after returning the id. One or two short retries on 404 right after submit is reasonable; after that, it really is missing.
5. Downloading the result URL too late. Many providers return signed URLs that expire. Copy the file to your own storage as soon as the job succeeds.

Sources

• RFC 9110: 202 Accepted and Retry-After
• IETF draft: The Idempotency-Key HTTP Header Field
• AWS Builders' Library: Timeouts, retries, and backoff with jitter

How do you handle long-running jobs in your own APIs: polling, webhooks, or both? Corrections welcome in the comments.