Some model work has no user waiting for it: embedding an uploaded document, summarising yesterday’s tickets, re-scoring a backlog. Putting that behind a queue turns provider failures from lost work into retried work, which is the whole reason to do it.
The shape: two Workers, one queue
A queue has producers, which send messages, and one consumer, which receives them in batches. The two can be the same Worker, but keeping them separate is worth the extra file: the producer is latency-critical and the consumer is not, they will want different CPU limits, and you will want to redeploy the consumer without touching the request path.
-
npx wrangler queues create ai-jobs -
npx wrangler queues create ai-jobs-dlq— create the dead-letter queue now, not after the first incident. - Add a producer binding to the request-handling Worker.
- Add a consumer binding to the processing Worker.
- Deploy both, then send one message and watch it arrive.
The producer
// wrangler.jsonc — producer Worker
{
"name": "api",
"main": "src/api.ts",
"compatibility_date": "2026-08-11",
"queues": {
"producers": [{ "queue": "ai-jobs", "binding": "AI_JOBS" }]
}
}
export default {
async fetch(request: Request, env: Env): Promise<Response> {
const { documentId, tenant } = await request.json<{ documentId: string; tenant: string }>();
await env.AI_JOBS.send({
documentId,
tenant,
kind: "embed",
attempt: 0,
enqueuedAt: Date.now(),
});
// 202: accepted, not done. Do not pretend otherwise.
return new Response(null, { status: 202 });
},
};
Send an identifier, not a payload. A message that carries the document itself is a message you cannot re-drive after a schema change and cannot inspect without reading customer data; a message that carries documentId lets the consumer fetch the current version. The attempt counter is there because it is useful in your own logs — the platform tracks retries independently.
The consumer and its batch
The consumer exports a queue handler instead of (or as well as) a fetch handler. It receives a batch, and the unit of success is the message, not the batch — which is the detail that makes this worth writing carefully.
// wrangler.jsonc — consumer Worker
{
"name": "ai-worker",
"main": "src/consumer.ts",
"compatibility_date": "2026-08-11",
"queues": {
"consumers": [
{
"queue": "ai-jobs",
"max_batch_size": 10,
"max_batch_timeout": 5,
"max_retries": 3,
"dead_letter_queue": "ai-jobs-dlq"
}
]
}
}
type Job = { documentId: string; tenant: string; kind: string; attempt: number };
export default {
async queue(batch: MessageBatch<Job>, env: Env): Promise<void> {
for (const msg of batch.messages) {
try {
const res = await fetch(env.GATEWAY_URL, {
method: "POST",
headers: {
"Content-Type": "application/json",
Authorization: `Bearer ${env.CF_API_TOKEN}`,
},
body: JSON.stringify({
model: "openai/text-embedding-3-small",
input: await loadText(env, msg.body.documentId),
}),
});
if (res.status === 429 || res.status >= 500) {
// Transient. Come back later rather than burning a retry now.
msg.retry({ delaySeconds: 60 });
continue;
}
if (!res.ok) {
// A 400 will fail identically forever. Ack it and record it.
console.error("permanent failure", msg.body.documentId, res.status);
msg.ack();
continue;
}
await storeVectors(env, msg.body.documentId, await res.json());
msg.ack();
} catch (err) {
console.error("job threw", msg.body.documentId, err);
msg.retry({ delaySeconds: 30 });
}
}
},
};
The continue after each branch is not stylistic. Cloudflare documents that the first call wins: once you have called ack() on a message, a later retry() on the same message is silently ignored, and vice versa. Falling through into a second decision is a bug that produces no error.
Retries, delays and the dead-letter queue
Cloudflare’s batching and retries documentation gives the defaults and bounds: max_batch_size defaults to 10 with a maximum of 100, max_batch_timeout defaults to 5 seconds with a maximum of 60, and max_retries defaults to 3. When a message exhausts its retries it is deleted, or written to the dead-letter queue if one is configured. Retry delays are set with delaySeconds, either per message via msg.retry({...}) or for the whole batch via batch.retryAll({...}), with a documented maximum delay of 24 hours.
Defaults and maxima are the vendor’s to change. Cloudflare, Queues batching and retries
Three decisions follow from those numbers for model work specifically:
- Distinguish transient from permanent before retrying. A 429 or a 503 will probably succeed later. A 400 for a malformed request, or a context-length error, will fail identically three more times and then land in the dead-letter queue having wasted four provider round trips. Ack the permanent failures and record them.
- Use a delay on a 429. Retrying instantly into a rate limit is how a backlog becomes a retry storm. A delay of 30 to 60 seconds costs nothing on background work and lets the window roll.
- Give the dead-letter queue a consumer. A DLQ nobody reads is a silent data-loss channel. The minimum is a consumer that writes each dead message to a D1 table so you can count them and re-drive them after a fix.
batch.retryAll() and batch.ackAll() exist for the cases where the whole batch shares a fate — the provider is down, or the batch is a no-op. Cloudflare documents ackAll() as behaving the same as a consumer that returns successfully. Reach for the per-message versions by default; batch-level calls throw away the successes alongside the failures.
Delivery is at least once
Everything in the previous section — retries, delays, redelivery from a dead-letter queue — has one consequence that has to be designed for rather than hoped away: a message can be delivered more than once, and your consumer will therefore sometimes process the same job twice.
For most background work that is harmless. For model calls it is expensive, because the duplicate costs a second inference. And there is one window where it is nearly guaranteed to happen: the consumer calls the provider, the call succeeds, and then something fails before ack() — the batch throws, the invocation is cut short, the write to your store fails. The provider has already generated and charged for the answer; the platform, correctly, redelivers the message because it never saw an acknowledgement.
Two habits remove most of the cost:
- Give every job a deterministic id and check before you spend. Not
crypto.randomUUID()— something derived from the work, so a redelivery computes the same id. A hash of the document id, the model and the prompt version is usually right, because it also changes when any of those changes and correctly re-runs the job. - Make the write idempotent. A unique constraint plus
ON CONFLICT DO NOTHINGturns a duplicate from a corrupted row into a no-op, which is a much better failure than reconciling afterwards.
const jobKey = await sha256(`${msg.body.documentId}:embed:v3`);
// Cheap read first: has this exact work already been done?
const done = await env.DB.prepare(
"SELECT 1 FROM embeddings WHERE job_key = ?1",
).bind(jobKey).first();
if (done) {
msg.ack(); // duplicate delivery, nothing to pay for
continue;
}
const vectors = await embed(env, msg.body.documentId);
await env.DB.prepare(
`INSERT INTO embeddings (job_key, document_id, vector, created_at)
VALUES (?1, ?2, ?3, ?4)
ON CONFLICT (job_key) DO NOTHING`,
).bind(jobKey, msg.body.documentId, JSON.stringify(vectors), Date.now()).run();
msg.ack();
The read-before-spend check is not free — it is a database round trip on every message — but it is orders of magnitude cheaper than the inference it guards, which is the same guard-call reasoning that applies to model routing generally. Note that it narrows the duplicate window rather than closing it: two deliveries in flight at the same moment can both read “not done”. The ON CONFLICT clause is what makes that case safe, and it is why both halves are needed rather than either alone.
Simultaneous deliveries are not hypothetical, either, because a queue consumer does not necessarily process one batch at a time — Cloudflare autoscales consumer invocations, so assume several batches may be in flight concurrently and check the consumer concurrency reference for how to bound it. That is also the number to look at before you size anything against a provider rate limit: the concurrency your provider sees is batches in flight multiplied by the in-flight requests per batch, not the number in your mapWithLimit call.
Budgeting a batch against the limits
A batch handler runs inside the same Workers runtime as everything else, so the constraints from the CPU time limit apply — with one difference that matters here. Cloudflare documents queue consumers as having a 15-minute duration limit, and that is wall clock, not CPU. Waiting is free for CPU accounting and is not free against a consumer’s 15 minutes.
That makes batch size a latency budget. Ten messages processed sequentially, each waiting five seconds on a model, is fifty seconds — comfortable. A hundred messages each waiting eight seconds is thirteen minutes, which is close enough to the limit that one slow provider takes the batch down. Either keep max_batch_size modest, or process the batch with bounded concurrency:
// Bounded concurrency: N in flight, not all of them.
async function mapWithLimit<T>(items: T[], limit: number, fn: (t: T) => Promise<void>) {
const queue = [...items];
const workers = Array.from({ length: Math.min(limit, queue.length) }, async () => {
for (let item = queue.shift(); item !== undefined; item = queue.shift()) {
await fn(item);
}
});
await Promise.all(workers);
}
// in the queue handler:
await mapWithLimit(batch.messages, 4, (msg) => handleOne(msg, env));
Four in flight is a starting point, not a recommendation: the right number is whatever keeps you under your provider’s rate limit, which you can find in your own logs. Firing all hundred at once will reliably produce 429s and convert a throughput problem into a retry problem.
Top comments (0)