Short answer: accept a thumbnail batch only after assigning it a durable idempotency key and a cost budget; retry transient work with deadline-bound full jitter, and treat cancellation as a state checked immediately before publication. That sequence keeps storage and cache growth bounded when a high-volume SaaS queue redelivers jobs.
The concrete workload is a B2B SaaS product that turns one uploaded image into several smart-crop aspect ratios. The queue may be busy for minutes. A submission response therefore means “recorded and resumable,” not “all derivatives exist.” That distinction belongs in the API contract and in the runbook.
I have been paged for missed jobs and duplicate deliveries. The lesson was uncomfortable: a green enqueue response can still be an incident if two workers publish the same crop or if a canceled batch creates another cacheable object.
One invariant helps: one final object per immutable input key.
How can a SaaS thumbnail queue make submission, backoff, and cancellation boring?
Create one batch record and one deterministic work item for each requested ratio. Persist the source object version, crop policy, output dimensions, schema version, and caller-supplied idempotency token. Return an asynchronous acceptance result with a status reference. Do not report completion from the admission path.
Admission is the first cost control. Reject a request that exceeds a tenant's derivative or pixel budget before it reaches the broker. Coalesce a repeated source version and crop policy by hashing those inputs. The hash must remain stable across retries; generating a fresh key on every attempt turns recovery into duplicate work.
The state machine can stay small: queued, running, succeeded, failed, and cancel_requested. A worker leases an item, decodes the source, writes a temporary object, verifies dimensions and media type, then performs a conditional publish to a content-addressed key. If that key already has matching metadata, the worker records success and acknowledges the delivery. Duplicate delivery becomes an ordinary code path.
Cancellation is a request, not a teleport. Mark the batch and unfinished items as cancel_requested; workers check that state before expensive decoding and again immediately before the conditional publish. An item published before the cancellation timestamp can remain readable under the documented retention policy, while no later derivative is exposed. Deleting shared objects during this race is riskier than leaving a referenced object to lifecycle cleanup.
Here is the boundary I keep small enough to unit-test. The interfaces are generic on purpose.
type BatchRequest struct {
Tenant string
SourceETag string
Ratios []string
CropPolicy string
Idempotency string
}
func submit(ctx context.Context, q Queue, store Store, req BatchRequest) (string, error) {
if len(req.Ratios) == 0 || len(req.Ratios) > 12 {
return "", fmt.Errorf("invalid derivative count")
}
id := stableID(req.Tenant, req.SourceETag, req.Ratios, req.CropPolicy, req.Idempotency)
if err := store.PutIfAbsent(ctx, id, req); err != nil {
return "", err
}
for _, ratio := range req.Ratios {
if err := q.Enqueue(ctx, WorkItem{BatchID: id, Ratio: ratio}); err != nil {
return id, err
}
}
return id, nil
}
The 12 limit is a policy knob, not a media standard. Set it from decode time, memory ceilings, and the storage lifetime you promise.
What should retry policy record before a worker sleeps?
Classify the failure first. An invalid crop request, malformed source, or unsupported media type is permanent. A lease loss, dependency timeout, or throttling response is transient. Retrying permanent input errors only amplifies queue pressure and object churn.
Use exponential backoff with full jitter and a deadline tied to the batch. Persist the next-attempt time, attempt number, error class, and source version so a process restart does not reset the delay. A retry budget measured in elapsed time is easier to explain to a customer than an arbitrary attempt count.
Broker semantics differ. Visibility timeouts, acknowledgements, and consumer offsets control delivery, but none of them supplies image idempotency or cancellation rules. Keep those rules in the batch record and test them independently of the broker. I'm not sure any staging queue reproduces production's exact redelivery timing, so load tests should use production-shaped file sizes and deliberately expire leases.
Put exhausted work in a dead-letter path with the original request, a safe error category, and correlation identifiers. Do not copy the full source image into every retry record; retain a reference to the immutable source instead.
How do storage keys and caches stop a successful batch from becoming a cost incident?
Define the derivative key from source version, ratio, crop policy, encoder settings, and output schema version. Change any input and the key changes. Keep every other input stable and duplicate workers converge on the same object.
Separate hot delivery from durable retention. An edge cache can serve popular derivatives while lifecycle rules move cold originals and outputs to cheaper storage. Cache-control headers should reflect the immutability of the key; mutable keys make invalidation both expensive and easy to get wrong.
During one incident, an operator removed a canceled batch's temporary prefix and nearly removed an object still referenced by another template. The cancellation had been accepted, the worker had stopped making new derivatives, and the cleanup job interpreted the batch id as ownership of every key below that prefix. That assumption was wrong because two templates had converged on the same content-addressed output. We changed the cleanup order: temporary prefixes are isolated, final objects are deleted only after a reference check, and a cancellation record keeps its timestamp even after the queue item is acknowledged. The long-term fix was content addressing, which made shared ownership visible in the data model and gave the runbook a concrete check instead of a guess.
That check is boring. Good.
Track bytes written, bytes served, cache-hit ratio, conditional-publish conflicts, canceled-before-render items, and retry delay by tenant. Alert when canceled work produces new final objects after its cancellation point. A dashboard showing only request count hides the storage tail.
The catch is that this queue design is not suitable for interactive, sub-second previews or arbitrary pixel-level editing. A synchronous image path with a small bounded cache fits those workloads better. Stick with a durable queue when work is bursty, replay matters, and a user can tolerate a status transition.
| Situation | Queue action | Storage action | User-visible result |
|---|---|---|---|
| New batch within budget | Enqueue deterministic items | Reserve no final objects | Accepted with a status reference |
| Transient dependency failure | Retry with jitter before deadline | Keep temporary output isolated | Batch remains running |
| Permanent input failure | Acknowledge and mark item failed | Store categorized error metadata | Batch reports partial or failed |
| Cancel before publish | Stop item and acknowledge cancellation | Remove temporary bytes only | No new derivative is exposed |
| Duplicate delivery after publish | Acknowledge idempotently | Reuse the matching content key | Same result, no second object |
Verification and rollback for the on-call runbook
Test the races deliberately: cancel while decoding, cancel after temporary upload, redeliver after publish, and retry after lease expiry. Assert invariants rather than timing. There must be at most one final object for a content key, and no new derivative after the cancellation point.
Canary a worker with a separate consumer group. Compare publish conflicts, retry ages, canceled-output counts, and storage bytes per completed ratio. If those signals move unexpectedly, pause admission for new batches, let in-flight leases expire, and roll back the worker version. Existing batch state remains authoritative; replaying the queue is safer than reconstructing jobs from logs.
The runbook should name the status reference format, cancellation command, deadline policy, and first dashboard panels. Keep one example request beside it. Small details matter.
Top comments (0)