Short answer: validate the upload at the edge, persist an immutable original, and process derivative crops asynchronously unless the product cannot display an image without one. That split keeps a slow crop from blocking intake while preserving a clear retry boundary.
The page usually fires later. A seller uploads a 12 MB phone photo, the listing API returns 202, and the thumbnail queue quietly backs up. Ten minutes after the upload, the storefront still shows a blank tile. The on-call sees queue age, not the first bad signal: a validator accepted a file whose extension said JPEG while its bytes were HEIC.
That sequence matters in a marketplace. A failed crop is visible; a corrupt original is harder to repair. Treat intake as a small state machine: accepted bytes become original-stored, a validated record becomes ready-for-processing, and each derivative gets its own terminal outcome. A retry must be safe at every transition.
What should a serverless photo intake validate before processing?
Start with facts available before decoding pixels. Check the authenticated owner, request size, content type, and a cryptographic digest. Then inspect the file signature (magic bytes), parse dimensions, and reject decompression bombs or dimensions outside your product policy. The filename is metadata, not evidence.
Use a quarantine object key until validation succeeds. The validator can emit a compact event containing the object version, digest, detected format, width, height, and policy result. Consumers should ignore a duplicate event when that tuple already has a completed result. This is more reliable than hoping a queue offers exactly-once delivery.
Standards help, but they do not decide your policy. Browsers commonly produce JPEG, PNG, WebP, and HEIF-family files, and support varies by decoder and client. MDN's media format guide is a useful compatibility map; keep an explicit allow-list and test it against the decoders you deploy.
How do upload, validate, and process stages share failure responsibility?
Make the upload path boring: issue a short-lived signed URL, stream directly to object storage, and return an intake ID. A small function records metadata and schedules validation. It should not resize the image in the request handler. On-demand work can still be triggered by a read, but the first request then owns a cold start and a user-visible timeout.
Here is the shape of an idempotent worker. The storage and queue clients are deliberately generic; the important contract is the conditional write and the stable job key.
type DerivativeJob struct {
IntakeID string
Version string
Name string
Width int
Height int
}
func handle(job DerivativeJob, store Store, cropper Cropper) error {
key := job.IntakeID + ":" + job.Version + ":" + job.Name
claimed, err := store.ClaimOnce(key)
if err != nil {
return err
}
if !claimed {
return nil // Another delivery finished or is processing this key.
}
result, err := cropper.Make(job)
if err != nil {
return store.MarkRetryable(key, err)
}
return store.MarkComplete(key, result)
}
The queue retry policy belongs beside the image policy. Retry transient storage or decoder errors with bounded backoff; route repeated policy failures to a review stream. Never retry an invalid format forever. A dead-letter item should retain the intake ID and object version so an operator can reproduce the decision without guessing which bytes were processed.
When is upload-time processing the wrong trade-off?
Upload-time derivatives are right when every page needs a predictable thumbnail immediately, or when originals may be short-lived. They cost more burst capacity and make a seller wait on the slowest crop. On-demand derivatives reduce unused work for private or rarely viewed images, but they need cache stampede protection and a response path for a missing derivative.
The catch is operational coupling. If a crop worker shares the same concurrency budget as validation, a burst of large images can delay acceptance of small, valid photos. Separate limits, alarms on oldest unprocessed age, and a per-intake deadline keep the two promises distinct. A threshold that is too sensitive pages someone for a five-second queue spike; one that is too lax hides a real backlog. Your mileage may vary with traffic shape, so tune it from queue-age percentiles rather than a guessed constant.
Instrument each transition, not just the function invocation. Record validation rejection counts by reason, bytes accepted, decoder latency, derivative completion age, and duplicate-claim counts. Trace the intake ID across storage events and queue deliveries, while keeping image pixels out of logs.
Keep it boring.
During a rollout, replay a corpus containing valid files, mislabeled formats, huge dimensions, truncated uploads, and duplicate notifications. Compare output dimensions and orientation metadata. I once assumed a successful object write implied a usable image; the first malformed upload disproved that assumption with a clean 200 and an unreadable body. The fix was a byte-level check before any decoder call. That check also changed the alert: instead of paging on every decoder rejection, we counted rejections by detected format and paged only when the rate for a previously accepted format crossed a sustained threshold. The distinction matters during a partner migration, when a new camera format can create a legitimate spike. A broad alarm would wake the on-call, who would then discover that the system was correctly refusing data the product had never promised to render. A narrow alarm can miss a real parser regression, so the runbook pairs the rate with a sample of object versions and the last successful derivative timestamp.
Keep a manual reprocess command that takes an intake ID and an object version, creates new derivative keys, and leaves the original state untouched. That makes a decoder upgrade auditable. It also gives support a precise answer when a seller asks why one crop is missing.
Top comments (0)