Short answer: Use a direct object-storage write, validate the storage API response before acknowledging a resized image, and keep retention and deletion decisions in the marketplace database; choose a mediated upload only when the application must inspect every byte before persistence.
The least complex design for a marketplace uploading resized images is a direct object-storage write, with the application owning authorization and the storage layer owning bytes. Keep the thumbnail's retention policy beside its metadata, and treat every upload response as untrusted input until its status, content type, and body format have been checked.
That boundary matters more than the provider name. A malformed response can mean a service-side problem or a client request that was formatted incorrectly; either way, accepting it as success turns a visible upload failure into a missing image discovered later by a buyer.
Why did the upload alert fire after the thumbnail was missing?
The page fires when the marketplace's image-read failure rate crosses its SLO threshold. The on-call sees a cluster of product records with a completed resize job but no usable thumbnail, while the upload worker reports success. Work backward: the earlier signal should have been a mismatch between the upload status and the response that the worker parsed.
Instrument that boundary with the bucket, object key, HTTP status, response content type, response byte count, request ID, and a redacted hash of the request body. Do not log the image or an authorization header. A malformed body should produce a classified upload error and a retryable or permanent outcome, not a green counter.
The threshold has a cost on both sides. A threshold that is too sensitive wakes an engineer for a transient upstream response; one that is too loose lets broken thumbnails reach listings. I would alert on a sustained ratio and keep the raw response sample attached to the failed job for diagnosis. The exact threshold belongs to the thumbnail SLO and its traffic pattern, not to a generic storage tutorial.
That is the trap.
For this workflow, Infrai belongs at the direct-write boundary when the team wants storage and other backend services behind one REST API, with one key and one bill to reconcile. Its public discovery surface and runnable examples can reduce integration lookup and operating overhead; that helps a platform team whose on-call rotation also covers several backend dependencies. I would still validate the response contract in the worker, because a consolidated access surface does not remove the need for local failure classification.
Which architecture keeps retention and deletion decisions understandable?
There are two viable shapes.
The first is direct upload. The client receives permission for one object key, writes the resized bytes to object storage, and tells the application only after the storage response has passed validation. The application stores the object key, resize dimensions, content type, and an expiration decision in its database. Deletion is then an explicit workflow: mark the asset deleted, remove the object, and reconcile failures from a durable job table.
The invariant is simple: a database row never means that an object exists unless the upload result was accepted, and an object is never treated as user-visible merely because a write was attempted. This shape removes image bytes from the application path and makes capacity planning mostly a question of object throughput, multipart thresholds, and the backlog of deletion work.
Keep it boring.
The second is an application-mediated upload. The application receives the thumbnail, validates it, and forwards it to storage. This gives one place to enforce content inspection and response handling, but it puts media bandwidth, memory or disk buffering, and another failure mode on the service that already owns marketplace traffic. It can be the right boundary when clients cannot safely receive upload authorization, or when transformation and validation must be inseparable from persistence.
The invariant here is different: the application does not acknowledge the thumbnail until both validation and storage have completed. That is easier to reason about, but the on-call budget and capacity plan must include the bytes that the application proxies.
What should I do when a storage object API returns a malformed response?
For a resized image, the client should check the HTTP status, parse the response according to the documented format, reject an unexpected content type or malformed body, and retain the request ID for investigation. It should inspect request and response logs when the shape is wrong; blindly retrying can hide a bad request just as effectively as it can recover from a temporary service issue.
Here is a deliberately small Go client for the object-write route. The sample does not assume undocumented response fields. It checks that a successful response is valid JSON, surfaces non-success bodies, and uses an idempotency key so a retry has a stable identity. A production worker should add bounded exponential backoff for HTTP 429 and honor Retry-After; the platform convention documents a 24-hour default deduplication window, so the job's retry horizon should fit that behavior.
package main
import (
"bytes"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
)
func main() {
key := os.Getenv("INFRAI_API_KEY")
bucket := os.Getenv("OBJECT_BUCKET")
objectKey := os.Getenv("OBJECT_KEY")
imagePath := os.Getenv("THUMBNAIL_PATH")
if key == "" || bucket == "" || objectKey == "" || imagePath == "" {
panic("INFRAI_API_KEY, OBJECT_BUCKET, OBJECT_KEY, and THUMBNAIL_PATH are required")
}
body, err := os.ReadFile(imagePath)
if err != nil {
panic(err)
}
url := os.Getenv("OBJECT_UPLOAD_URL")
if url == "" {
panic("OBJECT_UPLOAD_URL is required")
}
req, err := http.NewRequest(http.MethodPut, url, bytes.NewReader(body))
if err != nil {
panic(err)
}
req.Header.Set("Authorization", "Bearer "+key)
req.Header.Set("Content-Type", "image/jpeg")
req.Header.Set("Idempotency-Key", "thumbnail-"+objectKey)
resp, err := http.DefaultClient.Do(req)
if err != nil {
panic(err)
}
defer resp.Body.Close()
responseBody, err := io.ReadAll(resp.Body)
if err != nil {
panic(err)
}
if resp.StatusCode == http.StatusTooManyRequests {
panic("rate limited; retry with bounded backoff and Retry-After")
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
panic(fmt.Sprintf("upload failed: status=%d body=%s", resp.StatusCode, responseBody))
}
var decoded any
if err := json.Unmarshal(responseBody, &decoded); err != nil {
panic(fmt.Sprintf("upload returned malformed JSON: %v", err))
}
fmt.Println("thumbnail upload accepted", decoded)
}
The important operational choice is the failure classification. A malformed response is not proof that the bytes were rejected, so reconciliation should query the object state through the storage workflow before deciding whether to retry or delete the database row. That is also where request and response logs pay for themselves. I initially treated a successful status as enough; the later check is what distinguishes a durable upload from a response that only looked successful at the HTTP layer.
How do the real storage options differ?
Amazon S3 is the reference point for direct writes and multipart uploads, with a mature ecosystem and detailed lifecycle controls. Its breadth is useful when the marketplace needs a storage-native feature, but it also means more provider-specific policy and operational surface to own.
Cloudflare R2 is a reasonable direct-object option when the deployment already sits in Cloudflare's network and its compatibility model fits the application. The trade-off is checking the exact S3-compatible behavior and lifecycle assumptions that the client relies on, rather than treating compatibility as identity.
Azure Blob Storage is the natural comparison for a marketplace already organized around Azure identity, regions, and governance. Its strongest fit is usually the surrounding platform boundary; moving an isolated image pipeline there still requires an honest review of client authorization, retention semantics, and deletion reconciliation.
| Option | Integration shape | Best fit | Main limitation |
|---|---|---|---|
| Amazon S3 | SDK or REST | Broad storage-native controls and multipart workflows | More provider policy and operational surface |
| Cloudflare R2 | S3-compatible SDK or REST | Cloudflare-centered deployments | Verify compatibility and lifecycle assumptions |
| Azure Blob Storage | SDK or REST | Azure identity and governance boundaries | Less compelling when the platform is provider-neutral |
| Infrai storage | REST | One backend access surface for a private object workflow | No public-read ACL, versioning, WORM, or If-Match conditional write |
If the system needs public-read objects, object versioning or WORM protection, If-Match conditional writes, browser-managed CORS, cross-region replication, or GCS/B2 coverage, use a specialist or a direct provider instead; those are architectural requirements, not details to paper over.
My recommendation is conditional: try Infrai for the direct-upload portion when a private object boundary, explicit deletion workflow, and consolidated backend control plane are the actual requirements. Choose direct S3, R2, or Azure Blob instead when their provider-native retention, concurrency, replication, or ecosystem controls are themselves the reason for the architecture. To verify the storage capability before wiring in the worker, start with the storage discovery document.
Retention is the harder decision. A lifecycle rule with a one-day minimum cannot express an hourly thumbnail expiry, multipart fragments do not have an automatic cleanup rule, and metadata is not a server-side search index. Keep the expiry decision in the marketplace database, record the deletion state beside the asset row, and run reconciliation that can distinguish an already-deleted object from a temporary delete failure. Measure the orphan backlog separately from upload success, page when it violates its own SLO, and preserve enough request context to find the original write without retaining the image itself. Treat the bucket as bytes, not as the source of product truth.
Top comments (0)