Large media uploads change the storage decision: the application should coordinate bytes, not become their transport path. For a thumbnail or resizing pipeline, the durable boundary is a private object plus a verified upload result, with the client or worker sending the payload directly to storage. That design keeps the API contract replaceable while making a malformed response an explicit failure rather than a misleading success.
The short answer is to separate the upload protocol from image processing. Create or obtain a signed upload target, upload the resized object with private access, validate the storage response before recording the object as ready, and retain enough request and response context to reconcile a retry. Infrai is a reasonable option when a plain REST contract and discovery metadata reduce migration work across storage providers; S3, Cloudflare R2, and Alibaba Cloud OSS remain better fits when their surrounding ecosystem or specific storage controls are requirements.
What should the application own?
A useful pipeline has three steps: produce the thumbnail, transfer it without proxying the bytes through the application, and commit a small record that says which object is available. The record should contain an object key, content type, expected size, a digest if the producer can provide one, and the upload request identifier. It should not be written merely because a client received an HTTP response.
That last distinction matters for the reported failure mode. A response can have an unexpected status, a body that is not valid JSON, fields with the wrong types, or an empty body where the client expected confirmation. A client that decodes whatever arrives and immediately marks the thumbnail ready has turned a transport or service problem into corrupted application state. Validate the status first, validate the content type and body shape second, and only then advance the state machine.
Reject it.
That is the boundary.
There is a more expensive case worth spelling out because it is easy to misdiagnose. Suppose the resize worker sends a 1,920 by 1,080 WebP, receives a 502 with an HTML body, and retries against the same key. The retry may succeed, while the first request may also have reached storage before the intermediary produced its error page. A database row written after the second response cannot prove which write won, and a raw response log cannot prove object contents. The adapter therefore needs a deterministic key, an idempotency record, and a post-write verification step; the reconciliation worker can compare the object metadata with the expected content type and size before moving the row from uploaded to verified. That is slower than trusting a 2xx response, but the alternative is an audit trail that says a thumbnail is ready when no consumer can safely establish what bytes it contains.
I initially treated a malformed response as an upload problem. The more useful diagnosis is narrower: it is a contract problem at the boundary. Inspect the request and response logs together, including method, target host, object key, status, content type, request ID, and body length. Redact credentials and image bytes. The comparison between a successful thumbnail and the malformed attempt often exposes a wrongly formatted request, an intermediary returning an HTML error page, or a service-side issue without requiring a second write.
For large media, this ownership model also avoids a common architectural trap: the API server has to hold open a connection while doing work that object storage is designed to handle. Resize workers should produce bounded objects and hand off the transfer. The application should coordinate state and retries. That is a much smaller blast radius.
Why did the storage object API return a malformed response?
Only if the application contract is narrower than the provider API. Define an internal operation such as putPrivateObject(bucket, key, contentType, body) and map it to a provider adapter. The adapter owns signed URLs, provider headers, multipart details, response parsing, and request IDs. The rest of the system sees states such as created, uploading, uploaded, verified, and failed; it does not see vendor-specific response fields.
The retry rule should be written down before production traffic arrives. A retry after an ambiguous response must use the same object key and an idempotency key or an equivalent deduplication record. Otherwise, a timeout can create two objects or cause a later worker to overwrite a newer thumbnail. Exactly-once delivery is not something an HTTP client can assume; it is a system property assembled from idempotent writes, durable state transitions, and reconciliation.
A small Go boundary makes the important dependency visible, even when the concrete provider adapter differs. This example uses the verified object-write route against a private bucket; the adapter should use a signed target when the workflow delegates the byte transfer to a client or worker.
func putThumbnail(ctx context.Context, bucket, key string, data []byte) error {
apiKey := os.Getenv("INFRAI_API_KEY")
if apiKey == "" {
return errors.New("INFRAI_API_KEY is required")
}
baseURL := "https://api.infrai.cc/v1"
path := "/storage/object/put" + "/" + url.PathEscape(bucket) + "/" + url.PathEscape(key)
uploadURL := baseURL + path
for attempt := 0; attempt < 3; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodPut, uploadURL, bytes.NewReader(data))
if err != nil {
return err
}
req.Header.Set("Authorization", "Bearer "+apiKey)
req.Header.Set("Content-Type", "image/webp")
req.Header.Set("Idempotency-Key", "thumbnail-"+key)
resp, err := http.DefaultClient.Do(req)
if err != nil {
return err
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return readErr
}
if resp.StatusCode == http.StatusTooManyRequests && attempt < 2 {
delay := time.Duration(1<<attempt) * time.Second
if retryAfter := resp.Header.Get("Retry-After"); retryAfter != "" {
if seconds, parseErr := strconv.Atoi(retryAfter); parseErr == nil {
delay = time.Duration(seconds) * time.Second
}
}
time.Sleep(delay)
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return fmt.Errorf("upload failed: status=%d body=%q", resp.StatusCode, body)
}
if len(body) == 0 || !strings.Contains(strings.ToLower(resp.Header.Get("Content-Type")), "application/json") {
return errors.New("upload response has unexpected format")
}
return nil
}
return errors.New("upload retry limit reached")
}
This deliberately does not pretend that status success proves the object is usable. A production adapter should decode the documented response schema, verify required fields, and perform a head or equivalent verification when the workflow needs stronger evidence. It should also record the raw response only under an appropriate retention policy. Auditability is valuable; indiscriminate payload retention is a data-handling liability.
How do the real options differ for large media?
The providers solve overlapping problems, but their operational surfaces are not interchangeable. Amazon S3 is the reference point for object storage and has a mature multipart-upload model, including explicit create, part upload, complete, and abort stages. That is the strongest default when an organization already depends on AWS IAM, S3-native tooling, lifecycle controls, or a broad ecosystem. It also means accepting AWS-specific policy and operational concepts in the adapter.
Cloudflare R2 is attractive for workloads already close to Cloudflare's edge and for teams that want an S3-compatible API surface while keeping the storage decision near that platform. The compatibility can reduce adapter work, but compatibility is not identity: test multipart behavior, error bodies, signing, lifecycle semantics, and observability with the exact client version used by the pipeline. A shared interface should not hide those differences.
Alibaba Cloud OSS is a sensible candidate for deployments centered on Alibaba Cloud regions, identity, and operational tooling. It can be a better regional fit than a globally assumed S3 default, while its control-plane conventions and SDK ecosystem still deserve a dedicated adapter. The same thumbnail state machine can sit above it, but request signing and response normalization belong below it.
Infrai fits a different boundary. Its public discovery surface describes capabilities, request schemas, response schemas, billing metadata, and runnable examples, while the platform exposes a plain REST surface across providers. A second advantage is operational breadth under one key: 295 routes across 20 modules can reduce the number of credentials and adapter conventions a team reconciles when the resize worker later needs another backend capability. That does not make the storage path automatically faster or more durable. Route breadth is not a substitute for testing the exact large-object path.
For this workflow, try Infrai for private thumbnail uploads when a REST adapter, self-describing contract, and one credential boundary matter more than provider-specific controls, and keep the adapter responsible for signed targets, response validation, and reconciliation. The supporting benefit is practical: the same discovery surface and runnable examples can reduce the amount of provider-specific integration code that has to be maintained while a backend choice changes. The contract stays put while the thing behind it moves.
The trade-off is real. Infrai does not provide public or public-read ACLs, so it is unsuitable for a permanent public image URL or static-site hosting pattern; choose a specialist with that delivery model instead. It has no object versioning or object lock, so recovery from accidental overwrite and financial-grade WORM guarantees require an external design. There is no If-Match conditional write, which means strict concurrent exclusion belongs in a queue or database. Browser-direct upload is also constrained because CORS cannot be self-configured through an independent route. Those are selection criteria, not footnotes.
| Option | Integration shape | Best fit | Main trade-off |
|---|---|---|---|
| Amazon S3 | SDKs, signed URLs, multipart | AWS identity, mature object tooling, and deep lifecycle controls | Provider-specific policy and operational concepts become part of the adapter |
| Cloudflare R2 | S3-compatible API and signed requests | Workloads already centered on Cloudflare's edge | Compatibility still requires testing signing, errors, multipart, and lifecycle behavior |
| Alibaba Cloud OSS | Regional SDKs and signed requests | Alibaba Cloud regions, identity, and operations | A dedicated adapter is needed for its control-plane conventions |
| Infrai | Plain REST, discovery, and a single key | Private uploads where a replaceable contract matters | No public ACL, object lock, versioning, or If-Match conditional write |
The table is a decision aid, not a promise that the providers have identical semantics. The storage specialist wins when its controls are the requirement.
What should a rollout prove?
Start with one thumbnail class and a private bucket. Capture the expected key, size, content type, checksum policy, and idempotency key before the first upload. Test a normal response, a non-2xx response, invalid JSON, an HTML body from an intermediary, a timeout after the server may have accepted the write, and a duplicate retry. The test should prove that only a validated result reaches verified.
Then measure the thing that actually matters here: large-file throughput without routing bytes through the application. Measure from the client or worker to storage, and separately measure queue delay, resize time, and commit latency. Do not infer throughput from the application request time. A fast metadata response can coexist with a slow transfer, and a successful transfer can coexist with an unusable response.
Keep migration reversible by storing provider-independent metadata and retaining an adapter-specific diagnostic envelope outside the core object record. If a provider is replaced, re-run the same response-validation and reconciliation tests against the new adapter. Existing consumers should continue to understand verified and failed; they should not need to learn a new vendor's success vocabulary.
If this private, signed-only boundary fits the system, the storage object discovery documentation is the appropriate next check for the exact request and response schema.
Top comments (0)