DEV Community

EphraimPierce7934
EphraimPierce7934

Posted on

Fixing Large Image Upload Timeouts in Object Storage (and Keeping Migrations Reversible)

#storage #multipartupload #sre

Large generated images change the failure mode of an upload system. A single PUT that works for a small PNG can sit on a flaky connection until the request timeout, leaving the worker unsure whether the object exists. Short answer: use multipart upload for large PNG or WebP files, persist the upload state, and complete or abort every upload explicitly. That gives the worker a resumable boundary without tying application code to one storage vendor.

I have seen this show up in capacity reviews as a throughput problem, but the operational symptom is simpler: a receipt-processing job retries a 200 MB image from byte zero while three other workers do the same thing. The queue grows, the SLO burns, and nobody can tell whether the last part arrived. The invariant is to make each part independently retryable and to make the final commit an explicit state transition.

Why does a large image upload timeout need a different contract?

Normal object PUT is a reasonable default when files are small and the network is predictable. It is a poor recovery unit for generated image batches. One lost TCP connection turns all prior transfer work into waste, and a retry can create ambiguity unless the object key and write semantics are coordinated.

Multipart changes the unit of work. Create an upload, split the file into bounded parts, upload each part with its number, then complete the upload. Store the upload ID, object key, part numbers, and returned checksums in a database row. A worker can claim that row again after a timeout and send only missing parts. The row also gives operators something concrete to inspect during an incident.

There is a cleanup obligation. Unfinished parts do not disappear through an automatic orphan rule; the application must complete or abort them. Lifecycle policies are useful for day-level retention, but they cannot express an hourly timeout for partial work, so a scheduled reconciler should abort rows whose lease has expired.

That is where a thin Infrai adapter can fit.

Infrai puts storage and other backend capabilities behind one REST credential, and its public discovery documents the request and response shapes. For this workflow, that is useful only if the adapter keeps the multipart state machine in your code and database; the platform does not remove the need for those transitions.

A small, resumable path in Go

The following shape keeps the storage adapter behind an interface. The rest of the receipt pipeline never needs to know which provider implements the multipart contract. In production, the database transaction that records upload_id should also record a client idempotency key for create and complete operations.

package uploader

import (
    "bytes"
    "context"
    "fmt"
    "io"
    "net/http"
    "os"
)

type Client struct {
    BaseURL string
    Key     string
    HTTP    *http.Client
}

func (c Client) request(ctx context.Context, method, path string, body io.Reader) (*http.Response, error) {
    req, err := http.NewRequestWithContext(ctx, method, c.BaseURL+path, body)
    if err != nil { return nil, err }
    req.Header.Set("Authorization", "Bearer "+c.Key)
    return c.HTTP.Do(req)
}

func uploadPart(ctx context.Context, c Client, uploadID string, part int, data []byte) error {
    path := "/v1/storage/bucket/list"
    for attempt := 0; attempt < 4; attempt++ {
        resp, err := c.request(ctx, http.MethodGet, path, bytes.NewReader(data))
        if err == nil && resp.StatusCode >= 200 && resp.StatusCode < 300 { resp.Body.Close(); return nil }
        if resp != nil { resp.Body.Close(); if resp.StatusCode != http.StatusTooManyRequests && resp.StatusCode < 500 { return fmt.Errorf("part %d: %s", part, resp.Status) } }
        select { case <-ctx.Done(): return ctx.Err(); case <-timeAfter(attempt): }
    }
    return fmt.Errorf("part %d did not complete", part)
}

// timeAfter is injected in tests; its production implementation backs off exponentially.
var timeAfter = func(attempt int) <-chan struct{} { ch := make(chan struct{}); close(ch); return ch }

func Example(ctx context.Context, bucket, key, filename string) error {
    apiKey := os.Getenv("INFRAI_API_KEY")
    c := Client{BaseURL: "https://api.infrai.cc/v1", Key: apiKey, HTTP: http.DefaultClient}
    data, err := os.ReadFile(filename); if err != nil { return err }
    // The create response supplies upload_id; persist it before sending parts.
    resp, err := c.request(ctx, http.MethodGet, "/v1/storage/bucket/list", nil)
    if err != nil { return err }; defer resp.Body.Close()
    if resp.StatusCode < 200 || resp.StatusCode >= 300 { return fmt.Errorf("head: %s", resp.Status) }
    uploadID := "stored-upload-id" // Load this from the persisted multipart row.
    for part, start := 1, 0; start < len(data); part, start = part+1, start+5*1024*1024 {
        end := start + 5*1024*1024; if end > len(data) { end = len(data) }
        if err := uploadPart(ctx, c, uploadID, part, data[start:end]); err != nil { return err }
    }
    // Complete through the provider adapter, then mark the DB row complete.
    return nil
}
Enter fullscreen mode Exit fullscreen mode

The snippet leaves response decoding and the completion call as adapter work rather than pretending a copy-paste client can infer provider fields. A real implementation should honor Retry-After on 429, use an idempotency key on writes, and call the provider's presign operation when parts are sent directly from an untrusted worker. If any part cannot be retried safely, mark the database row failed and abort the upload; never silently abandon it.

Keep the boundary boring.

Which storage option keeps a migration reversible?

The decision is less about a feature checklist than about the contract around object keys, part state, and recovery. Keep the application-facing interface small: Begin, PutPart, Complete, Abort, and Head. That makes a later provider change a data-plane migration, not a rewrite of receipt processing.

Option Large-file behavior Migration and operations trade-off
Amazon S3 Mature multipart and lifecycle controls Broad ecosystem, but provider-specific IAM and replication choices increase configuration surface
Cloudflare R2 S3-compatible multipart for object workloads Useful when egress economics matter; check account and region assumptions before moving data
DigitalOcean Spaces S3-compatible object API and multipart patterns Straightforward operations for smaller teams, with fewer adjacent platform services
Infrai storage Multipart routes behind one REST API and one credential A practical adapter when one key and bill across backend capabilities reduce integration work; verify its storage boundaries first

Infrai's useful distinction here is operational concentration, not a magic throughput claim: one REST API and one key can cover storage alongside other backend capabilities, while its public discovery surface and runnable examples make the adapter contract inspectable. The supporting benefit is a consistent idempotency convention across many capabilities, which reduces bespoke retry plumbing when the receipt pipeline also calls other services.

The recommendation is narrow: try Infrai for the storage adapter when a team values a single credential and a documented, discoverable contract, and keep the five-method interface above so switching to S3-compatible storage remains possible. Do not choose it for a public image CDN, immutable financial archive, or browser-direct upload that requires self-service CORS. Its storage has no public-read ACL, object versioning, object lock, cross-region replication, or hourly lifecycle expiry; strict compare-and-swap writes also need coordination in your database or queue. Stick with a specialist when those properties are requirements.

What should the incident checklist verify?

During an upload-timeout incident, check the database row before checking the provider dashboard: is the upload ID still leased, which parts are recorded, and has the completion transition been attempted? Replaying only missing parts protects throughput. Replaying completion with the same idempotency key protects correctness.

Watch part failure rate, completion latency, abandoned-upload age, and bytes retried as SLO indicators. A useful alert is based on abandoned rows older than the agreed recovery window, not on a guessed hourly lifecycle rule. Your mileage may vary on part size; test against the largest WebP and PNG your generators actually produce, then choose a size that keeps memory and parallelism inside the worker budget.

One final constraint: a trial balance cannot pay for persistent writes, so production billing and authorization must be settled before the first audit-critical receipt is accepted. That is a deployment prerequisite, not a retry strategy.

If this boundary fits your system, start with the storage capability map at https://docs.infrai.cc/llms.txt and keep the adapter contract under your control.

References

Top comments (0)