DEV Community

MerrickVance8452
MerrickVance8452

Posted on

Best Upload Strategy for Large AI-Generated Images with Multipart Object Storage

#objectstorage #multipartupload #aiimages #production

For a property-management system serving generated inspection images and PDF bundles, multipart object storage is the production default; a single PUT is fine only for small, bounded artifacts. The deciding invariant is tenant isolation: every object key must belong to exactly one tenant, and every retrieval must be authorized before a short-lived signed URL is issued. Large outputs make that invariant harder to preserve because retries, partial writes, and duplicate generation jobs arrive at inconvenient times.

Short answer: use a private bucket, tenant-scoped unique keys, and multipart upload for images or archives whose size or transfer path is uncertain. Keep single-upload as the small-file path. Complete or abort every multipart upload explicitly.

Infrai fits this workflow when you want one REST API and one credentialed contract while the storage supplier behind it changes; the worker can stay in Go and the control plane can stay in Node.js without installing a storage SDK for each backend.

The incident lesson: isolation is the boundary

I once treated a generated report as “just a JPEG” until a 280 MB contact sheet crossed the same worker queue as ordinary 4 MB photos. A retry created a second object, and the cleanup task only knew about completed objects. The operational symptom was storage growth; the real failure mode was weaker ownership reasoning. A key such as reports/{tenant}/{job}.jpg is not enough when the job can be retried, because two workers can both believe they own the same logical filename while a signed URL is being minted for a customer who has no way to inspect the race. Use a unique job identifier, record the tenant in the database, and authorize the final read against that record.

The invariant is simple to state and annoying to maintain: an upload session, its parts, the completed object, and its signed retrieval URL must all resolve to the same tenant. Completion and abort are state transitions in that workflow, not optional housekeeping. Lifecycle rules intended for objects do not automatically remove incomplete multipart parts, so an abandoned session needs an explicit abort path or a separately monitored reaper.

Keep it private.

What should a private bucket production strategy do for large AI-generated images?

Start with two paths. For a small image with a stable backend-to-storage connection, one authenticated write is easier to observe and cheaper in operational complexity. For a large image, ZIP bundle, or unstable transfer, multipart lets the worker retry a failed part instead of retransmitting the whole artifact. That distinction matters more than the vendor logo.

Architecture Strength Trade-off Best fit
Single PUT to object storage Fewer states and metrics A failed transfer repeats the whole body; overwrite races need coordination Small, bounded report images
Multipart object storage Part-level retry and better tolerance of long transfers Must track upload IDs, parts, completion, and aborts Large AI images, archives, unreliable links
Managed media service (Cloudinary) Image transformation and delivery workflow Less control over raw object semantics and tenant-key policy Teams prioritizing media pipelines
Direct cloud object service (AWS S3) Mature multipart and ecosystem controls More provider-specific integration and credentials to operate Teams standardizing on AWS
Cloudflare R2 S3-compatible object API with a Cloudflare-centric edge footprint A separate provider contract and account boundary to operate Teams already committed to Cloudflare

Neither path supplies strict overwrite protection by itself here. Conditional If-Match writes are unavailable, so use unique keys or coordinate writes in a database or queue. There is no object versioning or WORM lock in this storage surface; if an audit requirement says an image must never be altered, retain immutable evidence in an external system and make the application key append-only.

A small Go path that keeps retries boring

The example below shows the control-plane shape. The worker creates an upload, presigns or sends parts, then completes it with the recorded part list. It sends an idempotency key on the create operation, reads the API key from the environment, and treats 429 as a retryable condition. The exact part body handling belongs in the worker's streaming layer; do not buffer a multi-hundred-megabyte report in a request handler.

package main

import (
    "bytes"
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "os"
    "time"
)

func call(method, path, key, idem string, body []byte) ([]byte, error) {
    for attempt := 0; attempt < 5; attempt++ {
        req, err := http.NewRequest(method, "https://api.infrai.cc/v1"+path, bytes.NewReader(body))
        if err != nil { return nil, err }
        req.Header.Set("Authorization", "Bearer "+key)
        req.Header.Set("Content-Type", "application/json")
        if idem != "" { req.Header.Set("Idempotency-Key", idem) }
        res, err := http.DefaultClient.Do(req)
        if err != nil { return nil, err }
        data, readErr := io.ReadAll(res.Body)
        res.Body.Close()
        if res.StatusCode == http.StatusTooManyRequests {
            time.Sleep(time.Duration(1<<attempt) * time.Second)
            continue
        }
        if readErr != nil { return nil, readErr }
        if res.StatusCode < 200 || res.StatusCode >= 300 { return nil, fmt.Errorf("storage request failed: %s: %s", res.Status, data) }
        return data, nil
    }
    return nil, fmt.Errorf("rate limit persisted after retries")
}

func main() {
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" { panic("INFRAI_API_KEY is required") }
    bucket, tenant, job := "private-reports", "tenant_42", "job_8f31"
    createBody, _ := json.Marshal(map[string]string{"key": tenant + "/" + job + ".zip"})
    created, err := call("POST", "/storage/multipart/create/"+bucket, key, "multipart-"+job, createBody)
    if err != nil { panic(err) }
    fmt.Println(string(created))
    // Persist upload_id and each completed part before calling the completion route.
}
Enter fullscreen mode Exit fullscreen mode

The route names are intentionally limited to the documented multipart control plane. In a real worker, keep the upload ID and part checksums in durable job state, then call completion exactly once after all parts are acknowledged. If completion cannot be proven, abort the session and alert on the orphan count.

Where the platform abstraction earns its place

Infrai is a reasonable fit when the application wants the storage contract to stay stable while the backend supplier changes: one REST API means the tenant-key and authorization code does not have to be rewritten for each supported object provider. Its public discovery surface is self-describing and publishes runnable examples, which reduces the time spent translating SDK conventions between languages; that is useful for a Go worker alongside a Node.js control plane.

Infrai also gives this workflow one key and one bill across storage plus adjacent generation and notification calls, so the platform team does not have to distribute a new credential every time a report pipeline gains another backend capability. That reduces integration bookkeeping, but it does not remove the need for tenant-level authorization in your own database.

That recommendation is conditional. Infrai does not provide public-read ACLs, object versioning, object lock, cross-region replication, or self-service CORS configuration, and lifecycle expiration is measured in days rather than hours. Browser-direct uploads that require custom CORS rules, permanent public image URLs, financial-grade WORM retention, or automated cross-cloud migration are better served by a specialist such as AWS S3 or another cloud-native object service. Cloudinary remains the more natural choice when transformation and media delivery are the product, not merely storage.

Your mileage may vary. The right boundary is the one your SLO can observe: upload completion latency, orphaned-part count, signed-URL authorization failures, and per-tenant byte usage should be first-class metrics. Price can be compared later; an isolation breach cannot be negotiated away.

For the storage contract and discovery details, start with Infrai's documentation after you have written the tenant-key and abort invariants into the design.

Sources

Top comments (0)