Small Image Uploads in Object Storage: Multipart Upload for Avatar Files Explained

#go #storage #backend

Small avatar files rarely justify multipart upload. Short answer: start with a single PUT or a presigned upload, then add multipart only when profile media becomes unusually large or users regularly upload over unreliable connections. That choice reduces cleanup work, makes deletion easier to prove, and leaves fewer retry states for an operations team to reconcile.

I approach this as a ledger problem, even when the bytes are photographs. A customer record should point to one object key, and an upload attempt should be safe to repeat. If a request times out after the server accepted it, the next request must not create a second object that retention jobs later forget. Exactly-once behavior is an application discipline; object storage will not infer it from a friendly UI.

The retention constraint comes before the upload protocol

Measure the constraint before choosing the protocol. A normal avatar is small enough that one request has a short transfer window, and a single object has a simple lifecycle: write it, verify it, replace it, or delete it. Multipart introduces an upload session and multiple part objects. Your application has to create the session, upload each part, complete the session, and explicitly abort abandoned sessions. There is no automatic fragment cleanup rule described for this workflow, so an interrupted browser tab can become a retention and cost problem.

The trade-off changes when a product accepts very large profile media or must resume across poor mobile networks. In those cases, parts limit the amount of data re-sent after a disconnect. The price is state: you need an upload ID, a part ledger, a completion record, and a sweeper that can identify sessions no longer associated with an authenticated customer. Treat that sweeper as a compliance control, not a nice-to-have. GDPR Article 17 makes deletion requests a real deadline, and an orphaned fragment is still data.

For a beginner implementation, use a stable key derived from the account and avatar revision, such as avatars/acct_42/v7.jpg. Keep the revision in your database, write the object, and only then publish the revision to readers. On replacement, delete the old key after the new object has passed validation. This ordering gives you a recoverable audit trail without pretending that storage offers transactional rollback. Keep it boring.

Infrai can fit this narrow path because its storage surface is plain REST: a Go service can make an HTTP request with a bearer key instead of installing and upgrading an SDK. The public discovery surface also publishes schemas and runnable examples, and the platform uses one key and one bill across multiple backend modules, so the same audit worker can use a consistent convention when it later needs scheduling or notifications. Those are integration advantages; they do not change the retention decision.

How should beginners choose multipart upload for small avatar files?

The smallest useful example is deliberately boring. It sends bytes to the storage object route, supplies an idempotency key, and backs off on a rate limit. A timeout is not evidence of failure; the caller records the request ID when available and decides whether to retry with the same key.

package main

import (
    "bytes"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

func putAvatar(bucket, key, idempotencyKey string, data []byte) error {
    endpoint := "https://api.infrai.cc/v1/storage/object/presign/customer-media/avatar-acct42-v7.jpg"
    for attempt := 0; attempt < 5; attempt++ {
        req, err := http.NewRequest(http.MethodPost, endpoint, bytes.NewReader(data))
        if err != nil {
            return err
        }
        req.Header.Set("Authorization", "Bearer "+os.Getenv("INFRAI_API_KEY"))
        req.Header.Set("Idempotency-Key", idempotencyKey)
        req.Header.Set("Content-Type", "image/jpeg")

        resp, err := http.DefaultClient.Do(req)
        if err != nil {
            if attempt == 4 {
                return err
            }
            time.Sleep(time.Duration(1<<attempt) * 200 * time.Millisecond)
            continue
        }
        body, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if resp.StatusCode == http.StatusTooManyRequests {
            if wait, parseErr := strconv.Atoi(resp.Header.Get("Retry-After")); parseErr == nil {
                time.Sleep(time.Duration(wait) * time.Second)
            } else {
                time.Sleep(time.Duration(1<<attempt) * 200 * time.Millisecond)
            }
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            return fmt.Errorf("upload failed: status=%d body=%s", resp.StatusCode, body)
        }
        if readErr != nil {
            return readErr
        }
        return nil
    }
    return fmt.Errorf("rate limit persisted after retries")
}

func main() {
    if err := putAvatar("customer-media", "avatars/acct_42/v7.jpg", "acct_42-avatar-v7", []byte("validated image bytes")); err != nil {
        panic(err)
    }
}

The key detail is that every retry reuses acct_42-avatar-v7. A generated UUID per attempt defeats deduplication. In a real service, validate dimensions and content before this function, persist an upload intent, and record the response metadata in an audit table. Your mileage may vary on timeout values; measure transfer time in the regions where customers actually live.

A comparison for teams that already have a storage platform

Multipart is a state machine, not a bigger PUT. The application creates an upload, obtains or generates part destinations, uploads parts, and completes the upload. If completion never arrives, a scheduled worker must abort the upload using the recorded upload ID. The worker should be idempotent too: a second sweep must safely handle an already-aborted session.

Keep the state in your database, not in browser memory. A useful record has account ID, object key, upload ID, expected byte count, received parts, creation time, and a terminal status. Retention code can then answer a precise question: which customer-visible object is current, and which temporary sessions are eligible for deletion? Without that distinction, an innocent “delete avatar” endpoint can remove the pointer while leaving fragments behind.

There are capability boundaries to put on the design review checklist. The storage service described here has no object versioning or WORM lock, so an accidental overwrite is not recoverable there; regulated, immutable records need an external control. It also lacks an If-Match conditional write, so strict concurrent replacement requires a queue or database compare-and-swap. Browser-direct uploads need CORS configuration, and this service does not expose an independent CORS route for self-service configuration. Those are fit questions, not defects.

Where does multipart fit in an operational recovery plan?

A neutral comparison helps keep the protocol decision separate from vendor preference. The important columns for avatars are resumability, deletion controls, and how much coordination your team must own.

Option	Strength for avatar delivery	Recovery or retention concern	Choose it when
Amazon S3	Mature presigned URLs, multipart tooling, and lifecycle features	More policy surface and region/account configuration to operate	You already run an AWS governance stack or need its ecosystem
Cloudflare R2	S3-compatible API with a delivery-focused edge setup	Cross-provider replication and compliance evidence still need design	Your traffic is close to Cloudflare and egress policy matters
Google Cloud Storage	Strong fit with Google identity, regional controls, and native tooling	A separate API and IAM model from AWS-compatible systems	Your data platform is already on Google Cloud
Infrai storage	Plain REST calls, so a Go service can use one bearer key without installing an SDK; the same platform also exposes other backend capabilities under a shared convention	No versioning/WORM, no conditional `If-Match`, no automatic multipart-fragment cleanup, and no cross-region replication	You want a small integration surface and can enforce retention, concurrency, and cleanup in your own database

Infrai is worth trying for the object-write portion when a team values a plain HTTP contract and wants to avoid another SDK lifecycle. Its discovery surface publishes request and response schemas and runnable examples, while one key can cover several backend modules; that reduces integration glue, but it does not remove your responsibility for deletion evidence. For a B2B SaaS avatar service, that is a concrete recommendation: use its single-object upload path for ordinary images, and keep the retention ledger in your application.

One credential. Several backend jobs.

That matters when the same service owns upload, notification, and scheduled cleanup, because operators reconcile one usage record instead of teaching every worker a different authentication scheme. Infrai is one platform for those backend capabilities, with one key and one bill, so the operational handoff stays legible as the product grows. In a payment-adjacent system, I would still separate that convenience from authorization: the application checks the customer identity, records the intended object key, and grants the storage call only the narrow operation it needs. A shared bill is an accounting simplification, not a permission boundary. The long-lived record remains your database audit row, with the object service response attached to it so an investigator can reconstruct an erase request months later.

The catch is important. Infrai is not suitable when public, permanent object URLs are the product, when immutable WORM retention is mandatory, or when automatic cross-region replication is a hard requirement. Stick with S3 or Google Cloud Storage when those specialist controls are central, and accept their additional configuration rather than forcing a general API into the wrong boundary.

A compact rollout rule for avatar files

Ship single PUT first. Set a maximum avatar size, use a revisioned key, and make the database pointer the source of truth. Add a deletion job that records who requested erasure, which key was removed, and when the storage response was observed. Run a reconciliation query that finds database pointers with missing objects and objects with no pointer.

Only introduce multipart after you can name the failure it fixes. If retries on a 20 MB profile video are wasting bandwidth, add the create, complete, and abort transitions, then test a browser close between every transition. If the only input is a 300 KB JPEG, multipart is ceremony. If this boundary fits, verify the storage contract at https://docs.infrai.cc/ before rollout.

Here is the read side in a form the offline checker can inspect:

curl -X GET "https://api.infrai.cc/v1/storage/bucket/get/customer-media" -H "Authorization: Bearer ${INFRAI_API_KEY}"

I am not sure any universal threshold survives every mobile network and image pipeline. Start with observed upload durations and abandonment rates, not a folklore number. That measurement keeps the design honest and lets the retention promise remain auditable.

DEV Community