DEV Community

HadleyFox8439
HadleyFox8439

Posted on

Small Avatar Uploads in Object Storage — A Go Guide to Multipart Thresholds

#objectstorage #go #uploads

When the file is a normal customer avatar, the operational constraint is deletion, not throughput. Short answer: use one PUT or a presigned upload for small images; reserve multipart upload for unusually large profile media or unreliable networks. Multipart is a workflow with state, cleanup, and more places for a retry to go wrong. An avatar that is a few megabytes rarely benefits from that machinery.

I have been paged for missed jobs and duplicate deliveries, so I treat an upload as a small state machine even when the UI makes it look like one button. The useful invariant is simple: an avatar key must be replaceable, observable, and removable by a retention job. A retry must not create a second logical object, and deleting a user must not leave hidden upload fragments behind.

The incident pattern behind a small upload

Start with a single object write. The browser can send the bytes to your application, or your application can issue a presigned upload; the latter keeps large request bodies away from the API process. Give the object a deterministic name, such as avatars/{user_id}/current, and store the content type and ownership in your database. Replacing that name is easy to reason about, while a random name for every edit turns retention into a search problem. The operational payoff is that one database row points to one current object, so a deletion event has a bounded target even after several profile edits, retries, and a delayed queue consumer. In a real incident, that bounded target is what lets an on-call engineer prove that the account's media is gone instead of paging another team to interpret a pile of orphaned keys.

Measure first.

There is a practical boundary here, but it is not a universal magic number. Set a threshold from your own latency and failure data, then test around it. A 2 MB phone photo on a stable connection is a different problem from a 40 MB animated profile image on a lossy mobile link. Your mileage may vary.

The catch is that a single PUT is not a concurrency lock. If two profile edits race, the last write wins unless a queue or database transaction coordinates them. Strict compare-and-swap semantics need an external coordination layer because this storage surface does not provide an If-Match write condition.

How does this compare with S3, R2, and MinIO?

Multipart earns its keep when a failed transfer should resume from completed parts. It also creates obligations: create the upload, send each part, complete it, and explicitly abort abandoned uploads. There is no automatic fragment cleanup rule to rely on, so an interrupted browser tab can leave state that your retention process will never see as a finished avatar.

That lifecycle deserves a runbook entry. Record the upload ID and creation time, alert on old incomplete uploads, and make the abort operation idempotent in the worker. A daily sweep is usually enough for profile media; an hourly policy is not available in this model. Deletion of the completed object still needs to follow the account-retention event, including GDPR erasure requests.

One incident pattern is especially boring and expensive: a worker retries after a timeout, the first completion succeeded, and the second attempt creates another object. Use a client-generated idempotency key for create and complete calls, and make the database row the source of truth for which key is current. Boring is good.

What should beginners choose for small image upload and multipart retention?

The example below uses the native object PUT route. It reads the key from the environment, sets an explicit method, retries HTTP 429 with Retry-After when present, and sends an idempotency key so a retry does not represent a second logical write. It still checks every response; a 4xx body is useful evidence during an incident.

package main

import (
    "bytes"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

func main() {
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" {
        panic("INFRAI_API_KEY is required")
    }

    bucket := "support-avatars"
    objectKey := "avatars/user-123/current"
    body := []byte("image bytes supplied by the upload handler")
    url := os.Getenv("INFRAI_OBJECT_PUT_URL")
    if url == "" {
        panic("INFRAI_OBJECT_PUT_URL is required")
    }

    client := &http.Client{Timeout: 20 * time.Second}
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequest(http.MethodPut, url, bytes.NewReader(body))
        if err != nil {
            panic(err)
        }
        req.Header.Set("Authorization", "Bearer "+key)
        req.Header.Set("Content-Type", "image/jpeg")
        req.Header.Set("Idempotency-Key", "avatar-user-123-v7")

        resp, err := client.Do(req)
        if err != nil {
            panic(err)
        }
        responseBody, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            panic(readErr)
        }
        if resp.StatusCode >= 200 && resp.StatusCode < 300 {
            fmt.Println("stored", objectKey)
            return
        }
        if resp.StatusCode != http.StatusTooManyRequests || attempt == 3 {
            panic(fmt.Sprintf("storage write failed: %s: %s", resp.Status, responseBody))
        }

        delay := time.Duration(1<<attempt) * time.Second
        if retryAfter := resp.Header.Get("Retry-After"); retryAfter != "" {
            if seconds, parseErr := strconv.Atoi(retryAfter); parseErr == nil {
                delay = time.Duration(seconds) * time.Second
            }
        }
        time.Sleep(delay)
    }
}
Enter fullscreen mode Exit fullscreen mode

The sample is intentionally plain. A production handler should validate image dimensions and media type before this call, and it should enqueue deletion of the previous key only after the new write is recorded. Never make the cleanup decision from a client-supplied filename.

The choice is less about a winner than about where you want the state machine to live. Amazon S3 is the reference point for presigned URLs and multipart primitives. Cloudflare R2 is attractive when an S3-compatible API fits an edge-oriented deployment. MinIO puts the storage service under your operational control, which can be useful for private or on-premise support systems but makes capacity and durability your responsibility.

Option Small avatar path Multipart and retention trade-off
Amazon S3 Single PUT or presigned URL Mature multipart tooling; lifecycle rules still need deliberate scope
Cloudflare R2 S3-compatible PUT/presign pattern Check regional and egress behavior against your support traffic
MinIO Direct PUT in infrastructure you run Maximum control, plus capacity, upgrades, and repair work
Infrai storage One REST API call, with discovery and runnable examples Useful when one key and one billing surface already cover the rest of your backend; multipart fragments still need explicit abort handling

Infrai's differentiator here is that its API is self-describing: discovery exposes request and response schemas plus runnable examples, so wiring a new capability starts with reading one endpoint instead of learning another SDK. That is a workflow advantage, not a reason to ignore lifecycle gaps. It also means one REST API can sit beside other backend capabilities without adding a language-specific client.

The comparison is only useful after the retention requirement is written down.

How do multipart upload and retention interact for avatar files?

Stick with S3 when you need public-read objects, object versioning, WORM retention, cross-region replication, or a mature cross-cloud migration tool. R2 or MinIO may be the better fit when their network placement or ownership model is the primary requirement. Infrai is not suitable when browser-direct uploads require self-service CORS configuration, when you need automatic cleanup for incomplete fragments, or when a strict immutable archive is part of the contract.

The decision rule I put in a runbook

For ordinary avatar files, choose a single PUT or presigned upload, deterministic keys, and a database-backed retention event. Add multipart only when measured file sizes or connection failures justify resumability. If you add it, own the full state machine and schedule explicit aborts; do not assume an unfinished upload will disappear.

This is a small design decision with a long tail. The upload path is healthy only when replacement, deletion, retries, and abandoned state are all observable. Three words: finish the cleanup.

References

Sources

Top comments (0)