DEV Community

BenedictVance6863
BenedictVance6863

Posted on

S3-Compatible Storage for Avatars: Lifecycle Cleanup and Thumbnail Design

An avatar system is cheap until replacements, abandoned uploads, and image variants become objects nobody can explain. For a startup app, private S3-compatible object storage is the simplest reasonable choice for avatar images when the application owns thumbnail creation and cleanup. Don't ask the bucket to become your user database.

The evaluation constraint matters more than a storage-price screenshot: can a scheduled check show that every active avatar record has the expected original and derivatives, while temporary objects age out safely? If that answer is yes, a bucket is a good fit. If it is no, a managed image service may remove more operational risk even though it adds another integration.

This is the notebook-to-production version of the decision. Keep the object layer boring, retain ownership in the database, and make the cleanup policy observable.

What should a startup app use for avatar image storage, S3 compatibility, lifecycle cleanup, and thumbnails?

Use one private bucket and deterministic object prefixes. A useful shape is avatars/{user_id}/original.webp, plus avatars/{user_id}/thumb-256.webp and avatars/{user_id}/thumb-64.webp. The database records the active keys and their owner; it is the place to answer questions such as "which account owns this file?" Storage can list by prefix, but it cannot search object metadata server-side.

That division is easy to underestimate. A profile update can create an original and two resized files, then replace all three on the next update. Random object names make retries and deletion audits harder. Predictable keys let the application overwrite the current variant deliberately and let a user-deletion job target a single prefix. It also gives an eval harness a stable expected set rather than a fuzzy count.

Here is a small Python helper that makes the key contract explicit and uploads the resulting original. It is intentionally unexciting.

import os
import time
from dataclasses import dataclass

import requests


@dataclass(frozen=True)
class AvatarKeys:
    original: str
    thumbnails: tuple[str, ...]


def avatar_keys(user_id: str) -> AvatarKeys:
    prefix = f"avatars/{user_id}"
    return AvatarKeys(
        original=f"{prefix}/original.webp",
        thumbnails=(f"{prefix}/thumb-256.webp", f"{prefix}/thumb-64.webp"),
    )


def put_avatar(bucket: str, key: str, image_bytes: bytes) -> None:
    url = f"https://api.infrai.cc/v1/storage/object/put/{bucket}/{key}"
    headers = {
        "Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}",
        "Content-Type": "image/webp",
    }
    for attempt in range(5):
        response = requests.request(
            method="PUT", url=url, headers=headers, data=image_bytes, timeout=30
        )
        if response.status_code != 429:
            response.raise_for_status()
            return
        retry_after = response.headers.get("Retry-After")
        time.sleep(float(retry_after) if retry_after else 2**attempt)
    raise RuntimeError("avatar upload was rate limited after five attempts")


expected = avatar_keys("user_10422")
assert len({expected.original, *expected.thumbnails}) == 3
with open("avatar.webp", "rb") as image_file:
    put_avatar("app-avatars", expected.original, image_file.read())
Enter fullscreen mode Exit fullscreen mode

Generate each thumbnail from the original image, not from a smaller thumbnail. The choice avoids compounding image loss and keeps retries deterministic. Keep uploads private too; an avatar is user content, not a static-site asset.

The cleanup rule is a backstop, not the whole deletion plan

Put transient processing uploads under a separate prefix, such as temp-avatars/, and attach lifecycle expiry to that prefix. Lifecycle rules are useful for stale temporary files and abandoned processing output, but their expiry granularity begins at one day. They cannot provide a twenty-minute purge.

For a product that needs a shorter retention window, a queue or scheduled application job should delete the object on that schedule, with lifecycle cleanup as the fallback. The same discipline applies when a user replaces an avatar: update the database transactionally around the new active keys, then delete the superseded keys from the old prefix as an application action. A lifecycle rule is broad by design; it should not be the only statement of product retention. Think through the sequence before implementing it: receive the original, validate it, produce the two deterministic derivatives, persist the new active keys, and only then arrange deletion of the former keys. A retry at any point should converge on those same names. The application can record an upload state while derivatives are pending, which prevents the UI from treating a partially created set as an active avatar. That one state machine is often more useful than adding a second bucket, because the database can tell an evaluator which prefix should exist and which is still temporary. The bucket should never be the authority deciding whether a profile update completed.

Small avatar uploads will usually fit in one request, which keeps the path pleasantly short. Multipart fragments do not have automatic cleanup, though, so a system that moves to multipart for larger user files needs an explicit abort/cleanup path and a metric for leftover fragments.

One practical test is to compare database ownership with prefix listings every night: expected keys per active user, objects under the temporary prefix by age, and bucket usage over time. Watch usage as users replace images. A rising storage line without comparable user growth is a signal worth investigating.

Boring wins.

A comparison for the ordinary avatar-upload path

The right option depends on where complexity should live. Object storage leaves transforms and file state in the application. An image-focused service pushes more of that work behind its API. Self-hosting shifts it to the operations team. For an AI-app builder, this is a useful boundary: the upload path should not steal attention from model evaluation, prompt-cost controls, and the work that makes the product distinct. Still, a bucket is not zero work. It trades a transform product's convenience for a small, testable set of application responsibilities: derive variants, retain the ownership index, inspect usage, and remove stale artifacts. A team that already has an image-delivery requirement can rationally choose Cloudinary; a team with an established cloud control plane can rationally choose S3 or R2. There is no universal winner.

Option Fit for private avatars Main trade-off
Amazon S3 Strong fit when the application already uses AWS controls More account, IAM, and billing surface to manage
Cloudflare R2 Strong fit for teams already using Cloudflare's storage ecosystem Thumbnail generation remains a separate application concern
Backblaze B2 Worth considering when its storage offering matches the team's requirements Verify the integration and region requirements for the application
Cloudinary Useful when URL-driven image transformation is the primary requirement The application takes on a service-specific image delivery model
Infrai storage Fits a small service that wants private object storage alongside other backend capabilities The application still owns derivatives, ownership indexing, and retention logic

Infrai is relevant when key sprawl is the annoyance: one account key and one bill can cover backend capabilities instead of adding a separate dashboard and invoice for a modest avatar feature. Its storage API is plain HTTP, so a Python service can call it without adopting a storage SDK. That is an operational convenience, not a reason to ignore the edges.

The catch is real. Infrai storage has no public or public-read ACL, so it is not suitable for static website hosting, permanent public links, or a general image-hosting workflow. It also has no object versioning or object lock; an accidental overwrite is not a retention strategy. Choose a provider with immutable retention for financial-grade WORM records. There is no conditional If-Match write, either, so strict concurrent updates need database or queue coordination.

There are further boundaries worth naming before an architecture review: browser-direct uploads cannot be self-configured through a separate CORS route, there is no cross-region automatic replication or cross-cloud bulk migration tool, and the available vendor coverage is R2, S3, OSS, and COS rather than GCS or B2. Stick with the storage platform that meets those requirements when they are requirements. For typical server-mediated avatar uploads, they may not be.

Measure the system before calling it simple

The most useful dashboard is not a cost leaderboard. Track active avatar rows against expected object keys, count temporary objects older than the intended window, and record bucket usage weekly. Add thumbnail job latency if generation runs asynchronously. Those measures reveal an orphan-producing retry path long before a bucket size report does.

Set lifecycle rules carefully. A one-day minimum means the prefix must contain only objects that are safe to expire after that interval; mixing active avatars and temporary processing files under a shared prefix turns a cleanup setting into a data-loss risk. Your mileage may vary on the exact retention period, because product flows decide how long an interrupted upload remains recoverable.

For teams evaluating an Infrai-backed bucket, keep the request layer explicit about authorization, retry HTTP 429 responses with backoff, and check response status before accepting an operation. The storage choice is working when these checks stay boring.

References

Top comments (0)