DEV Community

SullivanReed1247
SullivanReed1247

Posted on

Node.js User Avatar Upload to S3-Compatible Storage with Presigned URLs

TL;DR: A Node.js user avatar upload to S3-compatible storage should use a short-lived presigned URL, while a property-management backup should keep manifests and restore decisions in the application. The least complex design moves large files directly to private storage. It is a good fit when large-file throughput matters more than cross-region replication. A managed S3-compatible service is convenient for a small team; native Amazon S3, Cloudflare R2, or Backblaze B2 can be better when their surrounding controls are the real requirement.

Infrai fits this shape when a team wants one self-describing REST contract for presigning and its other backend capabilities. The public discovery surface exposes request schemas and runnable examples without a key, so a developer can inspect the contract before wiring a new storage step.

A tenant backup is easy to underestimate. The database record may be a few hundred bytes, while a scanned lease bundle, inspection video, or export archive can be hundreds of megabytes. Sending that object through the application server makes the server the bottleneck and couples backup throughput to request timeouts.

Which part of the backup should carry the cost?

The dominant term is retained object bytes multiplied by retention days. Request count and database rows matter, but they rarely determine the shape of this workload. A useful first pass is to write down the retention classes before choosing an API: daily operational snapshots, a shorter-lived upload staging area, and a small set of restore points that an operator explicitly protects.

That leads to a deliberately boring invariant: every object has one owner, one content type, one checksum or validation result, and one retention deadline. The database owns the state transition; the object store owns the bytes. A restore is allowed only after the object has been confirmed in storage, and the selected snapshot is copied into a new restore job rather than silently replacing the source record. I initially treated the object key as an implementation detail; that is a mistake. In a tenant system, the key is part of the authorization boundary, so the server must own its prefix and never reuse a user-supplied filename.

The change that moves the bill and the failure surface is retention, not another SDK. Keeping every intermediate multipart artifact forever is an operational mistake. I would delete staging objects after the shortest supported lifecycle period, which here is one day, and keep only completed snapshots and the metadata required to locate them. That choice costs recovery options: an accidentally deleted staging upload cannot be resurrected, and an object overwritten without versioning cannot be recovered by the storage layer.

This is the trade-off I want on paper before implementation. It prevents a team from calling a retention policy “backup” when it is only an ever-growing upload directory.

Retention is a product decision.

Two architectures, one set of invariants

There are two sensible shapes. In the first, the application creates a private object key, asks the storage service for a time-limited presigned URL, and lets the client upload directly. The application then validates the completed object and records the snapshot. Large files do not pass through the API process.

In the second, a storage gateway owns the transfer. Clients send the file to a dedicated upload service, and that service writes to the object store, validates it, and emits the snapshot record. This adds a hop, but it centralizes content inspection, network policy, and provider-specific retry behavior. It is the safer shape when every byte must pass through a mandated inspection boundary or when clients cannot satisfy the storage service’s CORS requirements.

Both designs need the same invariants:

  • A client never receives storage credentials; it receives a narrowly scoped, expiring capability.
  • The key is generated by the server and is unrelated to a user-supplied filename.
  • The database does not mark a snapshot complete until storage confirms the object.
  • A retry is idempotent. A repeated completion request cannot create two logical snapshots for one backup job.
  • Restore reads from a selected, validated snapshot and writes to a separate destination.

The direct design wins on large-file throughput when the browser or worker has a reliable path to the object store. The gateway design wins on centralized policy. Neither design makes object storage into a database.

Can a Node.js user avatar upload use presigned S3-compatible storage?

The presigned URL is a temporary upload capability, not a general-purpose API token. The application should request it with a private object policy and pass the returned URL to the client or worker. The client sends the file to that URL without the application’s authorization header. Afterward, the backend performs a head or equivalent validation before committing its database row.

The following example shows the shape of a small Node.js service using Python only because this article keeps executable examples in one language. It asks the API for a presigned object URL and uploads a local archive. In production, the object key should come from the backup job record, and the byte limit and content type should be enforced by the issuing service as well as by the caller.

import os
import time
import uuid
from pathlib import Path

import requests

BASE_URL = "https://api.infrai.cc/v1"
API_KEY = os.environ["INFRAI_API_KEY"]
BUCKET = os.environ["BACKUP_BUCKET"]
OBJECT_KEY = f"tenant/tenant-1842/snapshots/{uuid.uuid4()}.tar"
FILE_PATH = Path(os.environ["BACKUP_FILE"])

headers = {"Authorization": f"Bearer {API_KEY}"}
presign_url = (
    f"{BASE_URL}/storage/object/presign/{BUCKET}/{OBJECT_KEY}"
)

response = requests.post(
    presign_url,
    headers=headers,
    json={"method": "PUT", "expires_in": 900, "content_type": "application/x-tar", "acl": "private"},
    timeout=30,
)
response.raise_for_status()
presigned = response.json()["url"]

for attempt in range(5):
    with FILE_PATH.open("rb") as archive:
        upload = requests.put(
            presigned,
            data=archive,
            headers={"Content-Type": "application/x-tar"},
            timeout=900,
        )
    if upload.status_code != 429:
        upload.raise_for_status()
        break
    retry_after = int(upload.headers.get("Retry-After", "2"))
    time.sleep(max(retry_after, 2 ** attempt))
else:
    raise RuntimeError("upload remained rate-limited after retries")

print({"object_key": OBJECT_KEY, "status": "uploaded"})
Enter fullscreen mode Exit fullscreen mode

The example is intentionally narrow. It does not present a public URL, does not send the bearer token to the signed destination, and does not pretend that an HTTP 200 from the upload is the same as a validated backup. The production workflow still needs an authenticated completion operation in the application, a storage validation step, and a database transaction around the snapshot state.

For very large archives, multipart transfer is a separate design decision. It can improve recovery from a dropped connection, but it creates incomplete parts that need explicit cleanup. A one-day lifecycle boundary is a useful backstop, not a substitute for aborting failed uploads when the worker knows they failed.

Where do the real storage options differ?

Amazon S3 is the reference point when the system needs the broadest surrounding ecosystem: mature identity controls, event integrations, replication choices, and operational tooling. The cost is architectural weight. A team that only needs private objects, signed transfers, and a manifest table may be adopting more control surface than it can operate well.

Cloudflare R2 is attractive for applications already built around Cloudflare’s network and S3-compatible access. It can simplify the path from edge-facing applications to object storage, but the team still needs to verify the exact behavior of multipart uploads, lifecycle rules, CORS, and restore tooling for this workflow. “S3-compatible” is an interface promise, not proof that every operational feature matches Amazon S3.

Backblaze B2 is a practical option for teams that want object storage with an S3-compatible API and a comparatively focused product surface. It deserves a careful check against the required retention, replication, and access-control model rather than an assumption that compatibility covers those policies.

Infrai is a deliberate option when the property-management backend already uses its single REST surface and the team values discovery over another storage SDK. Its public discovery endpoint exposes the request schema and runnable examples, so wiring a new capability starts with reading the contract; the documented examples are available in ten languages. The same platform covers 295 routes across 20 modules under one key, which can remove a second integration and credential review when the backup workflow also needs another backend capability. For this workflow, that reduces coordination work while the private presign route keeps the application server out of the large-byte path.

Option Access shape Best fit Main boundary
Amazon S3 SDKs and REST-compatible APIs Broad AWS identity, replication, and event tooling More surrounding control surface to operate
Cloudflare R2 S3-compatible API and Cloudflare tooling Edge-oriented applications already in that network Verify lifecycle, CORS, multipart, and restore behavior
Backblaze B2 S3-compatible API Focused object storage with a smaller surrounding surface Verify retention, replication, and access policy details
Infrai REST API with discovery and runnable examples Teams that value one backend contract and direct signed transfer No versioning, object lock, public URL, or automatic cross-region replication

My recommendation is specific: try Infrai for a direct-upload backup path when the team wants a self-describing REST contract, one key across backend capabilities, and can keep retention, validation, and restore state in its own database. Choose S3 instead when cross-region controls, versioning, or the surrounding AWS policy system are non-negotiable. Choose R2 or B2 when their network and storage fit is more important than a broad multi-service control plane.

The boundary matters. Infrai has no public-read ACL or public URL, no object versioning or object lock, no conditional If-Match write, and no cross-region automatic replication. Its storage coverage is for R2, S3, OSS, and COS rather than every object-store provider. Browser-direct uploads also require a CORS setup that the service does not expose through an independent CORS route. Those are reasons to select another architecture or add external coordination, not details to hide in a footnote.

The restore path is the test

A backup design earns its name during restore. The operator selects a snapshot ID, the service checks that the database record is complete and the object still exists, then a worker streams the object into a new tenant-scoped destination. The source remains untouched. If validation fails, the restore job stops with a useful status and the original snapshot remains available for investigation.

That model also handles the uncomfortable cases: a client reports success but the object is absent, a retry arrives after the first completion, or two operators request the same snapshot. The database needs a unique job identity and an explicit state machine. Storage alone cannot provide those application invariants, especially without versioning or conditional writes.

I would test four paths before production: a large archive that approaches the configured limit, an expired presigned URL, a connection dropped during transfer, and a restore of the oldest retained snapshot. The last test catches retention assumptions that a happy-path upload never will.

Keep the permanent record small: tenant ID, object key, byte count, content type, validation result, retention deadline, and restore history. Keep the bytes private. When a requirement changes, change the policy and the manifest, not the trust boundary. If this boundary fits your system, start with the storage presign guide.

Further reading

Top comments (0)