Short answer: use a presigned URL for the browser-to-storage byte path, but keep snapshot creation, tenant authorization, retention assignment, and deletion behind a Python control plane; choose a proxy only when the backend must inspect or transform every byte before storage accepts it.
For an e-commerce backup system, upload speed isn't the first invariant. A backup is useful only if it belongs to the right tenant, has an immutable retention decision, and can be selected for restore without exposing another tenant's data. Direct upload changes where bytes travel. It must not move those decisions into the browser.
This is an architecture decision record for that boundary. The recommendation favors direct transfer for ordinary snapshot archives, with a narrow proxy path for cases that genuinely require synchronous byte inspection. The catch is operational: object arrival and snapshot readiness become separate events, so the system needs an explicit state machine rather than a hopeful upload complete toast.
An uploaded artifact is not a ready snapshot
Start with four invariants. A snapshot identifier is generated by the backend, bound to one authenticated tenant, limited to one object key, and assigned a retention class before any upload capability is issued. The browser can carry the bytes, but it can't choose a different tenant prefix, extend retention, or declare the snapshot restorable.
That distinction matters because a presigned URL is a time-limited capability. Anyone holding it may be able to perform the allowed operation until it expires, subject to the signed constraints. Treat it like an OTP: short-lived, scoped to one action, never logged in analytics, and useless after the intended transaction. Delivery discipline applies here too — a secret in a query string can leak through browser history, referrers, support screenshots, or observability tools unless those paths are deliberately controlled.
Keep the states boring and explicit: pending, uploaded, verified, ready, deleting, and deleted. An object appearing in storage moves nothing directly to ready. A trusted worker verifies the expected key and size, records a checksum when the storage contract exposes one with suitable semantics, and only then publishes the snapshot for restore. If verification doesn't finish before the intent expires, the object is quarantined from restore selection and cleaned up by policy.
Never trust the filename.
Use a generated object key such as tenant/{tenant_id}/snapshots/{snapshot_id}/archive, while preserving the original filename only as escaped metadata for display. This closes a surprisingly large class of path confusion and overwrite mistakes. It also makes deletion auditable: the database row points to one generated key, and the retention worker acts on that key rather than reconstructing it from user input.
The failure boundary is now clear. The control plane owns authorization and state. Object storage owns durable bytes. The browser owns transfer retries, but a retry uses the same snapshot intent only while that intent remains valid. A restore service reads only ready records and issues a separately scoped download capability after tenant authorization.
Model deletion and restore races first
Deletion is a workflow — not a single button. Consider a tenant whose 30-day snapshot reaches retain_until at 02:00 while an operator selects that same snapshot for restore at 01:59:59. The retention worker may already have read the row, and the restore service may already have authorized the tenant, so checking the clock twice doesn't settle ownership. Use a lease or transactional state transition instead: restore acquires a short lease on a ready snapshot, deletion changes the state to deleting only when no valid lease exists, and new restore requests stop at that state. Then request deletion using the exact stored key and confirm the resulting state before marking the row deleted. Repeating any step must be safe. An expired browser upload intent creates a related race; clean its object only after checking that no verified snapshot row owns the key. If storage configuration includes versioning or retention protection, define whether deletion means removing the current version, all versions, or merely making the snapshot unavailable to the application. A tenant purge should use the catalog as its work queue and reconciliation as a backstop, not a broad prefix deletion assembled from request text. These are different promises, and the audit log should say which one was fulfilled.
One key, one owner.
I'm not sure any retention design is complete until restore has been exercised against a representative archive. A checksum can show that bytes stayed consistent, but it can't prove the archive contains the tables, media references, and schema metadata needed by the current restore code. What resolves that uncertainty is a periodic restore drill in an isolated tenant, with timing and validation results recorded separately from upload success.
Assign retention before choosing the data path
Retention needs a recorded policy, not a scheduled guess. At snapshot creation, resolve the tenant's policy to concrete fields such as retain_until and deletion_mode, then store that decision beside the generated key. This prevents a later policy edit from silently changing the meaning of an existing backup. If policy changes must apply retroactively, make that a separate, authorized migration with an audit record.
Lifecycle rules in object storage are useful enforcement and cleanup tools. They can expire objects by age or transition them between storage classes. They do not replace the application catalog, because the restore picker still needs tenant ownership, snapshot status, and a comprehensible reason that an item is or isn't available. The catalog and lifecycle configuration should agree on the intended outcome, while reconciliation detects drift.
How should a web app choose presigned URL or proxy backend uploads?
Choose by asking where a decision about the bytes must happen. If the decision can occur after durable arrival, a presigned URL usually keeps the large payload off application workers and removes a duplicate network hop. If the request must be rejected based on streaming content before any byte reaches the destination, a proxy owns that inspection path and its capacity planning.
| Decision axis | Presigned URL path | Proxy backend path |
|---|---|---|
| Data path | Browser sends the archive directly to object storage | Browser sends the archive through application infrastructure |
| Latency shape | One wide-area upload hop after a small control request | An ingress hop plus an egress hop, with application buffering or streaming |
| Cost shape | Storage requests and transfer dominate | Storage costs remain, plus proxy compute and network capacity |
| Security boundary | Backend signs one constrained operation; post-upload verification gates restore | Backend can inspect the stream before forwarding, but also handles untrusted bytes |
| Failure handling | Upload completion and snapshot readiness are separate | Client connection, proxy capacity, and storage write share one longer critical path |
| Best fit | Large, opaque backup archives with asynchronous verification | Mandatory synchronous transforms, protocol translation, or inline inspection |
The presigned path is not automatically secure. Limit the method, object key, lifetime, and content constraints supported by the signing mechanism; configure cross-origin access for the exact web origins and methods in use; and make the finalization call idempotent. A client-provided MIME type is a hint, not proof. Archive expansion, malware scanning, schema checks, and restore rehearsal belong in a quarantined verification stage when they don't have to block ingestion.
A proxy isn't automatically safer either. It concentrates untrusted traffic on the application tier. Slow uploads occupy connections, retry storms multiply work, and a memory-buffering handler can turn a handful of large tenant backups into a capacity incident. Streaming avoids whole-file buffering, but backpressure, client disconnects, partial destination writes, and request timeouts still need explicit tests. Don't hide those concerns behind a framework's upload helper.
The practical beginner rule is short: direct for transport, backend for authority.
There are exceptions. A proxy is appropriate when regulation or policy says bytes must cross an inspection control before durable storage, when an archive must be encrypted with an application-held process before persistence, or when the destination lacks a browser-compatible signed operation. Your mileage may vary for very small snapshots because the extra hop may be operationally acceptable; measure with representative tenant sizes and network conditions rather than assuming the table decides everything.
The critical path in Python
The following code sketches the boundary, not a vendor SDK. sign_upload is a storage adapter that returns a constrained URL, while the repository enforces tenant ownership and legal state transitions. The browser never supplies an object key or retention deadline.
from dataclasses import dataclass
from datetime import datetime, timedelta, timezone
from typing import Protocol
from uuid import uuid4
class StorageSigner(Protocol):
def sign_upload(self, *, key: str, expires_in: timedelta) -> str: ...
@dataclass(frozen=True)
class UploadIntent:
snapshot_id: str
object_key: str
upload_url: str
expires_at: datetime
def create_upload_intent(
*, tenant_id: str, retention_days: int, signer: StorageSigner, repository
) -> UploadIntent:
if retention_days not in {7, 30, 90}:
raise ValueError("unsupported retention class")
now = datetime.now(timezone.utc)
snapshot_id = str(uuid4())
object_key = f"tenant/{tenant_id}/snapshots/{snapshot_id}/archive"
expires_in = timedelta(minutes=10)
repository.insert_snapshot(
snapshot_id=snapshot_id,
tenant_id=tenant_id,
object_key=object_key,
status="pending",
retain_until=now + timedelta(days=retention_days),
)
return UploadIntent(
snapshot_id=snapshot_id,
object_key=object_key,
upload_url=signer.sign_upload(key=object_key, expires_in=expires_in),
expires_at=now + expires_in,
)
def finalize_snapshot(*, tenant_id: str, snapshot_id: str, repository, storage) -> None:
snapshot = repository.get_owned_snapshot(tenant_id, snapshot_id)
if snapshot.status == "ready":
return
if snapshot.status != "pending":
raise ValueError("snapshot cannot be finalized from its current state")
metadata = storage.stat(snapshot.object_key)
repository.mark_uploaded(
snapshot_id=snapshot_id,
observed_size=metadata.size,
observed_at=datetime.now(timezone.utc),
)
repository.enqueue_verification(snapshot_id=snapshot_id)
Ten minutes and the three retention classes are example policy values, not universal defaults. Put them in versioned configuration, test boundary values, and make the signer adapter enforce the same key and method the database records. The finalize operation intentionally queues verification instead of marking the archive ready. Fast acknowledgement is useful; false readiness is not.
Observability should follow the snapshot identifier across intent creation, storage event processing, verification, restore, and deletion. Record byte counts and state-transition latency without recording signed URLs. Alert on old pending rows, verification backlog age, deletion lag beyond policy, and tenant authorization failures. For load tests, include disconnected clients, repeated finalization, expired capabilities, two concurrent restores, and a deletion attempt during an active restore lease.
The rejected option for the default path is a universal proxy. It remains the right choice when inline byte inspection is a hard requirement, and it may be simpler for tightly bounded payloads where the team already operates a streaming ingress tier. It is not suitable when multi-gigabyte tenant archives would compete with checkout, notification, or admin API traffic on the same workers. Keep the direct path when verification may happen asynchronously; keep the proxy when acceptance truly depends on a pre-storage decision.
Top comments (0)