Short answer: for large tenant backups, let the browser transfer bytes directly to private object storage, while a Python API and Postgres control the document identity, authorization, and restore state. Signed URLs are temporary capabilities, not your file model.
That decision matters in property management because a backup is rarely just a file. It belongs to a tenant, a building, a retention policy, and a restore request. A 4 GB lease archive should not occupy a Python worker while a user watches a progress bar. The application should arrange the transfer, then get out of the byte path.
The example below uses a generic S3-compatible client. The exact storage product can change; the control-plane rules should not.
Reliability failures in tenant backup workflows
The first failure is usually ownership, not storage. A tenant backup can exist as bytes while the product has no trustworthy answer to “which tenant may restore it?” The second is false completion: a browser reaches the end of its progress bar, the network response is lost, and the API cannot tell whether it should create a new backup or confirm the old one.
Measure first.
For a property-management SaaS, I use this decision table before choosing an upload path:
| Constraint | Direct browser transfer | Service-mediated transfer |
|---|---|---|
| Large-file throughput | Keeps payload bandwidth off the API workers | Makes the service carry every byte |
| Immediate content inspection | Happens after object verification | Can inspect before storage |
| URL revocation | Short expiry limits exposure | The service can decide on every read |
| Operational state | Needs upload and reconciliation records | Still needs records for retries and restore audit |
The table is intentionally dull. Dull is good for backup control paths.
No magic.
Consider a resident-management export that starts at 09:00, reaches 98 percent at 09:18, and loses its connection before the completion request reaches the API. The browser cannot safely conclude that the object is absent, and the API cannot safely create a second backup just because the client timed out. The durable answer is a database row that still says pending, an object verification step that can be repeated, and a completion transition protected by a row lock or an equivalent concurrency rule. If verification finds the object, the first completion changes the row to ready; if it does not, the stale row is eligible for cleanup. A retry therefore converges on one backup ID instead of producing two snapshots with similar filenames, two downstream extraction jobs, and an operator wondering which copy is authoritative. This is also why a UI message such as “upload finished” should be derived from server state, not from the last browser progress event. The user-facing detail is small; the recovery behavior is the product.
Start with two records: an immutable backup ID in Postgres and an object key that the server generates from that ID. The browser sends metadata to the API, the API checks the tenant and user, creates a pending row, and returns a short-lived signed upload URL. The browser then uploads directly to storage. Afterward, the API verifies the object and moves the row to ready.
The download path reverses the order. The API authorizes the restore request against the Postgres row, confirms that the backup is restorable, and returns a new signed download URL. A browser must never choose an arbitrary bucket key and ask the API to sign it. That makes tenant isolation a string-construction problem, which is a poor place to put a security boundary.
Here is a compact FastAPI-shaped implementation. Authentication is represented by a dependency because identity and session validation belong to the application. The important pieces are the server-owned key, the database state, and the direct transfer.
import os
import uuid
from datetime import datetime, timezone
import boto3
import psycopg
from fastapi import Depends, FastAPI, HTTPException
from pydantic import BaseModel
app = FastAPI()
storage = boto3.client("s3", endpoint_url=os.environ["OBJECT_STORAGE_ENDPOINT"])
bucket = os.environ["BACKUP_BUCKET"]
class BackupRequest(BaseModel):
filename: str
content_type: str
size_bytes: int
def database():
return psycopg.connect(os.environ["DATABASE_URL"])
def current_user_id() -> str:
# Replace this boundary with the application's verified session.
return "authenticated-user"
def tenant_for_user(user_id: str) -> str:
# Resolve membership from the application's authorization layer.
return "tenant-42"
@app.post("/tenants/{tenant_id}/backups/start")
def start_backup(tenant_id: str, request: BackupRequest, user_id: str = Depends(current_user_id)):
if tenant_for_user(user_id) != tenant_id:
raise HTTPException(status_code=403, detail="tenant access denied")
if request.size_bytes <= 0:
raise HTTPException(status_code=422, detail="size must be positive")
backup_id = str(uuid.uuid4())
object_key = f"tenants/{tenant_id}/backups/{backup_id}/original"
created_at = datetime.now(timezone.utc)
with database() as db:
db.execute(
"""INSERT INTO tenant_backups
(id, tenant_id, object_key, filename, content_type,
size_bytes, status, created_at)
VALUES (%s, %s, %s, %s, %s, %s, 'pending', %s)""",
(backup_id, tenant_id, object_key, request.filename,
request.content_type, request.size_bytes, created_at),
)
upload_url = storage.generate_presigned_url(
"put_object",
Params={
"Bucket": bucket,
"Key": object_key,
"ContentType": request.content_type,
},
ExpiresIn=900,
HttpMethod="PUT",
)
return {"backup_id": backup_id, "upload_url": upload_url}
@app.post("/tenants/{tenant_id}/backups/{backup_id}/complete")
def complete_backup(tenant_id: str, backup_id: str, user_id: str = Depends(current_user_id)):
if tenant_for_user(user_id) != tenant_id:
raise HTTPException(status_code=403, detail="tenant access denied")
with database() as db:
row = db.execute(
"""SELECT object_key, status FROM tenant_backups
WHERE id = %s AND tenant_id = %s FOR UPDATE""",
(backup_id, tenant_id),
).fetchone()
if row is None:
raise HTTPException(status_code=404, detail="backup not found")
if row[1] == "ready":
return {"backup_id": backup_id, "status": "ready"}
if row[1] != "pending":
raise HTTPException(status_code=409, detail="backup is not uploadable")
storage.head_object(Bucket=bucket, Key=row[0])
db.execute(
"UPDATE tenant_backups SET status = 'ready' WHERE id = %s",
(backup_id,),
)
return {"backup_id": backup_id, "status": "ready"}
@app.get("/tenants/{tenant_id}/backups/{backup_id}/restore")
def restore_backup(tenant_id: str, backup_id: str, user_id: str = Depends(current_user_id)):
if tenant_for_user(user_id) != tenant_id:
raise HTTPException(status_code=403, detail="tenant access denied")
with database() as db:
row = db.execute(
"""SELECT object_key FROM tenant_backups
WHERE id = %s AND tenant_id = %s AND status = 'ready'""",
(backup_id, tenant_id),
).fetchone()
if row is None:
raise HTTPException(status_code=404, detail="restorable backup not found")
download_url = storage.generate_presigned_url(
"get_object",
Params={"Bucket": bucket, "Key": row[0]},
ExpiresIn=300,
HttpMethod="GET",
)
return {"backup_id": backup_id, "download_url": download_url}
The code intentionally does not accept a client-supplied object key. It also makes completion idempotent: a second completion call returns the existing ready state instead of creating another row. That small detail saves a surprising amount of cleanup work when a mobile browser retries after a slow response.
What governance does large-file throughput require?
It changes the boundary, not the ownership model. The application should issue control requests and the browser should carry the payload. XMLHttpRequest exposes upload progress events in browsers, which gives a UI a useful signal without routing the file through the API process. The transfer still needs a timeout policy and a visible cancel state; a progress bar alone doesn't prove that the object is complete.
For genuinely large archives, use multipart transfer supported by the chosen storage system. The API creates the database intent and an upload session, the browser sends parts, and the API completes the session only after it has enough information to verify the final object. Keep the multipart identifiers and part metadata in a short-lived upload record, or use a queue-backed coordinator. Do not infer completion from the browser's last progress event.
There is a practical throughput trap here. A single signed PUT is easy to reason about, but a dropped connection near the end can force the browser to restart the whole object. Multipart transfer allows failed parts to be retried, at the cost of more state, more requests, and a cleanup policy for abandoned sessions. Choose the simpler path for small files; choose multipart when retransmitting a complete tenant archive is operationally painful.
The API should enforce the declared size before issuing a URL and compare the resulting object metadata at completion. That check is not a substitute for content validation. A backup importer still needs to inspect the archive type, validate its manifest, and reject content that does not belong to the expected tenant or backup format.
I put a 413 case in the test matrix, along with a connection drop at 98 percent. Those tests expose a different class of failure than a happy-path upload: one is a policy rejection, the other is a resumability problem. Keep them separate in metrics. Otherwise an operator sees “uploads are failing” and has no clue which decision to revisit.
Implementation for browser upload of private documents
Object storage is good at storing bytes and returning them under a key. It is not the right place to answer product questions such as “show the latest verified snapshot for this tenant,” “who requested this restore,” or “which parser version produced the manifest.” Those questions need transactions, indexes, and relationships.
A useful row includes the tenant ID, backup ID, server-generated object key, original filename, declared content type, declared and observed size, status, revision number, creation time, and retention deadline. Add a restore-request table when restores need approval or an audit trail. The byte object and the business record then have different responsibilities, but they share one stable ID.
Use an explicit state machine. pending means an upload was authorized but not verified. ready means the object passed the completion checks. expired means the product no longer permits a restore. restoring and restored describe the downstream job, not the upload itself. A worker can claim a restore with a database transition and retry it without issuing a new download capability on every internal attempt.
The hard case is deletion. A tenant may cancel a pending upload, an operator may shorten retention, or a restore may be rejected after the metadata has already been created. Mark the database state first according to the product rule, enqueue object deletion, and reconcile the result later. A periodic reconciliation job should find pending rows with old timestamps and objects with no matching row. It should produce an operator-visible report before it removes anything.
Never use a mutable filename as the identity of a revision. tenant-42/leases/latest.zip is convenient until two exports race. An ID-based key gives each revision its own address and lets Postgres decide which revision is current. That makes restore selection explicit rather than dependent on upload timing.
How does evaluation test signed download URLs?
A signed URL grants temporary access to a particular object. Treat it like a bearer credential during its lifetime. Keep its expiry short enough for the expected transfer, authorize before signing, and avoid placing it in durable logs, analytics payloads, or support tickets. The storage provider's documentation should be the authority for signing details and expiry behavior.
The URL must be scoped to the server-selected key and the intended operation. A download endpoint should not accept a raw bucket name, arbitrary prefix, or caller-selected HTTP method. For a restore, the API should check tenant membership, backup status, retention, and any approval rule in one decision before it mints the URL.
This is a capability design, so revocation deserves honest treatment. Invalidating a Postgres row does not necessarily invalidate a URL that was already issued. Keep expiries short, avoid issuing URLs before the final authorization point, and use a proxy download when immediate revocation is a hard requirement. A proxy gives stronger control but puts large-file bandwidth and connection management back on your service. That is a real trade-off.
Private browser uploads also require browser-origin configuration at the storage boundary. The allowed origin, method, and headers should be reviewed as deployment configuration, with credentials and origins kept narrow. The application still owns authorization; a permissive browser policy does not grant a tenant access by itself.
I'm not sure a direct upload is the right answer for every restore workflow. If files must be inspected byte-by-byte before storage, or if compliance requires every download to pass through a policy engine, a service-mediated stream may be the better fit. Your mileage will vary with network topology and retention rules, so measure the full transfer path with representative archive sizes instead of extrapolating from a 20 MB fixture.
Migration criteria for direct restore transfer
Test transitions, not just endpoints. A useful suite starts an upload for tenant A, attempts completion from tenant B, repeats completion, replaces a revision, lets a URL expire, and requests a restore after the retention deadline. It also verifies that a database timeout does not cause the client to invent a new backup ID. The expected outcome is a stable state and an actionable event for each case.
For observability, log the backup ID, tenant ID, state transition, byte count, elapsed transfer time, and request correlation ID. Do not log signed URLs. Record storage request identifiers when the storage system provides them, but keep them associated with the internal backup ID rather than exposing them in the UI. Alert on old pending rows, unusually high restart rates, verification-size mismatches, and restore jobs that exceed their expected duration.
The operational checklist belongs in prose. Before release, prove that row creation precedes URL issuance; authorization is checked on both start and restore; completion can be repeated safely; the observed object matches the declared contract; abandoned multipart sessions are cleaned up; and each revision gets a new key. Then run a load test with the largest realistic tenant archive from a browser network, because application-only benchmarks miss CORS, client memory, and last-mile behavior.
Keep parsing, indexing, and AI extraction after ready. That gives an eval harness a stable input revision and makes prompt-cost accounting visible: a failed extraction can be retried without re-uploading bytes, and a changed parser can be compared against the same backup. It also keeps the upload request small enough to reason about when the archive is measured in gigabytes.
The catch is that this pattern is not suitable when the storage system cannot provide the required signed operations, when multipart cleanup cannot be owned, or when every byte must pass through an application policy layer. In those cases, use a mediated transfer or a platform-native workflow with those controls, even if it costs more application bandwidth. Direct browser transfer is a decision about failure ownership as much as throughput.
Top comments (0)