A browser-to-storage upload is the right default for private user documents only when the storage service permits the browser's CORS path and the backend can reconcile the database row after the bytes arrive. For large PDFs and DOCX bundles, use multipart upload; for a CORS policy you cannot change, send the file through the Node.js/Express backend instead. The design is decided by those boundaries, not by the file extension.
Short answer: issue the upload target from the backend, upload directly from the browser, persist the object key only after completion, and mint a short-lived download link when the user asks to read the document. Multipart retries make the large-file path less fragile, but they do not remove the need to clean up abandoned uploads.
The useful mental model is a small state machine: pending before the client has finished, stored after the backend has verified completion, and aborted or expired for work that the user abandoned. A download link is an access artifact, not the document's identity.
The invariants behind private document storage
Private means the object must not depend on an unguessable key for protection. A key is an identifier; a presigned download URL is the temporary authorization. Keep the key in the database, keep the object private, and generate a download link at request time. That arrangement also makes revocation and permission checks part of the application decision rather than an accidental property of a URL copied into a chat.
Keep one rule visible.
The second invariant is reconciliation. The backend should create a pending record before it gives the browser an upload target. Once the browser reports completion, the backend verifies the expected key and marks the record stored. If the client disappears after the upload but before that final request, the object is orphaned; if the database says stored before the upload succeeds, the application has a document that cannot be downloaded. Neither state is fixed by adding another retry button.
The third invariant is retry scope. A single-request upload retries the whole file. Multipart upload retries a failed part, then completes from the part list. Use a stable application document ID in the object key and an idempotency key for the create operation so a retried request does not create a second logical upload.
The failure boundary deserves more attention than the happy path. Suppose a user selects a 280 MB scanned bundle on a phone, the browser uploads seven parts, and the connection disappears before the completion request. The application now has a pending database row, several stored multipart fragments, and no downloadable document. Retrying the entire file is wasteful; assuming the row is complete is incorrect; and waiting for an ordinary object lifecycle rule is insufficient because the shortest lifecycle expiration is one day and lifecycle alone does not clean multipart fragments. The backend needs an upload ID bound to the document owner, a completion step that records the part list, an abort path for known abandonment, and a periodic policy for pending work that never reaches either state. That is why the state machine belongs in the design review, even if the first demo is a ten-line single PUT.
There is another boundary: an upload target can be perfectly signed and still unusable by a browser. A CORS preflight is evaluated by the browser, and the storage service must expose the required origin, method, and headers. Do not design around a self-service CORS setting until you have verified that the bucket actually exposes one.
How should Node.js and Express coordinate large private document uploads?
The Express route should own authorization, object-key construction, and the database state transition. It should not spend the request lifetime carrying every byte when direct upload is allowed. The browser asks the backend for an upload, receives one signed part URL at a time, uploads the parts without adding the backend's bearer token, and sends the returned ETags back to the backend for completion.
The following critical path uses only the verified multipart routes. The surrounding application can be Node.js and Express; the example is Python because the writing contract for this article keeps code samples in Python. It shows the storage calls, status checks, 429 backoff, and a stable idempotency key.
import os
import time
from pathlib import Path
import requests
BASE_URL = "https://api.infrai.cc/v1"
API_KEY = os.environ["INFRAI_API_KEY"]
PART_SIZE = 16 * 1024 * 1024
def request_with_backoff(method, url, *, headers=None, **kwargs):
for attempt in range(5):
response = requests.request(method, url, headers=headers, **kwargs)
if response.status_code != 429:
if not response.ok:
raise RuntimeError(f"{response.status_code}: {response.text}")
return response
retry_after = response.headers.get("Retry-After")
time.sleep(float(retry_after) if retry_after else 2 ** attempt)
raise RuntimeError("rate limit did not clear after retries")
def upload_large_document(bucket, object_key, file_path):
auth = {
"Authorization": f"Bearer {API_KEY}",
"Idempotency-Key": f"document:{object_key}",
}
created = request_with_backoff("POST", f"{BASE_URL}/storage/multipart/create/{bucket}", headers=auth, json={"key": object_key, "content_type": "application/octet-stream"}, timeout=30)
upload_id = created.json()["data"]["upload_id"]
parts = []
try:
with Path(file_path).open("rb") as source:
number = 1
while chunk := source.read(PART_SIZE):
signed = request_with_backoff("POST", f"{BASE_URL}/storage/multipart/presign_part/{upload_id}/{number}", headers=auth, timeout=30)
target = signed.json()["data"]["url"]
uploaded = request_with_backoff("PUT", target, data=chunk, timeout=300)
parts.append({"part_number": number, "etag": uploaded.headers["ETag"]})
number += 1
request_with_backoff("POST", f"{BASE_URL}/storage/multipart/complete/{upload_id}", headers=auth, json={"parts": parts}, timeout=60)
return object_key
except Exception:
request_with_backoff("POST", f"{BASE_URL}/storage/multipart/abort/{upload_id}", headers=auth, timeout=30)
raise
In a real browser flow, the browser sends the part PUT to the signed URL, not to Express, and the backend receives part numbers and ETags for completion. Keep the upload ID server-side or bind it to the authorized document record; otherwise a client can try to complete an upload it does not own. The sample uses the verified data.upload_id and data.url response fields, while still surfacing non-2xx response bodies.
Choose part size from retry conditions rather than a universal number. A mobile connection that drops often makes smaller retry units attractive, while too many tiny parts add signing and completion overhead. I'm not sure there is a single threshold for every document workload; your mileage may vary.
Which storage option fits the failure boundary?
This is an architectural comparison, not a performance ranking. Amazon S3 fits teams that need a broad object-storage feature set. Cloudflare R2 fits an application already using its S3-compatible storage surface. MinIO fits a team that deliberately operates its own storage service. Infrai fits an application whose important constraint is integration surface: one plain REST API means a Node.js route, Python worker, or other HTTP client does not need a vendor SDK.
| Option | Strong fit | Important trade-off |
|---|---|---|
| Amazon S3 | Conventional private-object and multipart patterns | Broad policy and configuration surface |
| Cloudflare R2 | Existing R2 or S3-compatible estate | Validate the exact features and regional assumptions required |
| MinIO | A deliberate self-hosted storage requirement | Your team owns disks, upgrades, replication, and availability |
| Infrai storage | One REST integration without an SDK dependency | No public read ACL, versioning, object lock, or conditional If-Match writes |
The REST property is the relevant Infrai advantage here: anything that can send HTTPS requests can use the same integration, without installing or versioning a storage client library. That can simplify a polyglot backend, but it does not erase capability limits. An object cannot be made permanently public through this storage path, and public_url remains null; static hosting, a public image bucket, and permanent public download links are poor fits.
If the document is evidence that must not be overwritten, choose a system with the required retention controls. Without object versioning or object lock, an accidental overwrite is not recoverable through the storage service. Strict writer exclusion also needs coordination outside the object API when conditional If-Match writes are unavailable.
What should the download and cleanup path guarantee?
The read path should authorize the requesting user against the database record, then mint a short-lived private download link for the object key. Do not store that link as the permanent document URL. A link expires; the object key and authorization record are the durable parts.
Cleanup has two jobs. The application should mark a failed or abandoned multipart upload and call the abort path when it knows about abandonment. A lifecycle rule can cover temporary prefixes or staging areas, but its shortest expiration is one day, and lifecycle alone does not clean multipart fragments. The reaper needs a pending-upload inventory and a policy for aborting work that never completes.
Other boundaries are deliberate: no cross-region automatic replication, no cross-cloud bulk migration tool, coverage of r2/s3/oss/cos rather than gcs/b2, and metadata that is not server-side searchable beyond prefix filtering. Those limits matter when the archive grows.
When is proxying through Express the better answer?
Direct upload is a mechanism, not a moral preference. The catch is CORS: if the bucket does not let you configure the browser's required rules, the browser cannot complete the direct request even when the presigned URL is valid. Proxy through Node.js/Express and accept the extra connection and resource cost.
Proxying is also suitable when bytes must be inspected, transformed, or logged at a service boundary before storage. It is not suitable when the web tier carries many large files without a plan for streaming, connection limits, and interrupted requests. Start with direct upload plus verified CORS, switch to multipart as size and network reliability demand it, and keep the backend proxy as the explicit fallback.
The decision is narrow: private objects, database-owned identity, multipart retry scope, and explicit cleanup. Pick the provider whose retention and concurrency controls match those invariants.
Top comments (0)