Short answer: for private 100MB to 1GB documents, use a resumable multipart upload directly to object storage, keep a small metadata record in your application database, and make the finalization step verify size, checksum, and ownership before a receipt is processed. A single request through the Node.js API is easier to demo, but it turns a slow connection or a proxy limit into a lost file.
The bill is usually dominated by retained bytes and repeated transfer, not by the tiny metadata row that says who uploaded a PDF. That changes the design question. The useful choice is not “which storage brand has the nicest SDK?” It is how much original evidence you keep, how long you keep it, and what happens when a 1GB ZIP reaches 93 percent and the browser disappears.
What should a Node.js multipart upload path guarantee for private object storage?
Start with an upload session owned by your application. The session records an opaque object key, expected content length, media type, tenant, expiry, and a state such as created, uploading, or committed. The client receives short-lived part URLs or equivalent scoped credentials. It never receives a bucket-wide secret.
Parts should be large enough to avoid thousands of requests, yet small enough that one retry is cheap. For a 100MB PDF, 8-16MB parts are a reasonable starting experiment; for a 1GB ZIP, measure with 32-64MB parts on the networks your users actually have. Those are tuning points, not universal constants. Your storage service may impose its own minimum part size and maximum part count, so the contract must be checked against its API documentation.
The application should accept a completion request only after it has authenticated the session owner and checked every returned part number and checksum. Then it asks object storage to assemble the object. A database transaction can mark the session committed after a successful assembly; a worker can process the receipt only from that committed state. This ordering prevents a queue consumer from reading a key that exists in the database but is not yet a complete object.
Here is the shape I use for a provider-neutral completion handler. The storage client below is deliberately an interface: the important behavior is validation and state transition, not a particular SDK.
from dataclasses import dataclass
from hashlib import sha256
@dataclass
class UploadSession:
owner_id: str
object_key: str
expected_bytes: int
state: str
def complete_upload(session: UploadSession, owner_id: str, parts, store):
if session.owner_id != owner_id or session.state != "uploading":
raise PermissionError("upload session is not owned by caller")
if not parts or len(parts) > 10000:
raise ValueError("invalid part list")
if [part.number for part in parts] != list(range(1, len(parts) + 1)):
raise ValueError("part numbers must be contiguous")
store.complete_multipart(session.object_key, parts)
head = store.head(session.object_key)
if head.size != session.expected_bytes:
store.delete(session.object_key)
raise ValueError("assembled object has unexpected size")
return {"key": session.object_key, "bytes": head.size, "state": "committed"}
The head check is not a substitute for an end-to-end digest. If your threat model requires content integrity rather than transport integrity, record a client-computed SHA-256 and verify it in a worker that can stream the completed object. Do not trust a filename or a browser-supplied MIME type as proof of what was uploaded.
Where do retention and cost change the architecture?
Retention is a product decision with an engineering price tag. Keep the original receipt when an audit or legal hold requires it. Keep a derived thumbnail or OCR text only when it has a defined consumer. An unbounded “temporary” prefix becomes a second archive surprisingly quickly.
| Data | Default treatment | Failure you accept by deleting it |
|---|---|---|
| Original PDF or ZIP | Retain under a versioned, private key | Storage cost and deletion workflow remain |
| Upload session and part manifest | Expire after a short window | An abandoned upload must restart |
| Receipt parse result | Rebuild from the original | Reprocessing consumes worker time |
| Client preview | Regenerate on demand | A preview may be briefly unavailable |
Lifecycle rules help with age-based retention, but they do not answer whether a record is under legal hold. Put the hold state in your own database and make deletion jobs query it before issuing a storage delete. Versioning can protect against accidental overwrite, yet it also means “delete one file” may leave older versions consuming bytes.
I model the monthly estimate as retained gigabytes plus request and transfer terms, then run it against two deliberately boring scenarios: 10,000 receipts at 100MB and 1,000 receipts at 1GB. The arithmetic exposes the dominant term before a team argues about a few thousand completion calls. Prices and free allowances change, so I keep current numbers out of code and verify them against the selected service's current billing page.
Measure the retention path, not just the happy-path upload. Suppose a receipt arrives as a 1GB ZIP, the parser expands it into 4GB of temporary files, and a retry leaves an abandoned multipart session behind. A dashboard that counts only committed objects reports 1GB, while the storage bill can briefly reflect the original, the incomplete parts, and the expanded workspace. The exact multiplier depends on the parser and lifecycle settings, but the shape of the risk is stable: bytes accumulate in places the product owner cannot see. I put a daily job beside the application that totals committed bytes by tenant, lists incomplete multipart sessions by age, and compares parser workspace usage with the retention policy. When the numbers disagree, the job opens an operational ticket rather than deleting data automatically. That extra hesitation matters for audit records; an over-eager cleanup can destroy the only copy of a disputed receipt. The point is not to predict a perfect monthly bill. It is to make every retained byte explainable, including the byte that exists for ten minutes during a retry.
Keep it explainable.
The catch is that aggressive retention is not free operationally either. If you delete originals after parsing, a disputed receipt may have no evidence left; if you keep every intermediate part forever, a crashed browser becomes a quiet storage leak. Choose the loss you can explain to an auditor.
How do retries, checksums, and private URLs behave under failure?
Multipart upload gives you a retry boundary. A failed part can be sent again without replaying the whole document. That only works if the client persists the upload ID, part numbers, and returned checksums somewhere durable, such as IndexedDB or a server-side session. A process restart should resume or deliberately abandon the session; it should not create an invisible second object.
Use bounded retries with jitter for network timeouts and 429 responses. Do not retry an authorization failure as if it were congestion. A sweeper can list sessions that have passed their expiry and ask storage to abort their multipart uploads. This is a separate job from deleting committed objects, because an incomplete upload has no useful audit record.
Private access should be short-lived and purpose-specific. A download URL for an auditor can be scoped to one object and one method; an upload URL should not also permit deletion or listing. Log the session ID, tenant, object key hash, byte count, and outcome, but avoid logging the document contents or long-lived credentials.
Receipt processing needs a second trust boundary. Treat the PDF or ZIP as hostile input: enforce a maximum uncompressed size, scan archives for path traversal, and run parsers in a restricted worker. “Private” controls who can fetch an object; it does not make a file safe to parse.
What changes between 100MB PDFs and 1GB ZIP archives?
A 100MB PDF often completes in one foreground interaction, while a 1GB ZIP may outlive a tab, a mobile connection, or a deployment. The user interface should show committed parts and offer resume, not just a spinning percentage. The server should make completion idempotent: repeating the same valid completion request returns the committed result instead of creating a second receipt.
Large archives also deserve a different queue policy. Do not make the HTTP request wait for OCR, virus scanning, and receipt extraction. Publish a job after commit, include the immutable object version, and let workers claim it with a lease. If parsing fails, retain the original and record a classified failure for review; do not silently replace it with a zero-byte placeholder.
There are real interface differences among object stores. Amazon S3 documents multipart operations and checksum headers; Google Cloud Storage exposes resumable uploads; Azure Blob Storage has block-blob staging and commit; Cloudflare R2 follows an S3-compatible API. Compatibility reduces adapter code, not operational responsibility. Check each provider's limits for part size, checksum algorithms, lifecycle behavior, versioning, and signed URL scope before promising the same behavior everywhere.
Your mileage may vary across corporate proxies and consumer VPNs. I am not sure a benchmark from one region predicts your users' upload time, so I would measure p50 and p95 completion time, retry rate, abandoned sessions, and bytes retained for at least a week before changing part sizes.
A decision rule that survives audits
Use direct multipart upload when files can exceed your application server's request limits, users need resume, or the original must remain available for an audit. A simple single-request endpoint is suitable for small, controlled inputs where restart cost is negligible.
Stick with a managed object store when durability, lifecycle, and access logging are more important than placing bytes on a local disk. Choose self-hosted storage when data residency, offline operation, or a dedicated hardware boundary outweighs the maintenance burden. Neither choice removes the need for checksums, retention policy, and deletion evidence.
The approach is not suitable when you need low-latency random writes inside a document, transactional updates to individual bytes, or a database query over every field in the file. Use a database or a specialized content system for those requirements, and keep object storage for immutable blobs and their metadata.
The useful finish line is boring: one private key, one committed version, one traceable processing job, and a documented answer for what happens when the client vanishes at 93 percent. That is more durable than a clever upload widget.
Top comments (0)