Short answer: for large document upload selection in a media system, use multipart upload for private object storage when a 100 MB to 1 GB PDF or ZIP would be expensive to restart, and use a single request for small, repeatable transfers. The important choice is not a vendor or a fashionable threshold. It is the application state that proves an immutable original was completely received before receipt processing starts.
This is an architecture decision record. The invariants are stricter than “the object exists”: the received bytes must remain unchanged, one document revision must own the processing job, incomplete transfers must stay out of the queue, and abandoned transfers must have an expiry and cleanup path. Large-file throughput matters only after those claims can survive a timeout.
The audit record comes before the object
Start with failure cost and connection behavior. A 1 GB ZIP sent from a browser or a remote production team is a reasonable multipart candidate because losing the final response should not force the whole body across the network again. A controlled worker sending a small PDF over a dependable internal link may be better served by one request, because the simpler state machine is easier to inspect and operate.
The threshold belongs in configuration, alongside measurements for completion rate, retransmitted bytes, part retries, and upload latency. I’m not sure any team can choose the right threshold from file size alone; the answer changes with client networks, timeout budgets, and the storage contract. Treat 100 MB as a policy input, not a universal limit.
Reserve a document revision before opening the transfer. The reservation should contain the intended object key, expected size, media type, requester, and an idempotency key. Mark it ready only after storage confirms the completed object and the application has reconciled that result. Receipt extraction begins after that transition, never after the first successful request merely returns.
Keep it explicit.
| Option | Good fit | Failure boundary | Trade-off |
|---|---|---|---|
| Single request | Small files and dependable service-to-storage links | A failed request restarts the complete body | Minimal state, but a late failure wastes the full transfer |
| Multipart upload | 100 MB to 1 GB PDFs or ZIPs over failure-prone links | Retry a part, reconcile acknowledgements, then complete | Better recovery, with expiry and orphan cleanup to operate |
| Application proxy | A policy requires inspection before storage | The proxy owns connection, buffering, and backpressure failures | Central control, but the application carries more bandwidth and scaling work |
| Browser direct upload | Large client-side transfers with scoped authorization | Client and storage state must be reconciled by the application | Less server bandwidth, but a broader authorization and callback design |
The rejected default is multipart for every object. A 200 KB receipt pays for session creation, part bookkeeping, completion, and cleanup even when repeating the transfer is harmless. Use a single request there. The opposite mistake is making a 1 GB archive restart from byte zero because the first design had fewer database states.
Failure boundaries are part of the upload contract
Multipart changes the recovery unit from the whole object to an individual part. It also adds an upload identifier, part numbers, integrity evidence, acknowledgements, an expiry, and explicit completion or abort transitions. Those states belong in the application record; a private bucket is not an audit ledger.
The most damaging failure is a lost response. Storage may have accepted the final part while the worker timed out, so recording failed immediately can create a second logical revision or leave two plausible originals for one receipt. Reconcile the remote state with the reserved revision and acknowledged parts, then make exactly one terminal decision. In a real Node.js worker, that means the timeout handler must be allowed to ask for upload status, compare the returned parts with the reservation, and persist the result before a retry scheduler sees the item. A process restart between any two of those operations is ordinary, not an exotic edge case, so the records need enough information to resume without trusting in-memory state or a client callback.
The implementation below is a provider-independent policy seam. A Node.js service can stream from a request or file descriptor through its chosen HTTP client; it should not load a 1 GB ZIP into process memory just because an API accepts a buffer.
from dataclasses import dataclass
from enum import Enum
MIB = 1024 * 1024
MULTIPART_THRESHOLD = 100 * MIB
PART_SIZE = 16 * MIB
class UploadMode(str, Enum):
SINGLE = "single_request"
MULTIPART = "multipart"
@dataclass(frozen=True)
class UploadPlan:
object_key: str
size_bytes: int
mode: UploadMode
part_count: int
def choose_upload(object_key: str, size_bytes: int) -> UploadPlan:
if not object_key or size_bytes < 0:
raise ValueError("object_key is required and size_bytes must be non-negative")
if size_bytes < MULTIPART_THRESHOLD:
return UploadPlan(object_key, size_bytes, UploadMode.SINGLE, 1)
part_count = (size_bytes + PART_SIZE - 1) // PART_SIZE
return UploadPlan(object_key, size_bytes, UploadMode.MULTIPART, part_count)
def receipt_uploads() -> list[UploadPlan]:
return [
choose_upload("receipts/2026/camera-17.pdf", 100 * MIB),
choose_upload("receipts/2026/press-kit.zip", 1024 * MIB),
]
The 100 MiB and 16 MiB values demonstrate a policy; they are not claims about a provider’s hard limits. Persist each acknowledged part before allowing the next state transition, bound concurrency, and make retries idempotent for the reserved revision. An HTTP 429 should be treated as a scheduling signal: back off and honor Retry-After rather than spinning until the supposedly resilient worker becomes the bottleneck.
For a browser upload, the application should issue authorization tied to the reservation, then accept completion only for that object key and revision. Authorization to write an object is not authorization to replace a current receipt or read another account’s source file. Those are application decisions and should be logged as such.
How should a team choose multipart upload for large private documents?
Use an explicit state machine:
- Authorize the request and reserve a document revision.
- Select single-request or multipart transfer from the configured policy.
- Stream the bytes while recording observed size and acknowledged parts.
- Reconcile after a timeout instead of guessing from the client response.
- Mark the original
readyin the application database. - Enqueue receipt processing with the revision identifier and object key.
Only step five publishes work to the processor. If the worker dies after marking ready but before enqueueing, an outbox or equivalent durable handoff can publish the event later. If processing fails, retain the source and record the processing failure separately; deleting the original destroys the evidence needed to explain the result.
Consider a concrete recovery sequence: a browser has sent 63 of 64 parts for a receipt archive when its connection disappears. The reservation and 63 durable acknowledgements remain, but there is no ready event. A retry resumes the same revision and sends the missing part. If the final completion response is lost, a reconciler checks completion state before publishing the event. If the reservation has expired, it closes that revision and makes the incomplete bytes eligible for cleanup. The processor never has to infer business truth from a directory listing, a filename, or a timestamp. That boundary also makes review possible: an auditor can see which revision was reserved, which bytes were acknowledged, when completion was reconciled, and why a later processing attempt did or did not run, instead of receiving a plausible filename with no transfer history.
Concurrency needs the same discipline. Two tabs may upload different bytes for one logical receipt at the same time. Reserve revisions before transfer, and require completion to prove that its reservation is still current. If a revision is expired or superseded, abort or quarantine its incomplete upload instead of silently replacing the current original.
Where multipart stops being suitable
Multipart is not suitable when the storage contract cannot provide the integrity, expiry, and completion behavior required by the audit policy. It is also a poor fit for tiny files, strict low-latency metadata writes, or workflows where a proxy must inspect every byte before storage and cannot sustain the required bandwidth. Stick with a single request when the transfer is cheap to repeat and the simpler recovery path is more valuable than part-level retry.
Private object storage is not automatically an archive. If the requirement includes immutable retention, legal hold, searchable metadata, or a regulated authorization boundary, verify each capability separately and document which system is authoritative. FedRAMP is an authorization program; private objects alone do not establish compliance.
Measure more than total upload time: abandoned sessions, bytes retransmitted, completion latency, part retry counts, queue delay, and originals without processing results. Those signals show whether multipart improves the real bottleneck or merely adds operational states. Your mileage may vary, but the durable rule is stable: preserve the original, make completion explicit, and never confuse an incomplete transfer with an auditable document.
Top comments (0)