Choose the upload protocol from the retention state machine, not from the file extension: keep small documents on a single request, use multipart for transfers whose retry cost threatens the upload SLO, and make deletion of abandoned state an explicit, observable action. Short answer: for private 100 MB to 1 GB PDFs and ZIPs, multipart is usually the right transport once a failed whole-file retry would break the user-facing objective, but it does not decide how long the finished object should exist.
That distinction matters for a developer-tools team storing user documents or training artifacts. Uploading is a temporary workflow; retention is a policy. If those are represented by one boolean such as uploaded, an interrupted transfer can look complete, and a completed object can survive after its business purpose has ended. I've seen designs spend weeks tuning part sizes while leaving deletion as a cron job with no owner. The transport gets reviewed. The lifecycle gets forgotten.
Start with the retention contract
Start with states that can be named, queried, and audited: initiated, transferring, complete, expired, aborted, and deleted. The database record owns the business identity, retention deadline, object key, checksum or equivalent integrity evidence, and current transfer identifier. Object storage owns bytes. Keeping those responsibilities separate gives the platform team somewhere reliable to answer “why is this object still here?” during an incident.
The retention clock should begin from a business event you can defend, such as document acceptance or artifact publication, rather than from the first byte arriving. Temporary upload state needs a shorter independent deadline. A client that disappears halfway through a 1 GB ZIP must not keep a multipart session alive until the document's ordinary retention period expires.
Use a private namespace for both temporary and completed objects. Give a consumer a time-limited download authorization only after the database says the object is complete and the caller is allowed to read it. A browser upload URL is a capability, not proof that the document has passed validation; the service still needs to authenticate the workflow, constrain the destination, and record the transition.
The deletion worker should be idempotent. It can claim records whose deadlines have passed, request removal of the completed object or abort the incomplete transfer, then record the result with an audit timestamp. A retry of the worker must reach the same terminal state. Do not make a successful HTTP response from a client the only evidence that cleanup happened.
Delete twice.
The second attempt should be harmless, recorded, and boring. That property is more valuable during a recovery exercise than a clever one-shot cleanup script.
Make the selection a capacity decision
There is no universal byte threshold. A 100 MB PDF over a stable internal path may be less risky than a much smaller file sent by a browser through a constrained connection, so the selection should compare failure cost with operational cost. The table is a useful review artifact because it forces the team to name the burden it is accepting.
| Choice | Good fit | Cost or boundary to validate |
|---|---|---|
| Single request | The whole-file retry fits the upload SLO and memory budget | A late failure repeats all transferred bytes |
| Multipart | Large documents, unreliable paths, or a retry budget that cannot absorb a full resend | More states, part records, cleanup work, and concurrency to operate |
| Self-hosted storage | Direct control and existing on-call capacity are requirements | Capacity headroom, upgrades, backup, and restore drills stay with the team |
| Managed storage | The team wants to reduce storage operations work | API coupling, policy limits, and migration effort still need an owner |
This is a buy-versus-build decision as much as an upload decision. A managed service can reduce pager work, while a self-hosted component can provide control that is important for a particular compliance boundary; neither choice removes the need for an application-owned retention record. FedRAMP is a useful example of why an authorization boundary must be checked against the actual service and deployment, not inferred from the word “private.”
How should Node.js select multipart for private PDF and ZIP uploads?
Multipart changes the unit of retry. A broken connection near the end of a single-request upload can force the client to resend the entire object; with multipart, the application can retry the failed part and preserve completed progress. That is why the 100 MB to 1 GB range deserves a policy decision, while “PDF versus ZIP” does not: compression and format say little about network behavior.
The trade-off is state. A multipart workflow has creation, part transfer, completion, and abort paths, each of which needs logs and ownership. A single request is easier to reason about when a whole-file retry fits inside the latency budget. The threshold should come from measured object sizes, concurrency, retry frequency, connection quality, and the upload SLO. Your mileage may vary; an internal batch client and a browser on an office VPN do not have the same failure distribution.
Capacity planning must include the bytes in flight, not just the final bucket size. If the service buffers one part per active transfer, a 64 MB part size with 20 concurrent uploads implies roughly 1.28 GB of application memory before runtime and connection overhead. That may be acceptable on one worker and reckless on another. Set limits for active sessions, part size, request duration, and total retry bytes, then alert before the limits become a user-visible outage.
Measure it. A useful review exercise is to take the largest expected ZIP, the smallest worker memory limit, and the peak number of simultaneous transfers, then walk the numbers through a failure rather than a success. Suppose the policy admits 1 GB objects, the implementation keeps one 64 MB part in memory per transfer, and the service permits 20 active transfers. The nominal buffer is about 1.28 GB; now add the request bodies waiting for a retry, connection buffers, garbage-collector headroom, database work, and the rest of the process. If a failed part is retried without releasing the old reader, the nominal figure is no longer the capacity plan. If the client retries the whole object outside the service while the service also retries a part, network demand grows again. This is the kind of interaction that a benchmark with one successful upload will miss, and it is why the policy should cap both application retries and client-visible retries, record the part number for every attempt, and make the accepted object-size limit visible in the API contract.
The important invariant is simple: after creation succeeds, every exit is either completion or abort. A timeout, rejected validation, client cancellation, process shutdown, and failed completion all need a path to that invariant. A periodic sweeper remains useful for records left behind by a crashed process, but it is a backstop, not the primary cleanup mechanism.
What must the upload state machine guarantee?
The following Go example keeps provider calls behind an interface. A Node.js service can implement the same contract with its chosen storage client, while the policy and tests remain independent of that client. The example intentionally does not invent a provider-specific route or SDK behavior.
package retention
import (
"context"
"errors"
"fmt"
"io"
)
type Part struct {
Number int
Tag string
}
type Store interface {
Begin(ctx context.Context, key string) (string, error)
PutPart(ctx context.Context, uploadID string, number int, body io.Reader) (string, error)
Finish(ctx context.Context, uploadID string, parts []Part) error
Abort(ctx context.Context, uploadID string) error
Remove(ctx context.Context, key string) error
}
func Transfer(ctx context.Context, store Store, key string, parts []io.Reader) (err error) {
if len(parts) == 0 {
return errors.New("no parts to transfer")
}
uploadID, err := store.Begin(ctx, key)
if err != nil {
return fmt.Errorf("begin transfer: %w", err)
}
finished := false
defer func() {
if finished {
return
}
if abortErr := store.Abort(context.WithoutCancel(ctx), uploadID); abortErr != nil {
err = errors.Join(err, fmt.Errorf("abort transfer: %w", abortErr))
}
}()
completed := make([]Part, 0, len(parts))
for number, body := range parts {
tag, putErr := store.PutPart(ctx, uploadID, number+1, body)
if putErr != nil {
return fmt.Errorf("put part %d: %w", number+1, putErr)
}
completed = append(completed, Part{Number: number + 1, Tag: tag})
}
if err := store.Finish(ctx, uploadID, completed); err != nil {
return fmt.Errorf("finish transfer: %w", err)
}
finished = true
return nil
}
The production version should persist the transfer identifier and completed part tags after each successful part, so a process restart can resume or deliberately abort from a known record. It should also bound concurrency and use provider-documented retry rules for transient responses. Never treat a client retry as permission to create an untracked second upload; use an application idempotency key and a unique business record for that decision.
There is a second race that multipart does not solve: two requests can select the same object key. Serialize that choice with a database claim or another application-level ownership record. If the requirement is strict conditional replacement, verify that the selected storage system exposes the needed conditional-write behavior before committing to the integration. A transport protocol cannot supply a policy guarantee that its storage contract does not expose.
Which checks make deletion safe to operate?
Test the transitions, not just the happy-path bytes. The minimum table should include a successful small upload, a successful multipart upload, a failed part followed by a retry, cancellation after Begin, validation rejection before Finish, worker restart during deletion, and two writers racing for one key. For each case, assert the database state, object visibility, audit record, and cleanup request. A passing checksum test with an orphaned transfer is still a failed test.
Verification in production needs both business and storage signals: completion latency against the upload SLO, retry counts by part, bytes resent, active transfer age, completed-object age, deletion lag, and records whose deadline has passed without a terminal action. Keep the workflow database authoritative for these queries. A bucket listing can help reconciliation, but it is not a substitute for an indexed ownership record.
Roll out the policy behind a server-controlled flag. Begin with a bounded cohort of large documents, preserve the simpler single-request path for objects whose retry cost is acceptable, and compare the two paths using the same SLO and privacy checks. If memory pressure or abandoned-state age exceeds its budget, stop assigning new multipart work, let safe active transfers finish, abort records explicitly marked failed, and return the cohort to the simpler path while the model is corrected.
The rollback must not lengthen retention by accident. It must not make temporary objects public. It must not allow an unclaimed writer to replace a document. Those controls belong to the state machine, regardless of which upload protocol carries the bytes.
Top comments (0)