Short answer: for a large browser upload, send bytes directly to S3-compatible object storage with presigned multipart parts, but keep retention, deletion, and completion state in your application database; the upload is not finished until the server has made one explicit terminal decision.
That's the trade-off I would put in front of an edtech platform team. Direct upload keeps the API out of the byte path, while a server-owned state machine makes a tenant-scoped export auditable and deletable. The storage protocol matters, but it is not the part that decides whether a student record is retained for the right period or left behind after a tenant requests deletion.
What does retention mean before an export becomes an object?
The bounded failure scenario is familiar: a teacher uploads a 4 GB product-image archive, the browser sends several parts, and the tab disappears before completion. The object listing shows nothing useful because an unfinished multipart upload is not the completed object that the application reads. Waiting does not turn those parts into a tenant export.
I treat that as a control-plane failure, not a network anecdote. Before returning a presigned URL, record the tenant ID, object key, upload ID, intended retention class, part size, expected size, and an expiry deadline. The record needs an owner and a state. A cleanup job can then find open sessions without guessing from a storage prefix.
There is a sharp operational number here: a seven-day retention promise cannot be implemented by a cleanup process that only notices abandoned sessions after seven days. Storage lifecycle configuration is useful as a backstop, but the application still needs a faster deadline worker when deletion is part of the product contract. I have seen teams monitor object-read availability while never measuring open multipart sessions, and the resulting dashboard said “healthy” while unfinished writes accumulated. In a 12-part export, eleven successful uploads still represent zero usable exports until the final decision is recorded, which is why I would put the open-session gauge beside the completion SLO rather than in a storage-cost report. I'm not sure every provider exposes the same lifecycle timing, so the exact cleanup guarantee belongs in a test and in the service contract.
No hidden cleanup exists.
The invariant is straightforward: create and record the session, upload numbered parts, complete exactly once, or abort exactly once. A browser close can skip the final callback, so cancellation and deadline expiry must converge on the same server-side abort path. The worker should be idempotent: two workers may observe the same overdue row, but only one may win the transition away from an open state.
A completion state machine is the deletion boundary
Use a small state machine that both the browser callback and the expiry worker must respect:
-
planned: the database row exists and the browser has a bounded plan. -
uploading: parts may be uploaded and reported. -
completing: one worker owns the terminal completion attempt. -
completed: the object is available and its retention deadline is recorded. -
aborting: one worker owns the terminal abort attempt. -
aborted: no further browser work is accepted. -
deleting: an authorized tenant deletion is in progress. -
deleted: the application has recorded the deletion result.
Claim completing, aborting, or deleting with a database conditional update or a queue consumer. The storage operation may be retried, but the state transition must be safe to repeat and the audit record must identify the attempt. Keep metrics for each state age; a large deleting population is more actionable than a vague storage error rate.
Test the awkward paths before production: a duplicate completion callback, an abort racing with the final part, an expired presigned URL, a tenant deletion during export generation, and a worker restart after the storage call but before the database commit. These are not exotic cases in a browser workflow. They are the workflow.
No hidden cleanup exists.
How should a Node.js control plane handle browser direct upload, multipart presigned parts, complete, and abort?
Node.js is a reasonable control-plane choice because it can issue a compact upload plan while the browser talks directly to storage. The sequence is more important than the framework:
- Authenticate the tenant and validate the requested content type and size.
- Choose an object key that contains an opaque tenant identifier, never a user-supplied path.
- Create the multipart session through the storage API and persist its returned upload ID.
- Sign one URL per part, bounded by a maximum part count and a deadline.
- Let the browser upload parts in parallel and report each part number and checksum.
- Have the server validate ownership and state before completing the upload.
- On cancellation, a permanent failure, or expiry, abort the multipart session and mark the database row terminal.
The browser should not decide that a tenant export exists merely because the last PUT returned success. The server should require the expected part list, verify that the session belongs to the requesting tenant, and make the completion call from a transactionally claimed state. A retry of the completion request must return the recorded terminal result rather than create a second decision.
The following Go example shows the control-plane boundary without pretending that a generic S3-compatible endpoint has one universal authentication scheme. The application stores the returned upload ID before it signs any parts; the storage adapter behind CreateMultipart can use the selected provider's SDK or HTTP API.
package main
import (
"context"
"fmt"
)
type UploadPlan struct {
TenantID string
ObjectKey string
UploadID string
PartSize int64
ExpiresInSeconds int
}
type Storage interface {
CreateMultipart(context.Context, string, string) (string, error)
SignPart(context.Context, string, string, int) (string, error)
CompleteMultipart(context.Context, string, string, []int) error
AbortMultipart(context.Context, string, string) error
}
func StartUpload(ctx context.Context, storage Storage, tenantID, objectKey string, partCount int) (UploadPlan, error) {
if tenantID == "" || objectKey == "" || partCount < 1 || partCount > 10000 {
return UploadPlan{}, fmt.Errorf("invalid upload plan")
}
uploadID, err := storage.CreateMultipart(ctx, tenantID, objectKey)
if err != nil {
return UploadPlan{}, err
}
// Persist tenantID, objectKey, uploadID, partCount, retention, and deadline here.
// If persistence fails, abort the newly created session before returning an error.
return UploadPlan{
TenantID: tenantID,
ObjectKey: objectKey,
UploadID: uploadID,
PartSize: 16 * 1024 * 1024,
ExpiresInSeconds: 3600,
}, nil
}
The exact part size is a capacity decision, not a folklore constant. Smaller parts reduce retransmission but increase signing work, browser requests, and completion metadata. Larger parts reduce request count and increase the cost of a retry on a weak connection. Measure the file-size distribution, client throughput, peak concurrent uploads, and the time allowed by the export SLO. Your mileage may vary on mobile networks.
What should retention and deletion mean for a tenant-scoped export?
Retention needs two clocks. The first is the upload-session deadline, which bounds unfinished work. The second is the object retention deadline, which governs a completed image or export. Store both explicitly; an upload that completes at 23:59 should not inherit an ambiguous deadline calculated from when the browser opened the file picker.
Deletion is a workflow with evidence, not a DELETE button wired directly to an object key. Resolve the tenant's authorized object set from application metadata, mark the deletion request, revoke new presigned URLs, and delete completed objects through an idempotent worker. Open multipart sessions must be aborted as part of the same request. Record who requested deletion, which tenant scope was used, the number of objects found, and the final outcome.
For a tenant export, avoid putting mutable business meaning in the filename. Keep the tenant relationship in a database row and use an opaque key such as exports/{tenant-id}/{export-id}/archive.bin; the key is an index, not an authorization decision. A storage prefix alone cannot prove that the caller may read or delete every object below it.
The SLO should cover the user-visible operation and the cleanup tail. Useful measures include time to first part URL, upload completion latency, completion success rate, age of open sessions, abort latency, deletion completion latency, and objects whose retention deadline has passed. Alerting only on request availability misses the exact failure that retention policy is supposed to prevent.
Buy versus build: which boundary can the team actually operate?
The buy-vs-build choice is mostly an ownership decision. A managed S3-compatible service can reduce the storage team's burden for durability and regional operations, while a self-hosted deployment gives a team more control and adds responsibility for disks, upgrades, replication, and incident response. The direct browser pattern remains similar; the on-call boundary does not.
| Choice | What the team gets | What the team still owns | Retention question |
|---|---|---|---|
| Managed object storage | Multipart protocol and provider-operated storage infrastructure | Application authorization, session records, deletion workers, and policy tests | Can the service and application provide the required deletion evidence? |
| Self-hosted S3-compatible storage | Control over placement and operational policy | Capacity, durability, upgrades, replication, access control, and incidents | Can the team prove deletion and recover safely during a storage incident? |
| Application proxy for file bytes | Centralized request handling and simpler browser policy | Network egress, buffering, scaling, and the full byte-transfer SLO | Does the extra hop justify its cost and control? |
The catch is that direct upload is not suitable when every byte must pass through a content inspection or transformation pipeline before storage. In that case, keep the proxy or put a scanning stage in front of publication, and make the quarantine state explicit. Stick with a self-hosted system when regulatory placement or offline operation is a real requirement and the team has capacity for storage on-call. Choose managed storage when that operational ownership is not part of the platform's charter.
Do not make price the primary argument. The meaningful comparison is the total failure budget: storage requests, signing requests, database writes, deletion work, and human intervention all consume capacity. A low transfer bill does not compensate for an export that cannot be deleted on time.
Top comments (0)