Short answer: send each training artifact straight from the browser to private object storage with a short-lived signed upload URL, keep authorization and retention state in Postgres, and let Node.js coordinate the transfer without carrying the file bytes.
Bytes dominate.
Start with the bill, because retention multiplies every mistake. For a planning example, take 400 active courses, four 10 GB training artifacts per course, and three retained revisions. The retained payload is 48 TB before replicas or provider-specific billing units: 400 x 4 x 10 GB x 3. Storage grows with retained bytes and time; upload traffic, validation reads, student downloads, and deletion operations are separate terms. Put the real rates from the chosen deployment into that equation. Don't insert a guessed industry average.
The biggest adjustable term in this example is retained byte-months. Changing a signed URL lifetime from 15 minutes to 5 minutes doesn't move it. Reducing retained revisions from three to two removes 16 TB from the steady-state model, while shortening retention from 90 days to 30 days changes how long old revisions remain billable. Those are policy decisions, not storage tricks, and an edtech team should connect them to course audit and deletion obligations before touching a lifecycle rule.
What should a Node.js Postgres pattern for private browser upload and signed download URLs control?
Node.js should own intent, not bulk data. After authenticating the instructor and checking course membership, it creates an upload record, chooses an opaque object key, records the expected size and media type, and returns a narrowly scoped signed upload URL. The browser uploads to object storage. A later confirmation step verifies object metadata before the record becomes downloadable. A signed download URL is issued only after a fresh authorization check against the course and tenant.
Keep the bucket private. A presigned URL grants time-limited access using the permissions of the principal that created it, and it can be used more than once until it expires. Treat it like a bearer credential — don't log it, put it in analytics, or send it through email. That last detail matters in an edtech workflow: forwarding a convenient download link can quietly bypass the recipient and consent controls that would apply to an ordinary notification.
The Postgres row is the policy authority. The object store holds bytes; it doesn't decide whether an instructor still belongs to a tenant, whether a learner requested erasure, or whether a compliance hold overrides normal expiry. A compact record needs enough information to make those decisions without parsing object names:
| Field | Purpose |
|---|---|
tenant_id, course_id, artifact_id
|
Authorization boundary and stable domain identity |
object_key |
Random, server-selected storage locator |
state |
pending, ready, delete_due, or deleted
|
expected_bytes, observed_bytes
|
Admission control and post-upload verification |
content_type |
Expected type, still subject to validation |
created_at, retain_until
|
Reproducible retention decision |
policy_version |
Explains which rule produced retain_until
|
deleted_at |
Audit evidence that deletion completed |
That policy_version is easy to omit and painful to reconstruct. If a course moves from 90-day retention to 30-day retention, the service needs an explicit rule for existing artifacts: preserve their original deadline, recompute it, or place selected records on hold. A versioned decision makes a replay produce the same answer.
Make large-file throughput a capacity budget
Direct upload removes the Node.js application from the byte path, but it doesn't make throughput infinite. Browser egress, the storage endpoint, concurrent connection limits, and retry behavior still bound the system. Measure them separately. The useful test matrix varies object size and concurrency, then records completion rate, p50 and p95 duration, bytes per second, aborts, and retry counts. A 20 MB PDF tells you little about a 10 GB model bundle.
Use a small capacity model before load testing. This Python snippet calculates an idealized lower bound; it isn't a benchmark and deliberately excludes protocol overhead, contention, and retries.
def ideal_upload_minutes(size_gib: float, uplink_mbps: float) -> float:
bits = size_gib * 1024**3 * 8
return bits / (uplink_mbps * 1_000_000) / 60
artifact_gib = 10
for uplink_mbps in (10, 25, 100):
minutes = ideal_upload_minutes(artifact_gib, uplink_mbps)
print(f"{uplink_mbps:>3} Mbps: {minutes:.1f} ideal minutes")
At those explicit inputs, the ideal times are about 143.2, 57.3, and 14.3 minutes. Real transfers take longer. This immediately exposes a design error: a five-minute signed upload window can't cover a single-request 10 GB transfer on any of those modeled links. Choose expiry from measured p95 completion time plus a bounded margin, or use a resumable or multipart protocol whose authorization model survives individual part retries. I'm not sure what margin fits your population without regional and campus-network measurements; collect those before fixing the value.
Expiry isn't throughput.
For progress UI, XMLHttpRequest.upload provides upload progress events. Register listeners before sending the request; MDN also notes browser differences around when listeners should be attached. Progress is advisory, not proof of durable completion. The UI can display bytes sent, but only the server-side confirmation path should move the database row to ready.
Slow links happen.
Admission control should reject an upload intent before signing when the declared size exceeds the course or tenant limit. It should also cap concurrent pending uploads per tenant. This protects the control plane from a retry storm and keeps one training provider from consuming every connection slot, while the storage service handles the heavy stream. Use idempotency on intent creation and confirmation so a browser retry doesn't manufacture a second artifact revision.
Retention needs a state machine, not a bucket timer
An upload is not one atomic transaction across the browser, Postgres, and object storage. Design for the gaps. A client may request a URL and never upload; it may finish sending bytes and close before confirmation; two tabs may confirm the same intent; or a retention worker may race with a download request. None of those cases should make an unverified object visible.
The practical sequence is:
- Create a
pendingrow in a database transaction after authorization and quota checks. - Issue a signed upload URL for exactly that server-selected key, expected operation, and bounded lifetime.
- Upload directly from the browser while reporting local progress.
- Confirm through Node.js; inspect authoritative object metadata and compare it with the intent.
- Transition
pendingtoreadywith a conditional update, then permit signed downloads after authorization. - Mark expired rows
delete_due, delete the object idempotently, and finally recorddeleted_at.
Run a sweeper for abandoned pending rows and unreferenced objects. Give it a grace period so it doesn't delete a transfer that is still valid, and make every transition monotonic. If confirmation arrives after an artifact is already delete_due, it must lose the race. The database transaction protects the state transition; it cannot roll back an object-store operation, so reconciliation is part of the design rather than an emergency script.
For reproducibility, calculate retain_until once from an immutable event time and a named policy version. Don't repeatedly add “30 days” to the worker's current clock. Store timestamps in UTC, document whether “day” means 24 hours or a calendar boundary, and test daylight-saving transitions even if the database runs in UTC. Compliance disputes love ambiguous clocks.
Downloads need the same care. Authorize first, sign second, and keep expiry short enough for the exposure being protected but long enough for the expected download. Record that access was authorized without storing the signed query string. Range requests may be important for large artifacts, but their exact behavior is storage-specific; verify it against the selected service and include interrupted downloads in acceptance tests.
Know what this pattern deliberately gives up
The pattern fits large private artifacts when application servers shouldn't proxy gigabytes and the team can operate reconciliation. It is not suitable when every byte must pass synchronously through an application-controlled content inspection engine before landing. In that case, use a controlled ingest tier or a quarantine destination and accept its throughput and operational cost. Stick with application-proxied uploads for genuinely small, low-volume files when the simpler authorization boundary matters more than transfer efficiency.
The catch is that reducing retention removes recovery options. In the 48 TB planning example, keeping two revisions instead of three lowers the steady-state retained payload by 16 TB, but an instructor who discovers corruption after two newer revisions no longer has that third copy. A legal hold, active investigation, or published course may also require a different policy from an abandoned draft. Encode those exceptions as explicit states with auditable reasons; don't hide them in a worker's conditional branches.
There are other limits. Signed URLs are credentials, not revocation systems. Very short expiries punish slow campus networks, while very long expiries widen the window after leakage. Postgres can serialize the policy decision, but it can't prove that arbitrary bytes match a claimed media type. Content inspection, checksum verification, malware scanning, and quarantine promotion belong in the readiness pipeline, with downloads denied until the required checks pass.
Operate this as a delivery system. Alert on the age and count of pending and delete_due rows, confirmation failures by reason, upload duration by size band, reconciliation lag, and signed-download authorization denials. Avoid labels containing tenant IDs or object keys; those create high-cardinality telemetry and can expose private identifiers. Structured audit records belong in access-controlled storage with their own retention policy.
Finally, test the awkward boundaries: expiry during a slow transfer, duplicate confirmation, deletion racing with download authorization, tenant removal between intent and confirmation, clock skew, a changed policy version, and a browser retry after losing its response. The happy path is short. The proof lives at the edges.
References
- AWS S3 documentation, “Sharing objects with presigned URLs”: https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-presigned-url.html
- MDN, “Using XMLHttpRequest”: https://developer.mozilla.org/en-US/docs/Web/API/XMLHttpRequest_API/Using_XMLHttpRequest
- OWASP, “File Upload Cheat Sheet”: https://cheatsheetseries.owasp.org/cheatsheets/File_Upload_Cheat_Sheet.html
- PostgreSQL documentation, “Transaction Isolation”: https://www.postgresql.org/docs/current/transaction-iso.html
Further reading
- HTTP Semantics, RFC 9110: https://www.rfc-editor.org/rfc/rfc9110
Top comments (0)