Short answer: for an edtech system exporting signed documents, stream the archive into bounded multipart parts, publish it only after completion, and issue a short-lived signed download link only while the retention record permits access.
The deadline is the design constraint. A signed link is a temporary capability, not evidence that the bytes have been deleted. Keep the deletion deadline in the job or compliance database, keep the object private, and make the download endpoint check the export state before asking object storage to sign the exact key.
Large-file throughput is a separate constraint. A worker that creates a multi-gigabyte ZIP and sends it in one request has one large failure boundary: a process eviction, network reset, or exhausted disk can force the archive to be rebuilt. Multipart upload gives the worker smaller retry units, but it does not make an unfinished session a finished object, and it does not decide when deletion is allowed.
Start with the deadline, then budget the archive
Model an export as a state machine rather than a file path. A useful record contains an export ID, a unique object key, the retention deadline, an upload session identifier, completed part numbers and tags, an explicit status, and a completion timestamp. The worker writes state transitions; the download endpoint reads the completed state before issuing a capability.
The key should include the job identity, such as exports/{export_id}/documents.zip. That prevents two retries from treating a shared name as a lock. It also makes cleanup auditable: a lifecycle process can target completed objects by prefix, while an application job can remove one object when its deadline arrives. Do not put a signed URL in the authoritative record. It will expire; the key and deadline should not.
Throughput needs its own budget. Let P be the part size, W the number of concurrent uploads, and R the retry reserve. A worker needs room for the ZIP writer, W * P in-flight buffers, and enough local scratch space for its archive strategy. Bigger parts usually reduce request overhead, but they also increase the work lost when one part fails. Your mileage may vary; the right value comes from measurements of compression, memory pressure, link capacity, concurrent exports, and the cost of rebuilding a part.
That is the uncomfortable part.
A deadline that is precise in the database but vague in cleanup is not precise in production. For example, if a district administrator requests a signed-document ZIP at 09:00 and its deletion deadline is 18:00, a link expiring at 09:15 limits the capability window; it does not authorize the cleanup job to wait until tomorrow.
Keep it private.
How should a Node.js worker handle a large ZIP export with multipart upload and a signed download link?
Keep the browser out of the archive-writing path. The worker can stream source documents into a ZIP, buffer one bounded part, upload that part, and persist the returned part metadata before moving on. After every required part is acknowledged, complete the multipart session. Only then should the application create a signed download link for the object key and an expiry that matches the user's download window.
The following Python sketch shows the control flow without pretending that every object-storage system has the same SDK or request vocabulary. The adapter is the boundary to test against a real implementation; the job logic is where retention and publication rules belong.
from dataclasses import dataclass
from datetime import datetime
from pathlib import Path
from typing import Iterable
import zipfile
@dataclass
class ExportRecord:
export_id: str
object_key: str
delete_after: datetime
upload_id: str | None = None
completed_parts: list[dict] | None = None
class StorageAdapter:
def create_multipart(self, object_key: str) -> str:
raise NotImplementedError
def upload_part(self, upload_id: str, part_number: int, data: bytes) -> dict:
raise NotImplementedError
def complete_multipart(self, upload_id: str, parts: list[dict]) -> None:
raise NotImplementedError
def abort_multipart(self, upload_id: str) -> None:
raise NotImplementedError
def signed_download(self, object_key: str, expires_in_seconds: int) -> str:
raise NotImplementedError
def build_zip(paths: Iterable[Path], output: Path) -> None:
with zipfile.ZipFile(output, "w", zipfile.ZIP_DEFLATED) as archive:
for path in paths:
archive.write(path, path.name)
def publish_export(storage, record, source_paths, scratch_dir, part_size):
record.completed_parts = record.completed_parts or []
archive = scratch_dir / f"{record.export_id}.zip"
build_zip(source_paths, archive)
record.upload_id = record.upload_id or storage.create_multipart(record.object_key)
try:
with archive.open("rb") as stream:
part_number = 1
while chunk := stream.read(part_size):
part = storage.upload_part(record.upload_id, part_number, chunk)
record.completed_parts.append(part)
part_number += 1
storage.complete_multipart(record.upload_id, record.completed_parts)
return storage.signed_download(record.object_key, expires_in_seconds=900)
except Exception:
storage.abort_multipart(record.upload_id)
raise
In a real worker, persist the record after each successful part and make the retry path reconcile that record with the storage session before uploading more data. The sketch intentionally leaves the adapter abstract: request methods, part-size limits, checksum fields, and signed-link options vary, and inventing a universal endpoint would create false confidence. The completion call is the publication barrier. A link created before it can point at an object that is not ready.
There is a second failure boundary worth testing: the worker may finish uploading the final part and die before completion. On retry, the job should first load the export record, inspect the stored upload identifier and completed part metadata, and determine whether the completion transition was already recorded; if it was, the retry can verify the ready state and issue a link, while if it was not, the retry can reconcile the session and submit completion with the durable part list. Only after that state is clear should it consider a new archive. Otherwise a queue retry can rebuild a multi-gigabyte ZIP, leave an abandoned session behind, and create two possible objects for one export identity, which makes both cleanup and access review harder. Idempotency belongs in the job record and queue design, not in hope.
No premature link.
Failure modes to test before measuring throughput
A large-file design should name its failures before it names its provider. These are the cases I would put in a review:
| Failure mode | Required behavior | Evidence to retain |
|---|---|---|
| Worker stops during ZIP creation | Retry from source data or a durable intermediate, with bounded disk use | Job status and worker attempt ID |
| Part upload is interrupted | Retry only the affected part when its prior result is not durable | Part number, checksum or storage tag, attempt timestamp |
| Final part arrives before completion | Keep the object unpublished until explicit completion succeeds | Upload ID and completion state |
| Link is requested after the deadline | Refuse issuance, even if the object still exists briefly | Decision timestamp and retention deadline |
| Cleanup job is delayed | Alert and retry deletion; do not extend access silently | Deletion attempt and final deletion state |
| Duplicate job is enqueued | Use a stable export identity and unique object key | Queue ID, export ID, object key |
The deletion path deserves the same attention as the upload path. A scheduled lifecycle rule can help, but an application deadline still needs monitoring because cleanup timing, rule granularity, and retention configuration are separate from signed-link expiry. Never treat URL expiry as proof that the bytes were deleted.
For student records, log the decision to issue a link without logging the link itself. A URL containing a signature is a bearer capability. Apply least privilege to the worker credential, keep the object private, and make the download service check job ownership before asking storage to sign anything.
Which object-storage trade-offs belong in the design review?
Multipart support is only one row in the decision. Record the exact behavior for incomplete sessions, lifecycle deletion, checksums, conditional writes, versioning, object immutability, regional placement, egress, and signed-request expiry. An S3-compatible interface does not prove identical retention or authorization semantics across systems.
| Decision axis | Why it matters for this export | Question to answer |
|---|---|---|
| Part retry and completion | Determines rebuild cost after a worker failure | Can the job resume from durable part metadata? |
| Lifecycle deletion | Determines how closely storage cleanup follows the deadline | What is the documented minimum or granularity? |
| Immutability and versioning | Determines whether an overwrite can be recovered | Are accidental replacement and legal retention separate controls? |
| Signed retrieval | Determines who can read a completed archive and for how long | Can the service sign one private object without making the bucket public? |
| Operational surface | Determines what the team must monitor and rotate | Are credentials, regions, quotas, and audit events visible to operators? |
The catch is that this pattern is not suitable when the requirement is a permanent public asset, anonymous browser access, immutable legal hold, or built-in cross-region replication. Use a separately managed public delivery layer, an immutability-capable repository, or a replication design when those guarantees are mandatory. Stick with a simpler single upload when the archive is small enough that a retry is cheaper than multipart state, but record that choice as a measured constraint rather than a default.
I’m not sure a single numeric threshold can serve every school or district. Compression ratio, source locality, concurrent exports, and the user's download window change the answer. Run a representative archive through the worker, kill it at each state transition, and use the resulting memory, retry, and completion data to set the policy.
A compact rollout for signed-document exports
Ship the smallest state machine first: queued, building, uploading, completing, ready, expired, and deleted. Add metrics for bytes produced, part retries, active sessions, completion latency, link-issuance decisions, and deletion lag. Alert on sessions that remain incomplete beyond the job timeout and on objects past their deadline.
Then test the contract, not just the happy path. A worker should survive termination during ZIP creation, a failed part, and the gap after the final part. A download request after expiry should receive no signed capability. A cleanup retry should be visible and idempotent. A duplicate queue message should not publish two records for one export identity.
That rollout gives the team a decision rule: choose multipart when large-file throughput and restart cost dominate, choose a simpler path when archive rebuilds are cheap, and choose stronger retention controls when deletion deadlines are not the only obligation. The signed link is the last step in that chain, never the chain itself.
Top comments (0)