Short answer: for an edtech service that serves generated reports, define the deletion drill before selecting the multipart upload flow. A completed upload isn't permission to keep a report, and an aborted upload isn't evidence that every related object has disappeared.
The useful unit here is not a file transfer. It is a customer-visible custody promise: a report must be private while it is valid, restorable when it is published, and removable when retention or access policy says so. Large app backup files merely make the gaps easier to expose.
Start there.
The deletion drill should define the upload design
Take one generated report containing student work and its source attachments. Ask what happens when the customer cancels during part 7, when access is revoked after publication, and when the delete request times out. The answer should name a durable record, an authorization decision, a storage operation, and an observable terminal state. If it only says “retry the upload,” the design is missing its real contract.
The drill also separates two kinds of data. An edtech report archive may be deleted under the customer-facing retention policy while an internal database recovery backup remains under a different operational schedule. A bucket age rule can assist with cleanup, but it cannot know which tenant lost access or which report was superseded.
Keep the object private. The download handler should authorize from the report record, not from a guessed key. On deletion, revoke that authorization first, request removal of the published object, and record the result. For unfinished multipart work, record the reason and call the documented abort operation. A later reconciliation pass can find anything left in an ambiguous state.
That is the first design decision: retention is a state transition, not a side effect of a worker finishing its queue.
Consider the awkward sequence that makes this concrete. A report producer writes an archive, the worker sends several parts, and the customer cancels after the next part is accepted. The queue may redeliver the job while the cancellation transaction is still being observed by another worker. If that worker sees only a local “uploading” flag, it can continue sending bytes after the customer is no longer authorized to receive the result. If it sees only a remote multipart listing, it cannot know whether the source archive, tenant policy, or deletion request changed. The durable custody record has to win that race: the new worker checks the lease and policy, records the cancellation, aborts unfinished work, and leaves enough evidence for a reconciliation job to handle an ambiguous response. This is why the part ledger is an application concern even when the storage API already tracks parts.
The adapter should translate the storage service's documented multipart operations into a small application interface. It should not decide whether a tenant is allowed to retain an archive or whether a restore is safe. Those decisions belong above it, where the database row and authorization policy are visible.
The application record needs an opaque tenant identifier, report job identifier, deterministic object key, retention deadline, deletion state, upload identifier, part size, and acknowledged part metadata. Store a source archive identity or content digest when the producer can calculate one. If the producer cannot reproduce the same byte ranges after a restart, a new upload identifier is safer than combining two archive attempts.
The following policy object shows the boundary without pretending to be a provider SDK:
from dataclasses import dataclass, field
@dataclass
class ReportArchive:
tenant_id: str
object_key: str
upload_id: str
retention_until: str
parts: dict[int, str] = field(default_factory=dict)
state: str = "transferring"
def acknowledge_part(self, number: int, etag: str) -> None:
if self.state != "transferring":
raise RuntimeError("terminal archive cannot accept a part")
self.parts[number] = etag
def publish(self, expected_count: int) -> None:
expected = list(range(1, expected_count + 1))
if self.state != "transferring" or sorted(self.parts) != expected:
raise RuntimeError("the complete part set is required")
self.state = "published"
def request_deletion(self, reason: str) -> None:
if self.state == "deleted":
return
if not reason:
raise ValueError("deletion needs an auditable reason")
self.state = "deletion_requested"
def record_deletion(self) -> None:
if self.state != "deletion_requested":
raise RuntimeError("deletion was not requested")
self.state = "deleted"
This is a policy example, not a wire-protocol claim. The adapter owns request serialization; the job system owns leases, retries, retention, and the meaning of each terminal state. I don't want those responsibilities blurred by a convenient SDK call.
Writing the record first makes migrations and restarts legible. Create the backup row before the multipart creation request, then persist the upload identifier and part size. Each worker should hold a lease, use stable part numbers, and save an acknowledgement before releasing that lease. Two workers must not independently decide what part 7 means.
For a process restart, the next worker verifies that the object key, source archive identity, and part size still match. It discovers remote parts with the service's documented listing operation and reconciles them with the durable record. Missing ranges can be sent again; a changed source archive becomes a new attempt.
This is also the migration path from a filesystem archive. Keep the old artifact readable while a new record is created, copy or regenerate into a private object key, validate the restored result, and switch the download authorization only after publication. Delete the old copy in a separate, auditable step. Do not make a storage move and a customer-access change one irreversible operation.
The important failure modes are operationally different:
| Event | State to preserve | Next action |
|---|---|---|
| Worker exits after a part response | Upload identifier and known parts | Reconcile before resuming |
| Queue message is delivered twice | Lease and stable part number | Allow one owner to coordinate the job |
| Completion request times out | Publication uncertainty | Inspect object state before retrying |
| Customer cancels after publication | Revoked authorization and deletion reason | Apply the published-object deletion policy |
| Retention deadline passes | Deadline and deletion attempt | Deny downloads and reconcile removal |
| Restore validation fails | Quarantine state and diagnostic result | Keep the archive unavailable |
No single status field should pretend to capture all of that. published, restore_verified, and deleted answer different questions.
How do resume, complete, and abort work for large backup files?
Resume repairs transfer progress. Complete publishes a known part set. Abort disposes of unfinished multipart work. A generic retry wrapper around all three makes a timeout look like a deletion decision, which is precisely the ambiguity the record should prevent.
On resume, compare the durable record with remote multipart state and resend only missing or intentionally replaced ranges. A successful part response is evidence, but the local acknowledgement still needs to be committed. On complete, require a contiguous expected part set and correct part metadata, then update the report's published state only after the documented completion result.
On abort, record cancellation, validation failure, an expired lease, or another policy reason. Keep that record for audit even after the remote cleanup succeeds. If the request is repeated, the reconciliation job should reach the same interpretation instead of creating a new customer-visible state.
After publication, run a restore check in an isolated location through the same authenticated path a customer would use. Verify the archive manifest against the report job, and expose restore validation separately from transfer completion. 200 OK for one part is not proof that the backup is restorable.
Private object storage with multipart transfer fits a generated report that customers download through an authenticated application, as long as the team owns the ledger, authorization, deletion reconciliation, and restore checks. It is not suitable when the actual requirement is point-in-time recovery of operational database state and the team cannot maintain restore orchestration.
A database-native backup facility may be the better boundary for database recovery, but it does not automatically become a customer-facing report file. A local filesystem can be a staging boundary for one worker or a small export; it becomes a risky long-term boundary when durability, sharing, and cleanup span multiple workers.
The trade-off is governance. A provider-neutral adapter can hide differences in immutability, audit, region, or deletion controls that operators need to see. Stick with a native integration when those controls are mandatory and the team is prepared to keep that coupling. Your mileage may vary because the correct boundary follows the recovery and erasure promises, not the transfer vocabulary.
Start with one report type and a narrow tenant set. Define the state transitions and deletion outcomes before adding parallel workers. Then stop a worker after a part response, duplicate the queue message, interrupt completion, cancel after publication, run deletion twice, and validate a restored archive in isolation.
Only then tune concurrency. Bound in-flight parts per job and tenant, and expose transfer, publication, restore-validation, and deletion states separately. More workers may improve throughput, but they also increase file-descriptor, archive-producer, and coordination pressure.
The practical Node.js multipart upload example is therefore a retention-controlled workflow. The storage calls move bytes; the custody record decides when resume, complete, abort, restore, and delete are allowed to mean what the customer thinks they mean.
Top comments (0)