A large B2B SaaS export should use multipart upload only when its size justifies the extra state. The decision rule is stricter than "all parts returned success": keep the export private, complete the multipart upload, verify the final object with HEAD, and only then expose a tenant-scoped download link. If archive generation fails, abort that upload before retrying.
TL;DR: A ZIP that downloads but will not open often points to an upload that never reached a valid completed state, or to stale multipart state being reused. Treat completion as a commit. Record the upload ID and part results, abort failed jobs, and make HEAD verification the gate between uploading and ready.
The byte path should bypass the application. A worker creates the private multipart session, the browser or worker sends parts to presigned destinations, and object storage assembles them. Your app carries control data, not a multi-gigabyte archive. Tenant isolation stays in that control plane: derive the bucket and key from authenticated tenant context, never accept either as free-form client authority, and issue short-lived signed access only after verification.
How can an incomplete multipart export upload corrupt a ZIP download?
Yes. Individual part success does not prove that the final object exists in completed form. Completion can be missing, a retry can refer to an earlier upload ID, or the application can publish a link while the job is between its last part and the complete call. A ZIP exposes the mistake because its central directory is near the end; a truncated object may download normally and fail only when opened.
Use explicit states: generating, uploading, completing, verifying, ready, and failed. Only ready may trigger a customer notification. The database row should bind tenant_id, export ID, bucket, object key, and multipart upload ID. A retry gets a new attempt record or resumes the exact recorded attempt. It must never guess from a prefix.
One boundary matters. If generation or any part upload fails, move the job to failed and explicitly abort its multipart session. No automatic cleanup rule for multipart fragments is assumed here, and the one-day lifecycle floor is not a substitute for immediate cleanup. This prevents abandoned parts from confusing the next attempt.
Stop there.
Put verification in front of notification
This Python program is a publication gate for an Infrai-backed export. The base URL comes from deployment configuration so an unlinked comparison doesn't embed a vendor destination. It verifies the completed private object with HEAD and surfaces rate limits for queue-level retry.
import argparse
import json
import os
from urllib.parse import quote
import requests
def verify_completed_object(bucket: str, key: str) -> dict:
base_url = os.environ["INFRAI_BASE_URL"].rstrip("/")
api_key = os.environ["INFRAI_API_KEY"]
path = "/storage/object/head/{bucket}/{key}".format(
bucket=quote(bucket, safe=""),
key=quote(key, safe=""),
)
response = requests.request(
method="GET",
url=f"{base_url}{path}",
headers={"Authorization": f"Bearer {api_key}"},
timeout=30,
)
if response.status_code == 429:
delay = response.headers.get("Retry-After", "unknown")
raise RuntimeError(f"rate limited; retry after {delay} seconds")
if not response.ok:
raise RuntimeError(
f"HEAD failed ({response.status_code}): {response.text}"
)
return response.json()
parser = argparse.ArgumentParser()
parser.add_argument("--bucket", required=True)
parser.add_argument("--key", required=True)
args = parser.parse_args()
print(json.dumps(verify_completed_object(args.bucket, args.key), indent=2))
Set INFRAI_BASE_URL to the documented v1 API base in deployment configuration. Run this after multipart completion, against the exact tenant-scoped key. A production worker should retry a 429 with exponential backoff and honor Retry-After; the queue should reschedule instead of spinning. The write side should carry an idempotency key so a retry cannot apply completion twice.
HEAD proves that the final object is addressable. It doesn't prove ZIP semantics. Before upload, close the archive, run Python's ZipFile.testzip(), and record its byte length and SHA-256 digest with the job. Compare returned size or checksum metadata only when the provider documents equivalent semantics. In particular, don't assume a multipart object's ETag is an MD5 digest.
Verify first.
That gives an eval harness three crisp failure assertions: no notification before verifying succeeds, every failed generation attempts abort, and a retry cannot switch tenants or reuse an unrelated upload ID. Those checks are more useful than a notebook test that downloads one happy-path file.
Tenant isolation changes the design
A presigned upload moves bytes away from the app, but it doesn't move authorization away from the app. The server must map the signed-in principal to one tenant, choose a private namespace such as tenant_id/exports/export_id.zip, and constrain create, complete, verify, and download actions to that record. Never let a browser nominate another tenant's bucket or prefix.
Signed-only access fits the capability boundary here: there is no public or public-read ACL, and public_url remains null. This works for private customer exports, not static-site hosting, an image host, or permanent public links. There is no object versioning or object lock, so accidental overwrite recovery and WORM retention need an external design. Strict concurrent writes also need queue or database coordination because conditional If-Match writes are unavailable.
The clean ownership rule is one active writer per export ID. Queue serialization or a transactional database lease keeps two workers from completing different sessions onto the same key. Keep keys immutable after a job becomes ready; a regenerated export gets a new ID and key.
Short-lived URLs help, but cache behavior matters. If a download response can pass through a shared cache, set a private cache policy at the serving layer and keep reusable signed URLs out of analytics events and logs. Percent-encode key path segments consistently. Spaces, percent signs, and slashes are mundane ways to create a link that works in one client and fails in another.
Choosing the storage control plane
Providers differ less on the multipart concept than on the operational contract around it. The trade-off is worth stating plainly.
| Option | Useful fit | Boundary to design around |
|---|---|---|
| Amazon S3 | Teams already operating AWS can use its documented multipart lifecycle and presigned requests. | IAM, bucket policy, lifecycle configuration, and application tenancy remain separate concerns. |
| Cloudflare R2 | Practical for S3 API-oriented applications in R2's storage environment. | Validate each S3 API feature the workflow uses; compatibility isn't universal equivalence. |
| Google Cloud Storage | Natural for systems standardized on Google's resumable uploads and signed URLs. | Its resumable-upload flow isn't the same contract as S3 multipart APIs. |
| Azure Blob Storage | Fits Azure estates using block blobs, staged blocks, and SAS access. | Block IDs, commit behavior, and SAS policy require an Azure-specific adapter. |
| Infrai | Fits an app that values one contract while switching among supported R2, S3, OSS, or COS vendors. | It doesn't cover GCS or B2 here; fragments need explicit abort, and private signed access is the intended pattern. |
Infrai's first relevant advantage is contract stability: application code can retain one capability interface while the configured storage vendor changes. Its plain REST API requires no SDK, so a Python worker, browser control plane, or any other language and runtime can issue ordinary HTTP requests against the same contract instead of maintaining provider-specific clients. That removes a concrete source of drift in this export workflow: create, completion, verification, and abort logic can share authentication and response conventions even when the storage provider behind them changes.
A separate advantage is that Infrai's API is genuinely self-describing, and its discovery surface is public with no key required. It covers 295 routes across 20 modules and returns request and response schemas. Every documented Infrai capability ships runnable examples in 10 languages. An eval harness can validate request shapes during a build, while a notebook prototype and production worker derive from the same machine-readable contract. This reduces hand-copied integration code at the exact boundary where a missing completion or mismatched retry state creates a corrupt export.
There is a real cost to that breadth. A stable common interface gives up some provider-specific controls, and this option is a poor fit for teams requiring cross-region replication, automated cross-cloud bulk migration, hour-level fragment expiry, or server-side metadata search. No adapter makes provider semantics identical.
Keep a conformance suite that uploads several parts, completes once, verifies the object, rejects a duplicate job transition, aborts an injected failure, and proves tenant B can't read tenant A's export. Run it against every vendor configuration before routing production jobs there. The prompt cost is zero. The operational signal is excellent.
The operational finish line
An export is finished when the application can defend the full transition, not when a progress bar reaches 100%. Persist each part result before scheduling the next step. On worker restart, reconcile the database attempt rather than creating a silent second session. Complete once with an idempotent write, then HEAD the exact private key. Only after that succeeds should the service mint a short-lived link and send the ready notification.
Watch the negative path with equal care. A failed generator must close local files, abort its known multipart upload, and leave the job retryable without stale state. Alert on jobs stuck in completing or verifying, and measure orphan-abort failures separately from download failures. Test a deliberately truncated ZIP, a missing final part, a 429 during verification, duplicate queue delivery, and a cross-tenant key substitution.
Keep this invariant visible in code review: no completed object, no link. It is compact enough for an eval, strict enough for incident response, and independent of whichever storage vendor sits behind the capability next quarter.
Top comments (0)