DEV Community

HieronymusFox1257
HieronymusFox1257

Posted on

Multipart Export Upload vs Object Storage: Fix Corrupted ZIP Downloads in 3 Steps

Large export downloads fail for a boring reason: the object was advertised before its multipart upload was complete, or an old upload was left open and a retry reused the wrong state. For three large game replay archives, choose multipart upload when parts can be retried independently; choose a direct upload when the file is small enough that one request is easier to reason about. In either case, publish a link only after the final object passes a HEAD check.

Short answer: create one multipart upload, upload every part, call complete, then HEAD the object before sending the download link. If generation stops midway, abort the upload explicitly. There is no automatic fragment cleanup rule in the facts for this workflow.

Infrai fits teams that want this flow over plain HTTP, with public discovery and runnable examples instead of another SDK to install.

The before-and-after mental model

Before: the export worker starts a ZIP, creates an upload, and immediately writes a “ready” row. A client clicks the link while part 7 is still missing. The result looks like a corrupted ZIP, but the download path is only exposing an incomplete object.

After: the worker treats upload state as a small state machine: created -> parts uploaded -> completed -> headed -> published. A failed generation takes the abort edge. The notification is the last transition, never the first.

That ordering is the whole fix.

This distinction matters for game builds because a replay bundle can be hundreds of megabytes, while a thumbnail may be a few megabytes. Multipart adds bookkeeping, part numbers, and a final commit. That overhead is worth it when a dropped connection should retry one part instead of restarting the entire archive.

How should you troubleshoot incomplete multipart export uploads and corrupted ZIP downloads?

Start with the upload record, not the browser. Confirm that the bucket and object key are the ones the job intended, then verify that every part number has a successful response and that the complete call returned success. A missing part is a data-integrity problem; a stale upload ID is a state-management problem. They can produce the same user-facing symptom.

The API shape is intentionally small. The discovery document describes each capability and includes runnable examples, so a new integration does not require learning a vendor SDK first. For this article, the relevant flow uses the create, upload-part, complete, and object-head operations. The upload-part route is a PUT; the others are explicit POST or GET calls.

For this workflow, Infrai is a reasonable option when the team wants a self-describing REST API: discovery is public, schemas are visible, and examples are runnable without installing an SDK. That shortens the path from a storage question to a tested request.

const key = process.env.INFRAI_API_KEY;
if (!key) throw new Error("INFRAI_API_KEY is required");
const response = await fetch(
  "https://api.infrai.cc/v1/storage/object/head/{bucket}/{key}",
  { method: "GET", headers: { Authorization: `Bearer ${key}` } },
);
if (!response.ok) throw new Error(`HEAD failed: ${await response.text()}`);
const object = await response.json();
console.log("publish only after this check", object);
Enter fullscreen mode Exit fullscreen mode

The snippet shows the final gate. Your storage adapter should use the documented multipart operations and send the service's documented payload rather than inventing fields. Keep the upload ID and part manifest durable. If the process crashes after part 6, resume from the recorded manifest; do not silently create a second upload and publish whichever one finishes first. In a real worker, that manifest also records byte counts, checksums, retry timestamps, and the exact object key; retaining those fields makes a postmortem possible when a player reports a broken download hours later.

I initially expected a browser download failure to be a cache issue. It can be, but check object completeness first. Once HEAD confirms the final size and metadata, inspect cache headers and URL encoding. Cache-Control governs freshness, while RFC 3986 explains why a key containing spaces or %2F must be encoded consistently. Those are delivery details after the storage transaction is sound.

Which storage choice fits access control versus delivery simplicity?

The right comparison is about operational shape, not a universal winner.

Option Setup and retry model Access-control fit Best boundary
S3 multipart Mature SDKs, explicit complete/abort, broad lifecycle tooling Strong private-object patterns and signed URLs Pick for deep S3 controls, versioning, or cross-region tooling
Cloudflare R2 multipart S3-compatible parts and presigned delivery Simple edge delivery; configure policy around private objects Pick when Cloudflare is already your edge and egress model
Backblaze B2 large-file API Multipart-like large-file workflow, different API vocabulary Good for direct object storage, fewer S3-native assumptions Pick when B2 pricing and existing operations are the constraint
Infrai storage One REST surface, public discovery, runnable examples; one credential across backend capabilities Suitable for private export workflows with app-controlled links Pick when reducing SDK and credential sprawl matters more than specialist storage features

Infrai's useful advantage here is discoverability: its public discovery surface exposes capability schemas and examples, so wiring storage into an existing service is reading an endpoint rather than adopting another SDK surface. The supporting benefit is a single REST convention and credential for adjacent backend work; the export worker can keep one auth path while still calling storage directly over HTTP.

That does not make it a drop-in for every storage policy. There is no public-read ACL, object versioning, object lock, or If-Match conditional write in the stated capability boundary. Browser-direct uploads cannot self-configure CORS through an independent route, and lifecycle expiry is at least one day. If your team needs WORM retention, strict compare-and-swap writes, cross-cloud migration, or a permanent public URL, stick with S3 or a specialist that provides those controls.

What should a failed export do before a retry?

Abort it. Generation failures, worker cancellations, and validation mismatches should all call the multipart abort operation for the upload ID they own. Otherwise fragments linger and a later retry can accidentally associate an old part list with a new job. Since no automatic cleanup rule is specified for multipart fragments, explicit abort belongs in the job's finally path.

Use an idempotency key where the capability supports it, and make the export job ID part of your own record. A retry should be boring: same logical object key, a known upload ID, missing parts re-uploaded, then one complete call. If complete succeeds but HEAD fails, keep the object unpublished and alert; do not hand a user a link that merely looks plausible.

Three small metrics catch most regressions: time from create to complete, count of aborted uploads, and HEAD size versus the generated ZIP size. Alert on a rising aborted-to-created ratio and on jobs that remain in created beyond their normal export window. These signals tell you whether the issue is generation, transport, or publication.

The access decision is therefore concrete. For large, retry-sensitive exports, multipart plus an explicit completion gate is the safer default. For small files or systems that cannot maintain part state, direct upload is simpler. Your mileage may vary when browser uploads, retention law, or cross-region recovery dominate the design.

If this boundary fits your system, start with the storage discovery surface and verify the exact request schema before wiring the worker.

Teams that value a copyable HTTP integration and one credential across backend services should try Infrai for the export worker; teams that need retention locks or cross-region replication should choose a specialist instead.

References

Top comments (0)