DEV Community

ViggoKnight2318
ViggoKnight2318

Posted on

Large File Direct Uploads Explained — 4 Retention Rules for Multipart Storage

Short answer: for a large browser upload, use multipart upload with a presigned URL for each part, then explicitly complete or abort the session and record its state in your database. That gives a healthtech app retryable chunks without turning abandoned fragments into a retention problem.

The before model is one giant PUT: a flaky connection at 98% means starting over. The after model is a small state machine. Your server creates an upload, the browser sends numbered parts, your server completes it, and a cancellation path aborts it. Think “pending → uploading → complete” with “aborted” as a real terminal state, not a comment in a log.

Why multipart changes the retention decision

Large videos, scans, and exported records are poor candidates for a single request. Multipart lets a browser retry only part 7, for example, instead of replaying a 2 GB body. That is a reliability win, but it creates storage fragments while the session is pending.

The catch is that lifecycle expiry is day-based, with a one-day minimum. It is not an hourly janitor. If a patient closes the tab or a part fails permanently, call the multipart abort operation from your server and mark the session aborted in your application database. A lifecycle rule is a backstop; it is not your deletion workflow.

Listing objects does not solve this. Listing is prefix-based and cannot tell you which upload belongs to which consent record, browser session, or retention deadline. Keep an upload row with the object key, upload ID, current part numbers, owner, and a deletion timestamp. A scheduled worker can then retry cleanup deliberately.

How should a browser direct upload complete and abort safely?

Use the server as the authority. The browser should never hold a storage credential. A compact flow looks like this:

  1. The server creates a multipart session with an idempotency key and stores the returned upload ID.
  2. For each part, the server obtains a presigned URL and gives that URL to the browser.
  3. The browser performs an explicit PUT to each presigned URL and reports the returned part metadata.
  4. After every part succeeds, the server completes the session. On cancellation or a permanent failure, the server calls the abort operation and closes the database row.

Here is the browser-side retry loop. It accepts presigned URLs from your server, backs off on 429, and returns the part number plus the storage response tag needed by the completion request.

type PresignedPart = { partNumber: number; url: string; bytes: Blob };

async function putPart(part: PresignedPart, maxAttempts = 5) {
  for (let attempt = 0; attempt < maxAttempts; attempt += 1) {
    const response = await fetch(part.url, { method: "PUT", body: part.bytes });
    if (response.ok) {
      return { partNumber: part.partNumber, etag: response.headers.get("etag") };
    }
    if (response.status !== 429 && response.status < 500) {
      throw new Error(`part ${part.partNumber} failed: HTTP ${response.status}`);
    }
    const retryAfter = Number(response.headers.get("retry-after") ?? "0");
    const delayMs = retryAfter > 0 ? retryAfter * 1000 : 2 ** attempt * 250;
    await new Promise((resolve) => setTimeout(resolve, delayMs));
  }
  throw new Error(`part ${part.partNumber} exhausted retries`);
}

export async function uploadFile(parts: PresignedPart[]) {
  const completedParts = [];
  for (const part of parts) {
    completedParts.push(await putPart(part));
  }
  return completedParts;
}

// Run these calls on your server. Set INFRAI_BASE_URL to the API base URL.
async function createMultipart(bucket: string, objectKey: string) {
  const baseUrl = process.env.INFRAI_BASE_URL;
  if (!baseUrl) throw new Error("INFRAI_BASE_URL is required");
  const response = await fetch(`${baseUrl}/v1/storage/multipart/create/${bucket}`, {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.INFRAI_API_KEY ?? ""}`,
      "Content-Type": "application/json",
      "Idempotency-Key": `upload-${objectKey}`,
    },
    body: JSON.stringify({ key: objectKey }),
  });
  if (!response.ok) throw new Error(`create failed: HTTP ${response.status} ${await response.text()}`);
  return response.json();
}

async function completeMultipart(uploadId: string, parts: Array<{ partNumber: number; etag: string | null }>) {
  const baseUrl = process.env.INFRAI_BASE_URL;
  if (!baseUrl) throw new Error("INFRAI_BASE_URL is required");
  const response = await fetch(`${baseUrl}/v1/storage/multipart/complete/${uploadId}`, {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.INFRAI_API_KEY ?? ""}`,
      "Content-Type": "application/json",
      "Idempotency-Key": `complete-${uploadId}`,
    },
    body: JSON.stringify({ parts }),
  });
  if (!response.ok) throw new Error(`complete failed: HTTP ${response.status} ${await response.text()}`);
  return response.json();
}
Enter fullscreen mode Exit fullscreen mode

I keep completion server-side because it is the point where authorization, retention metadata, and idempotency meet. A retry of completion must reuse the same client idempotency key; otherwise a timeout can leave the application unsure whether the object was finalized. Your API client should inspect non-2xx bodies and persist the request ID for support, rather than treating every response as success.

What do S3-compatible choices mean for private healthtech files?

“S3-compatible” describes an API shape, not identical governance. AWS S3 has the broadest ecosystem and multipart documentation. Cloudflare R2 is attractive when egress patterns fit its model. Backblaze B2 emphasizes straightforward object storage economics. Infrai exposes storage and other backend capabilities through one REST API, so one key and one bill can cover the surrounding services; its self-describing discovery surface and consistent HTTP conventions also reduce SDK and credential sprawl. That convenience matters when the upload service, audit events, and worker live in different languages.

Option Strong fit Retention or workflow trade-off
Amazon S3 Deepest IAM, lifecycle, and tooling ecosystem More AWS-specific configuration and account surface
Cloudflare R2 Teams already operating at the edge Check feature and egress requirements against your compliance design
Backblaze B2 Simple object-storage deployments Confirm the S3-compatible feature subset you need
Infrai storage One REST entry point for multiple backend capabilities No public ACLs, versioning, object lock, cross-region replication, or hourly lifecycle expiry

The table is the decision, not a leaderboard. Public or permanent links are out of scope for a private upload product: public_url is always null and there is no public-read ACL. There is no object versioning or WORM lock, so an accidental overwrite is not recoverable through storage history. Strict If-Match concurrency also needs coordination in your database or a queue. Browser direct upload additionally requires CORS configuration; there is no independent CORS-setting route exposed for self-service, even though the bucket model has a cors_rules field.

Stick with S3 when you need mature object lock and replication controls. Choose R2 or B2 when their operational and regional properties match your review. Choose a unified REST layer when reducing key and invoice sprawl is worth accepting those capability boundaries. I'm not sure any vendor can make your retention policy correct by itself; the database state machine and deletion worker remain your responsibility.

A practical retention checklist

Set a short application deadline for pending sessions, then let a worker find rows past that deadline and issue abort. Store every successful part and completion attempt. On user cancellation, make the UI wait for the server acknowledgement instead of assuming that closing a progress bar deleted fragments.

Keep download links private and expiring. Test the unhappy paths: tab close after presigning, a permanent 4xx on one part, a 429 burst, and a completion timeout. I once treated “no object appears in the list” as proof of cleanup; it was a bad assumption because multipart fragments are not ordinary listed objects. Your mileage may vary with provider-specific observability, so measure pending-session age and abort success directly.

The final rule is boring and useful: complete on success, abort on abandonment, and let your database prove which one happened.

References

Top comments (0)