DEV Community

Fares Galal
Fares Galal

Posted on

Tracking File Upload Progress on AWS S3 – Lessons from Large File Uploads

#aws #frontend #tutorial #typescript

Post ThumbnailHave you ever tried uploading a large file to AWS S3 and wanted to show progress reliably?

I recently worked on a feature that involved uploading videos and large files, and I learned a lot about browser limitations, multipart uploads, and UX improvements. Here’s a breakdown of what I discovered.

Why onUploadProgress Alone Isn't Enough

For small files, Axios gives us a handy onUploadProgress callback:

axios.put(uploadUrl, file, {
  headers: { "Content-Type": file.type },
  onUploadProgress: (progressEvent) => {
    const percent = Math.round((progressEvent.loaded * 100) / progressEvent.total!);
    console.log("Progress:", percent, "%");
  },
});
Enter fullscreen mode Exit fullscreen mode

✅ Works perfectly for files ≤ 20MB

But when we tried large files (150MB+):

  • Progress either lagged behind
  • Or jumped directly to 100% at the end

Why? Because the browser sends the file in a single request, making onUploadProgress unreliable.

The Solution: Multipart Upload (Chunking)

With large files, the file must be split into chunks before uploading:

  • Each chunk is a separate request
  • Progress is tracked per chunk
  • Failed chunks can be retried individually

Example:

// 500MB file with 20MB chunks → 25 requests
const CHUNK_SIZE = 20 * 1024 * 1024;
Enter fullscreen mode Exit fullscreen mode

Some might ask: Isn’t 25 requests a lot?
Not really. Each request is short-lived and uploaded with controlled concurrency.
This makes uploads predictable, faster, and easier to retry on network failures.

Choosing the Right Chunk Size

AWS S3 requires minimum 5MB per chunk.

We define:

const MIN_CHUNK_SIZE = 5 * 1024 * 1024;       // 5MB
const DESIRED_CHUNK_SIZE = 20 * 1024 * 1024; // 20MB
Enter fullscreen mode Exit fullscreen mode

Then check the file size:

  • ≤ 20MB → Upload normally (no multipart)
  • > 20MB → Split into chunks and start multipart upload

Why 20MB? There’s no magic number where uploads fail. Factors like network speed, connection stability, and device performance all play a role.

Backend Workflow for Multipart Uploads

Previously, the backend returned a single S3 URL:

{
  "uploadUrl": "https://my-bucket.s3.amazonaws.com/file.mp4"
}
Enter fullscreen mode Exit fullscreen mode

Now it returns:

{
  "uploadId": "UPLOAD_ID_PLACEHOLDER",
  "key": "/FILE_NAME_PLACEHOLDER",
  "parts": [
    {
      "partNumber": 1,
      "uploadUrl": "https://my-bucket.s3.amazonaws.com/KEY_PLACEHOLDER"
    }
  ],
  "expiresAt": "EXPIRATION_DATE_PLACEHOLDER"
}
Enter fullscreen mode Exit fullscreen mode
  • Each chunk is uploaded to its corresponding uploadUrl
  • S3 returns an ETag for each chunk → acts as a fingerprint to confirm the chunk was uploaded successfully
  • ETags + part numbers are required for CompleteMultipartUpload to assemble the file in order

Tracking Progress for Large Files

  • Small files → browser events track progress by bytes
  • Large files → track progress based on the number of chunks uploaded

Benefits:

  1. Progress is stable and predictable
  2. Better UX
  3. No sudden jumps or delays
  4. Failed chunks can retry individually, no need to restart the entire upload

Final Step: Complete the Upload

Once all chunks are uploaded, send a final request with:

  • uploadId
  • key
  • All ETags

The backend calls CompleteMultipartUpload → file is assembled in S3 as if it were uploaded in a single request.

Takeaways

Switching to chunked uploads with progress tracking dramatically improves the experience for large files:

  • Predictable progress
  • Stable uploads
  • Retry for individual chunks
  • Better UX for your users

Top comments (0)