DEV Community

Cover image for How to Upload a 2 GB File Directly to S3 Without Touching Your Backend
Abhinav
Abhinav

Posted on

How to Upload a 2 GB File Directly to S3 Without Touching Your Backend

Imagine we're building a video-sharing platform.

A user wants to upload a 2 GB video. Our backend needs to authorize the upload, but does it really need to receive all 2 GB of data?

Not necessarily.

There is a much better approach: S3 Pre-Signed URLs with Direct-to-S3 Uploads.

This is a common architectural pattern used in production systems to handle large file uploads efficiently.

Let's understand how it works and why it matters from a backend engineering perspective.

1. The Problem With Traditional File Uploads

Consider a traditional architecture:

the infographic showing the traditional 2 GB upload flow, where the backend handles the entire file transfer between the client and Amazon S3.

Our backend acts as an intermediary between the client and S3.

What's wrong with this?

1. Unnecessary bandwidth consumption

The backend receives 2 GB and transfers another 2 GB to S3.

2. Memory pressure

If we buffer the entire file in memory, concurrent uploads can exhaust available memory.

Streaming reduces memory usage, but doesn't eliminate the network overhead.

3. Connection management

Long-running uploads occupy connections and consume backend resources.

4. Scalability

Imagine 100 users simultaneously uploading 2 GB files.

That's 200 GB of incoming traffic and another 200 GB of outgoing traffic through our backend.

Our application server becomes a bottleneck.

We need a different architecture.

2. The Solution: Pre-Signed URLs

Instead of making our backend handle the actual file transfer, we let the client communicate directly with S3.

Our backend only generates a temporary URL that authorizes the upload.

An infographic illustrating how to upload a large 2 GB file directly to Amazon S3 using a pre-signed URL.

Notice something important:

The backend never receives the actual file bytes.

It only handles authorization and upload initialization.

S3 handles the data transfer.

3. What Exactly Is a Pre-Signed URL?

A pre-signed URL is a temporary URL generated using AWS credentials that grants permission to perform a specific operation on a particular S3 object.

For example:

https://my-bucket.s3.amazonaws.com/videos/abc.mp4
?X-Amz-Algorithm=AWS4-HMAC-SHA256
&X-Amz-Credential=...
&X-Amz-Date=...
&X-Amz-Expires=900
&X-Amz-Signature=...
Enter fullscreen mode Exit fullscreen mode

Let's understand what's happening internally.

  1. Our backend has AWS credentials, preferably through an IAM role.
  2. It constructs a request containing the bucket, object key and HTTP method.
  3. AWS Signature Version 4 generates a cryptographic signature.
  4. The backend returns the signed URL.
  5. The client uses that URL to upload the file directly to S3.
  6. S3 validates the signature, expiration and permissions.

The client doesn't need AWS credentials.

Think of a pre-signed URL as a temporary access pass.

Anyone possessing that URL can use its granted permission until it expires, subject to its restrictions.

4. Let's Implement It Using Node.js

First, install the AWS SDK:

npm install @aws-sdk/client-s3 @aws-sdk/s3-request-presigner
Enter fullscreen mode Exit fullscreen mode

Backend: Generate a Pre-Signed URL

import { S3Client, PutObjectCommand } from "@aws-sdk/client-s3";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";
import { randomUUID } from "node:crypto";

const s3 = new S3Client({
  region: "ap-south-1"
});

async function generateUploadUrl(req, res) {
  const fileKey = `uploads/${randomUUID()}`;

  const command = new PutObjectCommand({
    Bucket: "my-bucket",
    Key: fileKey,
    ContentType: "video/mp4"
  });

  const signedUrl = await getSignedUrl(
    s3,
    command,
    { expiresIn: 900 }
  );

  res.json({
    uploadUrl: signedUrl,
    fileKey
  });
}
Enter fullscreen mode Exit fullscreen mode

Our backend generates a URL valid for 15 minutes.

Notice that we're generating the object key on the backend rather than accepting an arbitrary S3 path from the client.

This helps prevent unauthorized object overwrites.

Client: Upload Directly to S3

async function uploadFile(file) {
  // Step 1: Request upload authorization
  const response = await fetch("/api/upload-url", {
    method: "POST"
  });

  const { uploadUrl, fileKey } = await response.json();

  // Step 2: Upload directly to S3
  const uploadResponse = await fetch(uploadUrl, {
    method: "PUT",
    headers: {
      "Content-Type": file.type
    },
    body: file
  });

  if (!uploadResponse.ok) {
    throw new Error("Upload failed");
  }

  return fileKey;
}
Enter fullscreen mode Exit fullscreen mode

We now have two separate communication paths:

  • Client ↔ Backend: Authorization.
  • Client ↔ S3: Actual file transfer.

Our backend is no longer in the data-transfer path.

5. Is a Single PUT Request Enough for a 2 GB File?

Technically, yes.

S3 supports a single PUT operation for objects up to 5 GB.

However, for large files, we generally prefer Multipart Upload.

Consider this scenario:

We're uploading a 2 GB video.

The upload reaches 1.9 GB, but suddenly the network disconnects.

With a single PUT request, we generally need to retry the entire upload.

That's wasteful.

Multipart Upload solves this problem.

6. Understanding Multipart Upload

Instead of uploading one large file, we divide it into smaller parts.

For example:

A 2 GB file can be divided into 20 parts of approximately 100 MB each.

The important advantage?

If Part 15 fails, we only retry Part 15.

We don't need to upload the entire 2 GB file again.

Multipart Upload has three important operations

1. CreateMultipartUpload

We initiate a multipart upload.

S3 returns an UploadId.

This ID identifies the multipart upload session.

2. UploadPart

We upload individual parts using:

  • Bucket
  • Object Key
  • Upload ID
  • Part Number

Each successful upload returns an ETag.

3. CompleteMultipartUpload

Once all parts are uploaded, we send their part numbers and ETags to S3.

S3 assembles them into the final object.

There is also an important cleanup operation:

AbortMultipartUpload

If an upload is abandoned, we should abort it to clean up incomplete parts.

7. What Does the Multipart Architecture Look Like?

the Multipart Architecture Look Like

The client can upload multiple parts concurrently, subject to a reasonable concurrency limit.

For example, instead of uploading all 20 parts simultaneously, we might allow only 3–5 concurrent uploads.

This avoids overwhelming the client's network connection.

8. Production Engineering: What Can Go Wrong?

Generating a signed URL is relatively straightforward.

Designing a reliable upload system is the more interesting engineering problem.

Here are the concerns to consider

A. Security

Never blindly trust the client.

Our backend should:

  • Authenticate the user.
  • Verify upload permissions.
  • Generate unique object keys.
  • Restrict the allowed content type.
  • Enforce file-size limits.
  • Keep the S3 bucket private.

Remember: a pre-signed URL is a bearer capability. Anyone who obtains it can use its granted permissions.

B. Retry Mechanism

Network failures are inevitable.

Multipart uploads allow us to retry individual failed parts.

We should also implement exponential backoff:

const delay = (attempt) =>
  Math.min(1000 * 2 ** attempt, 10000);
Enter fullscreen mode Exit fullscreen mode

We should retry only appropriate transient failures, not every error blindly.

C. Resume Uploads

Suppose the browser closes after uploading 12 out of 20 parts.

If we persist the upload ID, part numbers and ETags, we can resume the upload.

We can use S3's ListParts API to inspect which parts have already been uploaded.

This is especially useful for unreliable networks.

D. Upload State Management

S3 stores the file, but our application still needs to understand its business state.

We can maintain an upload table:

uploads
--------------------------------
id
user_id
s3_key
upload_id
status
file_size
content_type
created_at
updated_at
Enter fullscreen mode Exit fullscreen mode

Possible statuses:

INITIATED
UPLOADING
COMPLETED
FAILED
ABORTED
Enter fullscreen mode Exit fullscreen mode

Why do we need this?

Because receiving an upload ID doesn't mean the file has been uploaded.

And receiving a completion request from the client doesn't automatically mean the object is valid.

Our backend should verify the completed object using S3 metadata, expected size and relevant checksums before marking it usable.

E. Duplicate Completion Requests

Consider this:

  1. Client requests upload completion.
  2. Backend calls S3.
  3. S3 successfully completes the upload.
  4. Backend response gets lost due to a network failure.
  5. Client retries completion.

Now we have a duplicate request.

Our completion API should handle this safely.

We can maintain upload state, verify the object in S3 and make the completion operation idempotent from the application's perspective.

F. Incomplete Upload Cleanup

Users may abandon uploads halfway through.

S3 retains incomplete multipart parts until the upload is completed or aborted.

We should:

  • Abort abandoned uploads.
  • Run periodic cleanup jobs.
  • Configure an S3 lifecycle rule to abort incomplete multipart uploads after a defined number of days.

9. A Very Important Architectural Concept: Control Plane vs Data Plane

This is probably the most important takeaway.

We have separated two responsibilities.

Control Plane — Our Backend

  • Authentication
  • Authorization
  • Upload initialization
  • Pre-signed URL generation
  • Metadata management
  • Upload lifecycle
  • Business validation

Data Plane — AWS S3

  • Actual file transfer
  • Multipart storage
  • Part management
  • Object assembly
  • Object persistence

Our backend controls the operation without handling the actual data transfer.

This separation makes the architecture more scalable.

10. When Should We Use This Architecture?

This pattern is useful whenever applications handle large files.

Examples:

  • Video-sharing platforms.
  • Document management systems.
  • Profile image uploads.
  • Backup services.
  • Medical document uploads.
  • Data import pipelines.

The backend should handle application logic, while dedicated storage infrastructure handles the actual file transfer.


Final Thoughts

The interesting part of this architecture isn't generating a pre-signed URL.

It's understanding why we don't want our application server to become a proxy for every large file upload.

By moving file transfer directly to S3, we reduce unnecessary backend bandwidth, improve scalability and separate application responsibilities from storage responsibilities.

And when we combine pre-signed URLs with multipart uploads, retries, resumability, validation and cleanup, we get something much closer to a production-grade upload system.

A good backend doesn't necessarily do all the work. Sometimes, good engineering means knowing which component should do the work.


Topics: #systemdesign #aws #s3 #nodejs #backend #distributedSystems #softwarearchitecture

Top comments (0)