Imagine we're building a video-sharing platform.
A user wants to upload a 2 GB video. Our backend needs to authorize the upload, but does it really need to receive all 2 GB of data?
Not necessarily.
There is a much better approach: S3 Pre-Signed URLs with Direct-to-S3 Uploads.
This is a common architectural pattern used in production systems to handle large file uploads efficiently.
Let's understand how it works and why it matters from a backend engineering perspective.
1. The Problem With Traditional File Uploads
Consider a traditional architecture:
Our backend acts as an intermediary between the client and S3.
What's wrong with this?
1. Unnecessary bandwidth consumption
The backend receives 2 GB and transfers another 2 GB to S3.
2. Memory pressure
If we buffer the entire file in memory, concurrent uploads can exhaust available memory.
Streaming reduces memory usage, but doesn't eliminate the network overhead.
3. Connection management
Long-running uploads occupy connections and consume backend resources.
4. Scalability
Imagine 100 users simultaneously uploading 2 GB files.
That's 200 GB of incoming traffic and another 200 GB of outgoing traffic through our backend.
Our application server becomes a bottleneck.
We need a different architecture.
2. The Solution: Pre-Signed URLs
Instead of making our backend handle the actual file transfer, we let the client communicate directly with S3.
Our backend only generates a temporary URL that authorizes the upload.
Notice something important:
The backend never receives the actual file bytes.
It only handles authorization and upload initialization.
S3 handles the data transfer.
3. What Exactly Is a Pre-Signed URL?
A pre-signed URL is a temporary URL generated using AWS credentials that grants permission to perform a specific operation on a particular S3 object.
For example:
https://my-bucket.s3.amazonaws.com/videos/abc.mp4
?X-Amz-Algorithm=AWS4-HMAC-SHA256
&X-Amz-Credential=...
&X-Amz-Date=...
&X-Amz-Expires=900
&X-Amz-Signature=...
Let's understand what's happening internally.
- Our backend has AWS credentials, preferably through an IAM role.
- It constructs a request containing the bucket, object key and HTTP method.
- AWS Signature Version 4 generates a cryptographic signature.
- The backend returns the signed URL.
- The client uses that URL to upload the file directly to S3.
- S3 validates the signature, expiration and permissions.
The client doesn't need AWS credentials.
Think of a pre-signed URL as a temporary access pass.
Anyone possessing that URL can use its granted permission until it expires, subject to its restrictions.
4. Let's Implement It Using Node.js
First, install the AWS SDK:
npm install @aws-sdk/client-s3 @aws-sdk/s3-request-presigner
Backend: Generate a Pre-Signed URL
import { S3Client, PutObjectCommand } from "@aws-sdk/client-s3";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";
import { randomUUID } from "node:crypto";
const s3 = new S3Client({
region: "ap-south-1"
});
async function generateUploadUrl(req, res) {
const fileKey = `uploads/${randomUUID()}`;
const command = new PutObjectCommand({
Bucket: "my-bucket",
Key: fileKey,
ContentType: "video/mp4"
});
const signedUrl = await getSignedUrl(
s3,
command,
{ expiresIn: 900 }
);
res.json({
uploadUrl: signedUrl,
fileKey
});
}
Our backend generates a URL valid for 15 minutes.
Notice that we're generating the object key on the backend rather than accepting an arbitrary S3 path from the client.
This helps prevent unauthorized object overwrites.
Client: Upload Directly to S3
async function uploadFile(file) {
// Step 1: Request upload authorization
const response = await fetch("/api/upload-url", {
method: "POST"
});
const { uploadUrl, fileKey } = await response.json();
// Step 2: Upload directly to S3
const uploadResponse = await fetch(uploadUrl, {
method: "PUT",
headers: {
"Content-Type": file.type
},
body: file
});
if (!uploadResponse.ok) {
throw new Error("Upload failed");
}
return fileKey;
}
We now have two separate communication paths:
- Client ↔ Backend: Authorization.
- Client ↔ S3: Actual file transfer.
Our backend is no longer in the data-transfer path.
5. Is a Single PUT Request Enough for a 2 GB File?
Technically, yes.
S3 supports a single PUT operation for objects up to 5 GB.
However, for large files, we generally prefer Multipart Upload.
Consider this scenario:
We're uploading a 2 GB video.
The upload reaches 1.9 GB, but suddenly the network disconnects.
With a single PUT request, we generally need to retry the entire upload.
That's wasteful.
Multipart Upload solves this problem.
6. Understanding Multipart Upload
Instead of uploading one large file, we divide it into smaller parts.
For example:
A 2 GB file can be divided into 20 parts of approximately 100 MB each.
The important advantage?
If Part 15 fails, we only retry Part 15.
We don't need to upload the entire 2 GB file again.
Multipart Upload has three important operations
1. CreateMultipartUpload
We initiate a multipart upload.
S3 returns an UploadId.
This ID identifies the multipart upload session.
2. UploadPart
We upload individual parts using:
- Bucket
- Object Key
- Upload ID
- Part Number
Each successful upload returns an ETag.
3. CompleteMultipartUpload
Once all parts are uploaded, we send their part numbers and ETags to S3.
S3 assembles them into the final object.
There is also an important cleanup operation:
AbortMultipartUpload
If an upload is abandoned, we should abort it to clean up incomplete parts.
7. What Does the Multipart Architecture Look Like?
The client can upload multiple parts concurrently, subject to a reasonable concurrency limit.
For example, instead of uploading all 20 parts simultaneously, we might allow only 3–5 concurrent uploads.
This avoids overwhelming the client's network connection.
8. Production Engineering: What Can Go Wrong?
Generating a signed URL is relatively straightforward.
Designing a reliable upload system is the more interesting engineering problem.
Here are the concerns to consider
A. Security
Never blindly trust the client.
Our backend should:
- Authenticate the user.
- Verify upload permissions.
- Generate unique object keys.
- Restrict the allowed content type.
- Enforce file-size limits.
- Keep the S3 bucket private.
Remember: a pre-signed URL is a bearer capability. Anyone who obtains it can use its granted permissions.
B. Retry Mechanism
Network failures are inevitable.
Multipart uploads allow us to retry individual failed parts.
We should also implement exponential backoff:
const delay = (attempt) =>
Math.min(1000 * 2 ** attempt, 10000);
We should retry only appropriate transient failures, not every error blindly.
C. Resume Uploads
Suppose the browser closes after uploading 12 out of 20 parts.
If we persist the upload ID, part numbers and ETags, we can resume the upload.
We can use S3's ListParts API to inspect which parts have already been uploaded.
This is especially useful for unreliable networks.
D. Upload State Management
S3 stores the file, but our application still needs to understand its business state.
We can maintain an upload table:
uploads
--------------------------------
id
user_id
s3_key
upload_id
status
file_size
content_type
created_at
updated_at
Possible statuses:
INITIATED
UPLOADING
COMPLETED
FAILED
ABORTED
Why do we need this?
Because receiving an upload ID doesn't mean the file has been uploaded.
And receiving a completion request from the client doesn't automatically mean the object is valid.
Our backend should verify the completed object using S3 metadata, expected size and relevant checksums before marking it usable.
E. Duplicate Completion Requests
Consider this:
- Client requests upload completion.
- Backend calls S3.
- S3 successfully completes the upload.
- Backend response gets lost due to a network failure.
- Client retries completion.
Now we have a duplicate request.
Our completion API should handle this safely.
We can maintain upload state, verify the object in S3 and make the completion operation idempotent from the application's perspective.
F. Incomplete Upload Cleanup
Users may abandon uploads halfway through.
S3 retains incomplete multipart parts until the upload is completed or aborted.
We should:
- Abort abandoned uploads.
- Run periodic cleanup jobs.
- Configure an S3 lifecycle rule to abort incomplete multipart uploads after a defined number of days.
9. A Very Important Architectural Concept: Control Plane vs Data Plane
This is probably the most important takeaway.
We have separated two responsibilities.
Control Plane — Our Backend
- Authentication
- Authorization
- Upload initialization
- Pre-signed URL generation
- Metadata management
- Upload lifecycle
- Business validation
Data Plane — AWS S3
- Actual file transfer
- Multipart storage
- Part management
- Object assembly
- Object persistence
Our backend controls the operation without handling the actual data transfer.
This separation makes the architecture more scalable.
10. When Should We Use This Architecture?
This pattern is useful whenever applications handle large files.
Examples:
- Video-sharing platforms.
- Document management systems.
- Profile image uploads.
- Backup services.
- Medical document uploads.
- Data import pipelines.
The backend should handle application logic, while dedicated storage infrastructure handles the actual file transfer.
Final Thoughts
The interesting part of this architecture isn't generating a pre-signed URL.
It's understanding why we don't want our application server to become a proxy for every large file upload.
By moving file transfer directly to S3, we reduce unnecessary backend bandwidth, improve scalability and separate application responsibilities from storage responsibilities.
And when we combine pre-signed URLs with multipart uploads, retries, resumability, validation and cleanup, we get something much closer to a production-grade upload system.
A good backend doesn't necessarily do all the work. Sometimes, good engineering means knowing which component should do the work.
Topics: #systemdesign #aws #s3 #nodejs #backend #distributedSystems #softwarearchitecture




Top comments (0)