In the early days, the web was mostly text. Forms, usernames, comments all just plain text data flying back and forth between browser and server.
But then people wanted more. They wanted to upload profile pictures, videos, PDFs, documents. And that's when things got interesting, because the web was never really built for that in the first place.
In this post, I want to walk you through how file uploads evolved, step by step, the same way I understood it myself. No fancy jargon, just the actual reasoning behind why each piece exists.
The First Problem: JSON Can't Handle Files
Let's say you want to send this to your server:
{
"username": "rohit",
"profilePic": "??"
}
Sounds simple, right? Except a JPEG is not text. It's raw binary bytes. And JSON was only ever designed to carry text-based values strings, numbers, booleans, arrays, null. It has no idea what to do with raw binary data.
So people found a workaround: convert the binary file into base64 text, and stuff that text inside the JSON.
It works. But it comes with real problems:
- The file size balloons (base64 adds roughly 33% overhead)
- Encoding and decoding costs extra CPU
- Huge JSON payloads become inefficient to send and parse
So the web needed a better way to move binary data around. That's how multipart/form-data was born.
Enter multipart/form-data
The idea here is simple: instead of forcing everything into one text format, split a single HTTP request into multiple independent parts. Each part can carry a different kind of data.
Here's roughly what that looks like on the wire:
----------------boundary
Content-Disposition: form-data; name="username"
rohit
----------------boundary
Content-Disposition: form-data; name="image"; filename="avatar.jpg"
Content-Type: image/jpeg
[BINARY FILE BYTES]
----------------boundary--
So one request can now carry:
- Part 1 → plain text
- Part 2 → plain text
- Part 3 → binary image
- Part 4 → another binary file, if needed
The browser sends a header like this to tell the server what's coming:
Content-Type: multipart/form-data; boundary=----abc123
Why You Need a Parser (and Why express.json() Isn't Enough)
If you've built even a basic form in Express, you've probably written:
app.use(express.json());
This tells Express how to understand JSON bodies. But here's the catch — express.json() has no idea what to do with uploaded files. It simply doesn't parse multipart data.
When a file is uploaded, the server receives a raw byte stream, and something needs to:
- Read the incoming HTTP stream
- Find the multipart boundaries
- Separate text fields from files
- Process the file bytes
- Hand clean, usable data to your controller
This is exactly why tools like Busboy, Formidable, and Multer exist. In the Express ecosystem, Multer became one of the most common ways to handle this.
Note: Multer does not magically upload your file to the internet. Its only job is to parse and handle the incoming multipart request. What you do with the file after that is entirely up to you.
The Simple (and Naive) Way to Store Files
The most basic approach looks like this:
User uploads image → App server receives it → Saves to /uploads folder
And your database might store something like:
| username | avatarUrl |
|---|---|
| rohit | /uploads/av1.jpg |
For a small app, this genuinely works fine. But the moment your app starts to grow, this approach starts falling apart.
The Problem With Local Storage at Scale
Imagine your app grows and you add a load balancer:
Users → Load Balancer → (Server A, Server B)
Now say a user uploads their picture, and it lands on Server A's disk. Later, that same user's request gets routed to Server B by the load balancer and the file is nowhere to be found.
Congratulations, you've just discovered a classic distributed systems problem. You could try to sync files between servers, but that adds a whole new layer of complexity you probably don't want to maintain.
The real fix is to separate compute from file storage.
Dedicated Object Storage
Users → Load Balancer → (Server A, Server B) → Object Storage
Instead of saving files inside the app server, you push them to dedicated storage think AWS S3, Cloudinary, Google Cloud Storage, or Azure Blob Storage.
This changes how your database is used too:
- Database → structured application data
- Object storage → the actual file bytes
This separation is fundamental. Once you internalize it, a lot of backend architecture starts making more sense.
Large Files Create a New Bottleneck
Object storage solves the "which server has my file" problem. But there's still an issue with how the file gets there in the first place.
Browser → 500MB video → Backend Server → same 500MB video → Object Storage
See the problem? Your backend server becomes a middleman for a file it doesn't even need to touch. For a single large upload, your app server now has to deal with:
- Network bandwidth
- Memory or temporary disk usage
- Connection duration
- Concurrent upload handling
Your backend becomes the bottleneck, even though it never actually needed to process those bytes. This is what led to the next big shift.
Direct-to-Storage Uploads
Most modern systems now use this flow instead:
- Browser → Backend: "I want permission to upload a file"
- Backend → Browser: "Here's a temporary upload permission"
- Browser → Object Storage: Uploads the file directly, skipping your backend entirely
- Browser → Backend: "Upload completed"
This is exactly where terms like presigned URLs, signed uploads, and direct uploads come from. Your backend just hands out temporary, scoped permission — it never has to carry the actual file weight.
Quick Recap
| Stage | What Happened |
|---|---|
| 1 | Web starts as text-only (forms, plain data) |
| 2 | Demand for files grows — base64 + JSON is tried, but it's inefficient |
| 3 |
multipart/form-data is introduced to carry mixed data in one request |
| 4 | Parsers like Multer, Busboy, Formidable appear to handle multipart streams |
| 5 | Small apps save files to a local /uploads folder |
| 6 | Multiple servers expose the local-storage problem |
| 7 | Dedicated object storage separates files from app servers |
| 8 | Backend-proxied uploads become a bottleneck for large files |
| 9 | Direct-to-storage uploads with temporary, presigned permissions solve it |
What's Next
I didn't want to cram everything into one giant post, so next time I'll cover:
- Memory vs disk storage during uploads
- MIME type validation
- Magic-byte validation (why trusting the file extension is a bad idea)
- Compression
See you in the next one. Keep learning.
Top comments (0)