Anirban Roy

Posted on Sep 23

How I Send 100GB+ Files Directly Between Browsers Without Cloud Storage

#javascript #networking #webdev

Most file-sharing products are built around the same basic idea:

Upload the file to a server.
Store it somewhere.
Generate a link.
Let the recipient download it later.

That model is simple, reliable, and works extremely well for a lot of products.

But it also creates a problem the moment you start dealing with very large files.

A 20MB PDF is easy.

A 500MB video is manageable.

A 5GB project starts to become noticeable.

At 50GB, 100GB, or more, suddenly everything becomes about storage quotas, upload times, bandwidth bills, server infrastructure, expired links, and paid plans.

I kept coming back to one question:

If the file already exists on one computer and the final destination is another computer, why does the entire file need to live on my server in the middle?

That question eventually became the foundation for ButterShare.

ButterShare is a browser-based peer-to-peer file transfer product. Instead of uploading the entire file to cloud storage first, the two browsers establish a connection and transfer the file directly.

The basic architecture is:

Traditional file sharing:

Sender
   |
   v
Cloud Storage
   |
   v
Receiver

ButterShare is closer to:

Sender
   |
   v
Receiver

Of course, the real implementation is more complicated than that diagram suggests.

There is signaling.

There is WebRTC.

There are NATs and firewalls.

There is buffering.

There is flow control.

There are browser memory limits.

There are mobile devices that decide your background tab is no longer important.

And there is the very uncomfortable question:

What happens when a 100GB transfer fails at 97%?

This post is about what I learned while designing large browser-to-browser file transfer and why sending 100GB directly between browsers is a very different problem from sending a normal attachment.

The first realization: file transfer and file storage are different products

This sounds obvious, but a lot of file-sharing software combines the two.

When someone says:

“Send this file to John.”

there are at least two possible interpretations.

Storage-based sharing

The sender uploads the file somewhere.

The server keeps it.

John downloads it whenever he wants.

This is asynchronous.

The sender can close their laptop and disappear.

Direct transfer

The sender and John are both online.

The file moves from one device to the other.

Once the transfer is complete, the infrastructure does not need to retain the actual file.

This is synchronous.

They solve different problems.

Cloud storage is better when persistence matters.

Direct transfer becomes interesting when both people are online and the only objective is moving data from A to B.

For large files, removing the storage layer changes a lot.

Why 100GB changes the architecture

Imagine a user wants to send a 100GB file.

With a conventional cloud architecture, the file usually crosses your infrastructure twice.

First:

Sender -> Your server/storage

Then:

Your server/storage -> Receiver

That means a single 100GB transfer can result in roughly 200GB of file traffic moving through infrastructure you operate.

And you still have to store the file somewhere in between.

Now multiply that.

100 users transferring 100GB:

10 TB uploaded
10 TB downloaded

1,000 users:

100 TB uploaded
100 TB downloaded

Suddenly your “simple file sharing app” becomes a storage and bandwidth business.

That can absolutely be the correct business model.

But it was not the model I wanted for ButterShare.

I wanted the infrastructure to coordinate the connection without becoming the permanent warehouse for the actual file.

Why WebRTC fits this problem

WebRTC is usually associated with video calls.

But underneath the video/audio features is something extremely useful for file transfer:

WebRTC Data Channels.

A data channel lets two peers exchange arbitrary data.

That can be:

text
JSON
binary data
file chunks
application messages

So instead of streaming video frames between browsers, you can stream pieces of a file.

Very simplified:

const peerConnection = new RTCPeerConnection()

const channel = peerConnection.createDataChannel("file-transfer")

channel.onopen = () => {
  console.log("Peer connected")
}

channel.onmessage = (event) => {
  // receive file data
}

The interesting part is that once the peer connection is established, the data can often travel directly between the two devices.

Your application server does not need to handle each file chunk.

That is the key architectural difference.

But WebRTC does not magically connect two browsers

Before two browsers can communicate, they have to find each other.

This is where signaling comes in.

The signaling server exchanges connection metadata between the peers.

Typical information includes:

SDP offers
SDP answers
ICE candidates

The rough process looks like this:

Sender Browser
     |
     | create offer
     v
Signaling Server
     |
     | forward offer
     v
Receiver Browser

Receiver Browser
     |
     | create answer
     v
Signaling Server
     |
     | forward answer
     v
Sender Browser

Both sides also exchange ICE candidates.

Eventually WebRTC attempts to establish the actual peer connection.

The important distinction is:

The signaling server helps the devices connect. It does not necessarily carry the file itself.

That distinction matters enormously when the file is 100GB.

A few KB of signaling traffic is trivial.

100GB of file traffic is not.

STUN, TURN, and the part everyone leaves out

This is where the nice architecture diagram starts becoming less clean.

You cannot assume every pair of devices can establish a perfect direct connection.

Users sit behind:

routers
NAT
carrier-grade NAT
corporate networks
restrictive firewalls
mobile networks
VPNs

WebRTC uses ICE to figure out how the peers can communicate.

A STUN server helps a device discover its public-facing network information.

In ideal conditions:

Browser A <----------------> Browser B

Direct.

Beautiful.

Cheap.

Fast.

But sometimes direct connectivity is impossible.

That is where a TURN server becomes important.

A TURN server relays traffic:

Browser A
    |
    v
TURN Relay
    |
    v
Browser B

At that point, your infrastructure may be handling the traffic again.

The important difference is that relaying encrypted traffic is not the same thing as permanently storing the file.

The relay forwards packets.

It does not need to create a downloadable cloud copy of the file.

Still, from an infrastructure perspective, TURN traffic is real bandwidth and needs to be treated seriously.

So “peer-to-peer” does not mean:

100% of every transfer will always bypass every server.

It means:

The system attempts direct peer connectivity and can avoid traditional cloud file storage.

That is a much more accurate description.

The naive implementation fails immediately

When developers first build browser file transfer, the natural approach often looks something like this:

const buffer = await file.arrayBuffer()
dataChannel.send(buffer)

This is fine for small files.

It is a terrible idea for 100GB files.

If the file is 100GB, you obviously do not want to load 100GB into browser memory.

Even far smaller files can create problems.

The browser may:

freeze
consume huge amounts of memory
kill the tab
become unresponsive
crash on mobile

Large file transfer needs to be streaming-oriented.

The file should be processed incrementally.

Chunking the file

Instead of treating the file as one huge object, divide it into manageable chunks.

Conceptually:

100GB file

[chunk 1]
[chunk 2]
[chunk 3]
[chunk 4]
...
[chunk 1,500,000]

The exact number depends on your chunk size.

A simplified sender might look like this:

const CHUNK_SIZE = 64 * 1024

let offset = 0

while (offset < file.size) {
  const chunk = file.slice(offset, offset + CHUNK_SIZE)
  const buffer = await chunk.arrayBuffer()

  dataChannel.send(buffer)

  offset += CHUNK_SIZE
}

This solves one major problem:

You no longer need the whole file in memory.

But this code still has another serious problem.

It sends data as fast as JavaScript can produce it.

The network cannot necessarily keep up.

Backpressure is one of the most important parts

A WebRTC data channel has an internal send buffer.

You can inspect it using:

dataChannel.bufferedAmount

If you continuously call:

dataChannel.send(...)

faster than the connection can transmit data, the buffer keeps growing.

Eventually you can create:

huge memory usage
poor performance
browser instability
transfer failure

So you need backpressure.

Instead of endlessly pushing data, you wait when the channel buffer gets too large.

Something like:

const MAX_BUFFER = 4 * 1024 * 1024

async function waitForBuffer() {
  while (dataChannel.bufferedAmount > MAX_BUFFER) {
    await new Promise(resolve => setTimeout(resolve, 10))
  }
}

Then:

while (offset < file.size) {
  await waitForBuffer()

  const chunk = file.slice(offset, offset + CHUNK_SIZE)
  const buffer = await chunk.arrayBuffer()

  dataChannel.send(buffer)

  offset += CHUNK_SIZE
}

A better implementation can use:

bufferedAmountLowThreshold

and:

onbufferedamountlow

rather than polling.

Conceptually:

dataChannel.bufferedAmountLowThreshold = 1024 * 1024

function waitUntilWritable() {
  if (dataChannel.bufferedAmount <= dataChannel.bufferedAmountLowThreshold) {
    return Promise.resolve()
  }

  return new Promise(resolve => {
    dataChannel.onbufferedamountlow = () => resolve()
  })
}

This is one of the biggest differences between a demo file-transfer project and something designed for genuinely large files.

Metadata comes before the bytes

The receiver needs to know what is coming.

Before transmitting file chunks, send metadata.

For example:

{
  "type": "file-start",
  "name": "project-final.mov",
  "size": 107374182400,
  "mimeType": "video/quicktime"
}

The receiver can then prepare for the incoming stream.

You may also send information such as:

{
  "id": "file_123",
  "relativePath": "project/videos/project-final.mov",
  "chunkSize": 65536,
  "totalChunks": 1638400
}

This becomes particularly important when sending folders containing multiple files.

The transport layer should understand the difference between:

file metadata

and:

binary file chunk

and:

transfer complete

A simple message protocol might include:

TRANSFER_START
FILE_START
FILE_CHUNK
FILE_END
TRANSFER_END
ERROR
ACK

Once you start thinking in terms of a protocol rather than random WebRTC messages, the system becomes much easier to reason about.

Why 100GB means you need BigInt thinking

JavaScript numbers can safely represent integers up to:

9,007,199,254,740,991

So 100GB itself is nowhere near the safe-integer limit.

But large-file systems still benefit from being disciplined about offsets, byte counts, chunk indexes, and serialization.

You do not want something like this hidden deep inside your protocol:

progress = chunksReceived * chunkSize

without considering the final partial chunk, reconnect offsets, multiple files, and aggregated folder size.

For real transfers, I prefer tracking actual byte counts:

bytesSent += chunk.byteLength

Then progress becomes:

const percentage = (bytesSent / totalBytes) * 100

Simple, but reliable.

Progress bars lie if you build them badly

A file-transfer progress bar looks trivial.

It is not.

Suppose you have sent data into the WebRTC buffer.

Does that mean:

JavaScript processed it?
the browser queued it?
the network transmitted it?
the receiver received it?
the receiver wrote it to disk?

Those are different stages.

If you update progress immediately after:

dataChannel.send(chunk)

you are measuring how much data you queued, not necessarily how much arrived.

For small files, nobody notices.

For a 100GB transfer, the difference can become substantial.

A more reliable system can use receiver acknowledgements.

For example:

Sender:
chunk 1000 sent

Receiver:
chunk 1000 received

Sender:
update confirmed progress

You probably do not want an acknowledgement for every tiny chunk because that adds overhead.

Instead, acknowledge ranges or byte offsets periodically.

Example:

{
  "type": "ack",
  "receivedBytes": 5368709120
}

Now the sender knows the receiver has confirmed roughly 5GB.

That is much more meaningful.

The receiver has the same memory problem

Chunking the sender is only half the solution.

A naive receiver might do this:

const chunks = []

channel.onmessage = event => {
  chunks.push(event.data)
}

const blob = new Blob(chunks)

Again, fine for small files.

Terrible for 100GB.

You have just moved the memory problem from the sender to the receiver.

You do not want to accumulate 100GB of chunks in RAM.

For truly large files, the receiver needs a way to write progressively.

Modern browser capabilities can make this possible, but support varies.

Depending on the browser and platform, you may use mechanisms such as:

File System Access API
writable file streams
browser-specific download strategies
temporary storage
service workers

This is where browser compatibility becomes a major design constraint.

Desktop Chromium gives you capabilities that mobile Safari may not.

If your product promises cross-platform transfer, you cannot design only for Chrome on your MacBook.

Desktop browsers are easy mode compared with mobile

Large file transfer on desktop is one problem.

Large file transfer on phones is another.

Mobile browsers have much tighter constraints around:

memory
storage
background execution
battery
tab suspension
screen locking
operating-system lifecycle

A user can start a transfer and then:

lock their phone
switch apps
put the browser in the background
receive a phone call
enter low-power mode

And suddenly your beautiful WebRTC connection is gone.

This creates UX requirements you do not think about in a desktop-only prototype.

For example:

Keep this tab open until the transfer finishes.

That message may sound primitive, but users need to understand the synchronous nature of peer-to-peer transfer.

You also need to handle disconnects gracefully instead of pretending they will never happen.

The hardest question: what happens when the connection dies?

Imagine transferring 100GB.

You reach:

97.4GB

Then Wi-Fi drops.

If your solution is:

“Please start again.”

the user is going to hate you.

For genuinely large files, resumability becomes extremely valuable.

That means the receiver and sender need to agree on how much data has already been transferred.

Conceptually:

{
  "fileId": "abc123",
  "receivedBytes": 104689827840
}

After reconnection:

Sender:
Okay, you already have 104,689,827,840 bytes.

Resume from there.

Then instead of:

file.slice(0)

you resume from:

file.slice(receivedBytes)

That sounds easy.

It becomes more complicated with:

multiple files
folders
changed files
chunk verification
interrupted writes
reconnecting to the wrong session

Once again, what started as a file-transfer feature becomes a protocol-design problem.

File integrity matters

A transfer showing “100%” does not necessarily mean the resulting file is correct.

Networks and transport protocols are reliable at lower layers, but once you build your own application-level chunking, reconnection, writing, and resume logic, verification becomes valuable.

One option is hashing.

Conceptually:

Sender hash:
abc123...

Receiver hash:
abc123...

If they match, great.

For huge files, hashing the entire file before transfer can itself take time.

So you may prefer chunk-level hashing or incremental hashing.

For example:

Chunk 1 -> hash
Chunk 2 -> hash
Chunk 3 -> hash

or calculate a streaming digest while data moves.

The right choice depends on how much integrity protection your product needs versus the CPU and complexity cost.

Folder transfer introduces another layer

Sending one 100GB file is actually simpler than sending a folder containing:

30,000 files

even if the folder is only 20GB.

Now you need to preserve:

filenames
relative paths
directory structure
file ordering
metadata

Conceptually:

project/
├── assets/
│   ├── logo.png
│   └── hero.jpg
├── video/
│   └── final.mov
└── project.json

You cannot just send bytes.

You need a manifest.

Something like:

{
  "files": [
    {
      "path": "assets/logo.png",
      "size": 204812
    },
    {
      "path": "assets/hero.jpg",
      "size": 5410289
    },
    {
      "path": "video/final.mov",
      "size": 42342353433
    }
  ]
}

Now the receiver knows how to reconstruct the directory structure.

This is why “send folder” is not just “send file but bigger.”

Encryption

One advantage of WebRTC is that encryption is part of the transport.

WebRTC data channels use secure transport mechanisms, so you are not sending arbitrary plaintext packets across the internet.

But you still need to think carefully about the broader security model.

For example:

How does someone join a transfer?

Maybe you use:

room code
secret link
QR code

Can someone guess the room?

Your identifiers need enough entropy.

A room system like:

ABC123

is convenient, but the security model depends on:

code length
expiration
rate limiting
session lifetime
whether links contain additional secrets

What does the signaling server know?

Ideally:

connection metadata
temporary room information

but not:

permanent copies of user files

How long do sessions live?

Temporary sessions should actually be temporary.

Security is not just encryption.

It is also minimizing what exists and for how long.

Why I did not want file storage in the architecture

There are obvious infrastructure reasons.

If ButterShare stored every file, I would eventually need to solve:

object storage
storage lifecycle
cleanup jobs
bandwidth costs
abuse
copyright complaints
malware storage
retention policies
privacy
expired transfers
storage quotas

Removing persistent file storage does not eliminate every problem.

But it dramatically changes the product.

The infrastructure becomes much more about:

connection coordination

than:

hosting arbitrary user data

That is a very different operational model.

P2P does not mean free infrastructure

This is worth emphasizing.

People sometimes hear peer-to-peer and assume:

No server costs.

Not true.

You still need infrastructure for things like:

signaling
authentication if applicable
room coordination
STUN
TURN
monitoring
abuse prevention
analytics
APIs

And TURN can become expensive if a meaningful percentage of transfers are relayed.

If a 100GB transfer has to go through your TURN infrastructure, you are moving a lot of bandwidth.

So the business model still needs to understand relay rates.

You should measure things like:

% direct peer connections
% relayed connections
average transfer size
average duration
connection failure rate
reconnect rate

Without this data, you do not really know how your P2P architecture behaves in the real world.

Connection speed is not magic

If two users have slow internet, P2P does not fix physics.

Suppose:

Sender upload: 20 Mbps
Receiver download: 500 Mbps

The transfer is still limited by approximately the sender's available upload bandwidth.

Likewise, Wi-Fi quality, mobile signal, congestion, TURN routing, and device performance can all affect throughput.

“No server upload step” does not mean:

instant 100GB transfer.

100GB is still 100GB.

What changes is where the data goes.

Instead of:

100GB to cloud
then
100GB from cloud

you are trying to make the destination the receiver from the beginning.

Why this architecture becomes attractive for large files

For small files, the benefits may not feel dramatic.

If you are sending a 3MB image, uploading it to a server is basically free from the user's perspective.

But consider:

5GB
20GB
50GB
100GB
500GB

As the file size increases, the architecture matters more.

Cloud storage adds:

an intermediate upload
storage requirements
infrastructure bandwidth
retention management

Direct transfer removes the need for permanent file hosting.

That is why I think P2P becomes increasingly interesting as file size grows.

The browser is becoming an operating system

One of the things I find most interesting about building ButterShare is how much browsers can now do.

A web application can access:

files
directories
WebRTC
streaming APIs
cryptography
persistent storage
service workers
filesystem capabilities on supported platforms

Years ago, a cross-platform file-transfer product almost certainly meant shipping:

Windows app
macOS app
Android app
iOS app
Linux app

Now a surprising amount can be done from:

https://...

That has huge distribution advantages.

The recipient does not necessarily need to:

download an installer
create an account
configure software
update an application

They can open a URL.

That is one reason I think browser-based file transfer is more interesting now than it used to be.

But browser compatibility keeps you humble

Just because an API exists does not mean it behaves identically everywhere.

Anyone building serious browser software learns this quickly.

You have:

Chrome
Edge
Firefox
Safari
iOS Safari
Android Chrome
in-app browsers
older devices

And each one comes with different behavior around:

WebRTC
file APIs
memory
downloads
filesystem writing
background tabs
device sleep
permission prompts

The real product is not:

“It works on my machine.”

The real product is:

“Can an iPhone user send a huge file to a Windows user without understanding WebRTC?”

That is a much harder standard.

UX matters as much as networking

Users do not care about:

ICE candidates
SCTP
DTLS
NAT traversal
bufferedAmount
TURN allocation

They care about:

“Is my file sending?”

“How long will it take?”

“Did it fail?”

“Can I try again?”

“Can I close this tab?”

This means a large part of the engineering work ends up in UX.

For example:

Bad

Peer connection state: disconnected

Better

Connection interrupted. Trying to reconnect...

Bad

bufferedAmount = 4194304

Better

42.7GB of 100GB transferred
Estimated time remaining: 18 minutes

The networking layer can be brilliant, but if the UI makes the user nervous, the product still feels broken.

What I would build differently if I started again

The biggest lesson is that I would treat the transfer engine as its own system from day one.

Not as:

some WebRTC code inside a React component

but as a proper state machine.

Something like:

IDLE
  |
  v
CREATING_SESSION
  |
  v
WAITING_FOR_PEER
  |
  v
CONNECTING
  |
  v
READY
  |
  v
TRANSFERRING
  |
  +----> PAUSED
  |
  +----> RECONNECTING
  |
  +----> FAILED
  |
  v
COMPLETED

That structure makes everything easier:

reconnection
retries
UI
analytics
debugging
resume support

Large file transfer has too many edge cases to manage with a couple of booleans like:

isConnected
isSending

Eventually those booleans turn into chaos.

I would also separate protocol from transport

Another design decision I strongly recommend:

Do not make your application protocol depend too tightly on WebRTC.

Think of WebRTC as the transport.

Your transfer protocol should define concepts such as:

session start
file metadata
chunk
acknowledgement
resume
file complete
transfer complete
error

Then the transport layer handles:

send(message)
receive(message)

That makes the architecture easier to test and potentially easier to extend later.

For example, you could theoretically support another transport in the future without rewriting the entire transfer model.

Measuring the right things

A transfer product needs good telemetry.

Not the files.

Not private user data.

But operational metrics.

I want to know things like:

connection success rate
time to connect
direct vs relayed connections
average throughput
transfer completion rate
disconnect rate
browser distribution
OS distribution
average file size
failure stage

For example, if Safari has:

62% completion

while Chrome has:

94% completion

that tells you exactly where to focus engineering effort.

If transfers larger than:

40GB

fail disproportionately often, that tells you something else.

Without metrics, you are debugging anecdotes.

The most important lesson: 100GB is not just a bigger 1GB

This is probably the main thing I learned.

You cannot build file transfer for 500MB and assume it will naturally scale to 100GB.

At very large sizes, every small weakness becomes visible.

A tiny memory leak matters.

A slightly inaccurate progress bar matters.

A reconnect bug matters.

A 0.5% failure rate matters.

A browser sleeping after 30 minutes matters.

The transfer may be running for hours.

The software has to survive that.

Large-file engineering is less about:

“Can bytes move?”

and more about:

“Can bytes keep moving reliably for a very long time under imperfect real-world conditions?”

That is the actual problem.

Where ButterShare fits

ButterShare is my attempt to make this architecture useful to normal people.

The user should not need to know what WebRTC is.

They should be able to:

Select a file or folder.
Get a link or QR code.
Send it to someone.
Transfer the file directly.

The product is built around:

browser-to-browser transfer
peer-to-peer networking
no traditional cloud file storage
cross-platform usage
large files
no mandatory account creation

The technical complexity should stay underneath.

The user should just see:

Send file.

When P2P is the wrong choice

I do not think peer-to-peer is automatically superior.

There are situations where cloud storage is clearly better.

For example:

Sender needs to go offline

Use cloud storage.

Recipient will download tomorrow

Use cloud storage.

50 people need the same file

Cloud/CDN distribution probably makes more sense.

You need permanent storage

Obviously use storage.

You need versioning and collaboration

Again, cloud storage.

P2P becomes compelling when:

one sender
+
one receiver
+
both online
+
large file
+
no need for permanent hosting

That is the sweet spot.

Final architecture

At a high level, the system looks something like this:

             +----------------------+
             |   Signaling Server   |
             +----------------------+
                  ^             ^
                  |             |
                  | metadata    |
                  | only        |
                  |             |
                  v             v

          +-------------+   +-------------+
          |   Sender    |   |  Receiver   |
          |   Browser   |   |   Browser   |
          +-------------+   +-------------+
                 \             /
                  \           /
                   \         /
                    \       /
                WebRTC Data Channel
                       |
                       |
                Direct if possible
                       |
                       v

               File data moves
             device-to-device

When direct connectivity is impossible:

Sender
  |
  v
TURN relay
  |
  v
Receiver

The important architectural goal remains:

The file does not need to become a permanently stored cloud object just to move between two people.

Closing thoughts

Before building ButterShare, “file sharing” sounded like a solved problem.

Then I started thinking about 100GB files.

The moment files get that large, you discover that the problem is not just uploading.

It is:

transport
memory
backpressure
browser limitations
NAT traversal
mobile lifecycle
reconnection
resume
integrity
filesystem writing
progress reporting
infrastructure economics

And that is what makes the problem interesting.

There is something satisfying about the simplicity of the final idea:

You have the file.

They need the file.

Connect the two devices.

Everything else is engineering.

I'm building this approach into ButterShare, a browser-based large-file transfer tool focused on direct device-to-device transfers without traditional cloud file storage.

If you work with WebRTC, browser streaming, file APIs, or large-file infrastructure, I'd genuinely be interested in hearing how you would approach the same problem.

ButterShare:

https://buttershare.com/

TL;DR

For very large browser-based transfers:

Don't load the entire file into memory.
Break files into manageable chunks.
Respect WebRTC backpressure.
Keep signaling separate from actual file transfer.
Use STUN and TURN appropriately.
Expect mobile browsers to behave differently.
Design reconnection and resume early.
Do not accumulate the complete file in receiver memory.
Track confirmed progress, not just queued bytes.
Treat file transfer as a protocol, not a loop calling send().
Remember that P2P does not mean servers disappear.
Most importantly: 100GB is not simply a larger version of 1GB.

That last point changes almost every engineering decision.