credits: https://xkcd.com/
Yes, another one
🤣 So here we are, there's like 50 different solutions for file storage/backup/transfer, but there's not one just "simple" solution.
One where you can just drag and drop on one side, download on the other and you're done.
Nothing to install, no account to create or email to send; Just a simple damn file transfer!
I'm not going to repeat all the edge cases of each existing solutions, I think we all already know them way too well and xkcd already summarized it.
Between sending yourself slack DMs and trying to send pics split into 10MB chunks because the mail server rejects attachments >10MB, transferring files in 2023 is still a strangely convoluted process.
The Problem
To summarize existing solutions:
It's either storage based, so you upload on someone's hard drive first, and then send a link to the receiving end which downloads from the said hard drive.
Or p2p, those usually rely on webRTC, the API now available on most browsers to establish peer-to-peer (direct) connections between two devices.
Storage solutions
Storage based services are a good pick when you want to send a single file to someone else. They're quite convenient but if you're just transferring files between two computers that are next to each other it's quite overkill.
Not only is the process way too heavy for the purpose (upload on server, enter email, open email, download etc..), but it's also a giant waste! Power consumption due to stale file storage makes no sense, and is an easy hanging fruit when it comes to lowering our favorite tool's carbon footprint.
Currently, many experts estimate that data storage and transmission in and from data centers use 1% of global electricity.
The other problem with storage is that, well.. it's stored, which is an obvious potential privacy issue. And I'm not talking about nosy admins or marketers, their servers can just get hacked 🏴☠️.
Btw, the semi official story telling for NoUSB is that we wanted to send 🍑/🍆 pics to each other with my wife, and couldn't find a way without copying them on Mark Zuckerberg's hard drives😅
P2P (webRTC) solutions
P2P solutions are cool too, but the first problem is that while the webRTC APIs are widely available now, their reliability is still variable (yes I'm looking at you Firefox), and it can easily be blocked by network hardwares.
But what makes them unusable for file transfer for now is the lack of File system APIs on most browsers, and namely the showSaveFilePicker function that allows to open a file for writing.
Sad developer noise
Without proper File system APIs all you're left with is a good ol' RAM buffer that you then turn to a blob URL to write the file to the file system.
In other words, the size of files you can download depend how much RAM the browser can consume.
The last big problem with P2P file transfers is IP leaking, so if you're using it to transfer to someone online, just keep in mind he will certainly end up with your IP address ☠.
What I wanted
So based on that, here's the solution I wanted, it had to be web-based to avoid the mobile app trap (becoming google/apple's bitch, appstore friction, "multi-platform" tech hell etc..).
And it had to ONLY rely on widely and reliably implemented browser APIs, so no cutting edge APIs that are barely implemented in just one variant of browsers.
Basically what's left is just the plain HTTP upload/download endpoints.
I would have a websocket connection to synchronize the peers to download/upload on "standard" http endpoints.
And the server would just have to "link" the two endpoints and transmit the data directly, without storage.
The UI should be as simple as possible, just drag&drop on one side, and download on the other. I also want multiple devices to be able to join the same files, and they should all be able to add their own files.
The UI should also provide some basic chat feature, because if it's opened from a chat app the user will be trapped in a webview, with no way to return to the chat app without disconnecting.
The system should also implement a lot of transfer control, to avoid abuse and control potential transfer costs.
How it went
As you might expect the http server part was quite straightforward, Golang seemed a perfect pick for a custom http server, just put a go channel between two upload/download endpoints and you have your base relay server.
Most of the backend work was about rate limiting to avoid abuses and control costs, but also make sure the system is at least a little bit scalable by load balancing the transfers among multiple relay server.
It should also handle different regions for fast transfers and energy preservation.
The biiiiig problem has been to deal with all browsers, and not only browsers, also all the sh💩tty webviews that are embedded on all messaging apps.
They're like stripped down browsers, for example the concept of "tabs" still exists on some, but behaves slightly differently on nearly all of them, amazing 👍.
The other one was to deal with Firefox's terrible network layer.
Fun fact: Firefox can't upload two files at the same time with the same speed in HTTP/2. Firefox also has other weird quirks, but they seem more related to process leaks as they only arise over time.
Where I'm at now
Well now I'm done with my favorite part which is developing the product and deploying it, so now it's time to show it to the world😱
It's already been extensively tested, btw I want to thank all those poor "social network friends" I spammed for live testing^^
And the reactions were very good😁 Basically the main question I've had was "But why tf did this not exist already?". No idea.
The main challenge now is to make it a new usage in people's mind, so I think finding good integrations could help with that.
The first integration is as a Discord bot, which allows to bypass discord's upload limits. I'm also planning a slack bot.
To test it 🙏
The fastest way to test it is between a laptop and a smartphone, which is a classic issue. Now it's a breeze:
- Open https://NoUSB.co on a laptop
- Click "Share link" and scan the QR code with your smartphone
- Add a large file from the laptop, it appears instantly regardless of its size
- Start the download
- Click "boost this!"
- Close the tab, everything disappears
One last word about webRTC
Actually it is implemented in NoUSB.co already, as an experimental feature, you need to add the &rtc=1 parameter to the URLs on both sides to enable it. Only chrome based desktop browsers can do the downloading end for now.
To conclude
There's no conclusion yet, please have a quick test and let me know how it went on your devices!
Thanks for reading!
Stant
Top comments (0)