There are a lot of ways to transfer files.1
Above the basic necessity of preserving the data during the transfer, sharing data should be fast, secure, and -- most of all -- easy. Most file transfer utilities I've used encompass two thirds of these qualities. I've wanted to make a utility that encompasses all three, without compromising any one of them.
relay > uploading
File transfers should be as fast as possible - the time it takes to transfer the file is often spent waiting. Faster file transfers means less waiting.
A common way to transfer a file is to first upload data to a server, and then, once uploaded, the link is shared with someone who goes to download it.2 This method is easy, but slow -- the transfer rate of the file is half the harmonic mean of the upload and download speeds which makes it slower than either just uploading or downloading.3
There are better ways than sequentially uploading and downloading. Instead, you can use a relay server to create a full-duplex real-time communication layer between the two computers so that "uploading" and "downloading" occur simultaneously between the two computers. This effectively increases the transfer rate because its not sequential and only limited by the slower of the two transfer modes (uploading or downloading).
key exchange > passwords
Transferring data should be secure. Whenever you share data, it goes through a network of servers that can "see" the data, so it should be encrypted to stay private. A common way to do end-to-end encryption is with a password. You use a password to encrypt your file, then send the password-protected file, and after which the other person uses the same password to decrypt it.
Passwords are easy, but passwords can be weak. It turns out many people use the same passwords so that breaking an encryption can be as easy as just trying each password in a list of pwned accounts.
Weak passwords can be used to make strong passwords using PAKE: password authenticated key exchange. PAKE is a cryptographic method where two people share a password which is then used -- via back-and-forth communication -- to generate a strong key. The strong key can then be used for all further encryption. Since the two people generate the strong key by exchanging information, no one else could possibly learn the strong key even if they have the original password.4
There are other ways to exchange a key, but in general you can get better security by exchanging a key rather than using a single password on both computers.
no port forwarding > port forwarding
Finally, transferring data should be simple, ideally with a single command and no maintenance or other programs needed.
Often it is not easy though.
Another common way to transfer a file is to use SSH or FTP. Both protocols are ubiquitous and relatively simple to use, but their simplicity relies on one of the computers to be running a server (like openssh) and it requires the server computer to have port forwarding enabled. It can be pretty much assumed that most pairs of computers don't meet these requirements (say, two computers in a café) so that makes these common utilities hard to use in most cases.
File transfers can be easier by eliminating the need for hosting a server or port forwarding. Again, using a relay server allows any two computers to connect to one another without resorting to port forwarding or fiddling with a server.
croc = fast + secure + easy
I've been working on an open-source tool that takes these ideas to heart: croc. Because its open-source and implemented in a language that easily cross compiles (Go) it can be implemented anywhere and thus will (hopefully) always be a fast, secure, and really easy way to send data.
AFAIK, croc is the only CLI file-transfer tool does all of the following:
- allows any two computers to transfer data (using a relay)
- provides end-to-end encryption (using PAKE)
- enables easy cross-platform transfers (Windows, Linux, Mac)
- allows multiple file transfers
- allows resuming transfers that are interrupted
- does not require a server or port forwarding
In developing croc I took a lot of inspiration from other CLI tools like toss and magic-wormhole which had some but not all of the qualities above.5
If you'd like to give it a try, the code is available online at https://github.com/schollz/croc.
-
I wrote about sending a file in 2017 and have tried to maintain a list of ways to send files since then. ↩
-
Some good server-based file-sharing tools: https://send-anywhere.com, https://send.firefox.com, https://ipfs.io/ ↩
-
For example, if you are transferring a file by uploading at 5 Mbit/s and then, after finishing the upload, downloading at 8 Mbit/s, then the effective transfer rate is 1/2 * 2/(1/5+1/8) = 3.1 Mbit/s - which is slower than either. ↩
-
PAKE additionally will prevent eavesdroppers. If anyone "listens in" on the information exchange, then all parties will end up with different strong keys and no one can decrypt anything between them, alerting the users that a eavesdropper is present. ↩
-
toss cleverly encodes port information in the code phrase, making it simple but it requires using connected computers (no firewalls) and the long random-ish code phrase is hard to "tell" someone. magic-wormhole has most everything (currently its missing capabilities for multiple file transfers and file resuming), but it requires installing lots of the Python ecosystem which is tricky for non-developers (and Windows users). ↩
Top comments (2)
Very cool tool! Thank you for creating it and sharing!
I'm eager to compare it against gFile
Thanks! Funny thing is that I've been wanting to do true peer-to-peer over UDP (no relay) for awhile. I actually saw gFile and used that to re-write croc. It has been an unreleased v5 of croc.
The v5 of croc is truly peer-to-peer using WebRTC however it does not appear to be very reliable. Most of the time it works, but sometimes it doesn't and I can't figure out why. Also I've found that the transfer speeds are not as fast as using parallel TCP sockets (which is v6 of croc, the current version).