Traditionally, we used methods like USBs (external drives) or email when transferring large-size files. On the one hand, you would have a hard time sending large files to remote areas with USBs. People would mail CDs, DVDs, or USBs to faraway places— not a very efficient and fast way to transfer large files. The email was the only famous, easy, and safe solution for remote file transfers. But still, email service providers would quickly put limitations on the size of attachments.
File Transfer Protocol (FTP) was designed primarily to do that, “transfer, send, and share large files.” But still, FTP had a lot of limitations (still does), especially when it came to security. Plus, it didn’t allow you to pause/resume, so large files were difficult to transfer, especially in unstable Internet connections.
What are the best modern ways to transfer and share large files?
- FTPS, and SFTP.
- File Syncing.
- WebDAV.
- HTTP transfer with cloud-based storage.
- Peer-to-Peer (P2P)
- Network File Shares
- Ship the External Drive.
1. FTPS and SFTP.
FTPS and SFTP are file transfer mechanisms. FTP is the original, fastest, and easy to deploy. It doesn’t have much overhead (no encryption), so it is often faster but less secure. Do not use FTP unless the server and client are in the same trusted internal network. FTPS (FTP-SSL or FTP Secure) is based on the standard FTP but wraps the connection with a TLS/SSL certificate. FTPS is slower but a lot more secure than FTP.
SFTP (SSH FTP) is based on the Secure Shell (SSH) mechanism. In fact, SFTP is the file transfer method based on SSH and not related to FTP. SFTP is very secure to use in external and internal networks. SFTP is not the same as FTP over SSH.
- Examples of SFTP (client/server) are OpenSSH and Bitvise.
- Examples of FTPS (client/server) are Filezilla and WinSCP.
2. File Syncing.
File Synchronization (File Sync) is a file transferring protocol that ensures that two or more end-points are automatically updated with the same information. File Sync works based on rules. If you add, modify, or remove a file from one place, the protocol will add, modify, or remove the file in the other place. This method is famous for performing backups and file versioning.
The Rsync protocol is a popular FileSync utility. It allows you to transfer a copy of your files from one location to another location; regardless it is on the cloud or on-premises. Rsync compares the modification times and file sizes.
Examples of File Sync Solutions are Resilio, Dropbox, sync.com.
3. WebDAV.
WebDAV (Web Distributed Authoring and Versioning) is an extension of the HTTP protocol. Although HTTP is a protocol designed to display websites and not specifically for transferring files, many web developers and designers use HTTP-based WebDAV for managing and editing their content into their web servers.
Cloud-based services such as Sharepoint use WebDAV to allow collaboration between users. With this solution, Sharepoint users can create, change, download, upload, and remove big files on the webserver. Some popular file transfer clients that offer WebDAV support are WinSCP and Cyberduck.
*Modern WebDAV solutions come with features like: *
- DAV locking
- Properties
- Namespace control
- Encryption (SSL/TLS)
- Authentication
- Caching and proxying.
4. HTTP transfer with cloud-based storage.
Before the cloud, it was pretty challenging and expensive to keep up an online file transfer server with ample storage units. Keeping big files stored for clients was not an easy solution. When cloud storage services came, they helped solve this big file hosting problem.
Today’s cloud-based storage services like Dropbox, GoogleDrive, or Files.com allow users to use their cloud storage to transfer and share big files. The easiest way is to set a shearable link, grant the necessary permissions, and send this link however you like in any way. In fact, as you may have realized, Gmail does this automatically when your attachment reaches the 25GB size limit.
To transfer big files from cloud storage, you usually use HTTP (but you can also use other methods).
5. Peer-to-Peer (P2P)
Peer-to-Peer (P2P) file-sharing mechanisms have been one of the best ways to transfer large files. Although P2P has been used before in different application domains, it wasn’t so popular until the file-sharing system Napster came into the scene. From there, P2P came to revolutionize the way big files are shared.
P2P opened the door to BitTorrent, one of the most popular file-sharing mechanisms that, at some point, used almost 80% of the entire Internet bandwidth. Today, BitTorrent is used in popular systems like Linux Ubuntu to allow their users to download large ISOs.
Robust seeding and torrenting solutions for sharing large files?
- A Seedbox - VPS servers designed to transfer, share, and seed torrent files.
- An optimized BitTorrent client. ISPs often throttle BitTorrent traffic, so optimizing your client will ensure that you get fast and private P2P transfers.
6. Network File Shares
Let’s suppose you need to transfer a large file within an internal network (a University campus or an office building). One of the standard solutions is to take a USB drive or external hard disk and walk, but many internal networks have security policies that restrict the I/O of USB drives (or some DevOps don’t like to walk). So what can you do? A Network File Share— either NFS or SMB, file-sharing solutions for internal networks.
File-sharing within internal networks is the safest, most convenient, and fastest to transfer large files. But today, not many people work within the same perimeter; they are outside working from home and moving around. And obviously, using an NFS over the Internet without some protection is not the best strategy. As a solution, corporate networks would create a VPN to allow users to access resources from a remote network and transfer using NFS, SMB, or NAS.
7. Ship the External Drive.
When you want to move massive amounts of data from one data center or server to another distant location, one of the only solutions is to use a carrier or shipping service to transport the external drive. Sometimes using the highway is, by far, a faster solution than using ISP’s infrastructure.
Obviously, it is not a good idea to use normal mail. Most likely, your data is valuable, so using a reliable and safe service is always the best idea. You don’t want your hard drive to get damaged or lost during logistics and transport. Some reliable services like the AWS Import/Export Disk are designed for massive transfers. Bear in mind that this option moves data into and out of AWS.
Final Words.
Transferring large amounts of data is not always an easy task. Ensuring speed and security makes big file sharing a difficult task to accomplish. Using any of the above seven methods will help keep a good balance between safety, speed, and convenience. For instance, Dropbox and Google Drive are easy to use, safe, and highly convenient.
HTTPS-based, FTPS, and SFTP are highly secure and fast. Plus, they can be cost-efficient if you have the skills to deploy and maintain the servers. FileSync is another very efficient way for large backups and transfers with low-overhead. P2P (BitTorrent) is an often-overlooked technology but is probably the best for reliable and fast transfers. Network File Shares via VPN is also challenging to configure and deploy, but it is also very safe and convenient. And finally, if you have Teras or Petabytes of data, better call your mail service!
So, what is your favorite method for transferring large files? If I didn’t mention it here, please go ahead and add it to your comments.
Top comments (0)