DEV Community

Cover image for Beyond SCP: Mastering VM File Transfers with Rsync and HTTPS Accelerators
Parth Agarwal
Parth Agarwal

Posted on • Originally published at blog.ra101.dev

Beyond SCP: Mastering VM File Transfers with Rsync and HTTPS Accelerators

#linux #programming #devops #opensource

Transferring files between your local machine and a virtual machine (VM) is one of the most common tasks in tech. We often use scp because it comes natively installed and has similar utilization to ssh, but it is the clunkiest tool for the job.

In this article, we’ll walk through the major CLI tools available, explain how they work under the hood, and highlight when you should use each.

We’ll cover:

  1. SSH (Secure Shell) Tools
    1.1 SCP (Secure Copy)
    1.2 SFTP (SSH File Transfer Protocol)
    1.3 SSHFS (SSH Filesystem Mount)

  2. Rsync

  3. HTTP(S) + Download Accelerators
    3.1 HTTP Single-thread server (Python)
    3.2 HTTP Multi-thread server (BusyBox httpd)
    3.2.1 Download Accelerators (aria2c, axel)
    3.3 HTTPS File Server with Basic Auth (Caddy)

1. SSH (Secure Shell) Tools

These tools leverage the established Secure Shell (SSH) protocol for secure, encrypted transfers and operations.

1.1 SCP (Secure Copy)

SCP is the classic method for transferring files over SSH. It opens a secure channel, encrypts data, and performs a raw byte stream transfer.

Protocol Encrypted Resumable
SSH Yes No 
# Upload a file
scp local_file user@remote:/path/to/destination

# Download a file
scp user@remote:/path/to/file local_dir/

# Copy between two remote SSH hosts
scp user1@remote1:/file user2@remote2:/destination
Enter fullscreen mode Exit fullscreen mode

✅ Advantages:

  • Installed by default on nearly all systems.
  • Simple syntax for quick transfers.
  • Can copy files directly between two remote SSH servers.

❌ Drawbacks:

  • No resumability; interrupted transfers start from scratch.

💙 Best Use Case:

  • scp is best for one-off, quick, and secure transfers where you don’t expect interruptions.

1.2 SFTP (SSH File Transfer Protocol)

SFTP is more feature-rich than SCP. It supports interactive sessions, directory traversal, file operations, and resumable transfers (when the client supports the feature).

Protocol Encrypted Resumable
SSH (subsystem) Yes Yes 
# Interactive session
sftp user@remote
sftp> get -a remote_file local_file  # -a for resume/continue or use reget instead.
sftp> put local_file remote_file

# One-liner download
sftp user@remote:/path/to/file local_file
Enter fullscreen mode Exit fullscreen mode

✅ Advantages:

  • Resumable transfers with most clients (using -a or reget).
  • Interactive mode for browsing and transferring.
  • Compatible with GUI clients like FileZilla and WinSCP.

❌ Drawbacks:

  • Slightly slower than SCP due to protocol overhead.
  • Cannot transfer between two remote servers.

💙 Best Use Case:

  • Since sftp is a maintained SSH session, it is useful when there are a lot of small file transfers, and the folder structure is unknown.

1.3 SSHFS (SSH Filesystem Mount)

SSHFS mounts a remote filesystem locally using SSH and FUSE (Filesystem in Userspace). This allows you to use your existing tools and editors as if the files were on your machine.

Protocol Encrypted Resumable
SSH (with FUSE) Yes N/A 
# Mount remote filesystem
sshfs user@remote:/remote/path /local/mountpoint

# Unmount
fusermount -u /local/mountpoint
Enter fullscreen mode Exit fullscreen mode

✅ Advantages:

  • Transparent: Edit remote files with your local IDE/editor seamlessly.
  • Secure by default via SSH.
  • Great for exploratory work or ad-hoc editing.

❌ Drawbacks:

  • High latency on every file operation (slow for reading many small files).
  • Poor throughput for large files or bulk transfers.
  • Requires sshfs and FUSE support.

💙 Best Use Case:

  • sshfs is ideal for development workflows where convenience and direct file access matter more than transfer speed.

2. Rsync

Rsync is optimized for synchronization and efficiency. Instead of re-copying everything, it calculates file checksums and only sends the differences (deltas). This delta-transfer algorithm, along with built-in compression and partial transfers, makes it a synchronization powerhouse.

Protocol Encrypted Resumable
SSH (default) or native TCP daemon Yes (if over SSH) Yes 
# Sync local directory to remote (Archive, Verbose, Compress and Partial Transfer)
rsync -avzP local_dir/ user@remote:/path/to/dir
Enter fullscreen mode Exit fullscreen mode

✅ Advantages:

  • Extremely efficient for repeated transfers (only sends deltas).
  • Built-in compression (-z) and resumability.
  • Preserves permissions, symlinks, and timestamps.

❌ Drawbacks:

  • Requires rsync on both machines.
  • More CPU-intensive due to checksum calculations.

💙 Best Use Case:

  • rsync is the go-to tool when you’re syncing directories with frequent updates or working with limited bandwidth where sending only changes matters most.

Performance Note: For initial full transfers, rsync overhead (checksum calculations) makes it slightly slower than raw HTTP transfers. However, for incremental syncs with only changed files, rsync is dramatically faster since it only transfers deltas.

3. HTTP(S) + Download Accelerators

These methods prioritize raw speed by leveraging the lightweight nature of HTTP and the ability of clients to open parallel connections and partial downloads.

3.1 HTTP Single-thread server (Python)

Python's built-in HTTP server is the quickest way to share files. It serves the current directory over HTTP with zero configuration.

Protocol Encrypted Resumable
HTTP No No 
# Server
python3 -m http.server 8000

# Client
wget http://<server-public-ip>:8000/filename.ext
Enter fullscreen mode Exit fullscreen mode

✅ Advantages:

  • Works out of the box.
  • Faster than any of the SSH Tools, No SSH encryption/protocol overhead.

❌ Drawbacks:

  • Single-threaded -> only one client at a time.
  • No authentication.
  • An SSH connection or a background job is required.
  • One-way setup (server -> client).

💙 Best Use Case:

  • Transferring non-sensitive medium-sized files on a trusted local network for quick testing.

3.2 HTTP Multi-thread server (BusyBox httpd)

BusyBox's httpd is a lightweight, multi-threaded HTTP server that handles multiple concurrent connections, making it ideal for serving files to several clients simultaneously.

Protocol Encrypted Resumable
HTTP No Yes (client-dependent) 
# Server
busybox httpd -f -p 8000

# Client
wget http://<server-public-ip>:8000/filename.ext

# download-accelerators http://<server-public-ip>:8000/filename.ext
Enter fullscreen mode Exit fullscreen mode

✅ Advantages:

  • Works out of the box.
  • No SSH encryption overhead.
  • Handles concurrent connections and multiple clients.

❌ Drawbacks:

  • No authentication.
  • An SSH connection or a background job is required.
  • One-way setup (server -> client).

💙 Best Use Case:

  • Transferring non-sensitive large-sized files on a trusted local network for quick testing.

3.2.1 Download Accelerators (aria2c, axel)

Download accelerators are client-side tools that work with any HTTP/HTTPS server. They split downloads into multiple parallel streams, saturating your available bandwidth for maximum speed. These tools can be used with busybox httpd (3.3), or Caddy (3.4).

Protocol Encrypted Resumable
HTTP/HTTPS Optional (HTTPS) Yes
  • aria2c: multi-source, segmented downloads, resumable (-c).
  • axel: lightweight, parallel streams, resumable, simpler than aria2c. 
# busybox httpd (HTTP) (3.2)
aria2c -c -x 8 -j 8 http://<server-public-ip>:8000/filename.ext

axel -n 8 http://<server-public-ip>:8000/filename.ext

# Caddy (HTTPS) (3.3)
aria2c -c -x 8 -j 8 --http-user=<username> --http-passwd=<password> --check-certificate=false https://<server-public-ip>/filename.ext

axel -n 8 -k -H "Authorization: Basic $(printf '<username>:<password>' | base64)" https://<server-public-ip>/filename.ext
Enter fullscreen mode Exit fullscreen mode

✅ Advantages:

  • Parallel streams saturate bandwidth.
  • Built-in resume support.

❌ Drawbacks:

  • Requires installation on client.
  • Background server process needed.

3.4 HTTPS File Server with Basic Auth (Caddy)

A multi-threaded HTTP server is fast, but it is not secure. The final solution to file transferring is to go into overdrive mode and use a production server. Here, we are using Caddy for simplicity; we could have used NGINX and CertBot to accomplish the same.

Protocol Encrypted Resumable
HTTPS Yes Yes 
# Server
caddy hash-password --plaintext <password>
# $2a$14$moXJYUFKt1w6d...
cat Caddyfile
Enter fullscreen mode Exit fullscreen mode 
https://localhost, https://127.0.0.1,  https://<server-public-ip> {
     tls internal

     root * /path/to/local_dir
     file_server browse

     basicauth {
        <username>  $2a$14$moXJYUFKt1w6d...
     }
}
Enter fullscreen mode Exit fullscreen mode 
sudo caddy run --config /path/to/Caddyfile
Enter fullscreen mode Exit fullscreen mode

Security Note: The example below uses --check-certificate=false because tls internal in Caddy creates a self-signed certificate. In production, use proper certificates with Let's Encrypt (for public IPs with domain names) or an internal Certificate Authority.

# Client

# aria2c
aria2c -c -x 8 --http-user=<username> --http-passwd=<password> --check-certificate=false https://<server-public-ip>/filename.ext

# axel
axel -n 8 -k -H "Authorization: Basic $(printf '<username>:<password>' | base64)" https://<server-public-ip>/filename.ext
Enter fullscreen mode Exit fullscreen mode

✅ Advantages:

  • Authentication and encryption built-in.
  • Supports parallel downloads.

❌ Drawbacks:

  • More complex setup.
  • One-way setup (server -> client).

💙 Best Use Case:

  • Sharing link for large files over the internet (with auth).

Security Considerations

Understanding the security implications of each tool is critical when choosing a file transfer method:

SSH-Based Tools (SCP, SFTP, SSHFS, Rsync over SSH)

  • Encrypted by default using SSH protocol
  • Uses public key authentication (SSH keys) or password authentication
  • Suitable for transferring sensitive data over untrusted networks
  • Requires SSH access and proper key management

HTTP Servers (Python, busybox)

  • No encryption - data transferred in plain text
  • No authentication - anyone with network access can download files
  • Use only on trusted local networks (VPNs, private subnets)
  • Never use for sensitive data without additional security measures

HTTPS (Caddy)

  • Encrypted via TLS
  • Basic Authentication sends credentials with every request
  • Self-signed certificates (using tls internal) trigger security warnings and require --check-certificate=false
  • For production use, obtain proper certificates via Let's Encrypt or internal CA
  • Suitable for transferring files over public networks when configured correctly

Best Practice:

  • Use SSH-based tools for sensitive data over untrusted networks.
  • HTTP servers are perfectly acceptable on trusted networks (VPNs, private subnets) where the network layer already provides security.
  • Use HTTPS only when transferring files over public internet without VPN protection.

Performance Comparison: Slowest to Fastest

  1. SSHFS: High latency per operation, every read/write is a round-trip over SSH.
  2. SCP/SFTP: SSH encryption overhead, single-threaded.
  3. Rsync over ssh: Efficient delta transfers and compression.
  4. HTTP Single-Threaded: Direct HTTP with no encryption overhead.
  5. HTTPS Multi-Threaded + Download Accelerators: Parallel streams that saturate available bandwidth. HTTPS adds ~10-20% overhead.

My Daily Workflow

Here is what I use on a day-to-day basis:

  • scp: When transferring small files between VMs (and when I forgot rsync flags.)
  • sftp: When I have many small files to operate on, and I also don't know the directory structure, the command has fewer flags and is easier on fingers.
  • sshfs: Never used.
  • rsync over ssh: I have majorly replaced scp with rsync -aivP (I don't use compression on the fly). It is especially handy with large files.
  • python3 -m http.server 8000: Not anymore for my setup, but I am debugging someone else's setup, then they probably don't have download accelerators.
  • busybox httpd -f -p 8000: Since most file transfer work happens over VPN with private endpoints. So for large files, this is my go-to choice.
  • HTTPS File Server (Caddy): Sending public links (outside VPN but still with user and password) for file download.

Each tool addresses the same problem differently. By understanding their trade-offs, you can pick the right tool for your specific requirements instead of defaulting to SCP.

Top comments (0)