DEV Community

Brandon Rozek
Brandon Rozek

Posted on • Originally published at brandonrozek.com on

Parallel SCP with LFTP

Segmented file transfer allows you to split up a file into multiple chunks and download them in parallel. There is a program written for Linux called LFTP which can accomplish this task and supports FTP, HTTP, SFTP, BitTorrent, among others. The syntax is a little funky, so I wrote a wrapper I call pget which allows for segmented file transfers using SCP.

Usage:

pget [user@host] [file] [segments?]

Enter fullscreen mode Exit fullscreen mode

where segments? is optional.

Example:

pget user@127.0.0.1 Documents/todo.txt 3

Enter fullscreen mode Exit fullscreen mode

The full script:

#!/bin/sh
# set -xv

show_usage() {
    echo "Usage: pget [user@host] [file] [segments?]"
    echo "Example: pget user@127.0.0.1 Documents/todo.txt 3"
    echo "Currently only supports sftp. Segments value are optional."
    exit 1
}

contains_help_flag() {
    if ["$1" = "-h"] || ["$1" = "--help"]; then
        return 0
    fi
    return 1
}

if ["$#" -lt 2] ||
   ["$#" -gt 3] ||
   contains_help_flag "$1" ||
   contains_help_flag "$2" ||
   contains_help_flag "$3" ; then
    show_usage
fi

REMOTE_HOST="$1"
FILE_LOCATION="$2"

if ["$#" -eq 3]; then
    NUM_SEGMENTS="$3"
else
    NUM_SEGMENTS="1"
fi

LFTP_COMMAND="pget -n $NUM_SEGMENTS $FILE_LOCATION;bye"
lftp -e "$LFTP_COMMAND" sftp://"$REMOTE_HOST"

Enter fullscreen mode Exit fullscreen mode

Sentry image

See why 4M developers consider Sentry, “not bad.”

Fixing code doesn’t have to be the worst part of your day. Learn how Sentry can help.

Learn more

Top comments (0)

A Workflow Copilot. Tailored to You.

Pieces.app image

Our desktop app, with its intelligent copilot, streamlines coding by generating snippets, extracting code from screenshots, and accelerating problem-solving.

Read the docs

👋 Kindness is contagious

Please leave a ❤️ or a friendly comment on this post if you found it helpful!

Okay