Answer: Using wget to recursively fetch a directory with arbitrary files in it

answer re: Using wget to recursively fetch a directory with arbitrary files in it

Oct 18 '17

2

This version downloads recursively and doesn't create parent directories.

wgetod() {
    NSLASH="$(echo "$1" | perl -pe 's|.*://[^/]+(.*?)/?$|\1|' | grep -o / | wc -l)"
    NCUT=$((NSLASH > 0 ? NSLASH-1 : 0))
    wget -r -nH --user-agent=Mozilla/5.0 --cut-dirs=$NCUT --no-parent --reject="index.html*" "$1"
}

Usage:

1. Add to ~/.bashrc or paste into terminal
2. wgetod "http://example.com/x/"

Open Full Answer