Day 03
10 examples in 11 days
Exclude and Include
Now it's time to explain the exclude and include option. This options are great if you want to get control of what needs to be sync.
With the --exclude option, you can exclude a file, a directory and even a list of files/directories listed inside a file. Let's check out the the rsync syntax with the --exclude option.
I will create a file called exclude-me.txt inside ~/Users folder
iamgroot@laptop:~$ touch ~/Users/exclude-me.txt && echo "Hello World" >> ~/Users/exclude-me.txt
Now Let's check the size and files inside ~/Users and ~/Sync/Users folders
iamgroot@laptop:~$ stat -c "%s %n" ~/Users/* && stat -c "%s %n" ~/Sync/Users/*
12 /home/iamgroot/Users/exclude-me.txt
151 /home/iamgroot/Users/user01.json
57 /home/iamgroot/Users/user02.json
69 /home/iamgroot/Users/user03.json
43 /home/iamgroot/Users/user.config
151 /home/iamgroot/Sync/Users/user01.json
57 /home/iamgroot/Sync/Users/user02.json
69 /home/iamgroot/Sync/Users/user03.json
88 /home/iamgroot/Sync/Users/user.config
4096 /home/iamgroot/Sync/Users/Users
To exclude a file, we need to wrap the file name in quotes after the --exclude option. This way, rsync leaves the file in source and doesn't include it inside destination.
iamgroot@laptop:~$ rsync -vhr --exclude 'exclude-me.txt' ~/Users ~/Sync/
sending incremental file list
sent 146 bytes received 17 bytes 326.00 bytes/sec
total size is 320 speedup is 1.96
Let's check if the file exclude-me.txt was copied from source to destination
iamgroot@laptop:~$ stat -c "%s %n" ~/Users/* && stat -c "%s %n" ~/Sync/Users/*
12 /home/iamgroot/Users/exclude-me.txt
151 /home/iamgroot/Users/user01.json
57 /home/iamgroot/Users/user02.json
69 /home/iamgroot/Users/user03.json
43 /home/iamgroot/Users/user.config
151 /home/iamgroot/Sync/Users/user01.json
57 /home/iamgroot/Sync/Users/user02.json
69 /home/iamgroot/Sync/Users/user03.json
88 /home/iamgroot/Sync/Users/user.config
4096 /home/iamgroot/Sync/Users/Users
It wasn't included. Like I said, this option comes very handy. It also works with directories. Let's create a folder and include multiple files in it
iamgroot@laptop:~$ mkdir ~/Users/exclude-dir && touch ~/Users/exclude-dir/e{1..5}.txt && ls -a ~/Users/exclude-dir
. .. e1.txt e2.txt e3.txt e4.txt e5.txt
Now we have a folder called exclude-dir with 5 files calles e1.txt, e2.txt, etc
Let's sync them excluding this folder
iamgroot@laptop:~$ rsync -vhr --exclude 'exclude-dir' ~/Users ~/Sync
sending incremental file list
Users/exclude-me.txt
Users/user.config
Users/user01.json
Users/user02.json
Users/user03.json
sent 731 bytes received 112 bytes 1.69K bytes/sec
total size is 332 speedup is 0.39
jorge@code-lab:~$ stat -c "%s %n" ~/Users/* && stat -c "%s %n" ~/Sync/Users/*
4096 /home/jorge/Users/exclude-dir
12 /home/jorge/Users/exclude-me.txt
151 /home/jorge/Users/user01.json
57 /home/jorge/Users/user02.json
69 /home/jorge/Users/user03.json
43 /home/jorge/Users/user.config
12 /home/jorge/Sync/Users/exclude-me.txt
151 /home/jorge/Sync/Users/user01.json
57 /home/jorge/Sync/Users/user02.json
69 /home/jorge/Sync/Users/user03.json
43 /home/jorge/Sync/Users/user.config
We excluded the exclude-dir folder but it copied the exclude-me.txt file that we previously did not wanted to include at destination. How can we tell rsync to exclude this file as well?, easy, just tell rsync you want to eclude it as well
I will remove the file from destination and repeat the rsync command and check what was sync into destination
iamgroot@laptop:~$ rm ~/Sync/Users/exclude.me-txt
iamgroot@laptop:~$ rsync -vhr --exclude 'exclude-dir' --exclude 'exclude-me.txt' ~/Users ~/Sync
sending incremental file list
sent 150 bytes received 17 bytes 334.00 bytes/sec
total size is 320 speedup is 1.92
iamgroot@laptop:~$ stat -c "%n" ~/Sync/Users/*
/home/iamgroot/Sync/Users/user01.json
/home/iamgroot/Sync/Users/user02.json
/home/iamgroot/Sync/Users/user03.json
/home/iamgroot/Sync/Users/user.config
Great, we achieve what we wanted.
If you really think about it, using mutiple --exclude options is not very clean and specially if you are working with multiple files/folders. There is a simple way to use --exclude in the same rsync line and its using curly braces
iamgroot@laptop:~$ rsync -vhr --exclude={'exclude-dir','exclude-me.txt'} ~/Users ~/Sync
sending incremental file list
sent 150 bytes received 17 bytes 334.00 bytes/sec
total size is 320 speedup is 1.92
Note: You have to watch out and not to include whitespaces between the files/folders isnide the curly braces, if you include a whitespace it will sync all files/folders
GOOD rsync -vhr --exclude={'exclude-dir','exclude-me.txt'} ~/Users ~/Sync
NOT GOOD rsync -vhr --exclude={'exclude-dir', 'exclude-me.txt'} ~/Users ~/Sync
Again, if we have multiple, by multiple a mean hundreds or even more, using this technique is not so handy.
The good news is that rsync have a way to do this easily hehe (well done rsync!!!).
Today, we are very used to use git and just like git has .gitignore with rsync is pretty much the same thing. We can create a file and list all the files we want to exclude. To achieve this, we have the --exclude-from option.
I will create a file called rsync-ignore.txt and inside of it I will list all files that we want to exclude, notice that I will include the rsync-ignore.txt file as well
iamgroot@laptop:~$ touch ~/Users/rsync-ignore.txt && echo -e "exclude-dir\nexclude-me.txt\nrsync-ignore.txt" >> ~/Users/rsync-ignore.txt && cat ~/Users/rsync-ignore.txt
exclude-dir
exclude-me.txt
rsync-ignore.txt
iamgroot@laptop:~$ rsync -vha --exclude-from='/home/iamgroot/Users/rsync-ignore.txt' ~/Users ~/Sync
sending incremental file list
Users/
sent 165 bytes received 20 bytes 370.00 bytes/sec
total size is 320 speedup is 1.73
notice I included the full path of the --exclude-from, using (~/) will not work
iamgroot@laptop:~$ ls -h ~/Sync/Users
user01.json user02.json user03.json user.config
As you can see, everything worked just as expected. Now I will point out some other options to be used with the --exclude option.
What about the Include option?
The include option behaves pretty much the same as --exclude, is just the opposite. It's useful when you mix them along the same line. Let's see an example
iamgroot@laptop:~$ rsync -avhm --include '*.json' --exclude '*.txt' ~/Users ~/Sync
building file list ... done
Users/
Users/user.config
Users/user01.json
Users/user02.json
Users/user03.json
sent 695 bytes received 95 bytes 1.58K bytes/sec
total size is 320 speedup is 0.41
What this did was:
- included all .json files
- excluded all the .txt files
- using the -m option it didn't copy the exclude-dir folder since it was empty because inside there were only .txt files and got empty
Below are some more options to be used.
Exclude Files using the wildcard (*)
This option let's you specify a pattern you want to exclude. To achieve this, you need to use the wilcard (*), it can also be used to exclude specific filetypes (it's all about how you use the wildcard. Let's see an example:
iamgroot@laptop:~$ rsync -avh --exclude '*.json' ~/Users ~/Sync
The above example will exclude all .json files. The * can be used at the beggining, at the end, in the middle, etc. i.e. rsync -avh --exclude '*ignore*' ~/Users ~/Sync will ignore the *rsync-ignore.txt* file
Exclude Files based in File Size
This option let's you specify between the maximum or minimum size of the files you want to exclude.
iamgroot@laptop:~$ rsync -avh --min-size=1K --max-size=50K ~/Users ~/Sync
The above example will exclude any file bigger than 1K and smaller than 50K
Here is what the man page says about it
--max-size=SIZE
This tells rsync to avoid transferring any file that is larger than the specified SIZE. The SIZE value can be suffixed with a string to indicate a size multiplier, and may be a fractional value
(e.g. "--max-size=1.5m").This option is a transfer rule, not an exclude, so it doesn’t affect the data that goes into the file-lists, and thus it doesn’t affect deletions. It just limits the files that the receiver requests to be transferred.
The suffixes are as follows: "K" (or "KiB") is a kibibyte (1024), "M" (or "MiB") is a mebibyte (1024*1024), and "G" (or "GiB") is a gibibyte (1024*1024*1024). If you want the multiplier to be 1000 instead of 1024, use "KB", "MB", or "GB". (Note: lower-case is also accepted for all values.) Finally, if the suffix ends in either "+1" or "-1", the value will be offset by one byte in the indicated direction.
Examples: --max-size=1.5mb-1 is 1499999 bytes, and --max-size=2g+1 is 2147483649 bytes.
Note that rsync versions prior to 3.1.0 did not allow --max-size=0.
--min-size=SIZE
This tells rsync to avoid transferring any file that is smaller than the specified SIZE, which can help in not transferring small, junk files. See the --max-size option for a description of SIZE and other information.
Note that rsync versions prior to 3.1.0 did not allow --min-size=0.
Ok, that'll be for today's example, thanks for reading!!!
Follow, ❤ or 🦄
Top comments (0)