Archer Allstars

Posted on Mar 16, 2023 • Updated on Mar 22

Replace Google Drive with Storj, plus Rclone and Kopia.

#tutorial #productivity #web3 #opensource

UPDATE 2023/03/21: The free tier is depricated. However, you can still test the service for free within 30 days period, see more about the pricing here. It has been like a roller coaster in the marketing part.

UPDATE: Now, the free tier is reduced to 25GB. Also note that the account created before this change will still get the 150GB free tier forever. See more here.

When I moved to Linux, I lost the ability to sync with Google Drive because Google Drive doesn't provide a Linux desktop client. Therefore, I moved to Storj, a Web3 decentralized cloud storage platform, and never looked back.

Why Storj?

~~First, because you'll get 150GB[opening] 25GB[first edit] of cloud storage from Storj "for free"!~~ Second, you'll get top-notch security since all of your files will be encrypted and then split into pieces across over 16,000+ server nodes around the globe. There's no way for data leaks to happen through zero-trust security, see more here. Lastly, the performance is very good @ 24 Gb/s download speed through S3 connection.

How is the cost (after the first 150GB[opening] 25GB[first edit])?

~~You get the first 150GB[opening] 25GB[first edit] of storage and 150GB[opening] 25GB[first edit] of bandwidth/month for free. After that,~~ it will be $4/TB for storage per month and $7/TB for bandwidth per month, which is pretty cheap compared to similar services. You can pay with their Storj coin or a credit card. See more here.

Other Web3 cloud storages?

There are many interesting Web3 cloud storages, for example, Sia, Filecoin, Internxt, and Arweave through Akord.

Sia is the closest competitor to Storj. It aims at a low-cost cloud storage market. It also supports S3 connection. But there's no free run to offer. And there's no pricing available on their website.
Filecoin has nothing to offer to the end-user at all. If you really want to use it, I would recommend Fleek. It's expensive but can do much more than just cloud storage.
Internxt has desktop and mobile clients available. However, it is very limited in term of features, for instance, file versioning is currently missing.
Akord has a different use case altogether as it offers a permanent storage. The price is high, but your files will be kept forever, hence unlimited file versioning, and available for download without additional cost. Therefore, it's not suitable for syncing a large number of files.
As a bonus, Züs (formerly 0Chain) is another interesting Web3 cloud storage project you might want to keep an eye on. They are not running on mainnet yet, but seem to be very promising. See their roadmap for more info.

Automatically sync your files on Storj with Rclone and Watchexec

You can upload and download file using web interface. However, the web interface can't sync your local files on the cloud. Therefore, we will add sync features with Rclone, and make it automatically with Watchexec.

This walkthrough will focus on Linux, openSUSE Tumbleweed to be specified. However, Windows users can use Rclone and Watchexec without issues, hence can set up sync functionalities with Storj. Please see Rclone and Watchexec websites for the install instructions on Windows.

Setup Rclone to sync files to Storj

Install Rclone in YaST Software Management, or simply install it using this command:
```
sudo zypper install rclone
```
Setup S3 connection and Rclone by following the official Storj Docs.

Importantly, in the process of creating the S3 credential, the UI will ask you to generate or enter your 12-words passphrase. You will have to enter your project's passphrase. Otherwise, you won't be able to access your S3 bucket.

In order to make it easy to remember which Rclone configuration is for which Storj bucket, I recommend to always name the connection name the same as the bucket name.

You can also create a master key for your S3 access. It's the key to accessing all your buckets in a specific project. However, I don't recommend using this on mobile. Because there's neither an official Storj app nor open source S3 storage browser on mobile yet. On PC, even without the official client, you can use a master key with ease since all the toolchains are open source.
Now, with a working Rclone connection, you can sync from any local folder, including the folders and files inside it, to Storj cloud with this command:
```
rclone copy --update --progress /path/to/local/folder/ connectionname:bucketname/
```
For example, I can sync my storj folder in my home directory to my storj-bucket using this command:
```
rclone copy --update --progress ~/storj/ storj-bucket:storj-bucket/
```
Note that I use rclone copy command instead of rclone sync command because rclone sync command can delete all your files on the cloud if you accidentally delete your local files.

The --update option will only overwrite files (with the same file name) on the cloud if your local file is newer.

The --progress option will show the transfer progress of the command.
You can sync from Storj cloud to your local folder by using this command:
```
rclone copy --update --progress storj-bucket:storj-bucket/ ~/storj/
```
You can mount Storj cloud (storj-bucket) to your local folder named storj-bucket (you need to create a folder to be used as a mount point) in your home directory by using this command:
```
rclone mount storj-bucket:storj-bucket/ ~/storj-bucket/
```
You can enable file caching to be able to work fully in sync on the cloud by using --vfs-cache-mode full option and --vfs-cache-max-age 24h0m0s (default 1h0m0s) to set the caching duration. See more about this option here.
You don't have to remember all of these lengthy commands by using the app called MenuLibre to easily create a desktop file to launch the commands, or even add it to run automatically at startup by using GNOME Tweaks. Both can be downloaded using GNOME Software. For example, I created rclone mount command in a desktop file using MenuLibre, so I can mount Storj cloud easily with a single click, as shown in the screenshot below:

Note that a desktop file doesn't support a relative path (a path with ~/ for /home/username/). See more here.

Setup Watchexec to sync files to Storj... Automatically!

Install Watchexec in YaST Software Management, or simply install it using this command:
```
sudo zypper install watchexec
```
Create a file watcher script to execute commands in response to file modifications in the synced folder:
```
#!/bin/bash
watchexec --watch ~/storj/ rclone copy --update --progress ~/storj/ storj-bucket:storj-bucket/
```
You can copy this script to a plain text editor and save it at your home directory as filewatcher.sh.
Make the script executable as a program in the script's right-click properties.
Run the script in the terminal with ./filewatcher.sh.
You can watch as many folders as you want by editing the ./filewatcher.sh script, for example we will watch ~/storj1/ and ~/storj2/ folders, and execute rclone copy commands in response to file modifications in these folders:
```
#!/bin/bash
watchexec --watch ~/storj1/ rclone copy --update --progress ~/storj1/ storj1-bucket:storj1-bucket/ &
watchexec --watch ~/storj2/ rclone copy --update --progress ~/storj2/ storj2-bucket:storj2-bucket/
```
Make the script run at startup by creating the desktop file with MenuLibre executing ./filewatcher.sh. Then, adding it to startup apps using GNOME Tweaks, as shown in the screenshot below:

Note that a desktop file doesn't support a relative path (a path with ~/ for /home/username/). See more here.

Advanced Usage Examples

Syncing a path with whitespace or non-English characters

The problem arises when you try to sync a folder naming with whitespace or non-English characters. This can easily be fixed by:

watchexec -n --watch ~/"Documents/Storj Backups"/ rclone copy --update ~/"Documents/Storj Backups"/ storj-backups:storj-backups/

We use -n option to run the command directly, instead of wrapping it in a shell (watchexec runs in a shell by default). Then, with the path, you encapsulated both ends with ". These eliminate the issues with whitespace and non-English characters.

Using a variable in your bash script

You're using a variable to make your script easy to manage. Imagine if you want to change the above path to another location, you will have to change it 2 times (for watchexec and for rclone), and that's only one path. If you use a variable, you'll only need to change the path you save in the variable. From the above example:

#!/bin/bash

# directory path variables
dir=~/"Documents/Storj Backups"/

# the monitor-sync command
watchexec -n --watch "$dir" rclone copy --update "$dir" storj-backups:storj-backups/

Note that when using a directory path variable which contains whitespace or non-English characters in your bash script, don't for get to encapsulate it in " as shown in the above example.

Automatically backup your files (with file versioning) on Storj with Kopia

Although we can sync files on Storj without any issue with Rclone. But it doesn't perform any file versioning, which would require at least a compression and an incremental backup mechanism. Otherwise, your file versions/history would fill up the entire storage quickly.

Many commercial cloud storage backup services provide this functionality through their desktop/mobile clients. However, Storj aims toward building an infrastructure for developers to build upon their backend. That's why they only provide basic functionalities through their web interface.

Why would you need file versioning?

If you work with many files, mistakes can happen, so a reliable backup system can save you a lot of time since you don't have to redo your work again. However, Storj only provides basic functionalities for you in their web interface, i.e. upload and download files. Therefore, you will have to build this feature yourself.

Fortunately, this is very easy with Kopia, a cross-platform backup tool that supports incremental backups, client-side end-to-end encryption, compression and data deduplication. It can backup to S3 cloud storage through a very intuitive GUI, thus supported Storj.

Setup Kopia for automatic backup with file versioning.

Install Kopia by going to Kopia's GitHub releases page. Download the installer file or simply using the AppImage file. I recommend the AppImage version for openSUSE users since the RPM's version update checking doesn't work.
Setup a new backup by creating a new backup repository. Choose Amazon S3 or Compatible Storage, as shown in the screenshot below:
Fill in your access key, secret key, and endpoint for your S3 connection that you generated from Storj. Please note that the endpoint can't contain https://, just fill in only gateway.storjshare.io. Otherwise, the connection won't be established.
Choose the folder you want to backup and take a new snapshot. You can change the snapshot policies at this point, the UI is self-explainable.
Wait for the first snapshot and upload to finish.
The snapshot will look like this on Storj. These snapshot files require Kopia to open.
You can click at the snapshot's path to manage each snapshots easily. You can also mount the snapshot on your system.

Note that Kopia is not available on mobile yet. Therefore, you can't open Kopia's snapshot on mobile. You'll have to make priority on your files, i.e. if it needs to be viewed on other devices (not on PC) and not required any file versioning, use Rclone. If it's a working file, for example, a Blender file that can't be opened on mobile anyway, use Kopia.

Mobile apps for Storj cloud

There are not many S3 file browser apps on mobile. However, there are some interesting apps that I use on mobile. They are S3Drive and FolderSync.

I hope this helps anyone who wants to leave Google Drive. I leave Google Drive because Google doesn't provide Drive client on Linux, so I lost the ability to sync when I left Windows. Storj doesn't let me down, as I can use all the tools that support S3 connection "easily". However, I am still missing the face recognition feature on Google Photos 🥲

Cover Photo by moren hsu on Unsplash

Storm Troopers Photo by Brian McGowan on Unsplash

In Space Photo by The New York Public Library on Unsplash

City Photo by Willian Justen de Vasconcellos on Unsplash

Kid Photo by Yuri Shirota on Unsplash

DEV Community