DEV Community: Sergey Kibish

How to Backup to Backblaze B2 with Rclone and keep the network alive

Sergey Kibish — Sat, 27 Nov 2021 15:11:53 +0000

Some time ago I concluded that it would be nice to have an offsite copy of the important data if something will go completely wrong.

I'm living in a place with not the fastest internet download/upload (39/12 Mbps) speeds. It took some time to tune all the parameters of the Rclone¹ to successfully create an initial backup (around 400GB) and keep the network alive.

Why to backup

When you work more and more in IT you understand the importance of a good level of redundancy. Everything breaks at some point (accidentally or on purpose), you need to have backups.

There is a 3-2-1 backup rule² which is nice to follow:

There should be at least 3 copies of the data, stored on 2 different types of storage media, and one copy should be kept offsite, in a remote location (this can include cloud storage).

I already had two pieces of the formula (3 copies, 2 of them on different external hard drives). One was missing: offsite backup.

When I was young I had not a pleasant experience of losing hard drive and all family photos, videos, etc... After this, I'm a little bit paranoid about where such information should be stored and how it should be backed up.

Currently, my storage environment looks as follows:

Self-hosted instance of Nextcloud³ as a main source of storage
Backups to two external hard drives

And the time came to introduce cloud backup.

Where to backup

I've wanted something simple, cheap, secure which I can execute in the terminal and it would just work.

One of the options was to go with some VPS and rsync⁴ classic setup. This setup is not so cheap. Each cloud provider has a cost for bandwidth, storage and compute. Price is very different, but with 400GB you might end up spending from $5 to $10, or more per month. I wanted something cheaper.

I've chosen Backblaze B2⁵. It's less than $2 ($0.0005 * 400) per month for me. Which is very good for me. If I will need to extract the full backup, it will cost me $4 ($0.01 * 400).

How to backup

Okay, how to backup?

rsync would not work in this case because it works with file system but with B2 you would need to use API, b2 CLI, S3 compatible tools (because it is compatible with S3 API) or something else.

And I've stumbled across Rclone¹:

Rclone is a command line program to manage files on cloud storage. It is a feature rich alternative to cloud vendors' web storage interfaces. Over 40 cloud storage products support rclone including S3 object stores, business & consumer file storage services, as well as standard transfer protocols.

What a beast is this tool. It can not only backup and restore, but also encrypt and decrypt the data, mount, analyse and other stuff. Worth checking out.

Configuring rclone

All configuration is stored in the rclone.conf file which can also be encrypted and stored in the git repo.

The configuration looks similar to the below:

[local]
remote = /var/storage

[b2]
type = b2
account = BACKBLAZE_ACCOUNT_ID
key = BACKBLAZE_ACCOUNT_KEY
hard_delete = true

[b2-crypt]
type = crypt
remote = b2:backup-bucket
password = CRYPT_PASSWORD

These are three remotes:

local is my local folder which should be backed up
b2 is a connection to my B2 account
b2-crypt is a special remote to save files in b2 bucket encrypted. It wraps b2 remote. More details can be found in the very nice documentation.

To do the encrypted backup, you do the following in the console:

rclone --config rclone.conf sync local: b2-crypt --progress

And it will start to backup. Slowly.
And remembering my network speeds, it will start to die because rclone will eat all the bandwidth. You will not be able to watch anything on YouTube or Netflix. Even loading some blogs will take a longer time than usual. Then rclone will tell you that it starts to drop connections sometimes.

Rclone crashing the internet

Search will tell that you are not the only one with such a problem⁶. If you don't have a very fancy router, you might end up with a phenomenon named bufferbloat⁷. The question is, what to do? We still need to backup our data.

Time for some tuning

Step 1.
Take measurements of your WAN download/upload speeds with Speedtest or embedded in the router speed test tool.

Let's take my numbers from the above:

Download: 39 Mbps
Upload: 12 Mbps

Step 2.
If you have some QoS settings in the router, you might want to manually set the limits for upload and download bandwidth 10% less than measured, e.g. 35/10,8 Mbps. This will help the router to have room to breathe.

This step is optional, but if you had some problems with the internet connection in general, this setting might help you solve the issue too.

Step 3.
Now we need to configure the rclone parameters.

rclone --config rclone.conf sync local: b2-crypt: \
  --transfers 20 \
  --b2-chunk-size 48M \
  --b2-hard-delete \
  --fast-list \
  --bwlimit "08:00,512k 12:00,10M 13:00,512k 23:00,10M" \
  --progress

--transfers — how many files to transfer simultaneously. Backblaze can handle a lot of simultaneous transfers and works better when there are many of them. According to the rclone backblaze module documentation⁸ the recommended size is 32. I've ended up with 20, experimentally.
--b2-chunk-size — the default is 96M which is must be fine, but because speed is not fantastic and I wanted to put many files online, sliced it in half.
--b2-hard-delete and --fast-list are optional but I've set them in my script because they are valuable during next backups. By default, B2 does not delete files when you delete them. In my case, I want to delete them and not store shadow copies. Fast list helps to execute fewer commands against the service (e.g. list all files in one request and not many).
--bwlimit. This one is interesting. Because I needed to work, I've limited rclone on the usage of the network. In the example above I've instructed rclone to use only 512kbps starting 08:00 in the morning, a little bit more during lunch, back to 512 after and back to the maximum at night.

If in Step 2 you enabled QoS settings on your router, then you can change 10M to off (use all that is available).

Final words

With the above recommendations, everyone in your household will be able to use the internet normally and backup will happen... but slowly. In my case, it took around 4-5 days. So, if you have access to the data centre or fibre network, you will have more luck!

Git Sweet Git

Sergey Kibish — Mon, 20 Jul 2020 16:47:56 +0000

Your Git repository is like your home. You spent time there.
You move things. You want it to be a nice and comfortable place for you. And you want to keep it clean and ready for guests. Yes, sometimes it's a mess after a party but still, you know how to clean everything and get back a shiny house.

README

When guests arrive to you what is the first thing they see? Door and a rug with "WELCOME" word printed on it.

This rug is a README.md file in your Git repository. This is a first place where users try to familiarize with your project and start to look around. Like in real life.

Usually your guests will take off shoes on this rug while inspecting what they can see.

In README.md you want to answer following main questions:

What is this?
How to install this?
How to use this?

If you would like your guests to contribute to today's dinner, then additional questions should be answered:

How to contribute?
What help is needed?

There are different homes. But Git repository should be a minimalist house.

What does it mean - "minimalist house"? My definition is following:

Minimalist house is a house that has only what is needed.

.gitignore

You know, when guests enter your room you don't want them to see socks scattered all around. You need to put everything in a locker. In the Git repository "socks" can be hidden with help of .gitignore.

What are those "socks"? Something that adds mess to your repository and should not be shown to your guests. Something like node_modules, vendor, .DS_Store, .vscode, etc.

In Git there are two types of .gitignore files. Global and local. Global is used to ignore files that expose your working environment. For example:

Files that are generated by the OS, e.g. .DS_Store
Files that somehow show which editor you are using, e.g. .idea, .vscode

Local is used to hide irrelevant parts of your project. For example:

node_modules
*.log
.env
vendor

Those files and folders that are not mandatory to save in the repository. node_modules are populated after npm install, thus are not needed. *.log files are added when you debug something. .env file is your specific environment configuration that should not be shared with others. I think you've got an idea.

Good collection of global and local .gitignore files can be found in github/gitignore repository.

Good repository should only consist of things that are needed for the project.

.gitattributes

While you cook guests wait for you in the main room and start to walk around touch things and then somebody asks a question: "Is this a picture or a book?" pointing their finger on a shelf. "This is a picture", you answer. "Looks so real", one of your friends whispers.

Sometimes you need to help Git the same way as you helped your friend to recognize what it is in front. With this helps .giattributes file. This file helps Git to understand the contents of other files to better diff / render them.

Nice collection of .gitattributes files can be found in alexkaratarakis/gitattributes.

Some web project .gitattributes file might look as follows:

# Graphics
*.gif    binary
*.jpg    binary
*.png    binary

You want to help Git to identify images as images and not text, for example.

At this point repository looks like this:

README.md
.gitignore
.gitattributes

This is a good start for any project that should go with an initial commit.

Commit message

Somebody found your family photo album. They start to look through, year by year, event by event. Childhood, youth, adulthood. What were the times... What is good about this album is that on each page there is a title and some description of what happened.

Git commit history is similar to this photo album. Event by event you can go through it. You can go from beginning till the end viewing commits with title and description.

Yes, commits have title and description. And it is nice to have them both. For example:

feat: add view counter to each post

Added view counters to each post
to understand what posts are popular
and which are not.

Closes: 906

When somebody in the future will open this commit, they can understand why this was done. In the commit description you can have some meta information to help find related content. In this case you see Closes trailer which point to an issue in some ticket system that can be closed by this commit. This also might be a link to some internal wiki, etc.

There are many different conventions on how to write commits and it is mandatory to agree on the approach within your team. Otherwise a beautiful story might be lost and you will need to do a lot of explanation to your guests. For the convention example you can take a look at Git Wiki and Conventional Commits Specification.

Treat Git commit as emails. There is a good summary on Git Wiki about how to write nice Git commits:

The "one-line summary plus body of the message" has a strong correlation with how we communicate [..] via e-mail.
You do not start a sentence on the "Subject: " header and continue on to the body of the message, starting the body halfway of the sentence.
Instead, you try to make sure you write something sensible by itself on the "Subject: " header to help the recipient when later scanning for it among bunch of messages, and you write a full paragraph that you can understand without reading the subject line first.

Commit history

Now you enter the room with 12 plates in your hands. Everyone looks at you shocked (because you handled so many). And then they fall...

It's better to serve plates one by one or have a few in hands. Stable and more control.

Same applies for Git commits. Smaller and granular ones are better.

Personally I would recommend to have commits on each move while you fix bugs or develop features. This will help reviewers to understand thinking behind final implementation. Your commit history might look like this in a feature branch:

* c911ba2 (HEAD) simplified method according to provided feedback
* 4692d22 added integration tests
* 7b84532 added unit tests
* 47b8452 implemented message parsing logic

When everything is reviewed and you are ready to merge - squash commits into one:

* c911ba2 (HEAD) feat: add message parser

With body having context of work done:

feat: add message parser

Implemented message parser according to #42 document.
Unit and integration tests cover all cases described
in the linked document.

Reviewed-By: D
Closes: 21

Rebase and Merge

While you make a beautiful dessert in the kitchen a lot of things are flying around. Many ingredients lie on a table, jars are open, everything smells tasty. Process is going on. When you are almost finished, you clean everything up and add final touch with a cherry on top.

It is good to do a rebase when you are almost finished and would like to present your beautiful work. Merge - to signal that work is completed.

Use both. The approach I prefer is:

Rebase on feature / fix branches
Merge to main branch when everything is finished

Rebase is a nice helper for cleanup after work. Highly recommend to read about "The Golden Rule of Rebasing" before using it.

Merge is a completion indicator. When work is finished and a milestone is achieved, it can be marked with a merge commit.

.gitconfig

When conversations are finished, you and your friend go toward the corner with piano and guitar and start jamming, having fun and singing songs together.

It is possible to start playing straight away because all instruments are in tune and fingers know how to play these instruments. To play even better on guitar some time ago you gave it to master which made it even better to play. Less movement for hands, more precise sound because strings were changed.

Git can also be treated as an instrument. You can play on it right now or you can tune it up a little bit to play even better.

Such tuning is done with help of configuration and aliases. For example, you would like to list tags and branches in your repository:

git tags
git branches

# Without configuration
# git tag -l
# git branch -a

Aliases hide complexity. Better to read and less possible mistakes to make.

Push branch and added release tag:

git push

# Without configuration:
#  git push --follow-tags or git push && git push --tags

With help of .gitconfig you can set this as default behavior (push tags on git push).

My recommendation is to share the same .gitconfig across your team. Good configuration will ensure that team members are on the same page.

Nice configuration can simplify work with Git and make it more predictable.

For a reference you can take a look at my .gitconfig in skibish/dotfiles.

rm -rf . && git init

If you did some home repairs you know that sometimes it is better to start from scratch.

Sometimes this happens to the Git repositories too. Even though ideally Git should store all the history of your repo, sometimes it makes sense to drop everything and start recording history of changes from scratch.
When this might happen:

You build an MVP or PoC of a product. It would be better to start the history of a "real" production product fresh
Repository is not in a good shape. A lot of artifacts are not in their places, some burden around, not easy to follow history, etc.

Don't be afraid to throw things away.
It just means that you are growing and freeing up space for future endeavors.

Final words

These little things helped me to simplify my Git workflow. Hope that something from suggestions will work for you too.

What are your approaches to keep Git repository clean and work with it easier?

About Those Behind The Curtains

Sergey Kibish — Wed, 24 Jun 2020 17:36:22 +0000

I want all involved in development to think about other users. Not about those who use your application to solve their problems. But those who extend your app, run it in production and integrate with other systems. About developers, operations, SREs and many others who make your business running. Those behind the curtains. I want you to think about the Developers Experience (DX).

The problem

Working with different businesses and in outsource companies I've spot following tendency: The main focus is always on business requirements and UI features of built application. Because this is the first thing that real users will see. A client who ordered a new feature would like to see a pretty demo and understand that money invested is working. Such priority looks logical and right. But problems are hiding behind such thinking.

There are many other users behind the curtains. Almost nobody thinks about developer experience (except IT-focused products). Giving priority only to business features and UI is not enough for some solutions.

Nowadays all businesses are digital. They integrate with many systems, chats, banks, you name it. There is always some development work happening and many of them don't focus on DX. Which is bad.

It is bad because engineers also should feel happy using your product. If they struggle, you as a business might have loss of trust or reputation. All users of your product are special. Yes, they all have different needs, but meet those needs.

Culture

DX is also about product vision. It is something that should be in your development team. It should become part of a culture. Understand what you create. Not from specifications but from the real users perspective.

Frontend developers are more product-oriented than backend developers. They focus on users and their experience. Backend developers focus on algorithms and data structures. I know that there are backend developers who also think about their users. This article is not about them.

The product team and people who have vision should teach others to see problems from the real users perspective.
If you would like to bring innovations and raise the quality of the product - such thinking should become part of you.

Bad DX examples

What experience should engineers avoid trying to use your application? This is an opinionated short list of bad DX examples. If you have something to add - welcome to the comment section below.

Bad API design

Take a look at the following:

GET     /users
GET     /users?id={userID}
NEW     /users/create
POST    /users/update?id={userID}
DROP    /user/{userID}

Do you feel pain looking at this?
I do.

It's a mix of different things
Semantics are not followed
Not easy to understand straight away what is happening
Not standard HTTP methods are in use (but this is fine according to RFC 2616).

What is good is that it works. But it would be a lot better if it looked like this:

GET       /users
GET       /users/{userID}
POST      /users
PUT       /users/{userID}
DELETE    /user/{userID}

Clean. Without looking at documentation such API already starts to give clues.

Many frameworks (Ruby on Rails, Laravel, Play, etc.) suggest following design of REST APIs by default. Actually, Roy Thomas Fielding was the first who proposed REST. To start developing taste in API design take a look at this resource or this article to get started.

Another thing that I would like to mention is deep nested objects in API responses. Objects are trees and sometimes those trees grow like real trees. Very tall and dense. It is not good when your response object looks like this:

{
  "id": "001",
  "value": "some value",
  "child": [
    {
      "id": "002",
      "value": "another value",
      "child": [
        { ... }
      ]
    }
  ]
}

To get needed values you need to traverse a graph. Algorithmic complexity to parse a graph is O(V+E) (DFS). And it's a lot. There might be too many nodes to traverse. It would be better to flatten such object.

[
  {
    "id": "001",
    "value": "some value"
  },
  {
    "id": "001:002",
    "value": "another value",
  },
  {
    "id": "001:002:xxx",
    "value": "and another value",
  }
]

Much better. And this might be a final variant if the return object is small. Parsing the following structure will take O(n) (because of one loop). Nice. And reading such structure is easier. Is it possible to lower search to O(1)?

{
  "001": {
    "id": "001",
    "value": "some value"
  },
  "001:002": {
    "id": "001:002",
    "value": "another value",
  },
  "001:002:xxx": {
    "id": "001:002:xxx",
    "value": "and another value",
  }
}

Now it is O(1). You can specify a key and extract value without iterating an object. Some may recall Redux State Shape Normalization by looking at the last example.

TIP: Use the following approach to store objects under keys in cache.

Solutions provided here are not silver bullets. Key takeaway:

Keep your objects as flat as possible.

Take a look at your APIs. Can somebody's life be simplified?

Configuration properties chaos

Sometimes configuration properties are treated not seriously. Take a look at this snippet:

base.url=http://localhost:8080
port=1313
jobs.disabled=true
extract.values.and.save.period.hours=5
send.email.each.12.hours=true

Naming is strange and hard to read
Hard to follow logic (because there is no logic)
Time unit is big. It might be that there are properties set in other time units than hours.
It's hard to figure out the meaning of some properties by their names
etc.

Treat configuration properties as first-class citizens.

Developers and Ops interact with your application using properties. If it is not clear how to use them - mistakes may arise. Additionally it is a sign about the bad quality of your product.

How to fix this?

# Network address (server.address) to which the server should bind
# and listen port for incoming HTTP requests (server.port).
# Default:
server.address=127.0.0.1
server.port=80

# Job to extract and save reports to storage.
# If enabled (jobs.extract_reports.enabled) job will
# run every 5 hours (jobs.extract_reports.interval in seconds) and save
# reports to defined destination (jobs.extract_reports.destination).
#
# Enable email notifications
# (jobs.extract_reports.email_notifications) to send summary
# of jobs done during the day.
# Default:
jobs.extract_reports.enabled=false
jobs.extract_reports.interval=18000
jobs.extract_reports.destination=/tmp/reports
jobs.extract_reports.email_notifications=false

Comments added - more context given
Sensible defaults are set
All properties are logically grouped
Application with set defaults will be able to start without extra configuration

For inspiration I would recommend to take a look at:

Logs with no meaning and action

Logs are the main interface of your app for the Ops team.

If logs lack context it might lead to long investigations of a problem.

[12:40:45] ERROR Exception happened
com.ekiras.exception.BaseException: Base Exception
at com.ekiras.controller.HomeController.ex1(HomeController.java:21) ~[bin/:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_45]
...
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:893) ~[spring-webmvc-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:969) [spring-webmvc-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:860) [spring-webmvc-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:622) [tomcat-embed-core-8.0.30.jar:8.0.30]
...

Such a log only states that something bad happened. For those who don't have access to the code of your app this is almost zero information. Stacktrace will not give information on how to solve the issue.

[12:40:45] ERROR Connection to the database (db.internal/app) failed. Check datasource property in application configuration

The message above is clearer because it states directly:

What is the problem - DB connection failed
How this error can be fixed - check datasource property

Usually, machines collect logs first and after humans read them. If this is the case, write log in JSON or structured text formats:

time="2020-06-14T01:27:38-04:00" level=error msg="Connection to the database (db.internal/app) failed. Check datasource property in application configuration" status_code=500 path="/about"

With the example above, logs can have more context like response status code and path that failed. The following format is readable by machines and by humans. Rule of thumb:

One event equals one log line.

No comments in code

When you want to use some method from the library and it doesn't have a description - it's a problem.

public List<User> users(Bool limit, String group, String sort) {
  // ...
}

Guess games, anyone? The assumption is that it will return the list of users. But what is limit, group and sort? What are their values? What can be set there?

Without comments we need to read code to understand what is happening there. After reading open Pull Request with the following comment:

/**
 * Return list of users.
 *
 * @param limit specifies to get users with limited access (default: true)
 * @param group user group to lookup (default: "regular")
 * @param sort  how to sort users (default: "asc"; possible values: "asc", "desc")
 * @return      list of users
 */
public List<User> users(Boolean limit, String group, String sort) {
  // ...
}

Congratulations! You've got angry developers who spent a lot of time understanding code that they should not even know.

Go is playing this game by making comments mandatory.
And another player is Rust, who go with documentation comments as tests approach.

Self-documenting code is a myth.

Document your code. Users and the future self will be very grateful for such effort.

Unpredictable responses

For example, you want to get a list of prices set for some item in the shop during the specified day. API response might look following:

{
  "item": "78956745",
  "day": "2020-06-14",
  "prices": [
    "8,72",
    "9,01",
    "8,02"
  ]
}

What if there were no price changes during the day? Item ID and day are valid. The only thing missing is the list of prices. Consider the following options:

Prices are returned as an empty list:

{
  "item": "78956745",
  "day": "2020-06-14",
  "prices": []
}

Prices are not returned:

{
  "item": "78956745",
  "day": "2020-06-14",
}

Prices are set to null:

{
  "item": "78956745",
  "day": "2020-06-14",
  "prices": null
}

Two last options are not predictable. Option 2 modifies response by removing prices object from the response. Not good. Option 3 makes it worse. You saw that in response there is a list and now you observe different type.

The best approach is to go with Option 1. You know that prices is a list. In future, you also expect that there will be a list. If there is no information, then return an empty list. Less confusion and meet expectations.

Only once executed configuration properties

You can give an option to the user to configure your service via configuration files or APIs. In some scenarios, this does make sense. But sometimes there is a case when configuration in a file is used only once to bootstrap an app. And then only the API is in use.

Don’t do that. If you provide both options - both of them should work the same way. Do change in file - changes apply. Do changes via API - changes apply.

Configuration operations that users do should be idempotent.

It doesn’t matter how you set configuration - it should always work the same way.

How to level-up DX

It's all about thinking. And thinking is something not easy to change. I would suggest to start with The Twelve-Factor methodology. It is a set of good practices to apply. It's more oriented towards backend applications, but ideas can apply to other areas as well. Take a look at Stripe for inspiration. This is one of the companies that is known for great DX. Another company that I like and it inspires is Digital Ocean. Also suggest to read fresh and nice article from Chris Coyer about meaning of DX for different people. Are you building CLI? Get inspiration from Carolyn Van Slyck talk on how to design Command-Line tools that people love.

And ask yourself questions:

Who are end users / stakeholders? Is it another team? Other developers who perform integrations?
How will this library be used?
Can this be simplified?
Is this readable and understandable?
Does it look simple?
Will my colleagues understand this?

And ask for feedback to become even better.

Hope you enjoyed it. Happy coding and nice mood!