DEV Community: Tryggvi Björgvinsson

How do you manage multi-project documentation?

Tryggvi Björgvinsson — Sun, 09 Feb 2020 12:07:50 +0000

At work we have lots and lots of software projects and we believe having good documentation makes us more efficient and the code more maintainable.

The problem is how to manage this documentation. We used to have a central wiki system where we would document projects but because it was disconnected from the code, the docs got outdated.

We run our own Gitlab instance so we decided to move documentation into the Gitlab projects where we'd write quick intros and get-the-project-running docs in a README file and more detailed docs in the project wiki. That allowed the docs to be closer to the code and at least the READMEs checked via merge requests.

But non-code docs (processes etc.) are then more difficult to manage. We have some documentation repos to get around it, and a central doc repo with links to all the other repos and their docs.

How do you manage documentation for multiple projects?

Photo by Viktor Talashuk on Unsplash

Jinja templates in Salt

Tryggvi Björgvinsson — Sun, 26 Jan 2020 23:58:59 +0000

We ended the second article in this series about the configuration management tool Salt, with a problem. Throughout that post we defined roles in grains and configured what packages we wanted to install in the pillar. But at the end of the post, we were still installing packages by targeting the grains in the state.

To make the states really dynamic, we don't want to hard-code roles and packages in them. We want the state logic to be flexible. We want to configure our environment in salt's own key value store (the grains and the pillar). To do that, we need to go deeper into templating.

Renderers in Salt

By default, every salt state file is passed through a renderer before being processed by salt. The default renderer in salt is a Python templating engine called Jinja. The output of the Jinja process is then passed to a YAML processor which parses the data in the files into a data structure usable by salt. This rendering pipeline is commonly represented as jinja|yaml (if you know Unix pipes, you'll understand why).

These represent the two types of renderers in salt. Jinja is a text renderer, that takes in some text and returns some text. YAML is a data renderer, that takes in some text and returns a data structure.

There are other text renderers available such as genshi and mako, both of them templating engines. There are also other data renderers, such as json.

In this post we'll stick to the default jinja|yaml and focus on the text renderer, namely Jinja.

Jinja

As a templating language, Jinja allows us to generate output text files by processing a bit of logic placed in a source text file, a template file as it is called. Jinja even allows us to use a bit of Python for the logic.

For example let's say we would want to turn a Python list into a Markdown list we could have a template like this:

Here is a markdown list for you:
{% for word in ['apple', 'grapple', 'wroom'] %}
* {{ word }}{% endfor %}

This would produce the following output when we run it through the Jinja renderer:

Here is a markdown list for you:

* apple
* grapple
* wroom

Let's break this example down a bit. As you can see, Jinja's templating language is based on curly brackets {} accompanied by another sign denoting statements, expressions, or even comments.

Statements

Statements are wrapped in curly brackets accompanied by %. The for loop statement in the example above is written as:

{% for word in ['apple', 'grapple', 'wroom'] %}

This is resembles (and is) the Python way of defining a for loop (excluding the colon needed in Python). An if statement follows the same curly percentage format, so an if statement to check if the word contains the letter a, would be written as:

{% if 'a' in word %}

Python uses indents to close of blocks for statements like these. That would raise problems in templates because it wouldn't be clear when you want the template to output white space or when you're just indenting a statement. To get around this problem, Jinja closes blocks with a Jinja-specific statement.

To close off a for loop statement you would use {% endfor %} while closing an if statement requires {% endif %}. So when you find a for loop in a Jinja template you will see where the block ends by looking for the endfor statement.

Closing statements like this is only required for code blocks like loops or conditionals. There are statements that do not require it, such as the set statement which assigns a value to a variable. If we would want to assign the string 'snow' to a variable called weather we'd write that as a statement:

{% set weather = 'snow' %}

We do not close a statement like this with a endset because it isn't a code block. It's just a single line statement.

Expressions

Expressions to print out something like a variable are expressed with double curly brackets {{ }}. In the example we want to print out in each line in our list an asterisk followed by the variable word so we write that as:

* {{ word }}

The output will be an asterisk followed by white space, because Jinja prints out all non-Jinja symbols as is. Thereafter it would print out the variable word (the Jinja expression). If the value of the variable word is apple it would output:

* apple

White space removal

Jinja prints out all symbols as is. That means white space becomes slightly problematic, because white space and line breaks (the invisible \n in our templates) all get printed out too. The example with the apple, grapple, wroom may be difficult to parse as it is. If we would instead write it as:

Here is a markdown list for you:

{% for word in ['apple', 'grapple', 'wroom'] %}
  * {{ word }}
{% endfor %}

We'd end up with an output containing a lot more white space:

Here is a markdown list for you:


  * apple

  * grapple

  * wroom

Jinja gives us a nice way to remove white space. If instead of opening the statement with a {% you open it with a {%- (that is, with an appended minus sign), you remove all white space in front of the statement. If you prepend the minus sign to the closing bracket %} to get -%} you remove all white space after the statement.

So to make our example more readable, we could add a single minus sign when we close our for statement:

Here is a markdown list for you:

{% for word in ['apple', 'grapple', 'wroom'] -%}
* {{ word }}
{% endfor %}

This would give us the output we want, but in a more readable way.

Other nice features

There are plenty of other features in Jinja that I won't go into but I do recommend you read the official Jinja documentation. There are two honorable mentions that I recommend you to check out.

One is filters. Filters are functions you can use to modify variables. Filters are applied using the pipe and can be chained (similar to Unix pipes), i.e. variable|filter|filter.

If we would for example want to print out a random word from our list we could use the built-in random filter:

{% set wordlist = ['apple', 'grapple', 'wroom'] -%}
Word of the day is: {{ wordlist|random }}

This would print out a string with one of the strings randomly chosen, for example:

Word of the day is: apple

I recommend going through the list of built-in filters in Jinja and get to know most of the filters.

Another feature which is nice is tests for variables, mostly because of syntactic sugar that makes the templates easier to read. Tests are done by adding an is behind the variable followed by a function. For example we can combine a lot of what we've discussed into our very own Jinja fizzbuzz generator:

{% for number in range(1, 22) -%}
    {% set modulo3 =  number is divisibleby(3) -%}
    {%- if modulo3 %}fizz{% endif %}

    {%- if number is divisibleby(5) %}buzz
    {%- elif not modulo3 %}{{ number }}{% endif %}
{% endfor -%}

This will output the fizzbuzz counter for a few numbers. In this we use a for loops, if and set statements. We print out variables and we use a built-in test called divisibleby that checks if the variable is divisible by a provided number. There are a few built-in tests in Jinja that can make the code easier to read.

But let's get back to the series. We're here to learn about Salt!

Using Jinja in Salt

In our last post we had defined what packages to install based on roles in our pillar. However, we were still hard coding these packages in our states. We should aim to keep configurations in our pillar and the state logic, not the configuration, in our states.

So to remind ourselves, the pillar has a variable called packages which contains a dictionary of package names as keys, and a description as the value. For example, this was the pillar for our cow host:

cow-e9e01577bbd8:
    ----------
    packages:
        ----------
        cowsay:
            A program with a cow
        fortune:
            A program for a fortune teller

Let's create a new state file and call it roles.sls. This file is going to have the logic we need to install the packages based on the pillar with the help of Jinja.

We still have the same problem as before, where Ubuntu installs in a different location than Alpine so we must create symbolic links. Now you however understand the logic a little bit better. This was the trick we did:

{% if grains['os'] == 'Ubuntu' %}
make cowsay available on ubuntu:
  file.symlink:
    - name: /usr/bin/cowsay
    - target: /usr/games/cowsay
{% endif %}

We used Jinja's if statement to check if the operating system defined in the grains is Ubuntu. If it is, then we create this state. If not, then we don't do anything.

Salt adds grains and the pillar to the global context of Jinja, and makes them available as variables. That's why we're able to write grains['os']. Similarly we can use the Jinja global variable pillar to access the pillar.

Back to the roles.sls. We'll put this Jinja/YAML logic into the file:

{% if 'packages' in pillar %}
install role tools:
  pkg.installed:
    - pkgs: {% for package in pillar['packages'] %}
      - {{ package }}
      {% endfor %}

{% if grains['os'] == 'Ubuntu' %}
{% for package in pillar['packages'] %}
make {{ package }} available on ubuntu:
  file.symlink:
    - name: /usr/bin/{{ package }}
    - target: /usr/games/{{ package }}
{% endfor %}
{% endif %}

{% endif %}

Let's go through this step by step. First we check if the pillar has some packages for us. If it does we define a state to install all role tools. This state calls the installed function in the pkg module and then supplies the list of all packages names by looping through the packages we get from the pillar.

Thereafter we create a separate state with a symbolic link for every package (again accessing the pillar packages, but only if we are on an Ubuntu machine). Now this assumes that all packages we install will be installed in /usr/games on Ubuntu. That may not be the case, but I'll leave it as an exercise for you to use the pillar value and some logic in the state to only do it if we explicitly tell salt to create this symbolic link.

In the next post I'll show you one way to achieve it.

Deploying our package logic

We can now safely drop the rest of the sls files we created for each of the roles. Go ahead and remove cow.sls, fortune.sls, and locomotive.sls from the state directory.

Then we need to update our top.sls state file to use the new roles.sls instead of the hard-coded state files we just deleted. Thankfully this is easy because we define packages in the pillar and only worry about state logic.

We could just apply this to all salt minions, because we do check if there are packages in the pillar before we apply them. However, let's just limit ourselves to those minions that have a role defined in the grains (so we can use this opportunity to remind ourselves about how to target with grains). Our top.sls file will then look like this now:

base:
  'roles:*':
    - match: grain
    - roles

Let's try it out. Again, we should kill all the machines and start with a clean slate. Type in these commands and answer yes when prompted.

docker-compose stop
docker-compose rm
docker-compose up -d
docker-compose exec salt bash

Once we're attached to the salt master, we accept the salt keys for the new minions (press enter to accept them when prompted):

salt-key -A

Then we simply apply the state to all machines:

salt '*' state.apply

You should see all states succeed. That's super.

So now we're at a place where, if we want to define new packages for a new role, we just define it in the pillar with some simple configurations and the rest is automatic.

Now we're starting to see the true potential of using Salt to manage IT infrastructure.

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

Cover image by Motokoka: Awashima jinja shrine

Grains and pillars in Salt

Tryggvi Björgvinsson — Thu, 16 Jan 2020 23:13:14 +0000

The first article in this series went over an example of how to set up the Salt configuration management tool to automate a computer infrastructure.

Salt is very powerful and in this next section of the series, we'll take a look at two important components, grains and the pillar. At a first glance grains and the pillar seem to serve the same purpose but there is an important distinction between them.

What are grains and the pillar?

Grains and the pillar are used by Salt to store data about the minion it is controlling. They are both in the end a key value store. In fact they are what is called in Python a dictionary (if you're into Perl or Ruby, they're known as a hash).

For example, the following is an example of some grains on the alpine minion we created in the first article in the series:



ipv4:
    - 127.0.0.1
    - 172.25.0.3
ipv6:
kernel:
    Linux
kernelrelease:
    4.15.0-65-generic

If you want to check out all of the grains, you can execute the following command in the salt master docker container (to get a shell, type this in the same directory as you keep your docker compose file: docker-compose exec salt bash):

salt 'locomotive-*' grains.items

The pillar data looks the same except the keys and values are different. However if you run the command to list the pillar items on your salt master, you won't get a similar result now:

salt 'locomotive-*' pillar.items

You'll get nothing as a result:



locomotive-03f0f2574592:
    ----------

That's because while many grains are automatically generated when the salt minion starts up, the pillar data isn't. In fact the salt minion doesn't ever generate the pillar data. That's the big distinction between grains and the pillar.

Grains are generated and/or stored on the minions themselves. Pillar data are generated and/or stored on the salt master.

This distinction allows the pillar data to contain sensitive data you don't want lying around on the minion. It could be minion configurations or variables you want to manage in a central place, well or just any other data.

Grains on the other hand are useful for information you won't necessarily know when you configure the minion like IP addresses, operating system, CPU information etc. It can also be arbitrary data, like a role you want to configure and store on the minion itself.

For example you could on the machine create a grain called role with value webserver. Then the salt master could pick up that grain and from there decide how to configure it, without changing anything in the salt master configuration files. The pillar data for that machine may then include information such as software licenses you need for a particular application, users allowed to log into web servers etc.

One good thing to keep in mind is that grains are fairly static, they change infrequently because they're just information about the system (either automatically generated or custom data). The pillar on the other hand is better suited for data that will change more frequently.

Configuring grains

Most of the grains you normally need are automatically generated. However, if you ever find yourself in a position where you want to configure them you can store them in the minion configuration file on the machine. Location depends on operating system, for example on Unix systems you find it at /etc/salt/minion except in FreeBSD where it's stored at /usr/local/etc/salt/minion. On Windows machines the configuration is found at C:\salt\conf\minion.

The configuration file is a YAML file and you can add grains as a root element in the configuration file with the grains you want to store. For example you could add this to the minion config:



grains:
  roles:
    - webserver

This would make the grain roles available with a list as its value, containing the webserver role (it doesn't have to be a list of strings but I try to choose a data structure applicable to the context and a server could have multiple roles so that's why I'd use a list for the roles).

Another place where you can store grains is in /etc/salt/grains. I like explicit so I like this location more than the minion config. It's a YAML file but if you want to use /etc/salt/grains you don't need the root grains element. So the equivalent /etc/salt/grains file to the one above is:



roles:
  - webserver

Let's adapt the example from the first post in this series and configure grains for the two salt minions.

Let's create two grains files, one for our alpine container and another for the Ubuntu container, defining roles for the machines. Create a file called alpine.grains in the same location as your docker-compose.yml file (we're going to mount it later on). Here are the contents of the alpine.grains file:



roles:
  - locomotive
  - fortuneteller

Create a similar file for the Ubuntu container called ubuntu.grains with these contents:



roles:
  - cow
  - fortuneteller

Now let's update our docker-compose.yml We just have to mount the two files we created at /etc/salt/grains on both machines so the docker-compose.yml file should look like this:



version: "3"
services:
  salt:
    image: salt-master
    volumes:
      - ./master.config:/etc/salt/master
      - ./states:/srv/salt
  alpine-minion:
    image: alpine-minion
      - ./alpine.grains:/etc/salt/grains
  ubuntu-minion:
    image: ubuntu-minion
    volumes:
      - ./ubuntu.grains:/etc/salt/grains

Let' get all of them up and running using these new grains and therefore their new roles. Run this command from the same directory:

docker-compose up -d

Still running first post containers?

If you shut down and removed the containers you created in the first post, you'll just have to accept keys like you did in the first post.

If you are still running the same containers, you should see the minions being updated (recreated), but not the salt master (it's up to date).

If you try to check things out with salt master now, you'll notice that it doesn't manage your salt minions any longer. The reason is that because we recreated the docker containers they now have a new minion id (it's a new machine basically). So if you're not following this guide adamantly and want to play around, you'll have to accept the keys as shown in the first blog post in the series.

Now we're ready to use those newly defined roles somehow.

Usefulness of grains

Targeting minions

Grains are very useful for targeting minions. Remember in the first article how we targeted the machines using the minion id? We can also target them using grains.

Let's update the configuration file for the salt master to target based on grains instead of minion ids. First let's create a special state file for the fortune program. Remove the line in both cow.sls and locomotive.sls where we tell salt to install the fortune package. Then create a similar state file called fortune.sls:



install fortuneteller tools:
  pkg.installed:
    - pkgs:
      - fortune

Alright. Now we can change the top.sls file to target based on grains instead of minion ids. The pattern for it is to say key:value. Then you can, before you list what state files to include, begin by telling salt what you want to match with your pattern (in our case the grain).

So to apply the apache.sls file to machines with the role webserver you would write something like this:



'roles:webserver':
  - match: grain
  - apache

Notice the ' around roles:webserver. This is so that the YAML file gets processed correctly. There is a shorthand version to achieve the same. You could instead write this:



'G@roles:webserver':
  - apache

Let's modify the top file so we target each role specifically and apply the appropriate state files. The top file could look like this then (I choose the more explicit targeting pattern):



base:
  'roles:cow':
    - match: grain
    - cow
  'roles:fortuneteller':
    - match: grain
    - fortune
  'roles:locomotive':
    - match: grain
    - locomotive

Logic in state files

Grains can also be useful in the state files themselves. For example, we can check the operating system and do different things based on the operating system. You'll see more on this when we go through templating in the Salt files but to give you a taste, let's solve the problem we had in the first article of the series, where the Ubuntu machine installed the packages in /usr/games/.

To solve that problem we'll put an if statement that checks the os grain to see if it is an Ubuntu operating system. If it is, then we make a symbolic link to /usr/bin. We'll need to change all state files because Ubuntu does this with all packages.

Before we change the files, let's first see how we look up the grain value in the state file. The grain we're after to check the operating system is the os grain. It becomes available as grains['os'] (if you know Python, you'll understand that this is just a dictionary lookup).

So the trick is to add a Python if statement inside {% %} then include the state configurations you want before you end with a {% endif %}. I'll cover this better in a later post but to start cow.sls should have this content now:



install cow tools:
  pkg.installed:
    - pkgs:
      - cowsay

{% if grains['os'] == 'Ubuntu' %}
make cowsay available on Ubuntu:
  file.symlink:
    - name: /usr/bin/cowsay
    - target: /usr/games/cowsay
{% endif %}

To dissect this a little bit, first you see the state we defined in the first post of the series and modified a few minutes ago (to remove fortune). Then we create our if statement which checks if the os grain is equal to Ubuntu. If it is, then we create a state called make cowsay available on Ubuntu. This states uses the file state module to create a symlink to /usr/games/cowsay (the target) at /usr/bin/cowsay (defined in name of the symlink). Then we close the if statement.

fortune.sls is going to be similar



install fortuneteller tools:
  pkg.installed:
    - pkgs:
      - fortune

{% if grains['os'] == 'Ubuntu' %}
make fortune available on Ubuntu:
  file.symlink:
    - name: /usr/bin/fortune
    - target: /usr/games/fortune
{% endif %}

So will locomotive.sls



install locomotive tools:
  pkg.installed:
    - pkgs:
      - sl

{% if grains['os'] == 'Ubuntu' %}
make sl available on Ubuntu:
  file.symlink:
    - name: /usr/bin/sl
    - target: /usr/games/sl
{% endif %}

Trying out the grain based configuration

Let's test our new grain based configuration. First let's stop and remove all of our docker containers because we want a fresh slate (this way we'll also get around the new minion ids discussed earlier and easily remove the older containers.

Type in these docker-compose commands to get stop containers, remove them, start up new ones and finally enter bash shell on the salt master (answering yes when prompted):

docker-compose stop
docker-compose rm
docker-compose up -d
docker-compose exec salt bash

Once you've entered the bash shell on the new salt master, accept the minion keys with this command (pressing Enter for yes when prompted):

salt-key -A

Now we're ready to test our grain based targeting in the top.sls. Let's apply the states to all machines (targeting them with the glob *):

salt '*' state.apply

This will install all the programs, based on the roles defined in the grains and create the symlinks on the Ubuntu machines thanks to the if statement in the states. If you look at the output report, you'll see 2 states succeeded in the Alpine container while four changed in the Ubuntu container.

Targeting in the command line

Another place you can use to target based on grains is on the command line. Instead of using the glob or minion id when you run the salt command on the salt master, you can target based on grain by using the -G option.

As an example, let's run the fortune command on all fortuneteller minions (both Ubuntu and Alpine containers). To run a command via the salt command on the salt master you type cmd.run instead of the state.apply we've used earlier and then you tell salt which command to run (in our case fortune). Go ahead and type this in your salt master's bash shell:

salt -G roles:fortuneteller cmd.run fortune

As you can see, we give salt the -G option followed by the grain key and value we want to target (much like we targeted them in the top.sls file. Then we tell salt to run the fortune command with the cmd.run fortune. The output should be some nice words of wisdom:



cow-e54f99c12817:
    Don't feed the bats tonight.
locomotive-68af4abb8a98:
    Put no trust in cryptic comments.

We can also only ask Alpine minions to deliver fortunes:

salt -G os:Alpine cmd.run fortune

This returns only a single wisdom because the grain only matched a single minion:



locomotive-68af4abb8a98:
    The meek shall inherit the earth -- they are too weak to refuse.

OK. That's enough about grains for now, let's quickly turn to the pillar.

Configuring the pillar

As I said previously. The pillar offers the same data structure as grains but instead of being generated and hosted on the salt minions, the pillar data are generated on the salt master and handed to the minion.

That means that one big difference between the pillar and grains is that the pillar needs targeting, so that the pillar data can be handed to the right minion. Besides that, they're just the same.

Let's move the exact programs we need to install for each role into the pillar. That makes it more flexible because we don't have to hard code the programs to install on each minion based on the role. It's going to simplify maintenance and our state code.

Configuring the pillar is a mix between states and grains. We have a top.sls file for the pillar where we do the targeting. There we include files, which should return a data structure similar to grains.

Pillar configuration files

We begin by creating a folder called pillar. This folder we're going to mount on our salt master. This is the folder that will hold our pillar data. Let's first create files for each role. The root element in each file will be packages (could be anything we want but we want it to be explicit right?). Followed by the name of the program as a key with a value which is just a description of the package we install. Something like this:



packages:
  package-name: package description

We won't really use the package description, we're only interested in the package name. However, salt will automatically merge dictionaries in a smart way (which is useful and cool) but unfortunately not append to lists. That's fine though because this gives us a nice place to put a description or comment with an Easter egg.

So first create a file called cow.sls with this content:



packages:
  cowsay: A program with a cow

The second file called fortuneteller.sls with this content:



packages:
  fortune: A program for a fortune teller

Lastly create the third file called locomotive.sls with this content:



packages:
  sl: Tech Model Railroad Club

Finally we need our top.sls file. It's going to be exactly the same as our state file (you can copy it if you want). First we define the environment we want it to be used in, then we target the roles and include the sls files we just created. So go ahead and create the top.sls with this content:



base:
  'roles:cow':
    - match: grain
    - cow
  'roles:fortuneteller':
    - match: grain
    - fortune
  'roles:locomotive':
    - match: grain
    - locomotive

As you can see we can match grains when targeting our pillar. That's pretty cool. We use the roles grain to match the roles.

Be careful, because matching grains in the pillar comes with a caveat. When you use grains to match because they can be configured on the minion itself which means a minion can send a grain to get data it shouldn't get. In our case we're just declaring packages to install, so that's not so harmful.

Configuring the master

Next we'll reconfigure our salt master to read from our newly created pillar directory. For that we have to mount the pillar directory with docker compose and configure the master to read the pillar files from that directory. We begin by changing the master configuration.

We only need to add a new configuration variable called pillar_roots which is similar to the preexisting file_roots. It tells in what folder we keep the pillar files for a particular environment (in our case base).

We will mount the directory with the pillar data at /srv/pillar. Update the master.config file and add the pillar_root and point the base environment to /srv/pillar, so your master.config file should look like this:



file_roots:
  base:
    - /srv/salt/
pillar_roots:
  base:
    - /srv/pillar

Then we need to change our docker-compose.yml file to mount our new pillar directory at /srv/pillar.



version: "3"
services:
  salt:
    image: salt-master
    volumes:
      - ./master.config:/etc/salt/master
      - ./states:/srv/salt
      - ./pillar:/srv/pillar
  alpine-minion:
    image: alpine-minion
    volumes:
      - ./alpine.grains:/etc/salt/grains
  ubuntu-minion:
    image: ubuntu-minion
    volumes:
      - ./ubuntu.grains:/etc/salt/grains

That's it. That's all you need.

Do we have pillar data?

Let's have a look at the pillar data we've created. Again, we want to start with a clean slate so we stop our docker containers and remove them (press y when prompted if we're sure we want to remove them). Then we spin them back up again and attach to the bash shell of salt master:

docker-compose stop
docker-compose rm
docker-compose up -d
docker-compose exec salt bash

Once we're attached to the salt master, we accept the salt keys for the new minions (press enter to accept them when prompted):

salt-key -A

To have a look at the pillar data we just created type the following command

salt '*' pillar.items

You should now see the packages listed based on the roles of the machines. Something like this:



cow-e9e01577bbd8:
    ----------
    packages:
        ----------
        cowsay:
            A program with a cow
        fortune:
            A program for a fortune teller
locomotive-023378ead2d9:
    ----------
    packages:
        ----------
        fortune:
            A program for a fortune teller
        sl:
            Tech Model Railroad Club

We still aren't using this pillar data. To do that properly, we'll have to dive into the next part of this series: templating (I've teased it a little bit in this article though).

Little by little we'll get a better understanding of many of Salt's features.

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

Cover image by Yair Aronshtam: Salt pillar at the Dead Sea, Israel

Let's get started with Salt

Tryggvi Björgvinsson — Tue, 07 Jan 2020 23:12:54 +0000

Configuration management is an essential piece of any computer system consisting of multiple components. If implemented correctly, it can make maintenance a lot easier and predictable for anyone involved.

This becomes especially powerful when the system is managed with machine-readable configuration files instead of using various interactive tools. Spinning up and configuring each component should be automatic and consistent.

SaltStack (also known by the less searchable name Salt) is one such configuration management system. It doesn't only support configuration management, but also remote execution and event driven automation but let's focus on getting started with Salt configuration management.

Simple architectural overview

Before we dive into a small example, let's take a look at a very brief and simplistic overview of how Salt works so you'll know what it's all about.

At its core Salt operates by setting up a small agent, called a minion, on all the system components you want to configure or operate (e.g. server). This minion talks to a Salt master which knows how the minion should be configured based on some configuration files written in YAML.

In the configuration management corner of Salt, these YAML configuration files define the state the component should find itself in. The minion itself then makes sure it is in the desired state by doing some Python-fu.

So basically to get Salt working, all you need to do is to set up a Salt master and define the configuration in a YAML file. Then you just install the minion on the system components you want the Salt master to control.

Example

To see it all in action let's set up a small Docker-based example. I'm fully aware that it's kind of weird to use Salt to configure Docker images because the Dockerfile is essentially a configuration file providing consistency, but Docker just makes it simpler to showcase the simplicity and for you to try out Salt.

We're going to create one Debian image with Salt master and create configuration files to control one Ubuntu image and one Alpine image.

First, let's create our two minions.

Alpine minion

The only thing you have to do really, is to install the Salt minion agent, it will automatically be configured with sane defaults although we'll change them slightly.

Installing Salt minion on Alpine is as easy as just running:

apk add salt-minion

(in Docker we'll add the --no-cache flag because we want to clean up after ourselves)

We have to make one small change to make the configuration files for the states easier. We're going to change something called the minion id.

Normally when the salt minion agent starts up it will by default take the hostname and set it as a minion id. The id is how the Salt master will recognize the different minions.

In the configuration management setup we need to target our minions which will allow us to define different states for different machines. There are many ways to target minions but we're going to target ours using the minion ids.

In Docker the hostnames are going to be a random string (e.g. 71665beda883) which makes predefined targeting difficult. What we're going to do is prefix the minion id with a string that makes it easier for us to target the machine.

For the Alpine example, we're going to use the prefix locomotive. To do that we'll create a docker entrypoint file which echoes "locomotive-$HOSTNAME" into the file where the salt minion stores its id, /etc/salt/minion_id. Afterwards we'll start up the salt minion.

So this is our docker-entrypoint.sh for alpine:

echo "locomotive-$HOSTNAME" > /etc/salt/minion_id
salt-minion

Now all we have to do to create the Alpine Salt minion, is install the Salt minion, copy our docker entrypoint file into the image, and set the command to run the docker entrypoint file.

FROM alpine

RUN apk add --no-cache salt-minion

COPY docker-entrypoint.sh /

CMD ["sh", "/docker-entrypoint.sh"]

Let's build the Docker image as alpine-minion (assuming we're in a folder where we have the alpine Dockerfile):

docker build -t alpine-minion .

Alright, onwards to the next minion.

Ubuntu minion

The Ubuntu minion is going to be almost exactly the same. The difference is going to be that we use apt to install the salt minion instead of apk, and we're going to use the prefix cow instead of locomotive (because we want to target the minions differently).

So the docker-entrypoint.sh is going to be like this:

echo "cow-$HOSTNAME" > /etc/salt/minion_id
salt-minion

The Dockerfile is going to look like this:

FROM ubuntu

RUN apt-get update && \
    apt-get install -y salt-minion

COPY docker-entrypoint.sh /

CMD ["sh", "/docker-entrypoint.sh"]

Let's build this one as ubuntu-minion (again assuming we're now in the ubuntu minion's directory):

docker build -t ubuntu-minion .

That's enough of the minions, let's move on to the states for our minions.

Salt states

You may be wondering why I chose the locomotive and cow prefix for the alpine and ubuntu images, respectively. That's because we're going to install two different software packages based on the prefix.

As I said before, we're going to target the minions (based on their minion id) and install the different packages based on the prefix. When compiling the state for a specific minion, Salt always starts looking in a file called top.sls. The sls file extension stands for SaLt State file, but don't let that fool you, it's just a YAML file. All state definition files should have the sls file extension.

We're going to create three state files. One to install two software packages: cowsay and fortune. One to install software packages sl and fortune (we install fortune in both this way for the example). Lastly we create the top.sls file where Salt starts its journey in.

Make sure a package is installed

Let's start by making sure the packages cowsay and fortune are installed. A Salt state file can contain multiple states, all defined by state names as the root elements of the state file. We're just going to have one in a file called cow.sls to make sure we install cowsay and fortune.

We can call the state what we want, but to be explicit let's call it install fun cow tools. The only thing we have to make sure when giving these names is that the for any given minion, no two states can have the same name.

The state element in the YAML file is a dictionary of the states we want. Salt comes with a lot of built in state modules we can use (that's the beauty of Salt) but the Salt state module we want to use is called pkg.

We use the pkg state module to define that the minions where this state applies need to have packages installed (a function in the pkg module called installed). The pkg.installed function can take in a keyword argument pkgs which for its value is a list of packages to install, in our case: fortune and cowsay.

The cow.sls file will look like this:

install cow tools:
  pkg.installed:
    - pkgs:
      - fortune
      - cowsay

The locomotive file is going to be similar, but instead of installing cowsay, we're going to install sl in addition to fortune. We'll also use the name install locomotive tools for the state. So the file locomotive.sls will look like this:

install locomotive tools:
  pkg.installed:
    - pkgs:
      - fortune
      - sl

As you can see, the files use the same salt modules and functions even if we know that behind the scenes one will be executed on an alpine machine and the other on an ubuntu machine. Salt will use the appropriate package manager.

top.sls

Let's tie this all together in the top.sls and target the machines based on their prefix. Salt allows use of an asterisk/glob when targeting so it's fairly easy to target the minion ids with these prefixes.

First we must define an environment we use. We'll use the default Salt environment called base. That's going to be the root element and the value is a dictionary. The key elements of that dictionary are the targeting patterns. Their values are what state files it should apply. So something like this:

<environment>:
  <target pattern>:
    - <state file>
    - <state file>

The state files are given as the name without the file extension. The state files can be in subdirectories but then we have to use the Python convention where directories and files are separated by a period. So a file cow.sls in the subdirectory farm is included as farm.cow.

Now this may seem complicated but it's really not. It's just hard to describe. If we assume the files are in the same directory as the top.sls file, our top.sls will look like this:

base:
  locomotive-*:
    - locomotive
  cow-*:
    - cow

This means any minion id that starts with locomotive- is going to apply the states found in the locomotive.sls file in the same folder as the top.sls file.

This is the gist of how we build up and define the state of a system with Salt, using YAML files. Now we all we need is the master that can tell the minions what to do.

Salt master

Let's use Debian for the Salt mater. Just like with the minions, The only thing we have to do to get Salt up and running with sane defaults, is to install the salt-master package. So it's going to be a simple Docker image we need (we'll do the magic that we need to make our Salt system operational, when we mount configuration files):

FROM debian:10-slim

RUN apt-get update && \
    apt-get install -y salt-master

CMD salt-master

Let's build this image as salt-master (assuming we're in the same directory as the Salt master Dockerfile):

docker build -t salt-master .

Now with all images we need let's make it operational!

Salt master configuration

First off, we're going to need a basic configuration file that will tell the Salt master where our state files are.

Thanks to Salt's sane defaults, the configuration file is super-simple. We only have to define a file root where our state environment can be found. It is possible to manage a lot of different Salt environments with different states using the same Salt master and configuration file but we're going to stick to the default base environment.

The configuration file just needs to point to the directory with the root of our state files for a base Salt environment. We'll create a file called master.config with only three lines:

file_roots:
  base:
    - /srv/salt/

Our state files (those we created earlier) need to be made available at /srv/salt and we need to mount our master.config in our Salt master as the file /etc/salt/master.

Docker compose environment

To make all of this easier we'll use Docker compose to set up all of our services. Before we create the Docker compose file let's first have a look at how our local file system should look when we've created our Docker compose file (assuming the Dockerfiles we used earlier are stored elsewhere):

salt-example/
  |
  |- docker-compose.yml
  |- master.config
  `- states/
       |
       |- cow.sls
       |- locomotive.sls
       `- top.sls

Now we can create our Docker compose file.

There are a few things we have to make sure we do. Two I have already mentioned: one is to mount the master.config as /etc/salt/master in the Salt master service, the other to mount our states/ directory as /srv/salt/ in the Salt master service.

The third thing I haven't mentioned, but it's very important. By default minions will look for the Salt master at the domain name salt. This is configurable but we are using defaults, so we have to make sure the Salt master service in the Docker compose file is called salt (which makes it available as salt to our minions who share the docker compose network).

So the docker-compose.yml file will define three services, salt (with files and directories mounted properly), our alpine minion and the ubuntu minion:

version: "3"
services:
  salt:
    image: salt-master
    volumes:
      - ./master.config:/etc/salt/master
      - ./states:/srv/salt
  alpine-minion:
    image: alpine-minion
  ubuntu-minion:
    image: ubuntu-minion

That is all. We're ready to start everything up. Just type:

docker-compose up -d

Congratulations! If there haven't been any typos, you now have your first Salt configuration management environment up and running. But it actually doesn't do anything.

Accepting minions

If Salt would just start handing out states to any machine that contacts the Salt master and fits into a specific minion id pattern, we'd have a problem on our hands. To avoid this we need to accept what minions we want the Salt master to manage.

When the minion contacts the Salt master it sends its public key. If we haven't accepted the minion before, we need to accept it (that is to say, if we trust it) using a tool installed on the Salt master called salt-key.

Before we continue, let's open up a shell on the Salt master docker container:

docker-compose exec salt bash

Now in the Salt master bash shell you can list all minion keys that are accepted, denied, rejected and unaccepted, by running salt-key with the -l all option:

salt-key -l all

You should see an output similar to this one:

Accepted Keys:
Denied Keys:
Unaccepted Keys:
cow-7d7f4faf9802
locomotive-1fbb4f5c409d
Rejected Keys:

As you can see there should be two unaccepted keys (one with a cow prefix and another with a locomotive prefix).

If we do not accept them we will not be able to manage their configurations so let's go ahead and accept all the keys we've gotten using the -A flag:

salt-key -A

You'll be asked to confirm if you want to accept the two listed keys. Just enter Y if all looks good (or simply press Enter because accepting is the default). Now you should see something like:

Key for minion cow-7d7f4faf9802 accepted.
Key for minion locomotive-1fbb4f5c409d accepted.

Now we can control our minions. They may have shutdown because it took some time to accept their keys (they don't want to wait forever). Let's just be sure they're up by exiting the Salt master:

exit

Then we run docker compose up again:

docker-compose up -d

Then we can attach to the Salt master again:

docker-compose exec salt bash

If you want to be sure we've accepted our keys and they're still there just type:

salt-key -l all

You should see something like:

Accepted Keys:
cow-7d7f4faf9802
locomotive-1fbb4f5c409d
Denied Keys:
Unaccepted Keys:
Rejected Keys:

Applying state

The states still haven't been applied to our minions. They don't know what packages they should install so they haven't installed anything. To trigger the configuration we will have to execute a state.apply on the minions we want to apply the state.

To do this we target our minions and we can safely just target all of our minions using the glob (asterisk) and tell it to apply the states. To do that we type the following command:

salt '*' state.apply

Notice the quotes around the glob. This is needed because we're passing the string with the targeting pattern.

You should see a lot of stuff on your screen but near the end of it all, you'll see a summary like:

Summary for locomotive-1fbb4f5c409d
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1
Total run time:   5.633 s

This tells you all states were run successfully and one change was made (it installed the packages) on the locomotive machine (Alpine). You'll see a similar report for the cow machine (Ubuntu)

You can run the state.apply function as often as you like. Because it is a state, it defines how your minion should look so what it actually does before installing packages in our case, is to check if all the packages are installed. If they aren't or some are missing, it goes ahead and installs the missing ones.

The state apply does the magic of making sure that the state is fulfilled. Running state.apply again should result in a report for (one for the Alpine machine and another for the Ubuntu machine) like this one:

locomotive-1fbb4f5c409d:
----------
          ID: install locomotive tools
    Function: pkg.installed
      Result: True
     Comment: All specified packages are already installed
     Started: 23:53:42.277146
    Duration: 723.556 ms
     Changes:   

Summary for locomotive-03f0f2574592
------------
Succeeded: 1
Failed:    0
------------
Total states run:     1
Total run time: 723.556 ms

This is awesome. Salt tells us everything is great, but is it? Let's check. First exit the Salt master

exit

Alright now we're ready to check if Salt did what we wanted it to.

Does it work?

The cowsay package in Ubuntu is installed in /usr/games/cowsay so to see it in action we'll run this:

docker-compose exec ubuntu-minion /usr/games/cowsay I think I need some Salt

You'll see a bovine ascii art telling you it needs some Salt. Let's make see if we have fortune installed (also in /usr/games/):

docker-compose exec ubuntu-minion /usr/games/fortune

Enjoy the wisdom!

Alpine should also have some packages (in this case we don't have to access it at /usr/games/ because it is already in the path):

docker-compose exec alpine-minion fortune

Woah! More wisdom! Wait, maybe we installed the cow on Alpine as well:

docker-compose exec alpine-minion cowsay Salt told me not to

Nope. You should get an error telling you that cowsay isn't found in the path. But sl is:

docker-compose exec alpine-minion sl

Here's a joke because you got this far: What sound does a sick train make? -- Ah-Ah-Ah Choo choo!

So these were the basics of Salt. In future posts I want to dive deeper into Salt but I needed to get the basics out of the way.

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

Cover image by Jorge Royan: Four salt shakers on a table with salt spilled from one of them

I wrote a book

Tryggvi Björgvinsson — Wed, 06 Jun 2018 00:15:02 +0000

Yes. I wrote a book. It hasn't been printed yet but now that I have sent it to my publisher I have time to kill. So I thought: Why not reflect on the last one and a half year I spent writing the book and share my experience. If you want to write a book you maybe you can get an idea of what the journey is like.

Here's the backstory; One day in September 2016 I got an email from someone claiming to be an assistant acquisitions editor at Manning publications. According to the email they were interested in publishing a book on data quality and wanted to know if I'd be interested in talking to them. My first thought was: "email scam", then after some thought I decided to reply which 1.5 year later ended in a complete book.

User oriented book writing

Manning publications has a really great approach to book writing. I can't compare it to other publisher because this is my first book and first foray in book writing (except for my Ph.D. thesis). Manning's approach is very user oriented and the whole process is designed to output a book that suits all kind of readers.

First thing I had to do after the initial call with Manning with was define my intended audience (data scientists) and what I wanted to teach that audience in my book. I scoped the book, weighed the difficulty of each chapter to see if the book really matched my audience. It was a kind of a requirement analysis for my book. As I worked on the book I fleshed out the lesson in each chapter, reweighed the chapters, and constantly aimed all my writing at the intended audience.

During the writing process the book was regularly reviewed by a focus group consisting of my intended audience. A group that gave me really valuable feedback and allowed me to better focus my book on what they found helpful and avoid any confusion or boring bits. This was often frustrating because I had to rethink, restructure and rewrite all of my chapters at least twice before I found the best approach to teach data quality without being limited by preconceptions of the intended audience.

Then once I had sent Manning a majority of the chapters my book went into something called Manning early access program or MEAP for short. This program publishes an early release of the book and allows the general public to provide feedback and help me further improve the book for all possible readers. As part of the MEAP Manning put up a book forum where I could converse with readers and adapt the book accordingly but I found private conversations with readers outside the forum most helpful.

Manning provided me with a development editor and a technical editor who helped me frame the book and keep me on track. My development editor was awesome and patient while I internalized Manning's learning system -- the way they have found out helps people learn best. That way I adapted my style to how my readers would read my book. I think this shift in how I write and structure my content was the most difficult thing to overcome. I had written a lot of text before, but unknowingly I wrote only for me and people like me. Obviously, I'm not everybody and Manning had a way to structure book to fit a wider audience.

For example I didn't like repeating explanations in each chapter but what I didn't realize is that while I read books from page 1 to the last page. Others fly into chapters in random order, based on what interests them. If I had never repeated things I said in previous chapters those readers would miss context and not get the same experience. So the trick was to write so that the book was approachable to both groups.

Given my experience in data quality and usability, which is all about understanding what users need in addition to software development which is also about understanding user requirements, it's almost embarrassing to admit how hard it was for me to realize this.

Balancing work, life, and writing

The second most difficult part of the book writing process was fitting the book writing into my life. I'm a father of two daughters. When I started writing the book my two daughters were 2 and 4 years old. My wife was finishing her Ph.D. and had recently started a full-time job. I had a full-time job as the head of IT and dissemination at Statistics Iceland. We had also planned our summer vacation trip in 2017. That plus all of my other interests had to be balanced somehow in addition to all the crazy things I decided to do also, like buying a new apartment and moving, facing a hard disk breakdown and restoring from backup to a new laptop, participating in the parent association of my daughters' preschool, undertaking a major strategy shift at work and whatnot. In retrospect I don't know what I was thinking.

So how did I write the book? First thing: You know how authors tend to thank their family and spouses for the support. I see that in a new light. It's not something they do out of courtesy to their spouses. If you have a family you need to have a supporting spouse. We had to plan how to split the weekends, who got to work in the morning and who got to work in the afternoon (as I said my wife was finishing her Ph.D. on the side). Our family life suffered. We often discussed how awful it was not to see each others for days on end. If one of us was picking up the daughters from preschool, cooking, reading to them, and getting them to bed; the other one was sitting in a coffee shop typing on a laptop.

Coffee shops were great. They became home away from home. A place where I could focus. If I was at home I was easily disturbed which caused me to lose focus. In coffee shops I was rarely disturbed and kept focus for hours. I especially like one coffee shop that was opened until late so I was dose up on coffee during the day and shift into beer in the evening. I wrote the book in a lot of different places that kept my me constantly interested while revising the book.

The writing rhythm I found most suitable was to spend a week or two to research further the chapter I was going to write; find interesting stories, dig into literature, material, create example programs for the chapter and other preparation work. Then take a weekend to bash out surrounding words. I then sent the first rough draft to my awesome development editor who read it and helped me finalize the chapter for user review with lots of back and forth editing and comments. This rhythm didn't always work -- some chapters were more difficult to plan and write than others so they needed more preparation -- but that's the setup that worked best for me and my family.

Git statistics

I wrote all of my book in Asciidoc, a typesetting text-based format to write books. I used Git to version control the book and pushed it to a Gitlab server hosted by Manning. Looking at the Git statistics and Gitlab charts reveals how I wrote the book.

Looking at the whole git repository I changed 1208 files, inserted 113137 lines and deleted 42037 lines (a line is is no more than 80 characters). To put those insertions and deletions into into context, the main chapters of the book are in total 9347 lines (70077 words). Some of the insertions and deletions are things like svg diagrams I created for the book and kept in Git, but it's still pretty clear that a lot of the work was rewriting, deleting, and adding text.

The setup of writing most of the content during the weekend is clear if we look at commits broken up by weekday. Majority of the 157 commits (in total) I made on Sundays (almost 40). Those were the big text blocks I hammered out after preparing for a week or longer. The Thursday commits are made up of two types of commits, when I finalized code for the chapters before the weekends or the chapter structure and when I made changes after getting edits from my editor.

Looking at the commit hours it's also pretty clear that I'm a family man working a full-time job (my timezone is UTC+0 so those are my working hours in the graph below). I seem to work best in the evenings. I start committing work around 4PM and continue until 2AM in the night. Then I sleep for a few hours and head into work. The small amount of commits around noon are from when I took the morning shift on weekends.

Was it worth it?

Oh yes it was. Even if I worked late, after my family went to sleep, sacrificing extra curricular activities I learned so lot, not only about how to teach and write, but also about my area of expertise. I had to come up with creative solutions to make my examples simple enough to be useful in a book.

I'm very happy with the outcome. The book is still in MEAP (now when I write these words) so it hasn't been printed yet but all the chapters have been published and it's going through the last review. If you're interested, you can buy it -- The Art of Data Usability on Manning's website (ebook available now, printed book/pbook once it's been published).

If I was able to do this with everything that was going on in my life at the same time, you can too! So if you have an idea for a book somewhere in your stomach there's nothing stopping you from just starting (although I do recommend that you have fewer things going on in your life than I had -- you're more likely to keep your sanity that way).

I hope this post is inspiring, even if it was mostly about me. I'm more than happy to answer any questions you may have in the comments to help you and encourage you to get your book written.