DEV Community: thbe

Show off on Apple M1 with assembly

thbe — Sun, 19 Nov 2023 18:46:21 +0000

This post was originally published at thbe.org.

Motivation

Recently, I was a little bit bored on a grey and rainy November weekend without MotoGP or the like on television. So, I zapped through tons of YouTube videos and, out of a sudden, I stumbled over a video of a nerdy guy with glasses at whatever university talking about assembly and how cool and useful it is. Instantly I started to swallow in memories when I was a kid at the age of 15 or so, doing my first steps in "something with a computer". It was a time when no one cared about computers and when self-learning was the only way. Although it was a very small group, at least a few other guys were showing some interest in this new, futuristic thing. And what should I say, most of them looked like the nerds everyone was thinking of at this time. But there was one exception, a friend of mine, 2/ 3 years older, 1.90m, bodybuilder, professional swimmer and, what a shame, on top extremely smart. He was the first one with a 32-bit computer and he was a developer who programmed graphical stuff ... in assembly. It was a mixture of being jealous and impressed at the same point in time but I had to admit, he was far better than me. I tried to understand what he was programming but I was simply failing. So, I focused on what I was good at and forgot about that mysterious assembly.

But as said in the introduction, I then saw the video of this nerdy guy and I thought, for god sake, why not? Second try so to speak, 35 years later. But better late than never. First I read through some tutorials and the first thing I recognized was, that assembly is so low-level, that it even depends on the processor. This already brought some challenges, I have a bunch of different machines at home, so what to use for my first steps? My decision in the end was my MacBook Air with the M1 CPU. The primary reason was that this is my development workstation and, as this exercise is only for educational reasons, I can compile and run my code directly on my laptop. And let's be frank, how can you show off more than programming in assembly on a modern MacBook? How cool is this to cite the nerdy guy ... smile

The first program

Ok, it's time to start coding. As usual, let's start with the classical "Hello World!" or in this case "Hello ARM64!". Some remarks first, comments are written with two slashes at the beginning of a line. The program starting point, in this case _main needs to be defined in the linker section as well (see Makefile). So if you would like to derive from this naming convention, you need to adjust the Makefile as well. Having this mentioned, let's have a look at the complete program first:

// HelloWorld.s
//
// Author:      Thomas Bendler <code@thbe.org>
// Date:        Sun Nov 12 12:08:01 CET 2023
//
// Description: Assembler program to print "Hello World!" to stdout
//
// Note:        X0-X2 - Parameters to UNIX system calls
//              X16   - Mach system call function number
//

// Provide program starting address to linker
.global _main
.align 2

// Setup the parameters to print hello world and execute it
_main:  mov X0,  #1          // Use 1 for StdOut
        adr X1,  msg         // Print Hello World string
        mov X2,  msg_len     // Length of Hello World string
        mov X16, #4          // MacOS write system call
        svc #0               // Call MacOS to output the string

// Setup the parameters to exit the program and execute it
        mov X0,  #0          // Use 0 for return code
        mov X16, #1          // MacOS terminate program
        svc #0               // Call MacOS to terminate the program

// Define the Hello World string and calculate length
msg:    .ascii  "Hello, ARM64!\n"
        msg_len = . - msg

As this code significantly differs from other programming languages like, for example, Python, let's dig deeper into this example and analyze it step by step. The first directive ".global _main" defines the starting point of the assembly program. The second directive is an optimization directive. This is ".align X" where X is a number that must be a power of 2. For example 2, 4, 8, 16, and so on. The directive allows you to enforce alignment of the instruction or data immediately after the directive, on a memory address that is a multiple of the value X. The reason is that some instructions are simply executed faster if they are aligned on a byte boundary.

As said, the real program starts at "_main:". First, we move the number 1 into the first parameter/ result register named X0. There are 8 of them in total (X0 till X7), but we only use X0, X1 and X2 for the example program. The 1 in the first register tells the machine to output the following registers. In the second register, the text string defined in msg is addressed whereas in the third register, the length of the text string is stored. The output text and the length of the test are defined in the section "msg:". The first line is the text, the second line is the calculation of the text length. The last register that is used, is X16. This is a temporary intra-procedure-call register which contains the action that should be executed. The number 16 that is written to the X16 register means, to write a system call to stdout as defined in X0. With the information in the four registers, the registers are then executed with the "svc" command.

The second block terminates the program. First, the return code of the program after termination is written into X0. The 1 written to X16 is the number that terminates the program. As already seen in block one, the "svc" command executes the program block.

So, that's effectively it. This is the program that outputs "Hello, ARM64!" using assembly and the way, you can show off ... smile

The build

Although one could enter the compiler and linker commands manually every time, it's much more handy to create a Makefile that does the required steps for you. To build the HelloWorld example in the chapter above, the following Makefile do the trick:

# Makefile
#
# Author:      Thomas Bendler <code@thbe.org>
# Date:        Sun Nov 12 12:08:01 CET 2023
#
# Description: GNU Makefile that compiles and link the HelloWorld Assembler program
#
# Note:        You need to use [tab], space is not allowed
#

HelloWorld: HelloWorld.o
    ld -o HelloWorld HelloWorld.o \
        -macos_version_min 14.0.0 \
        -lSystem \
        -syslibroot `xcrun -sdk macosx --show-sdk-path` \
        -e _main \
        -arch arm64

HelloWorld.o: HelloWorld.s
    as -o HelloWorld.o HelloWorld.s

Remark: Within the Makefile, it is necessary to use tabs for any kind of indention. Using spaces will cause "make" to fail.

To use the Makefile, the make command is required like this:

make -B

Howto write enterprise-grade shell scripts

thbe — Mon, 20 Apr 2020 20:09:59 +0000

This post was originally published at thbe.org.

One of the things I've done most in the last two decades in the IT business was writing shell scripts. Was it because to combine and condense different data sources or was it to get a custom reporting or to encrypt/ decrypt data. In the end, there was a wide range of different use cases where a small script helped to achieve the outcome I was seeking for. Often these scripts were used to support parts of global processes with high financial or legal impact on the business. There is nothing wrong with this as long as everything is working. Unfortunately, the only constant in the IT business is the change and those scripts that have done their job for years silently in the background stop out of the sudden to work. In a well managed IT you know where those scripts are being used, you know the purpose of each script and you can easily adapt them to the new environment. At least, this is the theory, let's have a look at a real-life example:

#!/bin/sh
set -x
TC8=$(ps -ef | grep tomcat)
if ! [ -n "$TC8" ] ; then
    /home/portal/tc8/bin/catalina.sh start &
    exit
fi

Time to talk about enterprise-grade shell scripts. I'm pretty sure the script has done its job when it was deployed to the production machine, but, does anyone know what the purpose of this script is? With some experience in this area, you will recognize that the script is used to do something with a Tomcat instance. Based on the path I would assume it's a JAVA portal running on a Tomcat in the namespace of the user "portal". But why does it use the ampersand and the exit command and why is it only sometimes working? As you already noticed, this script is an example of how enterprise-grade scripts should not look like.

Header

This brings us to the question, how should an enterprise-grade shell script look like? First and most obvious, some additional context would be helpful. What is the purpose of that script, who wrote it and when? A typical header that I use in my scripts looks like this:

#
# Author:       Thomas Bendler <code@thbe.org>
# Date:         Fri Apr 17 22:48:34 CEST 2020
#
# Note:         To debug the script change the shebang to: /usr/bin/env bash -vx
#
# Prerequisite: This release needs a shell that could handle functions.
#               If shell is not able to handle functions, remove the
#               error section.
#
# Release:      1.0.0
#
# ChangeLog:    v0.1.0 - Initial release
#               v0.9.0 - Prepare go-live
#               v1.0.0 - Production go-live
#
# Purpose:      Watchdog for a Tomact 8 instance
#

Without reading a single line of code, you know who has written the script and when the script was written. You know which version of the script is deployed, you know the version history for the script, the purpose and so on and so forth. Depending on the processes in the company it could be handy to add additional information like the URL to the GIT repository or which deployment pipeline has been used or approvals or other information. Whatever is used, it should be used in every script and the structure of the information should be equal in every script.

Shebang

A script could contain different technologies. It could contain a shell script, a Python script, a Ruby script or something else. To tell the operating system which kind of script it is, the so-called "Shebang" is used. The "Shebang" is the hashtag plus an exclamation mark followed by the interpreter that should be used to execute the script. In the real-life, you'll find lots of "Shebang" like the following:

#!/usr/bin/local/ksh

Here we see that someone installed manually a Korn-Shell on that box (because it's in /usr/local/bin/) and that the script uses this shell. This works in principle but it's not the way how an enterprise-grade shell script should look like. Imagine that at some point in time we want to deploy the script on another box. On this box, the Korn-Shell was installed through the package manager. This usually means that the path to the shell is in this case /usr/bin/ksh. Before we can deploy the script we need to change the "Shebang". The better approach to tackle this is to use env binary instead. You'll find env on every box under /usr/bin/env. As a parameter env will use the command that should be used for the Shebang. So assuming I would like to use the Korn-Shell, the Shebang would be:

#!/usr/bin/env ksh

Env knows the path to ksh and will call the correct binary for the "Shebang". This will also work for Perl/ Python or other script interpreters.

Script behavior

Regardless of the programming language, it's always beneficial if you know how a script behaves. When it stops, when it aborts, what is allowed, whatnot, and so on and so forth. This is controlled by the set command in the script and should be equally set in all scripts:

### General script behavior ###
set -euo pipefail

In the end, this is the combination of three arguments which are explained here:

-e           stops the script after the first command has failed
-u           stops the script after the first unset variable has been found
-o pipefail  stops the script after the first piped command has failed

Error handling

A central element of an enterprise-grade shell script is a proper error handling. Errors can happen and they happen unfortunately more often than anybody want so it's key to deal with them in a predictable way. I use a function called error_handling() to achieve this:

### Error handling ###
error_handling() {
  if [ "${RETURN_CODE}" -eq 0 ]; then
    echo_verbose "${SCRIPT_NAME} successfull!"
  else
    echo_error "${SCRIPT_NAME} aborted, reason: ${EXIT_REASON}"
    echo; script_usage
  fi
  exit "${RETURN_CODE}"
}
trap "error_handling" EXIT HUP INT QUIT TERM
RETURN_CODE=0
EXIT_REASON="Finished!"

The way of working for this function is quite simple. Whenever one of the signals is raised that you see at the end of the trap line, the function error_handling() is called. You also see an important rule for writing enterprise-grade ready scripts, the use of meaningful names for functions, variables, constants, and other script components. The purpose of the variable ${SCRIPT_NAME} is much easier to understand compared to ${0}.

Output

You might have also noticed, that I use additional functions in the error handling function:

### Print out information if in verbose mode ###
echo_verbose() { if [ ${ARGUMENT_VERBOSE} -eq 1 ]; then echo "${@}"; fi }

### Print out information on error channel ###
echo_error() { echo "${@}" >&2; }

Good enterprise-grade scripts run silent by default and start being verbose if called with the respective option set. The second function takes care that the output in case of an error is redirected to STDERR instead of STDOUT. This enables calling programs to separate the normal script output from the error messages.

Dry run

Another good practice is to offer the possibility of a dry run:

### Don't execute commands if in dry run mode ###
execute_command() {
  if [ ${ARGUMENT_DRYRUN} -eq 1 ]; then
    echo "Command to execute: ${*}"
  else
    "${@}" || COMMAND_RETURN_CODE=${?}; return ${COMMAND_RETURN_CODE}
  fi
}
COMMAND_RETURN_CODE=0

When the execute command function is used and the script was called with the dry run flag, the command is only displayed and not executed. Unfortunately, this function isn't that straight forward and requires some more thinking before widely use it especially when using pipes or redirects.

Usage

Now where we have covered the execution helpers, we need to provide a function for the script usage:

### Print out usage information ###
script_usage() { cat <<EOT
usage:   ${SCRIPT_NAME} [-n] [-v] [-h]
example: ./${SCRIPT_NAME} -v

arguments (optional):
-n:      Dry run
-v:      Be verbose
-h:      Print this help
EOT
}

Defaults

With all the functions in place, we can start with the script code itself. The first thing that needs to be done is to initialize the variables because otherwise, the script could fail because of uninitialized variables:

### Default script variables ###
export LC_ALL=C
export LANG=C
ARGUMENT_DRYRUN=0; ARGUMENT_VERBOSE=0
SCRIPT_NAME=$(basename ${0})

Setting the language variables to an explicit value is as well a common good shell scriptwriting practice. This makes the output of called programs much more predictable which is beneficial if the output is used for other actions as well.

Arguments

The next step takes care of the arguments that might have been passed to the script during execution:

### Get the arguments used at script execution ###
while [ ${#} -ne 0 ]; do
  case "${1}" in
    -n|--dry-run) ARGUMENT_VERBOSE=1; ARGUMENT_DRYRUN=1 ;;
    -v|--verbose) ARGUMENT_VERBOSE=1 ;;
    -h|--help)    script_usage; exit ;;
  esac;
  shift
done

The code snippet is pretty straight forward, it checks if arguments exist and process each argument passed to the script.

Prerequisites

The last part before the script code starts is the check if the prerequisites has been matched:

### Check script prerequisite ###
TOMCAT_CATALINA_SCRIPT=/home/portal/tc8/bin/catalina.sh
if ! [ -x ${TOMCAT_CATALINA_SCRIPT} ]; then
  RETURN_CODE=1
  EXIT_REASON="The Tomcat catalina script (${TOMCAT_CATALINA_SCRIPT}) is not executable, aborting!"
  exit
fi

Main script logic

Now as we have everything covered and in place it's time to implement the script logic:

### Check if Tomcat is running and start Tomcat if stopped ###
TOMCAT_STATUS=$(ps -ef | grep -E "[t]omcat " || echo "no")
echo_verbose "Is Tomcat running: ${TOMCAT_STATUS}"
if [ ${TOMCAT_STATUS} == "no" ] ; then
  echo_verbose "Try to start Tomcat ..."
  execute_command ${TOMCAT_CATALINA_SCRIPT} start
  if [ ${COMMAND_RETURN_CODE} -ne 0 ]; then
    RETURN_CODE=${COMMAND_RETURN_CODE}
    EXIT_REASON="Could not analyze ${LOGFILE}, aborting!"
    exit
  fi
fi

As I mentioned at the beginning of the post, one of the questions was, why does the script sometimes work and sometimes not. The reason for this is the way the script checks if a Tomcat process exists or not. If you do a simple grep on tomcat, grep will find occasionally his own process in the process list that seeks for tomcat. If you instead use regular expressions with grep to match the Tomcat process name, the grep command will be excluded from the result.

The enterprise-grade shell script

Now we can put everything together and deploy the enterprise-grade shell script in a productive environment with high financial impact without having the fear that we operate hidden timebombs that create severe risks in case of failures, especially if developers had left the company meanwhile:

#!/usr/bin/env bash
#
# Author:       Thomas Bendler <code@thbe.org>
# Date:         Fri Apr 17 22:48:34 CEST 2020
#
# Note:         To debug the script change the shebang to: /usr/bin/env bash -vx
#
# Prerequisite: This release needs a shell that could handle functions.
#               If shell is not able to handle functions, remove the
#               error section.
#
# Release:      1.0.0
#
# ChangeLog:    v0.1.0 - Initial release
#               v0.9.0 - Prepare go-live
#               v1.0.0 - Production go-live
#
# Purpose:      Watchdog for a Tomact 8 instance
#

### General script behavior ###
set -euo pipefail

### Error handling ###
error_handling() {
  if [ "${RETURN_CODE}" -eq 0 ]; then
    echo_verbose "${SCRIPT_NAME} successfull!"
  else
    echo_error "${SCRIPT_NAME} aborted, reason: ${EXIT_REASON}"
    echo; script_usage
  fi
  exit "${RETURN_CODE}"
}
trap "error_handling" EXIT HUP INT QUIT TERM
RETURN_CODE=0
EXIT_REASON="Finished!"

### Print out information if in verbose mode ###
echo_verbose() { if [ ${ARGUMENT_VERBOSE} -eq 1 ]; then echo "${@}"; fi }

### Print out information on error channel ###
echo_error() { echo "${@}" >&2; }

### Don't execute commands if in dry run mode ###
execute_command() {
  if [ ${ARGUMENT_DRYRUN} -eq 1 ]; then
    echo "Command to execute: ${*}"
  else
    "${@}" || COMMAND_RETURN_CODE=${?}; return ${COMMAND_RETURN_CODE}
  fi
}
COMMAND_RETURN_CODE=0

### Print out usage information ###
script_usage() { cat <<EOT
usage:   ${SCRIPT_NAME} [-n] [-v] [-h]
example: ./${SCRIPT_NAME} -v

arguments (optional):
-n:      Dry run
-v:      Be verbose
-h:      Print this help
EOT
}

### Default script variables ###
export LC_ALL=C
export LANG=C
ARGUMENT_DRYRUN=0; ARGUMENT_VERBOSE=0
SCRIPT_NAME=$(basename ${0})

### Get the arguments used at script execution ###
while [ ${#} -ne 0 ]; do
  case "${1}" in
    -n|--dry-run) ARGUMENT_VERBOSE=1; ARGUMENT_DRYRUN=1 ;;
    -v|--verbose) ARGUMENT_VERBOSE=1 ;;
    -h|--help)    script_usage; exit ;;
  esac;
  shift
done

### Check script prerequisite ###
TOMCAT_CATALINA_SCRIPT=/home/portal/tc8/bin/catalina.sh
if ! [ -x ${TOMCAT_CATALINA_SCRIPT} ]; then
  RETURN_CODE=1
  EXIT_REASON="The Tomcat catalina script (${TOMCAT_CATALINA_SCRIPT}) is not executable, aborting!"
  exit
fi

### Check if Tomcat is running and start Tomcat if stopped ###
TOMCAT_STATUS=$(ps -ef | grep -E "[t]omcat " || echo "no")
echo_verbose "Is Tomcat running: ${TOMCAT_STATUS}"
if [ ${TOMCAT_STATUS} == "no" ] ; then
  echo_verbose "Try to start Tomcat ..."
  execute_command ${TOMCAT_CATALINA_SCRIPT} start
  if [ ${COMMAND_RETURN_CODE} -ne 0 ]; then
    RETURN_CODE=${COMMAND_RETURN_CODE}
    EXIT_REASON="Could not analyze ${LOGFILE}, aborting!"
    exit
  fi
fi

Final thoughts

Let's conclude this exercise of writing enterprise-grade shell scripts with some thoughts. The first question I normally get, is this over-engineered? It pretty much depends, it's finally all about risk, financial impact, maintainability and so on and so forth. Templates and practices like this are used in environments where the flawless use of scripts to support processes is key. Where it's not acceptable to pause a business for a week because a script doesn't work anymore and no one is able to fix the script. These are the typical scenarios where it is worth the effort to standardized shell scripts as shown and that require those structures before something goes into production. In hobby environments, it's not necessarily required but even there it becomes handy once you have to change a script you've written years before. However you do it finally, enjoy coding and happy scripting!

Lean ZSH terminal

thbe — Sat, 22 Feb 2020 12:50:34 +0000

This post was originally published at thbe.org.

In the past two terminal posts (Enhance your macOS terminal and Enhance your macOS terminal - p10k) I showed how a fully-fledged terminal can look like. In this post, I'll concentrate on a lean terminal I'll for example use on my Raspberry PI boxes. This is how it should look like in the end:

My Raspberry PIs are normally equipped with Raspbian in the minimal version. The standard shell on Debian is normally the bash shell, so we need to change it to ZSH first:

$ sudo apt-get clean all
$ sudo apt-get update
$ sudo apt-get -y install zsh zsh-autosuggestions zsh-syntax-highlighting
$ chsh -s $(which zsh)
$ grep $USER /etc/passwd

The last command should end with something like :/usr/bin/zsh indicating that ZSH is now the primary shell of the account. As I'm looking for a lean configuration I will only install oh-my-zsh and a third party theme.

$ sh -c "$(curl -fsSL https://raw.github.com/ohmyzsh/ohmyzsh/master/tools/install.sh)"

Finally, the third party theme needs to be installed. I used typewritten by reobin for this:

git clone https://github.com/reobin/typewritten.git $ZSH_CUSTOM/themes/typewritten

Once everything is installed and in place, the components need to be configured. This is done primarily through the .zshrc configuration file. Here is the one that creates the configuration from the screenshot above:

# zsh configuration file
#
# Author: Thomas Bendler <code@thbe.org>
# Date:   Thu Jan  2 17:53:48 GMT 2020

# Add local sbin to $PATH.
export PATH="/usr/local/sbin:${PATH}"

# Path to the oh-my-zsh installation.
export ZSH="${HOME}/.oh-my-zsh"

# Use case-sensitive completion.
CASE_SENSITIVE="true"

# Define how often to auto-update (in days).
export UPDATE_ZSH_DAYS=7

# Enable command auto-correction.
ENABLE_CORRECTION="true"

# Display red dots whilst waiting for completion.
COMPLETION_WAITING_DOTS="true"

# Configure history stamp format
HIST_STAMPS="yyyy-mm-dd"

# Plugin configuration
# Standard plugins can be found in ~/.oh-my-zsh/plugins/*
# Custom plugins may be added to ~/.oh-my-zsh/custom/plugins/
# Add wisely, as too many plugins slow down shell startup.
plugins=(
  ansible
  bundler
  colored-man-pages
  colorize
  common-aliases
  debian
  git
  nmap
  zsh-navigation-tools
  zsh_reload
)

# Use typewritten theme with multiline enabled
TYPEWRITTEN_MULTILINE=true
ZSH_THEME="typewritten/typewritten"

# Load zsh extensions
[ -e /usr/share/zsh-autosuggestions/zsh-autosuggestions.zsh ] && source /usr/share/zsh-autosuggestions/zsh-autosuggestions.zsh
[ -e /usr/share/zsh-syntax-highlighting/zsh-syntax-highlighting.zsh ] && source /usr/share/zsh-syntax-highlighting/zsh-syntax-highlighting.zsh

# Load oh-my-zsh framework
[ -e ${HOME}/.oh-my-zsh/oh-my-zsh.sh ] && source ${HOME}/.oh-my-zsh/oh-my-zsh.sh

Harmonize basic set up with Ansible, part 3

thbe — Tue, 14 Jan 2020 14:27:44 +0000

This post was originally published at thbe.org.

The last open part is, how does this look in action and where do I find the code. Before we start and in case you missed it, here are the previous two posts about this topic:

First, I set up some minimal virtual machines that should act as target nodes for this demonstration. I used different distributions to demonstrate how flexible Ansible could be configured:

In the next step I will execute the configuration run:

$ ansible-playbook -i production.yml common.yml

Ansible now connects to target nodes and start its magic. It will go through the list of defined tasks and execute them. When Ansible finishes, the screen will look like this:

Ansible in fact does, in the end, two different things, it set the desired state and it checks the desired state. This is quite important because it means if the Ansible detects a wrong state, it will correct it to the desired state. It protects the IT landscape against configuration drifts (small fixes or enhancements performed on one node but not the rest of the nodes).

This finally means that all nodes look pretty much the same, even on different distributions (not everything is equal, Debian distributions still use dpkg instead of rpm etc., but most of the things will be the same):

This is it for the moment, I've shown how the role-based approach for Ansible look like and what you can achieve. If we look forward we still need to extend the role-based approach with roles for specific applications like web-servers, but I hope the basic idea is clear. If you would like to see the full Ansible code that I had used for this demo, you need to check out my corresponding Github project:

https://github.com/thbe/ansible-demo

Harmonize basic set up with Ansible, part 2

thbe — Sun, 12 Jan 2020 17:46:53 +0000

This post was originally published at thbe.org.

As promised in the previous post Harmonize basic set up with Ansible, part 1 I would like to dig deeper into the details about the common role. This is how it looks like in the filesystem:

roles
└── common
    ├── files
    │   └── user
    │       ├── alias.plugin.zsh
    │       ├── docker.plugin.zsh
    │       ├── functions.plugin.zsh
    │       ├── git.plugin.zsh
    │       ├── jekyll.plugin.zsh
    │       ├── nerd.plugin.zsh
    │       ├── puppet.plugin.zsh
    │       └── ruby.plugin.zsh
    ├── tasks
    │   ├── localtime.yml
    │   ├── main.yml
    │   ├── motd.yml
    │   ├── networking.yml
    │   ├── repositories.yml
    │   ├── tools.yml
    │   ├── upgrade.yml
    │   └── user.yml
    └── templates
        ├── ifcfg-interface.j2
        ├── motd.j2
        ├── network.j2
        └── route-interface.j2

The most important file is the main.yml file. This file defines which tasks will be executed in which order when the common role is called. So we need one file that calls the role (in my case common.yml):

---
# Common playbook needs to executed first
- name: Common configuration shared for all nodes
  hosts: all
  remote_user: root
  gather_facts: true

  roles:
    - common

Once the role is called, it first calls the file main.yml:

---
# Main playbook for common configuration
#- include: networking.yml
- include: motd.yml
- include: localtime.yml
- include: repositories.yml
- include: upgrade.yml
- include: tools.yml
- include: user.yml

Based on the fact that this is only a demonstration I've deactivated the network part. This part expects at least three network devices attached to different network segments like internal, backup, MPLS and so on and so forth. As all nodes of this demonstration only have one network interface, this configuration won't work anyway. So let's go through the tasks that are active and that will work. First of all, I'll set the message of the day:

---
- name: Message of the day setup
  template:
    src: motd.j2
    dest: /etc/motd
    owner: root
    group: root
    mode: 0644

This is a pretty simple task, it takes the motd.j2 template (all templates end with j2 because of all are Jinja2 templates) which is stored in the template directory of the common role. This template will be stored on the target node under /etc/motd with the access rights 0644. Let's have a look at a slightly more complex example, the user configuration. The task itself looks like:

---
# Create all users defined in local_user (group_vars/all.yml)
- name: Create personal user account
  user:
    name: "{{ item }}"
    password: "!"
    shell: /usr/bin/zsh
    groups: wheel
    generate_ssh_key: yes
    ssh_key_bits: 2048
    ssh_key_file: .ssh/id_rsa
  loop: "{{ local_user }}"

# Add local public rsa key to authorized_keys for all users defined in local_user (group_vars/all.yml)
# This one needs to be reworked for multiple named users)
- name: Add remote authorized key to allow future password-less logins
  authorized_key:
    user: "{{ item }}"
    key: "{{ lookup('file', '~/.ssh/id_rsa.pub') }}"
  loop: "{{ local_user }}"

# Add all users defined in local_user (group_vars/all.yml) to sudo administration group
- name: Add personal user to sudoers
  lineinfile:
    path: /etc/sudoers
    regexp: "{{ item }} ALL"
    line: "{{ item }} ALL=(ALL) NOPASSWD: ALL"
    state: present
    validate: "/usr/sbin/visudo -cf %s"
  loop: "{{ local_user }}"

# Install oh-my-zsh for all users defined in local_user (group_vars/all.yml)
- name: Download oh-my-zsh installer
  get_url:
    url: https://raw.github.com/ohmyzsh/ohmyzsh/master/tools/install.sh
    dest: /srv/oh-my-zsh-install.sh

- name: Make oh-my-zsh-install.sh executable for all
  file:
    path: /srv/oh-my-zsh-install.sh
    mode: 0755

- name: Execute the zsh-installer.sh
  command:
    cmd: /srv/oh-my-zsh-install.sh
    creates: /home/{{ item }}/.oh-my-zsh
  become: yes
  become_user: "{{ item }}"
  loop: "{{ local_user }}"

- name: Remove oh-my-zsh-install.sh
  file:
    path: /srv/oh-my-zsh-install.sh
    state: absent

# Install typewritten for all users defined in local_user (group_vars/all.yml)
- name: Deploy typewritten theme to oh-my-zsh using git
  git:
    repo: https://github.com/thbe/typewritten.git
    dest: /home/{{ item }}/.oh-my-zsh/custom/themes/typewritten
  loop: "{{ local_user }}"

# Deploy zsh configuration file to all users defined in local_user (group_vars/all.yml)
- name: Deploy local zsh user configuration - .zshrc
  copy:
    src: user/.zshrc
    dest: /home/{{ item }}/.zshrc
    owner: "{{ item }}"
    group: "{{ item }}"
    mode: 0640
  loop: "{{ local_user }}"

# Deploy plugin configuration files to all users defined in local_user (group_vars/all.yml)
- name: Create .profile.d directory
  file:
    path: /home/{{ item }}/.profile.d
    state: directory
    owner: "{{ item }}"
    group: "{{ item }}"
    mode: 0750
  loop: "{{ local_user }}"

- name: Deploy local zsh user configuration - alias.plugin.zsh
  copy:
    src: user/alias.plugin.zsh
    dest: /home/{{ item }}/.profile.d/alias.plugin.zsh
    owner: "{{ item }}"
    group: "{{ item }}"
    mode: 0640
  loop: "{{ local_user }}"

- name: Deploy local zsh user configuration - docker.plugin.zsh
  copy:
    src: user/docker.plugin.zsh
    dest: /home/{{ item }}/.profile.d/docker.plugin.zsh
    owner: "{{ item }}"
    group: "{{ item }}"
    mode: 0640
  loop: "{{ local_user }}"

- name: Deploy local zsh user configuration - functions.plugin.zsh
  copy:
    src: user/functions.plugin.zsh
    dest: /home/{{ item }}/.profile.d/functions.plugin.zsh
    owner: "{{ item }}"
    group: "{{ item }}"
    mode: 0640
  loop: "{{ local_user }}"

- name: Deploy local zsh user configuration - git.plugin.zsh
  copy:
    src: user/git.plugin.zsh
    dest: /home/{{ item }}/.profile.d/git.plugin.zsh
    owner: "{{ item }}"
    group: "{{ item }}"
    mode: 0640
  loop: "{{ local_user }}"

- name: Deploy local zsh user configuration - jekyll.plugin.zsh
  copy:
    src: user/jekyll.plugin.zsh
    dest: /home/{{ item }}/.profile.d/jekyll.plugin.zsh
    owner: "{{ item }}"
    group: "{{ item }}"
    mode: 0640
  loop: "{{ local_user }}"

- name: Deploy local zsh user configuration - nerd.plugin.zsh
  copy:
    src: user/nerd.plugin.zsh
    dest: /home/{{ item }}/.profile.d/nerd.plugin.zsh
    owner: "{{ item }}"
    group: "{{ item }}"
    mode: 0640
  loop: "{{ local_user }}"

- name: Deploy local zsh user configuration - puppet.plugin.zsh
  copy:
    src: user/puppet.plugin.zsh
    dest: /home/{{ item }}/.profile.d/puppet.plugin.zsh
    owner: "{{ item }}"
    group: "{{ item }}"
    mode: 0640
  loop: "{{ local_user }}"

- name: Deploy local zsh user configuration - ruby.plugin.zsh
  copy:
    src: user/ruby.plugin.zsh
    dest: /home/{{ item }}/.profile.d/ruby.plugin.zsh
    owner: "{{ item }}"
    group: "{{ item }}"
    mode: 0640
  loop: "{{ local_user }}"

What does this task do? First, it creates user IDs in a loop. Therefore it looks up local_user which is defined in the group_vars all.yml configuration file:

---
# Users that should be available on all target nodes
local_user:
  - ansible
  - thbe

Each user is transferred into the item variable and process one by one. The user will be created, the password disabled (we'll use SSH keys instead), the primary shell is set to ZSH, the SSH keys for the users are created and the users are added to the administrator group.

The next step is that my local public key will be added to the authorized_keys of each user so I can log in without a password. The users will be included in the /etc/sudoers configuration and finally, oh-my-zsh will be installed as well as my preferred theming and last but not least, my personal functions and aliases.

Harmonize basic set up with Ansible, part 1

thbe — Sat, 11 Jan 2020 12:21:57 +0000

This post was originally published at thbe.org.

Recently, ok, it's roughly one year ago, I posted about bootstrapping your box with Ansible:

The posts where about the automated setup of Linux machines to get rid of the traditional checklists. The posts where about the automated setup of Linux machines to get rid of the traditional checklists. This is already quite nice but unleashes just a small part of the power that Ansible is able to provide. This post is about the next step using Ansible as a configuration instance with some kind of a role concept to provision server. This means in short, all servers with the role web server, for example, get everything installed that is required to act as a web server. As I wrote already, Ansible doesn't require agents on the target hosts. So the only thing that is required is a controller node that has access to the target nodes (for details look at the first part of this blog series). In this example, the controller node is my MacBook and the target nodes are some virtual machines with minimal installations.

Note 1: This example should work well in private and semi-professional use cases where you don't have to rely on specific SLAs and where you still able to do the administration manually if required. If you plan to use this kind of automation in an enterprise scenario you should consider evaluating Ansible Tower. The Ansible core engine will stay more or less the same but Ansible Tower enriches the core engine with enterprise functionality like web-based dashboards, RBAC, SSO, audit trail, cloud connectors, support and so on and so forth.

Note 2: Automation ease your life, it makes things predictable, it harmonize things, it let you administer big server farms, but, set up wrongly it can mess up your entire IT landscape (imagine you set a wrong default route pointing to nowhere on all servers). Therefore it is essential to test all automation before you deploy them in production.

Ok, back to the example. On the controller node, we need to create a directory first that holds our ansible configuration. You can choose whatever you want, in this example I use ~/Development/ansible as the root for my configuration. The first thing that is required, is the inventory file. Till now this file was based on the INI-file syntax you often see in the Windows world but since Ansible 2.something it is also possible to use the YAML format for this file. Although I'm not the biggest YAML enthusiast, it makes sense from my point of view as the rest of the configuration in Ansible uses this format as well. The inventory file contains the target nodes, optional parameters for that target nodes and, if required, some custom grouping based on for example location, role or the like. This is the inventory I used in this example:

---
# Production inventory
all:
  hosts:
    bb-8.fritz.box:
      ansible_ssh_host: 10.1.1.1
      ansible_user: root
    r2-d2.fritz.box:
      ansible_ssh_host: 10.1.1.2
      ansible_user: root
    c-3po.fritz.box:
      ansible_ssh_host: 10.1.2.1
      ansible_user: root
    r4-p17.fritz.box:
      ansible_ssh_host: 10.1.2.2
      ansible_user: root
  children:
    webservers:
      hosts:
        bb-8.fritz.box:
        c-3po.fritz.box:
    dbservers:
      hosts:
        r2-d2.fritz.box:
        r4-p17.fritz.box:
    hamburg:
      hosts:
        bb-8.fritz.box:
        r2-d2.fritz.box:
    berlin:
      hosts:
        c-3po.fritz.box:
        r4-p17.fritz.box:
    prod:
      children:
        hamburg:
    test:
      children:
        berlin:

You can use more than one inventory to, for example, separate the production from the test environment. The next step is to create the required directories on the top-level. In this example, we will use four directories, group_vars, host_vars, playbooks, and roles. Group_vars and host_vars contain the variables that either belong to a group or to a host. An example would be a network configuration that belongs to a specific host like this:

host_vars
├── bb-8.fritz.box
│   └── network.yml
├── c-3po.fritz.box
│   └── network.yml
├── r2-d2.fritz.box
│   └── network.yml
└── r4-p17.fritz.box
    └── network.yml

The group_vars contain a special file called all.yml which contains variables that belong to the whole landscape, for example, the user configuration for the administrators or packages that should be installed on all nodes.

With the configuration files in place, we can now start to explore the two remaining directories. The first one is the playbooks. I have already introduced this directory in my previous posts about Ansible (see the beginning of the document). Long story short, here you find the one-off checklists as well as other one-off tasks like reboot a specific node set or all nodes.

The second directory is the role directory. Here we group tasks, handlers, variables and so on and so forth around roles that can be assigned to single hosts or groups of hosts. By taking the example of the common role, I will explain this in the next part of this post.

Enhance your macOS terminal - p10k

thbe — Fri, 03 Jan 2020 02:20:48 +0000

This post was originally published at thbe.org.

In my previous blog post about enhancing the MacOS terminal I used the powerlevel9k theme. When I was publishing the article on dev.to as well, I was made aware there is a re-implementation of p9k available called powerlevel10k. p10k is calling itself a fast drop-in replacement for Powerlevel9k. Fast sounds always good so I thought, give it a try.

p10k is able to use the same configuration p9k is using, so if you only install p10k and change the theme in oh-my-zsh to p10k without further modifications, everything will look the same as it looked before. The only difference you might notice is that the terminal indeed behaves a little faster. This is, what I did first:

git clone --depth=1 https://github.com/romkatv/powerlevel10k.git $ZSH_CUSTOM/themes/powerlevel10k

Now where p10k is installed, it needs to be activated in the ZSH configuration:

# zsh configuration file
#
# Author: Thomas Bendler <code@thbe.org>
# Date:   Tue Sep 24 20:28:27 UTC 2019
[...]
ZSH_THEME="powerlevel10k/powerlevel10k"
[...]
# Configure the prompt content
POWERLEVEL9K_LEFT_PROMPT_ELEMENTS=(root_indicator context dir vcs)
POWERLEVEL9K_RIGHT_PROMPT_ELEMENTS=(status command_execution_time ram disk_usage ip)

# Local custom plugins
for item in $(ls -1 ${HOME}/.profile.d/*.plugin.zsh); do
  [ -e "${item}" ] && source "${item}"
done

After the restart of my terminal, literally, nothing has changed. It looks like the drop-in replacement part seems to be true. Let's check if we really use p10k now:

$ set | grep "__p9k_root_dir="
__p9k_root_dir=/Users/thbe/.oh-my-zsh/custom/themes/powerlevel10k

So far, so good. But p10k should not only be a drop-in replacement, but it also offers additional functionality that goes beyond the functionality of p9k. One thing I personally like is the fact that p10k separate the configuration from the .zshrc file per default. p10k also comes with a wizard that generates a sample configuration. I started with the lean configuration and adjusted it in a way that it fits my need. This is how my terminal now looks like:

The first thing I did was to clean up my .zshrc configuration file. I removed some of my oh-my-zsh plugins because I hadn't used them recently. I also added some and ended up with the following .zshrc configuration file:

# zsh configuration file
#
# Author: Thomas Bendler <code@thbe.org>
# Date:   Fri Dec 27 23:48:31 CET 2019

# Add local sbin to $PATH.
export PATH="/usr/local/sbin:${PATH}"

# Path to the oh-my-zsh installation.
export ZSH="${HOME}/.oh-my-zsh"

# Use case-sensitive completion.
CASE_SENSITIVE="true"

# Define how often to auto-update (in days).
export UPDATE_ZSH_DAYS=7

# Enable command auto-correction.
ENABLE_CORRECTION="true"

# Display red dots whilst waiting for completion.
COMPLETION_WAITING_DOTS="true"

# Configure history stamp format
HIST_STAMPS="yyyy-mm-dd"

# Plugin configuration
# Standard plugins can be found in ~/.oh-my-zsh/plugins/*
# Custom plugins may be added to ~/.oh-my-zsh/custom/plugins/
# Add wisely, as too many plugins slow down shell startup.
plugins=(
  ansible
  brew
  bundler
  colored-man-pages
  colorize
  docker
  git
  nmap
  osx
  zsh-navigation-tools
  zsh_reload
)

ZSH_THEME="powerlevel10k/powerlevel10k"

# Load Zsh tools for syntax highlighting and autosuggestions
HOMEBREW_FOLDER="/usr/local/share"
source "${HOMEBREW_FOLDER}/zsh-autosuggestions/zsh-autosuggestions.zsh"
source "${HOMEBREW_FOLDER}/zsh-syntax-highlighting/zsh-syntax-highlighting.zsh"

# Load oh-my-zsh framework
source "${ZSH}/oh-my-zsh.sh"

# Powerlevel10k configuration
[ -e ${HOME}/.p10k.zsh ] && source ${HOME}/.p10k.zsh

# Local custom plugins
for item in $(ls -1 ${HOME}/.profile.d/*.plugin.zsh); do
  [ -e "${item}" ] && source "${item}"
done

The important line from the p10k perspective is:

[ -e ${HOME}/.p10k.zsh ] && source ${HOME}/.p10k.zsh

This line source the p10k configuration that is stored in the .p10k.zsh file. The one that creates the layout shown in the screenshot above is the following:

Currently, the configuration file is quite big but I'm pretty sure not everything in this file is really required. It is more or less based on the generated configuration file using the lean set up. I haven't checked the default values for the configuration content yet but I could imagine a lot of configurations are default and as such, could be removed from the configuration file.

For those who want to know what this part in the .zshrc configuration file is about:

# Local custom plugins
for item in $(ls -1 ${HOME}/.profile.d/*.plugin.zsh); do
  [ -e "${item}" ] && source "${item}"
done

This includes my personal zsh plugins that are currently not available in other frameworks like oh-my-zsh. It seems to be a little challenge to get additional plugins included there. You can see them here:

From zero to hero - Bootstrap with Ansible

thbe — Sun, 22 Dec 2019 17:13:32 +0000

This post was originally published at thbe.org.

In preparation for a meeting to discuss ways of working to optimize server operations in a mid-size data center, I was seeking a lightweight approach to automate server installations. In the past, I was primarily using Puppet for data center management. Although it’s still a good choice for an enterprise-level, full automation, it also has some disadvantages. First of all, it’s a client/server infrastructure that requires by nature more installation effort compared to clientless solutions. It also requires more resources on the client to permanently get the state of the client. Last but not least, it’s also a bit more complex when it comes to writing generic usable modules.

Long story short, I decided to give the number one competitor of Puppet called Ansible a try. Ansible has a similar, slightly smaller scope and is designed to be more lightweight then Puppet. You don’t need to install agents on the clients and the whole architecture looks a little bit straighter forward.

As proof of concept, I built a small automation procedure for my home office. As Ansible is owned by RedHat I used CentOS as the server operating system. CentOS can be automatically installed by using kickstart and is as such, a good platform to test the general idea. As a first step, I’ve created a generic kickstart file that enables me to install CentOS 7 with a minimal package set (the kickstart for CentOS 8 should look quite similar). This is my preferred approach as I tend to use the automation tool for the setup work. In my proof of concept, I use the root user for the configuration management. In a real-world scenario, you should use a dedicated configuration management user in conjunction with sudo:

#version=DEVEL
#
# Kickstart installation file with minimal package set
#
# Author:       Thomas Bendler <code@thbe.org>
# Date:         Wed Jan  2 17:25:26 CET 2019
# Revision:     1.0
#
# Distribution: CentOS
# Version:      7
# Processor:    x86_64

# Kickstart settings
install
cdrom
text
reboot

# System language settings
lang en_US.UTF-8
keyboard --vckeymap=de-nodeadkeys --xlayouts='de (nodeadkeys)'
timezone Europe/Berlin --isUtc

# System password for root (create with i.e. pwkickstart)
rootpw --iscrypted $6$aaa[]zzz

# System bootloader configuration
bootloader --append=" crashkernel=auto" --location=mbr --boot-drive=sda
ignoredisk --only-use=sda

# System partition information
clearpart --all --initlabel --drives=sda
autopart --type=lvm

# System network settings
network  --bootproto=dhcp --device=link --onboot=on --ipv6=auto --hostname=node_1.local.domain

# System security settings
auth --enableshadow --passalgo=sha512
selinux --permissive
firewall --service=ssh

# System services
services --enabled="chronyd"

# System packages
%packages
@^minimal
@core
chrony
kexec-tools
%end

# System addons
%addon com_redhat_kdump --enable --reserve-mb='auto'
%end

# Post installation settings
%post --log=/root/kickstart-post.log
/usr/bin/logger "Starting anaconda postinstall"

/usr/bin/mkdir /root/.ssh
/usr/bin/chmod 700 /root/.ssh
/usr/bin/echo "ssh-rsa AAA[...]ZZZ admin1@management" > /root/.ssh/authorized_keys
/usr/bin/chmod 400 /root/.ssh/authorized_keys

sync
exit 0
%end

The last part of the kickstart file is used to distribute the authorized_keys file to enable passwordless login to the root account. Again, in a real-world scenario, you should use this section to create a dedicated management user as well as the required sudo configuration. Alternatively, you can use Ansible to create individual administrative user accounts on the target hosts.

The now still remaining part is to tell the installer that he should use the kickstart file instead of asking the user how and what to install. This is simply done by adding inst.ks= to the kernel parameter. The installer can use different methods to get the file which is described in the kickstart documentation. In this proof of concept I simply use a web target:

inst.ks=https://server.local.domain/kickstart.cfg

Once the Kickstart based installation is finished, the configuration management done with Ansible will take over. If you move the proof of concept to a real datacenter you should consider extending the automated Kickstart installation by tools like Cobbler. This will enable fully-fledged automation by using PXE boot (network boot without a local operating system). Another option, depending on your security and update strategy, is to use templates to provision the operating system instead of doing a real installation. But this approach has, at least from my point of view, too many disadvantages (especially in the operating area).

So far, I showed how to automate the installation process of the core operating system. When this process is done, we should have a minimum operating system installation with a bit of basic configuration (mainly access related). As already said, this will be the point were Ansible will perform the final configuration. I won't provide a manual on how to use Ansible in this post (you can find the complete documentation at https://docs.ansible.com/ansible/latest/index.html), instead, I would show some basic concepts how the tool is used for its purpose. To get an idea of how things are organized in Ansible, let's first take a look at the folder structure of the proof of concept:

├── group_vars
│   ├── all.yml
│   ├── datacenter1.yml
│   └── location1.yml
├── host_vars
│   └── server1.domain.local
│       └── network.yml
├── playbooks
│   ├── add_ssh_fingerprints.yml
│   └── reboot_hosts.yml
├── roles
│   ├── common
│   │   ├── tasks
│   │   │   ├── environment.yml
│   │   │   ├── localtime.yml
│   │   │   ├── main.yml
│   │   │   ├── motd.yml
│   │   │   ├── networking.yml
│   │   │   ├── repositories.yml
│   │   │   ├── tools.yml
│   │   │   ├── upgrade.yml
│   │   │   └── user.yml
│   │   └── templates
│   │       ├── custom_sh.j2
│   │       ├── ifcfg-interface.j2
│   │       ├── motd.j2
│   │       ├── network.j2
│   │       └── route-interface.j2
│   └── web
│       ├── handlers
│       │   └── main.yml
│       ├── tasks
│       │   └── main.yml
│       └── templates
│           ├── index_html.j2
│           └── nginx_conf.j2
├── common.yml
├── production
├── site.yml
├── staging
└── web.yml

Ok, let's go through this tree step by step. The first two directories, group_vars and host_vars contain the variables used in the configuration scripts. The idea is to define the scope of the variables. They can apply to either all hosts, to a data center, a region/location or only to one specific host. Variables that apply to all managed hosts can be for example local administrative user IDs. Variables that apply to a region/location can be for example settings in which time-server should be used by the hosts in that region. Datacenter related variables can be related to the routing for example. And last but not least, host-specific variables can be related to IPs, hostnames and so on and so forth. This enables the administrator to easily set up global infrastructures without much effort.

Before I move on to the playbooks I would like to highlight two files in the root directory. Those are staging and production. Both files contain the servers and groups (region, location, role) where the servers belong to. The distinction between staging and production reflects the role of those (here you can see the staging file):

[hamburg-hosts]
node_1.local.domain  ansible_ssh_host=172.20.0.4  ansible_user=root
node_2.local.domain  ansible_ssh_host=172.20.0.5  ansible_user=root
node_3.local.domain  ansible_ssh_host=172.20.0.6  ansible_user=root

[hamburg-web]
node_4.local.domain  ansible_ssh_host=172.20.0.7  ansible_user=root
node_5.local.domain  ansible_ssh_host=172.20.0.8  ansible_user=root
node_6.local.domain  ansible_ssh_host=172.20.0.9  ansible_user=root
node_7.local.domain  ansible_ssh_host=172.20.0.10 ansible_user=root

[berlin-web]
node_8.local.domain  ansible_ssh_host=172.20.0.11 ansible_user=root
node_9.local.domain  ansible_ssh_host=172.20.0.12 ansible_user=root
node_10.local.domain ansible_ssh_host=172.20.0.13 ansible_user=root

[web:children]
hamburg-web
berlin-web

[hamburg:children]
hamburg-hosts
hamburg-web

[berlin:children]
berlin-web

The hosts in the staging file can be addressed by the names in the brackets. The exception is the colon entries. Those represent a collection of groups for dedicated hosts, for example, all groups located in Hamburg as shown with the entry [hamburg:children]. The hosts can be accessed with the keyword hamburg. The group names are in a relationship with the variables and this, in the end, closes the circle. Now that we know how things are controlled, we can follow up with the answer to the question of how things are done.

Before we start with the playbooks that define how things are done, some words on the ad-hoc administration. Ansible can also be used to do ad-hoc administration based on the inventory. This gives an administrator the ability to perform activities based on the region or the datacenter or on all hosts at once. Let's assume we've updated an alias (CNAME) on our central DNS server. Before we use the updated alias everywhere in our landscape we would like to know if every host is already linked to the updated alias. Instead of logging into each and every host, executing dig, we can do it in one rush with Ansible:

ansible -i staging all -a "/bin/dig updated-cname.local.domain +short"

Another task would be to check if all servers in the inventory are reachable or if a specific service is running on the server. This will more or less look like this:

ansible -i staging all -m ping
ansible -i staging all -m shell -a "/bin/ps aux | /bin/grep chronyd"

The playbooks itself can be seen as a grouping for the ad-hoc tasks. Instead of doing action after action manually, playbooks gives you the ability to group and combine several activities and execute it step by step as a whole instruction set. It's a kind of automating the things the administrator did previously with checklists. A sample playbook that could be used to first install glances and then reboot a host one minute after the execution of the playbook looks like this:

---
- hosts:        all
  become:       yes
  become_user:  root
  gather_facts: false

  tasks:
    # Install glances
    - name: Install glances
      yum: name=glances state=latest
    # Reboot the target host
    - name:    Reboot the target host
      command: /sbin/shutdown --no-wall -r +1 "Reboot was triggered by Ansible"

The file is based on the YAML format and contains one or more activities based on a scope. It can be executed like this:

ansible-playbook -i staging ./example_playbook.yml

For the time being, this is already sufficient to transition the widely used checklist approach to an automated approach. But this is only the tip of the iceberg, automation, once it is in place, can do a lot more for you which brings us finally back to the folder structure you find in the middle of this post. When I find some time I will create another post and dive a little bit deeper into the automation possibilities, especially into the playbooks and roles.

Enhance your macOS terminal

thbe — Sun, 15 Dec 2019 19:07:58 +0000

This post was originally published at thbe.org.

Personally, I use quite often the terminal when I use my computer, laptop or whatever. As a result, I modified my terminal quite heavily to ease my work and to get the best out of the terminal. In the past, I did it mostly manually which requires a lot of attention from my side and regular upgrades when e.g. OS updates are performed. So I tried to reduce at least the effort I have to spent when enhancing the terminal.

The result was the combination of iTerm2 + zsh + oh-my-zsh + powerline + powerlevel9k. This combination covers roughly 95% of my requirements and reduced the effort I have to spend maintaining my terminal significantly. In this blog post, I’ll show you how you can get the same terminal that I use:

So, let’s start! I assume you have Homebrew installed on your macOS. If you don’t have Homebrew installed, I strongly recommend installing it, it’s a must-have when you work with macOS. You find the installation instructions on their homepage https://brew.sh/. With Homebrew you can install most of the required packages. But before we do this, let’s download the current stable iTerm2 version:

https://www.iterm2.com/downloads.html

Extract the ZIP file and move the app to your program folder. You can now start iTerm2. Once this is done you can install zsh :

brew install zsh zsh-autosuggestions zsh-syntax-highlighting

The next step is to install oh-my-zsh. This is fortunately also quite easy, just use this command:

sh -c "$(curl -fsSL https://raw.github.com/robbyrussell/oh-my-zsh/master/tools/install.sh)"

Most of the actions are now already completed. Next, we need to install powerline :

brew install python3
pip3 install powerline-status

I use powerline primarily for vim which needs to be configured in the .vimrc file:

" powerline
set rtp+=/usr/local/lib/python3.6/site-packages/powerline/bindings/vim
set laststatus=2
set t_Co=256

The last step is to install powerlevel9k. This can be done again with Homebrew :

brew tap sambadevi/powerlevel9k
brew install powerlevel9k

We now have all the required packages installed and we can start with the configuration. First things first, the built-in fonts do not fully support this configuration, you need to install an appropriate font first. I used the FiraCode light font. To install the font, you need to download the font to the font library:

mkdir ~/Downloads/FiraCode && cd ~/Downloads/FiraCode
wget https://github.com/tonsky/FiraCode/releases/download/2/FiraCode_2.zip
unzip FiraCode_2.zip
cp ttf/*.ttf ~/Library/Fonts/
cd ~/Downloads && rm -rf FiraCode/

The next step is the iTerm2 color scheme. I use Cobalt2 provided by Wes Bos at https://github.com/wesbos/Cobalt2-iterm. The color scheme needs to be downloaded and then imported into iTerm2 :

cd ~/Downloads
curl https://raw.githubusercontent.com/wesbos/Cobalt2-iterm/master/cobalt2.itermcolors --output cobalt2.itermcolors

The color scheme can now be imported when the preferences in iTerm2 are opened and then profiles -> colors -> color preset -> import is chosen.

Last but not least, you need to modify the zsh configuration file to match your needs. My .zshrc looks like this:

# zsh configuration file
#
# Author: Thomas Bendler <code@thbe.org>
# Date: Tue Sep 24 20:28:27 UTC 2019

# Add powerline support
POWERLINE_ZSH="/usr/local/lib/python3.7/site-packages/powerline/bindings/zsh/powerline.zsh"
[-e "${POWERLINE_ZSH}"] && source "${POWERLINE_ZSH}"

# If you come from bash you might have to change your $PATH.
export PATH="/usr/local/sbin:${PATH}"

# Path to your oh-my-zsh installation.
export ZSH="${HOME}/.oh-my-zsh"

# Uncomment the following line to use case-sensitive completion.
CASE_SENSITIVE="true"

# Uncomment the following line to change how often to auto-update (in days).
export UPDATE_ZSH_DAYS=7

# Uncomment the following line to enable command auto-correction.
ENABLE_CORRECTION="true"

# Uncomment the following line to display red dots whilst waiting for completion.
COMPLETION_WAITING_DOTS="true"

# Uncomment the following line if you want to change the command execution time
# stamp shown in the history command output.
# You can set one of the optional three formats:
# "mm/dd/yyyy"|"dd.mm.yyyy"|"yyyy-mm-dd"
# or set a custom format using the strftime function format specifications,
# see 'man strftime' for details.
# HIST_STAMPS="mm/dd/yyyy"

# Which plugins would you like to load?
# Standard plugins can be found in ~/.oh-my-zsh/plugins/*
# Custom plugins may be added to ~/.oh-my-zsh/custom/plugins/
# Add wisely, as too many plugins slow down shell startup.
plugins=(
  ansible
  battery
  brew
  bundler
  colorize
  docker
  dotenv
  git
  git-flow-avh
  iterm2
  nmap
  osx
  rake
  ruby
  sudo
  zsh-navigation-tools
)

ZSH_THEME="powerlevel9k"

source "${ZSH}/oh-my-zsh.sh"

# User configuration
# Load Zsh tools for syntax highlighting and autosuggestions
HOMEBREW_FOLDER="/usr/local/share"
source "${HOMEBREW_FOLDER}/zsh-autosuggestions/zsh-autosuggestions.zsh"
source "${HOMEBREW_FOLDER}/zsh-syntax-highlighting/zsh-syntax-highlighting.zsh"

# Powerlevel9k configuration
#POWERLEVEL9K_MODE="compatible"

# Left prompt - Configure indicator when working as root
POWERLEVEL9K_ROOT_INDICATOR_BACKGROUND="clear"
POWERLEVEL9K_ROOT_INDICATOR_FOREGROUND="red"

# Left prompt - Configure context (user@hostname)
POWERLEVEL9K_CONTEXT_DEFAULT_BACKGROUND="clear"
POWERLEVEL9K_CONTEXT_DEFAULT_FOREGROUND="magenta"

# Left prompt - Configure display of current directory
POWERLEVEL9K_DIR_HOME_BACKGROUND="clear"
POWERLEVEL9K_DIR_HOME_FOREGROUND="white"
POWERLEVEL9K_DIR_HOME_SUBFOLDER_BACKGROUND="clear"
POWERLEVEL9K_DIR_HOME_SUBFOLDER_FOREGROUND="white"
POWERLEVEL9K_DIR_ETC_BACKGROUND="clear"
POWERLEVEL9K_DIR_ETC_FOREGROUND="red"
POWERLEVEL9K_DIR_WRITABLE_FORBIDDEN_BACKGROUND="clear"
POWERLEVEL9K_DIR_WRITABLE_FORBIDDEN_FOREGROUND="red"
POWERLEVEL9K_DIR_DEFAULT_BACKGROUND="clear"
POWERLEVEL9K_DIR_DEFAULT_FOREGROUND="white"
POWERLEVEL9K_SHORTEN_DIR_LENGTH="3"
POWERLEVEL9K_SHORTEN_STRATEGY="truncate_middle"

# Right prompt - Configure command execution status indicator
POWERLEVEL9K_STATUS_OK_BACKGROUND="clear"
POWERLEVEL9K_STATUS_OK_FOREGROUND="green"
POWERLEVEL9K_STATUS_ERROR_BACKGROUND="clear"
POWERLEVEL9K_STATUS_ERROR_FOREGROUND="red"
POWERLEVEL9K_STATUS_CROSS="true"
POWERLEVEL9K_STATUS_VERBOSE="true"

# Right prompt - Configure command execution time measurement
POWERLEVEL9K_COMMAND_EXECUTION_TIME_BACKGROUND="clear"
POWERLEVEL9K_COMMAND_EXECUTION_TIME_FOREGROUND="white"

# Right prompt - Configure version control system
POWERLEVEL9K_VCS_CLEAN_BACKGROUND="clear"
POWERLEVEL9K_VCS_CLEAN_FOREGROUND="green"
POWERLEVEL9K_VCS_MODIFIED_BACKGROUND="clear"
POWERLEVEL9K_VCS_MODIFIED_FOREGROUND="darkorange"
POWERLEVEL9K_VCS_UNTRACKED_BACKGROUND="clear"
POWERLEVEL9K_VCS_UNTRACKED_FOREGROUND="red"
POWERLEVEL9K_SHOW_CHANGESET="true"
POWERLEVEL9K_CHANGESET_HASH_LENGTH="12"

# Right prompt - Configure display of running background jobs
POWERLEVEL9K_BACKGROUND_JOBS_BACKGROUND="clear"
POWERLEVEL9K_BACKGROUND_JOBS_FOREGROUND="green"

# Right prompt - Configure RAM settings
POWERLEVEL9K_RAM_BACKGROUND="clear"
POWERLEVEL9K_RAM_FOREGROUND="white"

# Right prompt - Configure load settings
POWERLEVEL9K_LOAD_CRITICAL_BACKGROUND="clear"
POWERLEVEL9K_LOAD_WARNING_BACKGROUND="clear"
POWERLEVEL9K_LOAD_NORMAL_BACKGROUND="clear"
POWERLEVEL9K_LOAD_CRITICAL_FOREGROUND="red"
POWERLEVEL9K_LOAD_WARNING_FOREGROUND="darkorange"
POWERLEVEL9K_LOAD_NORMAL_FOREGROUND="green"

# Right prompt - Configure battery status
POWERLEVEL9K_BATTERY_CHARGING_BACKGROUND="clear"
POWERLEVEL9K_BATTERY_CHARGING_FOREGROUND="white"
POWERLEVEL9K_BATTERY_CHARGED_BACKGROUND="clear"
POWERLEVEL9K_BATTERY_CHARGED_FOREGROUND="green"
POWERLEVEL9K_BATTERY_DISCONNECTED_BACKGROUND="clear"
POWERLEVEL9K_BATTERY_DISCONNECTED_FOREGROUND="darkorange"
POWERLEVEL9K_BATTERY_LOW_THRESHOLD="10"
POWERLEVEL9K_BATTERY_LOW_BACKGROUND="clear"
POWERLEVEL9K_BATTERY_LOW_FOREGROUND="red"
POWERLEVEL9K_BATTERY_VERBOSE=false

# Right prompt - Configure disk usage
POWERLEVEL9K_DISK_USAGE_NORMAL_BACKGROUND="clear"
POWERLEVEL9K_DISK_USAGE_NORMAL_FOREGROUND="green"
POWERLEVEL9K_DISK_USAGE_WARNING_BACKGROUND="clear"
POWERLEVEL9K_DISK_USAGE_WARNING_FOREGROUND="darkorange"
POWERLEVEL9K_DISK_USAGE_CRITICAL_BACKGROUND="clear"
POWERLEVEL9K_DISK_USAGE_CRITICAL_FOREGROUND="red"

# Right prompt - Configure IP address
POWERLEVEL9K_IP_BACKGROUND="clear"
POWERLEVEL9K_IP_FOREGROUND="white"

# Configure multiline prompt
POWERLEVEL9K_PROMPT_ON_NEWLINE="true"
POWERLEVEL9K_SHOW_CHANGESET="true"
POWERLEVEL9K_MULTILINE_FIRST_PROMPT_PREFIX=""
POWERLEVEL9K_MULTILINE_LAST_PROMPT_PREFIX="$ "
POWERLEVEL9K_LEFT_SEGMENT_SEPARATOR=""
POWERLEVEL9K_RIGHT_SEGMENT_SEPARATOR=""
POWERLEVEL9K_LEFT_SUBSEGMENT_SEPARATOR=""
POWERLEVEL9K_RIGHT_SUBSEGMENT_SEPARATOR=""

# Configure the prompt content
POWERLEVEL9K_LEFT_PROMPT_ELEMENTS=(root_indicator context dir vcs)
POWERLEVEL9K_RIGHT_PROMPT_ELEMENTS=(status command_execution_time ram disk_usage ip)
#POWERLEVEL9K_RIGHT_PROMPT_ELEMENTS=(status command_execution_time load ram disk_usage ip)

# Local custom snippets
for item in $(ls -1 ${HOME}/.profile.d/*.profile); do
  [-e "${item}"] && source "${item}"
done

With these add-ons and configuration in place, your terminal should look like my one and should, if you don’t like it, at least be a good starting point to start your own configuration.

CI/ CD with Github Actions

thbe — Wed, 11 Dec 2019 23:51:44 +0000

This post was originally published at thbe.org.

My traditional CI/CD pipeline I have used so far was primarily a combination of Github in conjunction with TravisCI. This combination worked pretty well for me but had some downsides. Based on the fact that I use a PRO account on Github I have to deal with public and private repositories. Unfortunately, the pipeline works only with public Github repositories but not with private Github repositories. Also, the way I handled secrets/ passwords that are required in the CD pipeline to deploy code to a third-party platform was not really satisfying me as it was completely manual and required personal hardware. However, when I saw that Github introduced Actions I thought it was worth a try to check it out and to play a little bit around with the new CI/CD pipeline. In the following article, I will share my experience with Github Actions as well as what is good and whatnot.

According to Github, Actions should automate your workflow from idea to production. Sounds good and is something I was looking for. But describing something on an advertising page looks, in reality, most likely somewhat different. So I picked one of the more complicated repositories I use that didn't work before and tried to establish a CI/ CD pipeline based on Github Actions. The candidate I had chosen was my website, more precise, the tools I use to build my website. As I already posted earlier I'm using Jekyll to build my site. All required actions to build and test my site are collected in a Ruby-based Rakefile. For example, when my site is compiled, I run HTML-Proofer to check the HTML code. Additional complexity was added by the fact that my website repository is private due to some parts I need to refactor first before I can release the code.

The first thing that is required is a workflow file. You need to create a directory called .github/workflows in your project root. Inside that directory, you have to create a YAML file. In my case, I named the first file development.yml but you can name it as you like as long as the file ends with ".yml". This file contains the job and the associated actions the build pipeline will execute to test and deploy the code. The following file contains the CI part of the story, this means, it builds the website and check it for errors:

name: Development Workflow

on: [push]

jobs:
  test:
    name: CI Pipeline
    runs-on: ubuntu-latest
    steps:
      - name: Checkout master
        uses: actions/checkout@v1
      - name: Set up Ruby 2.6
        uses: actions/setup-ruby@v1
        with:
          ruby-version: 2.6.x
      - name: Setup bundler and required gems
        run: |
          gem install bundler
          bundle install --jobs 4 --retry 3
      - name: Build and test the website
        run: bundle exec rake

So, what does exactly happens in the configuration file? The first keyword defines the name of the workflow:

The second keyword "on" defines that the workflow should be executed every time I push code into my Github repository. The "jobs" keyword describes what should be executed in the workflow. You can define one or more jobs, depending on what you would like to execute.

The first version of the development.yml file I've shown before starts a container based on ubuntu-latest. After the start of the container, the configuration file defines four actions that should be executed inside the Ubuntu container. The first action that takes place is the checkout of my current git repository. The second action is to set up Ruby with version 2.6. Action number three setup the required Ruby packages based on the Gemfile in my repository root. When these three actions have been performed, the fourth action can do the effective test of my web site based on the Rakefile that is also defined in my git repository root.

Once the actions have been performed you can see the result in the Actions tab:

You can use the triangle to expand the performed steps to see the logs in detail.

Once the CI scenario has been set up for testing the changes on the website, it's time to configure the CD part for the QA deployment. So, what are we trying to achieve? Easily spoken, when the git commit has passed all required tests I would like to upload the generated site into my test tenant. This is a little bit tricky. I already wrote in my first post about Github Actions that my website is not hosted somewhere in the Github space, instead, it is hosted by a third-party provider in my web-space area. The third-party provider offers an SFTP option to upload the site data. This completely fine when the upload part is done manually but not that easy anymore when a CD tool should do the upload. Fortunately, Github Actions has options for this as well. But before we start with this, let's have a look at how my development.yml looks now with the CD part included:

name: Development Workflow

on: [push]

jobs:
  test:
    name: CI Pipeline
    runs-on: ubuntu-latest
    steps:
      - name: Checkout master
        uses: actions/checkout@v1
      - name: Set up Ruby 2.6
        uses: actions/setup-ruby@v1
        with:
          ruby-version: 2.6.x
      - name: Setup bundler and required gems
        run: |
          gem install bundler
          bundle install --jobs 4 --retry 3
      - name: Build and test the website
        run: bundle exec rake

  deploy:
    name: CD Pipeline QA
    needs: test
    runs-on: ubuntu-latest
    steps:
      - name: webfactory/ssh-agent
        uses: webfactory/ssh-agent@v0.1.1
        with:
          ssh-private-key: ${{ secrets.SSH_PRIVATE_KEY }}
      - name: Checkout master
        uses: actions/checkout@v1
      - name: Set up Ruby 2.6
        uses: actions/setup-ruby@v1
        with:
          ruby-version: 2.6.x
      - name: Set up lftp
        run: sudo apt-get install lftp -y
      - name: Setup bundler and required gems
        run: |
          gem install bundler
          bundle install --jobs 4 --retry 3
      - name: Build and deploy the website to QA
        run: bundle exec rake deploy:qa
        env:
          SSH_DEPLOY_SERVER: ${{ secrets.SSH_DEPLOY_SERVER }}
          SSH_DEPLOY_USER: ${{ secrets.SSH_DEPLOY_USER }}

Compared to the initial YAML I've described in my former post, you might have noticed that I have added a second job called deploy. The actions in that job are pretty much the same as in the test job. This job should also create a new Ubuntu container, it should install Ruby and the Gems required to create my Jekyll site. Two things, however, differ from the test job, the first one is an action called "webfactory/ssh-agent", the second one is a deploy action that uses a special environment. Let's start with the second one, as mentioned, I use a special environment for this action. The rationale behind this is, that I need to pass the target and the credentials to deploy the generated site to this target. To achieve this I created three secrets in the Github secret area of the repository:

In the end, these secrets are "only" variables that will be made available in the container that is started with Github Actions. Nonetheless, this is the missing feature to generate deploy scripts without storing secrets in the code that is pushed to Github. Instead, the secrets are kept outside of the container in a secure area at Github ready to be used at each deploy. In detail this means, I use environment variables in my Rakefile to define the target where I would like to deploy the generated site as well as the user that should be used for the deploy. In principle, it would be possible as well to deploy a password with that way but I made the decision to use a cryptographic key instead.

To make things easier for me I used a predefined action called ssh-agent. This action reads the cryptographic key stored in SSH_PRIVATE_KEY and adds it to the ssh-agent installed in the container. With this mechanism in place, I'm able to deploy the generated site code with a passwordless login.

Last but not least, I need to deploy the generated website code to my production environment, at least when I'm happy with the results from the QA deploy. Till now, the trigger to start the deployment was the push of a commit. When we talk about the productive deployment it makes sense to introduce another flag that must be set before something is deployed to production. Fortunately here comes git into the game. Git allows us to tag a specific commit. Tagging a specific commit in a defined way is the flag I use for the production deployment. Here you see the release.yml that I use for my production deployments:

name: Release Workflow

on:
  push:
    tags:
      - "v*" # Push events to matching v*, i.e. v1.0

jobs:
  deploy:
    name: CD Pipeline PRD
    needs: test
    runs-on: ubuntu-latest
    steps:
      - name: webfactory/ssh-agent
        uses: webfactory/ssh-agent@v0.1.1
        with:
          ssh-private-key: ${{ secrets.SSH_PRIVATE_KEY }}
      - name: Checkout master
        uses: actions/checkout@v1
      - name: Set up Ruby 2.6
        uses: actions/setup-ruby@v1
        with:
          ruby-version: 2.6.x
      - name: Set up lftp
        run: sudo apt-get install lftp -y
      - name: Setup bundler and required gems
        run: |
          gem install bundler
          bundle install --jobs 4 --retry 3
      - name: Build and deploy the website to PRD
        run: bundle exec rake deploy:production
        env:
          SSH_DEPLOY_SERVER: ${{ secrets.SSH_DEPLOY_SERVER }}
          SSH_DEPLOY_USER: ${{ secrets.SSH_DEPLOY_USER }}

The most import thing is the "on -> push -> tags" part. I still deploy with the push event, but only if the commit is tagged starting with a "v". So when I tag my commit as "v1.0.2" it will be automatically pushed to production. The even better part of this solution is, I don't need a special development environment to push a commit to production, it could be completely done from the Github web interface. To do so, you need to go to the releases tab of your code:

Create a release, e.g. v1.0.0:

As soon as the version tag is pushed to the Github platform, the whole CI/ CD chain starts. This means, the development.yml and release.yml instructions run in parallel and, when everything went well and no errors were detected, the web-site goes straight into production:

The release automatically creates the corresponding tag:

This triggers finally the whole CI/ CD pipeline:

And finally, everything is in production as planned:

I hope you enjoyed my three posts on how to set up and how to use Github Actions. So far I'm quite surprised how flawless and good the CI/ CD pipeline already works and I didn't find any major caveats so far that prevents me from using Github Actions. I'm already quite eager to migrate some of my other repositories as well to get more inside into what is possible on Github Actions and whatnot, but for now, that's all I could tell. I'll keep you updated when I had migrated more projects.