Automate your coding environment with Ansible, and make a simple GUI for it using only bash scripting

I'm not proud to admit that on two separate occasions I have nuked my linux dev environment. The first time was a dd gone wrong - yes, while trying to write some firmware onto a microcontroller connected to USB, I did in fact do the very thing people warn you about when using dd: accidentally write over my own computer's boot drive.

The second time was again a self-inflicted wound: a script-gone-wrong created a folder called '~' full of files that shouldn't be there (you can in fact create a folder called that, it would need to be enclosed in quotes though). Without thinking, I went to remove this folder by using rm -rf ~ and thus wiping my entire home folder including all my settings and some other apps.

While my code was fine - all my code projects live in git, my software installation and dev environment weren't. I still had to spend several hours setting up all my programs again.

Ever since then, I've decided to automate the setup of my dev environment so that if (or rather, when) I manage to destroy my installation again, I can easily reinstall everything I need.

In fact, this is the same stack that I use to deploy code to VMs, as well as maintain headless machine and colleague's dev laptops for work. It's served me well, so I hope this can help others who like me have tempted fate with a dd or rm -rf and came out on the losing side.

Introducing Ansible

Ansible is an open-source program for basically doing all manner of tasks on remote machines, ranging from installing software to copying over config files, to running shell commands.

There are a few reasons that Ansible is better than just having a bunch of bash scripts that you can throw around: one of the main benefit is that Ansible is bundled with a lot of extensions that allow you declaratively write what you want done, and it'll figure out what actually needs to be run, while skipping anything that is already fulfilled (it's mostly idempotent).

Another benefit is, Ansible just needs to be installed on the local machine that is doing the controlling, it doesn't need to be installed on the remote machine being managed. It's only requirements are an SSH connection, and Python. There's no need to install Ansible on the target machine (it's agentless).

Ansible's files are all YAML, meaning it can all live inside a git repository, making it easy to keep your scripts up to date, or to add or even collaborate on them. I've introduced this kind of setup to several teams, who have all adopted the idea of a shared Ansible repository with automation for many of their formerly manual tasks.

Install Ansible

Ansible only needs to be installed on the local machine that's doing the managing. You use this machine to reach into other machines over SSH and fiddle around. Ansible can also connect to the local machine in case you want to set up the machine you're on.

If you're using Ubuntu, you can install ansible via its ppa. I also install aptitude, as Ansible prefers to use this over apt or apt-get (though it can use apt as well). I also install dialog, a simple utility for creating interactive dialogues in the command line, you'll see why I do this later.

I throw all of this in an install_ansible.sh file that I keep in the repo in case I need to run this on a new machine and don't yet have ansible set up

#!/bin/bash

# install ansible ppa
apt-add-repository ppa:ansible/ansible -y

# install stuff
apt update
apt install -y aptitude ansible dialog

Define the hosts

Ansible uses a hosts file that contains a list of all the hosts you might want to connect with and manage. If you were just using Ansible to manage your own machine, then you would just need one local entry, but if you want to manage remote machines, you can add their SSH connection details in this file hosts.yml file. There are many more options available here, including specific connection options. See Ansible's documentation for those details.

all:
  hosts:
    local:
      ansible_host: 127.0.0.1
      ansible_connection: local
    wintel0:
      ansible_host: 192.168.1.121
    wintel1:
      ansible_host: 192.168.1.142
    ybox-vm:
      ansible_host: 192.168.1.128

The last three are a few of my remote machines that I want to manage.

Playbooks and Roles

Ansible's main "unit of work" is the playbook. This is a single YAML file. I like to keep at least one playbook per physical machine that I want to keep set up or up to date. This way, whenever I want to make sure a particular machine has the latest set of base software, I can run its playbook. Sometimes when there's a cluster of machines that all need the same setup, a host group can be added to the hosts.yml and a single playbook used that target all of them.

Ansible has another level of hierarchy called Roles. I like to use Roles for the individual things that a particular playbook needs to do, for example set up a single software package.

For example, I have a a set of software that I will often use on the machines, so I have a few Roles for setting them up, and I can include this Role across those playbooks. The first one I always run is one called: software-common-apt which sets up miscellaneous software that I like to have when I do any kind of remote admin work: vim, tmux, htop.

Then, I have a software-python which ensures I have the right version of python, pip, setuptools, and wheel. Ansible actually requires python to be installed already, but most linux distros will come with that. So my python Role is mostly about making sure pip and some common packages are available.

Finally, I have a software-docker which does all the steps needed to get docker, docker-compose, and docker-credential-helper installed. Almost all of my work involves docker, so this goes onto every machine I use.

Example Roles and Playbooks

As an example, the file playbooks/roles/software-common-apt/tasks/main.yml looks like this:

---
- name: Installing Common Apt packages
  become: yes
  become_method: sudo
  apt:
    pkg:
    - vim
    - tmux
    - htop
    - jq

Pretty simple. While the file playbooks/roles/software-docker/tasks/main.yml looks like this:

---
- name: Install packages
  become: yes
  become_method: sudo
  block:
  - name: Ensure built-in docker is removed
    apt:
      pkg:
      - docker
      - docker-engine
      - docker.io
      state: absent

  - name: Install docker GPG
    apt_key:
      id: 0EBFCD88
      url: https://download.docker.com/linux/ubuntu/gpg
      state: present

  - name: Install docker apt repository
    apt_repository:
      repo: "deb [arch=amd64] https://download.docker.com/linux/ubuntu {{ ansible_distribution_release }} stable"
      update_cache: yes

  - name: Install docker-ce
    apt:
      name: docker-ce
      state: latest

  - name: Install dependencies
    apt:
      pkg:
      - curl
      - apt-transport-https
      - ca-certificates
      - software-properties-common

  - name: Install Python docker package
    pip:
      name:
      - docker

  - name: Check if docker-compose is installed
    stat:
      path: "{{ compose_path }}"
    register: compose_is_installed

  - name: Check docker-compose version
    when: "compose_is_installed.stat.exists"
    command: "{{ compose_path }} version"
    register: compose_read_version
    changed_when: False
    failed_when: False

  - name: Install docker-compose
    when: "not compose_is_installed.stat.exists or compose_version not in compose_read_version.stdout"
    get_url:
      url: "https://github.com/docker/compose/releases/download/{{ compose_version }}/docker-compose-Linux-x86_64"
      dest: "{{ compose_path }}"
      owner: root
      group: root
      mode: +x

  - name: Check if docker-credential-helper is installed
    stat:
      path: "{{ credential_path }}"
    register: credential_is_installed

  - name: Check docker-credential-helper version
    when: "credential_is_installed.stat.exists"
    command: "{{ credential_path }} version"
    register: credential_read_version
    changed_when: False
    failed_when: False

  - name: Install docker-credential-helpers
    when: "not credential_is_installed.stat.exists or credential_version not in credential_read_version.stdout"
    unarchive:
      src: "https://github.com/docker/docker-credential-helpers/releases/download/v{{ credential_version }}/docker-credential-secretservice-v{{ credential_version }}-amd64.tar.gz"
      remote_src: yes
      dest: /usr/local/bin
      mode: +x

- name: "Remember to add docker group to users!"
  debug:
    msg: Remember to add docker groups to users with "usermod -aG docker <username>". Use "newgrp docker" to use the group immediately

Where the variables are pulled from playbooks/roles/software-docker/vars/main.yml and look like this:

---
compose_version: "1.25.4"
compose_path: /usr/local/bin/docker-compose
credential_version: "0.6.3"
credential_path: /usr/local/bin/docker-credential-secretservice

This lets me specify the exact versions of things I'm installing to ensure reproducibility of installs.

Ansible Roles expect files to be in a certain folder structure, definitely check out their documentation for these details.

These Roles need to be tied together into a playbook. So I have a playbooks/machine-wintel0.yml which exists to install all the things needed on that machine. It looks like this:

---
- hosts: wintel0
  tasks:
  - include_role:
      name: software-common-apt

  - include_role:
      name: software-python

  - include_role:
      name: software-docker

Note how this playbook specifically targets wintel0. I could also tell this script to target all, and running it would attempt to keep all the software up to date in the all group of my hosts file (which is everything). That is sometimes useful if you have large numbers of machines to maintain with the same base software.

Running Ansible from CLI

With these files in place, and having installed Ansible, I can now run it from the command line.

$ ansible-playbook playbooks/machine-wintel0.yml -i hosts.yml -K --ask-pass
SSH password:
BECOME password[defaults to SSH password]:

PLAY [wintel0] ****************************************************

TASK [Gathering Facts] ****************************************************
ok: [wintel0]

TASK [include_role : software-common-apt] *********************************

TASK [software-common-apt : Installing Common Apt packages] ***************
ok: [wintel0]

TASK [include_role : software-python] *************************************

TASK [software-python : Installing Python 3.8] ****************************
ok: [wintel0]

TASK [software-python : Install stuff that ansible needs] *****************
ok: [wintel0]

TASK [include_role : software-docker] *************************************

TASK [software-docker : Ensure built-in docker is removed] ****************
ok: [wintel0]

TASK [software-docker : Install docker GPG] *******************************
ok: [wintel0]

TASK [software-docker : Install docker apt repository] ********************
ok: [wintel0]

TASK [software-docker : Install docker-ce] ********************************
ok: [wintel0]

TASK [software-docker : Install dependencies] *****************************
ok: [wintel0]

TASK [software-docker : Install Python docker package] ********************
ok: [wintel0]

TASK [software-docker : Check if docker-compose is installed] *************
ok: [wintel0]

TASK [software-docker : Check docker-compose version] *********************
ok: [wintel0]

TASK [software-docker : Install docker-compose] ***************************
skipping: [wintel0]

TASK [software-docker : Check if docker-credential-helper is installed] ***
ok: [wintel0]

TASK [software-docker : Check docker-credential-helper version] ***********
ok: [wintel0]

TASK [software-docker : Install docker-credential-helpers] ****************
ok: [wintel0]

TASK [software-docker : Remember to add docker group to users!] ***********
ok: [wintel0] => {
    "msg": "Remember to add docker groups to users with \"usermod -aG docker <username>\". Use \"newgrp docker\" to use the group immediately"
}

PLAY RECAP ****************************************************************
wintel0: ok=16   changed=0    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0

In this case, there are a lot of ok (as I had already set up this machine before). This means the particular task is already fulfilled so there would have been no need to have done the task. If Ansible made any changes, it would highlight as changed. This is part of what makes Ansible nice, as it can decide whether it will even attempt to run a task. For custom tasks that you define, it is also possible to add in your own ways of detecting whether a task should run or not.

I also selected to --ask-pass -K when running the command, this will prompt for SSH password and sudo password. If you have key access configured and passwordless sudo configured, you wouldn't need to prompt for a passwords.

Making a simple GUI for Ansible

As you can see, with all the Roles and playbooks set up, whenever I want to run it, I can just issue the command. I've ended up with a big laundry list of different playbooks, each doing a different task. Some playbooks just do simple things like remotely shut down a machine (easier to run the playbook than to have to SSH in and then issue the shutdown command); others fetch log files.

This big list of playbooks becomes hard to remember, most of the time I have to look through my files first to see what I have before I can type out the command. So to help with this, I use a simple GUI to let me look at what playbooks I have and select them, sometimes I also want to be able to limit my run to certain hosts. This is where that dialog comes in handy. Forget having to learn any GUI/TUI libraries, dialog is the way to go for quick interactivity from bash scripts.

My script is simply:

#!/bin/bash

HEADER="Run Ansible Playbook"
TLINES=$(tput lines)
TCOLS=$(expr $(tput cols) \* 4 / 5)

# get playbook
FILE=$(dialog --backtitle "$HEADER" --stdout --clear --title "Select Playbook" --fselect playbooks/ $(expr $TLINES - 15) $TCOLS)

if [ ! -f "$FILE" ]; then
  dialog --backtitle "$HEADER" --clear --title "Error" --msgbox "The file $FILE was not found" 5 $TCOLS
  clear
  exit 1
fi

# get hosts
# HOST_LIST=$(yq r hosts.yml -j | jq ".all.hosts|keys[]" | tr -d '"')
# HOST_NUM=$(echo "$HOST_LIST" | wc -l)
# HOST_STR=$(echo "$HOST_LIST" | awk '{print $0" off"}' | nl -w1 -s" " | tr "\n" " ")
# HOSTS=$(dialog --backtitle "$HEADER" --stdout --clear --title "Select Hosts" --checklist "Use space to select" $(expr $TLINES - 10) $TCOLS $HOST_NUM $HOST_STR) # the xagrs trims spaces
#
# CHECK=$(echo $HOSTS | tr -d "\n")
# if [ -z "$CHECK" ]; then
#   dialog --backtitle "$HEADER" --clear --title "Error" --msgbox "No hosts selected, use space to select" 5 $TCOLS
#   clear
#   exit 1
# fi
#
# HOST_CSV=$(echo $HOSTS | tr " " "\n" | (while IFS=" " read -r line; do echo -n "$HOST_LIST" | sed -n "${line}p"; done;) | tr "\n" ",")
#
# override
HOST_CSV="all"

dialog --backtitle "$HEADER" --stdout --clear --title "Ask for SSH password?" --yesno "Yes: enter SSH password\nNo: local or key access available" 10 $TCOLS
RETVAL=$?

if [ $RETVAL -eq 0 ]; then
  ASK_PASS="-K --ask-pass"
fi

clear
ansible-playbook $FILE -i hosts.yml --limit "$HOST_CSV" $ASK_PASS $@

Running this script will bring up a file selection dialogue that'll let you select which playbook to run. It'll also ask you which host to limit the run to (if you uncomment out that section of my script); and also ask you whether you want to enter a password or not and try to use keyfiles.

This selection dialogue makes using Ansible a breeze, I just fire up ./run_ansible.sh (which contains the script above, and also lives in the git repo), select the playbook I want, and off it goes. No more having to remember exactly what the playbook is called or what the hostnames are.

The final folder structure at this point looks something like this:

Conclusion

Having used this way of maintaining my personal dev machines and server deployments for work for several years now, I've gradually built up a library of playbooks for a variety of tasks. Ansible has entirely replaced the collection of notes I used to keep to remind myself about "how to install x" for hard-to-remember installation procedure for some software.

For my devops tasks, Ansible works nicely alongside CI/CD and Infrastructure-as-Code too. Our repositories sometimes have a deploy folder that contains an Ansible playbook intended to be run by the CI/CD upon completion of the build; the playbook contains all the tasks needed to set up or update a particular server and deploy the code or docker image to it. And don't forget, Github Actions has Ansible pre-installed too, so you can get started immediately with deployment tasks from Github Actions without having to install Ansible first.

I highly recommend this kind of tool to any developer who is fed up with having to configure and maintain their dev environment or servers all the time.

(Cover Photo by Franck V. on Unsplash)

Comments

[Corentin Bettiol]

I love dialog.
There is even python-dialog which exposes the functions of dialog in python scripts, it's very useful!
We use it to manage our local devenv (composed of several dockers containers) :)

[Alexey Melezhik]

Ansible looks attractive when it comes to "static" and simple configurations. When one needs add more dynamics things get not that easy (yaml is not a language ). One may consider Sparrow as an alternative automation tool.

[gdahboy]

I have used a lot ansible many tasks and i like it all. I am very good at it, but creating a gui for it ... that's really fantastic