DEV Community: Owen Davies

Storing DevOps documentation in Git and managing CAB approvals with Pull Requests

Owen Davies — Mon, 23 Nov 2020 00:00:00 +0000

Often in a large enterprise you will need to submit changes to a CAB (Change Advisory Board) before getting approval to make any changes to your infrastructure. But managing versions of your diagrams and documents / infrastructure code / what has actually been deployed can be difficult.

Documentation is often stored in a seperate document store (like Sharepoint).

I’m using Azure DevOps to manage work items here, but you can insert your own (Jira, Github Issues).

Our plan today

Problems to address
Solution
New document creation process
Diagram
Results

1. Problems to address

Separating documents from infrastructure code often causes documents to be out of date as they’re not always up to date in Sharepoint.
Decisions of when/why changes were implemented is hard to keep track of.

2. Solution

Store documents within Git (Source Control), directly with infrastructure code.
Utilise Pull Requests in Azure DevOps for CAB approval of document changes (and track any changes that have been requested to the documents).

3. New document creation process

Each application folder has a docs folder (with LLDs, HLDs, Visio diagrams, etc)
An Engineer will have a Story in Azure DevOps to do the analysis and design work and create the document
They will create a branch to make the changes to the documents in that folder, commit the branch and submit a Pull Request when they are complete.
When a Pull Request is requested to master, a release pipeline copies the docs folder to the folder in Sharepoint.
The PR is assigned to a CAB member for CAB.
They then take this to CAB, and can view the documents in Sharepoint or in the repository.
If approved this is merged into master, and the new story can be created for the next sprint to do the implementation.
If denied, they CAB can create bugs against the Pull Request in Azure DevOps, those can be assigned to the engineer who can make the changes and the docs folder will be copied again to the Sharepoint folder

4. Diagram

5. Results

Versioned documents and code are stored together at the same time.
Any time code is changed you will see if documentation hasn’t been updated and also know what actually changed.
Part of the Pull Request process for committing code, can be to ensure documentation has been updated if needed.
All conversations around the document changes will be visible inside the Pull Request. Comments can be seen / discussed inside the Pull Request, and bugs created and assigned for any changes needed. (This will show us approved date, and will see the who/when decisions were made to make changes)

6. Conclusion

Now we have documentation and infrastructure code stored together and versioned in the same location.

Next post will be setting this up in Azure DevOps or maybe GitHub actions.

What do you think of this approach?

How do you store your documentation for you infrastructure code?

Super lightweight bookmarks manager - dmenu bookmarks with bm

Owen Davies — Tue, 12 May 2020 00:00:00 +0000

Storing my internet bookmarks to disk in a non-proprietry way and making them easily accessible was my main goal in the research for this blog post.

A while back I decided that I wanted to TRY to break away from Google in as many ways as I could, not using Chrome as a web browser was one of the biggies. But one of the most convenient things about Chrome is the bookmark sync. The downside to Chrome, is they are tracking everything you do. Whether this is a bad thing or not I leave up to you.

I tried so many different bookmark managers, but so many of them use databases and require complex setup, or special syncing software running.

I wanted something that:

Stored to disk in a text file
Let me use my Dropbox / Nextcloud or normal Cloud sync software to take care of syncing between devices
Works in Chrome / Firefox / Surf / anything
Easily accessible
Super lightweight

I settled on an amazingly simple bookmark manager called bm, and using dmenu (which I was already using as my main menu bar).

Simply hitting Super + Shift + B brings up the bookmark manager

Final result:

The plan

What is bm?
What is dmenu?
My dmenu-bm script
Link to the script on github
Conclusion

1. What is bm?

Enter bm for “Simple Bash CLI bookmarks” https://github.com/tj/bm

Format of the bookmarks file

| Each bookmark gets a single line in the text file, and is delimited by |

Example:

bd8b3eff7fa82a0382a3e7576c5363b6|2016-01-18T07:21:36Z|:1|https://github.com/tj/bm|bm a cool enhanced bookmark tool for your console|default,shell

First column is the GUID
Second column is the DateTime added
Third column… I’m not sure?
Fourth column is the URL link
Five column is the Description text
Sixth column are the tags comma delimited

CLI commands

You can access and manipulate your bookmarks from the CLI. Although I don’t feel this is the most useful way to access it.

Adding bookmarks

bm add https://www.duckduckgo.com "DuckDuckGo Search Engine" search privacy

Listing bookmarks

bm -l

Searching bookmarks

bm -s 'string'

Opening bookmarks

bm -o 'string'

The downside

It doesn’t make it very easily accessible, at least not enough for me. I want to be able to launch a webpage without having to go into a terminal.

2. What is dmenu?

dmenu is a super lightweight menu bar for linux.

dmenu was development by the guys at suckless. They build software “with a focus on simplicity, clarity and frugality.” Basically they create minimal software that is hyper-performant. Sounds cool eh?

https://tools.suckless.org/dmenu/

I love dmenu, it’s lightweight, and allows me to customise it with scripts. It’s become my main menu system on my Linux machine. I’m adding more and more scripts to it and optimising my workflow.

3. My dmenu bookmarks script

dmenu-bm.sh

#!/usr/bin/env bash

declare -A g bmarray;

while IFS=\| read -r guid date id url title tags;
do
  bookmark="$title "-" "$url" "-" "$tags"";
  bmarray["$bookmark"]="$url";
done < /home/owen/Nextcloud/bookmarks/bm.lnk

function load() {
  while IFS=\| read -r guid date id url title tags;
  do
    bookmark="$title "-" "$url" "-" "$tags"";
    printf "$bookmark\n";
  done < /home/owen/Nextcloud/bookmarks/bm.lnk
  printf
}

choice=$(load | dmenu -i -l 15 -p "Add/Open bookmark:")

case "$choice" in
  Add) dmenu-bm-add.sh ;;
  *) bm -o ${bmarray[$choice]} ;;
esac

dmenu-bm-add.sh

#!/bin/sh

result() {
  echo -n | dmenu -p "$1"
}

url="$result "URL:")"
title="$(result "Title:")"
tags="$(result "Tags (comma delimited):")"

bm -b '/home/owen/Nextcloud/bookmarks/bm.lnk' -a $url -T "$title" -t "$tags"

My shortcut in dwm

config.def.h

static const char *bookmarkscmd[] = { "dmenu-bm.sh", NULL };

TODO:

I still have some tidying of this up, some code duplication in the dmenu-bm.sh script. But for the most part this works, and I use this in my day to day workflow.

4. Link to the script on github

You can find the above files in my github at https://github.com/Owen-Davies/dmenu-bm/

5. Conclusion

There we have it, a very simple bookmark management solution that I use day to day.

I’m a big fan of dmenu, do you have any dmenu scripts that you use that you recommend?

Still a bit new to bash scripting, so please point me in the direction of any improvements, I will try to revisit this once I get better. But for now it works, so :-)

You can check out some other dmenu scripts on the subreddit https://www.reddit.com/r/dmenu/

Create an Azure Virtual Machine with Terraform

Owen Davies — Mon, 24 Feb 2020 00:00:00 +0000

To round off things nicely, I thought I would follow on from two previous posts about creating Azure Virtual Machines. First we went through how to create an Azure Virtual Machine using PowerShell, then we went through with ARM templates. Now we’re going to use Terraform for the third try.

Create an Azure Virtual Machine with PowerShell

Create an Azure Virtual Machine with an ARM template

Setting the scene

Let’s assume that we have nothing setup. No virtual network, no storage, nothing. So we will be using Terraform to define everything.

The plan

Create our terraform file
Create the AzureRM provider in terraform
Define the Azure resource group
Define a virtual network and subnet
Define a new public IP address
Define a Network Interface for our VM
Define the Virtual Machine
Build the Virtual Machine
The whole file in one
Conclusion

1. Create the terraform file

Let’s create our terraform file and name it main.tf

touch main.tf

2. Create the AzureRM provider in terraform

Open up main.tf in your editor of choice and add the azure provider to the top of the file

provider "azurerm" {
  version = "= 2.0.0"
  features {}
}

3. Define the Azure resource group

Now let’s create our new resource group that everything will live inside

resource "azurerm_resource_group" "rg" {
  name = "my-first-terraform-rg"
  location = "northeurope"
}

4. Define a virtual network and subnet

resource "azurerm_virtual_network" "myvnet" {
  name = "my-vnet"
  address_space = ["10.0.0.0/16"]
  location = "northeurope"
  resource_group_name = azurerm_resource_group.rg.name
}

resource "azurerm_subnet" "frontendsubnet" {
  name = "frontendSubnet"
  resource_group_name = azurerm_resource_group.rg.name
  virtual_network_name = azurerm_virtual_network.myvnet.name
  address_prefix = "10.0.1.0/24"
}

5. Define a new public IP address

resource "azurerm_public_ip" "myvm1publicip" {
  name = "pip1"
  location = "northeurope"
  resource_group_name = azurerm_resource_group.rg.name
  allocation_method = "Dynamic"
  sku = "Basic"
}

6. Define a Network Interface for our VM

resource "azurerm_network_interface" "myvm1nic" {
  name = "myvm1-nic"
  location = "northeurope"
  resource_group_name = azurerm_resource_group.rg.name

  ip_configuration {
    name = "ipconfig1"
    subnet_id = azurerm_subnet.frontendsubnet.id
    private_ip_address_allocation = "Dynamic"
    public_ip_address_id = azurerm_public_ip.myvm1publicip.id
  }
}

7. Define the Virtual Machine

resource "azurerm_windows_virtual_machine" "example" {
  name = "myvm1"  
  location = "northeurope"
  resource_group_name = azurerm_resource_group.rg.name
  network_interface_ids = [azurerm_network_interface.myvm1nic.id]
  size = "Standard_B1s"
  admin_username = "adminuser"
  admin_password = "Password123!"

  source_image_reference {
    publisher = "MicrosoftWindowsServer"
    offer = "WindowsServer"
    sku = "2019-Datacenter"
    version = "latest"
  }

  os_disk {
    caching = "ReadWrite"
    storage_account_type = "Standard_LRS"
  }
}

8. Build the Virtual Machine

Login to Azure with the CLI

There are several ways to authenticate with Azure to run our terraform file, for this example I’m going to suggest the Azure CLI

Make sure you have the Azure CLI installed, then run:

az login

Which should bring up a browser window for you to login to your Azure subscription.

Run terraform init

Now we need to run terrafrom init to prepare the directory and pull down the resources that we have defined in our file.

terraform init

Build our terraform file

terraform apply

Time taken: 3m10s

The whole file in one

provider "azurerm" {
  version = "= 2.0.0"
  features {}
}

resource "azurerm_resource_group" "rg" {
  name = "my-first-terraform-rg"
  location = "northeurope"
}

resource "azurerm_virtual_network" "myvnet" {
  name = "my-vnet"
  address_space = ["10.0.0.0/16"]
  location = "northeurope"
  resource_group_name = azurerm_resource_group.rg.name
}

resource "azurerm_subnet" "frontendsubnet" {
  name = "frontendSubnet"
  resource_group_name = azurerm_resource_group.rg.name
  virtual_network_name = azurerm_virtual_network.myvnet.name
  address_prefix = "10.0.1.0/24"
}

resource "azurerm_public_ip" "myvm1publicip" {
  name = "pip1"
  location = "northeurope"
  resource_group_name = azurerm_resource_group.rg.name
  allocation_method = "Dynamic"
  sku = "Basic"
}

resource "azurerm_network_interface" "myvm1nic" {
  name = "myvm1-nic"
  location = "northeurope"
  resource_group_name = azurerm_resource_group.rg.name

  ip_configuration {
    name = "ipconfig1"
    subnet_id = azurerm_subnet.frontendsubnet.id
    private_ip_address_allocation = "Dynamic"
    public_ip_address_id = azurerm_public_ip.myvm1publicip.id
  }
}

resource "azurerm_windows_virtual_machine" "example" {
  name = "myvm1"  
  location = "northeurope"
  resource_group_name = azurerm_resource_group.rg.name
  network_interface_ids = [azurerm_network_interface.myvm1nic.id]
  size = "Standard_B1s"
  admin_username = "adminuser"
  admin_password = "Password123!"

  source_image_reference {
    publisher = "MicrosoftWindowsServer"
    offer = "WindowsServer"
    sku = "2019-Datacenter"
    version = "latest"
  }

  os_disk {
    caching = "ReadWrite"
    storage_account_type = "Standard_LRS"
  }
}

10. Conclusion

So there we have it, a new Virtual Machine built in Azure using terraform. I personally really like the formatting and syntax compared to ARM templates. Of course I think terraform plan is where the magic is, which ARM template CURRENTLY has no answer for. I’m sure it will come soon enough to ARM, but in the mean time. Terraform really is a great solution for IaC (Infrastructure as Code).

Terraform - setting up domain DNS with cloudflare

Owen Davies — Mon, 17 Feb 2020 00:00:00 +0000

I need to verify my domain name with google to confirm that I own it. To do that I need to create a TXT record in the DNS, which is currently hosted in Cloudflare. I’ve been wanting to get my Cloudflare DNS into Terraform for a while, so in this article we’re going to import my existing DNS settings into the Terraform state file, and then add a new TXT record to verify with google.

If you need to install Terraform check out my article on Getting Started with Terraform.

Our plan today

Setup our new Terraform file
Get our Cloudflare API key
(Optional) Import our already existing DNS configuration from Cloudflare
Create our DNS resources in our Terraform file
Run our Terraform DNS configuration

1. Setup our new Terraform file

Create our new Terraform file in a new project directory

mkdir cloudflare-terraform
cd cloudflare-terraform
touch main.tf

Add the cloudflare provider to our terraform file

provider "cloudflare" {
  version = "~> 2.0"
  email = "you@example.com"
  api_key = "your-api-key"
}

Run terraform init

Run terraform init to setup the folder and download the providers declared in our main.tf terraform file.

terraform init

We can see that terraform has pulled down version 2.3 of the cloudflare provider.

2. Get our Cloudflare Global API key

Login to Cloudflare with your credentials
Click on My Profile at the top right
Click on the 3rd tab “API Tokens”
Click on View next to Global API Key
Type in your password again to confirm it’s you
Copy your API Key and store it somewhere super safe. Don’t give this to anyone that you don’t want to be able to access and modify your DNS.
Add the generated token to your main.tf file.

3. (Optional) Import our already existing DNS configuration from Cloudflare

1. Add the Zone to our Terraform file

Let’s query the Cloudflare API to get the Zone ID of the domain that we want to import the DNS records for. By querying https://api.cloudflare.com/client/v4/zones/ it will output all zones information.

curl -X GET "https://api.cloudflare.com/client/v4/zones/" -H "X-Auth-Key:YOUR_API_KEY" -H "X-Auth-Email:you@example.com" -H "Content-Type:application/json" | jq .

Find the domain (zone) that you want to import and copy the ID value, this is what we will use to import in to the state file.

2. Create a Zone entry in our Terraform file

Create a new resource with the type cloudflare_zone and name it something appropriate.

Mine looks like this:

resource "cloudflare_zone" "owendavies-net" {
 zone= "owendavies.net"
}

3. Import the zone in to the Terraform state file so that it knows about it.

Run the following command, replacing ZONE_ID with the ID that you copied in the first step.

terraform import cloudflare_zone.owendavies-net ZONE_ID

4. Check the Terraform file is correct

Now let’s run terraform plan and look at the output, if it complains and says that it’s going to make a change, that means that the cloudflare_zone resource you created in your terraform file is different to what terraform imported into the state file, it will tell you what changes there are, modify the resource until it says there are no changes to make when you run terraform plan.

terraform plan

5. Get DNS records from Cloudflare API

Run the following curl command replacing the YOUR_ZONE_ID and YOUR_API_KEY and email address.

curl -X GET "https://api.cloudflare.com/client/v4/zones/YOUR_ZONE_ID/dns_records" -H "X-Auth-Key:YOUR_API_KEY" -H "X-Auth-Email:you@example.com" -H "Content-Type:application/json" | jq . | tee importedcloudflare.json

This will query the Cloudflare api and return all the DNS records for the ZONE_ID that you specify, it will format it into JSON and save it to importedcloudflare.json

6. Create Terraform resource to match the above output

Open up the importedcloudflare.json file and create a new cloudflare_record resource matching the values in the importedcloudflare.json file.

When you have created the resource, find the id value in the importedcloudflare.json file.

7. Import the resource

Run the following command (replacing the ZONE_ID & DNS_RECORD_ID with your IDs) to import the resource into the terraform state file.

terraform import cloudflare_record.www-owendavies-net ZONE_ID/DNS_RECORD_ID

8. Test to see if the file matches the state

Run terraform plan which will show you what changes will be run (the differences between you state file and the main.tf file).

Make sure it says there are “No changes”, if there are any changes that it makes, go back and edit your resource in your main.tf file and try again.

terraform plan

My final terraform file:

In my case I have 2 CNAME records as well as the cloudflare zone itself (the domain).

provider "cloudflare" {
  version = "~> 2.0"
  email = "you@example.com"
  api_key = "your-api-key"
}

resource "cloudflare_zone" "owendavies-net" {
 zone= "owendavies.net"
}

resource "cloudflare_record" "owendavies-net" {
  zone_id = "ZONE_ID"
  name = "owendavies.net"
  value = "owen-davies.github.io"
  type = "CNAME"
  proxied = true
  ttl = 1
}

resource "cloudflare_record" "www-owendavies-net" {
  zone_id = "ZONE_ID"
  name = "www"
  value = "owen-davies.github.io"
  type = "CNAME"
  proxied = true
  ttl = 1
}

Now when I run terraform plan it says there are no changes to be applied

4. Create our DNS file in Terraform

Add a Cloudflare Zone resource

If you didn’t already do this during the (optional) import, you will need a cloudflare_zone resource.

resource "cloudflare_zone" "example" {
  zone = "example.com"
}

Add a record to the domain

In my case I’m adding a new TXT record to the domain that google wants to see to verify my domain ownership:

resource "cloudflare_record" "txt-owendavies-net" {
  zone_id = "ZONE_ID"
  name = "owendavies.net"
  value = "google-site-verification=JZMrzi_LveMVIyP8g5ZICIcPVYg39UPvyOjbLmRLuWU"
  type = "TXT"
  proxied = false 
  ttl = 1

5. Run our Terraform DNS configuration

Check the plan

Run terraform plan to see the changes that are going to happen

terraform plan

Check the output from the above terraform plan and make sure that all that terraform is going to do is add the new dns record, it shouldn’t be changing anything else if you added the resources correctly.

Make the changes

Now it’s time toe run terraform apply and make the change in Cloudflare.

terraform apply

Hopefully if everything has gone well, you now have your domain in Terraform, and a new record in the dns. You can now commit these changes to source control, and in the future can manage all your changes by Infrastructure as Code (IaC).

Using Docker for local SQL Server development

Owen Davies — Mon, 03 Feb 2020 00:00:00 +0000

Using docker images for development is a super power, you don’t need SQL installed on your development machine, yet you still have the full power of SQL there for you when you need it, and if you want specific versions for different projects you can simply spin up the version you want, and keep all the configuration inside a dockerfile in your repository for that project.

In this article we are going to:

Download the latest SQL Server docker image locally
Create a docker container and connect to it from SQL Server Management Studio
Setup a volume mount to store the database files outside of the container so we can persist the data when we tear down our container.
Set this up with a docker-compose file

1. Download the SQL Server docker image

We need to pull down the SQL Server docker image from the container registry with the following command:

docker pull mcr.microsoft.com/mssql/server

You should see something similar to the following image:

2. Run the container

Now that we have the sql server image locally, it’s time to run the container.

We need to specify several parameters to do this:

-e or –env - This is where we set the environment variables needed by the container

-p or –publish - This publishes a containers ports to the host ports specified

–name - Sets the name of the container

-d or –detach - Runs the container in the background (detached)

SQL Server needs us to set 2 environment variables, first to Accept the EULA agreement, and secondly to set the SA password.

docker run -e "ACCEPT_EULA=Y" -e "SA_PASSWORD=Sample123$" -p 1433:1433 --name sqlserver -d mcr.microsoft.com/mssql/server:latest

The output of this looks like this:

You can see that the command spun up the container and returned the new container ID, then stayed running in the background, leaving our terminal free to run more commands (this is the detached parameter we ran).

Check what containers are running

docker container ls

As you can see below, we now have our sqlserver container running with the same container ID that it displayed before, it shortened the ID tho, so we only see the first part.

Connect to SQL Server from Management Studio

Now lets connect to the container with SQL Management Studio, and create some tables.

Fire up Management Studio, and connect to localhost, with username sa and the password we set in the docker run command Sample123$

You should now be connected to the database running in the container, you can go ahead and add a database and a table, and insert some data. But when we remove the container, and then run the container again our data will be gone.

Remove the container

To remove the container we first must stop the container, and then we can use the docker container rm container_name command to remove it.

docker container stop /sqlserver 
docker container rm /sqlserver

Check the container has gone

Now when we run docker container ls we see the container is gone

docker container ls

Run the container again and check the data is no longer there

Now we can run the exact same docker run command as before:

docker run -e "ACCEPT_EULA=Y" -e "SA_PASSWORD=Sample123$" -p 1433:1433 --name sqlserver -d mcr.microsoft.com/mssql/server:latest

Check that it is running with docker container ls

Connect to SQL Server from Management Studio again as before

Connect to the database with Management studio and we can see that the database has gone. To persist this data we need to setup a volume mount point, and point our container there.

3. Create volume mount point for persistent data

We can create a volume mount point which will make a location on the local disk the stored location for the database files inside the container, this way each time we remove the container we will be able to retain

docker run -e "ACCEPT_EULA=Y" -e "SA_PASSWORD=Password123" -p 1433:1433 
--name sqlserver -v C:\Users\Owen\DockerVolumes\sqlserver:/var/opt/mssql/data 
-d mcr.microsoft.com/mssql/server:latest

4. Set this up with a docker-compose file

To save having to to remember/type all of these parameters each time we want to use our SQL database, we can do the exact same thing with a docker-compose.yaml file. This will mean we can just run a single simple command and all the parameters will be contained in the yaml file.

Realistically this is much easier, as generally I have so many containers I don’t want to have to remember everything about each of them.

Take the below code, save it as docker-compose.yml in your project root.

version: '3.4'

services:
  sqlserver:
    image: mcr.microsoft.com/mssql/server
    container_name: sqlserver
    environment:
      - ACCEPT_EULA=Y
      - SA_PASSWORD=Password123
    ports:
      - "1433:1433"
    volumes:
      - C:\Users\Owen\DockerVolumes\sqlserver:/var/opt/mssql/data

Now instead of having to type the long command from before all we have to type is the following:

docker-compose up

Connect to the database with Management Studio

Now you can connect to the database with Management Studio, add data and use as normal. You can then stop and remove the container and retain all the data in the volume you specified. This data can be test data that you store somewhere central so that your whole team can share it and of course version it, or somewhere local to you.

Getting started with Terraform

Owen Davies — Thu, 10 Oct 2019 00:00:00 +0000

Terraform allows you to setup Infrastructure as expressed by code, (Infrastructure As Code IaC). It’s platform independent, and supports a wide range of different things, from cloud providers like Azure, AWS, GCP, to databases to DNS, pretty much everything.

What we're going to do:

Install Terraform on Debian
Install Docker
Create a terraform script to download a docker image of jekyll
Modify our terraform script to spin up the jekyll container.

Setting up your workstation

I’m using Debian 10 (buster) for my installation here but Terraform can literally be installed on anything since it is written in golang.

Install Terraform

Download the most recent package for your operating system from the terraform download page

curl -O https://releases.hashicorp.com/terraform/0.12.10/terraform_0.12.10_linux_amd64.zip

Unzip the package into our /usr/local/bin to install it locally

sudo unzip terraform_0.11.10_linux_amd64.zip -d /usr/local/bin

Confirm it’s working

Now lets run the following command to check that it’s working and available

terraform -version

You should receive the following version output Terraform v0.12.10

As you can see Terraform is still not at version 1, it’s still in beta and is in active development and each version can and has change quite drastically. So just be aware of this.

Install Docker

If you already have docker installed locally you can skip these steps

Install the prerequisites to add a new repository over https

sudo apt-get update && apt-get install apt-transport-https ca-certificates curl software-properties-common

Import the GPG key from docker

curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add -

Add the docker apt repository to our repository list

sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian $(lsb_release -cs) stable"

update apt and install Docker CE

sudo apt-get update
sudo apt-get install docker-ce

Creating our first Terraform script

Pull a docker image from the Terraform Docker provider

Create our new script

mkdir terraform-gettingstarted
touch main.tf

Create the script content

Now open up the main.tf in your favourite text editor and add the following:

# Download the latest jekyll image
resource "docker_image" "image_id" {
  name = "jekyll:latest"
}

Initialise our Terraform directory

This command will initialise our directory by creating a .terraform directory. And download any providers that are necessary.

terraform init

This will download the terraform docker provider. The providers are stored in the .terraform directory. If you look in .terraform/plugins/linux_amd64/ you will see the terraform provider.

View the changes that will happen

terraform plan

By running terraform plan we will see all the changes that are going to be applied.

Run our new script

Once we have viewed the plan, and we are happy it isn’t doing something we didn’t expect we will run the following command:

terraform apply

This will download the docker image that we specified in the main.tf file.

See what was applied

terraform show

This will return the docker image id that we asked it to download

Interpolation syntax

Interpolational syntax is a way for you to reference variables and other objects from within your infrastructure.

The documentation on terraform.io explains this in depth. (Link)[https://www.terraform.io/docs/configuration/interpolation.html]

Create a docker container of the image

Let’s add to our main.tf file and actually create a container using the image above, referenced with interpolation syntax

Open main.tf and edit

Let’s go ahead and open the main.tf file that we created above, and add the following code below:

# Start the container
resource "docker_container" "container_id" {
  name = "blog"
  image = "${docker_image.image_id.latest}"
  ports {
    internal = "2368"
    external = "80"
  }
}

You can see the $ sign, this is how we are referencing the image above.

Run our changes

Now lets view the plan of the changes by running the below:

terraform plan

You’ll see that it is referencing our id of our image

Now lets apply our changes

terraform apply

Now if you run:

docker ps -a

You will see our container is running with the ports open.

If you now browse to port 80 in a web browser you will see the jekyll blog setup.

Clear terraform infrastructure

To remove all the infrastructure we just created above we can use the destroy command.

terraform destroy

It will tell you what you are about to destroy, and require you to type yes.

Replacing SSMTP with MSMTP in Docker

Owen Davies — Mon, 26 Aug 2019 00:00:00 +0000

Trying to build a new image of a Docker file that I’ve been running for months, and I’m greeted with this error:

Package ssmtp is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Package 'ssmtp' has no installation candidate

Looks like sSMTP is no longer maintained, and has been orphaned since 19th March 2019. MSMTP is the suggested replacement. So let’s get replacing.

The plan

Install mSMTP in our Docker image
Configure mSMTP to our smtp server (in this case Office 365)
Tell PHP to use mSMTP for email instead of sSMTP
Send a test email
Final source

1. Install mSMTP in our Docker image

We’re going to start with the php:7.30-apache docker image

FROM php:7.3-apache

Lets install msmtp with apt-get, and tidy up afterwards:

apt-get update && apt-get install ssmtp -y && \
rm -rf /var/lib/apt/lists/*

2. Configure mSMTP to our smtp server (in this case Office 365)

We need to setup a configuration file for msmtp to have the smtp details. The msmtp config location is in /etc/

Let’s create a new file called msmtprc, and add the following configuration:

account default
host smtp.office365.com
port 587
tls on
tls_starttls on
tls_trust_file /etc/ssl/certs/ca-certificates.crt
tls_certcheck on
auth on
user user@domain.com
password "YourAwesomeStr0ngP4zzw0rd"
from "user@domain.com"
logfile /var/log/msmtp.log

This is for Office 365, change the smtp details to gmail or your own smtp server.

Copy the config into our Docker image

COPY msmtprc /etc/msmtprc

Make the msmtprc config file readable/writable by its owner

RUN chmod 600 /etc/msmtprc

3. Tell PHP to use mSMTP for email instead of sSMTP

Now that we have mSMTP setup, we need to tell PHP to use it.

You can get a copy of the default php.ini file from the docker image we have just created. It’s located in /usr/local/etc/php/php.ini-production

docker cp <CONTAINER_ID>:/usr/local/etc/php/php.ini.production .

Open up your php.ini file, and make the following change to the sendmail_path variable:

sendmail_path = /usr/bin/msmtp -t

Copy your php.ini file to the Docker image php config location:

COPY php.ini $PHP_INI_DIR/php.ini

4. Send a test email

With all that setup we should now be able to send an email.

1. Build the Docker image

docker build . -t php-msmtp-setup:latest

2. Run the container

docker run -d -p 80:80/tcp php-msmtp-setup:latest

3. Log into the container

Run the following command to find the CONTAINER ID of our new container

docker container ls

Run the following command to login to the container

docker exec -ti <CONTAINER_ID> /bin/bash

4. Send an email directly with msmtp

echo -e "Subject: Test Mail\r\n\r\nThis is a test mail, let me know if this works" |msmtp --debug --from from@yourdomain.com -t to@someone.com

5. Send an email with php

php -r "mail('to@domain.com','Test Mail from PHP', 'This is a test mail from PHP, let me know if this works');"

5. Final source

Our Docker file

FROM php:7.3-apache

RUN apt-get update && apt-get install msmtp -y && \
    rm -rf /var/lib/apt/lists/*

COPY msmtprc /etc/msmtprc
RUN chmod 600 /etc/msmtprc

COPY php.ini $PHP_INI_DIR/php.ini

EXPOSE 80

The changes to our php.ini file

sendmail_path = /usr/bin/msmtp -t

Our msmtprc file

account default
host smtp.office365.com
port 587
tls on
tls_starttls on
tls_trust_file /etc/ssl/certs/ca-certificates.crt
tls_certcheck on
auth on
user user@domain.com
password "YourAwesomeStr0ngP4zzw0rd"
from "user@domain.com"
logfile /var/log/msmtp.log

Get latest Azure API Version in ARM templates

Owen Davies — Thu, 24 May 2018 00:00:00 +0000

The following script will crawl through all your ARM templates in a folder and check for the latest Azure API version.

This is useful if you want to make sure you are staying up to date with the latest API.

Currently there is no way to be notified if your ARM templates have a newer version of the API. And if you are anything like me you have hundreds of nested ARM templates to keep up to date. Keeping track of all the resources you are using in all your files is difficult.

This will take into account if you hold your resource API versions inside variables, parameters or directly in the apiVersion itself.

Examples of where it looks

I store Resource API versions in multiple places depending on a few different reasons in my ARM templates. I try to store in Variables as much as possible for reuse, but often I need to store in parameters so that I can pass in to a nested template.

The following JSON ARM Template file has examples of storing the API version in 3 places:

Directly in the resource
In Variables
In Parameters

{
  "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "apiVersionDisks": "2017-06-01"
  },
  "variables": {
    "apiVersionResourcesDeployment": "2017-05-10",
    "apiVersionDisks": "2017-04-01",
    "apiVersionStorage": "2017-04-01"
  },
  "resources": [{
      "apiVersion": "[variables('apiVersionResourcesDeployment')]",
      "type": "Microsoft.Resources/deployments"
    },
    {
      "name": "Name1",
      "type": "Microsoft.Storage/storageAccounts",
      "apiVersion": "[variables('apiVersionStorage')]",
    },
    {
      "name": "Name1",
      "type": "Microsoft.Compute/disks",
      "apiVersion": "[parameters('apiVersionDisks')]",
    },
    {
      "name": "Name1",
      "type": "Microsoft.Compute/disks",
      "apiVersion": "2015-06-15",
    }
  ]
}

The Script!

This script will check all 3 locations for API versions to check against.

$templatePath = 'C:\YOUR\ARM\PATH'

$listOfResources = @()

Get-ChildItem $templatePath -Recurse -Filter *.json |
Foreach-Object {

    $content = (Get-Content $_.FullName | ConvertFrom-Json)

    foreach($resource in $content.resources)
    {
        $resourceObject = New-Object -TypeName pscustomobject -Property @{
            "fileName" = $_.FullName
            "resourceName" = $($resource.type)
            "apiVersion" = ""
            "apiVersionVariableName" = ""
            "apiVersionParameterName" = ""
            "latestApiVersion" = ""
        }

        if($($resource.apiVersion) -like '*variables*')
        {
            $apiVariable = ($resource.apiVersion).replace("[variables('","").replace("')]","")

            $resourceObject.apiVersionVariableName = $apiVariable
            $resourceObject.apiVersion = $($($content.variables).$apiVariable)
        }
        elseif($resource.apiVersion -like '*parameters*')
        {
            $apiVariable = ($resource.apiVersion).replace("[parameters('","").replace("')]","")

            $apiParameter = ($resource.apiVersion).replace("[parameters('","").replace("')]","")

            $resourceObject.apiVersionParameterName = $apiParameter
            $resourceObject.apiVersion = $($($content.parameters).$apiParameter)
        }
        else
        {
            $resourceObject.apiVersion = $($resource.apiVersion)
        }

        $listOfResources += $resourceObject
    }
}

$resources = $listOfResources.resourceName | Sort-Object | Get-Unique

Foreach($i in $resources)
{   
    $providerNamespace,$resourceTypeName = $i.split("/",2)
    $latestApiVersion = ((Get-AzureRmResourceProvider -ProviderNamespace $providerNamespace).ResourceTypes | Where-Object ResourceTypeName -eq $resourceTypeName).ApiVersions | select -First 1

    $editObj = $listOfResources | where {$_.resourceName -eq $i}
    foreach($r in $editObj)
    {
        $r.latestApiVersion=$latestApiVersion   
    }

}

Foreach($resource in $listOfResources)
{
    if($resource.latestApiVersion -ne $resources.apiVersion)
    {
        Write-Host "`nThere is a new ApiVersion for $($resource.resourceName)"
        Write-Host "Filename: [$($resource.fileName)]"
        Write-Host "Current ApiVersion: [$($resource.apiVersion)]"
        Write-Host "Latest ApiVersion: [$($resource.latestApiVersion)]"
    }
}

Please let me know if you’ve found this useful! :-)