DEV Community: Graham Lyons

A Zero-Fricton Terraform Primer

Graham Lyons — Mon, 04 Jun 2018 16:30:05 +0000

What is Terraform and why should you care? How can you learn about it without having to provision real stuff in your Amazon account?

Infrastructure as Code

Back in the days when we started to get away from dealing with real servers in a rack somewhere a number of cloud infrastructure providers appeared, offering access to their virtual estate via a console and - if they were any good - an API.

I've worked in lots of different places which used these cloud providers (OK, mainly Amazon Web Services) and I've seen about as many different ways to manage the infrastructure.

The worst way is of course via the console. Clicking in a GUI is not a good way to make processes repeatable or scalable. Beyond that the options include: CloudFormation, a service from AWS (only works with AWS); HEAT templates from OpenStack, which is very similar to CloudFormation (only works with OpenStack); Chef Provisioning, which is no longer supported by Chef; and of course, Terraform.

All of these tools allow you to define what infrastructure you'd like - virtual machines, load balancers, block storage, databases etc. - as some kind of machine and human readable language (CloudFormation uses JSON, for example). The tool can interpret the code and create the desired resources; the code can be checked into version control and tracked like application code.

Terraform

Terraform is an Infrastructure as Code tool from Hashicorp, who produce other popular pieces of software such as Vagrant. It uses a declarative language, Hashicorp Configuration Language (HCL), to define the desired state of your cloud infrastructure. From this code it generates a dependency graph of the resources and, when run against one or more providers, walks that graph and ensures that the resources exist and are configured as defined.

Installation

The terraform executable is delivered as a single file so it just needs to be downloaded and put onto your system's path. On a *nix system /usr/local/bin/ is a good place as it's often already on your $PATH environment variable. From https://www.terraform.io/ find the 'Download' link and select the most appropriate version for your system. Download it, unzip it and put the terraform file somewhere you can run it, for example /usr/local/bin/.

Check that it's installed successfully and find out what version you're running - mine is:

$ terraform --version
Terraform v0.11.7

Defining Some Infrastructure

Now that Terraform is installed, we need to define what infrastructure we want it to create. We use HCL to define resources for different providers. A simple one is the random provider, which generates random data to use, for example, as server names. It doesn't operate against a cloud provider and requires no API keys etc. so is good to illustrate Terraform's workflow. We'll also use the local provider to write that random data out to a file.

Put the following into a file called example.tf:

variable "name_length" {
  type    = "string"
  default = "2"
  description = "The number of words to put into the random name"
}

resource "random_pet" "server" {
  length = "${var.name_length}"
}

resource "local_file" "random" {                                                   
  content     = "${random_pet.server.id}"                                        
  filename = "${path.module}/random.txt"                                         
}

output "name" {
  value = "${random_pet.server.id}"
}

Aside: Code Organisation

Terraform will look in the directory you tell it to (the current directory by default) and find all of the *.tf files - sub-directories are ignored. It'll treat the files it finds as one single definition and draw a graph of all the resources. It's common to see the variables and outputs split into different files to make it clear where to find them. It's also common, and a good idea, to split code into modules but we won't worry about that today.

In the same directory run: terraform init. You should see output which looks a bit like this:

$ terraform init

Initializing provider plugins...
- Checking for available provider plugins on https://releases.hashicorp.com...
- Downloading plugin for provider "random" (1.3.1)...
- Downloading plugin for provider "local" (1.1.0)...

The following providers do not have any version constraints in configuration,
so the latest version was installed.

To prevent automatic upgrades to new major versions that may contain breaking
changes, it is recommended to add version = "..." constraints to the
corresponding provider blocks in configuration, with the constraint strings
suggested below.

* provider.local: version = "~> 1.1"
* provider.random: version = "~> 1.3"

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

Terraform has looked at all of the *.tf files, determined which providers are being used and has downloaded the appropriate plugins. These are stored in the .terraform/ directory which has been created in the current path.

Planning changes

One amazing feature of Terraform is the ability to preview changes to get an idea of what's actually going to happen when you apply them. Is this change going to modify my loadbalancer in-place or is it going to destroy it and recreate it, taking my application offline for precious minutes?

Let's see what that looks like:

$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.


------------------------------------------------------------------------

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  + local_file.random
      id:        <computed>
      content:   "${random_pet.server.id}"
      filename:  "/home/vagrant/workspace/tfdemo/random.txt"

  + random_pet.server
      id:        <computed>
      length:    "2"
      separator: "-"


Plan: 2 to add, 0 to change, 0 to destroy.

------------------------------------------------------------------------

Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.

This is the first time we're running it so all the resources we've specified are being created - see the + next to their name in the output.

Also pay attention to the "Note" - we haven't saved this plan so whilst we've got a good idea what Terraform will do when we apply it, it's not guaranteed to try to do the same thing. We can store the plan in a file with a unique name by appending a timestamp i.e. terraform plan -out "plan-$(date +%s)".

Running that we instead get this at the end of the output:

...
This plan was saved to: plan-1527707537

To perform exactly these actions, run the following command to apply:
    terraform apply "plan-1527707537"

If we're happy with this plan then we can apply it for real.

Applying Changes

When we run terraform apply and pass it the plan file we get output which looks like the following:

$ terraform apply "plan-1527707537"
random_pet.server: Creating...
  length:    "" => "2"
  separator: "" => "-"
random_pet.server: Creation complete after 0s (ID: leading-piranha)
local_file.random: Creating...
  content:  "" => "leading-piranha"
  filename: "" => "/home/vagrant/workspace/tfdemo/random.txt"
local_file.random: Creation complete after 0s (ID: 681f312327eab60da028b397bc85af8682fdc185)

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

Outputs:

name = leading-piranha

The Random provider gave us a pet name consisting of 2 words and the output directive showed it at the end of the program. The local_file resource wrote the name into a file called random.txt in the current directory:

$ cat random.txt
leading-piranha[vagrant@localhost tfdemo]$

Making More Changes

Hmmm, there's no newline at the end of the file. I'd prefer it to be formatted with one so I'll add one into the HCL. The content in the local_file can be changed to, with a \n appended:

...
  content     = "${random_pet.server.id}\n"                                      
...

If we plan the changes we'll see that only the file is scheduled to change. There's no reason for the random_pet resource to be changed at all so Terraform uses it as it is.

$ terraform plan -out "plan-$(date +%s)"

Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

random_pet.server: Refreshing state... (ID: leading-piranha)
local_file.random: Refreshing state... (ID: 681f312327eab60da028b397bc85af8682fdc185)

------------------------------------------------------------------------

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
-/+ destroy and then create replacement

Terraform will perform the following actions:

-/+ local_file.random (new resource required)
      id:       "681f312327eab60da028b397bc85af8682fdc185" => <computed> (forces new resource)
      content:  "leading-piranha" => "leading-piranha\n" (forces new resource)
      filename: "/home/vagrant/workspace/tfdemo/random.txt" => "/home/vagrant/workspace/tfdemo/random.txt"


Plan: 1 to add, 0 to change, 1 to destroy.

------------------------------------------------------------------------

This plan was saved to: plan-1528126805

To perform exactly these actions, run the following command to apply:
    terraform apply "plan-1528126805"

Applying the changes from the plan we've just made can cating the file again shows that there's now a newline at the end:

$ terraform apply "plan-1528126805"
local_file.random: Destroying... (ID: 681f312327eab60da028b397bc85af8682fdc185)
local_file.random: Destruction complete after 0s
local_file.random: Creating...
  content:  "" => "leading-piranha\n"
  filename: "" => "/home/vagrant/workspace/tfdemo/random.txt"
local_file.random: Creation complete after 0s (ID: 82c2862c8ae7053eb94b7aa498265335c5d22b22)

Apply complete! Resources: 1 added, 0 changed, 1 destroyed.

Outputs:

name = leading-piranha

$ cat random.txt
leading-piranha

Variables

In the example.tf file you can see that we declared a variable called name_length and referenced it in the random_pet resource (length = "${var.name_length}"); why not just hard code that number?

To aid code reuse, Terraform lets us pass in different values for the variables we've defined. We use the -var flag and the name of the variable, like this:

$ terraform plan -var name_length=3
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

random_pet.server: Refreshing state... (ID: leading-piranha)
local_file.random: Refreshing state... (ID: 82c2862c8ae7053eb94b7aa498265335c5d22b22)

------------------------------------------------------------------------

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
-/+ destroy and then create replacement

Terraform will perform the following actions:

-/+ local_file.random (new resource required)
      id:        "82c2862c8ae7053eb94b7aa498265335c5d22b22" => <computed> (forces new resource)
      content:   "leading-piranha\n" => "${random_pet.server.id}\n" (forces new resource)
      filename:  "/home/vagrant/workspace/tfdemo/random.txt" => "/home/vagrant/workspace/tfdemo/random.txt"

-/+ random_pet.server (new resource required)
      id:        "leading-piranha" => <computed> (forces new resource)
      length:    "2" => "3" (forces new resource)
      separator: "-" => "-"


Plan: 2 to add, 0 to change, 2 to destroy.

------------------------------------------------------------------------

Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.

The plan tells us again what's going to happen - both resources will be destroyed and others created in their place. The file has to be recreated in this case because it's dependent on the value from random_pet. Terraform works this out from the dependency graph it generates - it can work out what it needs to recreate based on what's changed and what depends on that.

Aside: Dependency Graph

The dependency graph for your infrastructure can be seen in the [DOT language](https://en.wikipedia.org/wiki/DOT_(graph_description_language) by running terraform graph. If you've got Graphviz installed then you can render it by piping the output straight to the dot program:

$ terraform graph | dot -Tpng -o tfdemo.png

This is a really simple example and no critical infrastructure is at stake so we can apply these changes without saving to a plan file by simply running terraform apply and either typing "yes" at the prompt or passing the -auto-approve flag:

$ terraform apply -var name_length=3 -auto-approve
random_pet.server: Refreshing state... (ID: leading-piranha)
local_file.random: Refreshing state... (ID: 82c2862c8ae7053eb94b7aa498265335c5d22b22)
local_file.random: Destroying... (ID: 82c2862c8ae7053eb94b7aa498265335c5d22b22)
local_file.random: Destruction complete after 0s
random_pet.server: Destroying... (ID: leading-piranha)
random_pet.server: Destruction complete after 0s
random_pet.server: Creating...
  length:    "" => "3"
  separator: "" => "-"
random_pet.server: Creation complete after 0s (ID: scarcely-intense-mammoth)
local_file.random: Creating...
  content:  "" => "scarcely-intense-mammoth\n"
  filename: "" => "/home/vagrant/workspace/tfdemo/random.txt"
local_file.random: Creation complete after 0s (ID: a3f2f24388d1e4ddd72872a833469002f2ad5b75)

Apply complete! Resources: 2 added, 0 changed, 2 destroyed.

Outputs:

name = scarcely-intense-mammoth

Note that we need to pass the same parameters to the apply phase that we passed in planning. This is one very good reason to save the plan and use that when running apply.

State

Along with the .terraform/ directory which stores the provider plugins you'll notice that there's a terraform.tfstate file there too. A quick examination shows that it's text, which we can read!

$ file terraform.tfstate
terraform.tfstate: ASCII text

This is the state of our resources, in JSON format. The state represents Terraform's view of the defined resources. If you run the plan command with the same arguments in the same directory then Terraform will tell us that there's nothing to do:

$ terraform plan -var name_length=3

Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

random_pet.server: Refreshing state... (ID: scarcely-intense-mammoth)
local_file.random: Refreshing state... (ID: a3f2f24388d1e4ddd72872a833469002f2ad5b75)

------------------------------------------------------------------------

No changes. Infrastructure is up-to-date.

This means that Terraform did not detect any differences between your
configuration and real physical resources that exist. As a result, no
actions need to be performed.

If you're running Terraform to create your cloud infrastructure then make sure the state is committed to source control or - particularly if you're working with other engineers - persisted in one of the supported backends.

Wrapping Up

This illustrates a typical workflow for Terraform: code -> plan -> apply -> commit. To make it as easy as possible to follow along we've used providers which only operate locally, but if you added an aws_instance resource then the random server name we've generated could easily be used to set the Name tag on the EC2 instance. Terraform will pick up the standard AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables and your workflow remains unchanged as you provision real infrastructure.

Why My Development Environment is the Best

Graham Lyons — Wed, 11 Apr 2018 16:04:28 +0000

Why My Development Environment is the Best

Or more accurately, what works for me right now.

As software developers we spend a lot of our day at the keyboard, typing. It's natural that we spend time making that environment a pleasant and productive place in which to work.

Local development environments are often very personal and are tweaked and customised to the tastes of the individual developer. There are a finite number of editors and IDEs but an almost infinite combination of plugins, themes and customisations within those. In some of the more recent evolutions of my local environment I've rebelled against the culture of personalisation.

My Tools

Almost all the software I write gets deployed and run in production on a server running some flavour of Linux. I run everything locally on a virtual machine which is set up as close to production as I can get. Over the years this has been invaluable for catching bugs before they even get to a staging environment or reproducing production problems under safe conditions.

To manage and run virtual machines (VMs) I use the combination of VirtualBox and Vagrant. Once I've installed those I'm (nearly) ready to run a VM. I also use the vagrant-vbguest plugin which manages the installation of the VirtualBox Guest Additions in the VM itself. These are used to share a workspace directory between my host machine and the Linux VM: vagrant plugin install vagrant-vbguest

The configuration for a VM managed by Vagrant can be stored as code so here's an example on GitHub of the main environment I use at the moment: https://github.com/grahamlyons/centos-dev

To run it: clone the repo, start the VM and then connect to it:

git clone git@github.com:grahamlyons/centos-dev.git
cd centos-dev
vagrant up && vagrant ssh -c 'tmux attach || tmux'

The instructions in the Vagrantfile start from a CentOS 7 base image and:

specify a fixed IP address which can be used to refer to the VM
mount a local ~/workspace/ directory inside the VM (at the same path)
install some base packages, e.g. tmux, vim, git etc.
install and sets up Docker
copy some local configuration and credential files into the VM

The SSH connection command also puts me into either an existing tmux session or starts a new one. Tmux is a great tool for creating multiple tabs and panes in a single SSH session. I don't use any extra configuration for it beyond what's installed on CentOS with the package.

Benefits of a Virtual machine

The first time I was introduced to working inside a VM I was sold on it completely. It made so much sense to me to be as close to production as possible and installing software on Linux, using a proper package manager, is so much nicer than on OSX (or Windows - in the depths of my memory).

With the shared folder - ~/workspace/ in my case - I can use whatever editor I like on my host OS and the changes will always be inside the VM, ready to run.

Running a VM has also saved me from completely destroying my machine on more than one occasion, the worst of which was an accidental rm -rf / run as root. Always having a working machine that you can use to search for help with fixing problems is incredibly useful. If things get really back you can just destroy it and start again from your known good state.

Drawbacks of a Virtual machine

Using a VM is not a perfect solution and running code inside a directory shared between the host machine and the guest VM can give performance problems. The shared directory is great for use with a simple editor but with an IDE, which will want to run your code for you, it can be complicated or impossible to run the code inside the virtual machine.

My Editor

The first editor I ever used when I started working professionally was Homesite, which betrays my vintage. After I was no longer able to get hold of that I looked around for something else and saw something called Vim recommended. I was interested and downloaded it and opened it up. After a few minutes I managed to work out how to quit it and didn't open it up again for a year or two.

Vim

After throwing myself into Vim I now use it almost exclusively. Where I use something else I try to find a Vim key-bindings plugin for it. There is a big learning curve for Vim, and vimtutor was a big help, but now that I'm familiar with the movements and actions nothing lets me manipulate text faster.

Plugins and Configuration

My .vimrc file has gone through many iterations and it's now roughly 10 lines:

set expandtab
set shiftwidth=4
set tabstop=4

set modeline
set modelines=5

set nu
set colorcolumn=80

syntax enable

let g:netrw_liststyle=3
if exists("*netrw_gitignore#Hide")
    let g:netrw_list_hide=netrw_gitignore#Hide()
endif

I use spaces instead of tabs (who wouldn't?); (for OSX, where it was turned off) it's set to read Vim settings from the tops of files (http://vim.wikia.com/wiki/Modeline_magic); line numbers and syntax highlighting are on; there's a column at 80 characters to stop my lines from getting too long and I've set directory listings to look like a tree.

Everything else I use in Vim is vanilla. Just getting used to the defaults allows me to move between different machines more easily and there's less of my clever customisation and tweaking to remember and more widely available documentation to refer to.

Other Software

Over the past couple of years I've started running almost everything inside Docker containers so I'm yum installing less and less. Almost everything is available in an image from Docker Hub and it's so fast to start up once it's been pulled down that it just makes so much sense. Running different versions of e.g. Node, Ruby or Python side by side is much simpler.

I still use yum to install utility packages like telnet or jq, and they'll often make it into the configuration in the Vagrantfile.

This Works for Me for Now

So this is how my machine is set up at the moment, and it works really well for me. I run OSX and spend most of the time in Terminal, with one tab for my VM connection. I use Vim both on OSX and on the VM, and the same for Git.

It works well but I am always making changes. The introduction of Docker is more recent and is becoming more prominent. Let's see what this looks like in a year.

Machine Learning for the Lazy Beginner

Graham Lyons — Mon, 12 Feb 2018 13:33:18 +0000

Machine Learning for the Lazy Beginner

This article was prompted by a tweet I saw which asked for a walkthrough on training a machine learning service to recognise new members of 3 different data sets.

@rem: Being lazy here: I'm after a (machine learning) service that I can feed three separate datasets (to train with), and then I want to ask: "which dataset is this new bit of content most like".

Is there a walkthrough/cheatsheet/service for this?

My first thought was that this sounds like a classification task, and the idea that there are 3 sets of data should be the other way round: there is one set of data and each item in the set has one of 3 labels.

I didn't have a walkthrough in mind but I do know how to train a classifier to perform this exact task, so here is my walkthrough of classifying text documents using Javascript.

Do You Have Adequate Supervision?

Machine learning can be classified (no pun intended) as either supervised or unsupervised. The latter refers to problems where the data you feed to the algorithm has no predetermined label. You might have a bunch of text documents and you want to find out if they can be grouped together into similar categories - that would be an example of clustering.

Supervised learning is where you know the outcome already. You have set of data in which each member fits into one of n categories, for example a set of data on customers to your e-commerce platform, labelled according to what category of product they're likely to be interested in. You train your model against that data and use it predict what new customers might be interested in buying - this is an example of classification.

Get in Training

For the classification task we've said that we "train" a model against the data we know the labels for. What that means is that we feed each instance in a dataset into the classifier, saying which label it should have. We can then pass the classifier a new instance, to which we don't know the label, and it will predict which class that fits into, based on what it's seen before.

There's a Javascript package called natural which has several different classifiers for working with text documents (natural language). Using one looks like this:

const { BayesClassifier } = require('natural');
const classifier = new BayesClassifier();

// Feed documents in, labelled either 'nice' or 'nasty'
classifier.addDocument('You are lovely', 'nice');
classifier.addDocument('I really like you', 'nice');
classifier.addDocument('You are horrible', 'nasty');
classifier.addDocument('I do not like you', 'nasty');

// Train the model
classifier.train();

// Predict which label these documents should have
classifier.classify('You smell horrible');
// nasty
classifier.classify('I like your face');
// 'nice'
classifier.classify('You are nice');
// 'nice'

We add labelled data, train the model and then we can use it to predict the class of text we haven't seen before. Hooray!

Performance Analysis

Training a machine learning model with a dataset of 4 instances clearly isn't something that's going to be very useful - its experience of the problem domain is very limited. Machine learning and big data are somewhat synonymous because the more data you have the better you can train your model, in the same way that the more experience someone has of a topic the more they're likely to know about it. So how do we know how clever our model is?

The way we evaluate supervised learning models is to split our data into a training set and a testing set, train it using one and test it using the other (I'll leave you to guess which way round). The more data in the training set the better.

When we get the predictions for our test data we can determine if the model accurately predicted the class each item is labelled with. Adding up the successes and errors will give us numbers indicating how good the classifier is. For example, successes over total instances processed is our accuracy; errors divided by the total is the error rate. We can get more in-depth analysis by plotting a confusion matrix showing actual classes against predictions:

		Actual
		nice	nasty
Predicted	nice	21	2
	nasty	1	10

This is really valuable for assessing performance when it's OK to incorrectly predict one class but not another. For example, when screening for terminal diseases it would be much better to bias for false positives and have a doctor check images manually rather than incorrectly give some patients the all clear.

Train On All the Data

One way to train with as much data as possible is to use cross validation, where we take a small subset of our data to test on and use the rest for training. A commonly used technique is k-fold cross validation, where the dataset is divided into k different subsets (k can be any number, even the number of instances in the dataset), each of which is used as a testing set while the rest is used for training - the process is repeated until each subset has been used for testing i.e. k times.

Tweet Data Example

I've put together an example using the natural Javascript package. It gets data from Twitter, searching for 3 different hashtags, then trains a model using those 3 hashtags as classes and evaluates the performance of the trained model. The output looks like this:

$ node gather.js
Found 93 for #javascript
Found 100 for #clojure
Found 68 for #python

$ node train.js
{ positives: 251, negatives: 10 }
Accuracy: 96.17%
Error: 3.83%

The code is on Github: classification-js

Machine Learning is That Easy?!

Well, no. The example is really trivial and doesn't do any pre-processing on the gathered data: it doesn't strip out the hashtag that it searched for from the text (meaning that it would probably struggle to predict a tweet about Python that didn't include "#python"); it doesn't remove any stop words (words that don't really add any value, such as a or the. In fact, natural does this for us when we feed documents in, but we didn't know that...); it doesn't expand any of the shortened URLs in the text (learnjavascript.com surely means more than t.co). We don't even look at the gathered data before using it, for example graphing word-frequencies to get an idea of what we've got: are some of the "#python" tweets from snake enthusiasts talking about their terrariums?

To miss-quote Tom Lehrer, machine learning is like a sewer: what you get out depends on what you put in.

Wrapping Up

The aim of this article was to give an overview of how a machine learning model is trained to perform a classification task. Hopefully, for the beginner, this goes some way to lifting the lid on some of that mystery.

Cover image by: https://www.flickr.com/photos/mattbuck007/

Everything You Need To Know About Networking On AWS

Graham Lyons — Sun, 28 Jan 2018 19:40:07 +0000

Everything You Need To Know About Networking On AWS

Disclaimer: I'm not a network engineer and never have been - a tame network engineer has been consulted to ensure factual and terminological accuracy. The following is an in-exhaustive run down of everything I've learnt from building and using network infrastructure on Amazon Web Services. If you find you have no reference point for this information then have a poke around the "VPC" section of the AWS control panel (or get in touch to tell me I'm talking nonsense).

Parts of a Network You Should Know About

If you're running infrastructure and applications on AWS then you will encounter all of these things. They're not the only parts of a network setup but they are, in my experience, the most important ones.

VPC

A virtual private cloud - VPC - is a private network space in which you can run your infrastructure. It has an address space (CIDR range) which you choose e.g. 10.0.0.0/16. This determines how many IP addresses you can assign within the VPC. Each server you create inside the VPC will need an IP address so this address space defines the limit of how many resources you can have within the network. The 10.0.0.0/16 address space can use the addresses from 10.0.0.0 to 10.0.255.255, which is 65,536 IP addresses.

The VPC is the basis of your network on AWS and all new accounts include a default VPC with a subnets in each availability zone.

+---------------+
|     VPC       |     The Internet
|               |
|               |
|  10.0.0.0/16  |
|               |
|               |
+---------------+

Subnets

A subnet is a section of your VPC, with its own CIDR range and rules on how traffic can flow. Its CIDR range has to be a subset of the VPC's, for example 10.0.1.0/24 which would allow for IPs from 10.0.1.0 to 10.0.1.255 giving 256 possible IP addresses.

Subnets are often denominated as 'public' or 'private' depending on whether traffic can reach them from outside the VPC (the Internet). This visibility is controlled by the traffic routing rules and each subnet can have its own rules.

A subnet has to be in a specific availability zone within a region so it's good practice to have a subnet in each zone. If you plan to have public and private subnets then there should be one of each per availability zone.

+---------------------------+
|            VPC            |
|                           |
+------------+ +------------+
||  Subnet 1 | |  Subnet 2 ||
||10.0.1.0/24| |10.0.2.0/24||
||           | |           ||
|------------+ +------------|
+---------------------------+

Availability Zones

We've said that there should be subnets per availability zone, but what does that actually mean?

Each AWS region is divided into 2 or more different zones which, between them, aim to guarantee a very high level of availability for that region. Essentially, at least one zone should be able to operate, even if others suffer outages (🔥).

+----------+          +----------+
|us-ea)t-1a|          |us-east-1b|
|_____(____|          |__________|
|     )    |          |          |
|   ( &()  |          |    ✔     |
|  ) () &( |          |    8-)   |
+----------+          +----------+

Routing Tables

A routing table contains rules about how IP packets in the subnets can travel to different IP addresses. There is always a default route table which will only allow traffic to travel locally, within the VPC. If a subnet has no routing table associated with it then it uses the default one. These would be 'private' subnets.

If you want external traffic to be able to get to a subnet then you need to create a routing table with a rule explicitly allowing this. Subnets associated to that routing table would be 'public'.

All of the subnets in the default VPCs are associated with a route table which makes them public.

Internet Gateways

The routing table which makes a subnet public needs to reference an Internet gateway to allow the flow of external IP packets into and out of the VPC. You create your Internet gateway and then create a rule which says that packets to 0.0.0.0/0 - all IP addresses - need to go to there.

          Route table
         +-------------------+
         | 10.0.0.0/8: local | Requests within the VPC go over local connections.
      +--+ 0.0.0.0/0: ig-123 | Requests to any other IPs go via the Internet Gateway.
      |  |                   |
      |  +-------------------+
      |
      |
+-----+-------+          +-------------+
|  Subnet 1   |          |  Subnet 2   |
| 10.0.1.0/24 |          | 10.0.2.0/24 |
|             |          |             |
|             | 10.0.2.9 |             |
|             +--------->|             |
|             |          |             |
+-------+-----+          +-------------+
        | 8.8.4.4
        |
        |   +--------+
        +-->| ig-123 |
            |        +-----> The Internet
            +--------+

NAT Gateways

If you have an EC2 instance in a private subnet - one which doesn't allow traffic from the Internet to reach it - then there's also no way for IP packets to reach the Internet. We need a mechanism for sending those packets out, and then routing the replies correctly. This is called network address translation and is very likely done in your house by your wifi router.

A NAT gateway is a device which sits in the public subnets, accepts any IP packets bound for the Internet coming from the private subnets, sends those packets on to their destination and then sends the returning packets back to the source.

It's not necessary to have NAT gateways if you don't intend instances in your private subnets to talk outside if your VPC but if you do need to do that e.g. using an external API, SaaS database etc. then you can simply set up an EC2 instance (might be cheaper, depending on your traffic), configured appropriately, or use an AWS managed NAT gateway resource (will be easier to manage because you won't be doing it).

  +---------------------+
  |   Public Subnet     |
  |   10.0.1.0/24       |
  |                     |
  |    +------------+   |
  |    |            +----------->   The Internet
  |    |  nat-123   |   |
  |    |            |   |
  |    +-------^----+   |
  |            |        |
  +------------|--------+
               |
               | 8.8.4.4
               |                       Route table
  +------------+---------+    +---------------------+
  |  Private Subnet      +----+  10.0.0.0/16: local |
  |  10.0.20.0/24        |    |  0.0.0.0/0: nat-123 |
  |                      |    +---------------------+
  +----------------------+

The public subnet contains the NAT gateway
A request is made from the private subnet to an IP address somewhere on the Internet
The route table says that it needs to go to the NAT gateway
The NAT gateway sends it on

Security Groups

VPC network Security groups denote what traffic can flow to (and from) EC2 instances within your VPC. A security groups can specify ingress (inbound) and egress (outbound) traffic rules, limiting them to certain sources (inbound) and destinations (outbound). They are associated with EC2 instances rather than subnets.

By default all traffic is allowed out, but no traffic is allowed in. Inbound rules can specify a source address - either a CIDR block or another security group - and a port range. When the source is another security group then that must be within the same VPC. For example, a VPC is created with a default security group which allows traffic from anything which has that same security group. Assigning the group to everything created in the VPC (not necessarily the most secure practice) means that all those resources can talk to each another.

                   +---------------+
                   | sg-abcde      |
                   | ALLOW TCP 443 |
                   +----+----------+
                         |
                    +----+------+
                    |  i-67890  |
 10.0.1.123:22      |           | 10.0.1.123:443
------------------>X|           <----------------
                    |           |
                    +-----------+

An instance (i-67890) has a security group (sg-abcde) which allows TCP traffic on port 443
A request is made to its IP address (10.0.1.123) on port 22 which doesn't get through
A request is made to port 443 on the instance and the traffic is allowed

Putting it All Together

The complete picture of your virtual private network looks something like the picture below, with public and private subnets spread across availability zones, network address translation sitting in the public subnets and route tables to specify how packets are routed. EC2 instances are run in any subnet and have security groups attached to them.

                                        +-------+                                  
                                        | ig-1  |                                  
                                        |       |                                  
        vpc-123: 10.0.0.0/16  |         |       |        |                         
       +----------------------+---------+-------+--------+---------------------+
       |                      |                          |                     |   
       |  +-----+             |  +-----+                 |  +-----+            |   
       |  | NAT |             |  | NAT |                 |  | NAT |            |   
public |  |     |             |  |     |                 |  |     |            |   
subnets|  +-----+             |  +-----+                 |  +-----+            |   
       |                      |                          |                     |   
       |                      |                          |                     |   
       |                      |                          |                     |   
       |              +-------+                  +-------+             +-------+
       |              | rt-1a |                  | rt-1b |             | rt-1c |
       | 10.0.1.0/24  |       | 10.0.2.0/24      |       | 10.0.3.0/24 |       |   
-------+-----------------------------------------------------------------------+
       | 10.0.4.0/24  | rt-2a | 10.0.5.0/24      | rt-2b | 10.0.6.0/24 | rt-2c |
       |              |       |                  |       |             |       |   
       |              +-------+                  +-------+             +-------+
private|                      |                          |                     |   
subnets|                      |                          |                     |   
       |                      |                          |                     |   
       |                      |                          |                     |   
       |                      |                          |                     |   
       |                      |                          |                     |   
       |                      |                          |                     |   
       |                      |                          |                     |   
       +----------------------+--------------------------+---------------------+
       |         AZ 1         |          AZ 2            |        AZ 3         |

The Quickest Way to Run Python in Docker

Graham Lyons — Wed, 15 Nov 2017 16:41:13 +0000

I love Python. I think it's a beautifully designed language with a philosophy that I really appreciate as a developer trying to get stuff done. Just run this at a command prompt: python -m this

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!

What's not so nice is Python packaging. It's awkward, it's confusing - it's getting better but it's still not nearly as nice as it is in other languages (Ruby, Java, Javascript).

Docker is a collection of various Linux features - namespaces, cgroups, union file-system - put together in such a way that you can package and distribute software in a language-agnostic container. Docker is a great way to skirt the pain of Python packaging.

To install it, go to https://www.docker.com/ and under the "Get Docker" link choose the version for your operating system.

Just Enough Docker

So. What's the least we can get away with? Or, what's the least I can write to illustrate this? Well, if we use the onbuild Python Docker image, then not much.

Imagine we have a super-simple Flask app which just has one route, returning a fixed string (Hello, World?). We need very little, but we do have the dependency on Flask. Even on my Mac, with the latest OS (OK, not High Sierra just yet), the default Python installation doesn't include the pip package manager. What the hell? It does include easy_install, so I could easy_install pip, or rather sudo easy_install pip because it'll go in a global location. Then I could globally install the flask package too. Woop-de-doo. Which version is now globally installed on my system? Who knows!(?)

Let's not do that. Let's create our requirements.txt file:

echo Flask > requirements.txt

And our Flask app:

from flask import Flask

app = Flask(__name__)

@app.route('/')
def index():
    return 'Hello, World!'


if __name__ == '__main__':
    app.run(host='0.0.0.0')

And then let's have a Dockerfile too:

echo FROM python:onbuild >> Dockerfile

Build It

OK, so let's build it:

$ docker build -t myapp.local .
Sending build context to Docker daemon  4.096kB
Step 1/1 : FROM python:onbuild
# Executing 3 build triggers...
Step 1/1 : COPY requirements.txt /usr/src/app/
 ---> Using cache
Step 1/1 : RUN pip install --no-cache-dir -r requirements.txt
 ---> Using cache
Step 1/1 : COPY . /usr/src/app
 ---> Using cache
 ---> 79fdf87107de
Successfully built 79fdf87107de
Successfully tagged myapp.local:latest

Was that it? Is it built? Yup:

$ docker image ls myapp.local
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
myapp.local         latest              c58ad169cb28        4 seconds ago       700MB

Run It

Great, our app is inside a container! What next? Run the container like this:

$ docker run --rm myapp.local python server.py
 * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)

That tells the docker command to run the myapp.local container (which we built), to remove it when it stops (--rm) and to run the command python server.py inside it. Amazing! So can we see our app now?

The Final Piece

We can't see our app at the moment because although it's running perfectly inside the container it's not accessible anywhere else. The message we get when we run it says it's listening on port 5000, but if we try to access that on the same host we get an error:

$ curl localhost:5000
curl: (7) Failed connect to localhost:5000; Connection refused

We need to expose the port outside the container that it's running in:

docker run --rm -p 5001:5000 myapp.local python server.py

The -p argument maps your local 5001 port to 5000 inside the container (they can just be the same but I've made them different just to illustrate where the host and container ones are).

OK, not quite the end...

So I found out when I was writing this that:

The ONBUILD image variants are deprecated, and their usage is discouraged.

That's OK - you wouldn't really use the ONBUILD image for anything serious, and the Dockerfile that defines it is very easy to understand. Check it out for yourself and see how it works: https://github.com/docker-library/python/blob/f12c2d/3.6/jessie/onbuild/Dockerfile

We can replace the contents of our Dockerfile with the following and get the same result:

FROM python

RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app

COPY requirements.txt /usr/src/app/
RUN pip install --no-cache-dir -r requirements.txt

COPY . /usr/src/app

I've just removed the ONBUILD directives (plus the 3.6-jessie Python version - we can just take the latest).

Really The End

So that's our minimal example of how to run a Python app with its dependencies inside a Docker container. Docker is an amazing piece of technology and it's no surprise that the company is valued so highly. In this instance it provides a clean way for us to avoid pain with Python packaging, but we now also have a container image with our app inside which could be distributed and run on any other system which can run Docker.