DEV Community: tomwerneruk

Awesome HTTP Load Balancing on Docker with Traefik

tomwerneruk — Thu, 20 Jun 2019 14:00:00 +0000

Why Traefik?

Traefik is the up-and-coming 'Edge Router / Proxy' for all things cloud. Full disclosure, I like it.

The cut back features compared to products like F5 which I have used throughout my career is refreshing - these products still do have their place, and they can do some very cool stuff.

I'm a strong believer in avoiding technical debt when I'm building out my infrastructure and applications. Traefik is a production scale tool, while still being nimble enough to run in the smallest of deployments. There is very few reasons why you shouldn't consider incorporating it into your stack now - especially if you are self-hosting and don't have something like AWS ELB available to you, or if Traefik has a killer feature you need!

This Tutorial

Traefik supports multiple orchestrators (Docker, Kubernetes, ECS to name a few), however, for this tutorial I am going to cover Docker Swarm. Check out Why Single Node Swarm to see why I think you should be using Swarm over vanilla docker, even locally.

In case you didn't get the message above, I am going to cover a scale-ready configuration for Traefik here. This means HA clustered support, meaning that as you add nodes, your Traefik service will scale with you. Technical debt to a minimum!

What do I want to achieve?

I host a VPS server that hosts multiple websites, running multiple technologies. Thankfully they are all now containerised (it wasn't pretty).
I want TLS certificates by default on every URL.
I want to future proof this if I need to scale.

Treafik hits all of these points. Let's dive in....

Our HA Docker Swarm Ready solution has a few moving parts;

Traefik Container

This container will be scheduled as a global service in our Swarm. This means that for every Host in our Docker Swarm cluster, one instance of Traefik will be deployed. This will be our traffic workhourse. In most scenarios this should have more than sufficient throughput. If a given Traefik instance is getting saturated, then you might be getting to the point where you should be horizontally scaling (more Hosts, not more CPUs).

Consul Container

Consul is used in this scenario as a configuration store. Traefik needs a repository for config data and certificates which is accessible from all nodes in the cluster. Consul has other features - it overlaps a lot with Swarm's service mesh - but these don't need to be configured for this use case. This will be deployed in an HA manner.

Traefik Init Container

This is used to 'seed' our Traefik config into the Consul cluster when first starting up. Once this has done it's job, it is shutdown, and our scaled Traefik Container nodes take over.

Overall Traefik Cluster Design

Overall Docker Swarm Traefik HA Cluster

Our cluster is comprised of three docker swarm nodes, the recommended minimum number required for the raft consensus algorithm to provide resilience to node failure. There is no pre-requisite for this deployment except cluster communications working properly between nodes (as detailed in the Swarm setup guide - https://docs.docker.com/engine/swarm/swarm-tutorial/).

The Terraform and Compose files for this tutorial are located on my GitLab - https://gitlab.com/fluffy-clouds-and-lines/traefik-on-docker-swarm.git

The solution shall deploy 4 services;

A Traefik node on each manager host,
A Consul node on each manager host (maximum 3),
'whoami' to facilitate testing,
A standalone Traefik config generator.

Inside Traefik

Traefik has two main configuration areas; frontends and backends. Frontends respond to request from the outside world. This is where, if used, TLS is applied to requests. Backends group one or more containers to serve a frontend.

In this deployment, frontends are created dynamically as a result of the docker labels declared on the whoami service (the backend group).

Networking

Traefik and Consul shall expose their management UI ports in host mode. This means they will be exposed on each host they are deployed on. Only hosts that have these services deployed shall have 8080 and 8500 available.

In addition, Traefik shall expose the front-end ports of 80 and 443 on the ingress service mesh. This means that any Docker node will have ports 80 and 443, and will be routed internally by Docker to _ a _ Traefik instance, but not necessarily the one running on that node. This is to facilitate fail over in the case of Traefik container failure.

Deploying the Traefik Cluster

To deploy our stack we are going to use a Docker Compose file to define our services. Once deployed the Traefik instance will poll Docker (specifically the docker socket) for changes to deployed containers. Whenever a new container with appropriate metadata is started, Traefik will beginning routing traffic to it.

Environment Setup

A 3 node Docker Swarm deployment is required. If you don't have this, a Terraform manifest to deploy to AWS is available in this tutorial's repo.

You will need a DNS record that points to your Swarm cluster for the test service. I have chosen whoami2.docker.lab.infra.tomwerner.me.uk but this will need changing to a domain name you own. For maximum availability this should resolve to all of your Swarm nodes (but in theory only one is required due to the Traefik frontend being on the ingress mesh). I have achieved this by creating duplicate A records, resolving to each IP.

The Compose File

The full compose file is available in my GitLab repo, but breaking down the deployment;

  consul:
    image: consul
    command: agent -server -bootstrap-expect=3 -ui -client 0.0.0.0 -retry-join consul
    volumes:
      - consul-data:/consul/data
    environment:
      - CONSUL_LOCAL_CONFIG={"datacenter":"eu_west2","server":true}
      - CONSUL_BIND_INTERFACE=eth0
      - CONSUL_CLIENT_INTERFACE=eth0
    ports:
      - target: 8500
        published: 8500
        mode: host
    deploy:
      replicas: 3
      placement:
        constraints:
          - node.role == manager
      restart_policy:
        condition: on-failure
    networks:
      - traefik

Our Consul deployment is fairly out of the box. It ensures that;

consul is started in server mode,
3 servers are pre-defined as our required amount of nodes before a cluster election can be triggered,
that UI traffic can come from all sources,
that the remaining nodes can be contacted via the alias consul. This is a bit of 'trick' - normally this would be a list of IP addresses, however, using Docker's internal round robin DNS resolution, this means that with repeated attempts, all 3 nodes are returned to trigger an election.

  traefik_init:
    image: traefik:1.7
    command:
      - "storeconfig"
      - "--loglevel=debug"
      - "--api"
      - "--entrypoints=Name:http Address::80 Redirect.EntryPoint:https"
      - "--entrypoints=Name:https Address::443 TLS TLS.SniStrict:true TLS.MinVersion:VersionTLS12"
      - "--defaultentrypoints=http,https"
      - "--acme"
      - "--acme.storage=traefik/acme/account"
      - "--acme.entryPoint=https"
      - "--acme.httpChallenge.entryPoint=http"
      - "--acme.onHostRule=true"
      - "--acme.onDemand=false"
      - "--acme.email=hello@tomwerner.me.uk"
      - "--docker"
      - "--docker.swarmMode"
      - "--docker.domain=docker.lab.infra.tomwerner.me.uk"
      - "--docker.watch"
      - "--consul"
      - "--consul.endpoint=consul:8500"
      - "--consul.prefix=traefik"
      - "--rest"
    networks:
      - traefik
    deploy:
      placement:
        constraints:
          - node.role == manager
      restart_policy:
        condition: on-failure
    depends_on:
      - consul

This is our 'bootstrap' config for Traefik. This service only starts upon deployment. It defines the configuration for the deployment, including enabling Lets Encrypt certificates, enables the Docker provider and defines we will be using Consul as our Key-Value store.

  traefik:
    image: traefik:1.7
    depends_on:
      - traefik_init
      - consul
    command:
      - "--consul"
      - "--consul.endpoint=consul:8500"
      - "--consul.prefix=traefik"
    networks:
      - traefik
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    ports:
      - target: 80
        published: 80
      - target: 443
        published: 443
      - target: 8080
        published: 8080
        mode: host 
    deploy:
      placement:
        constraints:
          - node.role == manager
      mode: global
      update_config:
        parallelism: 1
        delay: 10s
      restart_policy:
        condition: on-failure

The Traefik service is again fairly out of the box. All configuration is pulled from consul (which is seeded by the traefik_init service). This container is deployed on all manager nodes at present, but this constraint is in place as Treafik requires access to data on the manager node to operate. Consider the decoupling options at the end of this post.

  whoami0:
    image: containous/whoami
    networks: 
      - traefik
    deploy:
      replicas: 6
      labels:
        traefik.enable: "true"
        traefik.frontend.rule: 'Host: whoami2.docker.lab.infra.tomwerner.me.uk'
        traefik.port: 80
        traefik.docker.network: 'traefik_traefik'

Our last service - whoami. This allows us to test the deployment. The labels defined here are used by Traefik to configure itself. Host: whoami2.docker.lab.infra.tomwerner.me.uk should be changed to match a valid DNS hostname for your environment.

networks:
  traefik:
    driver: overlay
    attachable : true

volumes:
  consul-data:
      driver: local

Finally our network and storage definition. The Traefik network is an overlay network to allow it to span the Swarm Cluster. It is also attachable to allow standalone containers to be attached to allow debugging (the default if false).

Our consul data is persisted, as not all of it is easily recreated (i.e our issued certificates).

Check our deployment

Assuming you have a working cluster already, deploy the cluster after checking out the repo;

$ docker stack deploy -c traefik_noprism.yml traefik

Verify the deployment;

$ docker service ls
NAME MODE REPLICAS IMAGE                     
traefik_consul replicated 3/3 consul:latest              
traefik_traefik global 3/3 traefik:1.7               
traefik_traefik_init replicated 0/1 traefik:1.7                
traefik_whoami0 replicated 6/6 containous/whoami:latest

The consul cluster election can take a minute or so. Monitor the Traefik and Consul logs to observe the progress of this. Navigating to the UI and having the consul and docker provider tabs available is a good indicator that this deployment has worked.

Test the deployment

Browse to any of your node's publicly accessible addresses to view the Traefik UI;

http://<public ip>:8080/

Under the Docker tab, the details of our test service should show.

Browse to your service hostname (in my case whoami2.docker.lab.infra.tomwerner.me.uk). This should resolve to one of your Swarm nodes and hand it off to a Traefik node. Refresh a few times to see it hit your chosen node under the Health tab (or open up all management UIs for each node).

That's it for the tutorial!

Additional Steps

All tutorials are a basis to start with. A couple of things to consider taking further;

Applying ACLs to restrict UI access to Consul (source IP and authentication),
Restricting access to the Traefik management UI, either by disabling it or firewall (i.e AWS VPC security group access from bastion only),
Docker Socket security...

If you're concerned around exposing a container to the internet which is bound to the Docker socket (a potential security issue), you may want to consider;

Deploying traefik-prism as per my previous post,
Using TCP to communicate to the Docker socket,
Consider Traefik Enterprise Edition if you are in a commercial environment (as this supports splitting configuration and routing roles).

As ever, comments and questions are welcomed below.

Deploying HA WordPress on AWS using Terraform and Salt

tomwerneruk — Tue, 21 May 2019 13:01:00 +0000

Love it or hate it WordPress is still around to stay. It is still the go-to for creating websites, thanks to a massive amount of plugins, themes and experience floating around for it.

While trying to learn anything I have found scripted deployments a great way to learn a product from a different perspective, requiring deeper thought to all the moving parts. This is the first of several posts around automating AWS deployment. I have experience with Cloudformation, but in the quest to always widen my skillset, I have started to work with Terraform instead. I will be using Salt to automate the deployment of WordPress onto the AWS infrastructure.

This article will get you up and running with a High Availability (HA) deployment of WordPress, ready for the next big thing™. It assumes some familiarity with AWS and WordPress.

Design is all about trade-offs

Designing IT Architecture is definitely not black and white, it is a complicated hairball of assumptions, estimates, previous experience and constraints (knowledge, financial and environmental to name a few). Beyond the simplest of design briefs, give two architects the same brief, and you will get two different designs. All design is inherently opinionated.

What does this have to do with our WordPress HA solution? It sounds simple on the surface but is it?...

For example, what would you mean by HA? Stays up during maintenance? Resilient to AWS failure? Resilient to WordPress issue (bad plugin, malformed update)?

What are your constraints? How much is your monthly spend budget? What expertise do you have to keep this solution fed and watered? Realistically, how much downtime can you tolerate? (it might be more than you think).

Out of the above, how much is certain? how much is an estimate (informed or guess)? how much do you think you need?

When approaching this project, I tried to break down each functional component and evaluate the best AWS product for the task. I am trying to evaluate each option based on the AWS Well Architected Framework Pillars, not just what 'sounds right'.

System Component	Low-Fi Solution	Generally used Solution	'Premium' / Exceptional Requirements Solution	What would I choose?
Edge	DNS Round Robin (Route53) + Health Monitor	AWS ELB (Application Load Balancer)	AWS ELB (Classic or ALB)	For most sites, AWS ELB in Application Mode. It is cost-effective, scales well, requires minimal integration effort and is conceptually well understood. Classic may be required where throughput is required at all costs. Round robin would be required when non-TCP traffic needs to be balanced.
Compute	EC2 Instances	EC2 Instances in Auto Scaling Group	EC2 Instances in Auto Scaling Group	Using an Auto Scaling group is a no-brainer here. The extra learning curve is worth the operational convenience. It won't attract additional cost unless mis-configured.
Database	EC2 hosted MySQL Instance	RDS Multi AZ deployment	RDS Multi AZ + cross region read replica	I would stump for a Multi AZ Deployment here. If you need to be tolerant of AWS Region failure, then you will need to consider cross-region read replicas (with a custom failover mechanism to promote a read replica and amending the WordPress configuration).
WordPress Root Storage	N/A, bake into AMI	Push / Pull from S3	EFS	EFS all the way. Higher cost but operationally slicker and less prone to errors. Syncing to S3 using cronjob or similar could have strange concurrency issues if filesystem updates occur on multiple WordPress hosts within a short time frame. Cost-efficient when considering operational advantage, especially when using the IA storage class.
Object Storage	No object storage, just serve static resource from EC2 instances	S3 + Cloudfront	S3 + Cloudfront	S3 all the way here to allow media to then be surfaced via Cloudfront Delivery Network.

Terraform Quickstart

Terraform is delightfully simple to get started with. It is simple to deploy and use, and the syntax is clean.

I'm not going to write a step-by-step how to get Terraform installed and running here, but head over to my channel for a tutorial for Windows and Linux. Instead, I want to cover the principles of the Terraform workflow and how to use modules. I also highly recommend the tutorial track from HashiCorp (Terraform's creator).

Terraform has a simple architecture. It comprises of the terraform tool which when run within a directory either; dry-runs (plan), deploys (apply) or removes (destroy) the infrastructure defined in one or more modules. These are the 3 core commands which will be required to manage this deployment. There are no special requirements for where the tool is installed or run from, except connectivity to your provider (in this case internet access to AWS).

Apart from the automation benefits of using an Infrastructure as Code tool like Terraform, we should be moving away from reinventing the wheel, and thinking more like a developer, using libraries wherever we can. In Terraform, infrastructure definitions can be wrapped up into a module to allow it to be reused elsewhere, just like a programming library. Terraform has access to a large repository of ready-made, battle-tested modules in the Terraform Registry. This tutorial will be making extensive usage of the AWS Modules available in the Registry. I have no problem in admitting that the authors of these modules likely know AWS and Terraform better than me! This tutorial will use a simplified folder structure which I would adapt to facilitate real-world usage. There are various ways to achieve a structure ready for real-world usage, check out the recommended reading at the end for links.

To this end, our structure is going to look like this;

├── main.tf
├── outputs.tf
├── modules
│   ├── compute
│   │   ├── main.tf
│   │   ├── userdata.tmpl
│   │   └── variables.tf
│   ├── database
│   │   ├── main.tf
│   │   ├── outputs.tf
│   │   └── variables.tf
│   ├── efs
│   │   ├── main.tf
│   │   ├── output.tf
│   │   └── variables.tf
│   ├── media
│   │   ├── main.tf
│   │   ├── outputs.tf
│   │   └── variables.tf
│   ├── network
│   │   ├── main.tf
│   │   └── outputs.tf
│   └── seeder
│   ├── main.tf
│   └── variables.tf
├── packer
│   ├── aws_vars.json
│   └── template.json
├── salt_tree
│   └── srv
│   ├── pillar
│   └── salt
├── terraform.tfvars
└── wordpressha.pem

This source tree contains;

main.tf this contains the entrypoint that all our modules will be called from.

modules contains our sections of functionality as per our design analysis above.

packer contains the template for our AMI base image. See Creating a LAMP AMI using Packer and Salt on how to use this.

salt_tree is used by both Packer and Terraform to configure our WordPress installation on our deployed EC2 instances. You could easily swap this out for a different tool i.e Chef or Puppet and change the provisioner in the Terraform code accordingly.

terraform.tfvars contains our configuration values to stand up our solution. Empty fields will need completing before running Terraform.

Deploying to AWS 🎉

That's the theory over, if you've got this far, you're on the home stretch!

Assuming you have installed Terraform and Packer correctly, checkout the code from my Git repository at https://gitlab.com/fluffy-clouds-and-lines/ha-wordpress-using-terraform-and-salt.git.

Before proceeding with the Terraform run, we need an SSH keypair (Terraform cannot currently create them). To create your keypair;

Logon to your AWS Console,
Change to your target region, and open EC2,
Network & Security > Keypairs > Create Key Pair
Name the Keypair 'wordpressha' and copy the downloaded wordpressha.pem to the directory where the Terraform code has been checked out into,
On Linux, change permissions to 400 (read only by user).

Next, ensure that terraform.tfvars is completed. Once done, execute;

> packer build -var-file=./packer/aws_vars.json ./packer/template.json
> terraform init
# Terraform modules for RDS and VPC don't resolve dependancies correctly, so explictly build VPC first
> terraform apply -target=module.network 
# Deploy Seeder dependancies
> terraform apply -target=module.database -target=module.efs 
# Deploy seeder
> terraform apply -target=module.seeder
# Deploy all to make state consistent
> terraform apply

This should take around 15 minutes end to end. This will;

Build our custom AMI image with all our LAMP (Apache, MySQL, PHP) dependencies baked in,
Download the external Terraform modules,
Build the AWS VPC,
Deploy the S3 bucket and CloudFront distribution,
Create the application load balancer and auto-scaling group,
Deploy the RDS MySQL Database instance,
Create the Elastic Filesystem,
Deploy the 'WordPress seeder'. This mounts the EFS and installs WordPress so that nodes that are started as part of the auto-scaling group already have the WordPress installation available to them,
Publish an A record to Route53, linked to the ALB.

You should then be able to browse to http://nextamazing.site/ and see your completed installation.

Don't like the use of -target? Yes, it's bad;

This targeting capability is provided for exceptional circumstances, such as recovering from mistakes or working around Terraform limitations. It is not recommended to use -target for routine operations, since this can lead to undetected configuration drift and confusion about how the true state of resources relates to configuration.

See below on suggestions on how to make this work in a real-life scenario. You really shouldn't take this approach going forward in production usage.

More Design Decisions

There are a few more design decisions that need to be made before this could be considered 'production ready'

The site should really be running on HTTPS, whether this is done with an AWS Managed Certificate (via ACM) or an externally signed CSR made available to the load balancer via ACM or IAM,
Although the infrastructure is in place for asset delivery via CloudFront, it is not setup in WordPress as part of this Terraform run. There are several options, both free and paid that will achieve this i.e plugins or custom cron jobs,
How will you maintain backups? At present, the RDS snapshot defaults will be used. How will you backup your WordPress installation?

Wrapping up...

That's it for now. This should have given a good introduction on how to use Terraform to deploy a full solution on AWS. Earlier I mentioned some simplifications made for the purposes of this blog article, a couple of things to consider;

This article was started before the release of Terraform 0.12, it should be considered whether a new project should be started on 0.12, assuming dependencies are compatible,
The major creative license I have taken here is to create one large module that needs separate components to be called in a specific order to achieve a specific end result. As some of the modules are dependant on each other (there is no reason why you couldn't run terraform apply twice and have a successful deployment), I would suggest either breaking this up into distinct modules i.e base, seeder and wordpress, or use a tool like Terragrunt,
One of the thought leaders in the IaC space, Gruntwork, has developed Terragrunt to improve your Terraform workflow to mitigate potential issues when running in production. One of the big advantages here is being able to compartmentalise Terraform state (the record of what Terraform has deployed) into smaller chunks, to reduce impact in cases of state corruption or loss (a definite possibility). This tool is worth considering in a large, multi-module deployment like this.

Creating a LAMP AMI using Packer and Salt

tomwerneruk — Thu, 02 May 2019 15:24:00 +0000

This is the first in a series of posts to create an Infrastructure as Code powered deployment of WordPress running on Amazon Web Services.

This one's going to be pretty short, partly down to the great tools on offer!

What create an image? Why Packer? Why Salt?

Creating images with as much of your installation and configuration baked are vital in DevOps environment, where predictability and agility is key. For example, if you have an autoscaling group which is creating a pool of WordPress application servers, installing the Apache, PHP and MySQL Client using a deployment script would mean a node would take too long to enter service. Whereas preparing a customised AMI will mean that the time to enter service is only restricted by the length of time to start the EC2 instance.

Packer is part of the wider Hashicorp toolset for controlling the cloud via IaC. Packer can create images for a wide range of cloud and on-premise platforms. Being part of the same family of tools there is a degree of similarity between how they work and your infrastructure code will be written.

Salt is one several configuration tools in the market. I have no real allegiance to any other tool after learning Puppet, Ansible and Salt. Greenfield environments are pretty rare, so you may well be restricted with your current toolset. The concepts used here are portable to other configuration tools, of which Packer supports many!

Packer and Salt Quickstart

Packer is extremely easy to get started with, like the rest of the Hashicorp products, it is simply a case of download, extract and run. Head over to my YouTube Channel to watch my how-to video.

Clone the tutorial repo from https://gitlab.com/fluffy-clouds-and-lines/packer-and-salt-lamp-ami. You should have a structure that looks like this;

├── aws_vars.json
├── README.md
├── salt_tree
│   └── srv
│   ├── pillar
│   │   ├── apache.sls
│   │   ├── mysql.sls
│   │   └── top.sls
│   └── salt
│   ├── apache
│   ├── mysql
│   ├── php
│   └── top.sls
└── template.json

aws_vars.json

This is used to provide variable data to avoid having to specify it on each invocation

salt_tree

Contains the Salt declarations to install and configure are LAMP stack components. This is copied by Packer to the remote host during the build, then executed by Salt.

template.json

The Packer build declaration that specifies the base AMI to use, and how Salt should be invoked during the build process.

Packer Run

Running Packer is as simple as it gets. After cloning the repo, all you need to decide is to whether put your credentials into a file or not.

To prompt (or use your AWS CLI credentials if configured), from the project root, run;

packer build template.json

or, if you have created a variables file;

packer build -var-file=./aws_vars.json template.json

The build should take around 5 minutes, to create the base instance, apply the Salt configuration and generate a final AMI.

Inside template.json

Packer templates have 3 main sections (excluding Post-Processors which are optional and are for specific use-cases);

Variables

Used to abstract from your code sensitive or dynamic information that should be provided to the script.

Builders

The heavy lifting of creating a machine with an appropriate base image to start building upon, and post provisioning, wrapping it up ready for use.

Provisioners

The way to actually to apply changes to your build, using scripts, configuration management tools (Chef, Puppet, Ansible, Salt etc).

Checkout the Packer documentation for the latest list of available Builders and Provisioners.

{
  "variables": {
    "aws_access_key": "",
    "aws_secret_key": ""
  }
...
}

Our variables are declared so they can be used in subsequent parts of the template. They can be provided at runtime, JSON file, environment variables, Consul or Vault (cool eh?).

{
...
  "builders": [
    {
      "type": "amazon-ebs",
      "access_key": "{{user `aws_access_key`}}",
      "secret_key": "{{user `aws_secret_key`}}",
      "region": "eu-west-2",
      "source_ami_filter": {
        "filters": {
          "virtualization-type": "hvm",
          "name": "*ubuntu-bionic-18.04*",
          "root-device-type": "ebs"
        },
        "owners": [
          "099720109477"
        ],
        "most_recent": true
      },
      "instance_type": "t2.micro",
      "ssh_username": "ubuntu",
      "ami_name": "wordpress-ha-node {{timestamp}}"
    }
  ],
 ...
}

Our builder specifies we want an EBS backed AMI, based upon the latest Ubuntu 18.04 image.

{
...
  "provisioners": [
    {
      "type": "salt-masterless",
      "local_state_tree": "./salt_tree/srv/salt",
      "local_pillar_roots": "./salt_tree/srv/pillar",
      "salt_call_args": "pillar='{\"role\":\"builder\"}'"
    },
    {
      "type": "shell",
      "inline": [
        "rm -rf /srv/salt",
        "rm -rf /srv/pillar"
      ],
      "execute_command": "sudo sh -c '{{ .Vars }} {{ .Path }}'"
    }
  ]
}

We use two provisioners here (you can have multiple per build), that run in sequence. The first provisioner uses Salt in masterless mode (no central server), uploads the pre-defined Salt tree to the host and applies it. The Second provisioner is to get round Terraform Bug #20323 which stops Terraform re-running Salt Masterless provisioner against this image.

Inside the Salt Tree

I have found Salt a strange beast to learn, in some cases it is very easy to understand, but some of the terminology takes time to get used too. This is not designed to be a full Salt intro, but more explain the decisions taken with the Salt definition this project uses.

Salt is a configuration management tool that takes configuration files and uses them to apply a desired state to a system. Salt at a high level uses States to define how to apply the configuration, and Pillars to provide variable data. It is similar to Packer having a template and a separate variables file.

/srv/salt/top.sls is the leader of the show here. It defines which States apply to any given Salt machine.

base:
  'role:builder': 
    - match: pillar # Match on 'role' passed in as additional Pillar data via salt_call_args
    - php
    - php.mysql
    - php.mysqlnd
    - apache
    - apache.config
    - apache.vhosts.standard
    - mysql # We don't need MySQL Server (using RDS instead), but can't be removed presently due to bug
    - mysql.config
    - mysql.client

The top.sls used in our tree;

Will apply PHP, Apache and MySQL states to the host. Each corresponding folder is a prebuilt Salt state called a Salt Formula. All were sourced from https://github.com/saltstack-formulas.
'role:builder' is a filter to decide which hosts to apply state too. In our next tutorial you will see how we use the same tree for different purposes based on role.
Each list item i.e php or mysql.client is a folder in the Salt tree. Periods mark subfolders. It is quite common for larger formulas to be split out like this.

/srv/pillar/top.sls is our Pillar configuration root. This;

Defines configuration for the states to be applied by /srv/salt/top.sls.
Some Salt Formulas have defaults that are sensible and therefore will not have a corresponding Pillar entry.

You will notice in our Packer Provisioner block we provide the salt_call_args with a value of "pillar='{"role":"builder"}'". This provides supplementary Pillar information that can then be used to decide which parts of the /srv/salt/top.sls are applied. There is a lot of flexibility around this, and there are other methods to filter this file.

I am certainly no Salt expert here. I have used pre-built Salt Formulas here and wired them together to create my desired setup.

Wrapping Up

If you've got this far, you should hopefully had a very quick intro to Packer and Salt, and successfully manged to build an image, like so;

amazon-ebs: -------------
    amazon-ebs: Succeeded: 27 (changed=17)
    amazon-ebs: Failed: 0
    amazon-ebs: -------------
    amazon-ebs: Total states run: 27
    amazon-ebs: Total run time: 69.343 s
==> amazon-ebs: Provisioning with shell script: /tmp/packer-shell598436168
==> amazon-ebs: Stopping the source instance...
    amazon-ebs: Stopping instance
==> amazon-ebs: Waiting for the instance to stop...
==> amazon-ebs: Creating AMI wordpress-ha-node 1558710754 from instance i-08f3e01d313901737
    amazon-ebs: AMI: ami-0477fc2kjw982c28e81
==> amazon-ebs: Waiting for AMI to become ready...
==> amazon-ebs: Terminating the source AWS instance...
==> amazon-ebs: Cleaning up any extra volumes...
==> amazon-ebs: No volumes to clean up, skipping
==> amazon-ebs: Deleting temporary security group...
==> amazon-ebs: Deleting temporary keypair...
Build 'amazon-ebs' finished.

==> Builds finished. The artifacts of successful builds are:
--> amazon-ebs: AMIs were created:
eu-west-2: ami-0477fc2kjw982c28e81

Happy infrastructure coding! Comments and questions welcome below!

Using django-allauth for Google Login to any Django App

tomwerneruk — Wed, 09 Jan 2019 13:33:00 +0000

If you're using django-cookiecutter for your new projects (and if you're not, you should) you may know that it comes with out of the box user login management, backed up by django-allauth. This package provides you the framework to build your own user management experience.

A recent app I have been working on, I wanted to allow login via Google, but wanted to ensure that only pre-approved users could login using social login - the application is for a closed user group, not designed for the random public. It's essentially access via invite only. I tried django-invitations, but it felt more geared towards non-social login (sorry guys if I am mis-selling what looks like a great library!). Therefore, it was back to customising django-allauth.

This tutorial isn't going to cover getting djang-allauth working from scratch - either take a peek at the django-cookiecutter source, or follow the installation from the docs here, we're looking at novel ways to change the django-allauth workflow.

Changing Behaviour

django-allauth allows for the over-riding default behaviour to built custom functionality via the use of adaptors. Using cookiecutter? a custom adaptor stub has already been created for you under <project name>/users/adapters.py. If not, go ahead and create an adapters.py file under one of your Django apps in your project - if you have a 'core' or 'settings' app this might be the best place for it. The location doesn't really matter, as you have to specify the location in your Django settings.

...
ACCOUNT_ADAPTER = 'myapp.users.adapters.AccountAdapter'
SOCIALACCOUNT_ADAPTER = 'myapp.users.adapters.SocialAccountAdapter'
...

_ settings.py _

from allauth.account.adapter import DefaultAccountAdapter
from allauth.socialaccount.adapter import DefaultSocialAccountAdapter
from django.conf import settings
from django.http import HttpRequest

class AccountAdapter(DefaultAccountAdapter):

    def is_open_for_signup(self, request: HttpRequest):
        return getattr(settings, "ACCOUNT_ALLOW_REGISTRATION", True)

class SocialAccountAdapter(DefaultSocialAccountAdapter):

    def is_open_for_signup(self, request: HttpRequest, sociallogin: Any):
        return getattr(settings, "ACCOUNT_ALLOW_REGISTRATION", True)

_ adapters.py _

All credit for this code snip goes to the django-cookiecutter devs, it's simple and get's the point across. At the moment both Account and Social based logins are open for business, this adaptor has no net effect on operation at the moment.

My requirement is to allow Social login and signup only, so first off, let's disable account based signup.

...
class AccountAdapter(DefaultAccountAdapter):

    def is_open_for_signup(self, request: HttpRequest):
        return getattr(settings, "ACCOUNT_ALLOW_REGISTRATION", False)
...

_ adapters.py _

If you head over to http://app/accounts/signup/, you should be told that signup is closed. Great. Next to get Social Logon working. To take the example of Google, you need to do the following;

ensure you have run ./manage.py migrate to ensure that the required social login tables have been created,
ensure you have added 'allauth.socialaccount.providers.google', to your INSTALLED_APPS
head over and create your Google OAuth Credentials,
Login to your Django Admin and create a new Social Application object of type 'Google'.

If you head to http://app/accounts/login/, Google should be listed as a login provider. Test your Google Login, it should allow you to login with your Google account without any issues. A new user will be created under Users and Social Account.

We're nearly there, last step is to restrict to only 'pre-approved' users only. Go ahead and delete via Django Admin the Social Account object created just now (leave the User in place). The way we are going to restrict to pre-approved users is to change the SocialAccountAdapter behaviour. We only want only social logins from an email address with a valid User object already created (either manually via Django Admin or another management screen).

class SocialAccountAdapter(DefaultSocialAccountAdapter):

    def pre_social_login(self, request, sociallogin):
        try:
            get_user_model().objects.get(email=sociallogin.user.email)
        except get_user_model().DoesNotExist:
            from django.contrib import messages
            messages.add_message(request, messages.ERROR, 'Social logon from this account not allowed.') 
            raise ImmediateHttpResponse(HttpResponse(status=500))
        else:
            user = get_user_model().objects.get(email=sociallogin.user.email)
            if not sociallogin.is_existing:
                sociallogin.connect(request, user) 

    def is_open_for_signup(self, request, sociallogin):        
        return True

This is our updated Social adapter. is_open_for_signup still returns true as even a pre-authorised User still hits the signup code path on their first login. pre_social_login is invoked when Social Login is completed, but before the conversion to a valid Django session is done. Therefore, this is the point to check for pre-authorisation.

The try block looks for an existing User against the email provided by the social login request. If a user exists, link the social logon and continue, if not, abort the attempt and leave an update in the 'message' queue.

That's it, this should give a basis for pre-approved social login. A few improvements to make this production ready;

ensure you force email verification, as some social networks (i.e Facebook) have been flagged as having questionable verification procedures for email addresses. By forcing email verification, it is an extra step of protection,
Start to override the default templates to provide logon buttons / logos for each provider you decide to use,
make the user experience slicker by suppressing the need for a username.

A suggested config to get you started;

ACCOUNT_AUTHENTICATION_METHOD = 'email'
ACCOUNT_EMAIL_REQUIRED = True
ACCOUNT_EMAIL_VERIFICATION = 'mandatory'
ACCOUNT_ADAPTER = 'myapp.users.adapters.AccountAdapter'
SOCIALACCOUNT_ADAPTER = 'myapp.users.adapters.SocialAccountAdapter'
SOCIALACCOUNT_QUERY_EMAIL = True
SOCIALACCOUNT_AUTO_SIGNUP = True
ACCOUNT_USERNAME_REQUIRED = False

That's all for now. As ever comments and questions below!

Hardening Traefik when using the Docker Provider

tomwerneruk — Thu, 06 Dec 2018 14:06:47 +0000

This issue on the Traefik GitHub tracker piqued my interest the other day. The Docker Socket does come up as the Achilles heel at times, with different mitigations to secure it - Proxy Containers, exposing via TLS with Authentication and Authorisation. What you choose is down to your risk appetite and your broader environment.

In this scenario, having the Docker Socket bound to an Internet-facing container wasn't a risk I wanted to take. A few options have been floated already (Sock Proxy, Switch to TLS etc., Switch to static file config). However, I wanted to take a stab at an alternative route.

The result is traefik-prism. It is a simple Python script that takes a valid Traefik Dynamic Config and publishes it to an Internet-facing container, which isn't attached to Docker.

Example Deployment for traefik-prism

It takes advantage of the built in APIs in Traefik, no magic here. It is designed to handle different Traefik providers, not just Docker (hint: comments from other provider users warmly welcomed). Roughly, the script;

retrieves the current Dynamic config (the frontends and backends) from /api on the API Endpoint (normally port 8080),
extracts the frontends and backends from the response, based off the PROVIDERS environment variable, then merge the potentially multiple blocks (say file and docker), into one config,
finally, pushes the new merged config to /api/endpoints/rest. The new config should be active immediately.

The security model of pushing from a higher security level zone (in this case a container with access to the Docker socket), and pushing to a lower one is a common pattern for securing systems. There isn't any 'sensitive' data to leak here, so I'm not too bothered about content checking as we move from high to low.

Check out the code on GitHub or pull the image straight from Docker Hub.

As ever, thoughts and comments always welcome!

Why you need Single Node Swarm for Development

tomwerneruk — Wed, 10 Oct 2018 14:54:00 +0000

Docker Swarm has matured enough that it's adoption is starting to pickup. Docker Captains are working with real clients rolling it out everyday. That's fine for clustered workloads, but what about locally?

Should you be using Swarm Mode even for local development? In my opinion, TL;DR - Yes.

Swarm Mode has many benefits which only make sense in production multi Docker host scenarios, but used locally it can add value and reduce differences between Development and Production (#win). What does Swarm actually do?

Swarm elevates a random collection of Docker hosts (or just one) into a cluster, which orchestrates (fancy word for automatic management) starting and stopping containers, managing cluster-wide information and providing transparent connectivity between hosts.

With one host, transparent connectivity isn't a win, but the same interface for cluster information management (Secrets, Configs, Labels) and the advanced service management is. In the same way that Compose is an evolution over plain Docker, Swarm is a step forward over Compose. Here are my top 3 reasons why you should be thinking about using Swarm locally.

Reduce Surprises - Simplification and Consistency

Let's face _ one of _ the elephants in the room. Docker Swarm (as at 18.09) doesn't currently have feature parity with docker run. Excluding concepts that just plain don't make sense for the service level of abstraction (i.e container names, restart policy), there are some of the more 'fringe' options that are not supported (sysctl Kernel tuning, host device mapping) - although some are currently work-in-progress. There is an excellent tracker of gaps (and progress) here.

This means that a compose file that works when targeting docker-compose isn't guaranteed to work 100% with Docker Swarm without tweaks. Sorry, but nothing is perfect.

If Swarm is still for you in production, then it makes sense to use Swarm locally to avoid you having to ;

effectively learn two product 'versions' at once,
have to maintain two or more compose files in parallel causing potential for more mistakes.

Secrets

Docker Secrets allow the secure publishing of sensitive details for an application to consume, taking the recommendations of the 12 Factor App to a new level by providing a much more granular approach compared to environment variables. They are great , but they do have limitations; nothing is entirely secure (the old adage, if you have host access, nothing is secure) and it needs to be supported by the application (or an appropriate entrypoint script).

Accepting these limitations, they are only available in Docker Swarm, as their implementation is tied to Swarm's internal cluster Raft database. That means, if you're using plain docker or Compose you can't access them (see below for an exception). But as we've just agreed above, your application will likely need changes to take advantage of Secrets, so do you really want to start adding if statements to code to handle Dev vs. Prod? (answer is no).

Yes, you can use Docker Secrets with Compose, but they get namespaced by Compose (<stack>_ is prepended) which means that you need to maintain two different references to secrets, depending on environment (again not good, as it causes differences between Dev and Prod) - it is possible, but it is down to you, your development flow and how your code is organised.

This means, in reality, if you want Secrets in your application, you should use Swarm locally. There are ways round it (simulating, injecting environment variables, branching in code), but they all point to the fact you are having to do extra work to get you to the same point.

Next hurdle, Secrets are immutable (they can't be viewed or amended once created). This is not development friendly, these values are likely to change as you iterate. Therefore, there is an option to source secrets from a file. Keeping them in files means it is much quicker and easier to tear down a stack and it's secrets and recreate them if you rebuild your environment or need to change a secret.

For example, on one of my current projects (which has separate development and production compose files - multi-stage builds planned!), I can cleanly reference the same secret in my code, but have a simple workflow to change it in Development.

development.yml

version: '3.6'

services:
    appserver:    
        image: appserver:latest    
        secrets:      
            - application_secret

secrets:  
    application_secret:    
        file: ./compose/local/secrets/application_secret

production.yml

version: '3.6'

services:
    appserver:    
        image: appserver:latest    
        secrets:      
            - application_secret

secrets:  
    application_secret:    
        external: true

The only difference here is the 'source' of the secret.

Sourcing from a file kind of defeats the purpose of a secret, but it preserves the interface.

Marking it as external in production requires manual intervention to create the secret, however, in larger organisations the person with the contents of the secret might be very different to the person deploying code. When it comes to deployment where security matters, separation of responsibility keeps the clipboards at bay.

Keeping as much consistency between environments reduces the complexity of the code that handles our secrets and keeps things clean.

Test Scaling

Horizontal scaling of an application requires thought. Just because Docker can scale a service multiple times, doesn't automatically mean your application will handle being run in this manner.

Run your code at scale locally (just because you have one host, doesn't mean you can't run multiple instances of a container) and rattle these issues out early. This will start to weed out issues with concurrent access to databases, message queues and files on shared storage (to name a few). Swarm will still load balance between multiple instances automatically, even on one host.

From the background I come from, predictability, simplicity and ease of handover between Development to Operations, trump novelty. In my eyes adopting Swarm locally helps reduce surprises and make life easier throughout the whole Develop and Deploy lifecycle. That's all for now!