Calming App Jams with Træfik

This was originally posted on the FlexMR Dev Blog.

At FlexMR we make heavy use of a ‘prototype-first’ approach. We find it’s easier to produce a stand alone implementation of an integration with a third party tool (or idea) than spend time folding it into what we have, only to decide to throw it away and try a different way.

As you’d expect this can produce a lot of apps. The configuration for NGINX (the ‘junction box’ of the server) started to become more unwieldy. New (and old) server configs and SSL certs started to pile up. Moving NGINX config to a separate git repository was tidier but it didn’t solve the underlying issue. I started to look around for an solution which better fitted our use case.

Our infrastructure setup for prototypes (and as it happens, an internal tool called Hairy Slackbot — more on that in the future) lives on a dedicated server which is running Docker Swarm mode (so it is ready to scale when we need to).

My wish list for a reverse proxy solution:

• Minimal configuration, ideally on the app side
• Scalable
• Easily use LetsEncrypt for certificate generation (provision and renew)
• Nice to have (only because we don’t need it right now): Load balancing

I’d looked at the popular nginx-proxy alongside docker-letsencrypt-nginx-proxy-companion however, in the spirit of experimentation I thought I’d give the new one a whirl.

Træfik

Træfik is a modern HTTP reverse proxy and load balancer that makes deploying microservices easy. Træfik integrates with your existing infrastructure components and configures itself automatically and dynamically. Pointing Træfik at your orchestrator should be the only configuration step you need.

We’re only going to touch a small portion of Træfik’s functionality in the example below. What we’re aiming to achieve here is:

• New services can be deployed to our development server without having to make any changes outside of that apps repository.
• New SSL certs are provisioned (and renewed) automatically by LetsEncrypt.

Show me some code!

Or, if you’re really keen there is a repository and example walkthrough in the next section.

What we had

In the example I described above, I defined separate stacks for both the hsb and proto1 apps as well as nginx. nginx used to part of the hsb stack by virtue of it being the first stack on the development server, this was split out once we started adding more apps.

Each app has its own git repository in which the docker stack file defines how it runs within the swarm (how many instances, memory and CPU restrictions, network details). We deploy via CI.

The new world

Stack 1: træfik

Deployed with: docker stack deploy --compose-file traefik-stack.yml --with-registry-auth traefik

	version: '3.4'
	services:
	traefik:
	# Use the apline image, its lovely and small - 22MB!
	image: traefik:1.7.3-alpine
	# This just exposes the ports to the host, the host firewall is configured separately
	ports:
	- "443:443" # Expose HTTPS port (and available externally)
	- "80:80" # Expose HTTP port (and available externally)
	- "8080:8080" # Expose port for the træfik UI (not available externally)
	# This is the network which traefik uses to communicate with other containers
	networks:
	- traefik
	# I've commented these out to make it simpler to get running and placed a volume mount on line 25 instead.
	#configs:
	# - source: traefik_v1.1.toml
	# target: /traefik.toml
	volumes:
	- /var/run/docker.sock:/var/run/docker.sock # Required to talk to Docker
	- /home/dev/reverse-proxy/traefik.toml:/traefik.toml # This is a volume mount for the træfik configuration (see line 18)
	- /home/dev/reverse-proxy/acme.json:/acme.json # This is used to store information generated by LetsEncrypt. There is
	# no config for this, just touch a blank file before starting the stack.
	deploy:
	placement:
	constraints:
	- node.role == manager # Only run this on a swarm manager
	# See comment on line 18, above
	#configs:
	# traefik_v1.1.toml:
	# external: true
	# Overlay network definition other apps will use this to talk to this service
	networks:
	traefik:
	driver: overlay

view raw reverse-proxy-stack.yml hosted with ❤ by GitHub

	debug = false
	logLevel = "ERROR"
	defaultEntryPoints = ["https", "http"]
	# https://docs.traefik.io/configuration/entrypoints/
	[entryPoints]
	[entryPoints.http]
	address = ":80"
	[entryPoints.http.redirect] # https://docs.traefik.io/configuration/entrypoints/#redirect-http-to-https
	entryPoint = "https"
	[entryPoints.https]
	address = ":443"
	[entryPoints.https.tls]
	[entryPoints.dashboard]
	address = ":8080"
	[entryPoints.dashboard.auth]
	[entryPoints.dashboard.auth.basic] # https://docs.traefik.io/configuration/entrypoints/#basic-authentication
	users = ["admin:$wibble"]
	[retry]
	# https://docs.traefik.io/configuration/backends/docker/
	[docker]
	swarmMode = true
	watch = true
	exposedByDefault = false
	# https://docs.traefik.io/configuration/acme/
	[acme]
	email = "[e-mail]"
	storage = "acme.json"
	entryPoint = "https"
	onHostRule = true
	[acme.httpChallenge]
	entryPoint = "http"
	# https://docs.traefik.io/configuration/api/
	[api]
	dashboard = true
	entrypoint = "dashboard"

view raw traefik.toml hosted with ❤ by GitHub

Stack 2: hsb

Deployed with: docker stack deploy --compose-file hsb-stack.yml --with-registry-auth hsb

	version: '3.4'
	services:
	hsb:
	image: 123456789012.dkr.ecr.eu-west-1.amazonaws.com/hairy-slackbot:deploy-20181015-0935-e583d12
	command: bundle exec puma -p 3000
	# This links to the network created in the proxy (træfik) stack
	networks:
	- traefik_traefik
	# This is where the 'magic' is defined. In a non-swarm environment this would be defined as plain labels.
	deploy:
	labels:
	- "traefik.enable=true" # Use træfik for this service
	- "traefik.backend=hsb" # What service to talk to
	- "traefik.frontend.rule=Host:hsb.our-dev-team-domain.co.uk" # What domain should this service use?
	- "traefik.docker.network=traefik_traefik" # The network to use
	- "traefik.port=3000" # The port to use for the service
	secrets:
	- source: hsb_secrets_1.8.yml
	target: /app/secrets.yml
	secrets:
	hsb_secrets_1.8.yml:
	external: true
	networks:
	traefik_traefik:
	external: true

view raw hsb-stack.yml hosted with ❤ by GitHub

Stack 3: proto1

Deployed with: docker stack deploy --compose-file proto1-stack.yml --with-registry-auth proto1

	version: '3.4'
	services:
	proto1:
	image: 123456789012.dkr.ecr.eu-west-1.amazonaws.com/prototoype:deploy-20181019-1050-a127c89
	command: bundle exec puma -p 9292
	# This links to the network created in the proxy (traefik) stack
	networks:
	- traefik_traefik
	# This is where the 'magic' is defined. In a non-swarm environment this would be defined as plain labels.
	deploy:
	labels:
	- "traefik.enable=true" # Use traefik for this service
	- "traefik.backend=proto1" # What service to talk to
	- "traefik.frontend.rule=Host:proto1.our-proto-domain.co.uk" # What domain should this service use?
	- "traefik.docker.network=traefik_traefik" # The network to use
	- "traefik.port=9292" # The port to use for the service
	secrets:
	- source: proto1_secrets_1.0.yml
	target: /app/secrets.yml
	secrets:
	proto1_secrets_1.0.yml:
	external: true
	networks:
	traefik_traefik:
	external: true

view raw proto1-stack.yml hosted with ❤ by GitHub

Visual view of configuration

Traefik comes with a rather bonny UI, which we’ve configured to serve on port 8080 of the host. I access this via a SSH port forward eg: ssh -L8080:localhost:8080 ... as I don’t make it available externally (and neither should you!).

The rather pretty traefik UI

Worked Example

The accompanying repo below uses a different setup to the hsb/proto1 setup above. I’ve done this to illustrate multiple services in the same stack and how single services can be assigned multiple domains.

neilbartley / traefik-swarm-example

Traefik Swarm (mode) Example

Traefik Swarm (mode) Example

Please see the accompanying blog post for background.

Running this example

Commands and Video.

View on GitHub

I’m making use of jwilder/whoami.. This provides a web service which just echos its container id.

Stack 1 is traefik, as above.

Stack 2 is proto1, this has admin (proto1.neil.bar) and backoffice (proto4.neil.bar) services.

Stack 3 is proto2, this has a single service, app (proto2.neil.bar and proto3.neil.bar).

As we’ve set up the [acme] block in our traefik.toml then Traefik will automatically request, provision and renew certificates for the hosts defined in the traefik.frontend.rule label in each service from LetsEncrypt.

Adding additional services doesn’t require any changes to the traefik stack and with the LetsEncrypt integration it really does make this task trivial.

asciinema demo

	# Created a server on DigitalOcean (anywhere will do). Just ensured it had docker installed.
	# Setup DNS entries for proto[1234].neil.bar pointing at the server.
	# Connected to the server.
	# Server has ports locked down. Open up the ones we need.
	ufw allow 80/tcp
	ufw allow 443/tcp
	# Wouldn't normally open this one up but its handy for this demo.
	ufw allow 8080/tcp
	# Initialise a Swarm
	docker swarm init --advertise-addr 209.97.138.242
	# Clone the example repo
	git clone https://github.com/neilbartley/traefik-swarm-example.git
	cd traefik-swarm-example
	# This is important, if the permissions aren't restrictive enough Traefik won't generate certificates.
	chmod 600 traefik/acme.json
	# Deploy the Traefik stack
	docker stack deploy --compose-file traefik/docker-stack.yml traefik
	docker service ls
	# Deploy the prototype stacks
	docker stack deploy --compose-file apps/proto1-stack.yml proto1
	docker stack deploy --compose-file apps/proto2-stack.yml proto2
	# Confirm the services are running
	docker service ls
	# Grab a list of the containers running
	docker ps

view raw commands.txt hosted with ❤ by GitHub