DEV Community: ohffs

Traefik v2 with Docker Swarm

ohffs — Fri, 06 Dec 2019 09:55:14 +0000

Traefik v2 with Docker Swarm

I've been a happy user of Traefik all through the v1.x series but with v2.1 coming out I began to have a proper look at upgrading. The docs are very thorough, but as with a lot of thorough docs also not very enlightening about 'how do I do the thing?'.

So after a bit of faffing about, watching yotube videos (the files here are modified versions of the compose-style ones attached to the video) etc I've got something running. This is a very basic 'just get it up and running' example - mostly as an aide-memoire for myself and hopefully to give some pointers to other people migrating from v1 to v2. I'm assuming familiarity with Traefik v1 so I'm not documenting everything line by line.

The stack files

Our setup is a traefik instance running listening on an overlay network called 'proxy'. Any web apps that need to talk to the outside world also sit on that network and have the magic traefik labels set so they get picked up. So the v2 traefik file I have so far is :

version: "3.3"

services:
  traefik:
    image: traefik:v2.0
    restart: always
    container_name: traefik
    ports:
      - "80:80"
      - "8080:8080" # traefik dashboard
      - "443:443"
    command:
      - --api.insecure=true # set to 'false' on production
      - --api.dashboard=true # see https://docs.traefik.io/v2.0/operations/dashboard/#secure-mode for how to secure the dashboard
      - --api.debug=true # enable additional endpoints for debugging and profiling
      - --log.level=DEBUG # debug while we get it working, for more levels/info see https://docs.traefik.io/observability/logs/
      - --providers.docker=true
      - --providers.docker.swarmMode=true
      - --providers.docker.exposedbydefault=false
      - --providers.docker.network=proxy
      - --entrypoints.web.address=:80
      - --entrypoints.web-secured.address=:443
      - --certificatesresolvers.mytlschallenge.acme.httpChallenge.entrypoint=web
      - --certificatesresolvers.mytlschallenge.acme.email=you@whatever.com
      - --certificatesresolvers.mytlschallenge.acme.storage=/letsencrypt/acme.json
    volumes:
      - letsencrypt:/letsencrypt
      - /var/run/docker.sock:/var/run/docker.sock
    networks:
      - proxy
    deploy:
      labels:
        - "traefik.enable=true"
        - "traefik.http.routers.api.rule=Host(`traefik.yourdomain.com`)"
        - "traefik.http.routers.api.service=api@internal" # Let the dashboard access the traefik api

networks:
  proxy:
    external: true

volumes:
  letsencrypt:

And a basic example wordpress stack file :

version: "3.3"

services:

  wordpress:
    image: wordpress
    restart: always
    container_name: wp
    environment:
      WORDPRESS_DB_HOST: db
      WORDPRESS_DB_USER: exampleuser
      WORDPRESS_DB_PASSWORD: examplepass
      WORDPRESS_DB_NAME: exampledb
    volumes:
      - wordpress:/var/www/html
    networks:
      - proxy
      - backend
    deploy:
      labels:
        - "traefik.enable=true"
        - "traefik.http.routers.wordpress.rule=Host(`wordpress.yourdomain.com`)"
        - "traefik.http.routers.wordpress.entrypoints=web"
        - "traefik.http.services.wordpress.loadbalancer.server.port=80" # it seems you always need to give traefik a port so it 'notices' the service
        - "traefik.http.routers.wordpress-secured.rule=Host(`wordpress.yourdomain.com`)"
        - "traefik.http.routers.wordpress-secured.entrypoints=web-secured"
        - "traefik.http.routers.wordpress-secured.tls.certresolver=mytlschallenge"

  db:
    image: mysql:5.7
    restart: always
    environment:
      MYSQL_DATABASE: exampledb
      MYSQL_USER: exampleuser
      MYSQL_PASSWORD: examplepass
      MYSQL_RANDOM_ROOT_PASSWORD: '1'
    volumes:
      - db:/var/lib/mysql
    networks:
      - backend

networks:
  backend:
  proxy:
    external: true

volumes:
  db:
  wordpress:

Using it

# assuming you are on a swarm master node
docker network create --driver=overlay proxy
docker stack deploy -c traefik.yml traefik
docker stack deploy -c wordpress.yml wordpress

After a short delay you should be able to visit the urls defined in the stack files on both http and https.

CI/CD

As each traefik-enabled service now has labels that have names to make them unique (eg, traefik.http.routers.wordpress.entrypoints=web) having a stack file with something like traefik.http.routers.${STACK_NAME}.entrypoints=web, traefik.http.routers.${STACK_NAME}-secured.entrypoints=web-secured is probably worth thinking about so you can do :

export STACK_NAME=wordpress
docker stack deploy -c wordpress.yml ${STACK_NAME}

and tie things together.

Further

Obviously this is a very basic setup. To take this into production you'd be looking at consul for the letsencrypt store, sensible deploy: flags, not giving traefik access
to the docker socket directly etc. But as a 'how on earth do I use v2' I hope it helps someone and saves them having to dig through things for
as long as I did.

Local slang or words in code

ohffs — Mon, 18 Mar 2019 18:27:48 +0000

I was looking back on a bit of Python code I wrote a year or two ago for a nanotechnology facility and remembered that as part of their process they move the 'device' back and forth a few nanometers before finalising their settings. I coded the function that did it as 'shoogle'.

(ˈʃʊɡəl ) or shoogie (ˈʃʊɡiː) dialect, mainly Scottish
verb

to shake, sway, or rock back and forth noun

a rocking motion; shake

I still feel absurdly pleased with myself that some £50m nanotech plant is 'shoogling' every few hours.

I got to wondering - what words have you snuck into code that you are secretly pleased with?

Story of a little bug

ohffs — Mon, 19 Nov 2018 11:02:46 +0000

One of our applications has a feature where a user can toggle an option which sends an email out to notify someone else. The email is delayed by 30 minutes so that the user can change their mind (or realise they toggled the wrong thing in the first place...). All well and good.

A while ago there was a small change to make to the code and email template, so we pushed out a new version. That push is via deployer which does 'zero downtime' deploys by symlinking a release directory to a 'current' one which the app is actually served from. It keeps the previous five releases by default so you can roll back if needed. All well and good.

The app uses a background queue to send out this particular email - the job is delayed by 30 minutes and before it finally sends it checks if the toggle is still in the 'yes - do that' mode. When we deployed the mentioned change we manually restarted the whole queue just to make sure it was in sync with the new code. Again, all was well and good - everything was working fine. We pushed out another handful of small updates over the course of the day.

Six weeks later...

We had an email from one of the office administrators saying they didn't think emails were going out from the system any more. We'd had a fairly brutal power-cut just a week or so before and my first thought was 'oh, damn - something hasn't restarted properly'. Although there hadn't been any errors caught by Sentry - our log monitor hadn't spotted anything either.

Checking the queue status showed that it was marked as 'paused'. I'd had that before where two apps were sharing a Redis instance and they got confused about which jobs they should be processing - sure enough there was another app on the server and it was pointing at the same Redis db. So we switched the db value up one on the problematic app and restarted the queue - which were now showing as healthy and a test message went out ok. "Phew!" I thought - nice easy fix and made a mental note to finally get round to making sure all the apps had a designated db so they never conflicted.

Then I started to think - 'hang on, that other app has Redis configured - but it doesn't actually use the queues. It was just a default we've got into the habit of including as "we'll probably use Redis at some point, include the config by default" kinda deal. So did a little more digging...

Checking the apps own error log showed nothing of interest. So checked the syslog on the server itself and saw lots of 'Could not open file /path/to/release/65/' messages from the queue worker before I had restarted them. Which was... odd. Especially as our log monitor hadn't picked it up.

The queue workers are started via systemd and point to the /path/to/current/ so that they should be using the latest release when they are run. But un-noticed by me, they actually see the symlink and resolve it to the 'real' directory so had gone to '/path/to/release/65/' which was where 'current' was pointing at the time.

Since the last manual 'restart the whole queue process' there had been five other small changes pushed out - each of those should have restarted the queue workers so it shouldn't have been a problem. But, again, un-noticed by me - the command used to restart the queues doesn't restart the master process - just tells the master process to restart the workers based on it's own idea of the path. So after the fifth push the master process was now pointing at a directory which the deployer command had removed as being the sixth-oldest release...

Sentry hadn't picked it up as it was never able to load enough of the code to register the handler. It also turned out that when the server had been built the person hadn't run Puppet on it so it hadn't installed the log forwarding/monitor setup. And the systemd process handler thought all was well as the master process continued to run quite happily as it had loaded all of it's code into RAM.

Wrapping up...

So all in all - the system hadn't sent out a copy of the email for almost six weeks. Thankfully it was a very quiet time of year for that particular feature so only a handful of these notifications had gone awol. But it was an interesting morning tracing back the errors, why all the layers of monitoring hadn't noticed, how the symlinking caught us out and piecing together the history of the server itself.

And now in addition to the existing monitoring, I've taken a bit of the code from the front-end that showed us that the queues were 'paused' to begin with and used it to make a little cron job that does alert Sentry if all isn't well. Probably... ;-)

Basic HA setup with (docker|kubernetes)

ohffs — Sat, 06 Oct 2018 14:30:01 +0000

We have a whole lot of web apps and services at work - most of them are not internet-facing as they are internal 'admin' type tools. At the moment we run them as fairly 'dumb' KVM virtual machines (dumb in the sense that there's no automatic fail-over and no shared storage so each vm is a disk image on the server it's running on).

I've been working on making the apps run inside of docker containers - initially just as a way of avoiding the annoying 'this needs PHP 5.3, that needs PHP 7.2, this needs python 2.7 + ...' issues. But I started getting more into the idea of using Docker Swarm or Kubernetes to give a more 'seamless' deployment process and allow for some kind of automatic fail-over of the apps.

After quite a lot of reading and experimentation - it feels like kubernetes is overkill for our needs. We basically have 10-20 apps running - and the world doesn't end if they're offline for a short while. The complexities of kubernetes seem quite high compared to Docker Swarm. On the other hand - it doesn't feel like Swarm is winning out in the container world.

Anyway - my current thinking is to go with the following setup :

3xServers running docker swarm (we have two nearby physical sites)
A virtual IP managed by something like keepalived to be the primary 'in-road' to the cluster
traefik to act as the swarm ingress
Probably ceph as a storage layer

The end goal is that if a server running App-X goes down, then App-X is started on another server and traefik takes care of routing to the new instance. If the primary server goes down, the keepalived brings up the IP address on another box and the traefik instance there starts handling the routing to the apps.

Most of the tutorials & guides I've found are either 'running hello world in swarm!' or 'bringing up 10,000 pods inside GKE!' with very little in the 'I'd just like stuff to kinda work if the power cord is pulled out of a box by accident' kind of thing ;-)

I'd be really interested in anyone's thoughts. I'm hoping to try and document the process to fill in the 'bigger than hello-world, smaller than Walmart' gap I seem to be falling into ;-)