DEV Community: Firmhouse

Our deployment strategy for on-premise customers (in China)

Michiel Sikkes — Thu, 10 Sep 2020 10:14:58 +0000

Last year (2019), one of our customers asked if it was possible to run our full recurring commerce platform to sell product subscriptions in China. That sounded like a great challenge. Hosting something in China is not as straightforward as hosting something for the rest of the world.

In this article, I'll show you how we've adopted our deployment pipeline to support on-premise deployments for our customers. I'm taking China as an example with some unique challenges. But the following article can be used to set up an on-premise deployment for your application on any type of infrastructure.

What's up with China?

Doing business in mainland China is fully locked down for any non-Chinese company. To do business in China, you need a domestic business license for a specific product category, and you need to deal with The Great Firewall. The Great Firewall causes your connection to be really slow or unavailable when you're hosting outside of mainland China. So getting local infrastructure in mainland China is kind of critical.

Unfortunately, hosting or deploying apps on mainland China servers is also fully locked down and only accessible for Chinese citizens and companies. So that's what we were facing.

We had set up an on-premise environment of our platform in our client's hosting account on Aliyun (Alibaba Cloud). And in doing this, make sure our existing deployment processes and hosting tooling would still work.

Our approach

This is the deployment approach we ended up with. I'll explain below the detailed configuration and setup of each important component. I'll also provide some examples and snippets on specific configuration setups.

On every PR merge into our main branch on GitHub, the following steps are taken:

CircleCI runs our tests, security checks, dependency scanners, etc.
On a successful build, CircleCI kicks off a special production Docker image build.
CircleCI uploads the new production Docker image to a repository on Docker Hub. The Docker image is tagged with the Git commit SHA.
A developer logs into the on-premise environment and deploys the image via Dokku's container-based deployment. The Git commit SHA is used to identify the release to deploy.
Dokku runs its regular deployment steps, application restarts, asset compilation, migrations, etc.
New release is live!

Server and database setup

The deployment process is pretty independent from the final server or database setup. All you need to be able to do is run container images and have your various database services available.

Here's what we used in China on Aliyun (Alibaba Cloud). Parts can easily be adapted or replaced with any other cloud infrastructure or local component via Dokku plugins.

Aliyun Elastic Compute Service instance with Ubuntu LTS.
Aliyun ApsaraDB RDS for PostgreSQL.
Aliyun ApsaraDB for Redis.
Terraform and Ansible for creating provisioning the servers with our standard server setup.
Dokku as on-server hosting and deployment platform.

As you can see, we used Aliyun Cloud's managed services for PostgreSQL and Redis. However, you can change this up if you're deploying to an on-premise environment on some kind of (virtual) server in a datacenter somewhere.

You can for example use Dokku's Redis and PostgreSQL plugins. These plugins allow you to run Redis and PostgreSQL on the same single server as your application container runs on. Additionally, they make sure that only your application can access these services and by default they are not accessible through the public internet.

Dokku with container-based deploys

We're quite a big fan of Dokku for relatively simple on-premise deployments. Dokku is very well supported, easy to set up, and you have your apps running in no time.

Dokku takes care of deployment access control, deploy and migration steps, versions, scaling, webserver hosting, etc. It also has great plugins for backups, various databases, and other components you might need. You can configure ENV vars per Dokku-managed application so that you can set configuration settings and database connections at runtime.

Dokku is your own little mini app deployment Platform as a Service running on your own infrastructure. Read here on how to deploy your app via Dokku.

With Dokku, you can either deploy your apps using the git push deployment strategy, or have it deploy your Docker containers. At first, we used the git push deployment method. Later we switched to a Docker container-based deploy method.

Why we don't use "git push" for deployment

One problem with the git push method is that every on-premise server your software runs on has a slightly different version of your app running. Even if it's based on the same commit. This is because git push in Dokku will build a new container image on every deploy on every server. So you cannot be certain that your application image is exactly the same on each on-premise environment you manage.

In addition, the Docker container build is triggered for every deploy on every server. And deploying an app can be quite CPU-intensive for a short period of time. You then risk pulling down our production app if you do not have access to a large enough server.

For China we had an additional problem with the git push method, related to The Great Firewall. The internet connection from Europe to China is very unreliable and/or very slow. It could sometimes take hours to deploy a single commit, as our codebase would have to be pushed to the server in China. But Dokku also needs to download a lot of images and dependencies during a deploy. We'd see connections, stalling, being paused for hours, or simply timing out.

Switching to Docker image-based deployment

So all these problems with git push -based deployment resulted in us switching to the Docker Image-based deployments with Dokku. It was definitely more work to set up but in the end resulted in a much smoother and faster deployment process.

The Dokku documentation can tell you how to use Docker images for your deployments.

After switching, a deployment from our side basically looked like running the following commands on the on-premise server:

$ docker pull firmhouse/platform:<Commit SHA> 
$ docker tag firmhouse/platform:<Commit SHA> dokku/platform:<Commit SHA> 
$ dokku tags:deploy platform <Commit SHA>

Commands to deploy a new release in Dokku

Building the production Docker image

CircleCI is our CI of choice and it runs our test suites and it builds our Docker containers. It sometimes even deploys our apps straight away.

Here are a few configuration snippets on how we set things up on CircleCI to build a Docker image and push it to our Docker Hub account.

CircleCI configuration for building and pushing the image

We have a special build step in our CircleCI workflow builds our production image and then pushes it to Docker Hub.

For extra security we have a separate Docker Hub user for every repository so that we can easily revoke access from CircleCI in the case of a breach.

Here's the relevant parts from our circle.yml configuration. This files lives in our application codebase and is automaticaly picked up by CircleCI on every push to the repository on GitHub.

version: 2
jobs:
  build:
    # Regular build steps. Redacted from this snipper.
  build_and_push_production_image:
    working_directory: ~/circleci-app
    docker:
      - image: circleci/ruby:2.5
    steps:
      - checkout
      - setup_remote_docker
      - run:
          name: Checkout on-premise branch
          command: git checkout master
      - run:
          name: Build Image
          command: docker build -t firmhouse/platform:$CIRCLE_SHA1 . -f Dockerfile-production
      - run:
          name: Tag latest
          command: docker tag firmhouse/platform:$CIRCLE_SHA1 firmhouse/platform:latest
      - run:
          name: Login to Docker Hub
          command: echo $DOCKER_PASSWORD | docker login -u $DOCKER_USER --password-stdin
      - run:
          name: Push commit-specific image to Hub
          command: docker push firmhouse/platform:$CIRCLE_SHA1
      - run:
          name: Push latest tag to Hub
          command: docker push firmhouse/platform:latest

workflows:
  version: 2
  main_flow:
    jobs:
      - build
      - build_and_push_production_image
        requires:
          - build

Snippet from our CircleCI configuration

Dockerfile for production

We have a Dockerfile-production in our codebase that is used specifically for building the image to be deployed to production. It uses the officially supported Ruby base images with Alpine as base distribution. It is also set up as a multi-stage build so that we don't leave any development/build dependencies in the final image.

You'll notice some Ruby on Rails-specific bits in here. Those can be taken out or replaced with what's needed for your framework.

FROM ruby:2.5.8-alpine AS build-env

ARG RAILS_ROOT=/app

RUN apk update \
  && apk upgrade \
  && apk add --update --no-cache \
  build-base curl-dev git postgresql-dev \
  yaml-dev zlib-dev nodejs yarn tzdata

ENV RAILS_ENV=production
ENV NODE_ENV=production
ENV BUNDLE_PATH=vendor/bundle
ENV BUNDLE_APP_CONFIG="$RAILS_ROOT/.bundle"
ENV BUNDLE_PATH__SYSTEM=false
ENV RAILS_SERVE_STATIC_FILES=true
ENV RAILS_LOG_TO_STDOUT=true
ENV APP_HOST=dispatch
ENV SMTP_DOMAIN=localhost
ENV SMTP_USERNAME=username
ENV SMTP_PASSWORD=password
ENV SMTP_ADDRESS=xxx
ENV SECRET_KEY_BASE=123
ENV BUNDLER_VERSION 2.0.2

WORKDIR $RAILS_ROOT

COPY Gemfile* package.json yarn.lock ./
COPY Gemfile Gemfile.lock $RAILS_ROOT/
RUN gem install bundler -v 2.0.2
RUN bundle config --global frozen 1 \
  && bundle install --without test:development:assets -j4 --retry 3 --path=vendor/bundle \
  && rm -rf vendor/bundle/ruby/2.5.0/cache/*.gem \
  && find vendor/bundle/ruby/2.5.0/gems -name "*.c" -delete \
  && find vendor/bundle/ruby/2.5.0/gems -name "*.o" -delete

RUN yarn install --production
COPY . .
RUN bin/rails assets:precompile

RUN rm -rf node_modules tmp/cache app/assets vendor/assets test

FROM ruby:2.5.8-alpine
ARG RAILS_ROOT=/app
ARG RUNTIME_PACKAGES="tzdata postgresql-client nodejs bash file imagemagick"

ENV RAILS_ENV=production
ENV BUNDLE_APP_CONFIG="$RAILS_ROOT/.bundle"
ENV RAILS_SERVE_STATIC_FILES=true
ENV RAILS_LOG_TO_STDOUT=true
ENV BUNDLER_VERSION 2.0.2

WORKDIR $RAILS_ROOT

RUN apk update \
  && apk upgrade \
  && apk add --update --no-cache $RUNTIME_PACKAGES
RUN gem install bundler -v 2.0.2

COPY --from=build-env $RAILS_ROOT $RAILS_ROOT
CMD ["bin/rails", "server"]

Our Dockerfile-production

Docker Hub for hosting our images

We currently use Docker Hub for hosting our container images. For additional security, we have all our applications in their own Docker Hub repositories. We create additional user accounts per repository/application so we can put their credentials in CircleCI.

A pretty decent on-premise deployment mechanism

For us, this is a pretty decent on-premise deployment mechanism. We don't do many on-premise setups anymore as this is truly an exceptional enterprise customer requirement.

Our main (European) platform runs on Heroku, and we leverage all their nice features to deploy and scale our platform.

However, having the setup described in this article in place allows us to very easily add any on-premise environments if required by our customers. Since it's based on a container image it is also quite easy to make a scalable version out of this on a Kubernetes cluster.

Happy to answer any of your questions about this setup!

Remove flickering flash messages on Turbolinks

Michiel Sikkes — Fri, 24 Jul 2020 10:06:21 +0000

If you're using Turbolinks, and flash messages in your Rails app, then this might come in handy.

You might have seen flickering flash messages on your pages when you re-visit them in your app. This is because Turbolinks caches the full page content in its own internal cache. And this cache includes your flash message if you don't explicitly take it out.

Thus, when you revisit a page where a flash message was just displayed, you first see that cached page for an instance. Then the flash messages dissapears because Turbolinks asynchronously loads your actual page content via AJAX.

Here's a snippet we're using to take any flash messages out of the page before sending it to the Turbolinks cache:

document.addEventListener("turbolinks:before-cache", function() {
   const flash_message_element = document.querySelector(".flash")
   if (flash_message_element) {
     flash_message_element.remove()
   }
 })

Remove flash message before sending to Turbolinks cache

Managing app secrets in Kubernetes at Firmhouse

Michiel Sikkes — Mon, 08 Jul 2019 19:58:32 +0000

At Firmhouse we're gradually migrating from a Dokku-based infrastructure onto a new Kubernetes-based infrastructure setup running on DigitalOcean Kubernetes. Out with manual work. In with automation!

In the new setup we heavily rely on Terraform for Infrastructure as Code collaboration and automation. One thing that we're adding to our Terraform repository is automated management of secrets and ENV vars for our app deployments.

In this article I'll show you how we're managing secrets and ENV vars via Terraform, Azure Key Vault, and Kubernetes Secrets.

App Deployment in Kubernetes

First of all, let's show you the thing that this is all about in the end: the application deployment. Our Ruby on Rails applications are deployed using a Kubernetes Deployment. Here is a snippet of one of our applications from our Terraform repository:

As you can see, we load the application's environment from a single Kubernetes Secret named dispatch-env.

We also put all Kubernetes objects for a given application in a specific Kubernetes Namespace. Following this practice makes sure that by default we don't let secret stuff from one application deployment leak over to a different application.

ENV vars via a single Kubernetes Secret

Simply loading the application's environment from a single Kubernetes secret makes it easy to manage the whole runtime environment via a single Terraform resource. Here is a redacted and shortened example of one of our apps:

Secret storage: Azure Key Vault

In our new infrastructure setup we store our secrets in an Azure Key Vault. Azure Key Vault is comparable to AWS Key Management Service or HashiCorp Vault.

We used to not store secrets at all or have only certain manually generated secrets stored in our 1Password business account.

Because Terraform can talk to Azure Key Vault via it's Azure provider, we can now start managing secrets without ever touching them or making then visible to a human eye. This allows us to read a secret key from Azure Key Vault in our Terraform repository:

Setting up the application ENV variables with secrets from Azure Key Vault

So uptil this point we have three things prepared.

Applications are deployed using Kubernetes Deployments in their own namespace.
An application uses one single Kubernetes Secret to load the environment.
Azure Key Vault is set up and we can load secrets via Terraform resources.

Now let's bring it all together.

In the following redacted sample you'll see how we're pulling in SMTP login credentials and a database password to load those into the application deployment's ENV:

Conclusion

Why this is great

Such a setup is great for the following reasons:

Anyone in the team can take a look at our Infrastructure as Code git repository to see what ENV vars are configured for a given application deployment.
Secrets are safely stored in Azure Key Vault, including versioning, timestamps, access logs, and access policies for team members and applications.
We don't have to manually log in to Dokku servers or into Heroku anymore to define the ENV variables for our apps.

What can be improved

One thing that still bugs me is that the actual secret values are stored in a Kubernetes Secret. In practice, this is not worse than storing them in Heroku or Dokku. But it would even be greater to have a setup where the secret value is not visible to humans with access to the Kubernetes cluster.

Jon Arild Tørresdal has a solution in place at Sparebanken Vest. He has written great blog post about this setup. Check it out here: Introducing Azure Key Vault to Kubernetes

Running-multi server Dokku: problems and options

Michiel Sikkes — Tue, 07 May 2019 19:31:18 +0000

This blog post is a collection of resources and thoughts about running applications via Dokku on a High Available (HA) or multi-server setup. Since Dokku doesn't support multi-server out of the box but there are some efforts to make it work, this post is a meant of an overview of options that are out there.

For me, there are two reasons for running applications in a multi-server setup:

Most important: being able to apply maintenance updates, update a kernel, and reboot a server without application downtime.
Making sure our applications can scale "horizontally" to multiple servers at increased load.

So which problems do we need to solve to scale Dokku deployments horizontally? To figure that out, let's first define the simplest imaginable multi-server Dokku setup.

The simplest imaginable multi-server Dokku setup

A simple imaginable setup would be: one load balancer (or reverse-proxy), and two "backend" servers that run Dokku and run the apps.

In this case, we won't expect Dokku to deal with routing or load balancing logic. That's what a load balancer is for. We simply want Dokku to play nice with having a brother on another server nearby, and we want to make maintaining that as easy as can be.

So what problems do we have to tackle to make this happen?

What Dokku "lacks" for multi-server

The most important thing would be that all app definitions, configuration options, ENV vars, domain names, SSL certificates, etc. are all stored on-server. This means that when running two Dokku servers, all application configuration would have to be defined and kept in-sync on both servers in the "simplest imaginable multi-server Dokku setup".

So we have problem one: keeping configuration in sync

The second problem is moving up domain mapping and SSL termination to the load balancer. Dokku provides awesome mechanisms for mapping domains to apps, installing SSL certificates, or using LetsEncrypt via the dokku-letsencrypt plugin.

However, since traffic will be coming in on the load balancer (or reverse-proxy), it needs to take over the role of recognizing the domain names, owning the SSL certificates and forwarding the rest of the traffic via an internal network or internally encrypted self-signed certificate to the backend servers.

So we have problem two: moving the routing and certificate part of the infrastructure to the load balancer

So what are the options?

Here's a list of options I think might be useful to start thinking about multi-server Dokku setups:

(Not really an option) Set up up the two Dokku servers, and keeping them in sync manually. Then putting a load blancer from your cloud provider, or a reverse-proxy like Treafik in front to deal with SSL certificates and routing to the two Dokku servers.
Automating maintenance of your Dokku servers via Ansible. There is a new repository live on GitHub where josegonzalez is working on Ansible scripts to maintain Dokku servers. By managing Dokku installs and their application settings via Ansible, it will already make it way easier to keep two Dokku servers and it's app settings "in sync". You will still need to put a load balancer or reverse-proxy in front of the two servers. One bigger challenge here is managing secrets. If you configuring your servers via Ansible, you'll need to set up some kind of secrets vault or other method to inject secrets into the Ansible runs when they update your server and app definitions.
Using different tools for setting up your own PaaS on your own server. This article on ServerFault seems to be updated with recent options, but I haven't taken a look at them personally: https://serverfault.com/questions/640038/scaling-out-dokku-infrastructure
Wait for Intercity to support multi-server setups. Intercity is the management panel for Dokku that we've built internally at Firmhouse. It's an open source project that you can run yourself on your own server. We're currently working on a feature to keep application settings in-sync across multiple servers.

CTO advice: A checklist for using cloud databases securely

Michiel Sikkes — Mon, 06 May 2019 18:31:32 +0000

Cloud-provided managed databases are great. Especially when you're CTO of a small company, like me. No sleepless night over (nonexistant) backup procedures, encryption-at-rest, firewalling, critical software updates. Production, Enterprise-grade Redis and PostgreSQL at your fingertips in a matter of minutes.

Sounds easy, but there are several things you need to consider. The database is not yours , it's theirs - so take good care who you entrust your (customers) data with and what you put in place to make it as secure as can be.

Here are 8 pieces of advice that are on my "managed database provider" checklist, in no particular order of importance. The following is pretty much a copy-paste from our own risk management assessment and security baseline documents at Firmhouse. Use at your own will (or peril!)

Ensure compliance with legislation and ensure secure standards

Picking just any provider because they offer the easiest and cheapest "one-click install" cloud databases is simply naïve. Always be sure that these providers have some information management certification, like ISO27001. Also make sure that their physical datacenters are operated under PCI standard and have SOC2 Type II reports available.

On top of that, if you cannot publicly get access to their security procedures or documentation, that's a bad sign.

Oh, and being able to sign a GDPR-compatible Data Processing Agreement/Addendum is also a pretty must-have! Must-have if you're a European company. And pretty important if you don't want your company to skip the whole European market, readily awaiting to give you money for your service.

We use Aiven.io as our managed database service, and they have pretty detailed information on both their compliance documentation (https://aiven.io/security-compliance) and logs of detailed information about their security, storage, and backup procedures (https://help.aiven.io/security/cloud-security-overview).

Service accounts and role-based authorization

Always create a service account/user to a database per system that's using it. If you only have one application that reads and write to your database, create a user for that application to your specific database.

Have some external or 3rd party reporting tool that just reads information from your database for dashboarding or business intelligence? Go ahead and create a read-only service account for just that user on the database.

Bottom line, just like for any other account: never share an account between multiple users or services.

Backup snapshots and point-in-time recovery

A database without automated backups cannot call itself a serious managed database service. Make sure your database provider offers automated backup snapshots and that they also support point-in-time recovery. With point-in-time recovery way you can quickly get your database back into a state from a few hours ago without loosing too much recent data.

I've never had the need for it luckily, but I'm sure I'm shooting myself in the foot by typing this now.

Have off-vendor backups

Yes backups are great. But what if your vendor goes bankrupt or due to some lawsuit is legally required to stop any active services they're providing? For these reasons, always export database snapshots to a 3rd party location and keep them stored there for 14 to 30 days. In the case that your vendor is wiped from the earth for whatever reason, you can at least start a recovery procedure to a different vendor that way.

Now, let's make sure an attacker can't do anything with the data in case it does get stolen somehow.

Encryption-at-rest

Making sure live data and backups are encrypted is a must-have. It is a good security measure against the once in a lifetime occurance that someone does get unpermitted access to a hard drive and rips it out of a server. But it's also just something practical: if you want to sell software to The Enterprise and Corporate, this is simply an important security requirement.

Encrypting your data "at rest" is pretty important. But encrypting it in-transit is even more important. Head over to the next paragraph.

SSLmode enabled by default

All serious databases (like PostgreSQL or Redis) allow you to connect to them over SSL. If your managed provider does not support this. Run away fast. Encryption of data-in-transit is just a must have to keep your data secure without people from sniffing around in your client's data.

However, don't think you're already done by simply using the SSLmode of your database connection! Nasty things can happen if you don't configure the thing from the next paragraph.

Configure SSL certificate pinning

Most managed database servers generate a self-signed certificate for securing the connection to the database. Without our applications verifying if the database they are talking to, is actually the database they were meant to be talking to, SSL encryption pretty much doesn't matter.

You need to make sure that your applications will only connect to the database service with the exact same SSL certificate as they are expecting. If your applications allow connections to any database service with an SSL certificate, you get caught by "the man in the middle". With this technique (and some additional hacks in/around your network) someone can spoof the database service and collect all the connection information that it needs. When this happens, the "man in the middle" essentially gets access to your full database, if you haven't applied the next and last security measure.

Private Networking/Firewall/IP whitelisting

Last but not least: make sure our managed database service allows some kind of networking constraints. This can either be a private network where your database lives in the same (virtual) network as your application servers. Or it can be a true public firewall with an IP whitelist if the service is accessed over the public internet.

That's it for now! Hope you enjoyed this post and that this "checklist" helps you in your day-to-day job. Or that it made you realize that you have a security gap somewhere. No biggy! Just calmly fill the hole and you're good to go for a good night's sleep again.

Upgrading an existing Intercity installation to the new installer

Michiel Sikkes — Mon, 29 Apr 2019 19:06:43 +0000

Two days ago, I posted about the updated and simpler installation method I shipped into Intercity's master branch: Updated installing Intercity to a single command. The new command works great on a fresh new Ubuntu LTS. 🕺💃

However, if you've previously installed Intercity via the intercity-server command as described in the previous docs/installation.md, you'll have to perform some additional steps as the database setup is different. You'll have to migrate your current database into the database of the new installation. You definitely don't want to loose al your precious app configuration settings and secrets!

What you'll have to do is the following. I'll explain each step in details below.

Export the current Intercity database via pg_dump.
Stop your current Intercity installation.
Install the new Intercity via the new installation procedure.
Import the database from your previous installation into the new one.

Before you perform the above steps: make sure you have a backup of your server. For example by using the backups/snapshots feature of your VPS provider or Cloud.

You can also copy the directory /var/intercity/shared/postgres_data to /var/intercity/shared/postgres_data_backup for example:

$ sudo cp -r /var/intercity/shared/postgres_data /var/intercity/shared/postgres_data_backup

Ok, so here we go:

Export the current database

Look up the Docker container ID of your current Intercity installation:

$ sudo docker ps

You should see something like this:

db76d2869d40    local_intercity/app "/sbin/boot"

"db76d2869d40" is the container ID of your current Intercity installation. Use it in the next few commands to access a shell in that running container, export the database, and bring it back to your host system:

$ sudo docker exec -it db76d2869d40 bash
(container) # su intercity
(container) $ cd /home/intercity
(container) $ pg_dump -U intercity -d intercity -f intercity.sql
(container) $ exit
(container) # cp /home/intercity/intercity.sql /shared
(container) # exit

You have now succesfully exported the database from your current Intercity environment into a file intercity.sql in /var/intercity/shared on your host system. We'll use this file in one of the next steps to import into your new Intercity installation.

Stop your current installation

You can safely stop your current Intercity with the following command, using the container ID you fetched in the previous steps:

$ sudo docker stop db76d2869d40

Install Intercity via the new installation procedure

$ mkdir intercity
$ wget https://raw.githubusercontent.com/intercity/intercity-next/master/scripts/bootstrap.sh
$ sudo bash bootstrap.sh

After a few minutes your installation is running and the "Create your first user" screen should be visible on the domain name you've configured. Please verify that you see the "Create your first user" screen to ensure your new installation is fully booted up!

Import your Intercity database

Now we're going to import the intercity.sql database into the new Intercity installation.

Run the following commands to do so:

$ sudo -s
# cat /var/intercity/shared/intercity.sql | docker-compose exec db psql -U postgres -d intercity

You should see a lot of output in your terminal. This indicates that the database is being imported. You probably see a lot of errors in this output, telling you relations or indexes already exist. This is fine, as the clean bootstrapped database from the new Intercity install already created those. This command is now only importing the data.

To check if the import was succesful: head over to the URL that your Intercity installation is hosted on. Instead of the "Create your first user" screen, you should now see the regular login screen again. If you see the login screen: import succesful!

Awesome! I hoped this procedure worked for you. If not, or if you're getting errors: let me know!

A new single command for installing Intercity

Michiel Sikkes — Sat, 27 Apr 2019 14:19:25 +0000

It just got extremely easy to install Intercity - the web UI for managing Docker applications, plugins, and application deployments. After the excellent initiative and support by Ariejan de Vroom and Jeroen van Baarsen from last November 2018, the big PR is finally, merged in! 🥳

The new way of installing and updating Intercity is as easy as running the following commands and following the setup procedure:

$ wget https://raw.githubusercontent.com/intercity/intercity-next/master/scripts/bootstrap.sh
$ sudo bash bootstrap.sh

Running the bootstrap.sh script will make sure Docker and Docker Compose are installed on your server. It will also ask you some setup questions and the configuration settings file. Finally, it will download and start the required servies to run Intercity, like an nginx-proxy with LetsEncrypt, PostgreSQL, and Redis.

In theory, running bootstrap.sh will also allow you to upgrade to the latest version of Intercity as we'll be publishing new releases of Intercity to the officially supported image on Docker Hub: https://hub.docker.com/r/intercity/intercity_next. Every time the image gets updated, you can run bootstrap.sh to have Docker Compose pull it down and restart your services.

Upgrading from the old way of installing Intercity is fairly easy, but you do need to make some file migrations of the PostgreSQL database files. An automated upgrade path or documentation for this will be ready as soon as I've tested this with my personal already running Intercity deployments.

Behind the scenes: Adding custom domain mapping to Airstrip via Traefik.io

Michiel Sikkes — Fri, 16 Nov 2018 09:55:04 +0000

At Firmhouse we have a product called Airstrip, which lets people quickly build a website to test and launch their new business proposition. In our current product sprint, we're improving a feature that allows people to automatically map a custom domain to the website they build with Airstrip. This post explains how we're doing that.

We've had a custom domain mapping feature in there for quite a while now, but it wasn't properly automated yet. How it would work is that we would instruct people to get in touch with our support team. We would then tell them to point their DNS records to our IP or CNAME and we would then manually configure their domain by adding a site to our Dokku instance and re-running $ dokku letsencrypt.

This way of working is long and requires quite some manual work. There's also no "instant gradification" for our user. It's quite the bummer if you have to wait more than a day before your custom domain is fully working!

We've also hit the limitations of using the Dokku Letsencrypt plugin for this. Mainly: if one of the domains added to our Airstrip app on Dokku has the wrong DNS records set up, then all domains would fail to renew the certificate.

So what's our new plan?

After some researching and consideration, we're going to add Traefik to our infrastructure. Traefik is a "Cloud Native Edge Router" (it says so on their website). Now I'm not fully sure what that means but it turns out it's pretty awesome as a thin layer in front of our current infrastructure for custom domain mapping and automatically LetsEncrypt'ing those domains at the same time! Everything we need for now.

Traefik will be put in front of our current infrastructure that is run on DigitalOcean droplets. The web and app server droplets are provisioned and maintained via Ansible and Intercity. The droplets run Dokku for app deployment and webserver configuration. We'll add another droplet that runs just Traefik.

We'll configure Traefik with a frontend per custom domain added by our users. Whenever someone adds their custom domain in the Airstrip user interface, we'll fire an API call to the Traefik REST api endpoint. All of the frontends that are created this way, are then connected to a single backend: the Airstrip app server we already have in place. Traefik will then take care of requesting a new LetsEncrypt certificate for every custom domain added to Traefik's configuration.

To illustrate, here's a Treafik configuration file to indicate how this would work:

# Backends
[backends]

  [backends.backend1]

    [backends.backend1.servers]
      [backends.backend1.servers.server1]
        url = "https://airstrip.firmhouse.com:443"
        weight = 1

# Frontends
[frontends]

  [frontends.frontend1]
    backend = "backend1"
    passHostHeader = true

    [frontends.frontend1.routes]
      [frontends.frontend1.routes.route0]
        rule = "Host:traefik.firmhouse.com"

  [frontends.frontend2]
    backend = "backend1"
    passHostHeader = true

    [frontends.frontend2.routes]
      [frontends.frontend2.routes.route0]
        rule = "Host:traefik2.firmhouse.com"

It's pretty awesome that Traefik can take all of this off our hands. We were initially considering writing some custom code to dynamically add/remove domains to Dokku or updating our webserver and LetsEncrypt configurations via other methods. Setting up a small Traefik droplet in front of our infrastructure is only a minor effort compared to that.

Next up is looking into how we can attach a Key/Value store to Traefik so that our dynamic configration doesn't get lost every time we restart the Treafik server or we need to upgrade the droplet.