How we improved our deployment tool

Original post: https://pantographe.studio/en/blog/how-we-improved-our-deployment-tool

Today we are going to talk about our deployment process. Around a year ago, we start working on a script that helps us to manage our deployments.
The idea is to have a unique script that manages our deployments.
At start, the script was a quiet simple shell script that ran needed commands to make new deployments. We quickly added some actions like communicating informations about new release to Sentry, running migrations if asked, then, at end, notifying us on our Mattermost chat when deployment is complete.

But last summer we decided to completely rebuild it, by coding it in Ruby.
The script is now split into different parts that we call layers, and we create a classic event manager to use them.
One of the main ideas by moving to a Ruby script, is to have something more maintainable.

At Pantographe we currently use Heroku and Dokku as hosting providers for our clients applications. So our deployer only manages this two providers, for now.

Our tool main needs are the following:

• Having more consistent notifications (notifying at start and when a command failed with informations)
• Being able to put our sidekiq workers in quiet mode at beginning of deployment
• Having a script structure that is easier to maintain

From Shell script to Ruby script

So, why that choice? I'm pretty sure that the Ruby choice is not the best one. There is better languages or tools to manage this kind of needs. But since we do Ruby all days at Pantographe, for Rails application, it's easier for our team to add new features.
It will also allow us to add a test coverage on our script, which is always good.

The new tool

Our new tool needs some features:

• A Logger object to print steps into STDOUT
• An event manager to easily execute commands or actions in specific order inside separate classes (layers)
• Separating files (layers) to store code related to Dokku, Heroku, notifiers, thirds parties, ...

Having simpler files, make our code a bit easier to read and maintain.

We have some events that can be handled in layers. Here are the different events names:

• configure: used to configure deployment (ex: setting git remote)
• pre_deploy: to do some action just before deploy starts (ex: notifying team that a new deployment is starting)
• deploy: code to run to deploy application depending on the provider (ex: git push)
• post_deploy: whatever script that needs to be run after deployment has been done (ex: finalize sentry release)
• deploy_succeeded: used to notify on successful deployment
• deploy_skipped: used when application is already deployed
• deploy_failed: used to notify when deployment has failed

Also, we have 4 kinds of layers:

• Providers: contains code relative to the provider like dokku and heroku.
• Applications: contains code relative to the application to deploy (ex: Rails)
• ThirdParties: contains code relative to some third party services like Sentry.
• Notifiers: contains code that will trigger notification (like on our Mattermost chat)

Since dokku and heroku both use git the same way to deploy an application, we moved git related code inside a concern. We also did the same for SSH related code.

Here an example of layer with our dokku provider layer.

module Deployer
  module Layers
    module Providers
      class Dokku < Base
        include Git # Git concern.
        include SSH # SSH concern.

        required_env :DOKKU_HOST # Set DOKKU_HOST has a required env var.

        on :configure, priority: 50 do
          check_env_vars!
        end

        on :configure, priority: 200 do
          logger.indent "Server is using #{dokku_version}"
        end

        # Those two methods are accessible from other layers. It helps to
        # apply migration or restart worker process for example.
        def run(cmd, &block)
          run_ssh :run, "'#{app_name}'", cmd, &block
        end

        def process(process, action)
          run_ssh "ps:#{action}", "'#{app_name}'", process
        end

        private

        def check_env_vars!
          return if server_uri.user

          raise EnvVarError.new("DOKKU_HOST", "must be [e-mail]")
        end

        # This is called inside SSH concern.
        # By default it calls a simple `whoami`.
        def test_ssh_connection!
          run_ssh :version
        end

        # Overwrites default SSH concern method.
        def server_uri
          @server_uri ||= begin
            uri = URI("ssh://#{ENV["DOKKU_HOST"]}")
            uri.port = 22 if uri.port.nil?
            uri.user = "dokku" if uri.user.nil?
            uri
          end
        end

        def dokku_version
          capture(:ssh, "#{server_host} version").strip
        end
      end
    end
  end
end

All layers look more or less the same way.

GitLab CI

Having a deployment tool is a great thing. But this script main goal is to deploy automatically our Rails apps with our GitLab CI.
Here is a configuration example, in which, we select the provider by choosing the correct image and we set required env vars for provider, application, third party and notifier layers we want to use.

deploy:staging:
  image: registry.domain.tld/devops/ci-deploy/dokku:latest
  stage: deploy
  variables:
    CI_DEPLOY: "true"
  script: deploy
  variables:
    APP_NAME: "my-project"               # Required var by Deployer
    DOKKU_HOST: "[e-mail]" # Required by Dokku layer
    AUTO_MIGRATE_ON_DEPLOY: "true"       # Optionnal in Rails layer

Then in GitLab project CI settings, we could configure the following variables:

• SSH_PRIVATE_KEY: The ssh private key that must be used for deployment with dokku
• MATTERMOST_WEBHOOK_URL: Mattermost webhook
• MATTERMOST_CHANNEL: Mattermost channel to notify
• SENTRY_AUTH_TOKEN: Sentry auth token. Used with sentry-cli
• SENTRY_URL: Sentry instance url
• SENTRY_ORG: Sentry organization
• SENTRY_PROJECT: Sentry project

Conclusion

Now we have a clean tool that every developers of our team could easily upgrade. Using a layers structure and an event manager give us the availability to easily add new providers and connect new third parties without breaking the script.

Our tool is now running for months in production without issues, it's a possibility that we push it soon as open source.
Let us know if it's something that you may be interested in 🙂
I guess it could be also easy to make it usable with the recent feature GitHub Actions.