How To Automate Deploys With Gitlab CI/CD And Deployer

The development of an application does not end with the programming itself. After its completion, you need to upload it to the server so as the users can use it. This can be done in many ways. Manually or automatically. In this post, I will show you how to upload the application and new functionalities automatically to the server. I will use Gitlab CI/CD for this process, which will run unit tests and then upload the code to the hosting using the Deployer.

Why Deployer?

Deployer is a simple tool which allows you to deploy the application to the server. It automates the entire process which we would have to do manually without it. Nobody wants to do repetitive activities manually. Thanks to Deployer, we can create our own tasks and run them in the selected order. The tool already has built-in presets for various frameworks such as Symfony, Laravel, Wordpress and many more. The undoubted advantage of Deployer is that it allows you to rollback the code to the previous version in case something goes wrong. It also has the ability to configure many deployment environments, so that before releasing the code to production, we can, for example, upload it to the test environment, so that the QA team can test the new functionality.

This tool allows you to deploy code in two ways. The first is to perform the entire build process on the server on which the application will be running. Deployer will then run commands on the target server where it will download the repository, dependencies, etc. Another way is to build the application once (locally or in the CI/CD) and upload the finished application to a selected place on the server. In this post I will show you both approaches.

Hosting Preparation

In order to upload an application to our hosting, the Deployer must have access to it. Connection to the hosting will be via SSH with key authorization. To enable such authorization on my hosting, it is required to generate a pair of keys using the ssh-keygen command, and then add the public key to the~/.ssh/authorized_keys file. We will pass the private key to Deployer so that it can connect to the hosting.

Additionally, as mentioned above, Deployer will download the code from the repository on our hosting. Therefore, you need to provide access to the repository and other dependencies on your hosting. To do this, put the appropriate private keys into the ~/.ssh directory. This step is required only if we want to build the application on the target server.

Finally, we need to grant the appropriate permissions to the created files, e.g .:

• chmod 700 ~/.ssh
• chmod 600 ~/.ssh/*

Deployer Configuration

Installation

To install Deployer, run the command composer require deployer/deployer --dev which will add it to the vendor directory.

Then create the deploy.php file in the root directory of the application. You can do it manually or by executing the following command:
vendor/bin/dep init.

Configuration

As I mentioned before, Deployer offers built-in presets for deploying applications. In this case, I used the Symfony 4 preset. It contains, among other things, such part:

desc('Deploy project');
task('deploy', [
    'deploy:info',
    'deploy:prepare',
    'deploy:lock',
    'deploy:release',
    'deploy:update_code',
    'deploy:shared',
    'deploy:vendors',
    'deploy:writable',
    'deploy:cache:clear',
    'deploy:cache:warmup',
    'deploy:symlink',
    'deploy:unlock',
    'cleanup',
]);

Here we can see the steps which will be performed after running the vendor/bin/dep deploy command. They are already implemented, but nothing prevents you from overwriting them or adding your own. You can even override the entire deploy command with a different list of steps in the argument.

Below, I paste the deploy.php file configured for my application. In the comments, I described what the individual elements are for.

<?php
namespace Deployer;

// Usage of built-in recipe
require 'recipe/symfony4.php';

// Here we set the name of the directory in which the particular releases will be located
set('application', 'myapp');

// Repository address from which the application will be downloaded
set('repository', '[e-mail]:xyz/app.git');

// Set the options with which the `composer install` command should be invoked
set('composer_options', '{{composer_action}} --prefer-dist --no-progress --no-interaction --no-dev --optimize-autoloader');

// Set how many releases should be kept by Deployer on the server.
// 3 means that we can go 3 releases back. -1 keeps all releases
set('keep_releases', 3);

// Specify which files are to be shared between the releases.
// In my case, it will only be a file with environment variables
add('shared_files', ['.env']);

// List which directories are to be shared between releases.
// In my case, it will only be a log directory
add('shared_dirs', ['var/log']);

// List of directories which must have write permission on the web server.
add('writable_dirs', ['var/log', 'var/cache']);

set('writable_mode', 'chown'); // we can choose from: chmod, chown, chgrp or acl.

// Web server user
set('http_user', 'xyz');

// Default stage. If no parameter is specified after calling the `dep deploy` command,
// the code will be deployed into the stage defined here
set('default_stage', 'prod');

set('ssh_multiplexing', true);

// Configure the server to which the code will be deployed.
// Provide here the parameters related to access, i.e. address, user or key path.
// Additionally, we choose which git branch will be deployed and provide the directory where the application will appear.
// We can define multiple such hosts in this file, e.g. additional one as a test environment
host(getenv('HOSTING_HOST'))
    ->stage('prod')
    ->user(getenv('HOSTING_USER'))
    ->port(getenv('HOSTING_PORT'))
    ->identityFile(getenv('HOSTING_SSH_KEY_PATH'))
    ->addSshOption('StrictHostKeyChecking', 'no')
    ->set('branch', 'master')
    ->set('deploy_path', '/home/xyz/domains/xyz.pl/{{application}}')
    ->forwardAgent()
;

// Set the path to the PHP version used by our application
set('bin/php', function () {
//   return locateBinaryPath('php7.4');
    return '/usr/local/bin/php74';
});

// I have overwritten the database migration command from the preset
task('database:migrate', function () {
    $options = '{{console_options}} --allow-no-migration --all-or-nothing --no-interaction';
    run(sprintf('{{bin/php}} {{bin/console}} doctrine:migrations:migrate %s', $options));
})->desc('Migrate database');

// In my case, to make the application available under the selected domain, the index.php file should be placed in the directory `/home/xyz/domains/xyz.pl/public_html`.
// The current version can be found in `/home/xyz/domains/xyz.pl/myapp/current/`
// Therefore, after a successful deployment, I create a symlink to the files in the deployment's public directory
task('deploy:symlink', function () {
    run( "ln -nfs {{deploy_path}}/current/public/* {{deploy_path}}/../public_html" );
});

// After the cache is warmed up, I perform a database migration
after('deploy:cache:warmup', 'database:migrate');

after('deploy:failed', 'deploy:unlock');

The above configuration is enough to upload the application to the server manually from the command line. The only thing to do is to call the vendor/bin/dep deploy command, and the application will be available to users after a while.

Failure handling

Sometimes there may be a situation where changes uploaded to the server cause an error, and the application is not working properly.
We can then revert code to the previous version using the command:

php vendor/bin/dep rollback prod

The Deployer will then point the symbolic link back to the previous version, and the application will start running again on the previous code.

Gitlab CI/CD Configuration

In modern applications, the whole process of deploying is usually automated. Before the release of the new version we usually want to run static analysis of code or tests. Or build JavaScript and CSS files. It can take some time. So let's go a step further and use the Gitlab environment so that the entire process is done for us.

Implementation Scheme

In order for Gitlab to run its CI/CD environment, the .gitlab-ci.yml file should be added to the application. It contains the steps we want to run to deploy the code. In this case, we will divide the process into three stages:

• building an application with dev dependencies - PhpUnit and Deployer are added to composer in the require-dev section.
• on the application built this way we will run tests.
• if everything goes well, we will run the Deployer script, which will upload the code to the server and build it there. As you can see in the configuration below, the first two steps will be performed on all branches of our repository. The third one will be launched only for the master branch, i.e. if you merge changes to this branch.

variables:
  COMPOSER_ALLOW_SUPERUSER: 1
  COMPOSER_DEFAULT_OPTIONS: '--optimize-autoloader --classmap-authoritative --no-progress --no-suggest --prefer-dist'
  DOCKER_IMAGE_PHP: 'registry.gitlab.com/xyz/app/php7.4:latest'

stages:
  - build
  - test
  - deploy

build:
  stage: build
  image: ${DOCKER_IMAGE_PHP}
  script:
    - composer install ${COMPOSER_DEFAULT_OPTIONS}
  variables:
    APP_ENV: prod
  artifacts:
    paths:
      - vendor/
      - bin/

phpunit:
  stage: test
  image: ${DOCKER_IMAGE_PHP}
  script:
    - php vendor/bin/phpunit
  variables:
    APP_ENV: test
  dependencies:
    - build

deploy_prod:
  stage: deploy
  image: ${DOCKER_IMAGE_PHP}
  script:
    - mkdir ~/.ssh
    # Private key to access hosting
    - echo "${PRIVATE_KEY}" > ~/.ssh/id_rsa
    - chmod 700 ~/.ssh
    - chmod 600 ~/.ssh/id_rsa
    # We put a specific commit - then in case of any errors in the application, we can redeploy the previous version
    - php vendor/bin/dep deploy prod --revision="${CI_COMMIT_SHA}" 
  only:
    # Branch on which this step will be performed
    - master
  dependencies:
    - build
  # Deploy can be launched automatically when the entire pipeline passes without errors, or manually
  when: on_success

Environment Variables

Some environment variables are set in the .gitlab-ci.yml file, e.g. DOCKER_IMAGE_PHP. Others are not, such as PRIVATE_KEY.
We do not put confidential information in this file, because everyone would have access to it. We add them in Gitlab in the section Settings -> CI/CD -> Variables. There they will be available only to granted users.

Docker Image

The Docker image is used in each step. This is where all the code will be run — download code from the repository, download dependencies, test and deploy. It must therefore contain the programs necessary for these activities. In this case, the image looks like this:

FROM php:7.4-fpm-buster

RUN apt-get -y update && apt-get -y --no-install-recommends install \
    git \
    zip \
    unzip \
    wget \
    openssl \
    curl \
    openssh-client \
    rsync

COPY --from=composer /usr/bin/composer /usr/bin/composer

You need to build such image before and put it in the Gitlab container registry. This can be done by following the three steps below:

docker login registry.gitlab.com
docker build -t registry.gitlab.com/xyz/app/php7.4 .
docker push registry.gitlab.com/xyz/app/php7.4

Building Application Within Gitlab CI/CD

There are cases when we might not want to build our application on the end server and do it on the Gitlab runner instead. This may be:

• we have multiple servers - then building application once is faster than doing it on each server
• we want to be sure that on each server the application will look exactly the same
• we cannot build assets because of lack of necessary tools like npm or yarn on the target server
• we want to keep the credentials in one place

The Deployer and Gitlab allows us to build application in such way. We only have to add small changes to the previous code.

Adjusting Deployer

To allow the Deployer to build the application in the Gitlab CI/CD, we need to change a bit our deploy.php file.

First of all, we have to add build task. It will tell Deployer to prepare the code in .build directory - it will download the repository and install dependencies.
It will also archive all files because sending one file is usually faster than sending thousands smaller. All those things will be done on Gitlab runner.

Secondly, in the upload task we have to tell Deployer where should it upload the prepared release. It will also unarchive the code on the target server.

task('build', function () {
    set('deploy_path', __DIR__ . '/.build');
    invoke('deploy:prepare');
    invoke('deploy:release');
    invoke('deploy:update_code');
    invoke('deploy:vendors');

    cd('{{deploy_path}}/releases/1/');
    run('rm -rfd deploy.php tests docker');
    cd('{{deploy_path}}');
    run("tar -cvf release.tar.gz -C {{deploy_path}}/releases/1/ $(find {{deploy_path}}/releases/1/ -maxdepth 1 -printf '%P ')");
})->local();

task('upload', function () {
    upload(__DIR__ . "/.build/release.tar.gz", '{{release_path}}');
    cd('{{release_path}}');
    run('tar -xf release.tar.gz');
    run('rm release.tar.gz');
});

task('release', [
    'deploy:info',
    'deploy:prepare',
    'deploy:release',
    'upload',
    'deploy:shared',
    'deploy:writable',
    'deploy:cache:clear',
    'deploy:cache:warmup',
    'deploy:symlink',
]);

task('deploy', [
    'build',
    'release',
    'cleanup',
    'success'
]);

Adjusting Gitlab Configuration File

In the .gitlab.yml file, the changes will be rather cosmetic. The only thing we have to add there is a private key to our repository, so as the runner has the access to it.
The deploy_prod part will now look like this:

deploy_prod:
  stage: deploy
  image: ${DOCKER_IMAGE_PHP}
  before_script:  
    - mkdir -p ~/.ssh
    - chmod 700 ~/.ssh
    # Private key to access repository
    - echo "$GITLAB_PRIVATE_KEY" > ~/.ssh/id_rsa
    # Private key to access hosting
    - echo "$HOSTING_PRIVATE_KEY" > ~/.ssh/id_hosting
    - ssh-keyscan gitlab.com >> ~/.ssh/known_hosts
    - chmod 600 ~/.ssh/*
  script:
    - php vendor/bin/dep deploy prod --revision="${CI_COMMIT_SHA}" 
  only:
    - master
  dependencies:
    - build
  when: on_success

Deploying Changes

Now, after merging the changes to the master branch, the runner will be launched and the application will be automatically deployed to our hosting. The entire process can be viewed in Gitlab in the CI/CD -> Pipelines tab. It looks like this:

On our hosting, in the directory /home/xyz/domains/xyz.pl/myapp we can see particular releases:

Deployer pushes changes to the releases/[release number] directory. As we set it in the deploy.php file, there will be three last versions here.
The current directory is a symbolic link to the last release.
In the shared directory there are files which are shared between individual releases.

Summary

Deployer is a simple tool which enables fast automatic deployment of applications. It is easy to configure and contains built-in settings we can use. In connection with Gitlab, it allows to automate the entire process of uploading the application, which saves a lot of time.

Comments

[Joda Stößer]

This approach is based on Deployer version 6, correct?
With version 7 some of the functions don't exist like that anymore. It is suggested, however not documented, to use a localhost host. Either with localhost('ci'), localhost() or the local: true in YAML.