The complete guide to setup a CI/CD for Rails 6+ on Gitlab

Continuous Integration/Deployment for Rails on Gitlab

In this blog post, we will through the necessary steps to setup Gitlab in order
to run Rails build, tests & deployment if everything will be okay.
I will put a particular attention on rails system test and how to make them works.

We will use Heroku to deploy our staging App.

What will we achieve ?

The build Stage

The build will contain:

• Installation of dependencies
• Database setup
• Precompile of assets (assets & webpacker)

The Test Stage

The Integration tests

In this stage, we will run all our integration tests, which basically turn to
run:
bundle exec rails test

The system tests

This is the most exciting and important part to have in our CI.
The system tests are very useful in term of testing complex UI requiring a massive use
of Javascript (React of Vue app) and interacting with external services like Google
Map Places.

The system test will mimic a regular user by clicking and filling inputs like a regular user on our App.
The main command executed in this stage is bundle exec rails test:system.

The interacting fact in this case is the use of container to embed the Selenium
Chrome browser to run real browser to fetch and tests our frontend.

The deploy Stage

This is an easy step, we will deploy our application to the staging environment.

---

The GITLAB-CI

Gitlab offer to everyone ( and we should be grateful to them for all the work
that have be done) a recipe that define how the code will be tested / deployed
and all the services needed for these tasks.
All the instructions are stored in .gitlab-ci present in the root of our repo.

This offer us a centralized and an easy way to manage our source code and
our continues integration for FREE.

How does it works

The CI follow these simple steps:

1. Booting one or several containers aka services that you have specified in the .gitlab-ci
2. Copy your repo in the main container.
3. Run all scripts wanted in it

Use the cache to speedup the CI

Gitlab allows us to cache folders and files and use them for the next jobs.
No need to recompile all dependencies or even download them.
In our case, caching all the gems and node_modules will save us several minutes.

Use artifacts to debug our tests

When the system tests fails, the test will save the screenshots in a temp folder.

The artifacts make possible for us to save those files and tie them to the job.

This will help us a lot when we want to debug a failing system tests.

---

Let's do it

1. The build

Prepare the build container

The build will be executed in a container, so we should have a container with
all the dependencies needed bundled inside.

For the modern rails app we should include:

• Ruby
• Node + Yarn
• Some system libraries

Here is the dockerfile

FROM ruby:2.4.3

RUN curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add -
RUN curl -sL https://deb.nodesource.com/setup_8.x | bash -
RUN echo "deb https://dl.yarnpkg.com/debian/ stable main" | tee /etc/apt/sources.list.d/yarn.list
RUN apt-get update -qqy && apt-get install  -qqyy yarn nodejs postgresql postgresql-contrib libpq-dev cmake

RUN rm -rf /var/lib/apt/lists/*

Easy yay !

Build the container

docker build .
Sending build context to Docker daemon  2.048kB
Step 1/6 : FROM ruby:2.6.5
2.6.5: Pulling from library/ruby
16ea0e8c8879: Pull complete
50024b0106d5: Pull complete
ff95660c6937: Pull complete
9c7d0e5c0bc2: Pull complete
29c4fb388fdf: Pull complete
069ad1aadbe0: Pull complete
e7188792d9dd: Pull complete
bae7e74440d1: Pull complete
Digest: sha256:2285f291f222e1b53d22449cc52bad2112f519bcce60248ea1c4d5e8f14c7c04
Status: Downloaded newer image for ruby:2.6.5
 ---> 2ff4e698f315
Step 2/6 : RUN curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add -
 ---> Running in abb67e50af3e
Warning: apt-key output should not be parsed (stdout is not a terminal)
OK
Removing intermediate container abb67e50af3e
 ---> 461e2dd2134d
Step 3/6 : RUN curl -sL https://deb.nodesource.com/setup_8.x | bash -
 ---> Running in 414f508a391c

## Installing the NodeSource Node.js 8.x LTS Carbon repo...

.....
Processing triggers for libc-bin (2.28-10) ...
Removing intermediate container af1183021a8d
 ---> 603cab5f6952
Step 6/6 : RUN rm -rf /var/lib/apt/lists/*
 ---> Running in 53c5950a25c1
Removing intermediate container 53c5950a25c1
 ---> 42b50699301e
Successfully built 42b50699301e

Tag it

docker tag 42b50699301e registry.gitlab.com/[ORG]/[REPO]/[CONTAINER]:v1

Now, we should publish this container to enable GitlabCI to use it.

Gitlab provide for us a container registry ! for free again !

So we just need to push this container in the project registry.

First, you should login to gitlab registry

docker login registry.gitlab.com
# use your gitlab credential

and PUSH

docker push registry.gitlab.com/[ORG]/[REPO]/[CONTAINER]:v1 # v1 is my version tag

If you have ADSL internet connection with a poor uploading speed, you can go
take a nap ;)

Once the push finishes, we are good to go to the next step.

The build script

This is the main build part in the gitlab-ci file

image: "registry.gitlab.com/[ORG]/[REPO]/[CONTAINER]:v1"

variables:
  LC_ALL: C.UTF-8
  LANG: en_US.UTF-8
  LANGUAGE: en_US.UTF-8
  RAILS_ENV: "test"
  POSTGRES_DB: test_db
  POSTGRES_USER: runner
  POSTGRES_PASSWORD: ""

# cache gems and node_modules for next usage
.default-cache: &default-cache
  cache:
    untracked: true
    key: my-project-key-5.2
    paths:
      - node_modules/
      - vendor/
      - public/

build:
  <<: *default-cache
  services:
    - postgres:latest
  stage: build
  script:
  - ruby -v
  - node -v
  - yarn --version
  - which ruby
  - gem install bundler  --no-ri --no-rdoc
  - bundle install  --jobs $(nproc) "${FLAGS[@]}" --path=vendor
  - yarn install
  - cp config/database.gitlab config/database.yml
  - RAILS_ENV=test bundle exec rake db:create db:schema:load
  - RAILS_ENV=test bundle exec rails assets:precompile

So we are using the previously created image to host the build.

We should add to the project a config/database.gitlab to replace the original
database config and use custom host and credential to connect to postgres
container booted by the GitlabCI.

  services:
    - postgres:latest

Gitlab when reading this line will bootup a database container (postgress) and
will use the variables defined before to setup the database

  POSTGRES_DB: test_db
  POSTGRES_USER: runner
  POSTGRES_PASSWORD: ""

The config/database.gitlab will tell our rails app how to connect to the
database, so before the app boots, the database.yml will be replaced by the
custom one.

test:
  adapter: postgresql
  encoding: unicode
  pool: 5
  timeout: 5000
  host: postgres
  username: runner
  password: ""
  database: test_db

2. The Integration Tests script

No need more explanation for this

integration_test:
  <<: *default-cache
  stage: test
  services:
    - postgres:latest
    - redis:alpine
  script:
    - gem install bundler  --no-ri --no-rdoc
    - bundle install  --jobs $(nproc) "${FLAGS[@]}" --path=vendor
    - cp config/database.gitlab config/database.yml
    - bundle install --jobs $(nproc) "${FLAGS[@]}" --path=vendor
    - RAILS_ENV=test bundle exec rake db:create db:schema:load
    - RAILS_ENV=test bundle exec rails assets:precompile
    - bundle exec rake test

3. The System Tests script

The infrastructure to make possible the system test is quite interesting.

To run the test we should start a browser (in a container) and fetch the page
from the rails server (from an other container).

system_test:
  <<: *default-cache
  stage: test
  services:
    - postgres:latest
    - redis:alpine
    - selenium/standalone-chrome:latest
  script:
    - gem install bundler  --no-ri --no-rdoc
    - bundle install  --jobs $(nproc) "${FLAGS[@]}" --path=vendor
    - cp config/database.gitlab config/database.yml
    - export selenium_remote_url="http://selenium__standalone-chrome:4444/wd/hub/"
    - bundle install  --jobs $(nproc) "${FLAGS[@]}" --path=vendor
    - RAILS_ENV=test bundle exec rake db:create db:schema:load
    - RAILS_ENV=test bundle exec rails assets:precompile
    - bundle exec rake test:system
  artifacts:
    when: on_failure
    paths:
      - tmp/screenshots/

We should tell to capybara to use the right IP instead of localhost, because here we have the browser
and the server in two different containers.

In the environment/test.rb, add these lines

  net = Socket.ip_address_list.detect{|addr| addr.ipv4_private? }
  ip = net.nil? ? 'localhost' : net.ip_address
  config.domain = ip
  config.action_mailer.default_url_options = { :host => config.domain }

  Capybara.server_port = 8200
  Capybara.server_host = ip

and we should tell the system test where to find the chrome driver to control the browser, update application_system_test_case.rb

require "test_helper"
require "socket"


def prepare_options
  driver_options = {
    desired_capabilities: {
      chromeOptions: {
        args: %w[headless disable-gpu disable-dev-shm-usage] # preserve memory & cpu consumption
      }
    }
  }

  driver_options[:url] = ENV['selenium_remote_url'] if ENV['selenium_remote_url']

  driver_options
end

class ApplicationSystemTestCase < ActionDispatch::SystemTestCase
  driven_by :selenium, using: :chrome, screen_size: [1400, 1400],
            options: prepare_options
end

The rails system test will take screenshots and save them to tmp/screenshots

{:class="img-responsive"}

As you can see, the screenshots are stored and attached to job, Neat!

4. The Staging deployment

This will deploy our code if build and tests stages succeed.

deploy_staging:
  stage: deploy
  variables:
    HEROKU_APP_NAME: YOUR_HEROKU_APP_NAME
  dependencies:
    - integration_test
    - system_test
  only:
    - master
  script:
    - gem install dpl
    - dpl --provider=heroku --app=$HEROKU_APP_NAME --api-key=$HEROKU_API_KEY

The HEROKU_API_KEY is stored in a safe place in the settings of the project

For more information about this, go to Gitlab variables documentation

---

Conclusion

Gitlab is an amazing project and provide a very nice spot where everything is
well integrated, the coding experience is enhanced.

Finally, let's hope that the migration to Google compute engine will provide a
better robustness to the project and less issues.

Longue vie à Gitlab !!

Cheers!

Here is the complete Gitlab CI file

Comments

[Dr Nic Williams]

Since my Gemfile already had gem "webdrivers" for local running of rails test:system, I needed to change this to disable webdrivers so as to stop the "Failed to find Chrome binary" error:

gem "webdrivers", require: !ENV['selenium_remote_url']

[CHADDA Chakib]

Hi Patrik,

When I print /etc/hosts inside a service and the main container, I get:

# Inside a service
2018-12-26T14:48:23.565202452Z 127.0.0.1    localhost
2018-12-26T14:48:23.565246527Z ::1  localhost ip6-localhost ip6-loopback
2018-12-26T14:48:23.565251755Z fe00::0  ip6-localnet
2018-12-26T14:48:23.565256138Z ff00::0  ip6-mcastprefix
2018-12-26T14:48:23.565260449Z ff02::1  ip6-allnodes
2018-12-26T14:48:23.565264704Z ff02::2  ip6-allrouters
2018-12-26T14:48:23.565268770Z 172.17.0.6   cc27ba611b73

# inside the main container
127.0.0.1   localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
172.17.0.5  postgres a35defa95dba runner-a252ce20-project-5796108-concurrent-0-postgres-0
172.17.0.6  runner-a252ce20-project-5796108-concurrent-0

So definitely, the ip of the main container is 172.17.0.6 and it's referenced inside the service but with an other name cc27ba611b73 and it's the container ID. so I cannot find an easy way to fix your issue.

May be you can create a issue in the repo, gitlab is opensource ;)

The chrome is CPU intensive, using it side by side with rails server can make your system test flaky.

Have a good day

[JC]

Hi, I followed this guide, but I am having problems with Chrome. When CI runs my tests, I get the following error:

Webdrivers::BrowserNotFound: Failed to find Chrome binary.

Any ideas why Chrome can't be found?

[CHADDA Chakib]

Hey,

I know that I have an issue with the latest selenium containers
Can you try with this one:

    - selenium/standalone-chrome:3.14.0-helium

[JC]

Hi,

I tried with that container version, but I'm still getting the same problem.

[CHADDA Chakib]

Are you sure that your system test is trying to connect with the remote chrome hosted inside the selenium container.

I think your issue is that the tests is trying to run in the local container and gets this error because there is no chrome installed in the build container.

Make sure that is used

 driver_options[:url] = ENV['selenium_remote_url'] if ENV['selenium_remote_url']

[JC]

My application_system_test_case.rb looks like this:

require "test_helper"
require "socket"

def prepare_options
  driver_options = {
    desired_capabilities: {
      chromeOptions: {
        args: %w[headless disable-gpu disable-dev-shm-usage] # preserve memory & cpu consumption
      }
    }
  }

  driver_options[:url] = ENV["selenium_remote_url"] if ENV["selenium_remote_url"]

  driver_options
end

class ApplicationSystemTestCase < ActionDispatch::SystemTestCase
  driven_by :selenium, using: :chrome, screen_size: [1400, 1400], options: prepare_options
end

And within the configure block ofconfig/environments/test.rb I have:

  net = Socket.ip_address_list.detect{|addr| addr.ipv4_private? }
  ip = net.nil? ? 'localhost' : net.ip_address
  config.domain = ip
  config.action_mailer.default_url_options = { :host => config.domain }

  Capybara.server_port = 8200
  Capybara.server_host = ip

[CHADDA Chakib]

Hello Patrik, thankyou for your comment.

To solve your issue, I will proceed like this:

1. Find the hostname of the main container that run your rails app in gitlab CI, in the rest of this comment, I will refer to it as [HOSTNAME]

2. setup a test domain with sub-domains using CNAME ( you can use route53 from aws for this) :

  subdomain1.example-ci.com CNAME [HOSTNAME]
  subdomain2.example-ci.com CNAME [HOSTNAME]
  ...

1. in your rails app, setup the links generator to use subdomains

The result will be:

1. Rails will generate domains using *.example-ci.com
2. Selenium will resolve this domain and get [HOSTNAME] as the first result it will try to resolve [HOSTNAME] a second time and will find the IP because all services and the main container are linked together gitlab service alias
3. selenium will send the http requests to the right container containing your rails app

Hope this help you

Merry Christmas

[Felipe Moyano]

Hi, thanks for your post! It was very useful. However I'm struggling a bit to get the system tests running.

I don't know if I'm missing something, but I'm getting a lot of errors when initializing Chromedriver.

Did you include it in your docker image? Is there any script you're executing on the build phase to install chromedriver and it's dependencies?

Thanks again for a great post

Felipe

[CHADDA Chakib]

Hey Felipe,

I think the error that you are getting is when the system test is trying to connect to the chrome driver in the separated container and failed.

I think this can help you by updating this file application_system_test_case.rb

require "test_helper"
require "socket"


def prepare_options
  driver_options = {
    desired_capabilities: {
      chromeOptions: {
        args: %w[headless disable-gpu disable-dev-shm-usage]
      }
    }
  }

  driver_options[:url] = ENV['selenium_remote_url'] if ENV['selenium_remote_url']

  driver_options
end

class ApplicationSystemTestCase < ActionDispatch::SystemTestCase
  driven_by :selenium, using: :chrome, screen_size: [1400, 1400],
            options: prepare_options
end

Take attention to the code that fetch the selenium_remote_url from the env variable

[Felipe Moyano]

Thanks! That did the trick.

[CHADDA Chakib]

Thank you for your comment, I updated the blog post.