DEV Community: CHADDA Chakib

Prevent ActiveStorage to kill your Rails app

CHADDA Chakib — Fri, 05 Feb 2021 20:50:00 +0000

ActiveStoage is a superb piece of code that makes uploads and file manipulation, resizing, and previews like a breeze on your ruby on rails application.

When you're, like me, hosting your app on Heroku, using s3 or other external storage to persist your file uploads, ActiveStorage makes it as easy as adding some lines on a config file.

Using ActiveStorage was for me a benediction, small efforts to maintain and it works reliably most of the time.

One of the main features of the ActiveStorage is the ability to resize pictures and create variants.
It uses ImagicMagic ou LibVibs libraries on the background to make this happen.

To create a variant we need to write:

# resize the picture to fit 200px by 200px by respecting the ratio. 
<%= image_tag picture.variant(resize: "200x200", auto_orient: true) %>

When the browser fetches the image for the first time, ActiveStorage will check if it has the asked variant of this picture ready, if not it will call in sync the ImagicMagic process to do the resizing and send it back to the browser.

If the picture is stored to S3 from aws for example, the process will follow:

Download the image from S3 into the server
Call the process to resize the picture
Upload the variant to S3 and update the database
Send an HTTP redirect to the new location of the image on S3

This a lot of work and it takes time (some seconds on small dynos).

Meanwhile, the process/thread is too busy and cannot work on other requests.

Imagine, one page contains 5 or 6 pictures, this will make your server busy for resizing all the pictures, may cause a significant slow down for the following requests.

You can tell me, it's not an issue, it's done only once, but when you have an app with uploads occurring all the day, running on a small dyno, this will be a serious issue.

If you care about the user experience and the confidence on your product, you should fix it.

You can throw more money on you Heroku, faster CPU the mitigate the issue, bare with me, I have a solution for you and will cost you only some lines of codes 😇

1. Picture resizing done on a background Job

To make your web server happy, overload all the heavy works with your background workers.

When an upload is done, start a background job to process the variants

class ResizePhotoJob < ApplicationJob
  queue_as :default

  def perform(file, resize_cmd:nil)
    if resize_cmd.nil?
      file.variant(auto_orient: true).processed
    else
      # the `.processed` will force the resizing to be done in sync
      file.variant(resize: resize_cmd, auto_orient: true).processed 
    end
  end
end

# and call it for your different needed variants
ResizePhotoJob.perform_later(file)
ResizePhotoJob.perform_later(file, resize_cmd: "250x250")

Of course, you need to write some tests for this! I will help you

require 'test_helper'

class ProcessPhotosFromAttachmentJobTest < ActiveJob::TestCase
  test 'resize photos' do

    profile = profiles(:jamal)
    profile.avatar.attach(io: File.open(file_fixture("photo_1.jpeg")), filename: 'image')

    file = profile.avatar

    key = file.variant(resize: '200x200', auto_orient: true).key
    refute file.service.exist?(key)

    ResizePhotoJob.perform_now(profile.avatar, resize_cmd: "200x200")

    key = file.variant(resize: '200x200', auto_orient: true).key
    assert file.service.exist?(key)
  end
end

Each variant on the ActiveStorage has a key ( a simple hash on the ImageMagic params )
The variant is stored and retrieved using this key.

# ==> retrieve the variant key 
file.variant(resize: '200x200', auto_orient: true).key

This test will see if the variant exists before the job executions and its existence after.

That's good, now we process the variants on the background, and hopefully when the user tries to access the picture, he will find the variant processed and ready to be served.

Unfortunately, it's not always the case 😔

Even worse, some pictures will fail to resize, for example on a small Heroku dyno, there is not enough memory allowed to ImageMagic to process some big pictures > 15Mb.

I got this error when resizing an image of 15mb (12000x9000) on a Hobby Dyno on Heroku

convert /tmp/ActiveStorage-36637-20210128-4-18i0017.jpg[0] -auto-orient -resize 250x250 -auto-orient
'/tmp/image_processing20210128-4-gnzbsa.jpg' failed with error:
convert-im6.q16: DistributedPixelCache '127.0.0.1' 
@error/distribute-cache.c/ConnectPixelCacheServer/244.
convert-im6.q16: cache resources exhausted
'/tmp/ActiveStorage-36637-20210128-4-18i0017.jpg' 
@error/cache.c/OpenPixelCache/3984.

Hopefully, in my case, this is not the regular size of pictures uploaded on my app.

For those cases, our job is useless, each request will start all the optimization task and it will take too much time.

These requests will end up with a 503 timeout and make worse the process/thread availability to process our regular requests.

2. Prevent resizing on the fly

My solution is dead simple:
When the variant is not available, do not resize and redirect to the original picture.

Okay, the user will download a big picture, that's good enough, the user will get his picture, okay it will take much longer to download but will use the S3 server resources, not mine 😆.

My app will only redirect the browser to the S3 servers when the actual download will occur.

We will use the same code used on the tests to check if the variant is present.

      <% variant = picture.variant(resize: "200x200" , auto_orient: true) %>
      <% if picture.service.exist?(variant.key) %>
          <%= image_tag variant %>   
      <% else %>
          <%= image_tag picture %>
      <% end %>

This small line of codes will make stay a little longer on small cheap Heroku dynos, make your app more resilient, and make you save money.

Glad to hear if you have done this kind of Budget optimization in another way.

Vim tips: Speed up with Vim tabs

CHADDA Chakib — Wed, 20 Jan 2021 16:18:46 +0000

I struggled a long time ago with vim when playing with multiple projects at the same time or playing with a code base with conflicting folders related to multiple languages used at the same time.

As an example, a typical rails code base will have a lot of ruby code and a bunch of javascript/typescript code ( stimulus, react ... ).

As an example, we can have a ruby service for invoicing for example
.../services/fetch_invoice_service.rb

and a typescript service used to fetch something from the API
.../javascript/src/services/fetchDataService.ts

On vim, using fzf I found my self using the Gfilescommande to find the desired file typing for example:
servicefetch

The Fzf will show ruby and javascript files on the same list.
Sometimes is really annoying when you're working on a big codebase.

Also, when you are working on multiple project on the same time with vim, usually you find your self having multiple vim instances.

We can fix this with Vim tabs

Vim tab to rescue

You can create a new tab on vim using the command: :tabnew

Navigation between tabs is done by:

gt cycling between tabs
{tab number}gt: switch to a specific tab.

Setting a working directory by tab:

To make your search for files really fast, you can set a working directory by tab.
:tcd [path to your folder]

Calling :Gfiles will use this path as a root directory

Rails workflow

I found my self for a rails app using three tabs:

The root tab on my project
The javascript tab within app/javascript
The terminals tabs

HEY Turbolinks! RJS is dead ...

CHADDA Chakib — Sat, 20 Jun 2020 14:30:42 +0000

I am super excited about the release of the new product crafted by the Basecamp team, the HEY email service.

With all the @dhh teasing about the upgraded frontend stack, I was pumped to know how the new internals work, especially how the HEY app handles real-time updates and all the HTML mutations.

I want to share with you my thoughts about this subject and to be clear, it's only how I have understood what I have observed on the browser, nothing official.

The starting point

I tweeted yesterday about what I am seeing from the Ajax requests on the hey.com app

chadda chakib

@zimski1990

Interesting html mutation pattern seen on hey.com, the same pattern is used for updates over the socket
@dhh we need to know more about that !

09:25 AM - 19 Jun 2020

And this was retweeted by a basecamp guy, well known for his works on rails, Trix ... etc

Javan Makhmali

@javan

twitter.com/zimski1990/sta…

01:42 AM - 20 Jun 2020

chadda chakib @zimski1990
Interesting html mutation pattern seen on https://t.co/tX5zyCrhIH, the same pattern is used for updates over the socket @dhh we need to know more about that ! https://t.co/Y0Uheoo5fG

So I tell my self, maybe I should dig more to understand how this works.

Everything starts with an HTML fragment

<template data-page-update="remove#posting_140----"</template>

<template data-page-update="prepend#postings">
    <article
        id="posting_-----"
        class="posting"
        data-list-target="item"
        data-bulk-actions-target="row"
        data-identifier="------"
        data-sort-code="-------"
        data-box-freshness-target="row"
        data-account-id="----"
        data-account-purpose="home"
        data-box-kind="imbox"
        data-topic="true"
    >
        <div class="posting__body">
        ...
        </div>
    </article>
</template>

When I dig on the obfuscated JavaScript available on the app, I understand this mechanism:

1. Receiving data

The payload received from the socket or an Ajax call.
The payload is a pure HTML.

2. Processing the payload

This payload will be ingested and transformed into a valid HTML fragment document.

... document.createRange().createContextualFragment(playload)

When we have a valid html document, it's easier to process each tag and extract the needed informations:

... querySelectorAll("template[data-page-update]"

3. Commands to mutate the HTML

Each tag will contain the command to be executed.
This command is present on the data-page-update attribute.

Commands I have seen on the code:

append
prepend
replace
update
remove

The command follow this syntax: [command]#[html_element_id]

The <template..></template> tag can be empty, for commands like remove or containing an HTML inside to be inserted somewhere.

So if I follow all of this, we can see a cleaner version of the RJS used today on the rails app.

No need to write and render a custom JS code to mutate the dom.
With the current version, we were doing something like

$('#posting_1409180').remove();
$('#postings').prepend(<%= render... %>)

Now, we have a clean HTML that contains the command to mutate the DOM.

This follows the Stimulus principals to add some attributes to the HTML to get sparkles ✨.

The Content Template element

The documentation says that the <template> is not rendered and we need some js to render it.

We can imagine here to mix a standard HTML with this mutation template and render them on the page.
The js processor will pick and execute the mutation.

We can also stack all the mutations needed

<tempate mutation 1   />
<tempate mutation 2   >..</tempate>
<tempate mutation 3   >..</template>

So we will get a 100% HTML over the wire, no Javascript, no JSON, and a clever client-side HTML renderer to execute these mutations.

I think The main candidate to embrace this new mission is our beloved Turbolinks.

Turbolinks and sockets for a Realtime without hassle

Using this pattern, I suppose on the next version of turbo links we can tell it to subscribe to some cable channel and will render all of this HTML mutations

Rails helper on the backend ??

I hope also this will come with some helper to simply edit these template mutations.

Stripe Strong Customer Authentication & Rails

CHADDA Chakib — Sat, 19 Oct 2019 11:40:19 +0000

You maybe heard about the migration of the Stripe CB payment to SCA Strong Customer Authentication
All payment from the European countries are concerned.

For more information, The stripe sca doc

The customer experience through the SCA

For your customer, after he puts the Credit card number, the bank can ask for a confirmation by SMS or a push notification in bank app in his smartphone.

Stripe now handle this and for better experience, they need to redirect the user to hosted page by stripe and will handle all the complexity.
When succeed or failed, the user is redirected to your webapp.

This is the flow that explain how this works

The main steps handled by the rails app are two

The preparation of the checkout session [generation of the key]
The validation of the payment [Verify the key]

1. The session preparation

This will prepare Stripe session for the checkout of the product and the redirection when the payment succeed or not.

So we will prepare

The name of the product
The price
The redirection when succeed or not

    secret = gen_secret(@product)
    Stripe::Checkout::Session.create(
      {
        payment_method_types: ['card'],
        line_items: [{
                       name: @product.name,
                       description: @product.description,
                       amount: @product.price,
                       currency: 'eur',
                       quantity: 1,
                     }],
        success_url: confirm_product_payment_url(@product, secret: secret),
        cancel_url: cancel_product_payment_url(@product)
      })

The interesting part here is the gen_secret

We need to generate this secret to be able to be sure and verify that the payment at Stripe succeed and the callback is not faked

We can generate this key like that in rails.
Thanks to rails to provide everything inside the framework and no need to add foreign dependency.

def gen_secret(product)
  key = Rails.application.secrets.secret_key_base
  @crypt = ActiveSupport::MessageEncryptor.new(key[0,32], cipher: 'aes-256-gcm')

  @crypt.encrypt_and_sign(product.token)
end

As I explained in the previous UML flow, the callbacks containing this secret will be stored at stripe using a secure connection from your server.

The Payment process

The payment process is classic, the customer will enter his credit card numbers and may use a strong authentication if needed (SMS code verification, push notification ... ect)

When the payment succeed

At this step, the stripe servers will send to the browser the redirection to the right callback url.
This callback is containing our secret generated before.
Time to validation !

2. The Validation process

Now, our rails app should verify this callback, in ruby it's easy
For this example I will use a simple verification.

I will only verify if decrypted token is the real token of the product.

  # in the product controller
  def confirm
    @product = current_product

    if PaymentService.new(product: @product)
                     .confirm(secret: params[:secret])
      flash[:success] = "Payment succeed"
    else
      flash[:error] = "Oups Error !"
    end

    redirect_to product_url(@product)
  end

  # inside the PaymentService
    ...
    def confirm(secret)
    if @crypt.decrypt_and_verify(secret) == @product.token
      # handle the success of the payment
      # notifications, emails ...ect

      return true
    else
      return false
    end
  end

The Front-end

Thanks to Stripe the front-end is easy !
So, when the customer want to pay a product, he will click on the payment button.

This will call our rails app to prepare the session, get back the redirection and follow it.

    <%= button_to product_pay_path(@product),
     method: :post,
     id: "pay_sca",
     data: { disable_with: '<i class="fa fa-spinner fa-spin"></i>'.html_safe },
     class: "btn btn-success btn-lg btn-block",
     remote: true do  %>
      <i style="float:left;color:white" id="notes-exists" class="fas fa-lock"></i>
      Payer
    <% end %>

man that's concise !, this will call our controller by ajax, show a cool animation when waiting for the response of the server and finaly follow the redirection to Stripe.

  # controller
  def create
    @company_travel = current_travel.company
    @session_id = PaymentService.new(product: current_product)
                                .create_session.id
  end

As you have maybe notice, we are using a CRUD resource, when a customer need to pay we create a charging resource.

The view part is simple also

# views/.../create.js.erb

const stripe = Stripe('<%= Rails.application.secrets.stripe_public_key %>');


stripe.redirectToCheckout({
  // We put here our session ID
  sessionId: '<%= @session_id %>'
}).then((result) => {
  // If `redirectToCheckout` fails due to a browser or network
  // error, display the localized error message to your customer
  // using `result.error.message`.
});

Easy yay !

I think now we are good to go.

Testing this will be in an other blog post with system tests.

As a gift, The complete service can look like this

class PaymentService
  include Rails.application.routes.url_helpers

  def initialize(product:)
    @product = product

    key = Rails.application.secrets.secret_key_base
    @cryptor = ActiveSupport::MessageEncryptor.new(key[0,32], cipher: 'aes-256-gcm')
  end

  def create_session
    secret = gen_secret

    Stripe::Checkout::Session.create(
      {
        payment_method_types: ['card'],
        line_items: [{
                       name: @product.name
                       description: @product.description,
                       amount: @product.price,
                       currency: 'eur',
                       quantity: 1,
                     }],
        success_url: confirm_product_payment_url(@product, secret: secret),
        cancel_url: cancel_product_payment_url(@product )
      })
  end

  def confirm(secret)
    if @cryptor.decrypt_and_verify(secret) == @product.token
      # handle a succeed payment
      # send notifications, invoices ... ect
      return true
    else
      return false
    end
  end

  private

  def gen_secret
    @cryptor.encrypt_and_sign(@product.token)
  end
end

A little of ELISP to improve your RAILS testing

CHADDA Chakib — Sat, 09 Feb 2019 19:25:09 +0000

Why spacemacs

I am a big big fan of vim, but I struggled with it when I was coding with multiple project in the same time.
The big pain was also when I need to see the diffs before committing.

So I moved to spacemacs (vim mode) several years ago and I am loving it.
I still use vim to edit some config files but the main code in on spacemacs.

Testing rails inside spacemacs sucks

Now, I am coding a lot of ruby mainly for rails projects and testing function included in the official ruby layer are not adapted for a modern rails test with minitest.

With rails now, when we call bundle exec rails test the code will try to find the spring process and call the tests inside.
The spring process improve a lot the boot-up time of ruby and the loading of all your rails project by caching it.

In spacemacs, your cursor is within test method and you hit space m t t, this will run only this test but he is calling your test by using ruby ..... [your test file] and this will not leverage all the optimization done in the rake test command shipped in rails.

More dangerous, this will flush your dev database and populate it with your test fixture.

bad bad .... very bad thing.

What I was doing before the help of elips

I have a terminal somewhere open, and when I need to run test, I switch to it and try to type the name of the test and line number of the test.

bundle exec rails test test/system/ ... / ... / ... / my_test_file_test.rb:123

Elisp, please help me !

I have written a small functions in Elisp because I want learn & practice this powerful language.

Here is the code, can be improved but who cares, it's doing the job :D

  (defun run-rails-test-file ()
    (interactive)
    (let ((filename (ruby-test-find-file)))
      (compile (format "bundle exec rails test %s" filename))))

  (defun run-rails-test-at-point ()
    (interactive)
    (let* ((filename (ruby-test-find-file))
           (buffername (get-file-buffer filename)))
      (with-current-buffer buffername
        (let ((line (line-number-at-pos (point))))
          (compile (format "bundle exec rails test %s:%s" filename line))))
      )
    )

  (spacemacs/set-leader-keys "mtt" 'run-rails-test-at-point)
  (spacemacs/set-leader-keys "mtT" 'run-rails-test-file)

Let's explain this code

(interactive): make your function callable with space space

ruby-test-find-file: this function will output the path of the current buffer if it's a test file and .... wait for it .... it will also return the last visited test file if you are in a none-test file.

This is very useful, imagine your current buffer is a test file, you run the test, it fails, so you switch to the actual file containing the code, you tweak it and you want to re-run the test, no need to switch again to the test file, this function will handle it for you.

I am reusing the functions defined in the spacemacs ruby layer.

Understand the code by yourself

In Emacs, Everything is a function, and you can read the actual code of all of them.
In spacemacs, the BEST helper is Space h d f and it will autocomplete the name of your function and show you the doc/code.

Let's try to understand what means with-current-buffer

type SPACE h d f and start type the name of the function.

Oh look to this this cutie, hit enter

Everything is documented !

Let's go to the code now, move your cursor on the filename and hit Enter

And we can recursively search and learn more things at each key's hit.

Hope you have learned something new and maybe this will help you with your rails workflow on this incredible piece of software Emacs !

The complete guide to setup a CI/CD for Rails 6+ on Gitlab

CHADDA Chakib — Sun, 19 Aug 2018 14:17:01 +0000

Continuous Integration/Deployment for Rails on Gitlab

In this blog post, we will through the necessary steps to setup Gitlab in order
to run Rails build, tests & deployment if everything will be okay.
I will put a particular attention on rails system test and how to make them works.

We will use Heroku to deploy our staging App.

What will we achieve ?

The build Stage

The build will contain:

Installation of dependencies
Database setup
Precompile of assets (assets & webpacker)

The Test Stage

The Integration tests

In this stage, we will run all our integration tests, which basically turn to
run:
bundle exec rails test

The system tests

This is the most exciting and important part to have in our CI.
The system tests are very useful in term of testing complex UI requiring a massive use
of Javascript (React of Vue app) and interacting with external services like Google Map Places.

The system test will mimic a regular user by clicking and filling inputs like a regular user on our App.
The main command executed in this stage is bundle exec rails test:system.

The interacting fact in this case is the use of container to embed the Selenium Chrome browser to run real browser to fetch and tests our frontend.

The deploy Stage

This is an easy step, we will deploy our application to the staging environment.

The GITLAB-CI

Gitlab offer to everyone ( and we should be grateful to them for all the work
that have be done) a recipe that define how the code will be tested / deployed
and all the services needed for these tasks.
All the instructions are stored in .gitlab-ci present in the root of our repo.

This offer us a centralized and an easy way to manage our source code and
our continues integration for FREE.

How does it works

The CI follow these simple steps:

Booting one or several containers aka services that you have specified in the .gitlab-ci
Copy your repo in the main container.
Run all scripts wanted in it

Use the cache to speedup the CI

Gitlab allows us to cache folders and files and use them for the next jobs.
No need to recompile all dependencies or even download them.
In our case, caching all the gems and node_modules will save us several minutes.

Use artifacts to debug our tests

When the system tests fails, the test will save the screenshots in a temp folder.

The artifacts make possible for us to save those files and tie them to the job.

This will help us a lot when we want to debug a failing system tests.

Let's do it

1. The build

Prepare the build container

The build will be executed in a container, so we should have a container with
all the dependencies needed bundled inside.

For the modern rails app we should include:

Ruby
Node + Yarn
Some system libraries

Here is the dockerfile

FROM ruby:2.4.3

RUN curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add -
RUN curl -sL https://deb.nodesource.com/setup_8.x | bash -
RUN echo "deb https://dl.yarnpkg.com/debian/ stable main" | tee /etc/apt/sources.list.d/yarn.list
RUN apt-get update -qqy && apt-get install  -qqyy yarn nodejs postgresql postgresql-contrib libpq-dev cmake

RUN rm -rf /var/lib/apt/lists/*

Easy yay !

Build the container

docker build .
Sending build context to Docker daemon  2.048kB
Step 1/6 : FROM ruby:2.6.5
2.6.5: Pulling from library/ruby
16ea0e8c8879: Pull complete
50024b0106d5: Pull complete
ff95660c6937: Pull complete
9c7d0e5c0bc2: Pull complete
29c4fb388fdf: Pull complete
069ad1aadbe0: Pull complete
e7188792d9dd: Pull complete
bae7e74440d1: Pull complete
Digest: sha256:2285f291f222e1b53d22449cc52bad2112f519bcce60248ea1c4d5e8f14c7c04
Status: Downloaded newer image for ruby:2.6.5
 ---> 2ff4e698f315
Step 2/6 : RUN curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add -
 ---> Running in abb67e50af3e
Warning: apt-key output should not be parsed (stdout is not a terminal)
OK
Removing intermediate container abb67e50af3e
 ---> 461e2dd2134d
Step 3/6 : RUN curl -sL https://deb.nodesource.com/setup_8.x | bash -
 ---> Running in 414f508a391c

## Installing the NodeSource Node.js 8.x LTS Carbon repo...

.....
Processing triggers for libc-bin (2.28-10) ...
Removing intermediate container af1183021a8d
 ---> 603cab5f6952
Step 6/6 : RUN rm -rf /var/lib/apt/lists/*
 ---> Running in 53c5950a25c1
Removing intermediate container 53c5950a25c1
 ---> 42b50699301e
Successfully built 42b50699301e

Tag it

docker tag 42b50699301e registry.gitlab.com/[ORG]/[REPO]/[CONTAINER]:v1

Now, we should publish this container to enable GitlabCI to use it.

Gitlab provide for us a container registry ! for free again !

So we just need to push this container in the project registry.

First, you should login to gitlab registry

docker login registry.gitlab.com
# use your gitlab credential

and PUSH

docker push registry.gitlab.com/[ORG]/[REPO]/[CONTAINER]:v1 # v1 is my version tag

If you have ADSL internet connection with a poor uploading speed, you can go
take a nap ;)

Once the push finishes, we are good to go to the next step.

The build script

This is the main build part in the gitlab-ci file

image: "registry.gitlab.com/[ORG]/[REPO]/[CONTAINER]:v1"

variables:
  LC_ALL: C.UTF-8
  LANG: en_US.UTF-8
  LANGUAGE: en_US.UTF-8
  RAILS_ENV: "test"
  POSTGRES_DB: test_db
  POSTGRES_USER: runner
  POSTGRES_PASSWORD: ""

# cache gems and node_modules for next usage
.default-cache: &default-cache
  cache:
    untracked: true
    key: my-project-key-5.2
    paths:
      - node_modules/
      - vendor/
      - public/

build:
  <<: *default-cache
  services:
    - postgres:latest
  stage: build
  script:
  - ruby -v
  - node -v
  - yarn --version
  - which ruby
  - gem install bundler  --no-ri --no-rdoc
  - bundle install  --jobs $(nproc) "${FLAGS[@]}" --path=vendor
  - yarn install
  - cp config/database.gitlab config/database.yml
  - RAILS_ENV=test bundle exec rake db:create db:schema:load
  - RAILS_ENV=test bundle exec rails assets:precompile

So we are using the previously created image to host the build.

We should add to the project a config/database.gitlab to replace the original
database config and use custom host and credential to connect to postgres
container booted by the GitlabCI.

  services:
    - postgres:latest

Gitlab when reading this line will bootup a database container (postgress) and
will use the variables defined before to setup the database

  POSTGRES_DB: test_db
  POSTGRES_USER: runner
  POSTGRES_PASSWORD: ""

The config/database.gitlab will tell our rails app how to connect to the
database, so before the app boots, the database.yml will be replaced by the
custom one.

test:
  adapter: postgresql
  encoding: unicode
  pool: 5
  timeout: 5000
  host: postgres
  username: runner
  password: ""
  database: test_db

2. The Integration Tests script

No need more explanation for this

integration_test:
  <<: *default-cache
  stage: test
  services:
    - postgres:latest
    - redis:alpine
  script:
    - gem install bundler  --no-ri --no-rdoc
    - bundle install  --jobs $(nproc) "${FLAGS[@]}" --path=vendor
    - cp config/database.gitlab config/database.yml
    - bundle install --jobs $(nproc) "${FLAGS[@]}" --path=vendor
    - RAILS_ENV=test bundle exec rake db:create db:schema:load
    - RAILS_ENV=test bundle exec rails assets:precompile
    - bundle exec rake test

3. The System Tests script

The infrastructure to make possible the system test is quite interesting.

To run the test we should start a browser (in a container) and fetch the page
from the rails server (from an other container).

system_test:
  <<: *default-cache
  stage: test
  services:
    - postgres:latest
    - redis:alpine
    - selenium/standalone-chrome:latest
  script:
    - gem install bundler  --no-ri --no-rdoc
    - bundle install  --jobs $(nproc) "${FLAGS[@]}" --path=vendor
    - cp config/database.gitlab config/database.yml
    - export selenium_remote_url="http://selenium__standalone-chrome:4444/wd/hub/"
    - bundle install  --jobs $(nproc) "${FLAGS[@]}" --path=vendor
    - RAILS_ENV=test bundle exec rake db:create db:schema:load
    - RAILS_ENV=test bundle exec rails assets:precompile
    - bundle exec rake test:system
  artifacts:
    when: on_failure
    paths:
      - tmp/screenshots/

We should tell to capybara to use the right IP instead of localhost, because here we have the browser
and the server in two different containers.

In the environment/test.rb, add these lines

  net = Socket.ip_address_list.detect{|addr| addr.ipv4_private? }
  ip = net.nil? ? 'localhost' : net.ip_address
  config.domain = ip
  config.action_mailer.default_url_options = { :host => config.domain }

  Capybara.server_port = 8200
  Capybara.server_host = ip

and we should tell the system test where to find the chrome driver to control the browser, update application_system_test_case.rb

require "test_helper"
require "socket"


def prepare_options
  driver_options = {
    desired_capabilities: {
      chromeOptions: {
        args: %w[headless disable-gpu disable-dev-shm-usage] # preserve memory & cpu consumption
      }
    }
  }

  driver_options[:url] = ENV['selenium_remote_url'] if ENV['selenium_remote_url']

  driver_options
end

class ApplicationSystemTestCase < ActionDispatch::SystemTestCase
  driven_by :selenium, using: :chrome, screen_size: [1400, 1400],
            options: prepare_options
end

The rails system test will take screenshots and save them to tmp/screenshots

{:class="img-responsive"}

As you can see, the screenshots are stored and attached to job, Neat!

4. The Staging deployment

This will deploy our code if build and tests stages succeed.

deploy_staging:
  stage: deploy
  variables:
    HEROKU_APP_NAME: YOUR_HEROKU_APP_NAME
  dependencies:
    - integration_test
    - system_test
  only:
    - master
  script:
    - gem install dpl
    - dpl --provider=heroku --app=$HEROKU_APP_NAME --api-key=$HEROKU_API_KEY

The HEROKU_API_KEY is stored in a safe place in the settings of the project

For more information about this, go to Gitlab variables documentation

Conclusion

Gitlab is an amazing project and provide a very nice spot where everything is
well integrated, the coding experience is enhanced.

Finally, let's hope that the migration to Google compute engine will provide a
better robustness to the project and less issues.

Longue vie à Gitlab !!

Cheers!

Here is the complete Gitlab CI file