DEV Community: Prabin Poudel

Setup RSpec Tests in Rails with Gitlab CI

Prabin Poudel — Tue, 02 Apr 2024 04:30:00 +0000

At Truemark, we are constantly looking to improve the code quality in our projects. And one way to do that is through the integration of CI into our workflow. CI can help in automating code reviews for linting and standard practices as well as for running tests to check if code change breaks any existing functionalities.

Today we will look at adding configurations to Gitlab CI for running RSpec tests in our Rails application.

Assumption

You have basic Gitlab CI configurations ready i.e. .gitlab-ci.yml exists in your project.

If it doesn't, you can refer to my other article Integrate Pronto with Gitlab for Rails App.
You are using PostgreSQL in your app though with minimal changes it should also work for any other databases like MySQL and SQLite (let me know in comments if it doesn't and I will help you!)
You are using RSpec as a test library but again with minimal changes it should work for MiniTest as well (let me know if it doesn't and I will help you!)

Tested and working in

Ruby 3.3.0
Rails 7.1.3
rspec-rails 6.1.1
selenium-webdriver 4.18.1

Configure Gitlab CI Variables

Firs of all, we need to add some configurations and files required by the CI to run tests. This should be done over at Gitlab. Let's looks at them one by one.

Add a variable for storing database.yml file content

It's not considered a good practice to use config/database.yml file for the CI so we will have to add a Gitlab CI variable and store the content required to setup PostgreSQL database inside it.

You can visit the official documentation to know a way to setup variables. You have to go to your project's setting in Gitlab and configure these in CI/CD variables.

Create a new variable for this:

Type: File
Flags

Uncheck all checklists here i.e. Protect variable, Mask variable and Expand variable reference
Description

You can write "Database YML" but it's optional and you can skip this as "Key" (just below this) is already clear enough on what this variable is storing.
Key: database_yml

Lastly in "Value" add the following:

test:
  adapter: postgresql
  encoding: unicode
  host: postgres
  database: test_ci_db
  username: postgres
  password: postgres
  pool: <%= ENV.fetch("RAILS_MAX_THREADS") { 5 } %>

For host make sure to use "postgres" instead of "localhost". For other services like MySQL and SQLite you will most probably have to use "mysql" or "sqlite" respectively as said in the official documentation:

The service container for MySQL is accessible under the hostname mysql. To access your database service, connect to the host named mysql instead of a socket or localhost.

Add a variable for storing environment variables

I normally use Figaro for storing environment variables which uses config/application.yml but just the plain .env file is also very popular. Anyway, just copy the content from whatever you are using and paste it inside the Value for this new variable.

Type, Flags and Description will be the same as described above for database.yml

You can add env for Key.

In "Value", add the copied content from your env file.

NOTE: Make sure to only copy what is under "test" block or ".env.test", you won't want to add production variables here as that will lead to security issues.

Require selenium-webdriver

We need to enable selenium-webdriver that is required to run tests in the browser, especially system tests.

Add the following to spec/rails_helper.rb if it's not already there:

require "selenium-webdriver"

Note: Previously "webdrivers" gem was required to automate the installation and update browser specific drivers. But Selenium 4 ships with webdrivers now leading to the webdrivers being deprecated. Quoting the webdrivers Github:

If you can update to the latest version of Selenium (4.11+), please do so and stop requiring this gem.

Configure Capybara with Selenium

We will configure Selenium with Chrome to be used both in CI and Local with Headless mode (by default) while also allowing to run in the browser if needed for debugging.

Create a new file "spec/helpers/capybara.rb" and add the following code:

Capybara.register_driver :selenium_chrome_custom do |app|
  options = Selenium::WebDriver::Chrome::Options.new

  options.add_argument("--headless=new") unless ENV["SELENIUM_HEADFUL"]

  options.add_argument("--window-size=1400,1400")
  options.add_argument("--no-sandbox")
  options.add_argument("--disable-dev-shm-usage")

  remote_url = ENV.fetch("SELENIUM_REMOTE_URL", nil)

  if remote_url
    Capybara::Selenium::Driver.new(
      app,
      browser: :remote,
      url: remote_url,
      options:
    )
  else
    Capybara::Selenium::Driver.new(app, browser: :chrome, options:)
  end
end

RSpec.configure do |config|
  config.before(:each, type: :system, js: true) do
    # Make the test app listen to outside requests, required for the remote Selenium instance
    Capybara.server_host = "0.0.0.0"

    if ENV.fetch("SELENIUM_REMOTE_URL", nil)
      # Use the application container's IP instead of localhost so Capybara knows where to direct Selenium
      Capybara.app_host = "http://#{IPSocket.getaddress(Socket.gethostname)}"
    end

    driven_by :selenium_chrome_custom
  end
end

Explanation

Let's look at what each of the code block above is doing.

Custom Selenium Chrome driver

Capybara.register_driver :selenium_chrome_custom

Since existing Selenium Drivers don't provide the custom options we want, we are creating a new driver selenium_chrome_custom which will handle Remote/Local connection as well as Headless/Headful mode.

Options

--window-size=1400,1400

Set the window size to 1400x1400 pixels. This is a reasonable size without being too large, but you can set it to whatever you like. This mostly impacts the size of debugging screenshots, but some tests may fail if you ask Capybara to click on an element which is not currently visible on the page.
--no-sandbox

Disables Chrome’s sandbox functionality because it has an issue with Docker version 1.10.0 and later.
--disable-dev-shm-usage

The "/dev/shm" shared memory partition is too small on many VM environments which will cause Chrome to fail or crash so we are disabling it.
--headless=new

Enable Chrome’s headless mode which will run Chrome without a UI.

SELENIUM_HEADFUL will control this option. In development, you may want to run Chrome and see what's happening in the browser; you can do so by running tests with SELENIUM_HEADFUL=true bundle exec rspec spec/system.

We will see list of other commands to run system tests at the end of this explanation section in a bit.

Some guides may suggest using the --disable-gpu flag, but this is no longer necessary on any operating system.

This explanation was shamelessly copied from Remote Selenium WebDriver servers with Rails, Capybara, RSpec, and Chrome 🙈.

Selenium remote URL

remote_url = ENV.fetch("SELENIUM_REMOTE_URL", nil)

Remote option is required mostly for CI but you can also test it out in local by running the Selenium Docker image e.g. with SELENIUM_REMOTE_URL=http://localhost:4444/wd/hub bundle exec rspec spec/system

Remote option is controlled by SELENIUM_REMOTE_URL which needs to be passed when running tests as seen above.

Another configuration related to the remote is the use of browser: :remote inside Capybara::Selenium::Driver.new which tells Capybara to run tests in remote Chrome browser instead of local one.

Capybara server and app host

Capybara.server_host = "0.0.0.0"

if ENV.fetch("SELENIUM_REMOTE_URL", nil)
  # Use the application container's IP instead of localhost so Capybara knows where to direct Selenium
  Capybara.app_host = "http://#{IPSocket.getaddress(Socket.gethostname)}"
end

server_host and app_host are required for Capybara to know how it can call driver in the Remote Server.

This piece of code was extracted from the official Rails Documentation.

Commands to run tests

Lastly, let's see various commands we can use to run system tests.

Run in headless mode (default): bundle exec rspec spec/system
Run in headful mode: SELENIUM_HEADFUL=true bundle exec rspec spec/system
Run in headless mode inside external docker image in local: SELENIUM_REMOTE_URL=http://localhost:4444/wd/hub bundle exec rspec spec/system

For CI, default command bundle exec rspec spec/system will work. But SELENIUM_REMOTE_URL will be http://selenium:4444/wd/hub and it will be passed an Environment Variable instead. We will look at how to do that next.

Update `.gitlab-ci.yml` to run all tests

We will be adding code to enable all the following tests and you can choose to pickup or ignore as per your requirement:

Unit and Integration tests (Model, Requests, Authorization, Services etc.) which don't require us to start browser
System Tests where we will start the Chrome browser and run tests inside it

Update your .gitlab-ci.yml with the configurations given below. Most of the configurations are accompanied by explanation, you can find clean configuration without comment at the end of the blog in the section "Final .gitlab-ci.yml"

# change to the ruby version your application uses
image: ruby:3.3.0

# explanation in next section
cache:
  paths:
    - vendor/
    - node_modules/
    - yarn.lock

stages:
  - test

# base configuration required for running tests
.base_db:
  # add-on docker images required for running tests
  services:
    - postgres:latest
  variables:
    # set Rails environment so we don't have to prefix each command with RAILS_ENV=test
    RAILS_ENV: test
    # Postgres runs in a separate docker image and requires authentication to connect. Disabling that here by using "trust" so it doesn't ask for authentication
    POSTGRES_HOST_AUTH_METHOD: trust
  before_script:
    # use same bundler version that was used in bundling the Gemfile
    - gem install bundler -v "$(grep -A 1 "BUNDLED WITH" Gemfile.lock | tail -n 1)" --no-document
    # install all gems to "vendor" folder which helps in caching of gem installation in between the execution of CI jobs
    - bundle config set --local path 'vendor'
    # install "nodejs" required for yarn and "cmake" required for pronto
    - apt-get update -qq && apt-get install -y -qq nodejs cmake
    # install gems in parallel, nproc returns the number of available processors
    - bundle install --jobs $(nproc)
    # install yarn
    - curl -o- -L https://yarnpkg.com/install.sh | bash
    # Make yarn available in the current terminal
    - export PATH="$HOME/.yarn/bin:$HOME/.config/yarn/global/node_modules/.bin:$PATH"
    - yarn install
    # copy all database configurations stored as the Gitlab CI variable to the file "config/database.yml"
    - cat $database_yml > config/database.yml
    # 👋 config/application.yml can be different for you. For e.g. if you are using ".env" then this content will be `cat $envfile > .env`
    - cat $env > config/application.yml
    - bundle exec rails db:create
    - bundle exec rails db:schema:load
    # Required for integration and system tests
    - bundle exec rails assets:precompile

unit_and_integration_tests:
  # reuse all configurations defined in .base_db above
  extends: .base_db
  stage: test
  # run this job only when merge requests are created, updated or merged
  only:
    - merge_requests
  script:
    # run all tests except system tests
    - bundle exec rspec --exclude-pattern "spec/system/**/*.rb"

system_tests:
  extends: .base_db
  stage: test
  services:
    # need to declare postgres again because "services" key will override the one defined in .base_db
    - postgres:latest
    # Docker image for Selenium with Chrome so test can run inside the browser
    - name: selenium/standalone-chrome:122.0
      alias: selenium
  variables:
    RAILS_ENV: test
    POSTGRES_HOST_AUTH_METHOD: trust
    # Location of the selenium docker image. "selenium" is an alias, you can also use http://selenium-standalone-chrome:4444/wd/hub or selenium__standalone-chrome (commonly seen in other guides)
    SELENIUM_REMOTE_URL: http://selenium:4444/wd/hub
  # store necessary files and folders in case of test failure for debugging the error
  artifacts:
    when: on_failure
    paths:
      - log/test.log
    expire_in: 1 week
  only:
    - merge_requests
  script:
    - bundle exec rspec spec/system

Explanation

Let's look at some configurations where explanation was missing and would be lengthy to add there.

cache

cache:
  paths:
    - vendor/
    - node_modules/
    - yarn.lock

This tells Gitlab CI to cache vendor folder where we are storing all our gems, node_modules where all JS packages are stored, yarn.lock which stores the information about installed packages with their versions.

Storing all these folders and files speed up the CI in subsequent runs. bundle install and yarn install will only install new packages that are not already inside the cache.

stages

stages:
  - test

Stages define when to run the jobs. For example, stages that run tests after stages that runs linting on new changes.

If you also have linting and continuous deployment configured then stages could look like this:

stages:
  - lint
    - test
    - staging_deploy
    - production_deploy

Jobs are run in the same order as configured here i.e. linting will run first then test and lastly deployments.

.base_db

This configuration is used by all jobs that require database access. All common configurations for such jobs are extracted here.

services are add-on docker images and provide capabilities like database, redis, selenium drivers, etc.

variables are environment variables used by Rails.

before_script runs before the script so anything that needs to be pre-configured can be added here.

unit_and_integration_tests

extends will extend the configurations defined in the .base_db and use those configurations for this job.

stage tells this job at what stage to run. Depending on stages defined just above this job configuration.

script are the series of command to execute for this job. We are running all tests except system tests by using the rspec command helper --exclude-pattern "spec/system/**/*.rb

system_tests

selenium/standalone-chrome:122.0 configures the docker image for Selenium with Chrome with the version 122.0, normally people use selenium/standalone-chrome:latest instead of this. But at the time of writing this blog, the latest version "123.0" had some issues and Chrome browser was not starting; I had to spend 6+ hours in debugging just for finding that out 🫠

artifacts is used to store necessary files and folders in case of test failure. This helps us in debugging failing tests when needed. We are storing test log files for this purpose.

Final `.gitlab-ci.yml`

If you also have Pronto or any other linter configured in CI then your final file could look like this:

image: ruby:3.3.0

cache:
  paths:
    - vendor/
    - node_modules/
    - yarn.lock

stages:
  - lint
  - test

pronto:
  before_script:
    - gem install bundler -v "$(grep -A 1 "BUNDLED WITH" Gemfile.lock | tail -n 1)" --no-document
    - apt-get update -qq && apt-get install -y -qq cmake
    - bundle config set --local path 'vendor'
    - bundle install --jobs $(nproc)
  stage: lint
  only:
    - merge_requests
  variables:
    PRONTO_GITLAB_API_PRIVATE_TOKEN: $PRONTO_ACCESS_TOKEN
  script:
    - git fetch origin $CI_MERGE_REQUEST_TARGET_BRANCH_NAME
    - bundle exec pronto run -f gitlab_mr -c origin/$CI_MERGE_REQUEST_TARGET_BRANCH_NAME

.base_db:
  services:
    - postgres:latest
  variables:
    RAILS_ENV: test
    POSTGRES_HOST_AUTH_METHOD: trust
  before_script:
    - gem install bundler -v "$(grep -A 1 "BUNDLED WITH" Gemfile.lock | tail -n 1)" --no-document
    - bundle config set --local path 'vendor'
    - apt-get update -qq && apt-get install -y -qq nodejs cmake
    - bundle install --jobs $(nproc)
    - curl -o- -L https://yarnpkg.com/install.sh | bash
    - export PATH="$HOME/.yarn/bin:$HOME/.config/yarn/global/node_modules/.bin:$PATH"
    - yarn install
    - cat $database_yml > config/database.yml
    - cat $env > config/application.yml
    - bundle exec rails db:create
    - bundle exec rails db:schema:load
    - bundle exec rails assets:precompile

unit_and_integration_tests:
  extends: .base_db
  stage: test
  only:
    - merge_requests
  script:
    - bundle exec rspec --exclude-pattern "spec/system/**/*.rb"

system_tests:
  extends: .base_db
  stage: test
  services:
    - postgres:latest
    - name: selenium/standalone-chrome:122.0
      alias: selenium
  variables:
    RAILS_ENV: test
    POSTGRES_HOST_AUTH_METHOD: trust
    SELENIUM_REMOTE_URL: http://selenium:4444/wd/hub
  artifacts:
    when: on_failure
    paths:
      - log/test.log
    expire_in: 1 week
  only:
    - merge_requests
  script:
    - bundle exec rspec spec/system

Conclusion

Phew, that was a lot of configurations and explanation.

The reason why I wrote this blog was because I faced various problems when trying out other blogs in the internet today and couldn't fully understand what was happening inside the configuration file because there were no explanations. I hope I have explained everything the code is doing and you don't have to waste time in researching about these things again.

With this, your Rails app now has all type of tests running in the Gitlab CI so you can now merge changes without any worry for them breaking the production application.

Thank you for reading. Happy coding!

References

Image Credits:

Cover Image by Jens Freudenau on Unsplash

Devise raise validations error when new and old passwords are same

Prabin Poudel — Wed, 21 Feb 2024 18:15:00 +0000

Authentication is a deal breaking feature in any applications nowadays. In Rails, Devise makes authentication a breeze; install a gem, run few commands and you have authentication working in your app.

Today, with the help of Devise we will look into a solution for a very common feature request in the app; throw validation error if user tries to change their password but add the same old password.

Skills required to follow the tutorial

Intermediate:

Rails

You should have

Existing Rails app with authentication already handled using Devise

Validation Implementation

Let's dive into the code now.

Add the following validation error to the model that is used for authentication:

class User < ApplicationRecord
  # ============ Custom Validation ============
  validate :new_and_old_password_must_be_different

  private

  def new_and_old_password_must_be_different
    return if changed.exclude?('encrypted_password')

    password_is_same = Devise::Encryptor.compare(User, encrypted_password_was, password)

    errors.add(:password, I18n.t('validations.not_allowed.old_password')) if password_is_same
  end
end

Please note that I am using "User" model for storing all users and authenticate them but table could be anything else like "Admin" as well.

We will understand what each line of code means in next section.

Code Explanation

changed.exclude?('encrypted_password')

ActiveModel stores changes that were made in the current transaction inside the variable "changed" and with this line of code we are returning early from the validation if user was updated but password wasn't updated.
Devise::Encryptor.compare(User, encrypted_password_was, password)

We are already using Devise for authentication so we are reaching out to the helper module "Encryptor" from Devise to compare new password with the old one. Here, current password will be in plain format and "Encryptor" will hash the password with relevant algorithm before comparing so we know if the password is same or different.

This line will return true if previous password is same as the new password or false if they are different.
errors.add(:password, I18n.t('validations.not_allowed.old_password'))

Lastly, we are adding validation errors to the User model if the password is same. And controller action will return the validation error to show it in frontend.

Conclusion

And with that we have successfully added a way for our app to throw validation errors when old password is used with the help of Devise. I hope you got to learn something new today.

Thank you for reading. Happy coding!

References

[Stack Overflow] Rails + Devise: How do I get an error message if password is not changed?

Image Credits

Cover Image by alexander ehrenhöfer on Unsplash

Run Cron Job Manually

Prabin Poudel — Wed, 17 Jan 2024 14:08:51 +0000

Cron Job has made the scheduling of various tasks super easy but debugging those scripts from Cron Job is equally harder. This is because Cron Job runs command you provide to it always at one specific time; which means there are only two ways for you to run the Cron Job:

Wait for the Cron Job to run at the time you have given, could be seconds, minutes or even hours
Run the Cron Job every minute and see if it throws any error (most people do this)

What if I tell you there is also a third way; run the Cron Job manually.

This approach will help us in running any of our script immediately without having to wait for Crontab to execute our job at specific time. And it will also run the job by replicating the same environment Cron uses while running it's jobs. Super helpful for debugging any Cron related issue if you ask me.

Steps

Let's dig into the approach of running the Cron Job manually then.

Step 1: Get PATHs your normal Cron Job uses

The biggest difference in running the script directly from the command line versus from the Cron Job is difference in it's PATH values.

A lot of times, Cron Job fails to execute because PATH doesn't include executables for e.g. for Ruby, PERL, etc. and due to these executables not being found script that was running properly in development or even from command line in the production server will no more work when running the same script from the Cron Job.

Run the command crontab -e to edit your crontab and add the following entry to the top (or anywhere in the file):

* * * * * /usr/bin/env > /home/deploy/cron-env

Please note that /home/deploy is the root path of my logged in user in the Production server, it could be different for you.

With the above entry in the crontab, we are telling the Cron Job to run every minute and add PATHs it gets from the command "/usr/bin/env" to a file named "cron-env"

Wait for a minute and check the content inside the file "cron-env" with the command cat cron-env, it should have the content similar to this:

HOME=/home/deploy
LOGNAME=deploy
PATH=/usr/bin:/bin
LANG=en_US.UTF-8
SHELL=/bin/sh
PWD=/home/deploy

Please note that "deploy" is the name of the logged in user.

You should remove the Cron Job we added previously from the Crontab since it has served it's purpose of providing us with PATHs we will require to run commands as a Cron Job.

Step 2: Create a bash script to run any command similar to Cron Job

Create a new file with the command nano run-as-cron and add the following inside:

#!/bin/bash

/usr/bin/env -i $(cat /home/deploy/cron-env) "$@"

Here is what the above command is doing:

/usr/bin/env is the path to the env command, which is used to find and execute a specified command with a modified environment
-i option tells env to start with an empty environment, ignoring the inherited environment variables.
$(cat /home/deploy/cron-env) uses the "cat" command to read the contents of the file located at /home/deploy/cron-env and then uses command substitution $(...) to include the contents of the file as arguments to the env command. The contents of the file are expected to be environment variable assignments for any commands that is run using this bash script.
"$@" is a special variable which represents all the arguments passed to the script or command. In this context, it allows the script or command executed by env to receive the same arguments that were passed to the original script or command containing this line.

For e.g. if you run the command ./run-as-cron /path/to/script -with arguments then everything after "./run-as-cron" is passed as it is here and executed by this script.

Step 3: Run any command you want by replicating the Cron behavior

Now we are ready to run any command we want using the same environment variables Cron Job uses during it's run.

You can run the command you want now with:

$ ./run-as-cron /path/to/script --with arguments --and parameters

For example if you have a script let's say to read a CSV file and parse them as JSON then you can run that command now with:

$ ./run-as-cron /home/deploy/production/scripts/parse-csv-as-json '/home/deploy/production/csv-files/employees.csv'

This assumes you take the file path as an argument inside the script.

And that's it, you should now be able to run the Cron Job manually and debug the result from it immediately!

If you are a Ruby on Rails developer follow along for a bit more as I have a bonus tip for you.

Bonus: Run Ruby on Rails tasks replicating the Cron behavior

Unrelated to this tutorial and being a Rails developer myself, I thought of attaching some bonus example on running the rake task by executing as the Cron Job manually.

I am also assuming that you also already have whenever gem configured and added some Rake tasks to "config/schedule.rb"

Now for example if you have a rake task that sends an email to you once a day with the list of users who registered yesterday to your app then the command could look like something below:

$ ./run-as-cron /bin/bash -l -c 'cd /home/deploy/production/your-app/current && RAILS_ENV=production /home/deploy/.rbenv/shims/bundle exec rake users:registered_yesterday --silent >> log/whenever.log 2>&1'

I have just copied the command whenever executes when we deploy the Rails app with "Capistrano", here is the short explanation of bash command and options used above:

/bin/bash specifies the Bash shell to be used for running the command.
-l option makes Bash act as if it had been invoked as a login shell. This means it will read certain login-specific configuration files.
-c '...' option allows you to pass a command as a string for Bash to execute. The command inside the single quotes is the actual command being executed.
cd /home/deploy/production/your-app/current && RAILS_ENV=production /home/deploy/.rbenv/shims/bundle exec rake users:registered_yesterday tells the script to change the working directory to the folder ".... /current" and execute the Rake task.
`>> log/whenever.log 2>&1 appends both standard output and standard error to the file "log/whenever.log". This is a common technique to capture both normal output and errors in a log file.

Conclusion

In this short and to the point tutorial we have learnt how we can run the Cron Job manually and immediately be able to debug issues instead of having to wait for some minutes or hours to debug the issue.

I hope you have also learnt about some new bash command options and techniques as I did while writing this blog. Lastly, I hope it helps you save a lot of your time into the future now when debugging any Cron Job.

Thank you for reading. Happy Tinkering!

References

Running a cron job manually and immediately [Server Fault]

Image Credits

Cover Image by Icons8 Team on Unsplash

[Solved] Ubuntu 22 Temporary failure in name resolution

Prabin Poudel — Fri, 05 Jan 2024 09:04:06 +0000

Background

In one of the client project, we had to upgrade our existing Ubuntu server from the version 20 to 22. Upgrade was smooth but after the upgrade was done, we started noticing the issue of "Temporary failure in name resolution".

When does this error occur?

The "Temporary failure in name resolution" error occurs when the system cannot translate a website name into an IP address. Somehow it got messed up during the upgrade

Fix

To fix the issue, you can hit the following commands; this fix was tested on an upgrade Ubuntu at version 22.04.2

$ apt install netplan.io
$ systemctl unmask systemd-networkd.service
$ systemctl unmask systemd-resolved.service
$ ENABLE_TEST_COMMANDS=1 netplan migrate
$ netplan try

# WARNING: This will immediately log you out of the server and restart it, if you are working with Production server; run with care.
$ reboot

$ apt purge ifupdown resolvconf
$ ln -sf /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf

Tada, and that should fix the issue 🎉

Conclusion

Are there any other solutions you tried and it worked? Let us know in the comments below.

Thanks for reading. Until next time! Happy tinkering.

References:

Image Credits

Cover Image by Albert Stoynov on Unsplash

Setup Active Job with Sidekiq in Rails

Prabin Poudel — Mon, 02 Jan 2023 23:58:30 +0000

Active Job is a framework for declaring jobs and making them run on a variety of queuing backends. These jobs can be everything from regularly scheduled clean-ups, to billing charges, to mailings. Anything that can be chopped up into small units of work and run in parallel.

Active Job comes pre installed in Rails.

Now on to Sidekiq; it is one of the most widely used background job frameworks that you can implement in a Rails application. It is multi-threaded and utilizes Redis for its queuing storage.

Today we will be working on setting up Active Job with Sidekiq while also following the official Rails documentation.

Skills required to follow the tutorial

Intermediate:

Rails
Linux skills to work with commands in server where your app has been deployed

You should have

Existing Rails app
Linux server already setup to run Rails app

Step 1: Install Capistrano Gems

Add the following to your Gemfile under development group:

gem 'sidekiq'

Run the following in command line:

$ bundle install

That should install the latest version of Sidekiq gem, now let's lock the version by looking at the "Gemfile.lock", this version was what I had in my lock file, yours could be different.

gem 'sidekiq', '~> 7.0.0'

Step 2: Enable Sidekiq

Add the following to "config/application.rb"

# Use sidekiq for active jobs
config.active_job.queue_adapter = :sidekiq

Step 3: Install Redis

Redis is an open source, in-memory data store used by millions of developers as a database, cache, streaming engine, and message broker. It is an in-memory key-value store known for its flexibility and performance.

Sidekiq is backed by Redis as a job management store to process thousands of jobs per second.

Official Redis Documentation has got you covered for the installation of Redis in any OS.

On MacOS via Homebrew

If you are on MacOS, you can follow instructions at Install Redis on MacOS (Official Documentation).

Linux

You can either choose to install from source or from APT repository.

I had installed Redis from the source and these were the instructions that I needed to perform from the command line:

$ wget http://download.redis.io/redis-stable.tar.gz
$ tar xvzf redis-stable.tar.gz
$ cd redis-stable
$ make

Copy the executables to you local usr bin folder so you can run the redis-server and cli commands from your home/user directory

sudo cp src/redis-server /usr/local/bin/
sudo cp src/redis-cli /usr/local/bin/

Automatically start on machine restart

NOTE: You can also find these instructions in the official Redis documentation at Getting Started > Installing Redis more properly

Create a directory in which to store your Redis config files and your data:
```
sudo mkdir /etc/redis
sudo mkdir /var/redis
```
Copy the init script that you'll find in the Redis distribution under the utils directory into /etc/init.d. We suggest calling it with the name of the port where you are running this instance of Redis. For example:
```
sudo cp redis-stable/utils/redis_init_script /etc/init.d/redis_6379
```
Edit the init script
```
sudo nano /etc/init.d/redis_6379
```
Make sure to modify REDISPORT according to the port you are using. Both the pid file path and the configuration file name depend on the port number. I had used the port "6379"
Copy the template configuration file you'll find in the root directory of the Redis distribution into /etc/redis/ using the port number as name, for instance:
```
sudo cp redis-stable/redis.conf /etc/redis/6379.conf
```
Create a directory inside /var/redis that will work as data and working directory for this Redis instance:
```
sudo mkdir /var/redis/6379
```
Edit the configuration file with sudo nano /etc/redis/6379.conf, making sure to perform the following changes:
- Change the port accordingly. In our example it is not needed as the default port is already 6379. It is under the section ## NETWORK ##
- Set daemonize to yes (by default it is set to no). It is under the section ## GENERAL ##.
- Set the pidfile to /var/run/redis_6379.pid (modify the port if needed). It is under the section ## GENERAL ##
- Set your preferred loglevel (notice by default). It is under the section ## GENERAL ##
- Set the logfile to /var/log/redis_6379.log ("" by default). It is under the section ## GENERAL ##
- Set the dir to /var/redis/6379 (very important step!). It is under the section ## SNAPSHOTTING ##
Finally add the new Redis init script to all the default runlevels using the following command
```
sudo update-rc.d redis_6379 defaults
```
You are done! Now you can try running your instance with:
```
sudo /etc/init.d/redis_6379 start
```
Test if it working correctly
- Try pinging your instance with redis-cli using the command redis-cli ping, you should see PONG
- Do a test save with redis-cli save and check that the dump file is correctly stored into /var/redis/6379/ (you should find a file called dump.rdb).
- Check that your Redis instance is correctly logging in the log file with tail -f /var/log/redis_6379.log, you can exit with CTRL + c
- If you are on a new machine/server where you can try things without taking anything else down in the machine/server make sure that after a reboot everything is still working (redis should start automatically on machine restart)

Windows

If you are on Windows, you can follow instructions at Install Redis on Windows (Official Documentation).

Step 4: Enable Web UI to Monitor Jobs

Add the following to your "config/routes.rb":

# config/routes.rb

require 'sidekiq/web'

Myapp::Application.routes.draw do
  # mount Sidekiq::Web in your Rails app
  mount Sidekiq::Web => "/sidekiq"
end

Step 6: Accessing the Sidekiq Web UI

Run rails server: rails s
Go to "http://localhost:3000/sidekiq"

You should now see the UI similar to this:

Step 6: Add Basic Authentication to Web UI for Preventing Unauthorized Access

By default there is no authentication and anyone that knows your API URL will be able to access the sidekiq UI.

You can allow any authenticated User to access the Sidekiq UI by adding following configurations to your "config/routes.rb":

# config/routes.rb

authenticate :user do
  mount Sidekiq::Web => '/sidekiq'
end

NOTE: This configuration is for the Devise gem, if you are using any other authentication gem then this configuration might be different.

You can view more options for authenticating users to restrict Sidekiq UI at Authentication (Sidekiq Gem Github).

Example Job

Now let's look at how a job can look like in the Rails App after setting up Sidekiq.

Application Job

Your "app/jobs/application.job" should look similar to the following:

class ApplicationJob < ActiveJob::Base
  queue_as :default
  # Automatically retry jobs that encountered a deadlock
  # retry_on ActiveRecord::Deadlocked

  # Most jobs are safe to ignore if the underlying records are no longer available
  # discard_on ActiveJob::DeserializationError
end

Only change from the default ApplicationJob that Rails generates by default is that we have added queue_as :default so we don't have to add it in all other jobs we add in our app.

If you want your job to be queued with the highest priority then in the new job you generate, you should update queue_as to high e.g. queue_as :high to override the queue_as :default inside the Application Job.

Expire User Note Job

Let's say you have a job that automatically removes the status of the user after certain time e.g. in the communication app called Slack we can add status and set expiration date and time. Your job could like the following for such feature:

class ExpireUserStatusJob < ApplicationJob
  def perform(user_status_id)
    status = UserStatus.find(user_status_id)

    return if status.blank? || !status.expired?

    status.update!(description: nil, expiry_date: nil, expiry_time: nil)
  end
end

And you could queue the job from UserStatus model by triggering the method "schedule_expiration" from the controller whenever the user status is updated:

class UserStatus < ApplicationRecord
  validates :expiry_date, presence: true, if: -> { expiry_time }
  validates :expiry_time, presence: true, if: -> { expiry_date }
  validates :description, presence: true, if: -> { expiry_date || expiry_time }

  def schedule_expiration
    return if expiry_date.blank? || expired?

    expiry_date_time = combine_date_time(expiry_date, expiry_time)

    ExpireUserStatusJob.set(wait_until: expiry_date_time).perform_later(id)
  end

  private

  def expired?
    return if expiry_date.blank? || expiry_date < Time.zone.today

    expiry_time.blank? || expiry_time.to_s(:time) <= Time.zone.now.to_s(:time)
  end

  def combine_date_time(date, time)
    DateTime.new(date.year, date.month, date.day, time.hour, time.min, time.sec)
  end
end

Bonus: Setup Sidekiq in Ubuntu Server

Setup

Create a file called sidekiq.service sudo nano /lib/systemd/system/sidekiq.service
Copy content from Example Sidekiq systemd configurations to this file.
Update the file:
- Change the WorkingDirectory to the folder where your app is deployed e.g. WorkingDirectory=/home/deploy/my-app/current
- Update the ExecStart path. This specifies the path and command to start Sidekiq. E.g. my current ExecStart path with rbenv is: ExecStart=/home/deploy/.rbenv/shims/bundle exec sidekiq -e production. NOTE: If you are using different Ruby version manager then "/home/deploy/.rbenv/shims/bundle" will be different.
- Add ExecReload=/usr/bin/kill -TSTP $MAINPID below ExecStart=....
- Remove comment from # User=deploy and update deploy to be the user you are using in the server, mine was deploy so I changed the line to User=deploy
- Uncomment # Group=sudo to make it Group=sudo
- Uncomment # UMask=0002 to make it UMask=0002
- Update WantedBy=multi-user.target to WantedBy=default.target
- Save the file

Run Sidekiq

You can enable the Sidekiq service with sudo systemctl daemon-reload

You can start the Sidekiq service with sudo systemctl start sidekiq

Here are some useful commands for changing status of Sidekiq:

"sudo systemctl {start,stop,status,restart} sidekiq" e.g. sudo systemctl status sidekiq

You can also use "sudo service sidekiq {start,stop,status,restart}" to perform commands e.g. sudo service sidekiq status

Sidekiq with Capistrano

If you are using capistrano for deployment, you can refer to the section "Bonus 2: Sidekiq for background jobs" to configure Sidekiq with Capistrano at Deploy API only Rails App with Capistrano.

Conclusion

Congratulation!!! You have successfully setup and deployed sidekiq if you followed the blog all the way to the end.

If you have any queries or confusions please let me know in the comments below and I will help you to clear those to the best of my ability.

Thanks for reading. Happy tinkering and happy coding!

Image Credits

Cover Image by Scott Blake on Unsplash

References

Deploy API only Rails App with Capistrano

Prabin Poudel — Thu, 08 Dec 2022 04:18:00 +0000

Capistrano is a deployment automation tool built on Ruby, Rake, and SSH. It allows you to deploy your app to a remote server in a single command after initial configurations are done.

Deployment with capistrano is as easy as cap production deploy. But to be able to hit this command, a lot of configurations need to be added first.

Today we will be looking into setting up capistrano in our API only Rails application for making it easier to deploy to any remote server.

Skills required to follow the tutorial

Intermediate:

Rails
Linux skills to work with commands in server where your app has been deployed

You should have

Existing Rails app
Linux server already setup to run Rails app

NOTE:
I am using rbenv for ruby so all configurations will be based on that, you can replace it as required for your version manager as required e.g. for rvm.

Step 1: Install Capistrano Gems

Add the following to your Gemfile under development group

group :development do
  gem "capistrano"
  gem "capistrano-rails"
  gem 'capistrano-rbenv'
end

Run the following from command line:

$ bundle install

That should install the latest version of capistrano gems, now let's lock those versions by looking at Gemfile.lock, these versions were what I had in my lock file, yours could be different.

group :development do
  gem "capistrano", "~> 3.17", require: false
  gem "capistrano-rails", "~> 1.6", require: false
  gem 'capistrano-rbenv', '~> 2.2', require: false
end

Step 2: Generate default configuration files

Run the generator to create a basic set of configuration files:

$ bundle exec cap install

Step 3: Require correct plugins in Capfile

Uncomment the following plugins in Capfile located at the root of the project

  # require "capistrano/rvm"
  require 'capistrano/rbenv'
  # require "capistrano/chruby"
  require 'capistrano/bundler'
  require "capistrano/rails/assets"
  require 'capistrano/rails/migrations'
  # require 'capistrano/passenger'

Step 4: Update Deploy file

Update the deploy file at config/deploy.rb with values as required:

application: Set to the application name or project name
repo_url: Set to the URL where your code is being stored, you can normally get this with git remote -v and copy the URL of origin
linked_files: Set to a list of files that should be shared and persisted over all releases, for e.g. config/database.yml, config/master.key, config/application.yml, etc. These files will not be deleted/reset in every release (other files will!!)
linked_dirs: Set to a list of folders that should be shared and persisted over all releases, for e.g. log, tmp/pids, node_modules, etc. These folders will not be deleted/reset in every release (other folders will!!)
keep_releases: Set to a number of previous releases you want to keep in the server after each release, I normally keep this to 3.
conditionally_migrate: Skip migration if files in "db/migrate" were not modified. I normally set this to "true" but default is "false"

Your deploy file could look something like below:

# frozen_string_literal: true

# config valid for current version and patch releases of Capistrano
lock '~> 3.17.0'

set :application, 'contract-template-editor-api'
set :repo_url, 'git@github.com:truemark/contract-template-editor/api.git'

# Default value for :linked_files is []
set :linked_files, %w[config/application.yml config/database.yml config/master.key]

# Default value for linked_dirs is []
set :linked_dirs, %w[log tmp/pids tmp/cache tmp/sockets vendor/bundle .bundle public/system public/uploads node_modules]

# Default value for keep_releases is 5
set :keep_releases, 3

# Skip migration if files in db/migrate were not modified
# Defaults to false
set :conditionally_migrate, true

Step 5: Update environment specific deploy files

I have normally worked on projects that has staging and production environment and configuration files for these two environment is provided default by Capistrano; each environment specific file is located under config/deploy/{environment}.rb.

In these files, there should be configurations that can change based on the environment of the Rails application.

Configurations for each environment will be the same with different value based on the environment e.g. server ip address will be different in production and staging.

We will only be looking at config/deploy/production.rb in this tutorial.

Update the deploy file at config/deploy/production.rb with values as required:

server: IP address of the server where you will deploy the app. It is a good idea to sore this value inside env file or rails credentials and take it from there.
user: name of the user in the server, normally it will be "deploy" but could be username e.g. prabin
roles: list of accessible roles for this user
deploy_to: path of the folder to deploy your app to, you can get the path by logging in to the server, going to the folder you want your app to be deployed to and enter the command pwd. If you haven't created the folder yet, you can create one e.g. contract-template-api and set the path of that folder.
branch: git branch that will be used to deploy the app, normally "main" or "master"
stage: environment of the app, should be "production"
rails_env: same as stage, should be "production"

Your file could look something like below:

server ENV['deploy_server_ip'], user: 'deploy', roles: %w[app db web], primary: 'true'
set :deploy_to, 'path to the folder where app should be deployed e.g. /home/deploy/contract-template-editor/api'
set :branch, 'main'
set :stage, :production
set :rails_env, :production

Step 6: Upload secret keys and files

Secret keys and files should never be committed to git. These files are required for app to function properly. Normally these files are always configured in the developer's machine so they can be uploaded directly from the project to the server using ssh.

Create a new rake task at lib/capistrano/tasks/config_files.rake and add the following content:

namespace :config_files do
  desc 'Upload yml files inside config folder'
  task :upload do
    on roles(:app) do
      execute "mkdir -p #{shared_path}/config"

      upload! StringIO.new(File.read('config/database.yml')), "#{shared_path}/config/database.yml"
      upload! StringIO.new(File.read('config/application.yml')), "#{shared_path}/config/application.yml"
      upload! StringIO.new(File.read('config/master.key')), "#{shared_path}/config/master.key"
    end
  end
end

We now need to tell capistrano to run this code during deployment, add the following to the end of config/deploy.rb

# ================================================
# ============ From Custom Rake Tasks ============
# ================================================
# ===== See Inside: lib/capistrano/tasks =========
# ================================================

# upload configuration files
before 'deploy:starting', 'config_files:upload'

Step 7: Create a database if deploying for the first time

Create a new rake task at lib/capistrano/tasks/database.rake and add the following content:

namespace :database do
  desc 'Create the database'
  task :create do
    on roles(:app) do
      within release_path do
        with rails_env: fetch(:rails_env) do
          execute :rake, 'db:create'
        end
      end
    end
  end
end

We now need to tell capistrano to run this code during deployment, add the following to the end of the deploy file.

# set this to false after deploying for the first time 
set :initial, true

# run only if app is being deployed for the very first time, should update "set :initial, true" above to run this
before 'deploy:migrate', 'database:create' if fetch(:initial)

Step 8: Reload the Rails application after successful deploy

Create a new rake task at lib/capistrano/tasks/application.rake and add the following content:

namespace :application do
  desc 'Reload application'
  task :reload do
    desc 'Reload app after deployment'
    on roles(:app), in: :sequence, wait: 5 do
      execute :touch, release_path.join('tmp/restart.txt')
    end
  end
end

We now need to tell capistrano to run this code during deployment, add the following to the end of the deploy file.

# reload application after successful deploy
after 'deploy:publishing', 'application:reload'

Final file

Your final "config/deploy.rb" file should look similar to this:

# frozen_string_literal: true

# config valid for current version and patch releases of Capistrano
lock '~> 3.17.0'

set :application, '{project name}'
set :repo_url, '{remote git repository where project for the code is stored}'

# Default value for :linked_files is []
append :linked_files, %w[config/application.yml config/database.yml config/master.key]

# Default value for linked_dirs is []
append :linked_dirs, %w[log tmp/pids tmp/cache tmp/sockets vendor/bundle .bundle public/system public/uploads node_modules]

# Default value for keep_releases is 5
set :keep_releases, 3

# Skip migration if files in db/migrate were not modified
# Defaults to false
set :conditionally_migrate, true

# ================================================
# ============ From Custom Rake Tasks ============
# ================================================
# ===== See Inside: lib/capistrano/tasks =========
# ================================================

# upload configuration files
before 'deploy:starting', 'config_files:upload'

set :initial, true

# run only if app is being deployed for the very first time, should update "set :initial, true" above to run this
before 'deploy:migrate', 'database:create' if fetch(:initial)

# reload application after successful deploy
after 'deploy:publishing', 'application:reload'

Deploy the app

From the command line you can now deploy the app to production using the following command:

cap production deploy

The app should be deployed in around 5-10 minutes, if you encounter any error while deploying and need any help, please post a comment below and I will do my best to help you resolve it.

Bonus 1: Whenever for cron jobs

whenever gem is used in Rails applications to schedule cron jobs e.g. send email notification about monthly expenditure on the 1st of each month.

Add the following to the deploy file just below "set :conditionally_migrate, true":

# Skip migration if files in db/migrate were not modified
# Defaults to false
set :conditionally_migrate, true

# Set unique identifier for background jobs
set :whenever_identifier, -> { "#{fetch(:application)}_#{fetch(:stage)}" }

whenever_identifier should be set to unique identifier for cron jobs, this is required if you have deployed multiple applications to same server (normally for staging servers) that require whenever gem so that those applications don't clash with one another for cron jobs. This is optional if you only have one application in the server.

Create a new rake task at lib/capistrano/tasks/whenever.rake and add the following content which will be responsible for updating cron tasks configured inside the "config/schedule.rb":

namespace :whenever do
  desc 'Update cron job'
  task :update_crontab do
    on roles(:app) do
      within current_path do
        execute :bundle, :exec, "whenever --update-crontab #{fetch :whenever_identifier} --set 'environment=#{fetch(:stage)}'"
      end
    end
  end
end

We now need to tell capistrano to run this code during deployment, add the following to the deploy file just below "before 'deploy:migrate', 'database:create' if fetch(:initial)":

# run only if app is being deployed for the very first time, should update "set :initial, true" above to run this
before 'deploy:migrate', 'database:create' if fetch(:initial)

# update cron job from whenever schedule file at "config/schedule.rb"
after 'deploy:finishing', 'whenever:update_crontab'

Bonus 2: Sidekiq for background jobs

Sidekiq gem is used in Rails applications to schedule background jobs so as to perform them at a later point without having to stop the execution of other codes.

Capistrano Configurations

Create a new rake task at lib/capistrano/tasks/sidekiq.rake and add the following content which will be responsible for updating cron tasks configured inside the "config/schedule.rb":

namespace :sidekiq do
  desc 'Quieten sidekiq'
  task :quiet do
    on roles(:app) do
      puts capture("pgrep -f 'sidekiq' | xargs kill -TSTP")
    end
  end

  desc 'Restart Sidekiq'
  task :restart do
    on roles(:app) do
      execute :sudo, :systemctl, :restart, :sidekiq
      execute :sudo, :systemctl, 'daemon-reload'
    end
  end
end

We now need to tell capistrano to run this code during deployment, add the following to the deploy file just below "after 'deploy:publishing', 'application:reload'":

# reload application after successful deploy
after 'deploy:publishing', 'application:reload'

# sidekiq related commands
after 'deploy:starting', 'sidekiq:quiet'
after 'deploy:reverted', 'sidekiq:restart'
after 'deploy:published', 'sidekiq:restart'

Now try to deploy the app to production and server will ask for password when running sidekiq commands. To fix that we need to add some more configurations in the remote server.

Server Configurations

We are assuming that sidekiq is already configured in your remote server. If you have not configured it yet, you can refer to the section "Bonus: Setup Sidekiq in Ubuntu Server" at Setup Active Job with Sidekiq in Rails

For capistrano to perform sudo actions without asking for the password, the user used by capistrano, normally "deploy" user should be in the sudo group and you should add commands that need to be executed in the server with sudo access but without using password to "/etc/sudoers" file.

Add deploy user to the sudo group

You can add your deploy user to the sudo group with the following command

# add "deploy" user to sudo group
$ sudo usermod -aG sudo deploy

# verify if the user has been added to the sudo group
# result should include "sudo" for the deploy user
$ groups deploy
deploy : deploy sudo

Add commands required for the sidekiq restart and daemon-reload to be performed without password
- Open the sudoers file for the edit with sudo EDITOR=nano visudo, this will ensure that the content inside the file is validated before saving so you don't end up with invalid file. If you do sudo nano /etc/sudoers then it doesn't validate the content so you should never do that.
- Add the following below the line root ALL=(ALL:ALL) ALL under "# User privilege specification"
```
  deploy ALL=NOPASSWD: /bin/systemctl restart sidekiq
  deploy ALL=NOPASSWD: /bin/systemctl daemon-reload
```
You can try running above two commands now in the command line of the server and it should run normally without asking for password:
```
$ sudo systemctl restart sidekiq
# doesn't ask for the password and executes the command
```

Now if you try to deploy again, capistrano won't stop to ask password when running sidekiq commands.

Conclusion

We have come to the finish line, now you should be able to deploy your API only Rails application to the server with one command.

Thanks for reading. Happy tinkering and happy coding!

Image Credits

Cover Image by NASA on Unsplash

[Solved] Error while Installing mysql2 Gem in M1 Mac

Prabin Poudel — Mon, 19 Sep 2022 15:21:53 +0000

In Ruby on Rails applications with mysql2 gem, mysql2 gem always threw error when trying in the new M1 Mac. The error always said "ld: library not found for -lzstd" and "make failed".

Let's resolve this issue today!

Assumptions

You are using rbenv for ruby
You are using homebrew

Error Message

Whenever I did bundle install inside the project, I got the following error:

linking shared-object mysql2/mysql2.bundle
ld: library not found for -lzstd
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [mysql2.bundle] Error 1

make failed, exit code 2

Solution

To resolve the issue, you will have to provide the location for mysql installation in homebrew and install the mysql2 gem separately.

Find out the version of mysql installed in your machine

Run the following command in the command line:
```
$ mysql --version
mysql  Ver 8.0.27 for macos12.5 on arm64 (Homebrew)
```
Note the mysql version; here 8.0.27 and move to next step.

Install mysql2 gem separately

To resolve the make error we will have to install mysql2 gem separately. To do that run the following command by changing the mysql version you got in previous step.

rbenv exec gem install mysql2 -- \
--with-mysql-lib=/opt/homebrew/Cellar/mysql/8.0.27/lib \
--with-mysql-dir=/opt/homebrew/Cellar/mysql/8.0.27 \
--with-mysql-config=/opt/homebrew/Cellar/mysql/8.0.27/bin/mysql_config \
--with-mysql-include=/opt/homebrew/Cellar/mysql/8.0.27/include

Run "bundle install" from your project root

Move to the project root and run bundle install in the command line.

Still getting the same error?

If you are still getting error, you need to also hit the command given below.

  # check zstd version
  $ ls /opt/homebrew/Cellar/zstd
  1.5.0

  $ bundle config --local build.mysql2 \
    "--with-ldflags=-L/opt/homebrew/Cellar/zstd/1.5.0/lib"

Run bundle install again from inside the project root and the error should now go away.

Downside

The downside of this solution is we have to run this command every time the mysql version is updated using homebrew.

So for example if your mysql version changes from 8.0.27 to 8.0.30 then you will again get this error. And in that case you will have to install the mysql2 gem separately again by using the steps above.

Conclusion

If you have any other solutions for this problem, do let us know in the comment section below.

Thank you for reading. Happy coding!

References

[Stack Overflow] An error occurred while installing mysql2 (0.4.8), and Bundler cannot continue
[Github] bundle install fails with Gem::Ext::BuildError

Image Credits

Cover Image by Rubaitul Azad on Unsplash

Setup Action Mailbox with SendGrid

Prabin Poudel — Sun, 22 May 2022 15:55:32 +0000

Rails 6 released with many awesome features and action mailbox was one of them that has come to make the life easier. From Official Action Mailbox Guide:

Action Mailbox routes incoming emails to controller-like mailboxes for processing in Rails. It ships with ingresses for Mailgun, Mandrill, Postmark, and SendGrid. You can also handle inbound mails directly via the built-in Exim, Postfix, and Qmail ingresses.

Basically, Action Mailbox can be used to forward all incoming emails to your Rails app and process it further as required like storing attachments, creating records from the email body in you database and many more.

And today, we will be implementing Action Mailbox with SendGrid.

Requirements

Setup Action Mailbox with SendGrid using the official Rails documentation
Update DNS records to forward emails received in the mailbox towards our Rails app
Test integration in development with built in UI provided by Rails
Test integration in development with NGROK to ensure seamless production release

Tested and working in

Ruby 3.0.0
Rails 7.0.2.4
Action Mailbox 7.0.2.4

You should have

Existing app built with Rails 7 or higher

Let's start integrating Action Mailbox with SendGrid in our Rails app now.

Step 1: Setup action mailbox

We will be following instructions from the Official Rails Guide for Action Mailbox.

Install migrations needed for InboundEmail and ensure Active Storage is set up:

$ rails action_mailbox:install
$ rails db:migrate

Step 2: Add Ingress Configurations

Tell Action Mailbox to accept emails from SendGrid by adding the following to both "development.rb" and "production.rb"

# config/environments/development.rb & config/environments/production.rb
config.action_mailbox.ingress = :sendgrid

Step 3: Generate Password for authenticating requests

First of all, we should generate a strong password that Action Mailbox can use to authenticate requests to the SendGrid ingress.

You can add any strong password or let Rails generate it for you. You can log into Rails console and generate a password for you:

> rails c
irb > SecureRandom.alphanumeric
# => "Kk9YGvzdPN69bfiu"

After that you can use rails credentials:edit in the command line to add the password to your application's encrypted credentials under action_mailbox.ingress_password, where Action Mailbox will automatically find it:

action_mailbox:
  ingress_password: YOUR_STRONG_PASSWORD

If you are using nano editor you can edit credentials with following command:

  $ EDITOR="nano" rails credentials:edit

Alternatively, you can also provide the password in the RAILS_INBOUND_EMAIL_PASSWORD environment variable.

If you are using figaro gem you can add the following to your "config/application.yml":

# config/application.yml

RAILS_INBOUND_EMAIL_PASSWORD: 'YOUR_STRONG_PASSWORD'

Step 4: Setup a Mailbox

Now we should setup a mailbox that will process all incoming emails through our Rails app.

You can generate a new mailbox with:

$ bin/rails generate mailbox forwards

This will create forwards_mailbox inside app/mailboxes

# app/mailboxes/forwards_mailbox.rb
class ForwardsMailbox < ApplicationMailbox
  def process
  end
end

Step 5: Whitelist email domains

We can configure our application_mailbox to accept all incoming emails to our Rails app and forward it to our forwards_mailbox for further processing.

Action Mailbox also accepts regex to whitelist domains or match certain emails. Let's look at how we can configure all these alternatives:

Accept all incoming emails

  # app/mailboxes/application_mailbox.rb
  class ApplicationMailbox < ActionMailbox::Base
    routing :all => :forwards
  end

Accept all emails from single domain

  # app/mailboxes/application_mailbox.rb
  class ApplicationMailbox < ActionMailbox::Base
    routing /.*@email-domain.com/i => :forwards
  end

Accept email from multiple domains

# app/mailboxes/application_mailbox.rb
class ApplicationMailbox < ActionMailbox::Base
  routing /.*@primary-email-domain.com|.*@secondary-email-domain.com/i => :forwards
end

This regex matching is telling application mailbox to forward all emails coming from @email-domain.com to our forwards_mailbox. For e.g. if we configure it to be /.*@gmail.com/i and our Rails app receives email to john-doe@gmail.com then it will be forwarded to our forwards_mailbox where we can further process it since this email matches with the pattern @gmail.com.

NOTE: Your mailbox name should match the name you've given it in the routing params i.e. forwards will route to forwards_mailbox.

Step 6: Test in development

Action Mailbox provides it's own set of UIs to test inbound emails in the development environment. To access this, let's fire up the Rails server first:

$ rails s

Visit Action Mailbox Inbound Emails Localhost URL and click on New inbound email by form. Fill in all required details like From, To, Subject and Body. You can leave other fields blank.

Before clicking on Deliver inbound email, let's add byebug (or any other debugging breakpoint e.g. binding.pry) to our process method so we know action mailbox is actually forwarding our emails to the right place.

# app/mailboxes/forwards_mailbox.rb
class ForwardsMailbox < ApplicationMailbox
  def process
    byebug
  end
end

You should make sure that email in From input box matches the email domain configured. Now when you click Deliver inbound email, the execution of the server process should stop at the process method since we have a breakpoint there. This means action mailbox is correctly forwarding incoming emails and our configurations are correct. You can perform further process as required in your app now.

But wait. Dang, there is an error from Rails while testing inbound emails in development!

Let's dig into what is happening.

Issue with Inbound Action Mailbox Testing in Development

Error reads: "undefined method 'original_filename' for "":String" and "NoMethodError in Rails::Conductor::ActionMailbox::InboundEmailsController#create"

Looking at the code in Action Mailbox of core Rails, I found out that this error is occurring because controller is trying to process the empty attachment further. But finding out why Rails was submitting empty attachment when we haven't chosen any attachment was hard. Note that, this also happens even if you choose one or multiple attachments.

In params, we get this "attachments"=>[""] and controller is trying to process it further with the following code:

private

def new_mail
  Mail.new(mail_params.except(:attachments).to_h).tap do |mail|
    mail[:bcc]&.include_in_headers = true
    mail_params[:attachments].to_a.each do |attachment|
      mail.add_file(filename: attachment.original_filename, content: attachment.read)
    end
  end
end

Here, we are getting error in the line mail.add_file(filename: attachment.original_filename, content: attachment.read) because "attachment" is empty string i.e. "" and not an object which has properties like "original_filename". Hence the error.

After looking into controller, my next stop for debugging the error was to look into the view because it shouldn't have sent the empty attachment in the first place.

View was just using a normal file_upload tag:

<div>
  <%= form.label :attachments, "Attachments" %><br>
  <%= form.file_field :attachments, multiple: true %>
</div>

There couldn't be any issue here, so I looked into the rendered HTML in the webpage and found out that there was a hidden tag for attachment:

<input name="mail[attachments][]" type="hidden" value="" autocomplete="off">
<input multiple="multiple" type="file" name="mail[attachments][]" id="mail_attachments">

Hence, the form is submitting empty attachment to the controller.

This problem could be solved in controller by filtering out attachments that are empty and I was near to submitting a PR to Rails. But then I thought, if I am getting this issue, there are obviously other developers who have been into this since this is an issue in Rails core and not in the code I have written.

Searching further, I found this issue titled Action Mailbox Conductor throws NoMethodError when creating inbound email submitted to Rails Core Github.

And if there is an issue, there must also be a PR. YES, there was one already titled Cannot deliver new inbound email via form but it hadn't been merged yet.

But for this tutorial and until PR is merged, we need this to work in our app. So, I was looking into how I can resolve it the best and searching for the solution that would work for all of us and not just me.

Scrolling further into the issue, I found a monkey patching very suitable for our use case.

Monkey Patching the issue

Add the following to your config/application.rb

# monkey patching to resolve the issue of action mailbox inbound email sending empty attachment
config.to_prepare do
  Rails::Conductor::ActionMailbox::InboundEmailsController.class_eval do
    private

    def new_mail
      Mail.new(mail_params.except(:attachments).to_h).tap do |mail|
        mail[:bcc]&.include_in_headers = true
        mail_params[:attachments].to_a.compact_blank.each do |attachment|
          mail.add_file(filename: attachment.original_filename, content: attachment.read)
        end
      end
    end
  end
end

Don't forget to restart the server and reload the page. After that you can submit the form again.

Voilà!! It works 🥳.

Now, your server should have stuck in the debugging breakpoint.

That's it, we have now successfully setup action mailbox and tested in development.

Now let's test using NGROK so we know that our configuration will work seamlessly (pretty much) in our production environment.

Step 7: Setup NGROK

Let's setup NGROK in our local machine:

Download the application

You can download the application from this download link.

If you are on MacOS, I highly suggest downloading NGROK using homebrew with the command brew install ngrok/ngrok/ngrok. It's easier than manual download and also don't normally give off any issue.
Serve your app using NGROK URL

While keeping the rails server running as it is, open a new tab in your command line.

You can then run the command ngrok http 3000, which will give you an URL connecting your local Rails app running on port 3000 to the internet. You should look at the URL besides the "Forwarding" option, it will be something similar to Forwarding https://e73a-27-34-12-7.in.ngrok.io -> http://localhost:3000

When running the NGROK, you should see a screen similar to the screenshot below:
Access the Rails app with NGROK URL

Open the URL you got before from NGROK e.g. https://e73a-27-34-12-7.in.ngrok.io in your browser and you should be able to see the Rails welcome screen or whatever your default page for the app is.

But, but, there is an error again 😭

Hah, don't worry. I have got you covered.

You should be seeing the Error UI similar to what is in the screenshot below:

This happens because of missing "auth token" which you can get after signing up to NGROK for free.
Sign up to NGROK

You can sign up to NGROK using this signup link.
Add NGROK auth-token to local configuration file

After signing up, you are presented with a dashboard and you can copy the auth-token from setup-and-installation step number 2 called "Connect you account"

Or you can follow this link to your auth token page.

Copy the token given and run the following in your command line:
```
$ ngrok config add-authtoken <your-authtoken>
```
Now restart your NGROK server and go to the new URL provided.

NOTE: URL changes each time you restart the NGROK server unless you use pro version and pay for the static URL.

What, Error? Again!!! 🤕
Resolving blocked host in Rails app

After accessing the NGROK URL, you should see an error page similar to the one below:

This is because Rails blocks https access in development and unauthorized URLs overall.

Let's add the NGROK URL to "config/environments/development.rb"
```
# config/environments/development.rb

config.hosts << "add-your-ngrok-url"
```
Restart the rails server and reload the page in the browser. Now, you should be able to see the default page of your Rails app. This is what I see in mine since it's a new application just for this blog:
Authorize all NGROK URLs in development

Above, we only allowed current URL provided by NGROK and as I have already said this URL changes each time we restart the server.

Changing it every time we restart the server is a hassle so we will add regex which will allow matching NGROK URLs to connect to our Rails app in development.

Let's replace previous configuration with the following:
```
config.hosts << /.+\.ngrok\.io/
```
Now, if you restart the rails server you should still be able to access the default page in your app. Also, try with restarting the NGROK server and accessing the page again which you should still be able to without getting blocked host error.

Step 8: Authenticate domain in SendGrid

You can follow the official SendGrid tutorial to authenticate your domain in SendGrid.

Part of the tutorial also goes through setting up MX records which we will go into detail here.

You can authenticate your domain by following this link to the Domain Authentication page

You will receive list of CNAME and Values similar to what is listed below in the process where prabinpoudel.com.np and em1181 will be different:

em1181.prabinpoudel.com.np
s1._domainkey.prabinpoudel.com.np
s2._domainkey.prabinpoudel.com.np

We will come back to this page after next step again so don't close the page yet.

Let's go to our DNS provider's dashboard and configure these records first.

Setup DNS Records

We need to add DNS records from SendGrid to our DNS provider so our email is actually being processed by SendGrid and routed to our Rails app with Inbound Parse Hook.

I use CloudFlare, so I will be showing you process to setup MX record using the settings from CloudFlare as an example.

Go to DNS tab from the left menu
Click on "Add Record" and choose MX from the dropdown then add the following values to each field

a. Name: "@"
b. Mail Server: "mx.sendgrid.net"
c. TTL: "auto"
d. Priority: "10"

You can also find the instruction for adding MX record in the tutorial to setup Inbound Parse Hook from SendGrid.
Click on "Add Record" again and add all three CNAME records we got previously while authenticating the domain one by one

Copy values from authenticating the domain page add them to CloudFlare:

a. Type: CNAME
b. Name: value from CNAME
c. Target: value from VALUE
d. Proxy Status: Turn the toggle button off (it will be on by default)
e. TTL: Auto
Go back to domain authentication page and click on "I've added these records" and click on "Verify" button

If everything was copied over correctly, you will see a page with the information "It worked! Your authenticated domain for prabinpoudel.com.np was verified."

Else you will get errors and you will have to fix those before moving forward.

Step 9: Configure Inbound Parse in SendGrid

We will be following the official SendGrid doc for configuring inbound parse hook to forward inbound emails to /rails/action_mailbox/sendgrid/inbound_emails with the username "actionmailbox" and the password we generated just before this.

From your SendGrid Dashboard click Settings, and then click Inbound Parse. You are now on the Inbound Parse page. Or you can click on this Inbound Parse Link to go there directly.
Click "Add Host & URL"
You can add/leave the subdomain part as required. I have left it blank because I don't have any subdomain just for receiving emails
Under "Domain", choose your domain name that you just verified
Under the URL we will have to construct one and add it

The format for the URL should be: https://actionmailbox:<your_action_mailbox_ingress_password>@<rails_app_nginx_url>/rails/action_mailbox/sendgrid/inbound_emails

For e.g. it will be https://actionmailbox:my_strong_password@5829-2400-1a00-b050-3fb6-b0ce-5946-b9be.in.ngrok.io/rails/action_mailbox/sendgrid/inbound_emails for my Rails app.

In production it can be a different URL so you should replace rails_app_nginx_url with the URL from where your Rails application is accessible to the internet.
Check "POST the raw, full MIME message" and click on Add

Now we are ready to test our integration with live email using SendGrid and Ngrok.

Step 10: Test if MX records are recognized by the internet

Before we test our integration with live email, we need to make sure that MX records are recognized by the Internet.

It may take some time for DNS records to be recognized throughout the world so email forwarding may yet not work for you. The maximum time period until this happens is 24 hours.

You can test if DNS records for your domain is working correctly and recognized from the website MX Toolbox

Add your domain name e.g. prabinpoudel.com.np
Click on "MX Lookup"

You should see "DNS Record Published" status in the test result table

Step 11: Test incoming email with SendGrid and NGROK

Finally, we are now at the last step. We will now send email to our mail server and we should receive it in our local Rails app and server should stop in our debugging breakpoint.

From your favorite email provider e.g. Gmail, send a test email to your domain e.g. for me I will test it via sendgrid-test@prabinpoudel.com.np.

It takes some time to process the email by SendGrid and receive in our Rails app, maximum ~1 minute.

You can check if the email is being received by SendGrid or not from Parse Webhook Statistics

Tada!! 🎉

You should have received the email and rails server must have stopped in the debugging breakpoint.

Conclusion

Congratulations!!! You have come a long way and gone through a lot of process to integrate Action Mailbox with SendGrid.

You can find a working app for this blog at Action Mailbox with SendGrid. You can view all changes I made for configuring SendGrid with Action Mailbox in the PR: Setting up Action Mailbox with SendGrid

Next, you can deploy the app to staging or production and add new Inbound Parse URL in SendGrid to point to the URL of those applications.

If you have any confusions, suggestions or issues while implementing any steps in this email, please let me know in comment section below and I will do my best to help you.

Thanks for reading. Happy coding and tinkering!

References:

Image Credits: Cover Image by erica steeves from Unsplash

Run ESLint on git commit with Husky and Lint-Staged in ReactJS

Prabin Poudel — Wed, 20 Apr 2022 01:50:41 +0000

How do you make sure that ESLint rules configured in your project are followed by all your team members and code with issues are not pushed to remote Git repository?

Answer to the question is; using Husky package with git hooks. Git hooks are one of the most popular way to trigger and enforce different side effects like ESLint rules. Husky depends on git hooks to trigger ESLint rules and make sure that all issues are resolved before you or anyone on your team can commit and push new changes to Git.

Assumptions

You have basic knowledge of ReactJS
You have worked with ESLint previously and have required configuration file for ESLint in your project

What are Git Hooks?

Git hooks are a set of scripts that Git executes before or after events such as: commit, push, and receive. Git hooks are a built-in feature - you don't need to download anything for them to work.

When you initialize git in your project with git init, git hooks are also automatically added. You can find sample files for each event under the folder your_project_path/.git/hooks.

To view list of sample files for various types of hooks, you can hit the following command:

  $ ls your_project_path/.git/hooks

At Truemark, we normally use it to enforce coding standards and code quality by running ESLint before "git commit".

What is Husky?

One important thing to note for Git Hooks is, it is not version controlled, meaning whatever you add to hooks folder is only in your machine and not configured in the GIT.

So, what happens when new member in your team clones the repository?
Nothing, they will get sample files like I mentioned above.

What?

Then "How do we as a team make sure that hooks are executed for everyone?"

The answer to that is husky package.

Husky package helps you and your team to manage and configure Git hooks in your projects.

With "husky" installed in your project; after you clone the repo, you just have to hit the command npm run prepare and all hooks are configured by husky in your machine.

Husky makes git hooks much more manageable because you don't have to write scripts for hooks manually. You can just add the command you want to run e.g. run ESLint before commit inside the configuration file provided by Husky and everything else will be taken care by the package.

Install Husky

Execute the following in the command line:

  npm install husky -D

This will add the husky package to your package.json under "devDependencies":

  "devDependencies": {
    // other dependencies here,
    "husky": "^7.0.4"
  }

Enable Git Hooks in your Project with Husky

You can enable git hooks in your project by running the command provided by husky package. In your project root path run following commands:

  npm set-script prepare "husky install"
  npm run prepare

After running above commands, you should see the following inside package.json:

  "scripts": {
    // other scripts here,
    "prepare": "husky install"
  }

This will also add required hooks in your project inside the folder .git/hooks/.

It will also add configuration files for Husky under the folder .husky inside your project root. This file is used to control all git hooks configured in your project, and this is also where you will be adding configurations for running ESLint before commit.

Enable ESLint as Pre-Commit Hook with Husky

Update scripts under package.json and add the script to run ESLint:

  "scripts": {
      // other scripts here,
      "lint": "eslint ."
    }

Add a pre-commit hook to run eslint with husky by running the following command:

  npx husky add .husky/pre-commit "npm run lint"

You should see the following code inside .husky/pre-commit file now:

  #!/bin/sh
  . "$(dirname "$0")/_/husky.sh"

  npm run lint

Run ESLint on git commit

After you are done making changes to your code, try committing your code:

  git add .
  git commit -m "your commit message"

Git hooks will run ESLint before commit and throw errors if any. If it didn't throw any error, add new code with issues manually and see the hook in action 🙈

This is something similar to what you will see in case there are issues in your code:

If there are no errors then your code will be committed to git and you can push to the remote repository.

What is lint-staged?

With Husky, ESLint is run on each and every file inside the project and if you ask me if that is a good idea; I will tell you that it's a very bad idea.

Why? Because running ESLint on code that was not changed as part of the feature can lead to various unforeseen bugs.

For big projects it can take a lot of time to run eslint on each and every file inside the project. Also in old projects, it doesn't make sense to sit and fix all best practice issues instead of shipping new features.

So, how do we run ESLint only on the code that we changed?

The answer is lint-staged. It is a package that helps in running pre-commit hooks only on files that have been changed in current commit.

Install lint-staged

Run the following command to install lint-staged in the project:

  npm install lint-staged --save-dev

You should see the following in your package.json:

  "devDependencies": {
    // other dependencies here,
    "lint-staged": "^12.3.8",
  }

Run ESLint on "git commit" with Husky and lint-staged

You can configure lint-staged in separate file or inside package.json itself, since there is only one command I felt that it was not worth it to have a separate file for the configuration.

You can view all supported options here.

Add following to package.json just below scripts:

  "scripts": {
    "build": "react-scripts build",
    "eject": "react-scripts eject",
    // other scripts here,
  },
  "lint-staged": {
    "*.{js,jsx}": "eslint --fix"
  }

We have added {js,jsx} so that staged files with only these extension are run through lint. You can update this to support other extensions like ts, tsx for typescript.

Update pre-commit file to run lint-staged and remove other commands, your file should look like this:

  #!/bin/sh
  . "$(dirname "$0")/_/husky.sh"

  npx lint-staged

This will run lint-staged script which will show ESLint issues only on staged files.

To test, you can now manually add new code with issues and see issues thrown only on changed files instead of in all files inside the project as what had happened previously before configuring lint-staged.

Conclusion

With Husky package configured in your project, you will never have to worry about having to comment on issues in merge requests which could already have been detected by ESLint in local machine of developers. This way, you and your team can focus on having meaningful discussion in merge requests which leads to overall growth of the project and members in your team.

Thanks for reading. Happy coding!

Image Credits

Cover Image by WOLF Λ R T on Unsplash

References

[Solved] .rbenv/shims/ruby: Argument list too long

Prabin Poudel — Sun, 24 Oct 2021 04:09:47 +0000

I couldn't access any ruby command in the linux server where we have hosted the Rails app for one of my client. I had to access rails console to update some database records manually but I couldn't and got stuck in this issue for over 2 days.

I encountered two issues along the way:

ruby version not found
.rbenv/shims/ruby: Argument list too long

Solving the second issue solved the first issue too!

Problem

Inside the root folder of the project , ruby -v always returned that ruby 2.7.0 was not found and need to be installed. When I tried installing ruby 2.7.0 via rbenv, it said version already exists.

I tried to execute ruby -v outside of the project just to be sure that it was not a problem with ruby version. It took a really long time to process the command which returned with following error:

/home/deploy/.rbenv/libexec/rbenv-exec: line 47: /home/deploy/.rbenv/shims/ruby: Argument list too long

Google searches didn't yield any relevant results.

In the process (after searching more for around two days), I stumbled upon the rbenv issue rbenv: cannot rehash. I had never imagined these issues to be related at all. I executed the command to rehash the rbenv to see if it executes successfully instead of throwing the error mentioned in the Github issue.

Boom! same issue; rbenv could not be rehashed.

Solution as mentioned in the replies was to delete the file ".rbenv-shim", and rehash rbenv again. It worked!

Solution

Execute the command rbenv rehash. It will return the location of the file you have to delete:

rbenv: cannot rehash: /home/deploy/.rbenv/shims/.rbenv-shim exists

Here, "/home/deploy/.rbenv/shims/.rbenv-shim" is the location of the file in my machine and it could be different in your local machine.

Remove the file to resolve the issue:

rm /home/deploy/.rbenv/shims/.rbenv-shim

Now when you run the command ruby -v it will return the version of ruby currently installed in your machine.

Reason behind the issue

It happens when previous rehash of the rbenv was killed prematurely.

Quoting from reply to the Github Issue:

During the rehash process, rbenv writes out the temporary file .rbenv-shim to indicate that the rehash is in progress. Then, if a parallel rbenv rehash process tries to run at the same time, it will fail because the file already exists. This guards against race conditions in parallel rehashes.

It seems like .rbenv-shim file was never cleaned up after a rehash that ran earlier. That earlier process might have been killed prematurely and never cleaned up after itself.

Conclusion

I am not sure if solution to the issue .rbenv/shims/ruby: Argument list too long is always to delete the ".rbenv-shim" file but hey, it's worth a try.

If the solution to your problem was different than this, please let us know below in the replies and I will be sure to include it in this blog so that it helps others.

Thank you for reading. Happy coding!

References

rbenv: cannot rehash [Github]

Image Credits

Cover Image by Artem Maltsev on Unsplash

Run RuboCop on git commit with Overcommit Gem

Prabin Poudel — Thu, 26 Aug 2021 12:09:08 +0000

How do you make sure that RuboCop rules configured in your project are followed by all your team members and code with issues are not pushed to Git repository?

Answer is using Overcommit gem with git hooks. Git hooks are one of the most popular way to trigger and enforce different side effects like RuboCop rules. Overcommit gem depends on git hooks to trigger RuboCop rules and make sure that all issues are resolved before you or anyone on your team can commit and push new changes to Git.

Assumptions

You have basic knowledge of Rails
You have worked with RuboCop previously and have required configuration files for RuboCop in your project

Shameless Plug:

If you are new to RuboCop, you can read about it with setup instructions in my another article - Beginner's Guide to RuboCop in Rails

In this blog

You will be learning the following:

What are git hooks?
Why use git hooks?
What is overcommit gem?
Install overcommit gem
Enable git hooks in your project with overcommit gem
Enable RuboCop as pre-commit hook with overcommit gem
Update git hooks with overcommit gem
Run RuboCop on "git commit"

What are Git Hooks?

Git hooks are a set of scripts that Git executes before or after events such as: commit, push, and receive. Git hooks are a built-in feature - you don't need to download anything for them to work.

When you initialize git in your project with git init, git hooks are also automatically added. You can find sample files for each event under the folder your_project_path/.git/hooks.

To view list of sample files for various types of hooks, you can hit the following command:

  $ ls your_project_path/.git/hooks

Why use Git Hooks?

There are various use cases for git hooks:

Check the commit message for spelling errors
Enforce pattern for commit messages
Enforce project coding standards like RuboCop
Email/SMS team members of a new commit
Push the code to production

At Truemark, we normally use it to enforce coding standards and code quality by running RuboCop before "git commit".

What is Overcommit Gem?

One important thing to note for Git Hooks is it is not version controlled, meaning whatever you add to hooks folder is only in your machine.

So, what happens when new member in your team clones the repository?
Nothing, they will get sample files like I mentioned above.

What?

Then "How do we as a team make sure that hooks are executed for everyone?"

The answer to that is overcommit gem.

Overcommit gem helps you and your team to manage and configure Git hooks in your projects.

With overcommit gem installed in your project; after you clone the repo, you just have to hit the command overcommit --install and all hooks are configured by overcommit in your machine.

Overcommit gem makes git hooks much more manageable because you don't have to write scripts for hooks manually. You can just add the command you want to run e.g. run RuboCop before commit, inside the configuration file provided by overcommit gem and everything else will be taken care by the gem.

Install Overcommit Gem

Add the following to your Gemfile:

  group :development, :test do
    # run rubocop before commit with overcommit and much more
    gem 'overcommit', '~> 0.58.0'
  end

Enable Git Hooks in your Project with Overcommit Gem

You can enable git hooks in your project by running the command provided by overcommit gem. From your project root path:

  overcommit --install

This will add required hooks in your project inside the folder .git/hooks/

It will also add configuration file .overcommit.yml inside your project root. This file is used to control all git hooks configured in your project, and this is also where you will be adding configurations for running rubocop before commit.

Enable RuboCop as Pre-Commit Hook with Overcommit

You can remove everything inside the configuration file .overcommit.yml and add the following inside:

# Use this file to configure the Overcommit hooks you wish to use. This will
# extend the default configuration defined in:
# https://github.com/sds/overcommit/blob/master/config/default.yml
#
# At the topmost level of this YAML file is a key representing type of hook
# being run (e.g. pre-commit, commit-msg, etc.). Within each type you can
# customize each hook, such as whether to only run it on certain files (via
# `include`), whether to only display output if it fails (via `quiet`), etc.
#
# For a complete list of hooks, see:
# https://github.com/sds/overcommit/tree/master/lib/overcommit/hook
#
# For a complete list of options that you can use to customize hooks, see:
# https://github.com/sds/overcommit#configuration
#
# Uncomment the following lines to make the configuration take effect.

PreCommit:
 RuboCop:
   enabled: true
   on_warn: fail # Treat all warnings as failures
   problem_on_unmodified_line: ignore # run RuboCop only on modified code

What's with all these configuration options?

"PreCommit"

Git hooks will run "rubocop" to check issues in code when you try to commit your changes.
on_warn: fail

All warnings will be treated as failures and you will have to resolve those warnings first before your code is committed to the remote git repository.
problem_on_unmodified_line: ignore

Tell overcommit to run RuboCop only on code that were changed in this commit.

This is specially useful when you are adding overcommit in old projects and you don't want to sit fixing all the issues inside the project. This lets you fix issues that you introduced or issues you want to refactor/fix while you can fix old issues in your own time (ohh, will that time ever come?).

Update Git Hooks with Overcommit

Now that you have added configurations for running RuboCop on pre-commit, you will need to tell overcommit gem to persist these changes to "pre-commit" script inside git hooks folder.

Add changes inside overcommit.yml to git
```
  git add .
```
Update scripts inside git hooks folder with overcommit gem
```
   overcommit --sign
```

Run RuboCop on git commit

After you are done making changes to your code and done with the part of "git add", try committing your code:

  git commit -m "your commit message"

Git hooks will run RuboCop before commit and throw errors if any. If it didn't throw any error, add new code with issues manually and see the hook in action 🙈

This is something similar to what you will see in case there are issues in your code:

If there are no errors then your code will be committed to git and you can push to the remote repository.

Conclusion

With overcommit gem configured in your project, you will never have to worry about having to comment on issues in merge requests which could already have been detected by RuboCop in local machine of developers. This way, you and your team can focus on having meaningful discussion in the merge requests which leads to overall growth of the project and members in your team.

If you use any other gem for managing git hooks, I would like to hear your opinions on what you think about the gem you are using in comparison to overcommit. Personally, I find overcommit gem much easier to use and with a lot of configuration options. (Oh boy! You haven't even tried other gems)

Happy coding!

Image Credits

Cover Image by Kelly Sikkema on Unsplash

References

Beginner's Guide to RuboCop in Rails

Prabin Poudel — Wed, 18 Aug 2021 12:24:55 +0000

RuboCop is a static code analyzer which analyzes the code based on the best practices followed by the Ruby developers around the world and defined on the community Ruby style guide.

Apart from analyzing the code, it also provides us the feature of automatically formatting the code and fix warnings inside our code.

If you are coming from Javascript background, you may have heard about ESLint.

RuboCop is ESLint for Ruby

Apart from Ruby, RuboCop also provides gems for implementing rules on various extensions like Rails, Minitest, RSpec, etc.

Why RuboCop?

It begs the question and curiosity among us, so why do we actually need RuboCop? What's the use of having RuboCop in our projects.

Here are some reasons on why we would want to use RuboCop in our projects:

Clean code

We all want to write clean code that adheres to best practices followed by developers around the world. Best practices comes from experience, it may take some years to know about the language and know the anti-patterns and good patterns to follow if we only rely on ourselves.

With RuboCop, we have the advantage of not having the experience because best practices have been bundled as rules and shipped to us inside the "rubocop" gem. RuboCop throws warnings whenever we violate rules configured for best practices and after fixing these issues, our code is most of the time clean and easy to understand.
Eases the code review process

The main purpose of code review is to fix logics in the code, or fix security vulnerabilities or discuss the path we took to develop the feature.

But hey, imagine a situation where we push a code with a typo and reviewer spots that, then comments on it for fixing because obviously no one wants to ship the code with typo to production!

What's wrong with that? It takes significant time to review the code, and with typo or discussion about best practices in merge requests, we as a developer are wasting a lot of time which could have easily be solved with the help of RuboCop by configuring rules.

RuboCop makes sure that code with issues never makes it to the merge/pull requests.
Best practice is no one size fits all

Normally best practices means what we like or dislike about the code or pattern we follow when we write the code and it differs for each one of us. If we focus our energy in discussing these practices in the code for every feature, when will we ship features?

With RuboCop, we can discuss with the team on what best practices should the team follow and disable or enable rules based on the conclusion, hence making everyone happy (well, you can never make everyone happy!).

Setup RuboCop in Rails

In this article, we will be installing main 'rubocop' gem for implementing rules in Ruby code along with the extension 'rubocop-rails' for Rails specific code.

Add Gems to Gemfile

Add the following to Gemfile inside the group :development, :test

group :development, :test do
  # enforce rails best practice with rubocop
  gem 'rubocop', '~> 1.18.0', require: false
  gem 'rubocop-performance', '~> 1.11.0', require: false
  gem 'rubocop-rails', '~> 2.11.0', require: false
end

NOTE: Update gem versions based on what is latest at the time you are installing these gems in our project

We have added the following gems to our Gemfile:

rubocop: For Ruby code
rubocop-performance: For code performance related rules
rubocop-rails: For Rails specific rules

Install Gems in the Project

Install rubocop globally

gem install rubocop

This will help us in running commands provided by 'rubocop' gem like auto formatting, running rubocop in the project, etc.
Install new gems with bundle install

Add Configuration Files

To control (enable/disable) rules, we need to create configuration files for each extension. If there is no file then RuboCop will enable default extensions. I like to have configuration files because it provides flexibility to team.

Let's create configuration files for RuboCop and it's extensions:

  $ cd /path/to/our/project
  $ touch .rubocop.yml
  $ touch .rubocop-performance.yml
  $ touch .rubocop-rails.yml

Add Rules to Configuration files

I also have a blog written specifically for configuration files of RuboCop, you can find it at RuboCop Configuration Files for Rails if you want more options.

Let's update configuration files and add rules for Ruby and installed extensions.

Ruby

# .rubocop.yml

# The behavior of RuboCop can be controlled via the .rubocop.yml
# configuration file. It makes it possible to enable/disable
# certain cops (checks) and to alter their behavior if they accept
# any parameters. The file can be placed either in your home
# directory or in some project directory.
#
# RuboCop will start looking for the configuration file in the directory
# where the inspected file is and continue its way up to the root directory.
#

inherit_from:
  - '.rubocop-performance.yml'
  - '.rubocop-rails.yml'

require:
  - rubocop-performance
  - rubocop-rails

AllCops:
  TargetRubyVersion: 2.7
  TargetRailsVersion: 6.0
  Exclude:
    - '**/db/migrate/*'
    - 'db/schema.rb'
    - '**/Gemfile.lock'
    - '**/Rakefile'
    - '**/rails'
    - '**/vendor/**/*'
    - '**/spec_helper.rb'
    - 'node_modules/**/*'
    - 'bin/*'

###########################################################
###################### RuboCop ############################
###########################################################

# You can find all configuration options for rubocop here: https://docs.rubocop.org/rubocop/cops_bundler.html

###########################################################
####################### Gemspec ###########################
###########################################################

Gemspec/DateAssignment: # (new in 1.10)
  Enabled: true

###########################################################
######################## Layout ###########################
###########################################################

Layout/ClassStructure:
  ExpectedOrder:
    - module_inclusion
    - constants
    - association
    - public_attribute_macros
    - public_delegate
    - macros
    - initializer
    - public_class_methods
    - public_methods
    - protected_attribute_macros
    - protected_methods
    - private_attribute_macros
    - private_delegate
    - private_methods

Layout/EmptyLineAfterMultilineCondition:
  Enabled: true

Layout/EmptyLinesAroundAttributeAccessor:
  Enabled: true

Layout/FirstArrayElementIndentation:
  EnforcedStyle: consistent

Layout/FirstArrayElementLineBreak:
  Enabled: true

Layout/FirstHashElementIndentation:
  EnforcedStyle: consistent

Layout/FirstHashElementLineBreak:
  Enabled: true

Layout/LineEndStringConcatenationIndentation: # (new in 1.18)
  Enabled: true

Layout/LineLength:
  Max: 150
  Exclude:
    - '**/spec/**/*'

Layout/MultilineArrayBraceLayout:
  EnforcedStyle: new_line

Layout/MultilineOperationIndentation:
  EnforcedStyle: indented

Layout/MultilineHashBraceLayout:
  EnforcedStyle: new_line

Layout/MultilineHashKeyLineBreaks:
  Enabled: true

Layout/MultilineMethodCallBraceLayout:
  EnforcedStyle: new_line

Layout/MultilineMethodDefinitionBraceLayout:
  EnforcedStyle: new_line

Layout/SpaceAroundMethodCallOperator:
  Enabled: true

Layout/SpaceBeforeBrackets: # (new in 1.7)
  Enabled: true

Layout/SpaceInLambdaLiteral:
  EnforcedStyle: require_space


###########################################################
######################## Lint #############################
###########################################################

Lint/AmbiguousAssignment: # (new in 1.7)
  Enabled: true

Lint/AmbiguousBlockAssociation:
  Exclude:
    - '**/spec/**/*'

Lint/AssignmentInCondition:
  AllowSafeAssignment: false

Lint/BinaryOperatorWithIdenticalOperands:
  Enabled: true

Lint/DeprecatedConstants: # (new in 1.8)
  Enabled: true

Lint/DeprecatedOpenSSLConstant:
  Enabled: true

Lint/DuplicateBranch: # (new in 1.3)
  Enabled: true

Lint/DuplicateElsifCondition:
  Enabled: true

Lint/DuplicateRegexpCharacterClassElement: # (new in 1.1)
  Enabled: true

Lint/DuplicateRequire:
  Enabled: true

Lint/DuplicateRescueException:
  Enabled: true

Lint/EmptyBlock: # (new in 1.1)
  Enabled: true

Lint/EmptyClass: # (new in 1.3)
  Enabled: true

Lint/EmptyConditionalBody:
  Enabled: true

Lint/EmptyFile:
  Enabled: true

Lint/EmptyInPattern: # (new in 1.16)
  Enabled: true

Lint/FloatComparison:
  Enabled: true

Lint/LambdaWithoutLiteralBlock: # (new in 1.8)
  Enabled: true

Lint/MissingSuper:
  Enabled: true

Lint/MixedRegexpCaptureTypes:
  Enabled: true

Lint/NoReturnInBeginEndBlocks: # (new in 1.2)
  Enabled: true

Lint/NumberConversion:
  Enabled: true

Lint/NumberedParameterAssignment: # (new in 1.9)
  Enabled: true

Lint/OrAssignmentToConstant: # (new in 1.9)
  Enabled: true

Lint/RaiseException:
  Enabled: true

Lint/RedundantDirGlobSort: # (new in 1.8)
  Enabled: true

Lint/SelfAssignment:
  Enabled: true

Lint/SymbolConversion: # (new in 1.9)
  Enabled: true

Lint/ToEnumArguments: # (new in 1.1)
  Enabled: true

Lint/TrailingCommaInAttributeDeclaration:
  Enabled: true

Lint/TripleQuotes: # (new in 1.9)
  Enabled: true

Lint/UnexpectedBlockArity: # (new in 1.5)
  Enabled: true

Lint/UnmodifiedReduceAccumulator: # (new in 1.1)
  Enabled: true

Lint/UnusedBlockArgument:
  IgnoreEmptyBlocks: false

Lint/UnusedMethodArgument:
  IgnoreEmptyMethods: false

Lint/UselessMethodDefinition:
  Enabled: true

###########################################################
######################## Metric ###########################
###########################################################

Metrics/AbcSize:
 Max: 45

Metrics/BlockLength:
  CountComments: false
  Max: 50
  Exclude:
    - '**/spec/**/*'
    - '**/*.rake'
    - '**/factories/**/*'
    - '**/config/routes.rb'

Metrics/ClassLength:
  CountAsOne: ['array', 'hash']
  Max: 150

Metrics/CyclomaticComplexity:
  Max: 10

Metrics/MethodLength:
  CountAsOne: ['array', 'hash']
  Max: 30

Metrics/ModuleLength:
  CountAsOne: ['array', 'hash']
  Max: 250
  Exclude:
    - '**/spec/**/*'

Metrics/PerceivedComplexity:
  Max: 10

###########################################################
######################## Naming ###########################
###########################################################

Naming/InclusiveLanguage: # (new in 1.18)
  Enabled: true

###########################################################
######################## Style ############################
###########################################################

Style/AccessorGrouping:
  Enabled: true

Style/ArgumentsForwarding: # (new in 1.1)
  Enabled: true

Style/ArrayCoercion:
  Enabled: true

Style/AutoResourceCleanup:
  Enabled: true

Style/BisectedAttrAccessor:
  Enabled: true

Style/CaseLikeIf:
  Enabled: true

Style/ClassAndModuleChildren:
  Enabled: false

Style/CollectionCompact: # (new in 1.2)
  Enabled: true

Style/CollectionMethods:
  Enabled: true

Style/CombinableLoops:
  Enabled: true

Style/CommandLiteral:
  EnforcedStyle: percent_x

Style/ConstantVisibility:
  Enabled: true

Style/Documentation:
  Enabled: false

Style/DocumentDynamicEvalDefinition: # (new in 1.1)
  Enabled: true

Style/EndlessMethod: # (new in 1.8)
  Enabled: true

Style/ExplicitBlockArgument:
  Enabled: true

Style/GlobalStdStream:
  Enabled: true

Style/HashConversion: # (new in 1.10)
  Enabled: true

Style/HashEachMethods:
  Enabled: true

Style/HashExcept: # (new in 1.7)
  Enabled: true

Style/HashLikeCase:
  Enabled: true

Style/HashTransformKeys:
  Enabled: true

Style/HashTransformValues:
  Enabled: true

Style/IfWithBooleanLiteralBranches: # (new in 1.9)
  Enabled: true

Style/ImplicitRuntimeError:
  Enabled: true

Style/InlineComment:
  Enabled: true

Style/InPatternThen: # (new in 1.16)
  Enabled: true

Style/IpAddresses:
  Enabled: true

Style/KeywordParametersOrder:
  Enabled: true

Style/MethodCallWithArgsParentheses:
  Enabled: true

Style/MissingElse:
  Enabled: true

Style/MultilineInPatternThen: # (new in 1.16)
  Enabled: true

Style/MultilineMethodSignature:
  Enabled: true

Style/NegatedIfElseCondition: # (new in 1.2)
  Enabled: true

Style/NilLambda: # (new in 1.3)
  Enabled: true

Style/OptionalBooleanParameter:
  Enabled: true

Style/QuotedSymbols: # (new in 1.16)
  Enabled: true

Style/RedundantArgument: # (new in 1.4)
  Enabled: true

Style/RedundantAssignment:
  Enabled: true

Style/RedundantBegin:
  Enabled: true

Style/RedundantFetchBlock:
  Enabled: true

Style/RedundantFileExtensionInRequire:
  Enabled: true

Style/RedundantSelfAssignment:
  Enabled: true

Style/SingleArgumentDig:
  Enabled: true

Style/StringChars: # (new in 1.12)
  Enabled: true

Style/StringConcatenation:
  Enabled: true

Style/SwapValues: # (new in 1.1)
  Enabled: true

Rails

# .rubocop-rails.yml

###########################################################
#################### RuboCop Rails ########################
###########################################################

# You can find all configuration options for rubocop-rails here: https://docs.rubocop.org/rubocop-rails/cops_rails.html

Rails/ActiveRecordCallbacksOrder:
  Enabled: true

Rails/AddColumnIndex: # (new in 2.11)
  Enabled: true

Rails/AfterCommitOverride:
  Enabled: true

Rails/AttributeDefaultBlockValue: # (new in 2.9)
  Enabled: true

Rails/DefaultScope:
  Enabled: true

Rails/EagerEvaluationLogMessage: # (new in 2.11)
  Enabled: true

Rails/ExpandedDateRange: # (new in 2.11)
  Enabled: true

Rails/FindById:
  Enabled: true

Rails/I18nLocaleAssignment: # (new in 2.11)
  Enabled: true

Rails/Inquiry:
  Enabled: true

Rails/MailerName:
  Enabled: true

Rails/MatchRoute:
  Enabled: true

Rails/NegateInclude:
  Enabled: true

Rails/OrderById:
  Enabled: true

Rails/Pluck:
  Enabled: true

Rails/PluckId:
  Enabled: true

Rails/PluckInWhere:
  Enabled: true

Rails/RenderInline:
  Enabled: true

Rails/RenderPlainText:
  Enabled: true

Rails/SaveBang:
  Enabled: true
  AllowImplicitReturn: false

Rails/ShortI18n:
  Enabled: true

Rails/SquishedSQLHeredocs: # (new in 2.8)
  Enabled: true

Rails/TimeZoneAssignment: # (new in 2.10)
  Enabled: true

Rails/UnusedIgnoredColumns: # (new in 2.11)
  Enabled: true

Rails/WhereEquals: # (new in 2.9)
  Enabled: true

Rails/WhereExists:
  Enabled: true

Rails/WhereNot:
  Enabled: true

Performance

.rubocop-performance.yml

###########################################################
#################### RuboCop Performance ##################
###########################################################

# You can find all configuration options for rubocop-performance here: https://docs.rubocop.org/rubocop-performance/

Performance/AncestorsInclude: # (new in 1.7)
  Enabled: true

Performance/BigDecimalWithNumericArgument: # (new in 1.7)
  Enabled: true

Performance/BlockGivenWithExplicitBlock: # (new in 1.9)
  Enabled: true

Performance/CollectionLiteralInLoop: # (new in 1.8)
  Enabled: true

Performance/ConstantRegexp: # (new in 1.9)
  Enabled: true

Performance/MapCompact: # (new in 1.11)
  Enabled: true

Performance/MethodObjectAsBlock: # (new in 1.9)
  Enabled: true

Performance/RedundantEqualityComparisonBlock: # (new in 1.10)
  Enabled: true

Performance/RedundantSortBlock: # (new in 1.7)
  Enabled: true

Performance/RedundantSplitRegexpArgument: # (new in 1.10)
  Enabled: true

Performance/RedundantStringChars: # (new in 1.7)
  Enabled: true

Performance/ReverseFirst: # (new in 1.7)
  Enabled: true

Performance/SortReverse: # (new in 1.7)
  Enabled: true

Performance/Squeeze: # (new in 1.7)
  Enabled: true

Performance/StringInclude: # (new in 1.7)
  Enabled: true

Performance/Sum: # (new in 1.8)
  Enabled: true

Run RuboCop

We have the option to run RuboCop on

Whole project
Files inside single folder
Only on single file

After running commands of RuboCop in the command line, we will be presented with issues found in our code inside the project, which we can then fix manually or also have option to auto correct issues in most cases.

Whole Project

  $ cd /path/to/your/project
  $ rubocop

Files inside single folder

  $ rubocop app

Single file

  $ rubocop app/models/user.rb

Auto fix warnings

RubCop also provides the feature of auto correcting issues in our code.

There are a couple of things to keep in mind about auto-correct:

For some offenses, it is not possible to implement automatic correction.
Some automatic corrections that are possible have not been implemented yet.
Some automatic corrections might change (slightly) the semantics of the code, meaning they’d produce code that’s mostly equivalent to the original code, but not 100% equivalent. We call such auto-correct behavior "unsafe"

We can run auto correction with the following command:

$ rubocop -a
# or
$ rubocop --auto-correct
# or
$ rubocop -A
# or
$ rubocop --auto-correct-all

Other RuboCop Extensions

RuboCop also has options for implementing rules on other extensions like:

rubocop-rspec For Rspec; a test framework popular for testing Rails code
rubocop-rake: A RuboCop plugin for Rake
rubocop-minitest: Another popular testing library for testing Ruby and Rails code

Style Guide

RuboCop is based on style guides which helps in maintaining best practices for each extension. If you are curious, you can view and read guidelines from links below:

Conclusion

RuboCop is very helpful in maintaining best practices and it's one of the gem that we include in all our project setup here at Truemark.

One thing to remember with Static Code Analyzers is we have the flexibility to enable and disable rules, hence we should always discuss with the team what to include, why to include and what to disable.

This is the guide I hope I had when I was starting our as a Rails developer. I hope you find it useful!

Happy coding!

Image Credits

Cover Image by Scott Webb on Unsplash

DEV Community: Prabin Poudel

Setup RSpec Tests in Rails with Gitlab CI

Assumption

Tested and working in

Configure Gitlab CI Variables

Add a variable for storing database.yml file content

Add a variable for storing environment variables

Require selenium-webdriver

Configure Capybara with Selenium

Explanation

Custom Selenium Chrome driver

Options

Selenium remote URL

Capybara server and app host

Commands to run tests

Update .gitlab-ci.yml to run all tests

Explanation

cache

stages

.base_db

unit_and_integration_tests

system_tests

Final .gitlab-ci.yml

Conclusion

Devise raise validations error when new and old passwords are same

Skills required to follow the tutorial

You should have

Validation Implementation

Code Explanation

Conclusion

References

Image Credits

Run Cron Job Manually

Steps

Step 1: Get PATHs your normal Cron Job uses

Step 2: Create a bash script to run any command similar to Cron Job

Step 3: Run any command you want by replicating the Cron behavior

Bonus: Run Ruby on Rails tasks replicating the Cron behavior

Conclusion

References

Image Credits

[Solved] Ubuntu 22 Temporary failure in name resolution

Background

When does this error occur?

Fix

Conclusion

References:

Image Credits

Setup Active Job with Sidekiq in Rails

Skills required to follow the tutorial

You should have

Step 1: Install Capistrano Gems

Step 2: Enable Sidekiq

Step 3: Install Redis

On MacOS via Homebrew

Linux

Automatically start on machine restart

Windows

Step 4: Enable Web UI to Monitor Jobs

Step 6: Accessing the Sidekiq Web UI

Step 6: Add Basic Authentication to Web UI for Preventing Unauthorized Access

Example Job

Application Job

Expire User Note Job

Bonus: Setup Sidekiq in Ubuntu Server

Setup

Run Sidekiq

Sidekiq with Capistrano

Conclusion

Image Credits

References

Deploy API only Rails App with Capistrano

Skills required to follow the tutorial

You should have

Step 1: Install Capistrano Gems

Step 2: Generate default configuration files

Step 3: Require correct plugins in Capfile

Step 4: Update Deploy file

Step 5: Update environment specific deploy files

Step 6: Upload secret keys and files

Update `.gitlab-ci.yml` to run all tests

Final `.gitlab-ci.yml`