DEV Community: Artur Trzop

DB max connection limits for Rails app and Postgres, Redis, Puma settings

Artur Trzop — Tue, 11 May 2021 14:04:15 +0000

Configuring the database connections pool for the Rails app might not be a straightforward task for many programmers. There is a constraint of max opened connections on a database level. Your server environment configuration can change in time and affect the number of connections to the database required. For instance number of servers you use can change when you autoscale it based on the web traffic. It means that the number of web processes/threads running for Puma or Unicorn servers could change. All this adds additional complexity. When you use two databases (e.g. Postgres + Redis), everything gets more complex. In this article, we will address that. You will learn how to estimate needed database connections for your Ruby on Rails production application.

Why do available database connections matter?

The first question is, why do you need to care about available database connections? The answer is simple. Suppose you configured your Ruby application to open too many DB connections. In that case, it could happen that you will get ActiveRecord::ConnectionTimeoutError exceptions from the application when the database cannot handle more new connections from your Rails app. It can result in 500 errors visible for your web app users.

This problem might not be apparent immediately. Often you will find out about it in production. Your application might work just fine until specific circumstances cause the Rails app to need more DB connections, which can trigger exception flood. Let's see how to avoid it.

RoR application configuration step by step

Let's break a typical Ruby on Rails application down into smaller components that use databases.

We have a Rails application that uses the Postgres database for ActiveRecord usage.
We also use the Redis database for background workers like Sidekiq.

It looks simple, isn't it? Let's start with that, and later on, we will add more complexity to the mix :)

Postgres database connections - how to check the limit?

How to check how many available connections do you have for Postgres?

If you use a dedicated server with Postgres installed, then most likely you have a default max_connections which is typically 100 connections.
If you use a Postgres instance on the AWS, then you need to check the AWS documentation to find out what's the max allowed connections to your database instance (it depends on if you use Amazon RDS or Aurora and what is server instance class)
If you use Heroku, you can check the Connection Limit for the Postgres Heroku add-on to check max acceptable connections.

ActiveRecord connection pool

In your Rails application, the config/database.yml file contains the pool option. As explained in the Rails docs:

Active Record database connections are managed by ActiveRecord::ConnectionAdapters::ConnectionPool, which ensures that a connection pool synchronizes the amount of thread access to a limited number of database connections.

Since the connection pooling is handled inside of Active Record by default, all application servers (Thin, Puma, Unicorn, etc.) should behave the same. The database connection pool is initially empty. As demand for connections increases, it will create them until it reaches the connection pool limit.

Any one request will check out a connection the first time it requires access to the database. At the end of the request, it will check the connection back in. This means that the additional connection slot will be available again for the next request in the queue.

The pool can be defined this way:

production:
  adapter: postgresql
  database: blog_production
  pool: 5

or as a part of a URL to the database:

development:
  url: postgresql://localhost/blog_production?pool=5

The URL option is popular when you host a database on an external server like Amazon RDS. Then you could define the URL this way:

production:
  url: postgres://blog_production:PASSWORD@blog-production.abcdefgh.eu-west-1.rds.amazonaws.com/blog_production?sslca=config/rds-combined-ca-bundle.pem&pool=5

Please note that for the production, you should not commit credentials in the config/database.yml file. Instead, store it in environment variables and then read the value at your Rails app's runtime.

production:
  url: <%= ENV['DB_URL'] %>

How does ActiveRecord connection pool affects Postgres max connections?

Let's start with a simple example. Your application may use one of the application servers like Puma or Unicorn. Let's focus on Puma because it's more complex as it has a separate configuration for several processes (known as workers in Puma terms) and threads. Unicorn runs in a single thread only. It works exactly like Puma with a single thread setting.

Puma config: 1 process and 1 thread

Let's say you use the Puma server to run the Rails application. The Puma is configured to run 1 process (worker) and it has only 1 thread.

The puma process can open up to 5 connections to the database because the pool option is defined as 5 in config/database.yml. Typically, there are fewer connections than that because when you run 1 process and only 1 thread, only 1 connection to the Postgres database will be needed to make a database query.

Sometimes the database connection might be dead. In such a case, ActiveRecord can open a new connection, and then you may end up with 2 active connections. In the worst-case scenario when 4 connections would be dead, then Rails can open 5 connections max.

Puma config: 1 process and 2 threads

If you use 2 threads in a single Puma process (worker) then it means those 2 threads can use the same pool of DB connections within the Puma process.

It means that 2 DB connections will be open out of 5 possible. If any connection is dead, then more connections can be opened until the 5 connection pool limit is reached.

Puma config: 2 processes and 2 threads per process

If you run 2 Puma processes (workers) and each process has 2 threads then it means that each single process will open 2 DB connections because you have 2 threads per process. You have 2 processes so it means at the start of your application, there might be 4 DB connections open. Each process has its pool, so you have 2 pools. Each pool can open up to 5 DB connections. It means that in the worst-case scenario, there can be even 10 connections created to the database.

Assuming you use 2 threads per Puma process, it's good to have the pool option set to 2 + some spare connections. It allows ActiveRecord to open a new connection if one of the DB connections is dead.

Puma config: 2 processes and 2 threads, and 2 web dynos on Heroku

If you use Heroku to host your application, it allows scaling your web application horizontally by adding more servers (dynos). Assume you run your application on 2 servers (2 Heroku dynos), each dyno is running 2 Puma processes, and each process has 2 threads. It means at the start, your application may open 6 connections to the database. Here is why:

2 dynos X 2 Puma processes X 2 Puma threads = 6 DB connections

2 dynos X 2 Puma process X Pool size (5) = Total pool size 20

It means that in the worst-case your application may open 20 DB connections.

Autoscaling web application

If you autoscale your web servers by adding more servers during the peak web traffic, you need to be careful. Ensure your application stays within the Postgres max connections limit. The above example shows you how to calculate expected opened DB connections and the worst-case scenario. Please adjust your pool size to ensure that you will be below the max connections limit for your database engine in the worst-case scenario.

What else can open DB connections?

We just talked about a webserver like Puma that can open connections and consume your max DB connections limit. But other non-web processes can do it as well:

You run Rails console on production in a Heroku dyno heroku run bin/rails console --app=my-app-name. It runs an instance of your Rails app, and 1 DB connection will be open. In the worst-case scenario, the number of connections defined in the pool can be opened. But it's unlikely that your DB connections would go dead. So the whole pool limit shouldn't be used.
You run scheduled rake tasks via Heroku Scheduler (cron-like tool). If the rake tasks are performed periodically, they need to open a connection to the DB so that at least 1 DB connection is used from the pool per rake task. Imagine you have 10 rake tasks that are started every hour. It means you need 10 available DB connections every hour. It can be easy to miss this if you base your estimation on just the web connections.
You use background workers like Sidekiq to perform async jobs. Your jobs may open DB connections. We will talk about it later.

Background worker - Sidekiq and ActiveRecord pool

Sidekiq process will use the pool defined in config/database.yml similarly as Puma. All Sidekiq threads in a Sidekiq process can use a common pool of connections.

If you run multiple servers (Heroku dynos), then it works similarly to the Puma example.

2 servers (dynos) X 1 Sidekiq process X 10 Sidekiq threads = 20 DB connections will be open.

You need to have a pool size of at least 10 in config/database.yml because Sidekiq by default, uses 10 threads.

If you use a pool size lower than 10 then Sidekiq threads will be fighting for access to limited connections in the pool. It could be fine for some time, but you should be aware that this can increase your job's processing time because not all Sidekiq threads will use DB connections in parallel. It can also lead to a problem described here.

Sidekiq and Redis database connections

Sidekiq uses the Redis database to store async jobs. It would be best if you calculate DB connections to Redis as well as Postgres connections. A Sidekiq server process requires at least (concurrency + 5) connections. The concurrency option is the number of Sidekiq threads per Sidekiq process.

Using the previous example:

2 servers (dynos) X 1 Sidekiq process X 10 Sidekiq threads = 2 servers (dynos) X 1 Sidekiq process X (10 + 5) = 30 Redis connections required.

Redis database connections

If you use Redis for processing background jobs, then it's not just the Sidekiq process that is using Redis connections. Your Puma process and threads can use Redis to add new jobs to the Sidekiq queue as well. Typically you will have 1 Redis connection per 1 Puma thread.

If you explicitly open a new Redis connection with Redis.new, this can create a new connection per the Puma thread as well.

Summary

We covered a few examples of Postgres and Redis on calculating DB connections needed by your Rails application. I hope this will give you a better understanding of how to estimate how many DB connections you need on your database level to serve your application's demands properly.

If you are looking to improve your Rails application workflow please consider checking how to run automated tests in parallel on your CI server with Knapsack Pro.

BitBucket parallel Cypress tests configuration for CI pipeline integration

Artur Trzop — Thu, 29 Apr 2021 09:36:42 +0000

Do you use BitBucket Pipeline as your CI server? Are you struggling with slow E2E tests in Cypress? Did you know BitBucket Pipeline can run parallel steps? You can use it to distribute your browser tests across several parallel steps to execute end-to-end Cypress tests in a short amount of time.

How to run tests in parallel

Distributing tests across parallel steps to spread the workload and run tests faster might be more challenging than you think. The question is how to divide Cypress test files across the parallel jobs in order to ensure the work is distributed evenly? But... is distributing work evenly what you actually want?

To get the shortest CI build time you want to utilize the available CI resources to the fullest. You want to avoid wasting time. This means that you want to ensure the parallel steps will finish work at a similar time as this would mean there are no bottlenecks in CI machines utilization.

Many things are unknown and unpredictable. This can affect how long it will take to execute tests on BitBucket Pipeline. There are things like:

boot time - time spent on loading your CI docker container
loading npm/yarn dependencies from cache
running Cypress tests
tests can run against different browsers and this can affect how long the tests are executed
sometimes tests can fail and their execution time is different
other times you may have flaky tests randomly failing and you could use Test Retries in Cypress to automatically rerun failed test cases. This results in running a test file for longer.

All of the above contribute to the uncertainty around execution time. It's hard to know how best to divide test files across the parallel steps to ensure the steps complete work at a similar time. But there is a solution to that - a dynamic test suite split during runtime.

Queue Mode - a dynamic tests split

To distribute tests work across BitBucket Pipeline parallel steps you can use Knapsack Pro with a Queue Mode. You can use @knapsack-pro/cypress npm package that will generate a Queue with a list of test files on the Knapsack Pro API side and then all parallel steps can connect to the queue to consume test files and execute them. This way parallel steps ask for more tests only after they finish executing a set of tests previously fetched from the Knapsack Pro API. You can learn about the details of Queue Mode from the article.

BitBucket Pipeline YML config

Here is an example of a BitBucket Pipeline config in YML. As you can see, there are 3 parallel steps to run Cypress tests via Knapsack Pro. If you would like to run your tests on more parallel jobs you simply need to add more steps.

image: cypress/base:10
options:
  max-time: 30

# job definition for running E2E tests in parallel with KnapsackPro.com
e2e: &e2e
  name: Run E2E tests with @knapsack-pro/cypress
  caches:
    - node
    - cypress
  script:
    # run web application in the background
    - npm run start:ci &
    # env vars from https://support.atlassian.com/bitbucket-cloud/docs/variables-and-secrets/
    - export KNAPSACK_PRO_CI_NODE_BUILD_ID=$BITBUCKET_BUILD_NUMBER
    - export KNAPSACK_PRO_COMMIT_HASH=$BITBUCKET_COMMIT
    - export KNAPSACK_PRO_BRANCH=$BITBUCKET_BRANCH
    - export KNAPSACK_PRO_CI_NODE_TOTAL=$BITBUCKET_PARALLEL_STEP
    - export KNAPSACK_PRO_CI_NODE_INDEX=$BITBUCKET_PARALLEL_STEP_COUNT
    # https://github.com/KnapsackPro/knapsack-pro-cypress#configuration-steps
    - export KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true
    - $(npm bin)/knapsack-pro-cypress
  artifacts:
    # store any generated images and videos as artifacts
    - cypress/screenshots/**
    - cypress/videos/**

pipelines:
  default:
  - step:
      name: Install dependencies
      caches:
        - npm
        - cypress
        - node
      script:
        - npm ci
  - parallel:
    # run N steps in parallel
    - step:
        <<: *e2e
    - step:
        <<: *e2e
    - step:
        <<: *e2e

definitions:
  caches:
    npm: $HOME/.npm
    cypress: $HOME/.cache/Cypress

If you are looking for an example with a custom docker container for a parallel step please see this one.

Please remember to add your API token in the KNAPSACK_PRO_TEST_SUITE_TOKEN_CYPRESS environment variable as a secure variable.

Summary

BitBucket Pipeline is a CI server that allows running scripts in parallel. You can use parallel steps to distribute your Cypress tests with Knapsack Pro to save time and run CI build as fast as possible.

Testing Ruby on Rails on Github Actions with RSpec

Artur Trzop — Mon, 19 Apr 2021 11:40:10 +0000

Are you thinking about migrating a Ruby on Rails project CI pipeline to Github Actions? You will learn how to configure the Rails app to run RSpec tests using Github Actions.

This article covers a few things for Github Actions YAML config:

how to use ruby/setup-ruby action to install Ruby gems with bundler and automatically cache gems. This way you can load Ruby gems for your project from the cache and run CI build fast.
how to use Postgres on Github Actions
how to use Redis on Github Actions
how to use Github Actions build matrix to run parallel jobs and execute RSpec tests spread across multiple jobs to save time

Github Actions YML config for Rails application

ruby/setup-ruby action

ruby/setup-ruby is an action that you can use to install a particular Ruby programming language version. It allows you to cache Ruby gems based on your Gemfile.lock out of the box.

- name: Set up Ruby
  uses: ruby/setup-ruby@v1
  with:
    # Not needed with a .ruby-version file
    ruby-version: 2.7
    # runs 'bundle install' and caches installed gems automatically
    bundler-cache: true

How to configure Postgres on Github Actions

To use Postgres on Github Actions you need to set up a service for Postgres. I recommend using additional options that will configure Postgres to use RAM instead of disk. This way your database can run faster in a testing environment.

In the config below, we also pass the settings for doing a health check to ensure the database is up and running before you start running tests.

# If you need DB like PostgreSQL, Redis then define service below.
# https://github.com/actions/example-services/tree/master/.github/workflows
services:
  postgres:
    image: postgres:10.8
    env:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: ""
      POSTGRES_DB: postgres
    ports:
      - 5432:5432
    # needed because the postgres container does not provide a healthcheck
    # tmpfs makes DB faster by using RAM
    options: >-
      --mount type=tmpfs,destination=/var/lib/postgresql/data
      --health-cmd pg_isready
      --health-interval 10s
      --health-timeout 5s
      --health-retries 5
{% endhighlight %}

### How to configure Redis on Github Actions

You can use Redis Docker container to start Redis server on Github Actions. See how simple it is:

{% highlight yml %}
services:
  redis:
    image: redis
    ports:
      - 6379:6379
    options: --entrypoint redis-server

How to use Github Actions build matrix to run tests with parallel jobs

You can use the build matrix in Github Actions to run multiple jobs at the same time.

You will need to split test files between these parallel jobs. For that, you can use Knapsack Pro with Queue Mode to distribute tests evenly between the jobs. This way you can ensure the proper amount of tests is executed on each job and the workload is well balanced between the jobs. Simply speaking this way you can make sure the CI build is as fast as possible - it has optimal execution time.

- name: Run tests
  env:
    KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC: ${{ secrets.KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC }}
    KNAPSACK_PRO_CI_NODE_TOTAL: ${{ matrix.ci_node_total }}
    KNAPSACK_PRO_CI_NODE_INDEX: ${{ matrix.ci_node_index }}
    KNAPSACK_PRO_FIXED_QUEUE_SPLIT: true
    KNAPSACK_PRO_RSPEC_SPLIT_BY_TEST_EXAMPLES: true
    KNAPSACK_PRO_LOG_LEVEL: info
  run: |
    bundle exec rake knapsack_pro:queue:rspec

You can see that for RSpec we also use a knapsack_pro Ruby gem flag KNAPSACK_PRO_RSPEC_SPLIT_BY_TEST_EXAMPLES. It allows to automatically detect slow test files and split them between parallel jobs.

You can learn more about it in a separate article explaining how to run slow RSpec files on Github Actions with parallel jobs by doing an auto split of the spec file by test examples.

Full YML config for Github Actions and Ruby on Rails project

Here is the full configuration of the CI pipeline for Github Actions. You can use it to run tests for your Rails project.

name: Main

on: [push]

jobs:
  test:
    runs-on: ubuntu-latest

    # If you need DB like PostgreSQL, Redis then define service below.
    # https://github.com/actions/example-services/tree/master/.github/workflows
    services:
      postgres:
        image: postgres:10.8
        env:
          POSTGRES_USER: postgres
          POSTGRES_PASSWORD: ""
          POSTGRES_DB: postgres
        ports:
          - 5432:5432
        # needed because the postgres container does not provide a healthcheck
        # tmpfs makes DB faster by using RAM
        options: >-
          --mount type=tmpfs,destination=/var/lib/postgresql/data
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
      redis:
        image: redis
        ports:
          - 6379:6379
        options: --entrypoint redis-server

    # https://help.github.com/en/articles/workflow-syntax-for-github-actions#jobsjob_idstrategymatrix
    strategy:
      fail-fast: false
      matrix:
        # [n] - where the n is a number of parallel jobs you want to run your tests on.
        # Use a higher number if you have slow tests to split them between more parallel jobs.
        # Remember to update the value of the `ci_node_index` below to (0..n-1).
        ci_node_total: [8]
        # Indexes for parallel jobs (starting from zero).
        # E.g. use [0, 1] for 2 parallel jobs, [0, 1, 2] for 3 parallel jobs, etc.
        ci_node_index: [0, 1, 2, 3, 4, 5, 6, 7]

    env:
      RAILS_ENV: test
      GEMFILE_RUBY_VERSION: 2.7.2
      PGHOST: localhost
      PGUSER: postgres
      # Rails verifies the time zone in DB is the same as the time zone of the Rails app
      TZ: "Europe/Warsaw"

    steps:
      - uses: actions/checkout@v2

      - name: Set up Ruby
        uses: ruby/setup-ruby@v1
        with:
          # Not needed with a .ruby-version file
          ruby-version: 2.7
          # runs 'bundle install' and caches installed gems automatically
          bundler-cache: true

      - name: Create DB
        run: |
          bin/rails db:prepare
      - name: Run tests
        env:
          KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC: ${{ secrets.KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC }}
          KNAPSACK_PRO_CI_NODE_TOTAL: ${{ matrix.ci_node_total }}
          KNAPSACK_PRO_CI_NODE_INDEX: ${{ matrix.ci_node_index }}
          KNAPSACK_PRO_LOG_LEVEL: info
          # if you use Knapsack Pro Queue Mode you must set below env variable
          # to be able to retry CI build and run previously recorded tests
          # https://github.com/KnapsackPro/knapsack_pro-ruby#knapsack_pro_fixed_queue_split-remember-queue-split-on-retry-ci-node
          KNAPSACK_PRO_FIXED_QUEUE_SPLIT: true
          # RSpec split test files by test examples feature - it's optional
          # https://knapsackpro.com/faq/question/how-to-split-slow-rspec-test-files-by-test-examples-by-individual-it
          KNAPSACK_PRO_RSPEC_SPLIT_BY_TEST_EXAMPLES: true
        run: |
          bundle exec rake knapsack_pro:queue:rspec

Summary

You've just learned how to set up your Rails application on Github Actions. I hope this will help you if you migrate your project from a different CI server to Github Actions.

You can learn more about Knapsack Pro and how it can help you run tests fast using parallel jobs on CI. It works with RSpec, Cucumber, Minitest, and other Ruby test runners. Knapsack Pro can also work with JavaScript test runners and has a native API integration.

How to run Minitest parallel tests on Github Actions

Artur Trzop — Tue, 13 Apr 2021 10:31:17 +0000

How to run Ruby on Rails tests in Minitest on Github Actions? What to do if tests are slow? How to manage complex workflows? You can use Github Actions build matrices to divide Minitest files between jobs and run the test suite much faster.

If your Minitest tests are taking dozens of minutes and you would like to save some time for your Ruby engineering team then you could use tests parallelization on your CI server.

To run tests as fast as possible you need to split them into equal buckets (into parallel jobs). But how to do it? Some of the test files can be super fast to execute, other Minitest files can take minutes if they run system tests (E2E tests).

There is also an aspect of preparing the test environment for each parallel job. By preparing I mean you need to clone a repository, install ruby gems or load them from a cache, maybe you need to load some docker container, etc. This can take various amounts of time on each parallel job. Random network errors happen like network delay to load cached gems, or maybe Github Actions from time to time will start one of your jobs late compared to others. It's an inevitable issue in the network environment and can cause your tests to run for a different amount of time on each parallel job. This is visible on the graph below and it causes the CI build to be slower.

In a perfect scenario you would like to cover all these problems and no matter what still be able to split Minitest work in parallel jobs in a way that ensures the tests on each parallel job completes at a similar time. This guarantees no bottlenecks. The perfect tests split is on the below graph.

Split tests in a dynamic way with Queue Mode

You can use Knapsack Pro Queue Mode to split tests in a dynamic way between parallel jobs. This way each job consumes tests from a queue until the queue is empty. Simply speaking this allows you to utilize your CI server resources efficiently and run tests in optimal time.

I described how Queue Mode splits Ruby and JavaScript tests in parallel with a dynamic test suite split. You can learn from that article about it.

Github Actions build matrix to run parallel tests

Github Actions has a build matrix feature that allows running many jobs at the same time. You can use it to run your Minitest tests between parallel jobs.

Below is a full Github Actions YML config for a Rails project and Minitest.
The tests are split with knapsack_pro Ruby gem and Queue Mode.

name: Main

on: [push]

jobs:
  test:
    runs-on: ubuntu-latest

    # If you need DB like PostgreSQL, Redis then define service below.
    # https://github.com/actions/example-services/tree/master/.github/workflows
    services:
      postgres:
        image: postgres:10.8
        env:
          POSTGRES_USER: postgres
          POSTGRES_PASSWORD: ""
          POSTGRES_DB: postgres
        ports:
          - 5432:5432
        # needed because the postgres container does not provide a healthcheck
        # tmpfs makes DB faster by using RAM
        options: >-
          --mount type=tmpfs,destination=/var/lib/postgresql/data
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
      redis:
        image: redis
        ports:
          - 6379:6379
        options: --entrypoint redis-server

    # https://help.github.com/en/articles/workflow-syntax-for-github-actions#jobsjob_idstrategymatrix
    strategy:
      fail-fast: false
      matrix:
        # Set N number of parallel jobs you want to run tests on.
        # Use higher number if you have slow tests to split them on more parallel jobs.
        # Remember to update ci_node_index below to 0..N-1
        ci_node_total: [8]
        # set N-1 indexes for parallel jobs
        # When you run 2 parallel jobs then first job will have index 0, the second job will have index 1 etc
        ci_node_index: [0, 1, 2, 3, 4, 5, 6, 7]

    env:
      RAILS_ENV: test
      PGHOST: localhost
      PGUSER: postgres
      # Rails verifies Time Zone in DB is the same as time zone of the Rails app
      TZ: "Europe/Warsaw"

    steps:
      - uses: actions/checkout@v2

      - name: Set up Ruby
        uses: ruby/setup-ruby@v1
        with:
          # Not needed with a .ruby-version file
          ruby-version: 2.7
          # runs 'bundle install' and caches installed gems automatically
          bundler-cache: true

      - name: Create DB
        run: |
          bin/rails db:prepare
      - name: Run tests
        env:
          KNAPSACK_PRO_TEST_SUITE_TOKEN_MINITEST: ${{ secrets.KNAPSACK_PRO_TEST_SUITE_TOKEN_MINITEST }}
          KNAPSACK_PRO_CI_NODE_TOTAL: ${{ matrix.ci_node_total }}
          KNAPSACK_PRO_CI_NODE_INDEX: ${{ matrix.ci_node_index }}
          KNAPSACK_PRO_FIXED_QUEUE_SPLIT: true
          KNAPSACK_PRO_LOG_LEVEL: info
        run: |
          bundle exec rake knapsack_pro:queue:minitest

Summary

As you can see the slow Minitest test suite doesn't need to be an issue for you. QA, Testers, or Automation Engineers could benefit from improving the CI build speed and allowing their software developers team to deliver products faster. You can learn more at Knapsack Pro.

ruby/setup-ruby or actions/cache - caching Ruby gems on Github Actions

Artur Trzop — Tue, 06 Apr 2021 18:51:25 +0000

How to start CI build faster by loading Ruby gems from cache on Github Actions? You can start running your tests for a Ruby on Rails project quicker if you manage to set up all dependencies in a short amount of time. Caching can be helpful with that. Ruby gems needed for your project can be cached by Github Actions and thanks to that they can be loaded much faster when you run a new CI build.

You will learn how to configure Github Actions using:

actions/cache - it's a popular solution to cache Ruby gems.
ruby/setup-ruby - it's a solution to install a specific Ruby version and cache Ruby gems with bundler. Two features in one action.

actions/cache - just cache dependencies

Actions/cache is a popular solution that can be used to save data into the cache and restore it during the next CI build. It's often used for Ruby on Rails projects that also use actions/setup-ruby for managing the Ruby version on Github Actions.

Let's look at the Github Actions caching config example using actions/cache.

# .github/workflows/main.yml
name: Main
on: [push, pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2

      - uses: actions/cache@v2
        with:
          path: vendor/bundle
          key: ${{ runner.os }}-gems-${{ hashFiles('**/Gemfile.lock') }}
          restore-keys: |
            ${{ runner.os }}-gems-

      - name: Bundle install
        env:
          RAILS_ENV: test
        run: |
          bundle config path vendor/bundle
          bundle install --jobs 4 --retry 3

You need to specify a directory path that will be cached. It's vendor/bundle in our case.
You also generate a unique cache key based on the OS version and Gemfile.lock file. When you change the operating system version or you install a new gem and Gemfile.lock changes then as a result the new key value will be generated.
You need to configure the bundler to install all your Ruby gems to the directory vendor/bundle.
You can use bundler options:
- --jobs 4 - install gems using parallel workers. This allows faster gems installation.
- --retry 3 - makes 3 attempts to connect to Rubygems if there is a network issue (for instance temporary downtime of Rubygems.org)

If you would like to see the full YAML config for the Github Actions and Rails project you can take a look at some of our articles:

ruby/setup-ruby - install Ruby and cache gems

In the previous section, we mentioned the actions/setup-ruby is often used with Ruby on Rails projects. The actions/setup-ruby has been deprecated so it's recommended to use ruby/setup-ruby action nowadays. It already has caching feature that you could use. Let's see how.

# .github/workflows/main.yml
name: Main
on: [push, pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2

    - uses: ruby/setup-ruby@v1
      with:
        # Not needed with a .ruby-version file
        ruby-version: 2.7
        # runs 'bundle install' and caches installed gems automatically
        bundler-cache: true

    # run RSpec tests
    - run: bundle exec rspec

As you can see using ruby/setup-ruby for managing the Ruby version and gems caching is much simpler. You just add an option bundler-cache: true and that's it.

You can read in ruby/setup-ruby documentation:

"It is also possible to cache gems manually, but this is not recommended because it is verbose and very difficult to do correctly. There are many concerns which means using actions/cache is never enough for caching gems (e.g., incomplete cache key, cleaning old gems when restoring from another key, correctly hashing the lockfile if not checked in, OS versions, ABI compatibility for ruby-head, etc). So, please use bundler-cache: true instead..."

Summary

You saw 2 ways of caching Ruby gems on Github Actions. There are also other ways to make your CI build faster like running tests in parallel. You can learn more about test parallelisation here or simply check the Knapsack Pro homepage.

Parallel scaling RSpec tests on Buildkite to increase CI build speed

Artur Trzop — Wed, 31 Mar 2021 13:53:45 +0000

If your RSpec test suite runs for hours, you could shorten that to just minutes with parallel jobs using Buildkite agents. You will learn how to run parallel tests in optimal CI build time for your Ruby on Rails project. I will also show you a few useful things for Buildkite CI like:

A real RSpec test suite taking 13 hours and 32 minutes executed in only 5 minutes 20 seconds by using 151 parallel Buildkite agents with knapsack_pro Ruby gem.
How to distribute test files between parallel jobs using Queue Mode in Knapsack Pro to utilize CI machines optimally.
A simple example of CI Buildkite parallelism config.
An advanced example of Buildkite config with Elastic CI Stack for AWS.
Why you might want to use AWS Spot Instances
How to automatically split slow RSpec test files by test examples (test cases) between parallel Buildkite agents

A real RSpec test suite taking 13 hours and executed in only 5 minutes

I'd like to show you the results of a real project for running RSpec parallel tests. The project we are looking at here is huge and its RSpec tests run time is 13 hours and 32 minutes. It's super slow. You can imagine creating a git commit and waiting 13 hours to find out the next day that your code breaks something else in the project. You can't work like that!

The solution for this is to run tests in parallel on many CI machines using Buildkite agents. Each CI machine has a Buildkite agent installed that will run a chunk of the RSpec test suite. Below you can see an example of running ~13 hours test suite across 151 parallel Buildkite agents.
This allows running the whole RSpec test suite in just 5 minutes 20 seconds!

The above graph comes from the Knapsack Pro user dashboard. 151 parallel jobs are a lot of machines. It would take the whole screen to show you 151 bars. You can only see the last few bars on the graph. The bars are showing how the RSpec test files were split between parallel machines.

You can see that each parallel machine finishes work at a similar time. The right edges of all of the bars are pretty close to each other. This is the important part. You want to ensure the RSpec work is distributed evenly between parallel jobs. This way you can avoid a bottleneck - a slow job running too many test files. I'll show you how to do it.

How to distribute test files between parallel jobs using Queue Mode in Knapsack Pro to utilize CI machines optimally

To run CI build as fast as possible we need to utilize our available resources as much as we can. This means the work of running RSpec tests should be split between parallel machines evenly.

The bigger the test suite, the longer it takes to run it and more edge cases can happen when you split running tests among many machines in the network. Some of the possible edge cases:

some of the test files take longer than others to run (for instance E2E test files)
some of the test cases fail and run quicker, some don't and run longer. This affects the overall time spent by the CI machine on running your tests.
some of the test cases take longer because they must connect with network/external API etc - this adds uncertainty to their execution time
some of the parallel machines spend more time on boot time:
- installing Ruby gems takes longer
- loading Ruby gems from cache is slow
- or simply the CI provider has not started your job yet
- or maybe you have not enough available machines in the pool of available agents

Multiple things can disrupt the spread of work between parallel nodes.

Our ultimate goal is to ensure all machines finish work at a similar time because this means every machine received a workload that was suitable to its available capabilities. This means that, if a machine started work very late it will run only a small part of the tests. If another machine started work very early it will run more tests. This will even out the ending time between parallel machines. All this is possible thanks to Queue Mode in knapsack_pro Ruby gem, it will take care of running tests in parallel for you. Queue Mode splits test files dynamically between parallel jobs to ensure the jobs completes at the same time.

You can see an example of running a small RSpec test suite across 2 parallel Buildkite agents for the Ruby on Rails project.

A simple example of CI Buildkite parallelism config

Here is a very simple example of Buildkite config to run 2 parallel jobs as you can see on the screenshot.

env:
  # You should hide you secrets like API token
  # Please follow https://buildkite.com/docs/pipelines/secrets
  KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC: "204abb31f698a6686120a40efeff31e5"
  # allow to run the same set of test files on job retry
  # https://github.com/KnapsackPro/knapsack_pro-ruby#knapsack_pro_fixed_queue_split-remember-queue-split-on-retry-ci-node
  KNAPSACK_PRO_FIXED_QUEUE_SPLIT: true

steps:
  - command: "bundle exec rake knapsack_pro:queue:rspec"
    parallelism: 2

Please note that you should hide your credentials like the Knapsack Pro API token and not commit it into your repository. You can refer to the Buildkite secrets documentation.

An advanced Buildkite config with Elastic CI Stack for AWS

When you want to run your big RSpec project on dozen or even hundreds of parallel machines you need powerful resources. In such a case, you can follow the Buildkite tutorial about AWS setup. The Elastic CI Stack for AWS gives you a private, autoscaling Buildkite Agent cluster in your own AWS account.

AWS Spot Instances can save you money

AWS offers Spot Instances. These machines are cheap but they can be withdrawn by AWS at any time. This means that you can run cheap machines for your CI but from time to time the AWS may kill one of your parallel machines. Such a scenario can be handled by the Knapsack Pro. It remembers the set of test files allocated to the AWS machine that was running the tests. When the machine is withdrawn and later on retried by the Buildkite retry feature then the proper test files will be executed as you would expect.

Buildkite retry feature

Buildkite config allows for automatic retry of your job. This can be helpful when you use AWS Spot Instances.
When AWS shuts down your machine during test runtime due to withdrawal then Buildkite can automatically run a new job on a new machine.

Another use case for the automatic retry is when you have flaky Ruby tests that sometimes pass green or fail red. You can use Buildkite to retry the failing job in such a case.

My recommendation is to use the rspec-retry gem as a first choice. RSpec-retry gem will retry only failing test cases instead of all test files assigned to the parallel machine.
The second option you can try is to rely on the Buildkite retry feature. It will retry the CI node and all tests assigned to it by Knapsack Pro API.

How to automatically split large slow RSpec test files by test examples (test cases) between parallel Buildkite agents

Slow RSpec test files are often related to E2E tests, the browser tests like capybara feature specs. They can run for a few or sometimes even dozens of minutes. They could become a bottleneck if the parallel job has to run a single test file for 10 minutes while other parallel jobs complete a few smaller test files in 5 minutes.

There is a solution for that! You can use Knapsack Pro with RSpec split by examples feature that will automatically detect slow RSpec test files in your project and split them between parallel Buildkite agents by test examples (test cases).

Summary

As you can see a combination of a few elements like Buildkite CI with cloud infrastructure solutions like AWS and an optimal split of test files using Knapsack Pro can improve significantly the work of your team.
With Knapsack Pro you can achieve great results and super fast CI builds. Feel free to try it and join other happy Buildkite users.

If you are looking for a Docker config you can also see repository examples at the end of the article: Auto balancing 7 hours tests between 100 parallel jobs on Buildkite CI.

Run Cucumber tests on Github Actions fast with parallel jobs

Artur Trzop — Fri, 26 Mar 2021 19:05:13 +0000

Cucumber employs Behavior-Driven Development (BDD) for testing your application. This type of test is often time-consuming when running in the browser. You will learn how to run Cucumber tests on Github Actions using parallel jobs to execute the test suite much faster.

Github Actions matrix strategy

You can use the Github Actions matrix strategy to run parallel jobs. You will need to divide your Cucumber test files between the parallel jobs in a way that work will be balanced out between the jobs.

It’s not that simple to do because often Cucumber tests can take a different amount of time. One test file can have many test cases, the other can have only a few but very complex ones, etc.

There are often more steps in your CI pipeline like installing dependencies, loading data from the cache and each step can take a different amount of time per parallel job before even Cucumber tests are started. The steps affect the overall CI build speed.

What you would like to achieve is to run parallel jobs in a way that they always finish the execution of Cucumber tests at a similar time. Thanks to that you will avoid lagging jobs that could be a bottleneck in your CI build.

Dynamically split Cucumber tests using Queue Mode

To get optimal CI build execution time you need to ensure the work between parallel jobs is split in such a way as to avoid bottleneck slow job. To achieve that you can split Cucumber test files in a dynamic way between the parallel jobs using Knapsack Pro Queue Mode and knapsack_pro ruby gem.

Knapsack Pro API will take care of coordinating how tests are divided between parallel jobs. On the API side, there is a Queue with a list of your test files and each parallel job on Github Actions is running Cucumber tests via the knapsack_pro Ruby gem. The knapsack_pro gem asks Queue API for a set of test files to run and after it gets executed then the gem asks for another set of test files until the Queue is consumed. This ensures that all parallel jobs finish running tests at a very similar time so that you can avoid bottleneck jobs.

You can learn more about the dynamic test suite split in Queue Mode or check the video below.

Github Actions parallel jobs config for Cucumber

Here is the full Github Actions YAML config example for the Cucumber test suite in a Ruby on Rails project using knapsack_pro gem to run Cucumber tests between parallel jobs.



# .github/workflows/main.yml
name: Main

on: [push]

jobs:
  test:
    runs-on: ubuntu-latest

    # If you need DB like PostgreSQL, Redis then define service below.
    # https://github.com/actions/example-services/tree/master/.github/workflows
    services:
      postgres:
        image: postgres:10.8
        env:
          POSTGRES_USER: postgres
          POSTGRES_PASSWORD: ""
          POSTGRES_DB: postgres
        ports:
          - 5432:5432
        # needed because the postgres container does not provide a healthcheck
        # tmpfs makes DB faster by using RAM
        options: >-
          --mount type=tmpfs,destination=/var/lib/postgresql/data
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5

      redis:
        image: redis
        ports:
          - 6379:6379
        options: --entrypoint redis-server

    # https://help.github.com/en/articles/workflow-syntax-for-github-actions#jobsjob_idstrategymatrix
    strategy:
      fail-fast: false
      matrix:
        # Set N number of parallel jobs you want to run tests on.
        # Use higher number if you have slow tests to split them on more parallel jobs.
        # Remember to update ci_node_index below to 0..N-1
        ci_node_total: [8]
        # set N-1 indexes for parallel jobs
        # When you run 2 parallel jobs then first job will have index 0, the second job will have index 1 etc
        ci_node_index: [0, 1, 2, 3, 4, 5, 6, 7]

    steps:
      - uses: actions/checkout@v2

      - name: Set up Ruby
        uses: actions/setup-ruby@v1
        with:
          ruby-version: 2.7

      - uses: actions/cache@v2
        with:
          path: vendor/bundle
          key: ${{ runner.os }}-gems-${{ hashFiles('**/Gemfile.lock') }}
          restore-keys: |
            ${{ runner.os }}-gems-

      - name: Bundle install
        env:
          RAILS_ENV: test
        run: |
          bundle config path vendor/bundle
          bundle install --jobs 4 --retry 3

      - name: Create DB
        env:
          # use localhost for the host here because we have specified a container for the job.
          # If we were running the job on the VM this would be postgres
          PGHOST: localhost
          PGUSER: postgres
          RAILS_ENV: test
        run: |
          bin/rails db:prepare

      - name: Run tests
        env:
          PGHOST: localhost
          PGUSER: postgres
          RAILS_ENV: test
          KNAPSACK_PRO_TEST_SUITE_TOKEN_CUCUMBER: ${{ secrets.KNAPSACK_PRO_TEST_SUITE_TOKEN_CUCUMBER }}
          KNAPSACK_PRO_CI_NODE_TOTAL: ${{ matrix.ci_node_total }}
          KNAPSACK_PRO_CI_NODE_INDEX: ${{ matrix.ci_node_index }}
          KNAPSACK_PRO_FIXED_QUEUE_SPLIT: true
          KNAPSACK_PRO_LOG_LEVEL: info
        run: |
          bundle exec rake knapsack_pro:queue:cucumber

Here is the view from Github Actions showing that we run 8 parallel jobs for the CI build.

Summary

I hope you find this example useful. If you would like to learn more about Knapsack Pro please check our homepage and see a list of supported test runners for parallel testing in Ruby, JavaScript, etc.

Originally published at https://docs.knapsackpro.com/2021/cucumber-bdd-testing-using-github-actions-parallel-jobs-to-run-tests-quicker

Best add-ons for Ruby on Rails project hosted on Heroku

Artur Trzop — Mon, 08 Mar 2021 14:41:15 +0000

After working for over 8 years with Heroku and Ruby on Rails projects I have my own favorite set of Heroku add-ons that work great with Rails apps. You are about to learn about the add-ons that come in handy in your daily Ruby developer life.

Heroku add-ons

Here it is, a list of my favorite Heroku add-ons from Heroku Marketplace and why I choose them for my Ruby on Rails projects hosted on Heroku.

Heroku Scheduler

Heroku Scheduler can run scheduled tasks every 10 minutes, every hour, or every day. I use it to run my scheduled rake tasks. For instance every day I run a rake task that will send a summary of users who signed up in the last 24 hours to my mailbox.

Heroku Scheduler add-on is free. The only limitation is that it has fewer options than the cron in the Unix system. If you need to run a rake task every Monday then you need to set up a rake task as a daily task in Heroku Scheduler and do a check of the day in the rake task itself to skip it when needed.

# lib/tasks/schedule/notify_users_about_past_due_subscription.rake
namespace :schedule do
  desc 'Send notification about past due subscriptions to users'
  task notify_users_about_past_due_subscription: :environment do
    if Time.current.monday?
      Billing::NotifyUsersAboutPastDueSubscriptionWorker.perform_async
    else
      Rails.logger.info("Skip schedule:notify_users_about_past_due_subscription task.")
    end
  end
end

New Relic APM

New Relic add-on does application performance monitoring. It's one of my favorite add-ons. It allows to track each process like puma/unicorn/sidekiq per dyno and its performance. You can see which Rails controller actions take the most time. You can see your API endpoints with the highest throughput and those which are time-consuming. New Relic helped me many times to debug bottlenecks in my app and thanks to that I was able to make Knapsack Pro API with an average 50ms response time. Who said the Rails app has to be slow? :)

Rollbar

Rollbar allows for exception tracking in your Ruby code and also in JS code on the front end side. It has a generous free plan with a 5000 exception limit per month.

You can easily ignore some common Rails exceptions to stay within the free plan limit.

# config/initializers/rollbar.rb
Rollbar.configure do |config|
  config.access_token = ENV['ROLLBAR_ACCESS_TOKEN']

  if Rails.env.test? || Rails.env.development?
    config.enabled = false
  end

  # Add exception class names to the exception_level_filters hash to
  # change the level that exception is reported at. Note that if an exception
  # has already been reported and logged the level will need to be changed
  # via the rollbar interface.
  # Valid levels: 'critical', 'error', 'warning', 'info', 'debug', 'ignore'
  # 'ignore' will cause the exception to not be reported at all.
  config.exception_level_filters.merge!('ActionController::RoutingError' => 'ignore')
  config.exception_level_filters.merge!('ActionController::InvalidAuthenticityToken' => 'ignore')
  config.exception_level_filters.merge!('ActionController::BadRequest' => 'ignore')
  config.exception_level_filters.merge!('ActiveRecord::RecordNotFound' => 'ignore')
  config.exception_level_filters.merge!('Rack::Timeout::RequestTimeoutException' => 'ignore')
  config.exception_level_filters.merge!('Rack::QueryParser::InvalidParameterError' => 'ignore')
  config.exception_level_filters.merge!('ActionDispatch::Http::MimeNegotiation::InvalidType' => 'ignore')

  config.environment = ENV['ROLLBAR_ENV'].presence || Rails.env
end

Logentries

Logentries - collects your logs from Heroku standard output so that you can browse them later on. If you need to find info about an issue that happened a few days ago in your logs then Logentries might be helpful.

Of course, you could use Heroku Command Line Interface and run heroku logs -n 10000 --app my-heroku-app command in terminal to browse logs for the last 10,000 lines but this method has limitations. You can't go that much in past logs or easily filter logs as in Logentries.

Logentries has a 5 GB and 7 days retention period in a free plan. This is enough for small Rails apps.

A nice feature I like in Logentries is an option to save the query and later on quickly browse logs by it. You can also display charts based on logs. Maybe you want to see how often a particular worker in Sidekiq has been called? You could visualize it.

Redis Cloud

If you use Redis in your Ruby on Rails app then Redis Cloud is your add-on. It has a free plan and paid plans are more affordable than other add-ons have.

Redis Cloud add-on does automatic backups of your data and offers a nice web UI to preview the live Redis usage and historical usage of your database instance.

I like to use Redis Cloud + sidekiq gem in my Rails apps. Also, Redis is useful if you need to cache some data quickly in the memory and expire it after some time.

redis_connection = Redis.new(
  # use REDISCLOUD_URL when app is running on Heroku,
  # or fallback to local Redis (useful for development)
  url: ENV.fetch('REDISCLOUD_URL', 'redis://localhost:6379/0'),
  # tune network timeouts to be a little more lenient when you are seeing occasional timeout
  # errors for Heroku Redis Cloud addon
  # https://github.com/mperham/sidekiq/wiki/Using-Redis#life-in-the-cloud
  timeout: 5
)

redis_connection.setex('my-key-name', 1.hour, 'this value will expire in 1 hour')

Twilio SendGrid

SendGrid is a free add-on that allows you to start sending emails from your Ruby on Rails. You can even connect your domain to it so your users get emails from your domain.

There are free 12,000 emails per month in the free plan.

Heroku Add-on to save you time & money

Here are a few of my favorite add-ons that will help you save money and time in your project.

AutoIdle

AutoIdle lets you save money by automatically putting your staging and review apps to sleep on Heroku. I use it to turn off my web and worker dyno for the staging app when there is no traffic to the app. No more paying for Heroku resources during the night and weekends. ;)

Rails Autoscale

Rails Autoscale is a powerful add-on that will help you save money on Heroku. It will measure requests queue time and based on that add or remove dynos for your web processes. If you have higher traffic during the day it will add more dynos. During the night when the traffic is low, it will remove dynos.

Rails Autoscale can also track your worker queue. For instance, if you have a lot of jobs scheduled in Sidekiq then Rails Autoscale will add more worker dynos to process your job queue faster. It can even shut down worker dyno when there are no jobs to be processed which can save you even more money.

Knapsack Pro

Knapsack Pro is a Heroku add-on and ruby gem that can run your Rails tests in RSpec, Cucumber, Minitest, etc, and automatically split the tests between parallel machines on any CI server. It works with Heroku CI, CircleCI, Buildkite, Travis CI, etc. It will help you save time by doing a dynamic split of tests with Queue Mode to ensure all parallel jobs finish work at a similar time. This way you optimize your CI build runs and save the most time.

Below you can see an example of the optimal distribution of tests, where each parallel CI machine performs tests for 10 minutes, thanks to which the entire CI build lasts only 10 minutes instead of 40 if you would run tests on a single CI server only.

I've been working on the Knapsack Pro add-on and I'd love to hear your feedback if you give it a try.

How to run fast RSpec tests on CircleCI with parallel jobs and have nice JUnit XML reports in CircleCI web UI

Artur Trzop — Wed, 03 Mar 2021 19:19:30 +0000

You will learn how to run RSpec tests for your Ruby on Rails project on CircleCI with parallel jobs to shorten the running time of your CI build. Moreover, you will learn how to configure JUnit formatter to generate an XML report for your tests to show failing RSpec test examples nicely in CircleCI web UI. Finally, you will see how to automatically detect slow spec files and divide their test examples between parallel jobs to eliminate the bottleneck job that’s taking too much time to run tests.

Ruby gems to configure your RoR project

Here are the key elements you need:

rspec_junit_formatter - it’s a ruby gem that generates an XML report for executed tests with information about test failures. This report can be automatically read by CircleCI to present it in CircleCI web UI. No more browsing through long RSpec output - just look at highlighted failing specs in the TESTS tab :)

knapsack_pro - it’s a Ruby gem for running tests on parallel CI jobs to ensure all jobs finish work at a similar time to save you as much time as possible and eliminate bottlenecks.
- It uses the Queue Mode to dynamically split test files between parallel jobs.
- Knapsack Pro can also detect your slow RSpec test files and divide them between parallel jobs by test examples. You don’t have to manually split your big spec file into smaller files if you want to split work between parallel container on CircleCI :)

Just add the above gems to your Gemfile.

group :test do
  gem 'rspec'
  gem 'rspec_junit_formatter'
end

group :test, :development do
  gem 'knapsack_pro'
end

For Knapsack Pro you will need an API token and you need to follow the installation guide to configure your project.

If you use knapsack_pro gem in Queue Mode with CircleCI you may want to collect metadata like JUnit XML report about your RSpec test suite. The important step for CircleCI is to copy the XML report to $CIRCLE_TEST_REPORTS directory. Below is a full config for your spec_helper.rb file (source code from FAQ):

# spec_helper.rb or rails_helper.rb

# This must be the same path as value for rspec --out argument
# Note: the path should not contain '~' sign, for instance path ~/project/tmp/rspec.xml may not work.
# Please use full path instead.
TMP_RSPEC_XML_REPORT = 'tmp/rspec.xml'
# move results to FINAL_RSPEC_XML_REPORT
# so that the results won't accumulate with duplicated xml tags in TMP_RSPEC_XML_REPORT
FINAL_RSPEC_XML_REPORT = 'tmp/rspec_final_results.xml'

KnapsackPro::Hooks::Queue.after_subset_queue do |queue_id, subset_queue_id|
  if File.exist?(TMP_RSPEC_XML_REPORT)
    FileUtils.mv(TMP_RSPEC_XML_REPORT, FINAL_RSPEC_XML_REPORT)
  end
end

You need the above logic in place to move the XML report from one place to another to avoid accidentally corrupting your XML file. When Knapsack Pro in Queue Mode runs your tests then it fetches a set of test files from Knapsack Pro Queue API and runs it and generates the XML report. After that, another set of test files is fetched from Queue API and the XML report is updated on the disk. If the report already exists on the disk it can get corrupted due to overriding the same file. That’s why you need to move the file to a different location after each set of tests from Queue API is executed.

CircleCI YML configuration for RSpec

Here is the complete CircleCI YML config file for RSpec, Knapsack Pro and JUnit formatter.

# Ruby CircleCI 2.0 configuration file
#
# Check https://circleci.com/docs/2.0/language-ruby/ for more details
#
version: 2
jobs:
  build:
    parallelism: 10
    # https://circleci.com/docs/2.0/configuration-reference/#resource_class
    resource_class: small
    docker:
      # specify the version you desire here
      - image: circleci/ruby:2.7.1-node-browsers
        environment:
          PGHOST: 127.0.0.1
          PGUSER: my_db_user
          RAILS_ENV: test
          # Split slow RSpec test files by test examples
          # https://knapsackpro.com/faq/question/how-to-split-slow-rspec-test-files-by-test-examples-by-individual-it
          KNAPSACK_PRO_RSPEC_SPLIT_BY_TEST_EXAMPLES: true

      # Specify service dependencies here if necessary
      # CircleCI maintains a library of pre-built images
      # documented at https://circleci.com/docs/2.0/circleci-images/
      - image: circleci/postgres:10.6-alpine-ram
        environment:
          POSTGRES_DB: my_db_name
          POSTGRES_PASSWORD: ""
          POSTGRES_USER: my_db_user
          # Rails verifies Time Zone in DB is the same as time zone of the Rails app
          TZ: "Europe/Warsaw"

      - image: redis:6.0.7

    working_directory: ~/repo
    environment:
      TZ: "Europe/Warsaw"

    steps:
      - checkout

      # Download and cache dependencies
      - restore_cache:
          keys:
          - v2-dependencies-bundler-{{ checksum "Gemfile.lock" }}-{{ checksum ".ruby-version" }}
          # fallback to using the latest cache if no exact match is found
          - v2-dependencies-bundler-

      - run:
          name: install ruby dependencies
          command: |
            bundle install --jobs=4 --retry=3 --path vendor/bundle

      - save_cache:
          paths:
            - ./vendor/bundle
          key: v2-dependencies-bundler-{{ checksum "Gemfile.lock" }}-{{ checksum ".ruby-version" }}

      # Database setup
      - run: bin/rails db:prepare

      - run:
          name: run tests
          command: |
            export CIRCLE_TEST_REPORTS=/tmp/test-results
            mkdir $CIRCLE_TEST_REPORTS
            bundle exec rake "knapsack_pro:queue:rspec[--format documentation --format RspecJunitFormatter --out tmp/rspec.xml]"

      # collect reports
      - store_test_results:
          path: /tmp/test-results
      - store_artifacts:
          path: /tmp/test-results
          destination: test-results

Summary

You’ve just learned how to make your CircleCI builds way faster! Now your RSpec tests can be automatically run on many parallel machines to save you time. Please let us know if it was helpful or if you have any questions. Feel free to sign up at Knapsack Pro or down below and try it yourself.

Originally published at docs.knapsackpro.com

How to run too slow RSpec test file in parallel on Github Actions - programmatically divide spec file by test examples

Artur Trzop — Mon, 29 Jun 2020 09:43:24 +0000

Splitting your CI build jobs between multiple machines running in parallel is a great way to make the process fast, which results in more time for building features. Github Actions allows running parallel jobs easily. In a previous article, we explained how you can use Knapsack Pro to split your RSpec test files efficiently between parallel jobs on GitHub Actions. Today we'd like to show how to address the problem of slow test files negatively impacting the whole build times.

Consider the split

Imagine you have a project with 30 RSpec spec files. Each file contains multiple test examples (RSpec "its"). Most of the files are fairly robust, fast unit tests. Let's say there are also some slower files, like feature specs. Perhaps one such feature spec file takes approximately 5 minutes to execute.

When we run different spec files on different parallel machines, we strive for similar execution time on all of them. In a described scenario, even if we run 30 parallel jobs (each one running just one test file), the 5 minute feature spec would be a bottleneck of the whole build. 29 machines may finish their work in a matter of seconds, but the whole build won't be complete until the 1 remaining node finishes executing its file.

Divide and conquer

To solve the problem of a slow test file, we need to split what's inside it. We could refactor it and ensure the test examples live in separate, smaller test files. There are two problems with that though:

First, it needs work. Although admittedly quite plausible in a described scenario, in real life it's usually not just the one file that's causing problems. Oftentimes there is a number of slow and convoluted test files, with their own complex setups, like nested before blocks, lets, etc. We've all seen them (and probably contributed to them ending-up this way), haven't we? ;-) Refactoring files like that is no fun, and there seem to always be more higher prio work to be done, at least from our experience.

Second, we belive that the code organization should be based on other considerations. How you create your files and classes is most likely a result of following some approach agreed upon in your project. Dividing classes into smaller ones so that the CI build can run faster encroaches on your conventions. It might be more disturbing to some than the others, but we feel it's fair to say it'd be best to avoid - if there was a better way to achieve the same...

Enter split by test examples

As you certainly know, RSpec allows us to run individual examples instead of whole files. We decided to take advantage of that, and solve the problem of bottleneck test files by gathering information about individual examples from such slower files. Such examples are then dynamically distributed between your parallel nodes and run individually, so no individual file can be a bottleneck for the whole build. What's important, no additional work is needed - this is done automatically by the knapsack_pro gem. Each example is run in its correct context that's set-up exactly the same as if you had run the whole file.

If you are already using Knapsack Pro in queue mode, you can enable this feature just by adding one ENV variable to your GitHub Actions workflow config: KNAPSACK_PRO_RSPEC_SPLIT_BY_TEST_EXAMPLES: true (please make sure you're running the newest version of knapsack_pro gem). After a few runs, Knapsack Pro will start automatically splitting your slowest test files by individual examples.

Here's a full example GitHub Actions workflow config for a Rails project using RSpec:

# .github/workflows/main.yaml

name: Main

on: [push]

jobs:
  test:
    runs-on: ubuntu-latest

    # If you need DB like PostgreSQL, Redis then define service below.
    # https://github.com/actions/example-services/tree/master/.github/workflows
    services:
      postgres:
        image: postgres:10.8
        env:
          POSTGRES_USER: postgres
          POSTGRES_PASSWORD: ""
          POSTGRES_DB: postgres
        ports:
          - 5432:5432
        # needed because the postgres container does not provide a healthcheck
        # tmpfs makes DB faster by using RAM
        options: >-
          --mount type=tmpfs,destination=/var/lib/postgresql/data
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
      redis:
        image: redis
        ports:
          - 6379:6379
        options: --entrypoint redis-server

    # https://help.github.com/en/articles/workflow-syntax-for-github-actions#jobsjob_idstrategymatrix
    strategy:
      fail-fast: false
      matrix:
        # Set N number of parallel jobs you want to run tests on.
        # Use higher number if you have slow tests to split them on more parallel jobs.
        # Remember to update ci_node_index below to 0..N-1
        ci_node_total: [8]
        # set N-1 indexes for parallel jobs
        # When you run 2 parallel jobs then first job will have index 0, the second job will have index 1 etc
        ci_node_index: [0, 1, 2, 3, 4, 5, 6, 7]

    steps:
      - uses: actions/checkout@v2

      - name: Set up Ruby
        uses: actions/setup-ruby@v1
        with:
          ruby-version: 2.6

      - uses: actions/cache@v2
        with:
          path: vendor/bundle
          key: ${{ runner.os }}-gems-${{ hashFiles('**/Gemfile.lock') }}
          restore-keys: |
            ${{ runner.os }}-gems-
      - name: Bundle install
        env:
          RAILS_ENV: test
        run: |
          bundle config path vendor/bundle
          bundle install --jobs 4 --retry 3
      - name: Create DB
        env:
          # use localhost for the host here because we have specified a container for the job.
          # If we were running the job on the VM this would be postgres
          PGHOST: localhost
          PGUSER: postgres
          RAILS_ENV: test
        run: |
          bin/rails db:prepare
      - name: Run tests
        env:
          PGHOST: localhost
          PGUSER: postgres
          RAILS_ENV: test
          KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC: ${{ secrets.KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC }}
          KNAPSACK_PRO_CI_NODE_TOTAL: ${{ matrix.ci_node_total }}
          KNAPSACK_PRO_CI_NODE_INDEX: ${{ matrix.ci_node_index }}
          KNAPSACK_PRO_FIXED_QUEUE_SPLIT: true
          KNAPSACK_PRO_RSPEC_SPLIT_BY_TEST_EXAMPLES: true
          KNAPSACK_PRO_LOG_LEVEL: info
        run: |
          bundle exec rake knapsack_pro:queue:rspec

You can find more details in the video below:

Let us know in the comments what you think about this solution. If you'd like to give this setup a try, you can also consult our FAQ entry explaining how to split slow RSpec test files.

As always, don't hesitate to ask questions if you encounter any troubles with configuring GitHub Actions in your project - we'd be happy to help!

Originally published at: https://docs.knapsackpro.com/2020/how-to-run-slow-rspec-files-on-github-actions-with-parallel-jobs-by-doing-an-auto-split-of-the-spec-file-by-test-examples

DEV Community: Artur Trzop

DB max connection limits for Rails app and Postgres, Redis, Puma settings

Why do available database connections matter?

RoR application configuration step by step

Postgres database connections - how to check the limit?

ActiveRecord connection pool

How does ActiveRecord connection pool affects Postgres max connections?

Puma config: 1 process and 1 thread

Puma config: 1 process and 2 threads

Puma config: 2 processes and 2 threads per process

Puma config: 2 processes and 2 threads, and 2 web dynos on Heroku

Autoscaling web application

What else can open DB connections?

Background worker - Sidekiq and ActiveRecord pool

Sidekiq and Redis database connections

Redis database connections

Summary

BitBucket parallel Cypress tests configuration for CI pipeline integration

How to run tests in parallel

Queue Mode - a dynamic tests split

BitBucket Pipeline YML config

Summary

Testing Ruby on Rails on Github Actions with RSpec

Github Actions YML config for Rails application

ruby/setup-ruby action

How to configure Postgres on Github Actions

How to use Github Actions build matrix to run tests with parallel jobs

Full YML config for Github Actions and Ruby on Rails project

Summary

How to run Minitest parallel tests on Github Actions

Split tests in a dynamic way with Queue Mode

Github Actions build matrix to run parallel tests

Summary

ruby/setup-ruby or actions/cache - caching Ruby gems on Github Actions

actions/cache - just cache dependencies

ruby/setup-ruby - install Ruby and cache gems

Summary

Parallel scaling RSpec tests on Buildkite to increase CI build speed

A real RSpec test suite taking 13 hours and executed in only 5 minutes

How to distribute test files between parallel jobs using Queue Mode in Knapsack Pro to utilize CI machines optimally

A simple example of CI Buildkite parallelism config

An advanced Buildkite config with Elastic CI Stack for AWS

AWS Spot Instances can save you money

Buildkite retry feature

How to automatically split large slow RSpec test files by test examples (test cases) between parallel Buildkite agents

Summary

Related articles

Run Cucumber tests on Github Actions fast with parallel jobs

Github Actions matrix strategy

Dynamically split Cucumber tests using Queue Mode

Github Actions parallel jobs config for Cucumber

Summary

Best add-ons for Ruby on Rails project hosted on Heroku

Heroku add-ons

Heroku Scheduler

New Relic APM

Rollbar

Logentries

Redis Cloud

Twilio SendGrid

Heroku Add-on to save you time & money

AutoIdle

Rails Autoscale

Knapsack Pro

How to run fast RSpec tests on CircleCI with parallel jobs and have nice JUnit XML reports in CircleCI web UI

Ruby gems to configure your RoR project

CircleCI YML configuration for RSpec

Summary

How to run too slow RSpec test file in parallel on Github Actions - programmatically divide spec file by test examples

Consider the split

Divide and conquer

Enter split by test examples