Brian Kephart

Posted on Aug 15, 2018 • Originally published at briankephart.com on Aug 15, 2018

Adding Code Quality Checks To Your Ruby Test Suite

#ruby #rails #testing

Over time I've integrated a few static analysis tools into my workflow. These have been extremely helpful in maintaining high quality code, but running them manually is tedious. For many projects, this would be automated in some kind of CI pipeline. For my current project, however, there is no automated CI pipeline. This app is deployed to Heroku straight from the command line. So, I decided to see if I could integrate these tools into my test suite, reducing manual quality checks to a single step.

Tool #1: Brakeman

Since I started learning Rails development, I have been using Brakeman to scan my code for security vulnerabilities. This tool uses static analysis to check for vulnerabilities to attacks like SQL injection and cross-site scripting. Brakeman was designed to be run from the command line, but a little digging into the docs revealed that it can be run as a Ruby library within an app. Following these instructions, I opened up a Pry console and ran:

require 'brakeman'
=> true
Brakeman.run Rails.root.to_s
=> #<Brakeman::Tracker:0x00007fde2d546da8 ... tons of output in this object

That object had way too much output to read. Next, I ran the same command, but stored the result as a variable...

result = Brakeman.run Rails.root.to_s
=> # same result as before

... so I could query its methods and attributes:

result.methods.sort
=> [:warnings] # There were tons of other methods in this array
result.warnings
=> []

result.instance_variables.sort
=> [:@errors] # Lots more here too
result.errors
=> [{:error=>"/Users/briankephart/Sites/some_app/app/jobs/some_job.rb:14 :: parse error on value \"**\" (error)",
   :backtrace=>["Could not parse /Users/briankephart/Sites/some_app/app/jobs/some_job.rb"]}]

As you can see, I had no security warnings, but one error due to Brakeman failing to parse a ** operator. This was helpful in showing me that the errors are readable in this form. Note that both the errors and warnings are returned as Arrays.

Using what I learned in my Pry console, I wrote the following test:

class BrakemanTest < ActiveSupport::TestCase
  require 'brakeman'

  test 'no brakeman errors or warnings' do
    result = Brakeman.run Rails.root.to_s
    assert_equal [], result.errors
    assert_equal [], result.warnings
  end
end

This worked perfectly. I could see the Brakeman output during my test run, and the test responded correctly to the error seen above.

Tool #2: RuboCop

I've recently been using RuboCop in a lot more of my work to check for syntax errors, good styling, and general readability. Like Brakeman, Rubocop is designed as a command line tool. Unlike Brakeman, the docs were not helpful to me in using it any other way. Instead, I had to dive into the source code. The most important line I found was here, in the file that defines the command line executable. The important points:

require 'rubocop'
cli = RuboCop::CLI.new
result = cli.run

What I needed to do was instantiate a RuboCop::CLI object and call its #run method. Looking at those files, I found that the #initialize method takes no arguments, while the #run method takes all arguments from the command line as an array (I usually run rubocop on the command line without arguments, though). Back to the Pry console:

require 'rubocop'
=> true

RuboCop::CLI.new.run
# Inspecting 397 files
# ...
#
# 397 files inspected, no offenses detected
=> 0

Notice that the return value is 0. Knowing this, I wrote the following test:

# Do not use this code
class RubocopTest < ActiveSupport::TestCase
  require 'rubocop'

  test 'no rubocop offenses' do
    assert_equal 0, RuboCop::CLI.new.run
  end
end

I noticed something strange, though. When I ran my full test suite, RuboCop checked all my files, but when I ran the test above on its own, RuboCop only checked that test file. The context in which I called the test became the context for RuboCop. The fix for this was to add a path argument to the run method, just as RuboCop allows a path argument on the command line.

# Use this instead of the code above
class RubocopTest < ActiveSupport::TestCase
  require 'rubocop'

  test 'no rubocop offenses' do
    cop = RuboCop::CLI.new
    args = [Rails.root.to_s]
    assert_equal 0, cop.run(args)
  end
end

Tool #3: Bundler Audit

Bundler Audit checks your Gemfile.lock for known-vulnerable versions of gems. This tool is pretty wedded to the command line, to the point where it proved easiest to just run system commands.

class RubocopTest < ActiveSupport::TestCase
  require 'bundler/audit/cli'

  test 'no vulns in bundle' do
    `bundle audit update -q` # Update vulnerability database
    result = `bundle audit`  # Run the audit
    code = `echo $?`.squish  # Returns '0' if successful, otherwise '1'

    # Print the scan result as the error message if it fails.
    assert_equal '0', code, result

    # If successful, output the success message
    puts "\nMessage from bundler-audit: #{result}"
  end
end