Rails: Minitest results output in TAP-format for Heroku CI

When using Heroku CI for automatic testing, the default output is just pass or fail without any parsing of how many tests passed or what errors where found.

This can be improved if you output the results in the Test Anything Protocol-format (TAP-format) according to Heroku.

This format is not built into the test frameworks used in Ruby on Rails. Therefore I had to write my own test reporter.

I chose to implement this for Minitest and decided to use the gem
minitest-reporters.

Expected outcome

I have a small project where the test suit looks like this:

To adhere to the TAP-format I need it to also output:

TAP version 13
1..7
ok 1 - AccountsControllerTest - test_renders_users_account
ok 2 - AccountsControllerTest - test_require_login
ok 3 - CallbacksMessagesControllerTest - test_handle_status_callback
ok 4 - MessagesControllerTest - test_should_create_message
ok 5 - ContactsControllerTest - test_should_create_contact
ok 6 - ContactsControllerTest - test_should_update_contact
ok 7 - ContactsControllerTest - test_show_contact

Solution

I created a custom reporter called TapReporter by inspecting a standard reporter from the minitest-reporters gem called ProgressReporter (see source):

	module Minitest
	module Reporters
	# Based on https://github.com/kern/minitest-reporters/blob/master/lib/minitest/reporters/progress_reporter.rb
	# Tries to adhere to https://testanything.org/tap-specification.html
	class TapReporter < DefaultReporter
	def report
	super
	puts("TAP version 13")
	puts("1..#{tests.length}")
	tests.each_with_index do |test, index|
	test_str = "#{test_class(test)} - #{test.name.tr("#", "_")}"
	if test.passed?
	puts "ok #{index + 1} - #{test_str}"
	elsif test.skipped?
	puts "ok #{index + 1} # skip: #{test_str}"
	elsif test.failure.present?
	puts "not ok #{index + 1} - failed: #{test_str}"
	message_for(test).each_line do |line|
	print_padded_comment(line)
	end
	unless test.failure.is_a?(MiniTest::UnexpectedError)
	trace = filter_backtrace(test.failure.backtrace)
	trace.each do |line|
	print_padded_comment(line)
	end
	end
	puts
	end
	end
	end
	private
	def print_padded_comment(line)
	puts "##{pad(line)}"
	end
	end
	end
	end

view raw tap_reporter.rb hosted with ❤ by GitHub

The solution on Heroku

A failing test

Successful tests

Keywords

• Ruby on Rails.
• Heroku - Platform as a Service, useful for hosting web applications.
• Continuous integration (CI) - in this post it mostly refers to automated testing.
• Test anything protocol(TAP) - a protocol for test results.
• Minitest - a test framework for ruby.