Parameterized testing with ExUnit

#elixir #exunit

UPDATE: hauleth mentions me a better solution in ElixirForum. So, I updated this article.

I found a way which is parameterized testing with ExUnit.

ExUnit.start()

defmodule ParameterizedTest do
  use ExUnit.Case, async: true

  for {lhs, rhs} <- [{"one", 1}, {"two", 2}, {"three", 3}] do
    test "#{lhs} convert to #{rhs}" do
      assert unquote(lhs) === unquote(rhs)
    end
  end
end

  1) test one convert to 1 (ParameterizedTest)
     parameterized_test.exs:7
     Assertion with === failed
     code:  assert "one" === 1
     left:  "one"
     right: 1
     stacktrace:
       parameterized_test.exs:8: (test)



  2) test three convert to 3 (ParameterizedTest)
     parameterized_test.exs:7
     Assertion with === failed
     code:  assert "three" === 3
     left:  "three"
     right: 3
     stacktrace:
       parameterized_test.exs:8: (test)



  3) test two convert to 2 (ParameterizedTest)
     parameterized_test.exs:7
     Assertion with === failed
     code:  assert "two" === 2
     left:  "two"
     right: 2
     stacktrace:
       parameterized_test.exs:8: (test)



Finished in 0.03 seconds (0.03s on load, 0.00s on tests)
3 tests, 3 failures

Randomized with seed 169786

The Probrem

We can't pass variables from outside into the block of tests directly.

ExUnit.start()

defmodule ParameterizedTest do
  use ExUnit.Case, async: true

  for {lhs, rhs} <- [{"one", 1}, {"two", 2}, {"three", 3}] do
    test "#{lhs} convert to #{rhs}" do
      assert lhs === rhs # We can't use their variables here.
    end
  end
end

** (CompileError) parameterized_test.exs:8: undefined function lhs/0
    (stdlib) lists.erl:1338: :lists.foreach/2

Other solutions

Use assertions instead of tests

Assertions and tests work well. But, ExUnit shows only one message per test. So, If you desire showing outline of tests. It does not match well.

ExUnit.start()

defmodule ParameterizedTest do
  use ExUnit.Case, async: true

  test "convert" do
    for {lhs, rhs} <- [{"one", 1}, {"two", 2}, {"three", 3}] do
      assert lhs === rhs
    end
  end
end

  1) test convert (ParameterizedTest)
     parameterized_test.exs:6
     Assertion with === failed
     code:  assert lhs === rhs
     left:  "one"
     right: 1
     stacktrace:
       parameterized_test.exs:8: anonymous fn/2 in ParameterizedTest."test convert"/1
       (elixir) lib/enum.ex:1925: Enum."-reduce/3-lists^foldl/2-0-"/3
       parameterized_test.exs:7: (test)



Finished in 0.03 seconds (0.03s on load, 0.00s on tests)
1 test, 1 failure

Randomized with seed 303559

Use module attributes

Tests work well. But, module attributes breaks scope. So, you have to keep module attribute in mind after using it.

ExUnit.start()

defmodule ParameterizedTest do
  use ExUnit.Case, async: true

  for {lhs, rhs} <- [{"one", 1}, {"two", 2}, {"three", 3}] do
    @pair {lhs, rhs}

    test "#{lhs} convert to #{rhs}" do
      {l, r} = @pair
      assert l === r
    end
  end

  test "pair should not have any value" do
    assert nil === @pair # We don't expect getting any values
  end
end

  1) test pair should not have any value (ParameterizedTest)
     parameterized_test.exs:15
     Assertion with === failed
     code:  assert nil === @pair
     left:  nil
     right: {"three", 3}
     stacktrace:
       parameterized_test.exs:16: (test)



  2) test three convert to 3 (ParameterizedTest)
     parameterized_test.exs:9
     Assertion with === failed
     code:  assert l === r
     left:  "three"
     right: 3
     stacktrace:
       parameterized_test.exs:11: (test)



  3) test two convert to 2 (ParameterizedTest)
     parameterized_test.exs:9
     Assertion with === failed
     code:  assert l === r
     left:  "two"
     right: 2
     stacktrace:
       parameterized_test.exs:11: (test)



  4) test one convert to 1 (ParameterizedTest)
     parameterized_test.exs:9
     Assertion with === failed
     code:  assert l === r
     left:  "one"
     right: 1
     stacktrace:
       parameterized_test.exs:11: (test)



Finished in 0.04 seconds (0.04s on load, 0.00s on tests)
4 tests, 4 failures

Randomized with seed 936081

Use ex_parameterized

https://github.com/KazuCocoa/ex_parameterized makes allow parameterized testing following:

defmodule MyExampleTest do
  use ExUnit.Case, async: true
  use ExUnit.Parameterized        # Required

  test_with_params "add params",  # description
    fn (a, b, expected) ->        # test case
      assert a + b == expected
    end do
      [
        {1, 2, 3},                 # parameters
        "description": {1, 4, 5},  # parameters with description
      ]
  end
end

I like this idea. But this time, I have wanted to write ExUnit test cases with only standard libraries.

Use ExUnit.Case.register_test/4

ExUnit.Case.register_test/4.

Registers a new attribute to be used during ExUnit.Case tests.

The attribute values will be available as a key/value pair in context.registered. The key/value pairs will be cleared after each ExUnit.Case.test/3 similar to @tag.

ExUnit.start()

defmodule ParameterizedTest do
  use ExUnit.Case, async: true

  ExUnit.Case.register_attribute __ENV__, :pair

  for {lhs, rhs} <- [{"one", 1}, {"two", 2}, {"three", 3}] do
    @pair {lhs, rhs}

    test "#{lhs} convert to #{rhs}", context do
      {l, r} = context.registered.pair
      assert l === r
    end
  end

  test "pair should not have any value" do
    assert nil === @pair
  end
end

warning: undefined module attribute @pair, please remove access to @pair or explicitly set it before access
  parameterized_test.exs:18: ParameterizedTest (module)

.

  1) test two convert to 2 (ParameterizedTest)
     parameterized_test.exs:11
     Assertion with === failed
     code:  assert l === r
     left:  "two"
     right: 2
     stacktrace:
       parameterized_test.exs:13: (test)



  2) test one convert to 1 (ParameterizedTest)
     parameterized_test.exs:11
     Assertion with === failed
     code:  assert l === r
     left:  "one"
     right: 1
     stacktrace:
       parameterized_test.exs:13: (test)



  3) test three convert to 3 (ParameterizedTest)
     parameterized_test.exs:11
     Assertion with === failed
     code:  assert l === r
     left:  "three"
     right: 3
     stacktrace:
       parameterized_test.exs:13: (test)



Finished in 0.05 seconds (0.05s on load, 0.00s on tests)
4 tests, 3 failures

Randomized with seed 385498

It works well, outline of tests, scope about variables and only using standard libraries. Actually, this solution is written on the top of this article before I update. But I feel codes a little bit verbose. for example: {l, r} = context.registered.pair.