DEV Community: Ulisses Almeida

Test Data Libraries for Elixir

Ulisses Almeida — Tue, 02 May 2023 11:07:31 +0000

In part one of this series, we introduced Elixir test factories and fixtures. Then in part two, we explored using data generation functions.

Now we'll look at some of the best Elixir libraries to use for your test data.

But before we do, let's quickly discuss why test data libraries can be helpful.

Why Choose a Test Data Library for Your Elixir App?

Elixir's built-in language features are more than sufficient for writing simple helpers to build your application test data.

For example, refer to the Ecto Guide to write a factory method pattern API like the one provided by ExMachina.

However, existing test data libraries are convenient as you don't have to write them yourself.

Elixir Libraries for Your Test Data

Here's a quick overview of existing test data libraries and why you might want to use them. We'll look at ExMachina, ExZample, Faker, and StreamData.

Let's start with the most famous one: ExMachina.

ExMachina

The factory library ExMachina created by Thoughbot uses function names and generates an atom used to call factories. For example:

defmodule MyApp.Factory do
  use ExMachina

  def github_repo_factory do
    repo_name = sequence(:github_repo_name, &"repo-#{1}")

    %GitHub.Repo{
      id: 1296269,
      name: repo_name,
      full_name: "octocat/#{repo_name}",
      description: "This your first repo!",
      owner_id: 1,
      owner_url: "https://api.github.com/users/octocat",
      private: false,
      html_url: "https://github.com/octocat/#{repo_name}",
      url: "https://api.github.com/repos/octocat/#{repo_name}"
    }
  end
end

You can invoke that factory using the functions provided by ExMachina. For example, here's a function that generates a list of resources:

MyApp.Factory.build_list(3, :github_repo)
[
  %GitHub.Repo{
    name: "octocat/repo-1"
    # ...
  },
  %GitHub.Repo{
    name: "octocat/repo-2"
    # ...
  }
    %GitHub.Repo{
    name: "octocat/repo-3"
    # ...
  }
]

The github_repo_factory/0 can be called using the :github_repo atom, which can call any utility function provided by ExMachina and injected by the use ExMachina macro.

One of the most powerful features of ExMachina is the sequence function, which guarantees you'll get the next increment of a sequence each time you call it, and the number will always be unique. It simplifies the caller's job by eliminating the need to explicitly pass a unique name. Using the sequence function automatically avoids unique database constraint issues when calling the same factory multiple times.

ExMachina also lets you add more building strategies, such as a JSON strategy that transforms your data example into JSON. The documentation suggests that you create smaller modules (with macros defining your factories) to break down a large factory module and still serve everything under the same module API. However, I wouldn't necessarily recommend this approach: it's harder to track the factory code definition if it breaks. I would rather have multiple factory modules when using ExMachina.

ExMachina is a well-established, actively maintained library that's widely used in the Elixir community. If invoking factories using the atom naming mechanism fits your preference, it's a solid choice!

Now let's take a look at another tool — ExZample.

ExZample

ExZample is a factory library I wrote which allows you to use defined examples in your struct modules. My goal was to enable developers to organize their factories in any way they want while still having access to convenient functions. For example, if you don't define an example function in GitHub.Repo, you can still build the struct with its default values:

ExZample.build_list(3, GitHub.Repo)
[
  %GitHub.Repo{
    name: nil
    # ...
  },
  %GitHub.Repo{
    name: nil
    # ...
  }
    %GitHub.Repo{
    name: nil
    # ...
  }
]

You can define an example function in the struct module, and ExZample will automatically use it if it's available.

# github/repo.ex
  def example do
    repo_name = sequence(:github_repo_name)

    %GitHub.Repo{
      id: 1296269,
      name: repo_name,
      full_name: "octocat/#{repo_name}",
      description: "This your first repo!",
      owner_id: 1,
      owner_url: "https://api.github.com/users/octocat",
      private: false,
      html_url: "https://github.com/octocat/#{repo_name}",
      url: "https://api.github.com/repos/octocat/#{repo_name}"
    }
  end

# test/support/test_helper.exs
# Sequences are created on app start
ExZample.create_sequence(:github_repo_name, &"repo-#{1}")

# then you can invoke the data example with ExZample module:
ExZample.build_list(3, GitHub.Repo)
[
  %GitHub.Repo{
    name: "octocat/repo-1"
    # ...
  },
  %GitHub.Repo{
    name: "octocat/repo-1"
    # ...
  }
    %GitHub.Repo{
    name: "octocat/repo-1"
    # ...
  }
]

If you prefer the atom naming mechanism of ExMachina, ExZample.DSL has you covered. You can define your factory in a module like this:

defmodule MyApp.Factory do
  use ExZample.DSL

  factory :github_repo do
    example do
      repo_name = sequence(:github_repo_name)
      %GitHub.Repo{
        id: 1296269,
        name: repo_name,
        full_name: "octocat/#{repo_name}",
        description: "This your first repo!",
        owner_id: 1,
        owner_url: "https://api.github.com/users/octocat",
        private: false,
        html_url: "https://github.com/octocat/#{repo_name}",
        url: "https://api.github.com/repos/octocat/#{repo_name}"
      }
    end
  end

  def_sequence :github_repo_name, return: &"repo-#{1}"
end

# Then use the aliased factory in your tests:
ExZample.build_list(3, :github_repo)
[
  %GitHub.Repo{
    name: "octocat/repo-1"
    # ...
  },
  %GitHub.Repo{
    name: "octocat/repo-1"
    # ...
  }
    %GitHub.Repo{
    name: "octocat/repo-1"
    # ...
  }
]

Here, we use macros provided by ExZample.DSL to define factories using the factory directive. We need to explicitly define the atom name :github_repo. Inside the factory body, we define the example block that builds a struct example.

Although the ExZample library may not be as well-known or extensively tested as ExMachina, ExZample was designed to pick up the exact features you need and ignore the rest. You may want to consider giving it a try. If you have any feedback or suggestions for improvements, feel free to contribute.

Next up, let's see what Faker has to offer.

Faker

Faker generates sample data that looks realistic but is fake. So, instead of using a sequence like those provided by ExMachina or ExZample, we can use Faker to generate false but genuine-looking data:

# test/support/test_helper.exs
# Ensure you start Faker before running your tests
Faker.start()

Faker.Internet.slug()
"sit_et"

Faker.Internet.slug()
"deleniti-consequatur"

Faker.Internet.slug()
"foo-bar"

The random nature of data generation can sometimes result in duplicate names, which may cause flaky tests. However, this is likely to occur only in rare situations, so it's still worth relying on Faker. Faker has an impressive collection of data samples and generation algorithms, including IP addresses, e-mails, URLs, addresses, and even Pokémon names. It's a great library to use in combination with the others presented here.

Finally, let's take a quick look at StreamData.

StreamData

StreamData generates a bunch of data samples for you. Unlike Faker, StreamData doesn't aim to generate realistic values. It creates raw, random data. You ask for a raw data type, and you get it. For example:

Enum.take(StreamData.string(:alphanumeric), 3)
#=> ["AcT", "9Ac", "TxY"]

StreamData returns a data generator that produces an infinite amount of random data. It implements the Enumerable protocol, so you can filter data using the Enum functions:

StreamData.string(:alphanumeric)
|> Enum.filter(&(String.length(&1) >= 5 and String.length(&1) <= 10))
|> Enum.take(1)
["hygT78ch"]

In the example above, we're only interested in grabbing strings between 5 and 10 characters long. StreamData was designed to be used with property testing, but you can also use its powerful data samples to build data examples for regular tests.

General Thoughts and Suggestions

I often wonder if it's worth deviating too far from the Elixir standard library when using a certain pattern. So I would like to invite you to aim for the following in your next Elixir project:

Try using your application's functions and rules to build test data. Create wrappers to simplify their usage if needed, with convenient defaults for testing.
When your application doesn't control the accuracy of your data, try to create example functions on the struct modules and use them to test your code.
Use Faker or StreamData to add some randomness, enriching your test data and making your tests less dependent on specific data.

If these three points still aren't enough for your data testing needs, embrace a factory library to bring more convenience and structure to your test
suite.

If you try the lean suggested approaches, you might realize how little you really need to be able to generate test data and have a healthy test suite.

Wrapping Up

In part one of this series, we summarised the ins and outs of Elixir test factories and fixtures. In the second part, we focused on generating data functions.

Finally, in this third and last part, we explored some test data libraries you can use for your Elixir application, including ExMachina, ExZample, Faker, and StreamData.

I hope you found this series helpful.

Good luck with your testing!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Generating Data Functions in Your Elixir App

Ulisses Almeida — Tue, 28 Mar 2023 10:56:25 +0000

In the first part of this series, we explored the ins and outs of Elixir test factories and fixtures. However, test factories bypass the rules of your Elixir application. Let's turn our attention to how we can avoid this by using data generation functions.

Let's dive straight in!

Creating Data With Your Elixir App's Public APIs

We'll use application code to generate test data. One of the biggest benefits of this method is that the generated data will be consistent with your application's rules. However, you need to write an extra layer of functions that your tests can easily use.

The Test Pyramid Strategy for Your Elixir App

Before jumping into the code, it's important to think about your application's public APIs and how data will be generated for each test strategy.

Saša Juric wrote about how to keep a maintainable test suite in Elixir. In the post, he described how the most important public API he needed to test in his app was the HTTP API layer, which is how his users interact with the app. For his app's context, it was worth having a lot of test HTTP requests, even though it might have some drawbacks on the test speed performance. In his test pyramid strategy, Saša favored interface-level tests over lower-level ones.

The image above shows different strategies you might choose for your app.

You might want a balanced pyramid with a good amount of tests in all layers or favor unit tests above everything else.

The tests in the top layer are slower and more difficult to maintain, and they overlap with lower-layer tests.

The high-level tests ensure that all modules work together, while the lower-level tests can tell you exactly which module or function is not working properly.

Your application's test pyramid strategy might have more layers with different names — for example, the term "unit" might mean completely different things depending on who you ask. Each strategy has its own trade-offs, and discussing them is out of the scope of this post.

The most important thing to take away, though, is that you need to understand the layers of your Elixir application and the type of data needed to write accurate functions that generate test data.

I'll give you some examples, but ultimately, it's up to you to decide what works best for your team and source code.

Extract Business Logic from Your Web Code

It's a well-known and common practice to split business logic from web code in web applications built with the Phoenix framework. In Phoenix, we usually call these business modules "context" modules.

These modules typically interact with databases, external services, other context modules, and a lot of other functions. The functions in context modules are usually the public API for your web layer.

An Example Using Helpers in Elixir

If you want to invest in a lot of tests on the context level, you can create helpers for the most-needed resources. For example, let's say you have these modules:

# lib/my_app/accounts.ex
defmodule MyApp.Accounts do
  def signup_user(username, password) do
  # ...
  end
end

# lib/my_app/profiles.ex
defmodule MyApp.Profiles do
  def create_author_profile(user) do
  # ...
  end
end

# lib/my_app/profiles.ex
defmodule MyApp.News do
  def create_post(post_contents, author) do
  # ...
  end
end

Here, we have the Accounts, Profiles, and News contexts. Suppose we want to test the MyApp.News.create_post/2 function. We first need to create an Author profile. To create an Author profile, we need a User account.

If we have more functions in the News context, or if more contexts need an Author, we always have the tedious task of creating these chained struct relationships.

We can create helpers and make them available to our tests that need valid Author and User structs in the system. To make these helpers only available in the :test environment, we need to tell Mix, Elixir's project build tool, where to find these files. You can update mix.exs with the following configuration:

elixirc_paths: (
  case Mix.env() do
    :test -> ["lib", "test/support"]
    _ -> ["lib"]
  end
 )

We tell Mix to compile the files in the test/support directory with the lib directory when building the :test environment. Of course, you can use any directory name you want, but test/support is a widely used convention. This way, you can create modules only available for a :test environment.

Here's an example of some helpers:

# test/support/helpers.ex
defmodule MyApp.Helpers do
  def create_user(opts \\ []) do
    username = Keyword.get(opts, :username, "test_user")
    password = Keyword.get(opts, :password, "p4ssw0rd")

    {:ok, user} = MyApp.Accounts.signup_user(username, password)
  end

  def create_author(opts \\ []) do
    user = Keyword.get_lazy(opts, :user, &signup_user/0)

    {:ok, author} = MyApp.Profiles.create_author_profile(user)
  end
end

In the MyApp.Helpers module above, we create wrappers over our application's core API with convenient defaults to use in tests. In real life, we wouldn't create users with the password "p4ss0wrd" by default, but for our test scripts, it's fine.

We also use Keyword.get/3 to overwrite important attributes of the created resources. This avoids unnecessary side effects, especially when the caller provides us with a user to the create_author/1 helper. That's why we use Keyword.get_lazy/3 in the create_author/1 helper. get_lazy will only invoke the signup_user function if the :user key doesn't exist.

Invoking and Customizing Helpers in Elixir

We can also be strict with data patterns here since we don't expect the creation of users or authors to fail. The caller can invoke and customize these helpers for the needs of the test. For example:

user = create_user(username: "test_user_2")

create_author(user: user)

When writing your tests, you can use the examples above and write something similar:

# test/news/news_test.exs

# module definition and stuff

alias MyApp.Helpers

# maybe other tests in the middle

test "creates new post with given the contents" do
  author = Helpers.create_author()

  assert {:ok, post} = News.create_post("My first post!", author)
end

In the code above, we use alias MyApp.Helpers to make our helper module functions accessible with a few characters. It makes the use of these functions as convenient as the test factories provided by ExMachina.

A cool advantage of this approach is that if your editor uses a language server like ElixirLS, you can quickly discover or navigate these functions easily. The build(:user) pattern doesn't allow the editors of today to track the definition code directly.

Breaking the Helpers Module Down Into Other Modules

As the Helpers module grows and gets more complex, you might want to break it down into different modules.

For example, you could break it down by context - AccountsHelper, ProfilesHelpers, etc. The best naming and file organization depends on each application's needs, so I'll leave it up to you.

The Helpers example discussed here might have left you thinking: "This looks like factories." And you're not wrong!

This is a different implementation of test factories. Instead of creating examples of data uncoupled with your application's rules, here we tie your application's rules and your test examples together.

It works like the factory pattern because your tests aren't coupled with the way the underlying struct is built.

This example satisfies the demands of the context modules layer, but what about the other layers? In the next section, we'll explore when using public APIs isn't enough and how to generate data examples for these cases.

Beyond Public APIs: Data Examples

One of the main advantages of using your Elixir app's public API to generate data for your tests is that the data will comply with your app's rules and database constraints.

While the speed and coupling with the database may not always be ideal, this is a small price to pay to ensure your data's validity.

As your app grows in complexity, you may need to add a layer of tests that run in memory to improve performance. It is also not uncommon for systems to talk to other systems through network APIs. In these cases, your application likely won't have enough control over the rules to build valid data. You may need to rely on API specifications provided by the remote system and snapshot some examples to use in your tests.

Data examples built in memory and decoupled from database or application rules can be extremely helpful.

Let's explore a lightweight — and somewhat controversial — method of generating data examples within your application modules.

Writing Data Examples in Data Definition Modules

Years ago, a colleague invited me to watch an episode of Ruby Tapas presented by Avid Grimm.

Inspired by the book Practical Object-Oriented Design by Sandi Metz, Grimm talks about writing data examples in the modules where the data definition lives. This can serve as executable documentation and also be used in tests.

Some people might not be big fans of mixing test data with application code. But, as long as it is clear that the data is meant as an example and not for production use, it can be a useful technique.

For example, let's say you're writing a GitHub client, and you're defining the Repo struct:

# github/repo.ex
defmodule GitHub.Repo do
  defstruct [
    :id,
    :name,
    :full_name,
    :description,
    :owner_id,
    :owner_url,
    :private,
    :html_url,
    :url
  ]

  @typespec t :: %__MODULE__{
    id: pos_integer(),
    name: String.t(),
    full_name: String.t(),
    description: String.t(),
    owner_id: pos_integer(),
    owner_url: String.t(),
    private: boolean(),
    html_url: String.t(),
    url: String.t()
  }
end

Here, we define the GitHub.Repo struct and document the keys type with typespec. While this provides a lot of information, extra documentation can help readers understand the data's nuances.

In this example, can you tell the difference between the name and full_name? Or url and html_url? It's hard to tell, right?

We can make it clearer. Let's add an example of the values:

# github/repo.ex
def example do
  %GitHub.Repo{
    id: 1296269,
    name: "Hello-World",
    full_name: "octocat/Hello-World",
    description: "This your first repo!",
    owner_id: 1,
    owner_url: "https://api.github.com/users/octocat",
    private: false,
    html_url: "https://github.com/octocat/Hello-World",
    url: "https://api.github.com/repos/octocat/Hello-World"
  }
end

Defining the example/0 function, which returns an example of the data structure, can be useful in various ways.

For example, if you're exploring the code in IEx (the Elixir interactive shell), you could invoke the function to quickly experiment with a complex function call. Livebook material (interactive documents that allow users to run code) could use these functions to show an example of the data shape. In production, the example values could be used as hints for form fields.

Customize Key Values of the `example` Function

One of the main purposes of an example function, however, is to create data for tests. You can make the example function even more useful by allowing the caller to customize the key values. Here's an example:

def example(attributes \\ []) do
  struct!(
    %GitHub.Repo{
      id: 1296269,
      name: "Hello-World",
      full_name: "octocat/Hello-World",
      description: "This your first repo!",
      owner_id: 1,
      owner_url: "https://api.github.com/users/octocat",
      private: false,
      html_url: "https://github.com/octocat/Hello-World",
      url: "https://api.github.com/repos/octocat/Hello-World"
    },
    attributes
  )
end

We use Elixir's struct!/2 kernel function to build structs dynamically. The great thing about struct! is that it fails if any unexpected keys are set. Now we can invoke the example/1 function and customize the data in any way we want:

test "renders anchor tag to the repository" do
  repo = GitHub.Repo.example(html_url: "http://test.com")

  assert hyperlink_tag(repo) =~ "<a href=\"http://test.com\""
end

In the example above, we can easily create a GitHub.Repo and customize its html_url key for our test.

Similarities to Test Factories for Elixir

"Wait, isn't this just factories again?" you might be wondering. And you're right! It's similar to the factory mechanism found in other libraries.

While this can make the functions easy to navigate and localize using your editor, if you rename the GitHub.Repo module, you'll need to find and replace a bunch of tests. However, modern editors are usually powerful enough to handle the task in a single command, so this shouldn't be a big issue.

Another interesting aspect of providing data examples in your struct modules is that you don't need to organize your factory files, as they are organized together within your application code.

The popular ExMachina factories, your own helper functions that use your application's rules, and data example functions in your structs are all examples of factory pattern implementations.

Using ExMachina

Using ExMachina to create your factories helps you separate
test data from production code and gives you convenient functions. For example, when you define a factory using ExMachina, you can use that definition to generate test data with different strategies, like:

build to generate in-memory structs
insert to load data in a database
build_list to generate multiple items in a list
string_params_for to create map params with keys as strings like you would receive in a Phoenix controller

These are a few examples of functions that ExMachina can offer — and it has more! The convenience of these functions is debatable, though. For example, having the insert function so conveniently available could make you unnecessarily insert things into a database. And how often do your controller parameters' keys and value formats match the schema attributes generated by the string_params_for to make it really worth it? However, these convenient functions do the job for simple cases and offer a foundation for your entire test suite.

Up Next: Elixir Libraries for Test Data

Now that you understand the fundamental techniques for generating test data in Elixir, you should be able to do this for your own project without much trouble.

In the third and final part of this series, we'll dive into some Elixir libraries for your test data, including ExMachina, ExZample, Faker, and StreamData.

Happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

An Introduction to Test Factories and Fixtures for Elixir

Ulisses Almeida — Tue, 21 Mar 2023 11:31:28 +0000

Writing tests is an essential part of any Elixir developer's routine. We're constantly chasing knowledge on how to write better tests, improving their speed and readability.

In this three-part series, we'll explore test data generation in Elixir. Whether you are a mid-level or senior-level Elixir developer, this series will provide valuable insights to help improve the testing process for your projects.

Let's start by digging into where test factories come from and why they are so popular in Elixir.

What are Test Factories in Elixir?

Test factories are functions for generating data, commonly used in the :test environment.

Test factory libraries like ExMachina are widely used among Elixir developers. If you look at the hex.pm stats, you'll see that ExMachina downloads aren't far behind the most popular Elixir database library: Ecto.

They allow you to create complex data structures using a very convenient interface that requires few inputs. Here's an example using ExMachina:

iex> user = ExMachina.insert(:user)
%User{
  id: "usr_xlkt"
  name: "Abilidebob"
  accounts: %Account{
    id: "acc_xktt"
    #....
  }
  #...
}

With a few characters, I can grab an example of a :user in the system. Test factories are inspired by the factory method pattern, a design pattern that allows a caller to create objects without knowing the specific module or class of the data that will be created.

Okay, this might sound a little bit complicated! Let's look at the following ExMachina code:

user = ExMachina.insert(:user)

do_something(user)

In this example, ExMachina is using the :user atom to dispatch to a function that can create examples of %User{} structs. If you write your tests in a dynamic style, you can rename the User struct without refactoring any tests, except the factory function.

The factory pattern isn't only helpful in tests. You can also use it to create or dispatch functions from dynamic sources, such as user inputs.

For example, you could use a factory function to create an account based on a user's input. A factory function is useful here because your application's users don't know the name of your source code's modules or functions — and you should keep it that way. You need to create a mechanism that links a user's input with the correct constructor:

def new_account("hobbyist"), do: HobbyAccount.new()
def new_account("professional"), do: ProfessionalAccount.new()
def new_account("enterprise"), do: EnterpriseAccount.new()
def new_account(account_type), do: raise ImpossibleAccount, account_type

In the previous example, I built a factory function named new_account/1 that takes a string as an argument and uses pattern matching to determine which function to call. For example, if a user selects the "Hobbyist" option from a selection box, the application will receive a "hobbyist" string as the input. The new_account/1 function can use the string to dispatch the HobbyAccount.new/1 function and generate a HobbyAcccount struct.

Why Use Test Factories for Elixir?

Test factories are good options for long-term test suite maintainability because of the following:

Learnability - factories document what data structures can look like in production
Reusability - data examples generated by factories can be reused in many different tests
Productivity - developers can invoke complex data examples from factories by typing a few characters instead of building new examples from scratch every time
Changeability - tests that rely on central factories always have the most up-to-date examples

These properties aren't inherently part of the factories alone. A software development team's discipline is required.

For example, if you don't provide relevant real-world examples in your factories, they will not be a source of knowledge for developers. If your factories produce values that other developers tend to override, they will more likely stop using those factories because they are not convenient anymore.

But when a factory pattern is well-maintained, it helps developers to maintain a sustainable test suite that lasts for a long time.

Now we'll turn our attention to test fixtures.

Test Fixtures in Elixir

In testing, a fixture is data that we prepare before running a test. Factories are functions that generate test data on demand.

Factories can complement fixtures — you don't necessarily
have to opt for one over the other. For example, you can prepare a test fixture that uses a factory function to insert data in a database.

That's why we can sometimes use these terms interchangeably and still be understood. Often, people associate the term "fixtures" with the feature in Ruby on Rails. Ruby On Rails fixtures have the following properties:

Data is defined in YAML files, not inline in the tests.
All defined data is loaded in the database before each test run.
You can reference these fixtures through dynamic methods generated by the framework.

However, different frameworks or libraries implement test fixtures differently. After all, the main purpose of test fixtures is to provide a common starting point for your tests.

There are basically two main kinds of test fixtures: inline
and implicit.

Inline Test Fixtures

The simplest inline fixture you can do in Elixir is the following:

test "creates an author profile" do
  user = MyApp.Repo.insert!(%User{id: "usr_123", name: "Abilidebob"})

  # rest of the test code
end

With a basic Repo.insert!/1, we make a database fixture for a test. That test can rely on the user with the id "usr_123" always being present in the database.

This example code assumes the test suite uses Ecto sandbox setup, which always reverts database changes made during a test after its execution.

So we don't need to worry at the beginning of the test if the user with the same id is already inserted in a previous execution. We can even use a factory function to insert the data, and it still would be correct to call it a test fixture. This fixture is explicit and inline with the tests.

Implicit Test Fixtures

Like in Ruby on Rails, it's possible to create implicit fixtures that multiple tests can share using the ExUnit.CaseTemplate test template:

# test/support/data_case.ex
defmodule MyApp.DataCase do
  use ExUnit.CaseTemplate

  # ... bunch of other code here

  setup _context do
    user = %User{id: "usr_123", name: "Abilidebob"}
    %{user: MyApp.Repo.insert!(user)}
  end
end

The code above uses the setup directive to make all tests that use this template insert a user. Then we return a map, where the key is :user, and the value is our recently created user. Any test can quickly reference this by using the key :user and grabbing it from the test context. For example:

defmodule MyApp.MyTest do
  use MyApp.DataCase

  test "hello world", %{user: user} do
    # then I use my `user` here
  end
end

ExUnit Contexts

ExUnit test contexts are a flexible mechanism where you can build things to share between multiple tests.

The biggest disadvantage of fixtures is their impact on test suite speed and test isolation if you share them too much. The speed can decrease because all the code you put in setup blocks of test templates always runs before each test, even when you don't need the data from the test context.

So be careful how much data you add and how multiple modules use the templates. Otherwise, as the speed decreases, implicit coupling between multiple tests on the same test fixtures can also increase. A small change to a test fixture shared in this way can make multiple tests fail for no obvious reason.

Sharing Bypass Instances Between Multiple Tests

Developers favor factory libraries over Ruby On Rails fixtures because they'd rather have testing data explicitly invoked from tests than implicit data generated far from where it is needed. In other words, they prefer inline test fixtures.

Explicitness contributes to better long-term maintenance. Implicit test fixtures are better for things that are very cheap to build and have a very general purpose, and low coupling with testing specifics.

A good example is the Bypass instance:

# test/support/data_case.ex
defmodule MyApp.GithubIntegrationCase do
  use ExUnit.CaseTemplate

  # ... bunch of other code here

  setup _context do
    bypass = Bypass.open()
    client = GitHub.new(endpoint: "http://localhost:#{bypass.port}/")
    %{bypass: bypass, client: client}
  end
end

In the example above, I prepare a test template for my GitHub client integration tests. The setup code creates a fake HTTP server using Bypass. The GitHub client uses this server to make HTTP requests during the tests using the client settings.

Now, all tests that use MyApp.GithubIntegrationCase can use the :bypass and :client keys to pull the fake server and its settings, respectively, from the test context.

Here is how these fixtures might be used in a test:

defmodule MyApp.MyTest do
  use MyApp.GithubIntegrationCase

  test "user registration", %{bypass: bypass, client: client} do
    Bypass.expect(bypass, :post, "/users") do
      # ... setup expectations and return a success response
    end

    # Use the `client` tag to make HTTP requests with the `GitHub` client
    {:ok, user} = GitHub.create_user(client, name: "Abilidebob")

    # Assert that the user is correct
    # ...
  end
end

If you've worked with other frameworks, such as Ruby on Rails, you might be surprised that Elixir doesn't have a popular fixture library.

I believe that's because the ExUnit test context can share test fixtures pretty easily. The fixture pattern is usually discouraged due to its impact on test suite speed and maintainability.

Test Factories and Your Elixir App's Rules

In his excellent post Towards Maintainable Elixir: Testing, Saša Jurić provides valuable tips on how to maintain a test suite. The focus of the post is on increasing your confidence in tests and reducing test overlap. There is no mention of test factories. Instead, Saša uses his own application interface to prepare test data. The reason he avoids test factories is because of their biggest problem: they bypass your application's rules.

When you set up a test factory for your test suite, the data examples it generates are completely detached from your application code's rules.

For example, let's say you have a User entity with an is_author flag, but that flag is only set to true if the user has one author profile. Here's an example of how the production code would look:

def create_author_profile(user) do
  author_profile = build_author_profile(user)
  user = Ecto.Changeset.change(user, %{is_author: true})

  Multi.new()
  |> Multi.update(:user, user)
  |> Multi.insert(:author_profile, author_profile)
  |> Repo.transaction()
end

Let's say that you now want to create the :user and :author factories for your test factory.

Here's an example using the ExMachina library:

def user_factory do
  %User{name: "Abili"}
end

def author_factory do
  %Author{
    name: "Mr. DeBob",
    user: build(:user, is_author: true)
  }
end

In the example above, the factories are straightforward. They build structs and assign values that make sense. However, have you noticed the is_author: true part? That tiny part is actually the biggest problem with factories: we have to duplicate our business rules!

Imagine having to remember to replicate every single aspect of multiple data modifications in your factories. Not only does this add extra code that needs to be maintained, but it can also mislead developers into trusting invalid data that your app cannot produce.

The invalid data generated by factories can lead you to write unnecessarily defensive code. For example, if someone forgets to include is_author: true in the factory, you might write code like this:

if user.is_author || Profiles.find_author(user_id) do
# logic for users that are author here

In the previous example, a check is written for the is_author flag. It is assumed the flag isn't always reliable — after all, the factory generates authors with is_author: false users. As a result, we decide to make a query in the database to double-check if the user is an author.

As you can see, factories that are decoupled from application rules aren't always perfect and can cause problems if they are not properly maintained.

Despite their issues, factories are a popular pattern for generating test data in Elixir. They offer convenient functions to build complex data structures, and you can invoke them on demand.

However, using factories adds an extra layer of maintainability, an extra worry to put in your team's minds. The worry is usually negligible when your application code is small, but can become problematic as your business and code grow more complex.

Up Next: Generating Data Functions in Your Elixir App

In this first part of our three-part series, we introduced test factories and test fixtures for Elixir. We then explored the biggest issue with test factories: the fact that they bypass your application's rules.

In part two, we'll explore how you can combat this by creating data generation functions using your application's rules.

Until then, happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

DEV Community: Ulisses Almeida

Test Data Libraries for Elixir

Why Choose a Test Data Library for Your Elixir App?

Elixir Libraries for Your Test Data

ExMachina

ExZample

Faker

StreamData

General Thoughts and Suggestions

Wrapping Up

Generating Data Functions in Your Elixir App

Creating Data With Your Elixir App's Public APIs

The Test Pyramid Strategy for Your Elixir App

Extract Business Logic from Your Web Code

An Example Using Helpers in Elixir

Invoking and Customizing Helpers in Elixir

Breaking the Helpers Module Down Into Other Modules

Beyond Public APIs: Data Examples

Writing Data Examples in Data Definition Modules

Customize Key Values of the example Function

Similarities to Test Factories for Elixir

Using ExMachina

Up Next: Elixir Libraries for Test Data

An Introduction to Test Factories and Fixtures for Elixir

What are Test Factories in Elixir?

Why Use Test Factories for Elixir?

Test Fixtures in Elixir

Inline Test Fixtures

Implicit Test Fixtures

ExUnit Contexts

Sharing Bypass Instances Between Multiple Tests

Test Factories and Your Elixir App's Rules

Up Next: Generating Data Functions in Your Elixir App

Customize Key Values of the `example` Function