DEV Community: Nyoman Abiwinanda

Solving Data Consistency in Elixir with Ecto's prepare_changes

Nyoman Abiwinanda — Sun, 26 Jan 2025 16:58:39 +0000

When developing an application, it's common for the state of one piece of data to depend on the state of another. This interdependence introduces complexity, as changes in one dataset must remain consistent with changes in others to avoid conflicts or inaccuracies. This challenge is often known as the data consistency problem.

Intro to Data Consistency Problem

Consider the following simple example app where you have a bank and an account that belongs to a bank.

defmodule ElixirApp.Bank do
   use Ecto.Schema

   schema "banks" do
      field(:name, :string)
      field(:is_active, :boolean)

      timestamps()
  end
end

defmodule ElixirApp.Account do
   use Ecto.Schema

   schema "accounts" do
      field(:name, :string)
      field(:currency, :string)
      field(:is_active, :boolean)

      belongs_to(:bank, Bank)

      timestamps()
   end
end

In this app, a bank has an "active" status, and if the bank becomes inactive, all accounts associated with it must also be set to inactive. This data consistency rule can be implemented within the bank's domain or context module. For instance, you might implement an update_bank/2 function as shown below:

defmodule ElixirApp.Banks do
   alias ElixirApp.{Account, Bank}
   alias ElixirApp.Repo

   import Ecto.Query, warn: false

   def update_bank(%Bank{} = bank, attrs) do
      with {:ok, updated_bank} <- do_update_bank(bank, attrs),
           {:ok, _num_of_deactivated_accounts} <- maybe_deactivate_bank_accounts(updated_bank) do
         {:ok, updated_bank}
      end
   end

   defp do_update_bank(%Bank{} = bank, attrs) do
      bank
      |> Bank.changeset(attrs)
      |> Repo.update()
   end

   defp maybe_deactivate_bank_accounts(%Bank{active: true}) do
      {:ok, 0}
   end

   defp maybe_deactivate_bank_accounts(%Bank{id: bank_id, active: false}) do
      {updated_accounts_count, _} =
         from(a in Account,
            where: a.bank_id == ^bank_id,
            update: [set: [is_active: ^false]]
         )
         |> Repo.update_all([])

      {:ok, updated_accounts_count}
   end 
end

Within the update_bank/2 function, each time a bank's status is updated, it’s passed to the maybe_deactivate_bank_accounts/1 function. This function ensures that if the bank is now inactive, all associated accounts are also deactivated, maintaining consistency across related data.

The previous implementation works but it has one major drawback that needs to be handled by developers consistently.

All functions that perform an update to a bank must be ensured to implement the same data consistency logic.

This means that:

The data consistency logic code can potentially be duplicated across different modules as the application evolves. I.e. the update operation to a bank takes place outside the Banks context module itself.
The data consistency logic becomes “unknown unknown” in the long run. This manual step introduces a risk of inconsistency if it’s overlooked, potentially leading to data integrity issues.

The key takeaway is that this implementation could lead to an increase in "unknown unknowns" over time. This risk arises because developers may unintentionally overlook the need for consistent data handling, leading to hidden bugs and data inconsistencies. To mitigate this, it’s essential to encapsulate or abstract this data consistency logic in a more centralized, reliable place. By doing so, we reduce code duplication and ensure that new developers don't need to be aware of this requirement upfront when adding features. This is where Ecto's prepare_changes/2 function comes in, offering an ideal solution for enforcing consistency at the database layer automatically.

How to Use Ecto `prepare_changes/2`

As explained in the official documentation, the prepare_changes/2function allows you to specify a function that the repository will execute upon insert, update, or delete operations. This means that any function passed to prepare_changes/2 will only run when the changeset is submitted to the Repo module. This approach is especially beneficial for abstracting or encapsulating data consistency logic directly within the schema's changeset function, allowing us to enforce consistency while keeping the changeset function itself pure and focused on validation and data transformation.

Let’s use the example app before to see how this function can be used inside the Bank changeset function to maintain consistency between banks and accounts.

defmodule ElixirApp.Bank do
   use Ecto.Schema
   alias Ecto.Changeset
   import Ecto.Changeset
   import Ecto.Query, warn: false

   schema "banks" do
      field(:name, :string)
      field(:is_active, :boolean)

      timestamps()
   end

   @doc false
   def changeset(bank, attrs) do
     bank
     |> cast(attrs, [:name, :is_active])
     |> validate_required([:name, :is_active])
     |> prepare_changes(&ensure_accounts_status/1)
   end

   defp ensure_accounts_status(%Changeset{action: :update} = changeset) do
      bank_id = get_field(changeset, :id)
      bank_active_status = get_field(changeset, :is_active)

      if bank_active_status == false do
         from(a in "accounts",
            where: a.bank_id == ^bank_id,
            update: [set: [is_active: ^false]]
         )
         |> changeset.repo.update_all([])
      end

      changeset
   end

   defp ensure_accounts_status(%Changeset{action: _} = changeset) do
      changeset
   end
end

In the implementation above, the code inside the ensure_accounts_status/1 function is only executed when the bank's status is updated to "inactive." This ensures that account status changes occur only when necessary, maintaining efficiency. Additionally, it's important to note that the changeset provided by prepare_changes/2 includes access to the repository (repo), allowing the developer to perform CRUD operations as part of the data consistency logic. This makes it possible to handle complex business rules directly within the changeset, without needing to manually invoke operations outside of it.

The use of prepare_changes/1 has the following pros and cons

Pros:

Encapsulating the data consistency logic down to the changeset function means that any updates to banks that use the bank’s changeset function will be guaranteed to produce a consistent result with other data which in this case accounts.
Less chance of duplicating the same data consistency logic across different domain functions as the application evolves.
Using the changeset function to perform a data validation will not make any database call (at least yet until it is passed to the Repo module) therefore keeping the changeset function pure.

Cons:

Performing the data consistency logic inside the changeset could increase the sense of “magic” when performing CRUD operations to the bank. It can become slightly unobvious of the source of the operation that makes accounts inactive. However, this could justified as long as it simplifies your application development in a way that reduces bugs or errors.
Logic duplication could still happen if the Bank module has multiple changeset functions and each function must perform the same data consistency operation. This however can be slightly simplified by abstracting the data consistency logic inside a private function and then re-using it across different changeset functions.

Conclusion

As effective as the prepare_changes/2 function can be for addressing data consistency problems, it’s important to know that there should be limits to its use. For example, it’s not advisable to overload prepare_changes/2 with complex logic or side effects, such as sending notifications, creating new records of bank, or other operations that could obscure the flow of the application. Keeping the function focused on its core responsibility—ensuring data consistency—will help maintain clarity and make your application’s behavior more predictable to other developers.

Now that you have an alternative approach for tackling data consistency issues with Ecto, you can try it out in your project and assess whether it helps reduce the likelihood of data inconsistency in your application. By keeping your code clean and leveraging Ecto's powerful tools like prepare_changes/2, you’ll be able to build more reliable and maintainable applications. Happy coding!

GitHub Action: Adding Post Steps in Composite Actions

Nyoman Abiwinanda — Sun, 23 Jul 2023 11:48:58 +0000

Often when we create CI/CD pipelines, we create some steps where the purpose is to setup something for the upcoming steps. For example, you might add a step to install aws cli, install commands with apt-get, restoring build artifact from a cache, and etc.

These setup commands tend to be copy-pasted across different pipelines making them less DRY. However, in many CI/CD tools or platforms such as GitHub Action, we could define an action, much like a function, to extract the repeated steps into a single reusable action. This process where we extract a bunch of steps to make them reusable can be implemented with a composite action in the GitHub platform.

To quickly demonstrate GitHub composite actions, let say we have the following CI workflow under .github/workflows directory that performs a unit test

# Defined in .github/workflows/ci.yml
name: Some CI workflow

...

jobs:
  unitTest:
    name: Run Unit Test
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      # Setup step to speed-up dependencies installation 
      # and app compilation
      - name: Restore build artifacts
        run: ... # run commands to restore app build artifacts

      - name: Install app dependencies
        run: ... # run commands to install app dependencies

      - name: Compile app
        run: ... # run commands to compile app

      - name: Run test
        run: ... # run commands to run unit test

      ...

You realize later that the step Restore build artifacts can be used for other existing or future workflows but instead of copy-pasting the code to other workflows, you instead create a composite action called cache under .github/actions/cache directory

# Defined in .github/actions/cache/action.yml
name: Cache

description: "Restore build artifacts"

runs:
  using: "composite"
  steps:
    - name: Cache
      run: ... # run commands to restore app build artifacts

by this stage we have the following file tree

.
├── .github
│   ├── actions
│   │   └── cache
│   │       └── action.yml 
│   │   
│   └── workflows
│       └── ci.yml
└── ...

we could reference our defined cache action inside our CI as follows

name: Some CI workflow

...

jobs:
  unitTest:
    name: Run Unit Test
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      # Reusing the cache action
      - name: Restore build artifacts
        uses: ./.github/actions/cache

      ...

So far, composite action works beautifully to make your workflows DRY-er however there is one drawback which GitHub composite action don't natively support (at least by the time of writing this post) which is to run post steps.

Post steps are simply steps that GitHub will run after a workflows has reach its final steps. For the cache example, we might want to run a post step to rebuild the app build artifacts for future CI. We could define this step as another action however this gives us inconvenience as we have to keep remembering to put the rebuild action at certain step in the workflows. How can we do better and let the composite action handle this post steps seamlessly for us?

After doing a bit of research, I came across the following GitHub issue: Support pre and post steps in Composite Actions in the GitHub action runner repo. In the discussion, I found out that we could use a node action to indirectly perform post steps in a composite actions.

To run post steps in composite actions, first we will re-use the with-post-step action that is defined in here and in this post we wouldn't look into the implementation detail of the code.

.github/actions/with-post-step/action.yml

# Copied from: https://github.com/pyTooling/Actions/blob/main/with-post-step/action.yml
name: With post step

description: "Generic JS Action to execute a main command and set a command as a post step."

inputs:
  main:
    description: "Main command/script."
    required: true
  post:
    description: "Post command/script."
    required: true
  key:
    description: "Name of the state variable used to detect the post step."
    required: false
    default: POST

runs:
  using: "node16"
  main: "main.js"
  post: "main.js"

.github/actions/with-post-step/main.js

// Ref: https://github.com/pyTooling/Actions/blob/main/with-post-step/main.js
const { spawn } = require("child_process");
const { appendFileSync } = require("fs");
const { EOL } = require("os");

function run(cmd) {
  const subprocess = spawn(cmd, { stdio: "inherit", shell: true });
  subprocess.on("exit", (exitCode) => {
    process.exitCode = exitCode;
  });
}

const key = process.env.INPUT_KEY.toUpperCase();

if ( process.env[`STATE_${key}`] !== undefined ) { // Are we in the 'post' step?
  run(process.env.INPUT_POST);
} else { // Otherwise, this is the main step
  appendFileSync(process.env.GITHUB_STATE, `${key}=true${EOL}`);
  run(process.env.INPUT_MAIN);
}

The file tree should now looks something like this

.
├── .github
│   ├── actions
│   │   ├── cache
│   │   |   └── action.yml 
│   │   └── with-post-step
│   │       ├── action.yml 
│   │       └── main.js
│   │   
│   └── workflows
│       └── ci.yml
└── ...

once we have with-post-step action defined, go back to cache action.yml definition and use the with-post-step action to define the post steps of the cache

# Defined in .github/actions/cache/action.yml
name: Cache

description: "Restore and rebuild build artifacts"

runs:
  using: "composite"
  steps:
    - name: Cache
      uses: ./.github/actions/with-post-steps
      with:
        main: ... # run commands to restore app build artifacts
        post: ... # run commands to rebuild app build artifacts

Once the post steps is in place for the cache action, using the action in any workflow will add an additional post steps when the workflow is running.

If you are interested to try out this yourself, the code in this post can be found in my composite-post-gha repo.

Reference

Controller-Free Approach for File Export in Phoenix LiveView

Nyoman Abiwinanda — Tue, 04 Jul 2023 21:46:31 +0000

In a recent project involving LiveView, I was tasked with building a page to display an accounting report. Alongside this requirement, it was important to enable users to export the report as an Excel file. While the common approach involves creating a regular Phoenix controller with an API to generate the Excel file, I sought to explore a more efficient and seamless solution. In this post, I will share an alternative method that allows us to export or generate an excel (or any other) file directly from the LiveView itself, eliminating the need for a separate controller.

The Traditional Approach

Traditionally, the export feature is implemented by creating a Phoenix controller responsible for building the Excel file and sending it as a response. Here's an example of how it's typically done:

defmodule SomeController do
  use AppName, :controller

  # This function is used to generate and export the report Excel file
  def export(conn, params) do
    # Fetch the report data from the DB...
    report_data = fetch_report_data_from_db(params)

    # ...and then build the excel file
    raw_content = generate_excel_file(report_data)

    conn
    |> put_resp_content_type("text/xlsx")
    |> put_resp_header("content-disposition", "attachment; filename=report.xlsx")
    |> text(raw_content)
  end
end

While this method is widely used, there is one thing that bothers me. Since the controller independently fetches the report data from the database and generates the Excel file, it duplicates the data retrieval process already performed in the LiveView page. This redundancy raises a question: Can we export the report data to an Excel file without relying on a Phoenix controller? Is it possible to construct the export file directly within the LiveView, reusing the report data stored in the LiveView socket?

An Alternative Solution

To address this, my co-workers and I tried to devised an approach that enables exporting and downloading files using Phoenix LiveView alone. In our LiveView process, which handles report rendering and houses the download report button, we made a modification. Instead of making an HTTP request to a Phoenix controller, we altered the download button to trigger a phx-click event named export_xlsx and we also add a phx-hook attribute set to PushFile to ensure that it triggers the required hook callback in the clients as you will see later.

<button id="export-btn" phx-click="export_xlsx" phx-hook="PushFile">
   Export to excel
</button>

def handle_event("export_xlsx", _, socket) do
   # Reusing the report data fetched during the LiveView access
   report_data = socket.assigns.report_data

   # Building the Excel file
   raw_content = generate_excel_file(report_data)

   {:noreply, push_file(socket, raw_content, "report.xlsx")}
end

defp push_file(socket, binary, filename) do
   prefix = "data:#{MIME.from_path(filename)};base64,"

   Phoenix.LiveView.push_event(socket, "download-file", %{
      base64_data: prefix <> Base.encode64(binary),
      filename: filename
   })
end

By constructing the Excel file within the LiveView process, we can conveniently utilize the pre-existing report data stored in the LiveView socket's assigns, which is fetched during the process mount.

What happens after that is where we spent the most time with. We take raw data of the excel file and then send it to client via liveview js (hook) event called download-file. This is done by the function push_file/3 which is just a wrapper we build around the function Phoenix.LiveView.push_event/3. In this case we send the file raw data in the form of base64 string.

Once we sent the file raw data to client, the client has to handle the event and do something with it. To handle such event, we define a hook inside the phoenix-live-view.js file as follows

// inside phoenix-live-view.js
const Hooks = {};

// Create a new hook that handle the "download-file" event
Hooks.PushFile = {
  mounted() {
    this.handleEvent("download-file", function (event) {
      var element = document.createElement('a');
      element.setAttribute('href', event.base64_data);
      element.setAttribute('download', event.filename);
      element.click();
      element.remove();
    });
  }
}

let liveSocket = new LiveSocket("/live", Socket, {
  hooks: Hooks,
});

essentially what the hook does is to create an HTML link tag, click the tag (to download the file), and then removes it. The created link tag looks something like this.

<!-- The excel file is stored using data URL -->
<a href="data:text/xlsx;base64,{base64_data}" download="filename.xlsx"></a>

Once the hooks is in place, we were able to download or export the report into an excel file without the need of Phoenix controller 🎉

Pros & Cons

By eliminating the use of a controller and integrating the Excel file creation directly within the liveview process, we could reduce the download time equivalent to the time it takes to load the report (from the database) in the liveview.

While the impact of this improvement may not be substantial in certain cases, it becomes more noticeable when the report page takes some time to load. Once the report is loaded, users no longer have to wait an additional amount of time to download the report file. This enhancement contributes to a more enjoyable user experience, lending a sense of responsiveness to the download feature.

However, it's important to consider that removing the controller and relying solely on liveview introduces a few drawbacks. Two immediate concerns come to mind:

Various browsers may impose restrictions on the size of files transmitted using data URLs. Consequently, if the file size exceeds these limitations, the data URL approach may not function as intended.
In line with the previous drawback, as file sizes increase, transmitting the file using data URLs could consume additional server CPU resources due to the base64 processing involved.

Both the controller-based approach and the approach mentioned in this post have their merits. If download time and responsiveness are not critical factors, the controller approach may be preferable. Ultimately, the choice of approach is up to you, depending on the specific requirements and priorities of your project.

VSCode Snippet for Reducing Boilerplate Memorization in Elixir

Nyoman Abiwinanda — Fri, 11 Nov 2022 19:51:23 +0000

If you have been writing software in Elixir for quite some time, you might feel that Elixir code can feel like it has a lot of boilerplates or configurations code. Elixir is one of the most fun programming languages I have ever used! And I wish I could use it for all of my software projects but to think that I could just rely on Elixir to do all of my projects is kind of naive for a software developer. I have been building software in many programming languages throughout my software development career and when I found myself switching between these languages, I often forgot the boilerplates or configurations code that exist in Elixir.

I have tried to anticipate this by creating sample projects so the next time I need to write code in Elixir (such as GenServer or GenStage code) I could just copy-paste from them. Initially, I thought this solution is enough however when I found myself needing to open other projects just to copy-paste boilerplates or configurations code that I forgot, it kind of lowered my mood to code a little bit. Have you experienced this as well?

In order to solve this simple issue yet impacting my coding productivity significantly, I decided to create my own snippet to assist me to code in Elixir that could reduce the need for me to memorize these boilerplates or configurations.

The Snippets

The snippets that I created are as follow:

`genserver`

`genstage.producer`

`genstage.consumer`

`genstage.producerconsumer`

Notice that VSCode is quite clever in that you don't have to type the complete snippet for it to show the autocomplete.

GitHub Repo

If you found the snippet to be useful and perhaps can help you in becoming more productive as well, feel free to find and use the snippet definition in this GitHub Repo. You might not need all the code that is generated from the snippet so feel free to remove some of them once generated.

For now, the snippets could help in generating boiler plates when creating GenServer and GenStage code however other elixir modules such as Supervisor, ConsumerSupervisor, Broadway, etc have their own boiler plates as well hence I am definitely going to add more snippets in the near future. If you think you have some ideas to improve them, you are welcomed to add more snippets or modify them :).

Limiting Concurrency in Elixir Using Registry

Nyoman Abiwinanda — Thu, 10 Nov 2022 10:35:13 +0000

When building applications with high performance in mind, we might be tempted to perform concurrent tasks as much as possible. However, performing tasks concurrently, especially in an uncontrolled manner might put your application's reliability at risk. For example, if there are one million of data that you want to process concurrently and you carelessly spawn a million processes suddenly, it might overwhelm your application resources.

Elixir makes it easy for us to perform tasks concurrently however it is also important for us to control the concurrency in a controllable or sensible manner. This is often referred to as Limiting Concurrency or Controlling Back-Pressure in Elixir.

There are several ways to limit concurrency in Elixir and some libraries such as GenStage, Flow, and Broadway have an API for this however what happens if you have a GenServer process that is spawned dynamically on runtime in which you want to limit the number of running concurrent processes yourself. In this case, one way to achieve this is to use Elixir Registry.

1. What is Registry

We won't go into too much detail about what Registry is since it is better to read through the official documentation for this but in a simple way, you could think of Registry as a key-value storage. It is common in elixir to index a GenServer process pids in the registry and later retrieve them in some other part of the application.

2. Setting Up Registry

Setting up a registry is simple and most of the time we don't even need to write additional code. Just like any other thing in elixir, a registry run in its own process hence to start a registry process, simply go to your OTP app supervision tree (this is inside the Application module) and then add the following code under the supervised children.

children = [
  {Registry, keys: :unique, name: YourOTPApp.Registry},
]

Note that you could name your registry with any name you want, in the code above we use the YourOTPApp.Registry as the process name.

When creating a registry, we could specify whether we want the keys option to be :unique or :duplicate. If we set it to :unique then any key can only be registered once. None of the options will prevent us to use the registry to limit concurrency. In this example, we will see how it is done by using the :unique option key.

3. Register Keys to Registry

Any processes in Elixir can be named. Assume we have a genserver process that acts as a worker. We could pass a :name keyword in the GenServer.start_link/3 function to name the worker.

defmodule YourOTPApp.Worker do
  use GenServer

  def start_link(args) do
    GenServer.start_link(__MODULE__, args, name: __MODULE__)
  end

  # ...
end

You could pass an atom, string, and tuple to the :name. You could even pass module names such as __MODULE__ which is common to use in genserver processes. One drawback of using the value mentioned is that hardcoding the name to a single value will only limit the process to a single running instance. It is similar to a singleton in object-oriented programming. For example, once a process is running with the name __MODULE__ then no other process can run with the same name again.

In this case, we don't want to hardcode to a single value, we want to be able to provide the name during runtime when we create the process. To do that we could use a registry to uniquely name the genserver process.

To register the genserver process to a registry, we could use a special :via tuple.

defmodule YourOTPApp.Worker do
  use GenServer

  def start_link(args) do
    id = Keyword.get(args, :id)

    GenServer.start_link(__MODULE__, args, name: via(id, "worker"))
  end

  # ...

  defp via(key, value) do
    {:via, Registry, {YourOTPApp.Registry, key, value}}
  end
end

The {:via, Registry, {YourOTPApp.Registry, key, value}} tuple will register the process to the YourOTPApp.Registry with a combination of key-value pair. In this case, only the key has to be unique while the value can be the same. Hence, we are hardcoding the value to "worker" and you will see later why we do this.

4. Limit the Number of Concurrently Running Processes

Once we register the worker processes to the registry, we would be able to query the worker pid from the registry. Now, we are not interested in the pid of the workers, we are just interested to know how many instances of the worker are running. To query how many workers we have, we could use the Registry,select/2 function to find out how many workers are running. The following code demonstrates the select query.

match_all = {:"$1", :"$2", :"$3"}
guards = [{:==, :"$3", "worker"}]
map_result = [%{pid: :"$2"}]

Registry.select(YourOTPApp.Registry, [{match_all, guards, map_result}])

The select query might look a little bit cryptic so let's take a look a little bit in detail:

match_all represents a {key, pid, value} tuple.
guards is a list of filters to query data from the registry. In this case, we only use one filter which is to select key, pid, and values registered where the value is "worker".
map_result is a list that represents how we want to map the data retrieved from the registry.

In this case, how we map the data retrieved from the registry does not matter. What matters is the number of running worker processes.

Once we could count the number of running workers, we could limit the number of concurrent workers by adding a simple conditional logic. The following code demonstrates limiting the running workers to only 5 maximum.

defmodule YourOTPApp do
  def start_worker(args) do
    if Enum.count(running_workers()) <= 5 do
      # code to start the worker
    else
      {:error, :maximum_workers_reached}
    end
  end

  def running_workers() do
    match_all = {:"$1", :"$2", :"$3"}
    guards = [{:==, :"$3", "import"}]
    map_result = [%{pid: :"$2"}]

    Registry.select(YourOTPApp.Registry, [{match_all, guards, map_result}])
  end
end

Note that registry will only returned running pids. If a worker crashed or stopped then its pid will not be returned.

5. Summary

In this post, you just see how Registry can be used to limit the number of concurrent processes in Elixir. Depending on your application or use cases, the way you register or query a process from the registry might differ but the concept is still the same which is to keep track of running processes using the registry and use that tracking to limit the number of them.

Kaffy - Admin Interface for Phoenix Applications

Nyoman Abiwinanda — Sun, 06 Dec 2020 16:16:40 +0000

If you are using elixir right now, chances are that this isn't your first programming language that you have used to build your apps. Probably you are coming from other programming languages such as Python, Java, C#, Golang, etc. Speaking of this, anyone coming from Python or at least has used Python as their main programming language to build your apps? I am one of these people.

As most people already know, the most used web framework in Python is Django. One of the Django's feature that I love the most or I think is unique is the Django admin interface. It provides you with a web interface where you could perform CRUD operations over you applications data out of the box. This is really useful especially if you or someone are trying to test your application and need to have a way of preparing a mock or test data quickly.

Unfortunately there isn't a similar feature in Phoenix nor a third party module that allows you to have this kind of admin interface, at least when I first use elixir. However, there is one now and I feel there are still not many people know about this tool in the elixir community. I thought this tool deserve more recognition so I decided to let people know about it through this post.

Kaffy

Kaffy is a tool that is inspired by Django admin that allows you to have a simple, flexible, and customizable admin interface in Phoenix application. With some minimal configurations, it ables to understand your application schemas and then make a CRUD page out of it.

I won't deep dive to much into how to use or set it up since it's pretty much documented in its GitHub repo but in summary not only it makes a CRUD page for your schema but it also enables you to:

Create a custom dashboard page with widgets.
Create a custom static page.
Register your own scheduled tasks.
And more.

Not only that it provides a similar functionality just like Django admin, but I personally think that the admin design looks much more pleasant in Kaffy :D.

Original Creator

First of all I didn't create this tool. I am just a guy who thought that this tool has a potential to be used by a lot of people in the future so I decided to share it. I originally found this tool around August 2020 from the following post in elixirforum. In that post you could find who is the original creator and some points that encouraged the creation of Kaffy in the first place. I encourage you to check it out since it contains a lot of information about the tool it self. Last but not least, Kaffy is pretty much new and it is still actively developed by people. If you are interested, maybe you could contribute to its GitHub repo and help make it a better tool for the elixir community to use :). Cheers ~

Speed-up Building your Elixir Unit Tests with Fictitious

Nyoman Abiwinanda — Sun, 29 Nov 2020 12:47:58 +0000

Imagine the following situation, you have several APIs to build or complete in a day or days. You started to work on them one by one, finish each of them in about an hour or two, feels satisfied when you finish one and that satisfaction makes you feel ready to move on to another tasks but only to realise at the end that you forgot to make the unit tests and making them could cost you about the same time or even more time than you need to finish up your APIs. Have you ever been in this kind of situation? Well... I do, sometimes it bother me so much that I decided to skip the unit test only to realised in the long run that this degrade my productivity because things just breaks unnoticedly as your applications are becoming bigger, complex, and more people are working on the same code base.

Why Unit Test can be Tedious?

Before we even touch our keyboard and just think about unit test briefly in our head, unit testing a code seems to be a simple task especially in a functional world. Typically these are the steps when we test our code:

Prepare your input to be fed to your function or system under test.
Call your function or invoke what ever things that you trying to test with the test input that you generate.
Check whether your function produce the expected output.

Basically these steps can be summarised with a phrase Prepare, Test, Assert. Sounds pretty simple right? Sure, but, this is usually what you think about it before you touch your keyboard and start making your real test though.

Sometimes the step that eats up a lot of your time isn't really the testing part but preparing the test or mock data for your system under test. Consider the following application. You have a function, lets called it like_a_comment() that increment the number of like for a blog post's comment by 1

The testing part is rather simple, you pass a sample comment to the function like_a_comment() and then checked at the end whether the number of likes for that comment is incremented by one. However we might not realised that in obtaining the input comment it self, it might require us to walk through several steps. For example, before we have a comment we need to:

Have a post that the comment is belonged to.
Even before that a user must exist as the owner of that post.

So you realised that the scope of your code is kind of more illustrated by the following diagram

You might be saying Hey, can we just wrapped the steps to create a comment in a function and reuse it through out other tests? Sure we can, in fact that is exactly what I had been doing previously. However, it only cover that one specific case. As my application gets bigger, more complex, and new data are introduced into the system, more data (utility) functions that I need to prepare before I could start writing the actual test. Would it be nice if there is an out of the box tool where I could just say I want a comment with no like to be created and I don't care how the comment is supposed to created? Turns out there is one in elixir.

Make your Unit Test Data with Fictitious

To address the issue that we discussed previously, I created a library called Fictitious which I open source to hex.pm for anyone to use. Fictitious, like its name, is a tool that enables you to create a fictitious data in elixir. It helps you to create mock data for your unit test without having the hassle of preparing the data in an convoluted order according to their associations that they have. Fictitious will ensure that whatever schema that you give as an input will create the related data for you.

Consider the following two schemas Person and Country

Person Schema

defmodule YourApp.Schema.Person do
  use Ecto.Schema
  import Ecto.Changeset
  alias YourApp.Schema.Country

  schema "persons" do
    field :name, :string
    field :age, :integer
    field :gender, :string
    field :email, :string
    belongs_to :nationality, Country, references: :id, foreign_key: :country_id, type: :id

    timestamps()
  end
end

Country Schema

defmodule YourApp.Schema.Country do
  use Ecto.Schema
  import Ecto.Changeset
  alias YourApp.Schema.Person

  schema "countries" do
    field :name, :string
    has_many :people, Person, foreign_key: :country_id

    timestamps()
  end
end

where Person is belonged to a Country. Let say in this case a person must belonged to a country. This means that country_id can't be null and when there is a value for it, it must references to an existing id of a country. This kind of data schemes could make data preparation in a test becomes tedious however this is one problem where fictitious could help you.

For example, calling fictionize/1 to Country will makes a fictitious country:

iex> Fictitious.fictionize(YourApp.Schema.Country)
{:ok, %YourApp.Schema.Country{
  __meta__: #Ecto.Schema.Metadata<:loaded, "countries">,
  id: 67,
  name: "B8LemwxB8ULP4NLUaFnKfwWkMmBYy8BTytkSN2PiL1UTO47yRM",
  people: #Ecto.Association.NotLoaded<association :people is not loaded>,
  inserted_at: ~U[2020-04-31 06:19:27Z],
  updated_at: ~U[2020-04-31 06:19:27Z]
}}

however calling fictionize/1 to Person will creates a person by creating the country as well:

iex> {:ok, person} = Fictitious.fictionize(YourApp.Schema.Person)
{:ok, %YourApp.Schema.Person{
  __meta__: #Ecto.Schema.Metadata<:loaded, "persons">,
  id: 725,
  name: "bElHKj9zVwnkLRpO4Y23yon9n80gm1yeAEL4PgtgkxBc0p2Y7C",
  age: 364,
  gender: "dF1O5Eq4ombjzah",
  email: "hpOXdOriGA9xaMhnwese40PqqL2Ine",
  nationality: #Ecto.Association.NotLoaded<association :nationality is not loaded>,
  nationality_id: 401,
  inserted_at: ~U[2020-04-31 06:19:27Z],
  updated_at: ~U[2020-04-31 06:19:27Z]
}}

iex> YourApp.Repo.preload(person, :nationality)
%YourApp.Schema.Person{
  __meta__: #Ecto.Schema.Metadata<:loaded, "persons">,
  id: 725,
  name: "bElHKj9zVwnkLRpO4Y23yon9n80gm1yeAEL4PgtgkxBc0p2Y7C",
  age: 364,
  gender: "dF1O5Eq4ombjzah",
  email: "hpOXdOriGA9xaMhnwese40PqqL2Ine",
  nationality: %YourApp.Schema.Country{
    __meta__: #Ecto.Schema.Metadata<:loaded, "countries">,
    id: 401,
    name: "lcb1e86TY6RSccL6vPGjXOv43gnp1t",
    people: #Ecto.Association.NotLoaded<association :people is not loaded>
    inserted_at: ~U[2020-04-31 06:19:27Z],
    updated_at: ~U[2020-04-31 06:19:27Z]
  },
  nationality_id: 401,
  inserted_at: ~U[2020-04-31 06:19:27Z],
  updated_at: ~U[2020-04-31 06:19:27Z]
}

In this case, having the country to be created automatically removes the trouble of having to prepare country data before a person could be created or in other word Fictitious ensures that you get the targeted or wanted entity to be created.

You might be thinking, I don't want the data to be completely random, I want to override the data with some values. How can I do that? No problem, fictitious allows you to override any field in the schema by simply providing the value at the second argument.

iex> {:ok, country} = Fictitious.fictionize(YourApp.Schema.Country, name: "Indonesia")
{:ok, %YourApp.Schema.Country{
  __meta__: #Ecto.Schema.Metadata<:loaded, "countries">,
  id: 7914,
  name: "Indonesia",
  people: #Ecto.Association.NotLoaded<association :people is not loaded>,
  inserted_at: ~U[2020-04-31 06:19:27Z],
  updated_at: ~U[2020-04-31 06:19:27Z]
}}

it could even override the related entity by passing it directly as follows.

{:ok, person} = Fictitious.fictionize(YourApp.Schema.Person, nationality: country)
{:ok, %YourApp.Schema.Person{
  __meta__: #Ecto.Schema.Metadata<:loaded, "persons">,
  id: 451,
  name: "ZFvtidsGOPh6OymYJk529bL2QT9KMZic2A0ietddl2RWy",
  age: 150940,
  gender: "rHZYpbDgJQokDX2vSpSfWUmELrTb9f",
  email: "xmcuHrJvotjAQz6itQnZtoMp",
  nationality: #Ecto.Association.NotLoaded<association :nationality is not loaded>,
  inserted_at: ~U[2020-04-31 06:19:27Z],
  updated_at: ~U[2020-04-31 06:19:27Z]
}}

More About Fictitious

Now that you know what fictitious is capable of, I hope it could help you just like it helps me in making unit test faster in elixir and becoming more productive. It definitely one of my additional or alternative tools to help me in creating unit test in elixir and hopefully it can be added to yours as well. If you are interested to know more about fictitious feel free to read the official documentation here or if you turns out to have an idea to improve it and interested to make it better feel free to contribute in the following GitHub repository.

Happy unit testing :)