DEV Community: calvinsadewa

Case Study: Reducing toil of resolving issue in Node JS - Introduction

calvinsadewa — Tue, 16 Apr 2024 07:33:38 +0000

Hi there! Welcome to the first of a four-part journey where I share the transformation we achieved in addressing and resolving issues within our Node.js projects at Kargo. This adventure led us to dramatically reduce the time and effort required to tackle these challenges head-on.

In this opening segment, we delve into the core problems that initially confronted us and outline the innovative solutions we proposed.

Background

Some time ago, the Tech team at Kargo embarked on an interesting project: to build a new Node.js backend service designed to support a feature requiring specific integration with a third-party platform. This venture presented a unique challenge for Kargo's engineers, as NodeJS backend was a new addition to our technology stack, which usually was based on Elixir or Go. Despite the initial hardships stemming from our unfamiliarity with this tech stack, we persevered, successfully delivering the required functionality. The project not only met but exceeded our success metrics, leading to a well-deserved celebration among the team.

However, as time progressed, the project's scope expanded, and with it, an increase in operational issues and bugs became apparent. The time required to resolve these issues began to increase significantly. On average, it took the team approximately four hours to address each problem, with around three issues arising per week. This not only led to considerable time loss but also began to affect team productivity adversely. More importantly, the user experience and the product's overall success suffered, as frequent issues and extended downtime became more common.

Diagnosis

After some investigation, the NodeJS project lack observability & debugging tools that we have in usual Kargo's backend service (Elixir/Go). This result in increased effort across maintenance lifecycle (Detect, Diagnose, Debug). To illustrate, here's an hypothetical example of how a particular issue is resolved:

Issue detection: The team received a report from user that the feature is not working as expected. An engineer are assigned to investigate the issue. There are no dashboard for the project, so the engineer usually need to look at relevant code for impacted feature, then look at log file, custom query to database, and other ad-hoc action for confirming the issue.
Issue diagnosis: To determine what's the most likely cause of the issue, the engineer need to be familiar with codebase involved. Occasionally, they are lucky and there's log that's explicit enough to point out the issue. But most of the time, they need to add more log into codebase then redeploy it to get more information.
Issue debugging: After the engineer have some idea on what's the issue, though it's not usually 100% certain. They would implement probable fix, then redeploy the code through CI/CD. If the issue still persist, they would need to repeat the process again. Usually it takes 3 to 4 retry before the issue is resolved.

Solution

We brainstorm on how to improve the situation, also looking at what the tools & technique we have in Elixir/Go backend service that might help in this situation. We come up following solutions that we think will help addressing problem in each of the stage (Detect, Diagnose, Debug):

Monitoring Dashboard + Canonical Log: By implementing Action-oriented Dashboard, we've created a quick and centralized access point for vital aspects of the project, significantly speeding up our ability to detect and validate issues. This dashboard, built on a Canonical Log record approach, offers the flexibility needed for performing custom advanced analyses with ease. Our aim with this system is to verify common issues within a mere 5 minutes of engineering effort. For example, we successfully reduced the time to check for a frequent session connection issue from 20 minutes to just 1 minute.
On Demand Diagnostic Logging: We developed a feature for dynamic, detailed logging of operations, enabling us to drill down into issues as they happened. This was particularly effective in a recent incident where typical logs were inconclusive. By enabling on-demand logging, we pinpointed the problem within minutes, a process that could have otherwise taken multiple redeployments.
Developer Code Execution: Just like how Elixir's remote shell capability help Kargo's engineer rapidly diagnosing & debug for issue happened in backend, we would like to have similar capability in NodeJS project.
Operational Handbook: We compiled a comprehensive handbook detailing common issues and their solutions, which served as a first reference point for our engineers. One memorable success story involved a team member who are relatively new to the project, was able to resolve an issue independently by following the handbook when the usual maintainer taking day off. This not only make issue resolving more efficient, but also give assurance to engineer.

Action-oriented dashboard help engineer quickly understand & troubleshoot issue and decide what's the fix

Ideal Outcome

We target we were able to reduce the time needed to resolve issue from 4 hour to 30 minutes. Thus reducing the impact of issue to user experience and product success, and help the team to focus on delivering more value to the user.

An ideal scenario of how the issue resolved in NodeJS project looks like:

Issue detection
For 50% common issue, the issue would be automatically detected and alerted from metric dashboard. The engineer could easily validate the issue exist by looking at metric dashboard and resolve the issue by following the operational handbook.
For the rest of 50% issue, engineer could do analysis based on dashboard + extended query from canonical log to confirm the issue and it's scope. If the issue is persistent, engineer could easily create new metric dashboard for the issue from canonical log data.
Issue diagnosis
When there’s an unknown issue, engineer could turn on diagnostic logging for user they are interested in (based on user ID) and get relevant debug log of WhatsApp bot server to form reasonable guess on where’s the issue is. Then engineer could use developer remote execution capability to validate whether the guess is correct or not.
Issue debugging
Once root cause of issue is identified, engineer could implement fix to the solution, then test it out first using developer remote execution. Once it's confirmed then engineer could deploy the fix through CI/CD.
Now, because root cause of issue is well identified, engineer could resolve the issue in just one try.

Conclusion

That's the background of the problem that we are facing, and next write up will be focusing on how we implement each of the solution that we proposed.

What do you do to make sure your team is able to resolve issue quickly? Share your experience in the comment below!

Does high performing organization doesn't care about best practice / architecture?

calvinsadewa — Wed, 29 Dec 2021 07:19:56 +0000

Hi all, recently i read Puppet State of Devops 2021 report link. Beside gaining a lot of insight from the report, i am also fascinated by a very interesting chart about aggregate team composition for low vs medium vs high performing organization from one of the slide.

Beside telling the usual story about high DevOps evolution characteristic like high performing team invest a lot in automating infrastructure / internal platform or low performing team has significant silo between developer & operation team, i also notice something different.

High performing team has less team composition which emphasis is on architecture or best practice, and has higher team composition which focus on providing reusable assets (tools, library). Below is subset of related statistic.

Teams	Low	Medium	High
A team that creates reusable assets (e.g., libraries, tools, or services) for other teams to assemble into solutions	3%	6%	10%
A team that defines the standards, processes, practices, frameworks or architectures that other teams must follow	16%	10%	7%
A team whose primary mission is to help define and encourage the adoption of good practices by other teams	7%	9%	4%

If understood naively, these statistics would imply that high performing organization doesn't invest much in architecture or best practice, and invest more in tooling/library, hence the click-bait title of this post. It would mean that high performing organization doesn't invest much on DevOps Center of Excellence.

Of course, this naive conclusion is wrong when we look at example of high DevOps evolution company, like Google or Netflix, as frequently they have exemplar best practice (like code stylesheet, or coding practice) and deals with really complicated architecture, like highly available, fault tolerant, zero trust security.

Then maybe, just thinking out loud, there is other reason that this statistic happen. maybe it's:

High DevOps organization thought that centralized best practice / architecture is slow or ineffective, and prefer shared / decentralized approach to it like frequent pair programming
Hiring & Screening of Engineers, FANG companies is notorious for their screening of engineers to have high standard on best practice / architecture

But these reasons is just hypothesis from me, i want to understand what's the actual difference or reason of this discrepancy, could anyone with relevant experience (i.e has work experience in high devops evolution team before) share their thought / experience on this issue?

Uncertainty increase development effort

calvinsadewa — Thu, 19 Aug 2021 08:26:28 +0000

Background

Currently i am working in a sprint-based squad, and recently at work i have been involved in quarterly planning on how to schedule sprints for squads for the rest of the quarter. You might think this is tedious process, but it is helpful in term of coordinating effort between multiple division, so that other division.

One of the problem we encounter is translating from estimated story point of each effort, to a precise schedule on when effort would be finished. We could do this by making translation between Story Point to Man-Days, and then from each Man-Days to precise schedule according to each squad composition.

Story Point	Man-days
1	0.05
2	0.1
3	0.25
5	0.5
8	1
13	2
21	4

Uncertain Estimation

We can say uncertainty in estimation as chance for an estimation to deviate from original estimation. Like 10 % chance for story point to go up or down from original estimation.

How would this uncertainty effect development effort? Turn out it generally increase effort due to higher story point generally take longer time to finish than lower story point.

For example, let's take estimation of a task as 8 with 10% chance ± 5 story point. Expected Man-Days for the task to finish would be:

80% * Man-Days(8 SP) + 10% * Man-Days(3 SP) + 10% * Man-Days(13 SP).
80% * 1 + 10% * 0.25 + 10% * 2.
1.025 Man-Days, which is higher than perfectly certain Man-Days(8 SP) = 1
Hence uncertainty increase effort, even though Average expected complexity still same.

Effect of Higher Uncertainty toward increased effort

We could also label these chances of change and calculate expected increase of effort, like:

Label	Chances	Avg Increase of effort
Perfectly certain	0% chance of change	0%
Certain	10% chance of 1 level of change	2.5%
Uncertain	20% chance of 1 level of change, 5% chance of 2 level of change	12.5%
Highly Uncertain	25% chance of 1 level of change, 10% chance of 2 level of change	27.5%

Afternote

This article is inspired this talk about statistical consequence of fat tail: https://www.youtube.com/watch?v=_8j1XZ0N_wE
I am using normal distribution here, giving equal chance for story point to be added and to be reduced. Given well recorded tendency for software developer to underestimate, probably log normal distribution would be better

Detail of calculations for Higher Uncertainty vs Effort

In which 1 level change is based of Fibonacci change, F(N+1) - F(N) = F(N-1) which for 8 Story Point it is 5 Story Point.
2 level change is F(N+2) - F(N) = F(N+1), which for 8 SP it is 13(!) SP, up and down (for down, it would make it goes to -5, for Manday we will treat it as negative Man-Day(5)).

Story Point	Perfectly Certain MD	Certain MD	Uncertain MD	Highly Uncertain MD
1	0.05	N/A	N/A	N/A
2	0.1	N/A	N/A	N/A
3	0.25	0.26	N/A	N/A
5	0.5	0.51	0.545	N/A
8	1	1.025	1.125	1.275
13	2	2.05	2.25	2.55
21	4	4.1	4.5	5.1

Hacking Ecto: Constructing Ecto Query from Raw SQL

calvinsadewa — Wed, 13 Jan 2021 13:24:40 +0000

Hi all, here's sharing "hack" tips for constructing Ecto.Query from Raw SQL. You would want to do this in case you want all good feature of Ecto.Query but want to bypass safeguard put in place.

A bit of introduction

Ecto is really cool database layer in Elixir, for SQL it let you define SQL query inside Elixir in readable and concise way, and execute it with automatic mapping to defined data structure.

example of SQL query and execution in Ecto

Ecto.Query is main module and data structure for defining a query in Ecto. Ecto.Query is composable and dynamically programmable, which means that we can extend a lot and reuse query functionality.

we define with_search_keyword query filtering function and use it as part of bill_of_ladings function

Constructing Ecto from Raw Query

By default, from, usual construction method of Ecto.Query doesn't support complicated SQL. from support only Ecto.Schema data structure and table name in string, like:

from(t in "transporter",
  where: t.name == "STAR TRANSPORT COMPANY")

However, you can specify Raw SQL in join, via fragment, which usually used in case you have complicated join query.

from(t in "transporter",
  join: i in fragment("SELECT * FROM invoice WHERE invoice.transporter_id = ?", t.id),
  where: i.date == ^date
)

We can leverage this Raw SQL join to get Raw SQL as base object of Ecto.Query.

Define dummy query

We need a dummy query, need to has exactly one result. as example we can query from "constants" table and limit it to 1

dummy_query = from x in "truck_types", as: :dummy, select: x.ksuid, limit: 1

Join dummy query with raw query

By using subquery, we can reuse the dummy query to be used as join component, which we use then to join with raw data

aggregate_subquery = 
  from(d in subquery(dummy_query))
    |> join(:inner, [], raw in fragment(@raw_sql), on: true, as: :raw)
    |> select([raw: raw], %{
      id: raw.id,
      name: raw.name
    })

Example of @raw_sql would be below. @raw_sql need to be constant string.

SELECT id, name FROM transporter
UNION ALL
SELECT id, name FROM shipper

Wrap joined query in subquery (again)

By wrapping the joined query again in subquery, we will have a query which is just like Ecto.Query but with custom SQL

query = from(q in subquery(aggregate_subquery))

Now you can use query as in usual Ecto way

Like in below, which search from transporter and shipper who has name "ANDI"

from(q in query,
  where: name == "ANDI"
)

Why would you want to hack it?

Ecto has really flexible and expressive way to write SQL, why then you would want to resort to such hacking as writing Raw SQL?

For me, the case come when i need to retrofit an existing SQL query, ~500 line - full with Postgres specific SQL feature, with existing functionality which is already built with Ecto.Query. Rewriting it in Ecto would need to decipher such 500 line of SQL, and would be full of fragment for safety hatch of writing Raw SQL. I reason that doing it all in Raw SQL would be easier to comprehend, which then led to this hack.

Property-based testing with StreamData

calvinsadewa — Wed, 10 Jun 2020 16:07:30 +0000

Recently, i have been (re)implementing KSUID library on elixir (called ExKsuid). KSUID is a way to generate identifier (like UUID) which result can be sorted by time, i have been using KSUID a lot, and i like how it can make DB operation like find all notification after X timestamp really efficient if used as primary key. There is an old KSUID library (elixir-ksuid) but some of feature that i want hasn't implemented yet (like generate KSUID from a timestamp).

During implementing ExKsuid, i remember there is StreamData elixir library for Property-based testing, which then i took a fancy of using in testing.

Suprise to me! StreamData actually help me tremendously during implementing to find out edge cases.

What is Property-based testing

When we are thinking of testing, usually what come to mind is testing by example, in which we test process input example and expect result to conform to output example, like:

Describe Find Longest Pair Point of Set

Given set of 2D-point [{1.3, 1.6}, {4.2, 1.0}, {0.3, 0.6}, {2.0, 4.5}, {5.3, 4.3}]
When Find Longest Pair Point of the set
Then return [{0.3, 0.6}, {5.3, 4.3}]

or maybe

Describe Transfer

Given User A with Account Balance 20000,
  AND User B with Account Balance 10000
When User A transfer to User B with amount 5000
Then User A should have Balance 15000,
  AND User B should have Balance 15000

In contrast, Property-based testing try to leverage property of the tested code or model to then generate a bunch of input and try to find when property is broken. Property can be any rule the code/model is expected to hold if it is implemented correctly.

for example, if we have an rectangle with height A, and width B, then pair {(0, A), (B, 0)} is going to be longer then all of set of point taken inside of rectangle

using StreamData, we can generate list of point inside rectangle like (here generate 10 list)

a = 4
b = 5
StreamData.list_of(
  StreamData.tuple({
    StreamData.float(min: 0, max: a), 
    StreamData.float(min: 0, max: b)}
  )
) |> Enum.take(10)
# result
[
  [],
  [],
  [{2.5, 0.0}],
  [{1.875, 2.0}, {2.5, 4.0}, {1.25, 0.0}, {0.625, 2.0}],
  [{0.0, 4.0}],
  [{3.75, 0.0}, {4.375, 1.5}, {5.0, 1.125}, {1.25, 0.0}],
  [
    {3.046875, 4.0},
    {4.375, 2.3125},
    {2.5, 1.875},
    {0.0, 2.0},
    {0.390625, 4.0},
    {0.46875, 0.5}
  ],
  [
    {0.68359375, 0.21875},
    {3.4375, 4.0},
    {0.78125, 0.0},
    {0.9765625, 2.0},
    {0.0, 4.0},
    {1.25, 1.484375},
    {4.2578125, 2.8125},
    {0.0, 3.5}
  ],
  [{5.0, 3.6875}, {2.5, 0.5}, {4.765625, 3.625}, {2.44140625, 1.25}],
  [
    {0.46875, 2.4765625},
    {3.125, 1.2734375},
    {1.7578125, 2.484375},
    {2.5, 0.46875},
    {0.9375, 2.25}
  ]
]

then from this generated test data, we can test if find_longest_pair_point maintain the property or not

defmodule TestFindLongestPair do
  use ExUnitProperties

  property "pair corner of a rectangle is longer than all pair of point inside rectangle" do
    check all a <- StreamData.positive_integer(),
        b <- StreamData.positive_integer(),
        inside_points <- StreamData.list_of(
          StreamData.tuple({
            StreamData.float(min: 0, max: a),
            StreamData.float(min: 0, max: b)}
          )
        )
    do
      points = Enum.shuffle([{0, a}, {b, 0}] ++ inside_points)
      assert [{x1, y1}, {x2, y2}] = find_longest_pair_point(points) |> Enum.sort()
      assert a == y1
      assert b == x2
      assert x1 == 0
      assert y2 == 0
    end
  end
end

which if not implemented correctly, would fail like this

We can also make a property from Transfer process, we can say that during transferring should maintain total account balance before and after transfer.

Describe Transfer Maintain Total Account Balance

Given User A with Account Balance X,
  AND User B with Account Balance Y
When User A transfer to User B with amount Z
Then X + Y == currentBalance(User A) + currentBalance(User B)

However, Transfer process is what we usually say stateful process, in which it change state of user A and user B. Property-based testing (and StreamData in particular) sadly doesn't have much support in testing stateful process.

Example in Real Life

There are two main module in ExKsuid, ExKsuid (which handle how to generate/parse KSUID) and ExKsuid.Base62 (which handle how to encode/decode data in Base62 format.

Helping clarify by giving edge cases

Property i would like to test for Base62 is that data which is encoded and decoded should return the same, so i write

  property "encode and decode binary data return same binary data" do
    check all(data <- StreamData.binary()) do
      assert data == Base62.decode(Base62.encode(data))
    end
  end

but it return error like,

turn out binary data is <<0, 245>>, which after decode & encode return <<245>>. After considering it, i would like to only care integer value of binary data (in this case both is 245) due to seemingly hard problem to match leading 0 position in binary form and base62 form, so i update test to reflect on it.

  property "encode and decode binary data return same binary data" do
    check all(data <- StreamData.binary()) do
      x = :binary.decode_unsigned(data)
      y = :binary.decode_unsigned(Base62.decode(Base62.encode(data)))
      assert x == y
    end
  end

Actually finding bugs in edge cases

On main ExKsuid, i test property "KSUID generated from lower timestamp is lower that KSUID generated"

  property "Generated KSUID from earlier time lexicographically smaller than later time" do
    check all(int1 <- StreamData.positive_integer(), int2 <- StreamData.positive_integer()) do
      assert ExKsuid.generate(timestamp: int1 + @epoch) <
        ExKsuid.generate(timestamp: int1 + int2 + @epoch)
    end
  end

and immediately it fail with

  1) property Generated KSUID from earlier time lexicographically smaller than later time (ExKsuidTest)
     test/ex_ksuid_test.exs:37
     Failed with generated values (after 4 successful runs):

         * Clause:    int1 <- StreamData.positive_integer()
           Generated: 5

         * Clause:    int2 <- StreamData.positive_integer()
           Generated: 3

     Assertion with < failed
     code:  assert ExKsuid.generate(timestamp: int1 + @epoch) < ExKsuid.generate(timestamp: int1 + int2 + @epoch)
     left:  "e3RlO2qT8eNY7rPCfQ8Ku0"
     right: "10OPH1xqiF0vG3XHXcAAFYj"
     stacktrace:
       test/ex_ksuid_test.exs:39: anonymous fn/3 in ExKsuidTest."property Generated KSUID from earlier time lexicographically smaller than later time"/1

after looking closely at test data,i realize that KSUID should have 27 character, and it seems 0-encoding of Base62 have connection with the problem. Turn out that binary data for low timestamp has a lot of leading 0

ExKsuid.generate_raw(timestamp: 1400000008)
#result
<<0, 0, 0, 8, 40, 147, 107, 12, 82, 97, 52, 48, 149, 155, 109, 162, 194, 203,
  98, 182>>

When it is encoded to base62, those extra leading 0 is ignored. The fix is simple, just add leading 0 until it is 27 character, and the test goes green.

Conclusion

Does Property-based test help finding out bugs and giving confidence with low effort? Absolutely!

Can it be used to wholly replace example based test though? Not in my case, I still use example test use to cross check my implementation with existing reference library to give even more confidence.

The confidence and ease is what's matter for me in testing after all.

Mocking in Elixir: Comparison between Mox, Mockery, Mimic, Syringe, and Lightweight DI

calvinsadewa — Sat, 02 May 2020 15:58:52 +0000

Unit test in elixir is fantastic using ExUnit! It is simple and easy! However sometimes we need to test function/module which access outside resources our code, like third party services or API.

These outside resource may

take time to be accessed (due to network)
require additional money (in case of pay per request API)
not return stable data (in case of live updating data)

These make accessing outside resource directly when unit testing is not desirable, hence the need of mock these resources in our unit test so it can still be fast, cheap, and stable.

This article aims to explore 5 different ways of mocking in elixir: Mox, Mockery, Mimic, Syringe, and Lightweight DI

Code to test

Here is our sample code to test, corona_news. corona_news is a simple application to display latest update regarding Corona Virus / COVID-19 by accessing https://api.covid19api.com. We would like to test mocking CoronaNews.Gateway

defmodule CoronaNews do
  @moduledoc """
  Module for displaying Corona Virus (COVID-19) data update
  output of this module is expected to be readable by human user
  """

  @doc """
  Return human-readable text summary of corona update for a country

  ## Example
    iex> text_news_for(CoronaNews.Country.global)
    Total Case: 3340989
    Total Recovered: 1052510
    New Case: 86819
    Latest Update: 2020-05-02 12:50:40.760326Z
  """
  def text_news_for(country \\ CoronaNews.Country.global()) do
    result = CoronaNews.Gateway.fetch_data_for_country(country)
    case result do
      {:ok, data} ->
        """
        Total Case: #{data.total_case}
        Total Recovered: #{data.total_recovered}
        New Case: #{data.new_case}
        Latest Update: #{data.latest_update}
        """
      {:error, error} ->
        """
        Failed fetching data
        Error: #{inspect error}
        """
    end
  end

  @doc """
  Display human-readable text summary of corona update for a country
  """
  def display_news_for(country \\ CoronaNews.Country.global()) do
    IO.puts(text_news_for(country))
  end
end

defmodule CoronaNews.Country do
  @moduledoc """
  Module for storing constants identifier of each country
  Country Identifier is CountryCode from https://api.covid19api.com
  """

  @type t :: String.t()

  def global, do: "Global"
  def indonesia, do: "ID"
  def china, do: "CN"
  def united_states_of_america, do: "US"
end

defmodule CoronaNews.Gateway do
  @moduledoc """
  Module for contacting API at https://api.covid19api.com
  for latest data regarding corona
  """

  @base_url "https://api.covid19api.com"
  @summary_url @base_url <> "/summary"

  @typedoc """
  Result for fetch API,
  latest_update is always UTC
  example:
  %{
    latest_update: ~U[2020-05-02 12:41:23Z],
    new_case: 433,
    total_case: 10551,
    total_recovered: 1591
  }
  """
  @type result :: %{
    total_case: number(),
    total_recovered: number(),
    new_case: number(),
    latest_update: DateTime.t()
  }

  @doc """
  Fetch latest summary data for a country
  ## Example
    CoronaNews.Gateway.fetch_data_for_country(CoronaNews.Country.indonesia)
  """
  @spec fetch_data_for_country(CoronaNews.Country.t) :: {:ok, result} | {:error, any()}
  def fetch_data_for_country(country) do
    with {:request, {:ok, response}} <- {:request, HTTPoison.get(@summary_url)},
      {:status_code, %{status_code: 200}} <- {:status_code, response},
      {:body_decode, {:ok, json_map}, _response} <- {:body_decode, Jason.decode(response.body), response}
    do
      parse_summary_data_for_country(json_map, country)
    else
      {:request, failed} -> {:error, "Request failed, #{inspect failed}"}
      {:status_code, response} -> {:error, "Response code not 200, #{inspect response}"}
      {:body_decode, failed, response} -> {:error, "JSON decode failed, #{inspect response}, #{inspect failed}"}
    end
  end

  # parse JSON result for get summary data to result type for country
  defp parse_summary_data_for_country(json_api_data, country) do
    data = if country == CoronaNews.Country.global do
      json_api_data[CoronaNews.Country.global]
    else
      json_api_data["Countries"]
      |> Enum.find(fn slug -> slug["CountryCode"] == country end)
    end

    if data == nil do
      {:error, "Country data not found for #{inspect country}"}
    else
      latest_update = if data["Date"] == nil do
        DateTime.utc_now()
      else
        {:ok, datetime, _} = DateTime.from_iso8601(data["Date"])
        datetime
      end

      {:ok, %{
        total_case: data["TotalConfirmed"],
        total_recovered: data["TotalRecovered"],
        new_case: data["NewConfirmed"],
        latest_update: latest_update
      }}
    end
  end
end

Here is example of result of corona_news

Using Mox

Mox is library for defining concurrent mocks in Elixir.

Mox principle is that mock should be an object and not a verb. You should create behaviour of module to be mocked, and then during test change using module (at compile time) to mocked module conforming to behaviour.

It is perhaps the most used library for mock in elixir, with 541015 recent download.

For using Mox to test CoronaNews, we need to:

Define behaviour for CoronaNews.Gateway (CoronaNews.Gateway.Behaviour)
add mox to our dependency
Write the unit test

Here is the diff of code when i use mox for testing

# at test/corona_news_test.exs
+defmodule CoronaNewsTest do
+  use ExUnit.Case
+  import Mox
+
+  describe "CoronaNews" do
+    test "text_news_for/1 on Gateway Success" do
+      CoronaNews.GatewayMock
+      |> expect(:fetch_data_for_country, 1, fn country ->
+        assert country == CoronaNews.Country.indonesia()
+        {:ok, %{
+          total_case: 6,
+          total_recovered: 3,
+          new_case: 1,
+          latest_update: ~U[2020-05-02 13:37:37Z]
+        }}
+      end)
+
+      assert """
+      Total Case: 6
+      Total Recovered: 3
+      New Case: 1
+      Latest Update: 2020-05-02 13:37:37Z
+      """ == CoronaNews.text_news_for(CoronaNews.Country.indonesia())
+    end
+
+    test "text_news_for/1 on Gateway Failed" do
+      CoronaNews.GatewayMock
+      |> expect(:fetch_data_for_country, 1, fn country ->
+        assert country == CoronaNews.Country.indonesia()
+        {:error, "Error due to Mock"}
+      end)
+
+      assert """
+      Failed fetching data
+      Error: \"Error due to Mock\"
+      """ == CoronaNews.text_news_for(CoronaNews.Country.indonesia())
+    end
+  end
+end

# at test/test_helper.exs
+Application.put_env(:corona_news, :gateway, CoronaNews.GatewayMock)
+Mox.defmock(CoronaNews.GatewayMock, for: CoronaNews.Gateway.Behaviour)
+Application.ensure_started(:mox)
ExUnit.start()

# at mix.exs
  defp deps do
    [
      {:httpoison, "~> 1.6"},
      {:jason, "~> 1.2"},
+      {:mox, "~> 0.5", only: :test}
    ]
  end

# at lib/corona_news.ex
defmodule CoronaNews do
  @moduledoc """
  Module for displaying Corona Virus (COVID-19) data update
  output of this module is expected to be readable by human user
  """
- 
+  def gateway(), do: Application.get_env(:corona_news, :gateway, CoronaNews.Gateway)

  @doc """
  Return human-readable text summary of corona update for a country

  ## Example
    iex> text_news_for(CoronaNews.Country.global)
    Total Case: 3340989
    Total Recovered: 1052510
    New Case: 86819
    Latest Update: 2020-05-02 12:50:40.760326Z
  """
  def text_news_for(country \\ CoronaNews.Country.global()) do
-    result = CoronaNews.Gateway.fetch_data_for_country(country)
+    result = gateway().fetch_data_for_country(country)
...

+defmodule CoronaNews.Gateway.Behaviour do
+  @moduledoc """
+  Behaviour of Gateway (API data request module) for corona data
+  """
+  @type result :: %{
+    total_case: number(),
+    total_recovered: number(),
+    new_case: number(),
+    latest_update: DateTime.t()
+  }
+
+  @doc """
+  Fetch latest summary data for a country
+  """
+  @callback fetch_data_for_country(CoronaNews.Country.t) :: {:ok, result} | {:error, any()}
+end

defmodule CoronaNews.Gateway do
  @moduledoc """
  Module for contacting API at https://api.covid19api.com
  for latest data regarding corona
  """
- 
+  @behaviour CoronaNews.Gateway.Behaviour
...

Using Mockery

Mockery is Simple mocking library for asynchronous testing in Elixir.

It define several type of mock, but what i am using is macro based one in which the macro will change behaviour of module at testing

to use Mockery, we need to:

add mockery to depedency
modify CoronaNews module to able to mock CoronaNews.Gateway
write test

Here is the code diff for testing using Mockery

# at test/corona_news_test.exs
+defmodule CoronaNewsTest do
+  use ExUnit.Case
+  import Mockery
+
+  describe "CoronaNews" do
+    test "text_news_for/1 on Gateway Success" do
+      mock CoronaNews.Gateway, [fetch_data_for_country: 1], fn country ->
+        assert country == CoronaNews.Country.indonesia()
+        {:ok, %{
+          total_case: 6,
+          total_recovered: 3,
+          new_case: 1,
+          latest_update: ~U[2020-05-02 13:37:37Z]
+        }}
+      end
+
+      assert """
+      Total Case: 6
+      Total Recovered: 3
+      New Case: 1
+      Latest Update: 2020-05-02 13:37:37Z
+      """ == CoronaNews.text_news_for(CoronaNews.Country.indonesia())
+    end
+
+    test "text_news_for/1 on Gateway Failed" do
+      mock CoronaNews.Gateway, [fetch_data_for_country: 1], fn country ->
+        assert country == CoronaNews.Country.indonesia()
+        {:error, "Error due to Mock"}
+      end
+
+      assert """
+      Failed fetching data
+      Error: \"Error due to Mock\"
+      """ == CoronaNews.text_news_for(CoronaNews.Country.indonesia())
+    end
+  end
+end
+

# at mix.exs
  defp deps do
    [
      {:httpoison, "~> 1.6"},
      {:jason, "~> 1.2"},
-
+      {:mockery, "~> 2.3.0", runtime: false}
    ]
  end

# at lib/corona_news.ex
defmodule CoronaNews do
  @moduledoc """
  Module for displaying Corona Virus (COVID-19) data update
  output of this module is expected to be readable by human user
  """
-
+  import Mockery.Macro

  @doc """
  Return human-readable text summary of corona update for a country

  ## Example
    iex> text_news_for(CoronaNews.Country.global)
    Total Case: 3340989
    Total Recovered: 1052510
    New Case: 86819
    Latest Update: 2020-05-02 12:50:40.760326Z
  """
  def text_news_for(country \\ CoronaNews.Country.global()) do
-    result = CoronaNews.Gateway.fetch_data_for_country(country)
+    result = mockable(CoronaNews.Gateway).fetch_data_for_country(country)
#

Using Mimic

Mimic is A sane way of using mocks in Elixir.

It's usage is the most simple among other library in comparison due not needing any kind of change in codebase

to use Mimic, we need to:

add Mimic to dependency
add Mimic.copy(CoronaNews.Gateway) at test/test_helper.exs
write test

Here is the code diff for testing using Mimic

# at test/corona_news_test.exs
+defmodule CoronaNewsTest do
+  use ExUnit.Case
+
+  describe "CoronaNews" do
+    test "text_news_for/1 on Gateway Success" do
+      Mimic.expect(CoronaNews.Gateway, :fetch_data_for_country, fn country ->
+        assert country == CoronaNews.Country.indonesia()
+        {:ok, %{
+          total_case: 6,
+          total_recovered: 3,
+          new_case: 1,
+          latest_update: ~U[2020-05-02 13:37:37Z]
+        }}
+      end)
+
+      assert """
+      Total Case: 6
+      Total Recovered: 3
+      New Case: 1
+      Latest Update: 2020-05-02 13:37:37Z
+      """ == CoronaNews.text_news_for(CoronaNews.Country.indonesia())
+    end
+
+    test "text_news_for/1 on Gateway Failed" do
+      Mimic.expect(CoronaNews.Gateway, :fetch_data_for_country, fn country ->
+        assert country == CoronaNews.Country.indonesia()
+        {:error, "Error due to Mock"}
+      end)
+
+      assert """
+      Failed fetching data
+      Error: \"Error due to Mock\"
+      """ == CoronaNews.text_news_for(CoronaNews.Country.indonesia())
+    end
+  end
+end

# at mix.exs
  defp deps do
    [
      {:httpoison, "~> 1.6"},
      {:jason, "~> 1.2"},
-
+      {:mimic, "~> 1.0.0", only: :test}
    ]
  end

# at test/test_helper.exs
-
+Mimic.copy(CoronaNews.Gateway)
ExUnit.start()

Using Syringe

Syringe is a injection framework that also opens the opportunity for clearer mocking and to run mocked test asynchronously

to use Syringe, we need to:

add Syringe to dependency
add config :syringe, injector_strategy: MockInjectingStrategy to config/test.exs
add config :syringe, injector_strategy: AliasInjectingStrategy to config/config.exs
Reshuffle CoronaNews.Gateway to be above CoronaNews because Syringe rely on macro (hence need to define CoronaNews.Gateway before CoronaNews
add Injector to CoronaNews.Gateway at CoronaNews
add Mocker.start_link() at test/test_helper.exs
write test

Here is code diff using syringe

# at test/corona_news_test.exs
+defmodule CoronaNewsTest do
+  use ExUnit.Case, async: true
+  import Mocker
+
+  describe "CoronaNews" do
+    test "text_news_for/1 on Gateway Success" do
+      mock(CoronaNews.Gateway)
+      intercept(CoronaNews.Gateway, :fetch_data_for_country, nil, with: fn country ->
+        assert country == CoronaNews.Country.indonesia()
+        {:ok, %{
+          total_case: 6,
+          total_recovered: 3,
+          new_case: 1,
+          latest_update: ~U[2020-05-02 13:37:37Z]
+        }}
+      end)
+
+      assert """
+      Total Case: 6
+      Total Recovered: 3
+      New Case: 1
+      Latest Update: 2020-05-02 13:37:37Z
+      """ == CoronaNews.text_news_for(CoronaNews.Country.indonesia())
+    end
+
+    test "text_news_for/1 on Gateway Failed" do
+      mock(CoronaNews.Gateway)
+      intercept(CoronaNews.Gateway, :fetch_data_for_country, nil, with: fn country ->
+        assert country == CoronaNews.Country.indonesia()
+        {:error, "Error due to Mock"}
+      end)
+
+      assert """
+      Failed fetching data
+      Error: \"Error due to Mock\"
+      """ == CoronaNews.text_news_for(CoronaNews.Country.indonesia())
+    end
+  end
+end
+

# at test/test_helper.exs
-
+Mocker.start_link()
ExUnit.start()

# at mix.exs
  defp deps do
    [
      {:httpoison, "~> 1.6"},
      {:jason, "~> 1.2"},
+      {:syringe, "~> 1.0"}
    ]
  end

# at lib/corona_news.ex
# Reshuffle CoronaNews.Gateway module to be above CoronaNews module
defmodule CoronaNews do
  @moduledoc """
  Module for displaying Corona Virus (COVID-19) data update
  output of this module is expected to be readable by human user
  """
+  use Injector
+
+  inject CoronaNews.Gateway, as: Gateway

  @doc """
  Return human-readable text summary of corona update for a country

  ## Example
    iex> text_news_for(CoronaNews.Country.global)
    Total Case: 3340989
    Total Recovered: 1052510
    New Case: 86819
    Latest Update: 2020-05-02 12:50:40.760326Z
  """
  def text_news_for(country \\ CoronaNews.Country.global()) do
-    result = CoronaNews.Gateway.fetch_data_for_country(country)
+    result = Gateway.fetch_data_for_country(country)

# at config/config.exs
+config :syringe, injector_strategy: MockInjectingStrategy

# at config/test.exs
+config :syringe, injector_strategy: MockInjectingStrategy

Using Lightweight DI

Lightweight Depedency Injection is done by giving module/function as argument to called function.

on simple case this is easy, however it will become cumbersome on many nested function because each function need to pass the module/function to nested function after it

to use Lightweight DI, we need to:

modify function CoronaNews.text_news_for to accept gateway module
define mock module on test
write test

Here is code diff using Lightweight DI

# at test/corona_news_test.exs
+defmodule CoronaNewsTest do
+  use ExUnit.Case, async: true
+
+  describe "CoronaNews" do
+    test "text_news_for/1 on Gateway Success" do
+
+      defmodule GatewayMock do
+        def fetch_data_for_country(country) do
+          assert country == CoronaNews.Country.indonesia()
+          {:ok, %{
+            total_case: 6,
+            total_recovered: 3,
+            new_case: 1,
+            latest_update: ~U[2020-05-02 13:37:37Z]
+          }}
+        end
+      end
+
+      assert """
+      Total Case: 6
+      Total Recovered: 3
+      New Case: 1
+      Latest Update: 2020-05-02 13:37:37Z
+      """ == CoronaNews.text_news_for(CoronaNews.Country.indonesia(), GatewayMock)
+    end
+
+    test "text_news_for/1 on Gateway Failed" do
+      defmodule GatewayMock do
+        def fetch_data_for_country(country) do
+          assert country == CoronaNews.Country.indonesia()
+          {:error, "Error due to Mock"}
+        end
+      end
+
+      assert """
+      Failed fetching data
+      Error: \"Error due to Mock\"
+      """ == CoronaNews.text_news_for(CoronaNews.Country.indonesia(), GatewayMock)
+    end
+  end
+end
+

# at lib/corona_news.ex
defmodule CoronaNews do
  @moduledoc """
  Module for displaying Corona Virus (COVID-19) data update
  output of this module is expected to be readable by human user
  """

  @doc """
  Return human-readable text summary of corona update for a country

  ## Example
    iex> text_news_for(CoronaNews.Country.global)
    Total Case: 3340989
    Total Recovered: 1052510
    New Case: 86819
    Latest Update: 2020-05-02 12:50:40.760326Z
  """
-  def text_news_for(country \\ CoronaNews.Country.global()) do
-    result = CoronaNews.Gateway.fetch_data_for_country(country)
+  def text_news_for(country \\ CoronaNews.Country.global(), gateway \\ CoronaNews.Gateway) do
+    result = gateway.fetch_data_for_country(country)

Conclusion

You have now see multitude of way to mock in elixir, personally my favorite is Mimic due to simplicity of setup and doesn't need to change tested codebase. Do go out and try, your mileage may vary!

Implementing Message Outbox Pattern with Oban

calvinsadewa — Sat, 08 Feb 2020 12:01:54 +0000

Hey guys, today i want to share Message Outbox pattern and implementation with Oban in Elixir
I have used them extensively and hope that it can benefit you too.

Problems

What happen if as part of database transaction, you need to send email / push notification / outside communication?
Implemented naively, we could just send email / push notification as part of transaction.

Imagine we have transfer_money function which send money from a sender to receiver,
as part of operation, transfering money need to also send email notification to sender and receiver.

Naively, we could make transfer_money a transaction which do:

substract sender balance
send email to sender regarding substracted money
add receiver balance
send email to receiver regarding added money

Here is pseudo-code snippet implementing transfer_money function in elixir.

# model/payment.ex
defmodule Model.Payment do
    alias Repo
    import Ecto.Changeset, only: [change: 2]

    # Transfer money from sender to receiver
    # as part of transfer, we will send email notification
    def transfer_money(%{
        sender: %Account{} = sender,
        receiver: %Account{} = receiver,
        amount: %Decimal{} = amount
    }) do
        Repo.transaction(fn -> 
          Repo.update!(change(sender, balance: Decimal.sub(sender.balance, amount))) # operation 1
          send_transfer_email!(sender, "substracted by", amount) # operation 2
          Repo.update!(change(receiver, balance: Decimal.add(receiver.balance, amount))) # operation 3
          send_transfer_email!(sender, "received", amount) # operation 4
        end)
    end

    # Send transfer email notification
    def send_transfer_email!(account, operation, amount) do
        to = account.email
        from "payment@mycompany.com"
        body = """
        Your bank account #{account.account_number} have #{operation} #{amount}
        """

        {:ok, _} = send_email(to, from, body)
    end
end

Despite seeming okay, it is possible bug that can happen. Let me show a scenario:

transfer_money function initiated
Open a transaction
substract sender balance by amount via DB operation
send email notification to sender that balance has been substracted
add receiver balance by amount via DB operation
try send email notification to recevier that balance has been added, however failed due to transient network failure
Transaction rolledback

In above failure scenario, even though transfer_money function rolledback,
sender already sended email notifing s/he bank account subtracted emoji.

While mis-sent email may seem harmless (though will probably shock sender that s/he subtracted twice),
These kind of cases can also happen with Message Queue (RabbitMQ, Kafka, SQS) using system,
which would mean subscribing service get wrong data, in turn may spawn more problem.

The problem is that operation 2 and 4 cannot rolledback, Message Outbox pattern help us with it.

Message Outbox

Message Outbox is a pattern in which we insert intention to send message to Outbox in transactional manner,
Outbox is usually a table which then polled/listened regulary by Worker. Worker would send any message waiting to be sended.

With that in mind, let us patch transfer_money to:

substract sender balance
insert intent to send email for sender to outbox
add receiver balance
insert intent to send email for receiver to outbox

We also need to implement Worker which poll/listen to our outbox, and send email if any pending.

Here is snippet of pseudo function implementing Message Outbox

# model/payment.ex
defmodule Model.Payment do
    ...
    # insert Send transfer email notification to outbox
    def send_transfer_email!(account, operation, amount) do
        Repo.insert!(%SendMessage{account: account, operation: operation, amount: amount})
    end

    # Real implementation of sending email
    def real_send_transfer_email!(account, operation, amount) do
        to = account.email
        from "payment@mycompany.com"
        body = """
        Your bank account #{account.account_number} have #{operation} #{amount}
        """

        {:ok, _} = send_email(to, from, body)
    end
end

# outbox/worker.ex
defmodule Outbox.Worker do
    # Perform is done regularly by worker, either via polling/listening
    def perform(%SendMessage{} = send_message) do
        Model.Payment.real_send_transfer_email!(account)
    end
end

With Message Outbox, outside communication become part of transaction, and can be rolledback in case transaction failed.
However, implementing worker which listen/poll for message can be hard,
not to mention we still need to handle cases where we failed sended message and retry logic.

Oban

Oban is a transactional job queue in elixir built on PostgreSQL. It handle many of the operation relating to job queue such as enqueuing job,
distribute job to worker, handle job processing throughout lifecycle, log job status, and more. You can see more at link https://hexdocs.pm/oban/1.0.0/Oban.html

Because of it's transactional feature in PostgreSQL, we can leverage Oban as our Outbox.

Using Oban, we will do:

Setup Oban in config.exs and Migration (I left out this part, see https://hexdocs.pm/oban/1.0.0/Oban.html )
Implement an Oban worker which would send email for us, Outbox.Email, and implement processing for job send_email
modify send_transfer_email! to insert job send_email to Outbox.Email

Let's modify our snippet to use Oban.

# model/payment.ex
defmodule Model.Payment do
    alias Repo
    import Ecto.Changeset, only: [change: 2]

    # Transfer money from sender to receiver
    # as part of transfer, we will send email notification
    def transfer_money(%{
        sender: %Account{} = sender,
        receiver: %Account{} = receiver,
        amount: %Decimal{} = amount
    }) do
        Repo.transaction(fn -> 
          Repo.update!(change(sender, balance: Decimal.sub(sender.balance, amount))) # operation 1
          send_transfer_email!(sender, "substracted by", amount) # operation 2
          Repo.update!(change(receiver, balance: Decimal.add(receiver.balance, amount))) # operation 3
          send_transfer_email!(sender, "received", amount) # operation 4
        end)
    end

    # insert intent to send transfer email notification
    def send_transfer_email!(account, operation, amount) do
        to = account.email
        from "payment@mycompany.com"
        body = """
        Your bank account #{account.account_number} have #{operation} #{amount}
        """

        # Insert args to Outbox.Email
        %{
            to: to, 
            from: from, 
            body: body, 
            event: "send_email"
        }
        |> Outbox.Email.new()
        |> Oban.insert!()
    end
end

# outbox/email.ex
defmodule Outbox.Email do
    use Oban.Worker, queue: :email

    @impl Oban.Worker
    # Process sending email
    def perform(%{"event" => "send_email"} = args, _job) do
        {:ok, _} = send_email(args["to"], args["from"], args["body"])
    end
end

It's pretty simple doesn't it? Though args for Oban worker is interesting.
When i insert args to Outbox.Email, i insert it as Atom Map. However when i process args at perform function, it become String Map.
This is because when Oban inserting job to Database, it transform args to JSON, and when Worker try to process job, args is parsed as JSON.

While Message Outbox is good, one weakness is that Message Outbox cannot depend on outside communication result inside transaction.
If your transaction really need to depend on outside communication, you may consider implementing two phase (or three phase) commit.

**Note: Oban is built upon PostgreSQL, so if you use MariaDB/MySQL you may consider other Job Queue such as Rihanna or implement polling by using cron-based Quantum

**Special thanks for Nanda Girindratama and Rifki Adrian for early review