DEV Community: Alex de Sousa

ICON 2.0: Realtime Decentralized Quote Prices

Alex de Sousa — Tue, 08 Mar 2022 12:49:44 +0000

ICON 2.0 is a smart contract platform with a unique interoperability proposition: Blockchain Transmission Protocol (BTP). BTP will enable decentralized and trustless interoperability between ICON 2.0 and all blockchain that implement the protocol.

At the time of publishing this article, the full BTP is not yet released. However, some of its features are already available to everyone. One of these features is a websocket to receive realtime updates on either (or both) blocks produced and event logs emitted.

This article will focus on how we can leverage this websocket connection to retrieve quote prices from Balanced Decentralized Exchange by using the Elixir ICON 2.0 SDK I wrote in the past few months.

Without further ado, let's go!

Prerequisites

This article assumes you have downloaded icon either within a new project using mix new or using a script e.g:

#/usr/bin/env elixir

Mix.install([
  {:icon, "~> 0.1"}
])

# ... rest of the script ...

You can also find the full script for getting sICX/bnUSD quotes here.

Quote Prices

When we want to exchange one token for another, we're presented with token pairs. These pairs have a base token, a quote token and, a quote price:

If we, somehow, capture the event logs emitted every time someone exchanges one token for another, then we'll have realtime quote prices for any pair we want. In this article, we'll subscribe to Balanced sICX/bnUSD prices.

Note: sICX is staked ICX token and bnUSD is an algorithmic stablecoin pegged to the dollar called Balanced Dollars.

Getting the Swap Log

Balanced is built from several SCOREs (Smart Contract On Reliable Environment) and one of these SCOREs handles all operations related to the decentralized exchange (DEx). The main DEx operation is swapping tokens e.g. we can use the contract to exchange our sICX with bnUSD.

Furthermore, every time someone swaps a token, the SCORE will emit the following event:

Swap(int,Address,Address,Address,Address,Address,int,int,int,int,int,int,int,int,int)

We can query the SCORE's API with Icon.get_score_api/2 and get the meaning of each input. For that, we'll need the SCORE address, which is cxa0af3165c08318e988cb30993b3048335b94af6c in the Mainnet:

dex_score = "cxa0af3165c08318e988cb30993b3048335b94af6c"
identity = Icon.RPC.Identity.new(network_id: :mainnet)

{:ok, api} = Icon.get_score_api(identity, dex_score)

swap_definition =
  Enum.find(api, fn %{"name" => name, "type" => type} ->
    name == "Swap" and type == "eventlog"
  end)

In the variable swap_definition, we'll have the Swap event log definition:

%{
  "inputs" => [
    %{"indexed" => "0x1", "name" => "_id", "type" => "int"},
    %{"indexed" => "0x1", "name" => "_baseToken", "type" => "Address"},
    %{"name" => "_fromToken", "type" => "Address"},
    %{"name" => "_toToken", "type" => "Address"},
    %{"name" => "_sender", "type" => "Address"},
    %{"name" => "_receiver", "type" => "Address"},
    %{"name" => "_fromValue", "type" => "int"},
    %{"name" => "_toValue", "type" => "int"},
    %{"name" => "_timestamp", "type" => "int"},
    %{"name" => "_lpFees", "type" => "int"},
    %{"name" => "_balnFees", "type" => "int"},
    %{"name" => "_poolBase", "type" => "int"},
    %{"name" => "_poolQuote", "type" => "int"},
    %{"name" => "_endingPrice", "type" => "int"},
    %{"name" => "_effectiveFillPrice", "type" => "int"}
  ],
  "name" => "Swap",
  "type" => "eventlog"
}

Analyzing the Swap Event

With the previous definition, we can start making some assumptions about each event input and what things we need to get from them in order to have the price for sICX/bnUSD every time the event is emitted:

_id is the token pair identifier (indexed 1st position).
_baseToken is the SCORE address of the base token of the pair (indexed 2nd position).
_fromToken is the SCORE address of the token being sold (data 1st position).
_toToken is the SCORE address of the token being bought (data 2nd position).
_timestamp is the UNIX epoch timestamp in UTC in microseconds (data 7th position).
_endingPrice is the quote price after the swap in loop (data 12th position).

Either partially or fully, we can use the previous inputs to find the quote price we're looking for.

Protip: We can query the pool stats with the DEx function getPoolStats and search by ID. This will give us sICX/bnUSD pair has the ID 2. You can check it out yourself by running the following:
Icon.call(identity, dex_score, "getPoolStats", %{_id: 2},
  call_schema: %{_id: :integer}
)
At the time of writing this article, there are 39 pools, but not all of them have a name, so in this case using _baseToken, _fromToken, and _toToken addresses can help us figure out the actual pair name.

Realtime Updates

Elixir ICON 2.0 SDK has an Yggdrasil adapter for ICON's websocket. Thus we can use an Yggdrasil process to subscribe to the Swap event. In this case, the most important part is to define out channel correctly:

defmodule Quotes do
  use Yggdrasil
  alias Icon.Schema.Types.EventLog

  @dex_contract "cxa0af3165c08318e988cb30993b3048335b94af6c"
  @sicx_bnusd_id 2
  @signature "Swap(int,Address,Address,Address,Address,Address,int,int,int,int,int,int,int,int,int)"

  @channel [
    adapter: :icon,
    name: %{
      source: :event,
      data: %{
        addr: @dex_contract,
        event: @signature,
        indexed: [@sicx_bnusd_id, nil]
      },
      from_height: 47_077_000
    }
  ]

  def start_link, do: Yggdrasil.start_link(__MODULE__, [@channel])

  # ... handle_event/3 definition ...
end

Protip: I added from_height: 47_077_000 field to the channel's name for subscribing to an older block height. This way, we don't need to wait for a Swap event to happen and we can see some results right away. In general, this is only needed for testing purposes, but it wouldn't be needed if we just want the latest price.

Showing the Prices

The events we'll receive will have:

The _timestamp in microseconds in the 7th position of data.
The _endingPrice in the 12th position of data (_endingPrice * 10¹⁸).

So we can now define our handle_event/3 callback and finally print our quote price in the console:

defmodule Quotes do
  use Yggdrasil
  alias Icon.Schema.Types.EventLog

  # ... channel definition ...

  @impl Yggdrasil
  def handle_event(_channel, %EventLog{} = swap_event, _state) do
    [_, _, _, _, _, _, timestamp, _, _, _, _, price, _] = swap_event.data

    datetime =
      timestamp
      |> DateTime.from_unix!(:microsecond)
      |> DateTime.to_iso8601()

    price = price / 1_000_000_000_000_000_000

    IO.puts("[#{datetime}] sICX/bnUSD price: #{price}")

    {:ok, nil}
  end
end

If we execute this process:

Quotes.start_link()

then we'll get the following output:

[2022-03-07T16:17:15.095799Z] sICX/bnUSD price: 0.6762914917929946
[2022-03-07T16:33:31.390351Z] sICX/bnUSD price: 0.6765223727152583
[2022-03-07T16:19:25.140181Z] sICX/bnUSD price: 0.6763077935623969
[2022-03-07T16:06:10.885982Z] sICX/bnUSD price: 0.6696113966372721
[2022-03-07T16:28:17.210851Z] sICX/bnUSD price: 0.6765180152082558
[2022-03-07T16:27:41.288288Z] sICX/bnUSD price: 0.6764913565725723
... continues ...

Note: You can check out the full script here.

Conclusion

Though BTP is not fully released, we can already start leveraging some of its core features for our own advantage. Given ICON's block time is 3 seconds, this websocket connection gives us a tremendous advantage for building realtime bots for different purposes: from maximizing our yield to securing our favorite NFT latest drop.

ICON 2.0 definitely has a great advantage over many other slow blockchains and I believe its DeFi ecosystem will have a bright future!

Cover image by Maxim Hopman

My Worst Bug

Alex de Sousa — Wed, 22 Apr 2020 19:08:20 +0000

Some bugs pull you to the abyss and leave you there. They're the Balrog to our Gandalf. Once you beat them though, you're wiser and more powerful!

I was building a small plugin in C++ for an MT4 Server (ForEx trading server). The output of the project was a Windows DLL. Using the server's protocol, I managed to get JSON strings and parse them with RapidJSON. Everything ran smoothly in my virtual machine and in some development servers. Even Valgrind couldn't find memory leaks. I thought the plugin was ready for production...

Oh boy was I so wrong!

Note: RapidJSON is an amazing library and if I need to parse JSON in C/C++ again, I would use it without hesitation.

Debugging The Problem

Once I deployed the plugin, everything seemed fine... until the next day! A nasty segmentation fault killed the server. The plugin made us loose some money and I had to roll back the deploy.

After several days of testing, I realized the production server always died with the same set of data. I was able to pinpoint the error to RapidJSON. Something weird was happening when the memory was allocated, but none of the tools I was using to debug this were reporting any problems.

I was desperate, so I compiled the DLL with debug symbols and then I de-compiled it using OllyDBG.

I started reading the DLL assembly code ... for a week and a half! Reading assembly was horrible. I considered switching careers. But then I got to the instruction that failed! Eureka! I couldn't believe it! It felt good to finally understand the bug!

The Bug!

The problem was that RapidJSON's custom allocator:

Compressed the data in memory.
Allocated only what it needed.

The production machine architecture:

Allocated the memory RapidJSON asked for.
Ignored the way RapidJSON wanted the data to be structured.

It's easier to see with an image:

If RapidJSON needed to store an integer, a string and a boolean value, then it could fail randomly depending on the length of the string. e.g. given the integer 42 and the boolean true:

For the string Hey, id would succeed:

For the string Hi, it would fail:

A debugging nightmare!

The Solution

I just made RapidJSON use the machine's allocator instead of the custom one. Spent two weeks debugging something and changed just a single word in the code!

Conclusion

Now I avoid C/C++ at all costs!

I hope whatever bug you're dealing with at the moment gets solved soon!

Cover photo by Jez Timms

Yggdrasil as Distributed PubSub

Alex de Sousa — Thu, 16 Apr 2020 16:23:24 +0000

Yggdrasil's default adapter supports multi-node subscriptions out-of-the-box thanks to Phoenix PubSub. This distributed capabilities can be extended to any adapter compatible with Yggdrasil v5.0 without writing a single line of extra code.

Before We Start

I've used this example project for the code in this article. You can skip this section safely as long as you remember the following:

Basic project has only Yggdrasil.
Rabbit project has Yggdrasil for RabbitMQ.
A RabbitMQ server is available.
The host name is called matrix. Your machine's will be different.

If you want to follow along with the examples in this article, you can download the example project using the following command:

git clone \
  --depth 2 \
  -b blog \
  https://github.com/alexdesousa/alexdesousa.github.io.git examples && \
cd examples && \
git filter-branch \
  --prune-empty \
  --subdirectory-filter examples/matrix HEAD

In the folder you'll find:

Basic project that has a basic version of Yggdrasil.
Rabbit project that has Yggdrasil for RabbitMQ.
A docker compose with a RabbitMQ server:

   $ cd rabbit && docker-compose up

Basic Message Distribution

Yggdrasil's default adapter piggybacks on Phoenix PubSub for the message delivery, inheriting its distributed capabilities e.g. let's say we have the following:

The node :neo@matrix using Basic project:

   $ iex --sname neo -S mix

The node :smith@matrix also using Basic project:

   $ iex --sname smith -S mix

Both nodes are interconnected:

   iex(smith@matrix)1> Node.connect(:neo@matrix)
   true

Then :smith@matrix can subscribe to any channel where :neo@matrix is publishing messages e.g:

In :smith@matrix, we subscribe to the channel "private":

iex(smith@matrix)2> Yggdrasil.subscribe(name: "private")
:ok
iex(smith@matrix)3> flush()
{:Y_CONNECTED, %Yggdrasil.Channel{...}}
:ok

In :neo@matrix, we publish a message in the channel "private":

iex(neo@matrix)1> channel = [name: "private"]
iex(neo@matrix)2> Yggdrasil.publish(channel, "What's the Matrix?")
:ok

Finally, we can flush :smith@matrix mailbox and find our message:

iex(smith@matrix)4> flush()
{:Y_EVENT, %Yggdrasil.Channel{...}, "What's the Matrix?"}
:ok

Distributed pubsub as simple as that.

Bridged Message Distribution

The bridge adapter makes a bridge between any Yggdrasil adapter and the default adapter. This allows adapters to inherit the distributed capabilities of the default adapter e.g. let's say we have the following:

The node :neo@matrix using Basic project:

   $ iex --sname neo -S mix

The node :trinity@matrix using Rabbit project:

   $ iex --sname trinity -S mix

The node :trinity@matrix has access to a RabbitMQ server.
Both nodes are interconnected:

   iex(trinity@matrix)1> Node.connect(:neo@matrix)
   true

So our final infrastructure would look like the following:

Through :trinity@matrix, the node :neo@matrix can now subscribe to
a RabbitMQ exchange:

iex(neo@matrix)1> channel = [name: {"amq.topic", "private"}, adapter: :rabbitmq]
iex(neo@matrix)2> Yggdrasil.subscribe(channel)
:ok
iex(neo@matrix)3> flush()
{:Y_CONNECTED, %Yggdrasil.Channel{...}}
:ok

Or even publish messages:

iex(neo@matrix)4> Yggdrasil.publish(channel, "What's the Matrix?")
:ok
iex(neo@matrix)3> flush()
{:Y_EVENT, %Yggdrasil.Channel{...}, "What's the Matrix?"}
:ok

The good thing about this feature is that it works with any adapter that supports Yggdrasil v5.0.

Conclusion

Yggdrasil bridge adapter allows you to convert any adapter in a distributed one by relying in Phoenix PubSub.

I hope you found this useful and happy coding!

Cover photo by Quino Al

My Road to 8 Week Streak

Alex de Sousa — Fri, 10 Apr 2020 19:06:57 +0000

9 weeks ago, I challenged myself to consistently write an article every week for at least 8 weeks. Today, I can say I did it. It was hard. Not the hardest thing I've done, but hard nonetheless.

It wasn't just about writing something and get done with it for the week. The internet is already filled with shallow useless content. I didn't need to contribute to that. I wanted to write good content. Enjoyable and memorable content. Content that made me proud.

Writing is hard. A blank page is intimidating. Hitting the publish button and exposing your thoughts to the masses is scary. Quitting is easy, but unforgivable.

Nonetheless, I wrote.

Good Content

Beating the challenge wasn't the goal. The challenge kept my eyes on the goal. The goal was to promote my personal brand and my open source projects. I chose content marketing as my tool to reach the goal.

Everything I know about content marketing I learnt from my girlfriend. I'm not a marketing expert, but rule number one of content marketing is: write good content. The vaguest rule ever. So, what's good content?

Know Your Audience

It's nice when someone shares your content. You reached them. Your content was valuable enough for them. They are your target audience.

I love Elixir. I feel comfortable sharing my knowledge about it. Most of my projects are written in Elixir. My target audience was anybody interested in Elixir. So... I mostly wrote about Elixir.

Shareable Content

Writing for your target audience is not enough. Your content needs to be shareable. And the most important thing: it needs to be shareable by the right people.

The right people are the experts in the area you chose. They have read a gazillion times about list comprehensions in Elixir. They haven't read about that time you implemented Prolog's backtracking using list comprehension in Elixir.

Write about the cool stuff you have done. The things you're proud of.

Enjoyable Content

In the era of short attention span, you're worst enemy is sudden boredom. Use GIFs. Use images. Use graphics. Entertain while sharing knowledge.

Your article's content is competing with a sudden WhatsApp message with a meme or that Prime Video notification saying The Expanse new season is available now.

Findable Content

It doesn't matter you're the best technical writer in the world if nobody reads you. You need to distribute your content.

This greatly depends on the area you chose. In my case, Elixir's community is small and getting into the newsletters is not that hard as long as your content is valuable.

Don't wait for people to discover you. Reach out for them.

Important: Don't spam. Nobody likes spammers.

Measuring Success

Thinking you're writing good content is different than writing good content. I needed a way to measure my success. I tracked several metrics along the way:

Number of DEV views.
Number of DEV reactions.
Number of DEV followers.
Number of Github stars.

Additionally to those quantitative metrics, I also took into consideration whether the articles were published in a newsletter or not:

I created a spreadsheet in Google Sheets and with this script I periodically gathered the data. This approach was inspired by/stolen from this article.

I wasn't that rigorous with Twitter metrics, but I still kept them in mind when writing a tweet sharing my next article:

Number of Twitter likes.
Number of Twitter retweets.
Number of Twitter followers.

Results

My articles were well received by the community. I got useful feedback that has improved both my projects and blog posts. I even got great pull requests to some of my projects and that makes me happy. People not only find my articles, but also my projects, useful!

The following is a performance summary for all blog posts and open source projects:

Metric (Number of)	Before	Week 9	Change
DEV views	0	3319	+3319
DEV reactions	0	125	+125
DEV followers	0	164	+164
Github stars	49	170	+121
Twitter followers	2	29	+27
Elixir Weekly publications	0	4	+4
Elixir Radar publications	0	1	+1

Lessons

Consistently writing good content is hard. The following are some of the lessons learnt these past 8 weeks:

Choose a publishing day: My days were Thursdays. I needed to hit the publish button no matter what. Pressure is good.
Pick your article's topic one week before you publish it: Some articles require experimentation and research. Good content requires time. Don't rush it.
Pick a backup topic: If your main article is not ready for publishing, then publish a good article about the backup topic.
Write in beast mode: Write what's on your mind. Don't try to write a perfect article. Just write. Once your ideas are written, you can edit.
Distribute your content: Once you publish the article, distribute it in every relevant channel you have at your disposal.

Conclusion

Writing is hard. Writing requires practice. The more you practice, the easier it gets. Just do it!

Happy writing!

Cover image by Aaron Burden

AyeSQL: Writing Raw SQL

Alex de Sousa — Thu, 02 Apr 2020 16:20:00 +0000

Most developers consider that writing raw SQL (Structured Query Language) is a bad practice. The main arguments against it are that Object Relational Mappers (ORMs):

Abstract several SQL dialects.
Have a flat learning curve unlike SQL.
Optimize queries.

ORMs allow you to abstract queries in an almost magical way:

# INSERT INTO user (name, age)
#   VALUES ('Bob', 18)
#   ON CONFLICT(name)
#   DO UPDATE SET name = EXCLUDED.name
#   RETURNING name, age
User user = new User("Bob", 18)

# UPDATE user
#   SET age = 17
#   WHERE name = 'Bob'
user.age = 17

Before we start

The contents of this article are a mix between my work experience and the great insights Mastering PostgreSQL in Application Development book provides.

The code for this article can be found here and it can be downloaded with the following command:

git clone \
  --depth 2 \
  -b blog \
  https://github.com/alexdesousa/alexdesousa.github.io.git examples && \
cd examples && \
git filter-branch \
  --prune-empty \
  --subdirectory-filter examples/f1 HEAD

"My ORM abstracts several SQL dialects"

In my experience, I've found that it's more likely for an application to be re-written in another language e.g. Ruby to Elixir than actually migrating the data from one database to another e.g. MySQL to PostgreSQL.

Additionally, running the database migrations in a different database is not a walk in the park either. Complex applications end up adding custom SQL to their migrations in order to optimize or automate certain processes: database extensions, indexes and triggers.

And then, the data needs to be migrated e.g. pgloader is an amazing tool that migrates data from MySQL, SQLite or MSSQL to PostgreSQL. However, sometimes these tools don't work correctly or they are nonexistent.

"My ORM has a flat learning curve unlike SQL"

SQL is different. It's a functional language and most programmers are used to imperative languages. It's also hard to master.

Ironically, the queries are almost in natural language e.g. if we show the following sentences to someone who doesn't know SQL, probably they'll know they have the same objective:

Get the name and age of every client older than or equal to 18.
SELECT name, age FROM client WHERE age ≥ 18

ORMs specifically work great for simple queries. However, when you need a more complex query, you end up doing certain workarounds to bypass the ORMs limitations:

Writing raw SQL directly.
Querying the database several times (sometimes unknowingly) to retrieve the data we need and doing the computation in our language instead of the database.

For now I'll focus on the second approach and I'll talk about the first one later. It's easier to see it with a small example:

Given the following subset of tables from the Chinook database:

We want to know the duration of every album Pearl Jam has ever made (or at least the ones listed in the database). In an OO language we would do something like:

Artist artist = Artist.one(name: "Pearl Jam")

for album in Album.all(artistId: artist.id)
  ms = 0

  for track in Track.all(albumId: album.id)
    ms += track.milliseconds

  print("#{album.title} | #{format_time(ms)}")

For a developer who doesn't know SQL, this would seem like reasonable code. In reality, it's highly inefficient:

We're requesting one artist by name:

SELECT * FROM "Artist" WHERE "Name" = 'Pearl Jam'

Then requesting all albums by artist id:

SELECT * FROM "Album" WHERE "ArtistId" = artist.id

For every album, we're requesting the tracks by album id:

SELECT * FROM "Track" WHERE "AlbumId" = album.id

This database contains 5 Pearl Jam albums, so we would be querying the database 7 times. The magic behind the ORM is deceiving us. In every round trip to the database, we need to consider two things:

Time the query takes to execute.
Time the network transmits the data.

Good ORMs introduce ways of dealing with this type of problem. However, knowing we need to use those tools in order to have good performance in our queries requires knowing SQL.

So, going back to the title of this section: Do ORMs really have a flat learning curve? Considering programmers need to know SQL to understand the tools ORMs offer and SQL is hard, then I guess they have not.

The previous problem can be solved with one query:

Note: Chinook database has capitalized table and column names. That's why they need to be between double quotes.

    SELECT "Album"."Title" AS album,
           SUM("Track"."Milliseconds") * interval '1 ms' AS duration
      FROM "Album"
INNER JOIN "Artist" USING("ArtistId")
 LEFT JOIN "Track" USING("AlbumId")
     WHERE "Artist"."Name" = 'Pearl Jam'
  GROUP BY album
  ORDER BY album

and our result would be already formatted:

          album          |   duration  
-------------------------+-------------
 Live On Two Legs [Live] | 01:11:18.954
 Pearl Jam               | 00:49:43.857
 Riot Act                | 00:54:16.468
 Ten                     | 00:53:25.871
 Vs.                     | 00:46:17.674

"My ORM optimizes queries"

Good ORMs do optimize queries. Usually they optimize queries for the most common cases. However, if the problem we're solving is not covered by these optimizations, it can be a real head-scratcher.

In the end, we end up with a subpar solution or writing raw SQL to overcome the limitations.

Language Mappers

There are alternatives to ORMs. In Elixir, we have Ecto. It's not an ORM, but a language mapper. It gives you a Domain Specific Language (DSL) for dealing with SQL queries in Elixir e.g. again if we want to know the duration of every album Pearl Jam has ever made (or the 5 listed in this database), we would do the following:

query =
  from al in Album,
    inner_join: ar in Artist, on: al.artistId == ar.artistId,
    left_join: tr in Track, on: tr.albumId == ar.albumId,
    where: ar.name == "Pearl Jam"
    group_by: al.title
    order_by: al.title
    select: %{album: al.title, duration: sum(tr.milliseconds)}

Repo.all(query)

Language mappers require developers to know SQL, but allows them to almost never leave the comfort of their language. That "almost" is important.

Though not everything is sunshine and rainbows. Language mappers try to abstract several SQL dialects in an unified DSL. This means that not all the SQL of our database is going to be included in this DSL.

To overcome this limitation, Ecto introduces the concept of fragments. Fragments are pieces of custom SQL code that can be added to our queries e.g. given the following subset from Ergast Developer API database:

Let's say we want to get the percentage of accidents per participant in F1 seasons between the years 1974 and 1990. In SQL, we would have the following:

WITH accidents AS
(
    SELECT EXTRACT(year from races.date) AS season,
           COUNT(*) AS participants,
           COUNT(*) FILTER(WHERE status = 'Accident') AS accidents
      FROM f1db.results
      JOIN f1db.status USING(statusid)
      JOIN f1db.races USING(raceid)
  GROUP BY season
)
  SELECT season,
         ROUND(100.0 * accidents / participants, 2) AS percentage
    FROM accidents
   WHERE season BETWEEN 1974 AND 1990
ORDER BY season

Now, when translating this query to Ecto, we'll have to use fragments:

# NOTE: The schema definition is omitted.

from = 1974
to = 1990

accidents =
  from re in Result,
    join: st in assoc(re, :status),
    join: ra in assoc(re, :race),
    group_by: fragment("season"),
    select: %{
      season: fragment("EXTRACT(year from ?)", ra.date),
      participants: fragment("COUNT(*)"),
      accidents: fragment("COUNT(*) FILTER(WHERE status = 'Accident')")
    }

query =
  from a in subquery(accidents),
    where: fragment("? BETWEEN ? AND ?", a.season, ^from, ^to),
    select: %{
      season: a.season,
      percentage: fragment("ROUND(100.0 * ? / ?, 2)", a.accidents, a.participants)
    }

F1.Repo.all(query)

The code above is hard to read, hard to write and error prone. Yet we don't have any performance improvements in our query. A maintenance nightmare.

We could re-write the previous Ecto query differently by encapsulating them in Elixir macros and then importing our custom DSL e.g:

defmodule CustomDSL do
  import Ecto.Query

  defmacro year(date) do
    quote do
      fragment("EXTRACT(year from ?)", unquote(date))
    end
  end

  ...
end

But again, we wouldn't get any improvements performance-wise. Just readability... in Ecto. It was perfectly readable in SQL.

The consequences are clear. Developers need knowledge of:

The specific SQL dialect (in this case PostgreSQL dialect).
Ecto's API and its limitations.
Elixir's macros for fragment encapsulation.

Additionally, now they need to maintain a new custom DSL API with its documentation. If we only have a few of this complex queries in our project, is it worthy?

What's worse, after the refactor, we could end up with a subpar solution or wasting our time entirely.

Raw SQL

So far we've seen ORMs and language mappers are good for general problems we might encounter. However, some other problems are better left in raw SQL.

ORMs, language mappers and database adapters usually provide an API for running raw SQL. In Elixir, Ecto and Postgrex give us the function query/2 e.g. in Ecto we would do the following:

query =
  """
  WITH accidents AS
  (
      SELECT EXTRACT(year from races.date) AS season,
             COUNT(*) AS participants,
             COUNT(*) FILTER(WHERE status = 'Accident') AS accidents
        FROM results
        JOIN status USING(statusid)
        JOIN races USING(raceid)
    GROUP BY season
  )
    SELECT season,
           ROUND(100.0 * accidents / participants, 2) AS percentage
      FROM accidents
     WHERE season BETWEEN $1 AND $2
  ORDER BY season
  """

Ecto.Adapters.SQL.query(F1.Repo, query, [1974, 1990])

There are some things I'd like to point out from the previous code:

$1 and $2 are the query parameters: The numbers indicate the position in the parameter list. If we add status as a variable instead of the constant 'Accidents', we would need to update the other indexes.
The query is a string: usually editors wouldn't highlight the SQL syntax inside the string.
The columns and rows are separated in the result:

   {:ok,
    %Postgrex.Result{
      columns: ["season", "percentage"],
      command: :select,
      connection_id: 206,
      messages: [],
      num_rows: 17,
      rows: [
        [1974.0, #Decimal<3.67>],
        [1975.0, #Decimal<14.88>],
        [1976.0, #Decimal<11.06>],
        [1977.0, #Decimal<12.58>],
        [1978.0, #Decimal<10.19>],
        ...
   }}

The complexity it's still there.

Meet AyeSQL

alexdesousa / ayesql

Library for using raw SQL in Elixir

Inspired by Clojure library Yesql, AyeSQL tries to find a middle ground between raw SQL strings and SQL language mappers by:

Keeping SQL in SQL files.
Generating Elixir functions for every query.
Working out of the box with PostgreSQL using Ecto or Postgrex.
Being extendable to support other databases and outputs.
Allowing some query composition.

Defining the same query in AyeSQL, we would need the following:

A file for defining our query in raw SQL:

   -- file: lib/f1/query/season.sql

   -- name: get_accidents
   -- docs: Gets accidents in a season range.
   WITH accidents AS
   (
       SELECT EXTRACT(year from races.date) AS season,
              COUNT(*) AS participants,
              COUNT(*) FILTER(WHERE status = 'Accident') AS accidents
         FROM f1db.results AS results
         JOIN f1db.status AS status USING(statusid)
         JOIN f1db.races AS races USING(raceid)
     GROUP BY season
   )
     SELECT season,
            ROUND(100.0 * accidents / participants, 2) AS percentage
       FROM accidents
      WHERE season BETWEEN :from AND :to
   ORDER BY season

A file for declaring our queries as Elixir functions:

   # file: lib/f1/queries.ex

   import AyeSQL, only: [defqueries: 3]

   defqueries(F1.Query.Season, "query/season.sql", repo: F1.Repo)

Modify our configuration to run the queries by default:

   # file: config/config.exs

   import Config

   config :ayesql, run?: true

Then we can call our query as follows:

iex> F1.Query.Season.get_accidents(from: 1974, to: 1990)
{:ok,
 [
   %{percentage: #Decimal<3.67>, season: 1974.0},
   %{percentage: #Decimal<14.88>, season: 1975.0},
   %{percentage: #Decimal<11.06>, season: 1976.0},
   %{percentage: #Decimal<12.58>, season: 1977.0},
   %{percentage: #Decimal<10.19>, season: 1978.0},
   ...
 ]}

I don't know about you, but IMHO this seems like a maintainable way of dealing with complex SQL queries.

AyeSQL provides query runners for Ecto and Postgrex out-of-the-box. In the previous example we used Ecto's runner. If we need a different output or query a different database e.g. MSSQL, we could implement the behaviour AyeSQL.Runner for our use case.

Note: In the past year, my team and I had to write an Elixir application with a pre-existent MSSQL database. AyeSQL.Runner behaviour and TDS library allowed us to query the database with ease.

Conclusion

I'm hoping this article helps change the negative reputation raw SQL queries have.

Yesql inspired libraries like AyeSQL, can help you to be more productive when your ORM or language mapper have failed you.

I'll write more about AyeSQL features in future articles, but for now happy coding!

Cover image by Austin Neill

Elixir Pubsub In Less Than 50 Lines

Alex de Sousa — Thu, 26 Mar 2020 20:50:35 +0000

:pg2 is a mostly unknown, but powerful Erlang module. It provides an API for creating process groups.

Process Group

So, what's a process group? Well... it's a group of Erlang/Elixir processes.

Perhaps, the correct question would be, why do we care about process groups? Well, process groups are the foundation for publisher-subscribers (pubsubs for short).

PG2

Understanding :pg2 API and how it relates to a pubsub API will make it easier to understand:

Every process group is a channel e.g. a group called :my_channel is created:

   iex> :pg2.create(:my_channel)
   :ok

Every process in a group is a subscriber e.g. self() is part of :my_channel group:

   iex> :pg2.join(:my_channel, self())
   :ok

A publisher can send/2 messages to a channel e.g. the publisher gets all the members of the group :my_channel and sends "Some message":

   iex> members = :pg2.get_members(:my_channel)
   :ok
   iex> for member <- members, do: send(member, "Some message")

A subscriber will receive the messages in its mailbox:

   iex> flush()
   "Some message"
   :ok

A subscriber can unsubscribe from a channel e.g. self() leaves the group :my_channel:

   iex> :pg2.leave(:my_channel, self())
   :ok

A channel can be deleted:

   iex> :pg2.delete(:my_channel)
   :ok

And that's it! That's the API. And you know what's the best thing about it? It can work between connected nodes. Keep reading and you'll see :)

Implementing a PubSub

A PubSub has three main functions:

subscribe/1 for subscribing to a channel:

   def subscribe(channel) do
     pid = self()

     case :pg2.get_members(channel) do
       members when is_list(members) ->
         if pid in members do
           :ok                     # It's already subscribed.
         else
           :pg2.join(channel, pid) # Subscribes to channel
         end

       {:error, {:no_such_group, ^channel}} ->
         :pg2.create(channel)      # Creates channel
         :pg2.join(channel, pid)   # Subscribe to channel
     end
   end

unsubscribe/1 for unsubscribing from a channel.

    def unsubscribe(channel) do
      pid = self()

      case :pg2.get_members(channel) do
        [^pid] ->
          :pg2.leave(channel, pid)   # Unsubscribes from channel
          :pg2.delete(channel)       # Deletes the channel

        members when is_list(members) ->
          if pid in members do
            :pg2.leave(channel, pid) # Unsubscribes from channel
          else
            :ok                      # It's already unsubscribed
          end

        _ ->
          :ok
      end
    end

publish/2 for sending a message to a channel.

   def publish(channel, message) do
     case :pg2.get_members(channel) do
       [_ | _] = members ->
         for member <- members, do: send(member, message)
         :ok

       _ ->
         :ok
     end
   end

For a full implementation of PubSub you can check this gist.

I usually create a .iex.exs file in my $HOME folder and then run iex. You could do the same with the previous gist by doing the following:

~ $ PUBSUB="https://gist.githubusercontent.com/alexdesousa/4d592fe206cca17393affaefa4c8fd33/raw/4d84894f016bd9eef84bba647c77c62b9c9a6094/pub_sub.ex"
~ $ curl "$PUBSUB" -o .iex.exs
~ $ iex

Distributed PubSub

For our distributed experiment we'll need two nodes. My machine is called matrix and both nodes will be neo and trinity respectively:

:neo@matrix:

   alex@matrix ~ $ iex --sname neo
   iex(neo@matrix)1>

:trinity@matrix:

   alex@matrix ~ $ iex --sname trinity
   iex(trinity@matrix)1> Node.connect(:neo@matrix) # Connects both nodes

Now :neo@matrix can subscribe to :mainframe channel:

iex(neo@matrix)1> PubSub.subscribe(:mainframe)
:ok

And :trinity@matrix can send a message:

iex(trinity@matrix)2> PubSub.publish(:mainframe, "Wake up, Neo...")
:ok

Note: Sometimes it takes a bit of time for nodes to synchronize their process groups, so you might need to publish/2 your message twice.

Finally, :neo@matrix should receive the message:

iex(neo@matrix)2> flush()
"Wake up, Neo..."
:ok

And that's it. A powerful pubsub in a few lines of code thanks to :pg2.

Conclusion

Erlang has several built-in hidden gems like :pg2 that make our lives easier.

Happy coding!

Cover image by Nicolas Picard

Managing Dotfiles with Ansible

Alex de Sousa — Thu, 19 Mar 2020 17:32:05 +0000

When developers get tired of configuring again and again our machines, we tend to create a dotfiles repository.

A Dotfiles Repository

Dotfiles are commonly used for storing user preferences or preserving the state of a utility, and are frequently created implicitly by using various utilities.

It all starts with a small repository that contains configuration files for common tools e.g. .zshrc, .vimrc, etc. However, every time we configure a new machine, we need to copy those files by hand.

We pride ourselves with our ability to automate any task. Therefore, the next logical step for a dotfiles repository is to create a "small" shell script for automating tool installation and machine configuration.

All is well at first, but that "small" script ends up being hundreds of lines long and hard to maintain.

Ansible to the Rescue

Ansible is an configuration management tool. It provides its own language to describe system configuration.

This is not a new idea, but it feels like one: configure your own machine the same way you configure your servers... With Ansible.

Note: I'm using Debian throughout this tutorial, though the same can be accomplished with any architecture and operative system as long as Ansible is available for it.

Bootstrapping Ansible

For installing and configuring stuff with Ansible, we need to first install Ansible. We can use a small shell script to accomplish this. Some of the variables like $HOSTS and $PLAYBOOK will make sense in the next sections:

#!/usr/bin/env bash

set -e

# Dotfiles' project root directory
ROOTDIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# Host file location
HOSTS="$ROOTDIR/hosts"
# Main playbook
PLAYBOOK="$ROOTDIR/dotfiles.yml"

# Installs ansible
apt-get update && apt-get install -y ansible

# Runs Ansible playbook using our user.
ansible-playbook -i "$HOSTS" "$PLAYBOOK" --ask-become-pass

exit 0

Running the previous script as our sudoer user will effectively install Ansible and run our main playbook with all our roles.

Playbook: A file that defines several tasks to be executed in a target machine.
Role: Organizes multiple, related tasks with the data needed to run those tasks (variables, files, templates).

Basic Folder Structure

Once we have our bootstrap script in place, we can start writing Ansible configuration files.

In our dotfiles repository, our target is our own machine. This is very easy to define in our hosts file (with local connection so it doesn't require an ssh key):

# file: hosts
[local]
localhost

[local:vars]
ansible_connection=local

Note: In our bootstrap.sh script, this file is found in the variable $HOSTS.

Then we need a playbook that lists every role we want to deploy and configure in our machine.

# file: dotfiles.yml
- name: Set up local workstation
  hosts: local
  roles:
    - role: zsh
      tags:
        - zsh

In our example, we'll install zsh role.

Note: In our bootstrap.sh script, this files is found in the variable $PLAYBOOK.

In general, our folder structure would look something like:

└ dotfiles
  - bootstrap.sh
  - dotfiles.yml
  - hosts
  └ roles
    └ zsh
      └ files
        - zshrc.link
      └ tasks
        - main.yml

ZSh role

The following would be our zsh role tasks for:

Installing zsh.
Setting it up as our default shell.
Installing Oh-My-ZSH.
Linking our .zshrc configuration file to our home folder.

--------
# file: roles/zsh/tasks/main.yml

- name: Installs zsh | apt
  become: yes
  become_user: root
  apt:
    name: zsh
    state: present

- name: Installs curl | apt
  become: yes
  become_user: root
  apt:
    name: curl
    state: present

- name: Sets zsh as default shell for my user | command
  become: yes
  become_user: root
  command: chsh -s /bin/zsh {{ lookup('env' 'USER') }}
  register: zsh_for_user
  failed_when: zsh_for_user.rc >= 1
  changed_when: zsh_for_user.rc == 0

- name: Checks for oh-my-zsh installation | stat
  stat:
    path: "{{ lookup('env', 'HOME') }}/.oh-my-zsh"
  register: oh_my_zsh_stat

- name: Installs oh-my-zsh | raw
  raw: 'sh -c "$(curl -fsSL https://raw.githubusercontent.com/robbyrussell/oh-my-zsh/master/tools/install.sh)"'
  when: not oh_my_zsh_stat.stat.exists

- name: Links .zshrc file | file
  file:
    src: "{{ lookup('env', 'ROOTDIR') }}/roles/zsh/files/zshrc.link"
    dest: "{{ lookup('env', 'HOME') }}/.zshrc"
    state: link
    force: yes

Running Our Bootstrap Script

Finally, we can run our script:

~/dotfiles $ chmod +x bootstrap.sh
~/dotfiles $ sudo bootstrap.sh

This script will:

Install Ansible.
Run our playbook, which means installing and configuring zsh.

Conclusion

There are more cool features you can use to customize your system using Ansible. You can check my dotfiles repository if you want to see a fully working example.

Additionally, if you want to know more about Ansible, you can check this amazing tutorial.

I hope you find this useful :)

Happy coding!

Cover image by Josh Riemer

Oath: Don't Loose Your Keys!

Alex de Sousa — Thu, 12 Mar 2020 22:25:32 +0000

Recently, I needed to reset my phone. I spent some time backing up my pictures and documents. Everything was going great. But then, I hit a roadblock.

The problem

One-time passwords have become very handy for logging into several sites from Twitter to Coinbase. This passwords are 6 digit tokens generated using the current time and a private key. I was using Google Authenticator for getting my one-time passwords. Sadly, this app does not provide a way to backup the private keys.

The damage was done. I couldn't retrieve the private keys, so I needed to regenerate all of them in every site individually. I thought about my future self dealing with the same issue and I knew I needed a sustainable solution.

The research

I wanted a one-time password solution that:

Didn't rely on my phone or any app.
Could also be used in my computer.
Was offline (no private keys stored in the cloud).

That's when I discovered oathtool: a command line tool for generating 6 digit tokens given a private key.

I installed it using sudo apt install oathtool

Generating a 6 digit token with oathtool is as easy as doing the following:

$ oathtool -b --totp 'MyPrivateKey'
798946

Discovering this tool was a good start, but I needed a good way of dealing with the private keys. Then I stumbled upon this article. The author created two scripts:

One for encrypting the private key into a file using gpg2.
One for decrypting the private key and retrieving the 6 digit token using oathtool.

Additionally, the 6 digit token was automatically copied to the clipboard using xclip.

I installed both tools by running sudo apt install gnupg2 xclip

I loved the solution! Though it had some flaws like storing temporarily an unencrypted file with the private key, it was a great idea :)

The plugin

I wrote Oath ZSH plugin by gathering the best ideas from that article. I ended up with the following commands:

Adding a private key:

   $ oath add twitter.com
   Private key:
   [SUCCESS]  Key created for twitter.com

Showing a 6 digit token (it'll ask for the gpg password):

   $ oath twitter.com
   123456
   [SUCCESS]  Code copied to clipboard

Deleting a private key (it'll ask for the gpg password):

   $ oath delete twitter.com
   [WARN]     Deleting /home/user/.oath/twitter.com/B743BC73B5F90E2305142D226BBCD02E89ABBC79.gpg.gpg
   [WARN]     Deleting /home/user/.oath/twitter.com
   [SUCCESS]  Key deleted for twitter.com

The same private keys I added to oath, I also added them to my phone's Google Authenticator app. That way both, my computer and phone, generate the same 6 digit token at a given time.

The only difference is that now I can backup everything. I just need to copy the following folders:

$HOME/.gnupg/: GPG folder with all the gpg keys.
$HOME/.oath/: Oath folder where all the private keys are stored.

For more info, visit Oath Github repository.

Conclusion

Though this solution might not be for everyone, it solves the problem I had. Now I can reset my phone at any time and not worrying about my private keys, because they're safely backed up.

Happy hacking!

Cover image by Chunlea Ju

Yggdrasil and RabbitMQ Subscriptions

Alex de Sousa — Thu, 05 Mar 2020 21:39:47 +0000

One of the features I really like about RabbitMQ is its queue routing. Its flexibility allows you to do interesting things without much of a hassle. But before I dig deep into RabbitMQ's routing capabilities, I would like to mention some concepts.

Connections and Channels

RabbitMQ uses not only connections, but virtual connections called channels. The idea of channels is to introduce multiplexing in a single connection. A small system could establish only one connection with RabbitMQ while opening a channel for every single execution thread e.g:

The rule of thumb would be to use:

One connection per application.
One channel per process in the application.

Note: Once our connection starts to be overloaded, we can start adding more connections to our connection pool.

With a normal RabbitMQ setup, we need to deal with:

Connection pools: avoiding over consuming resources.
Channel cleaning: avoiding channel memory leaks when they are not closed properly.
Fault-tolerant connections: supporting re-connections in case of failure or disconnection.
Re-connection back-off time: avoiding overloading the database on multiple re-connections.

Exchanges and Queues

An exchange is a message router. Every queue attached to it will be identified by a routing key. Typically, routing keys are words separated by dots e.g. spain.barcelona.gracia.

Additionally, routing keys support wildcards, for example: spain.barcelona.* will match messages with routing keys like spain.barcelona.gracia and spain.barcelona.raval.

It's easier to see these concepts with an image example:

In the previous image:

Publisher X and Publisher Y are sending messages to Exchange logs.
Subscriber A is subscribed to logs.*.
Subscriber B is subscribed to logs.error.

Then:

Publisher X message will end up in Queue logs.info.
Publisher Y message will end up in Queue logs.error.
Subscriber A will receive Publisher X and Publisher Y's messages.
Subscriber B will receive Publisher Y's message.

Handling Subscriptions in Yggdrasil

Handling RabbitMQ's complexity might be intimidating. Fortunately, Yggdrasil for RabbitMQ generalizes the complexity in order to have a simpler API.

The biggest difference with previous adapters is the channel name. Instead of being a string, it's a tuple with the exchange name and the routing key e.g:

A subscriber would connect to the exchange amq.topic using the routing key logs.* as follows:

iex(subscriber)> Yggdrasil.subscribe(name: {"amq.topic", "logs.*"}, adapter: :rabbitmq)
iex(subscriber)> flush()
{:Y_CONNECTED, %Yggdrasil.Channel{...}}

Note: The exchange must exist and its type should be topic. The exchange amq.topic is created by default in RabbitMQ.

Then a publisher could send a message to the exchange amq.topic using logs.info as routing key:

iex(publisher)> Yggdrasil.publish([name: {"amq.topic", "logs.info"}, adapter: :rabbitmq], "Some message")
:ok

Finally, the subscriber would receive the message:

iex(subscriber)> flush()
{:Y_EVENT, %Yggdrasil.Channel{...}, "Some message"}

Additionally, the subscriber can be written using the Yggdrasil behaviour:

defmodule Logs.Subscriber do
  use Yggdrasil

  def start_link(options \\ []) do
    channel = [
      name: {"amq.topic", "logs.*"},
      adapter: :rabbitmq
    ]

    Yggdrasil.start_link(__MODULE__, [channel], options)
  end

  @impl true
  def handle_event(_channel, message, _state) do
    ... handle event ...
    {:ok, nil}
  end
end

Lost Messages

Yggdrasil will acknowledge the messages as soon as they arrive to the adapter, then it will broadcast them to all the subscribers. If the adapter is alive while the subscribers are restarting/failing, some messages might be lost.

Though it's possible to overcome this problem with exclusive queues, this feature is not implemented yet.

Conclusion

Yggdrasil for RabbitMQ handles RabbitMQ complexity let's you focus in what really matters: messages.

Cover image by Aswathy N

Yggdrasil and PostgreSQL Notifications

Alex de Sousa — Wed, 26 Feb 2020 16:29:01 +0000

One thing I really like about PostgreSQL is its notifications via pg_notify. This feature is very useful when trying to get real-time notifications for certain changes in a databases.

PostgreSQL notifications

Creating notifications in PostgreSQL is very easy e.g. let's say we have a table for books:

-- User table creation
CREATE TABLE books (
  id SERIAL PRIMARY KEY,
  title TEXT NOT NULL UNIQUE
);

and we want JSON notifications in the channel new_books every time a new book is created in our database e.g:

{
  "id": 1,
  "title": "Animal Farm"
}

The trigger could be implemented as follows:

-- Trigger function creation
CREATE OR REPLACE FUNCTION trigger_new_book()
  RETURNS TRIGGER AS $$
  DECLARE
    payload JSON;
  BEGIN
    payload := json_build_object(
      'id', NEW.id,
      'title', NEW.title
    );

    PERFORM pg_notify('new_books', payload::TEXT);
    RETURN NEW;
  END;
  $$ LANGUAGE plpgsql;

-- Sets the trigger function in 'books' table
CREATE TRIGGER books_notify_new_book
  BEFORE INSERT ON books
  FOR EACH ROW
  EXECUTE PROCEDURE trigger_new_book();

Then, the following query would trigger our JSON message in the channel new_books:

INSERT INTO books (title) VALUES ('Animal Farm');

The Problem

Though subscribing to our database notifications can be done easily with Postgrex library, handling the connections to the database is a bit of a hassle. We need to ensure:

Connection multiplexing: avoiding over consuming database resources.
Fault-tolerant connections: supporting re-connections in case of failure or disconnection.
Re-connection back-off time: avoiding overloading the database on multiple re-connections.

The Solution

Yggdrasil for PostgreSQL is an adapter that supports all the features mentioned above while maintaining Yggdrasil's simple API e.g:

For our example, we could subscribe to the database messages by doing the following:

iex> Yggdrasil.subscribe(name: "new_books", adapter: :postgres, transformer: :json)
iex> flush()
{:Y_CONNECTED, %Yggdrasil.Channel{...}}

Running the following query:

INSERT INTO books (title) VALUES ('1984');

We will get the following message in IEx:

iex> flush()
{:Y_EVENT, %Yggdrasil.Channel{...}, %{"id" => 2, "title" => "1984"}}

Note: Yggdrasil comes with built-in message transformers. We've used
:json transformer for this example in order to get a map from the JSON
data.

Additionally, our subscriber could also be an Yggdrasil process e.g:

defmodule Books.Subscriber do
  use Yggdrasil

  def start_link(options \\ []) do
    channel = [
      name: "new_books",
      adapter: :postgres,
      transformer: :json
    ]

    Yggdrasil.start_link(__MODULE__, [channel], options)
  end

  @impl true
  def handle_event(_channel, %{"id" => id, "title" => title}, _state) do
    ... handle event ...
    {:ok, nil}
  end
end

It's also possbible to use Yggdrasil.publish/2 with PostgreSQL:

iex> message = %{"id" => 3, "title" => "A Brave New World"}
iex> Yggdrasil.publish([name: "new_books", adapter: :postgres, transformer: :json], message)

Conclusion

Yggdrasil for PostgreSQL simplifies subscriptions to PostgreSQL notifications and let's you focus in what really matters: messages.

Cover image by Nasa

Yggdrasil: Easy Pub-Sub in Elixir

Alex de Sousa — Thu, 20 Feb 2020 08:58:17 +0000

When I started coding in Elixir (around 2016), I was working for a financial company. Our product automatically invested money in the Forex market by copying traders' actions (market orders) in real time. We had the following:

In words, our system:

Subscribed to PostgreSQL for receiving trader actions.
Published to RabbitMQ for:
- Categorizing trader actions.
- And enqueuing trader actions in the proper queue.
Subscribed to Redis for receiving updates on prices.
Subscribed to several RabbitMQ queues for:
- Receiving the categorized trader actions.
- And deciding whether it should open/close some market orders or not.
Opened and closed market orders.

We needed to be able to communicate with three systems (PostgreSQL, RabbitMQ and Redis). However, in general, we only needed three actions:

subscribe/1 to a channel.
publish/2 a message in a channel.
unsubscribe/1 from a channel.

If we could generalize those three actions into an API, we could then implement three individual adapters for every system to handle the annoying stuff like disconnections, failures, resource management, protocols, etc.

Meet Yggdrasil

Handling subscriptions should be easy and, in an ideal world, we would only need to know where to connect and start receiving messages right away.

We shouldn't need to worry about secondary (yet relevant) things like disconnections, failures and managing resources.

Yggdrasil is an immense mythical tree that connects the nine worlds in Norse cosmology.

Yggdrasil was our pub-sub generalization. Using the strong foundations of Phoenix pub-sub library, we built an agnostic publisher/subscriber application that has:

Multi node support.
Simple API: subscribe/1, unsubscribe/1 and publish/2.
A GenServer wrapper for handling subscriber events easily.
A basic adapter for using Elixir message distribution.
Fault-tolerant adapters for:
- Redis.
- PostgreSQL.
- RabbitMQ.

One API to rule them all

Yggdrasil's API is very simple:

A process subscribes to "my_channel":

   iex> :ok = Yggdrasil.subscribe(name: "my_channel")
   iex> flush()
   {:Y_CONNECTED, %Yggdrasil.Channel{...}}

A process (in this case the same process) publishes the message "my message" in "my_channel".

   iex> :ok = Yggdrasil.publish([name: "my_channel"], "my message")

The message should be in the mailbox of the subscriber process:

   iex> flush()
   {:Y_EVENT, %Yggdrasil.Channel{...}, "my message"}

The subscriber can unsubscribe from the channel to stop receiving messages:

   iex> :ok = Yggdrasil.unsubscribe(name: "my_channel")
   iex> flush()
   {:Y_DISCONNECTED, %Yggdrasil.Channel{...}}

flush() cleans the IEx process mailbox. In general, receiving Yggdrasil messages should be the same as receiving messages when the sender uses send/2.

Yggdrasil behaviour

Yggdrasil provides a behaviour for writing subscribers easily. Following the previous example, the subscriber could be written as follows:

defmodule Subscriber do
  use Yggdrasil

  def start_link do
    channel = [name: "my_channel"]
    Yggdrasil.start_link(__MODULE__, [channel])
  end

  @impl true
  def handle_event(_channel, message, _state) do
    IO.inspect {:mailbox, message}
    {:ok, nil}
  end
end

This subscriber will print the message as it receives it:

iex> {:ok, _pid} = Subscriber.start_link()
iex> :ok = Yggdrasil.publish([name: "my_channel"], "my message")
{:mailbox, "my_message"}

An interesting side-effect is that now we can send messages to any process as long as they are subscribed to the right channel without needing to know the process PID or name.

Conclusion

Yggdrasil hides the complexity of a pub/sub and let's you focus in what really matters: messages.

Hope you found this article useful. Happy coding!

Cover image by Todd Quackenbush

Skogsrå: Simplifying Your Elixir Configuration

Alex de Sousa — Thu, 13 Feb 2020 10:14:22 +0000

Once an Elixir project is large enough, maintaining config files and configuration variables becomes a nightmare:

Configuration variables are scattered throughout the code so it's very easy to forget a configuration setting.
OS environment variables must be casted to the correct type as they are always strings.
Required variables must be checked by hand.
Setting defaults can sometimes be a bit cumbersome.
No type safety.

Ideally though, configurations should be:

Documented.
Easy to find.
Easy to read.
Declarative.

In summary: easy to maintain.

The problem

We'll elaborate using the the following example:

config :myapp,
  hostname: System.get_env("HOSTNAME") || "localhost",
  port: String.to_integer(System.get_env("PORT") || "80")

The previous code is:

Undocumented: hostname and port of what?
Hard to read: Too many concerns in a single line.
Hard to find: where are these hostname and port used?
Not declarative: we're telling Elixir how to retrieve the values instead of what are the values we want.

Conclusion: it's hard to maintain.

Writing a config module

We could mitigate some of these problems with one simple approach:

Create a module for your configs.
Create a function for every single configuration parameter you app has.

The following, though a bit more verbose, would be the equivalent to the previous config:

defmodule Myapp.Config do
  @moduledoc "My app config."

  @doc "My hostname"
  def hostname do
    System.get_env("HOSTNAME") || "localhost"
  end

  @doc "My port"
  def port do
    String.to_integer(System.get_env("PORT") || "80")
  end
end

Unlike our original code, this one is:

Documented: Every function has @doc attribute.
Easy to find: We just need to to look for calls to functions defined in this module.

However, we still have essentially the same code we had before, which is:

Hard to read.
Not declarative.

There's gotta be a better way!

There is a better way - Meet Skogsrå

Skogsrå is a library for loading configuration variables with ease, providing:

Variable defaults.
Automatic type casting of values.
Automatic docs and spec generation.
OS environment template generation.
Run-time reloading.
Setting variable's values at run-time.
Fast cached values access by using :persistent_term as temporal storage.
YAML configuration provider for Elixir releases.

The previous example can be re-written as follows:

defmodule Myapp.Config do
  @moduledoc "My app config."
  use Skogsra

  @envdoc "My hostname"
  app_env :hostname, :myapp, :hostname,
    default: "localhost",
    os_env: "HOSTNAME"

  @envdoc "My port"
  app_env :port, :myapp, :port,
    default: 80,
    os_env: "PORT"
end

This module will have these functions:

Myapp.Config.hostname/0 for retrieving the hostname.
Myapp.Config.port/0 for retrieving the port.

With this implementation, we end up with:

Documented configuration variables: Via @envdoc module attribute.
Easy to find: Every configuration variable will be in Myapp.Config module.
Easy to read: app_env options are self explanatory.
Declarative: we're telling Skogsrå what we want.
Bonus: Type-safety (see Strong typing section).

How it works

Calling Myapp.Config.port() will retrieve the value for the port in the following order:

From the OS environment variable $PORT.

From the configuration file e.g. our test config file might look like:

# file config/test.exs
use Mix.Config

config :myapp,
  port: 4000

From the default value, if it exists (In this case, it would return the integer 80).

The values will be casted as the default values' type unless the option type is provided (see Explicit type casting section).

Though Skogsrå has many options and features, we will just explore the ones I use the most:

Explicit type casting.
Defining custom types.
Required variables.
Strong typing.

Explicit type casting

When the types are not any, binary, integer, float, boolean or atom, Skogsrå cannot automatically cast values solely by the default value's type. Types then need to be specified explicitly using the option type. The available types are:

:any (default).
:binary.
:integer.
:float.
:boolean.
:atom.
:module: for modules loaded in the system.
:unsafe_module: for modules that might or might not be loaded in the system.
Skogsra.Type implementation: a behaviour for defining custom types.

Defining custom types

Let's say we need to read an OS environment variable called HISTOGRAM_BUCKETS as a list of integers:

export HISTOGRAM_BUCKETS="1, 10, 30, 60"

We could then implement Skogsra.Type behaviour to parse the string correctly:

defmodule Myapp.Type.IntegerList do
  use Skogsra.Type

  @impl Skogsra.Type
  def cast(value)

  def cast(value) when is_binary(value) do
    list =
      value
      |> String.split(~r/,/)
      |> Stream.map(&String.trim/1)
      |> Enum.map(String.to_integer/1)
    {:ok, list}
  end

  def cast(value) when is_list(value) do
    if Enum.all?(value, &is_integer/1), do: {:ok, value}, else: :error
  end

  def cast(_) do
    :error
  end
end

And finally use Myapp.Type.IntegerList in our Skogsrå configuration:

defmodule Myapp.Config do
  use Skogsra

  @envdoc "Histogram buckets"
  app_env :buckets, :myapp, :histogram_buckets,
    type: Myapp.Type.IntegerList,
    os_env: "HISTOGRAM_BUCKETS"
end

Then it should be easy to retrieve our buckets from an OS environment variable:

iex(1)> System.get_env("HISTOGRAM_BUCKETS")
"1, 10, 30, 60"
iex(2)> Myapp.Config.buckets()
{:ok, [1, 10, 30, 60]}

or if the variable is not defined, from our application configuration:

iex(1)> System.app_env(:myapp, :histogram_buckets)
[1, 10, 30, 60]
iex(2)> Myapp.Config.buckets()
{:ok, [1, 10, 30, 60]}

Required variables

Skogsrå provides an option for making configuration variables mandatory. This is useful when there is no default value for our variable and Skogsrå it's expected to find a value in either an OS environment variable or the application configuration e.g. given the following config module:

defmodule MyApp.Config do
  use Skogsra

  @envdoc "Server port."
  app_env :port, :myapp, :port,
    os_env: "PORT",
    required: true
end

The function Myapp.Config.port() will error if PORT is undefined and
the application configuration is not found:

iex(1)> System.get_env("PORT")
nil
iex(2)> Application.get_env(:myapp, :port)
nil
iex(3)> MyApp.Config.port()
{:error, "Variable port in app myapp is undefined"}

Strong typing

All the configuration variables will have the correct function @spec definition e.g. given the following definition:

defmodule Myapp.Config do
  use Skogsra

  @envdoc "PostgreSQL hostname"
  app_env :db_port, :myapp, [:postgres, :port],
    default: 5432
end

The generated function Myapp.Config.db_port/0 will have the following @spec:

@spec db_port() :: {:ok, integer()} | {:error, binary()}

The type is derived from:

The default value (in this case the integer 5432)
The type configuration value (see the previous Explicit type casting section).

Conclusion

Skogsra provides a simple way to handle your Elixir application configurations in a type-safe and organized way. Big projects can certainly benefit from using it.

Hope you found this article useful. Happy coding!

This article is also available here: https://thebroken.link/skogsra-simplifying-your-elixir-configuration/.

Cover image by Lukasz Szmigiel