DEV Community: José Diogo Viana

Sign in with Metamask using LiveView

José Diogo Viana — Fri, 05 May 2023 09:38:24 +0000

There's a chance that you have heard of Metamask in the past year or two. But, if you haven't, don't worry, as I'll try to explain it quickly. Metamask is an Ethereum crypto wallet, and serves as a gateway to dApps (decentralized Apps). Coinbase has the following definition for a crypto wallet:

Crypto wallets store your private keys, keeping your crypto safe and accessible. They also allow you to send, receive, and spend cryptocurrencies like Bitcoin and Ethereum.

For a more complete explanation, I recommend reading their full blog post, but for the ones that won't read it, I want to highlight that Metamask is just one option between several existing Hot Wallets (Online Wallets). However, for various reasons that don't matter here, Metamask grew into one of the most used wallets.

The main idea of my discussion today is not limited to the Metamask wallet only but can be applied to any other wallet as well. However, you'd need to adjust the code a bit for other wallets, as they may handle Javascript events differently. If you want to support multiple wallets in your dApp instead of doing all the manual work for each one individually, it's probably best to look into things like RainbowKit, wagmi, etc.

My teammate David Lange did an awesome talk for Coimbra.Blockchain where he talks about some of the hurdles and tools to use when developing a frontend for a dApp. I definitely recommend watching the presentation video. (It takes about 40 minutes.)

Why is it relevant?

By using a solution such as Sign in with Metamask (or any other wallet, as highlighted before) the user doesn't need to worry about an extra username + password combo for our App. Yes, I know that there are password managers, and it makes these worries a bit redundant, but let's be honest... End users of these apps don't want to trust us with their emails and passwords, even if they are random. Also, they already need to have a wallet to interact with other Blockchain features, so we might as well take advantage of that.

This can be used as a reliable authentication method because we know that only the person with access to that wallet's private key will be able to cryptographically sign a message, that when verified it'll correspond to the public key. Side note: in Ethereum, and some other blockchains, the public key corresponds to the Wallet's Address. We'll see below how this works with a bit more detail.

How does it work?

If you are still here, then I'm going to assume that you're familiar with how Asymmetric Encryption works (read this post if not).

The image below tries to represent how a crypto wallet can be used as a valid sign-in mechanism. The application's backend gives the User a message to sign with the wallet's private key. The backend then receives the encrypted message and uses the Wallet's Public Key (its Address, for Ethereum) to decrypt it and verify it against the original one.

Some of the more trained eyes may have seen a possible issue with the scheme above. Bear with me for a little more, because I'm going to address it in some paragraphs below.

An important note for this blog post is that the flow I'm talking about is for situations where we have a dedicated backend, and we want to maintain some kind of session between our App and the User. It's very frequent to have dApps that don't need a dedicated backend, so the User just connects its Wallet to the frontend, and can then interact directly with Smart Contracts running in the blockchain.

In the same way, Metamask gives you an interface to make transactions in the blockchain, it also allows the User to just sign simple text messages, using their private keys. The following image is one example. This action is completely off-chain, so therefore it doesn't have associated costs or delays.

Why Elixir and Liveview?

Well, why not? That's a valid reason, but I like to think that I have a more elaborate answer for this. To start, most projects in the Web3/Crypto space have separate frontends from the backend (at least the ones that have a dedicated backend). The reason for this is mainly because there are a lot of tools that can help with quick development if using any modern Javascript Framework.

In my case, I just wanted to do a simple API Playground where any user could interact with the project I was working on, but where it'd be possible to have an authenticated place. The API was all built with Elixir, and Phoenix 1.7 was a couple of days from being released, so I decided to take the opportunity and do something using LiveView, and have a learning experience.

A look into the desired Flow

I can't say I have used many crypto Wallets, but most of them work pretty much the same way. Before interacting with a dApp, you first need to connect the wallet to that website. I'll explain better what this is, down the line, but it's a required step regardless if you want the app to trigger any action from the wallet.

Focusing only on our goal of implementing a Sign In with Metamask flow, the base flow should look like this:

Connect the wallet;
Sign a message;
Validate the signature on the backend;
Create a Session token, if the signature is valid.

Above, we have already seen how we can sign messages using Metamask, but there is a problem with the flow I described... In that example, the user signs a message saying Here is a message!, and then the backend validates the signature, checking if it matches the original message. Even if you don't know much about cryptography, you can understand that if you always sign the same message contents, the resulting signature is the same, also. So, if your key is A and you are signing B, the final signature is always C. This way, if an attacker can intercept the Signature going from Metamask to the backend, then it'd be able to impersonate the actual wallet owner and sign in with our Application because the signature would be valid!

To solve that problem, we need to have a Nonce associated with each User in the database. With it, we make sure that every time a user is signing a message to log in to our app, that message has an extra number/string to make it different every time, therefore resulting in different signatures. Then, as our backend knows the base message, and the nonce given to the respective user trying to sign in, it's still possible to fully validate if the signature is valid or not.

All good up until now? It's time to dive into some code and see how all of this glues together.

Getting our hands dirty

The first step I took was to look for existing examples using Elixir and LiveView. I end up finding this interesting presentation, by Crystal Adkins, talking about what I wanted to learn about! The presentation links to the respective Github Repo of the project demonstrated during it. After exploring for a bit, I was able to understand how the interactions between LiveView and Metamask worked through JS, so I started working on my own version of the code.

Note: There are some differences between the project presented by Crystal Adkins and our use case. For example, you first had to sign in/up using email and password, and only then you could associate and connect Metamask with your account. In that codebase, you always sign the same message (the problem I mentioned), but I assume that it was just a Proof of Concept of the capabilities of LiveView. The same project also did more complex stuff like actually interacting with Smart Contracts, something that I'm not talking about today, and it wouldn't be present in our situation.

Second Note: The message structure being signed in my version, doesn't follow any Ethereum Improvement Proposals (EIPs) like EIP-4361. EIPs describe standards for the Ethereum platform, including core protocol specifications, client APIs, and contract standards. To implement something like that, you'd just need to change the message contents and add more information for User signing it.

To manage sessions and all the corresponding logic, I decided to use the mix phx.gen.auth Accounts User users generator, and remove all of the unwanted parts (like account confirmation, reset password flows, etc...). It's a pretty standard approach and works just fine.

Then I started the actual implementation of the Sign In button. This button is a LiveView component, and has the following mount function:

defmodule DoraWeb.MetamaskButtonLive do
  use DoraWeb, :live_view

  @impl true
  def mount(_params, _session, socket) do
    {:ok,
     assign(socket,
       connected: false,
       current_wallet_address: nil,
       signature: nil,
       verify_signature: false
     )}
  end

  # render logic and event handlers...
end

Looking at the assigns and explaining what each of them represents, is very simple:

connected: if the Metamask Wallet is already connected to our website or not. (The connect thing, I mentioned before). We can get this information from window.ethereum, and I'll explain what it is, next.
current_wallet_address: if the Wallet is already connected, or the User just connected, we can retrieve which Address it corresponds to. (Don't forget that here the Address is also the Public Key.)
signature: after triggering a signing request, this assign will hold the signature to be sent to the backend, for verification.
verify-signature: is supposed to work as a flag to trigger the verification process or not.

The render function is also understandable at first look. If we have @connected as true, then we can render the Sign In message, trigger a new Signature Request, and then the respective verification. If @connected is false, then we render Connect, and clicking on the button triggers the Wallet connection, instead.

@impl true
def render(assigns) do
  ~H"""
  <span title="Metamask" id="metamask-button" phx-hook="Metamask">
    <%= if @connected do %>
      <.form
        for={%{}}
        action={~p"/users/log_in"}
        as={:user}
        phx-submit="verify-current-wallet"
        phx-trigger-action={@verify_signature}
      >
        <.input type="hidden" name="public_address" value={@current_wallet_address} />
        <.input type="hidden" name="signature" value={@signature} />
        <.button class={button_css()}>
          <span class="w-6"><.metamask_icon /></span> Sign in
        </.button>
      </.form>
    <% else %>
      <.button class={button_css()} phx-click="connect-metamask">
        <span class="w-6"><.metamask_icon /></span> Connect
      </.button>
    <% end %>
  </span>
  """
end

Let's start by inspecting phx-click="connect-metamask" for the case where the Wallet is not connected. It relies on the push_event function. It's a small function, but very powerful in that way that enables us to easily communicate with the Javascript side of the App, by pushing an event to the client, and where we can interact directly with Metamask.

@impl true
def handle_event("connect-metamask", _params, socket) do
  {:noreply, push_event(socket, "connect-metamask", %{})}
end

How to interact with Metamask from Javascript?

MetaMask injects a global API into websites visited by its users at window.ethereum. This API allows websites to request users' Ethereum accounts, read data from blockchains the user is connected to, and suggest that the user sign messages and transactions.

Consequently, to interact with window.ethereum, we have Javascript libraries like Ethers or web3.js. I end up using Ethers, but you can select whatever you see fit. To handle the event above, you need something like:

const web3Provider = new ethers.providers.Web3Provider(window.ethereum)

window.addEventListener(`phx:connect-metamask`, (e) => {
    web3Provider.provider.request({method: 'eth_requestAccounts'}).then((accounts) => {
      if (accounts.length > 0) {
        signer.getAddress().then((address) => {
            this.pushEvent("wallet-connected", {public_address: address})
        });
      }
    }, (error) => console.log(error))
})

Upon successful connection of the Wallet, a new event "wallet-connected" is pushed with the respectively connected address. We can handle this event in our LiveView file, just like any other emitted by our template. When receiving that event, we set @connected to its new state, and we also update @current_wallet_address. By changing these two assigns the button will now render a different thing entirely because the Wallet is already connected to our app.

@impl true
def handle_event("wallet-connected", params, socket) do
  {:noreply,
   assign(socket,
     connected: not is_nil(params["public_address"]),
     current_wallet_address: params["public_address"]
   )}
end

Since we are already connected, when clicking our new button, the event emitted when trying to submit the form is phx-submit="verify-current-wallet". Even though this form is a post to ~p"/users/log_in", this action will only be triggered when @verify_signature is true, due to phx-trigger-action={@verify_signature} (check the px-trigger-action docs for more info on how it works).

The handler for this event looks for an existing account to return its current nonce, or if an account for that Address doesn't exist, then it generates a new nonce in the spot. This way, we don't distinguish if a User trying to sign a message to log in will have a valid account or not (the process will fail later instead). This nonce is then pushed to the Javascript client side and will be used for the signing process.

@impl true
def handle_event("verify-current-wallet", _params, socket) do
  nonce =
    case Accounts.get_user_by_eth_address(socket.assigns.current_wallet_address) do
      nil -> Accounts.generate_account_nonce()
      user -> user.nonce
    end

  {:noreply, push_event(socket, "get-current-wallet", %{nonce: nonce})}
end

The Javascript code is actually very similar, and through signer.signMessage(message) a window, similar to the image shown in the introduction part, will appear and the User will be able to sign that message. If the user ends up signing the message, we can push a new event with the result in the payload, and listen to it on the LiveView component side.

As a separate note, there are several events triggered by Metamask (and not our app) that you can also listen to, and make everything feel even snappier. I didn't dive into details, but here you can find docs for API description and the full list of events.

window.addEventListener(`phx:get-current-wallet`, (e) => {
    signer.getAddress().then((address) => {
        const message = `You are signing this message to sign in with Dora. Nonce: ${e.detail.nonce}`

        signer.signMessage(message).then((signature) => {
            this.pushEvent("verify-signature", {public_address: address, signature: signature})

            return;
        })
    })
})

Then, the event pushed by Javascript is handled again on the LiveView Component, and @verify_signature is marked as true. This way, the form submission is triggered at this time, and the post to ~p"/users/log_in" is done.

@impl true
def handle_event("verify-signature", params, socket) do
  {:noreply,
   assign(socket,
     signature: params["signature"],
     verify_signature: true
   )}
end

On the Session controller's side, we receive the form params (public_address and signature), and we verify the signature. If it's valid we call UserAuth.log_in_user(user, params), if it's not, we redirect back to ~p"/".

To verify the User's signature, ExWeb3EcRecover is being used. This is a "simple" library that exports one function recover_personal_signature/2. This function returns the address that created the signature for a personally signed message by an Ethereum Wallet. So, in our case, we can reconstruct the original messages by the User (we have its nonce stored in the DB still), and then we check if the Address trying to sign in with that signature is the same that signed the message in the first place. If so, we update the nonce to a new random value, and we return the user model as success.

def verify_message_signature(eth_address, signature) do
  with user = %User{} <- find_user_by_public_address(eth_address) do
    message = "You are signing this message to sign in with Dora. Nonce: #{user.nonce}"

    signing_address = ExWeb3EcRecover.recover_personal_signature(message, signature)

    if String.downcase(signing_address) == String.downcase(eth_address) do
      update_user_nonce(eth_address)
      user
    end
  end
end

ExWeb3EcRecover depends on ex_keccak, which means you'll need to have Rust installed to compile it. For a quick and easy Rust install follow these instructions.

As you probably have noticed, in our flow no new accounts are created. That's because, for this project, only pre-configured wallets should be able to sign in. However, the change needed to have that working as a sign up should be pretty simple. For example, in the code snippet for the wallet connected, we can do something like this:

@impl true
def handle_event("wallet-connected", params, socket) do
  address = params["public_address"]
  # also validate if is not nil! The User may not connect the wallet
  Accounts.create_user_if_not_exists(address) 

  {:noreply,
   assign(socket,
     connected: not is_nil(address),
     current_wallet_address: address
   )}
end

Wrapping up

This post had a bunch of code that may be hard to follow, but I tried to explain it as best as I can. Now, to fill in the gaps, feel free to look at the full code where this was implemented and the final result:

Github Repo
Example App deployed with Fly.io.

If you have any questions or comments, reach out in one of my socials. And as always, stick around for the next one 👋

How to build a Concurrent & Resilient Service in Elixir

José Diogo Viana — Thu, 16 Feb 2023 17:05:42 +0000

During the last 12 months, more or less, I had to build 3 different services that were highly concurrent and required some degree of resilience and fault tolerance. These services had different requirements and did completely different things, but after the third one, I noticed that I'm pretty much using the same recipe for the structure. That's what I'll talk about in this blog post, and at the end, I expect to have given enough insight into this recipe, so that you can also follow it in your next project or even make it better (if you have suggestions feel free to contact me at one of my socials).

Resilience & Fault Tolerance

In this title, I mention Resilience and there's also Fault Tolerance in the first paragraph. After all, what are these concepts about? I learned about them, more in-depth, back when I was getting my Master's Degree in Distributed Systems, and I've had them at the back of my mind every time I need to build a system that requires high availability and correctness.

If you want to be Fault Tolerant, then as the name says, it needs to be able to deal with possible failures that may occur. This ranges from writing defensive code, and being prepared to handle these errors, to the ability to maintain functionality when portions of a system break. The Wikipedia definition has the following analogy:

An example in another field is a motor vehicle designed so it will continue to be drivable if one of the tires is punctured, or a structure that is able to retain its integrity in the presence of damage due to causes such as fatigue, corrosion, manufacturing flaws, or impact.

The example above can also enter the realm of Resilience, but I see it more as the ability to recover from a partial or total crash. If you have a running service that shuts down for some reason, you may not need to do anything special when restarting, but for some occasions, it can be necessary to recover the previous known state and resume from that. These systems are very common when the previous state matters.

I think this Post from Uwe Friedrichsen makes a good summary of what Fault Tolerance is, and it also makes a comparison between both concepts, as they have a lot in common.

Please don't get confused or think I'm talking about Byzantine Fault Tolerance, as that's a matter for actual Distributed Systems, and not for Concurrent ones. Concurrency is also not the same thing as Parallelism, so what is Concurrency after all?

Concurrency

In computing, concurrency refers to multiple things that overlap in time, so that one starts before the other finishes, and they "fight" each other over CPU time. Parallelism is multiple things happening at the same time. Multiple processes or threads on a single time-sliced CPU might exhibit concurrency but not parallelism. Parallelism is about multiple tasks that literally run at the same time on hardware with multiple computing resources like a multi-core CPU. As per Rob Pike:

Concurrency is about dealing with lots of things at once. Parallelism is about doing lots of things at once.

In this blog post by Manning Free Content Center there's a great explanation including two images that inspired me to do my own below. You should definitely read it if you're interested in the matter. (Or if you just want the 5 year old explanation check Joe Armstrong's Concurrent and Parallel Programming.)

As for Distributed systems, well, they're distributed... Who would have guessed? 😅 This means that a distributed system is one in which components are decoupled from one another. Distribution also introduces issues of consistency and separate failure domains (that's where those Byzantine Faults enter). In a non-distributed system, consistency is often handled by that single machine and everything lives or dies together. If the hardware fully fails, then the service is completely down.

So here is a small summary just to consolidate knowledge before jumping on to the next phase of this post.

Concurrency (A robot only Singing and Dancing at each point in time).

Parallelism (A band performer Singing and Dancing at the same time).

Distributed (A full band performing together).

The BEAM & Elixir

As almost everybody at this point knows, Elixir compiles into BEAM (the Erlang Virtual Machine) byte code (via Erlang Abstract Format). This means that Elixir code can be called from Erlang and vice versa, without the need to write any bindings.

For the ones that don't know Erlang or its background, it's a language developed back in the ‘80s by Ericsson for better development and stability of telephony applications, and that was made open source in 1998.

This way, Elixir has all of the benefits of the battle-proven system that is Erlang, and it is possible to use the existing Erlang libraries with no performance penalty. Another strong point for Elixir is concurrency (you probably see where I'm going with the line of thought), which, in most languages, is a bit of a pain. Not only is dangerous, but it's also hard to achieve synchronisation. In Elixir it is effortless to create a new process, and it performs very well. It is a core feature of the platform and does not require a separate library. Note that Erlang/Elixir processes are not native processes; they are much more lightweight and are scheduled by the BEAM.

So, introductions aside, from this point I'm going to assume that you have at least the basic knowledge about the Erlang and Elixir ecosystems.

If you're reading this, there's a chance that you already built some Phoenix Apps and know how amazing it is to build upon it while keeping the concepts above somewhat abstracted under it. However, that's not why I'm here today. Have you ever needed to build a resilient and concurrent system, just like Phoenix, for example, inside your usual Backend (if you're into Monoliths) or as a stand-alone app (if you're into Microservices)? In the case you already have, you may see a lot of similarities between them, and it's those that I want to talk about and include in my recipe.

Taking advantage of Supervisors

Erlang has the Let it crash philosophy, and that's where Resilience and Fault Tolerance enter the scene. Don't get confused, the idea behind is not to let it crash and burn, but to let it crash and recover. Instead of having uncontrolled failures, Erlang turns failures, exceptions and crashes into tools that we can use and control.

The language comes with a lot of different tools, and you should definitely learn more about them. Here, I'll mostly prioritise Supervisors, but in this Medium post you can find more info on Processes, Monitors, and others.

Even though Supervisors come from Erlang, I want to clarify that from now on I'm talking of Supervisors on Elixir. Looking at the definition in the Elixir docs:

A supervisor is a process which supervises other processes, which we refer to as child processes. Supervisors are used for building a hierarchical process structure called a supervision tree. Supervision trees provide fault tolerance and encapsulate how our applications start and shut down.

The Supervisor process has the job of looking at the health of its child processes and acting accordingly to the specified configuration and strategy. These can range from configuring:

:shutdown: defines how a child process should be terminated.
:restart: defines when a terminated child process should be restarted.
:strategy: how the supervisor should act when a process terminates.
:max_restarts: the maximum number of restarts allowed in a time frame.

There are several more, and each of these has various options, but I feel that with the ones above you can already grasp what we can achieve with Elixir to quickly develop a Fault Tolerant and Resilient service.

Dynamic and Partition Supervisors

DynamicSupervisor: As the name implies, it's a supervisor that starts children dynamically. The Supervisor module (in Elixir) was designed to handle mostly static children that are started in the given order when the supervisor starts. A DynamicSupervisor starts with no children. Instead, children are started on demand. When a dynamic supervisor terminates, all children are shut down at the same time, with no guarantee of order. We'll see them in my simple framework for these services.
PartitionSupervisor: A supervisor that starts multiple partitions of the same child. Certain processes may become bottlenecks in large systems. If those processes can have their state trivially partitioned, in a way there is no dependency between them, then they can use the PartitionSupervisor to create multiple isolated and independent partitions. It's also possible to combine a PartitionSupervisor with a DynamicSupervisor (check this amazing post by AppSignal for a concrete example).

A simple Framework

We have the concepts in place, and we also know about Erlang/Elixir tools to manage processes, so it's time to jump into some actual code.

The following image tries to represent the base structure I've been using for these last 3 services I had to build/incorporate into an Elixir app. It all starts by starting a DynamicSupervisor under the main App Supervisor (usually where the Application code is). Then this DynamicSupervisor starts and manages all worker processes that should do all the heavy and concurrent lifting. Note that as stated above, the DynamicSupervisor can be replaced or mixed in with a PartitionSupervisor if the use case requires (in my situation, I haven't had to use them yet, but they are next in line if there is any bottleneck in the process).

With this approach, you can also guarantee that an unhealthy Worker Process won't affect the others, as they are separate entities. If you want to ensure synchronization between them, then adding a new Manager process can be an easy solution (check my previous blog post on Simulations with Elixir for a quick example). For the hard way but more robust, you probably need to look into Consensus Algorithms, but that's more in the realm of Distributed and not Concurrent only systems.

You probably noticed a new term in the image, theETS or Erlang Term Storage. An ETS allows us to store any Elixir term in an in-memory table and use it as a cache mechanism (learn more about them). I usually use this table to keep track of Worker Processes' PIDs and other metadata about each one of them (depending on the use case and the amount of information that each Worker needs to know about itself, or others). Be careful not to use it as a cache prematurely! It's an in-memory store only! For persistent solutions, we'll see other possibilities, below.

Usually, I also add some client functions to the DynamicSupervisor module, in order for Worker Processes to be able to update info on the ETS or trigger the start/stop of other Workers if needed. These functions can be abstracted into a separate Module if this one starts to grow indefinitely, and you feel they should be separated.

defmodule MyApp.MyModule do
  require Logger

  use DynamicSupervisor

  def start_link(init_arg) do
    with {:ok, pid} <- DynamicSupervisor.start_link(__MODULE__, init_arg, name: __MODULE__) do
        # Extra logic here if needed
        {:ok, pid}
    else
      _error ->
        Logger.error("Error starting MyApp")
    end
  end

  @impl true
  def init(_init_arg) do
    DynamicSupervisor.init(strategy: :one_for_one)
  end

  def start_worker_instance(args) do
    spec = {MyApp.MyModule.Worker, args}

    with {:error, error} <- DynamicSupervisor.start_child(__MODULE__, spec) do
      Logger.error("Error starting new Worker: #{inspect(error)}")
    end
  end

  def stop_worker_instance(some_id) do
    with {:ok, pid} <- get_pid_worker_instance(some_id) do
      Worker.stop(pid)

      {:ok, :closed}
    end
  end

  def get_pid_worker_instance(some_id) do
    case :ets.lookup(:my_ets_table, some_id) do
      [] -> {:error, :not_found}
      [{_some_id, pid, _rest}] -> {:ok, pid}
    end
  end

  def delete_worker_from_ets(some_id) do
    :ets.delete(:my_ets_table, some_id)
  end

  def insert_worker_in_ets(some_id, pid, rest) do
    :ets.insert(:my_ets_table, {some_id, pid, rest})
  end

  # ...
end

With the piece of code above, we already have a very basic supervision tree that can help us achieve a lot of things without much hassle. If a Worker Process dies, it's restarted, and the same thing for the DynamicSupervisor itself. Don't forget to start this Supervisor under the Application, or another already there.

The code for the Worker usually looks something like this:

defmodule MyApp.MyModule.Explorer do
  require Logger
  use GenServer, restart: :transient

  def start_link(args) do
    {:ok, pid} = GenServer.start_link(__MODULE__, args)

    MyApp.MyModule.insert_worker_in_ets(args.some_id, pid)
    send(pid, :start)
    Logger.info("Starting #{args.some_id}.")

    {:ok, pid}
  end

  def stop(some_id) do
    GenServer.stop(some_id)
  end

  @impl true
  def init(init_arg) do
    # new_state = some kind of computation on init_arg

    {:ok, new_state}
  end

  @impl true
  def handle_info(:start, state) do
      # start work
  end

  # ...
end

Note the usage of use GenServer, restart: :transient. This piece of code tells the Supervisor to not restart this process if it stopped in a normal/controlled way. Only in a case of a crash, it will be automatically restarted. (You can then play around with other options.)

Going the Extra mile

I want to reiterate that the approach, or "recipe" as I called it previously, is as simple as you can get in order to start. There are obviously a lot of extra things you can add to make your system even more robust to faults. Here are some of them, that I've used in the past.

Using a `:dets` and other disk storage mechanisms

Imagine the situation that our service is indexing Events that happen in a separate place. If the service completely fails, when restarting we want to resume/continue indexing in the place where we left off before the downtime. This way we won't have any duplicated events, and won't miss others while down.

We can achieve this with the help of disk storage-based mechanisms, like using a :dets or the Database itself. A :dets is very similar to the :ets, but it's disk-based instead. It features a lot of helper functions that you can find in the docs, including ways to quickly turn it into an ETS, if we want.

IMPORTANT NOTICE: if you use services like Heroku or Fly.io, relying on this file approach may not work! Changes to the filesystem on one machine may not be propagated to others and are not persisted across deploys and/or restarts (depending on service specifics, they don't work the same way).

To avoid the problem above, you can rely on your dedicated Database or other services like Redis to keep track of it. The start_link code for the Supervisor would look like this now:

def start_link(init_arg) do
  with {:ok, pid} <- DynamicSupervisor.start_link(__MODULE__, init_arg, name: __MODULE__) do
    :dets.open_file(:my_dets, [])

    restore_previous_state()

    {:ok, pid}
  else
    _error ->
      Logger.error("Error starting Dora")
  end
end

defp restore_previous_state do
  if :ets.whereis(:some_pid) == :undefined do
    :ets.new(:my_ets, [:set, :public, :named_table])
  end

  case :dets.match_object(:workers, {:_, :_, :_}) do
    [] ->
      MyApp.Workers.list_workers()
      |> Enum.each(&start_worker_instance(&1.some_id, &1.extra_info))

    list when is_list(list) ->
      Enum.each(list, fn {some_id, extra_info} ->
        start_worker_instance(some_id, extra_info)
      end)

    _ -> Logger.error("Some unexpected error happened!")
  end
end

Note that it's also required to add/remove workers' entries from these tracking systems when a new Worker Process is started/stopped with success.

You can check our Dora, the TipsetExplorer repo with this approach, as one of the projects we submitted for the FVM Space Warp Hackathon.

Process Pool

If your use case requires a lot of Worker Processes, and you don't want to flood the BEAM with new processes, you should definitely go with a Process Pool solution. You can easily exhaust your system resources if you do not limit the maximum number of concurrent processes that your program can spawn. The solution is to use a set of workers (processes) to limit the number of connections instead of creating a process for every individual action. For this, I would suggest two different approaches:

Poolboy: widely used lightweight, generic pooling library for Erlang that addresses this issue. This Elixir School guide is very helpful to get started quickly.
Custom solution: if Poolboy doesn't fit well within your constraints, you can always build your custom solution. For a previous personal project, I took the opportunity to build my custom solution, not because I had special requirements, but because I wanted to try my own thing. You can find the code for this custom pool here. The file code I linked is a "special type" of Worker Process, under the DynamicSupervisor supervision tree, but only keeps track of the pool and other Process management-related things.

Different Handlers that can be plugged and played

Suppose we continue our Indexing example, highlighted in the persistent-store mechanisms chapter, and we have different ways to listen/query for those events being emitted. We could have for example a queue/socket for passive listening, but the other service could also feature an HTTP API that we could poll.

This way if we want an even more fault tolerant and resilient service, we can make our Worker Processes use different Handlers to interact with the external system. So, in case we start listening to events via a socket/queue and the external service has a failure on that section, we can switch to automatically polling it through the HTTP API. This approach requires some overhead to manage which handlers are being used, and when to switch between them.

Wrapping up

We've now reached the end of this post... It was probably a bit too technical, but I tried to explain all base concepts in a way that any developer, with different degrees of experience, could understand the problem and the proposed approaches. If you have any suggestions to add under the Extra mile section or even the base Framework, feel free to reach out!

As hinted in the previous sub-section, there are a lot more things to be explored and looked at according to your needs. I don't believe this is a hammer, and all my problems are nails, and you shouldn't believe it also! However, I think it should give a good place to start for personal/side projects.

Image from: https://raspstephan.github.io/blog/hammer-and-nail/#

And as always, thank you for taking the time to read this, and see you in the next one 👋

Genetic Algorithms to optimize an Asset Portfolio

José Diogo Viana — Fri, 23 Dec 2022 21:54:58 +0000

On the last weekend of October, we (finiam) participated in ETHLisbon, an Ethereum-related hackathon, and our submitted project, Matusalem, resembled a pension fund scheme but only for Crypto Assets - I won't dwell here into much details of the project, as we'll probably have a dedicated blog post explaining it.

So, why am I talking about an Ethereum hackathon and Genetic Algorithms? Well, by the title of this post, you can probably guess why. For the last few weeks I had been reading Genetic Algorithms in Elixir by Sean Moriarity, and I was hooked on giving them a try in a use case outside the book. Fast forward to ETHLisbon, and our idea for the project, we found a place where Genetic Algorithms could be a nice path to explore, trying to figure out from a set of Crypto Assets which ones should make it into our pension fund portfolio, and I became responsible for it.

The base idea

Forgetting the details of how the whole thing would work, in this blog post, we'll focus only on the process of calculating the percentages of the funds we should allocate to each asset from a given list, using Genetic Algorithms.

Starting from the simplest use case, imagine we have 4 Assets to choose from and invest the money. Let's call them assets A, B, C, and D. Supposing the table below represents their expected returns and associated risk (1-7).

	Returns (%)	Risk Level
A	4%	1
B	2%	3
C	8%	7
D	5%	4

If we wanted to calculate the percentages we should invest in each asset, just by looking at the expected returns, it's clear that we should put all eggs in the basket C. However, things are just not that simple regarding investments, as we can understand that different assets have different risk levels. Normally, the higher the expected return, the higher the risk, but that's not always the case. So, our final goal, for this hackathon, was to optimize for a portfolio allocation that would maximize returns while bearing minimal risk. How can we do that? (At least in a mathematical way).

Sharpe Ratio

Here comes in Sharpe Ratio. We can evaluate the investment performance based on the risk-free return. The risk-free return is the return of an investment with zero risks for the investor (for example US Government Bonds, etc.).

A Higher Sharpe metric is always better than a lower one because a higher ratio indicates that the portfolio is making a better investment decision.

Since Genetic Algorithms are an optimization problem, and I'll explain further down how they work, we can use Sharpe Ratio as the metric we want to optimize. It takes into account expected returns and also associated risks. But how does it work, then?

The image is from WallStreetMojo's explanation of the Sharpe Ratio, and I recommend reading it if you're curious about these subjects.

If you're not familiar with Sharpe Ratio or Statistics in general, you may be asking how does the formula above take into account the assets' risks? Well, that part is handled by the portfolio's standard deviation.

Portfolio Standard Deviation refers to the volatility of the portfolio, which is calculated based on three important factors that include the standard deviation of each of the assets present in the total Portfolio, the respective weight of that individual asset in the total portfolio, and the correlation between each pair of assets of the portfolio.

This way, if a portfolio has a high standard deviation it means that the risk is high, it has more volatility and therefore not so stable returns. A portfolio with a low standard deviation has less associated risk.

We will see below how we can calculate the standard deviation for a portfolio and use it inside our Genetic Algorithms' fitness function. But what is a fitness function?

Genetic Algorithms

As stated above, Genetic Algorithms are used to solve optimization problems. An optimization problem is a problem of finding the best solution from all feasible solutions. Usually, the solution to that problem is hard to obtain using conventional mathematics, requiring a lot of computation (brute force).

One example of an optimization problem is the Knapsack problem.

Genetic Algorithms help us overcome this brute force approach, by using certain heuristics that enable us to find a "good enough" solution, but not necessarily the best one.

Despite the cool name, how does a Genetic Algorithm actually work? It tries to mimic evolution the same way it happens in nature. Concepts like Natural Selection are known by almost everyone, and if you're reading this post, you're probably familiar with it. Individuals with certain variants of a trait tend to survive and reproduce more than individuals with other less successful variants. This gives that phenotype (with valuable traits) a reproductive advantage and may become more common in a population - survival of the fittest. A key part of this is also Sexual Selection, where this form of selection, together with Natural Selection, means that some individuals have greater reproductive success than others within a population, for example, because they are more attractive or prefer more attractive partners to produce offspring. A Genetic Algorithm takes these concepts and runs a sort of simulation for an evolution of a population.

In his book Genetic Algorithms in Elixir, Sean provides a simple scheme like the one below, that summarizes really well the different components that take part in these algorithms.

What does it mean, and how does it relate to Darwin's theories? Let's imagine we have a population for a species of Unicorns and follow the diagram above.

Initialize Population: In our case the starting population doesn't evolve out of a different species, it just gets "initialized randomly" (we'll see how we can represent a population later). That means that we start with a population of individuals we know are unicorns (four legs, a spiralling horn projecting from its forehead, etc), but their traits (genotypes) would be randomized across the entire population. Traits like height, horn size, muscle density, etc. A good starting population for these algorithms should be random enough (but still inside valid boundaries) so that we don't simulate evolution with a starting bias. The population size also needs to be a finite number, for example, 100 individuals.
Evaluate Population: In this phase, we give each individual in our population a rating, accordingly to a pre-defined fitness function. After the rating process, all individuals are sorted by the rating they got. The best one is checked against the termination criteria. What are the termination criteria? It's a function that verifies if we have reached an optimal solution (it compares with a reference value) or could check if the rating is getting better each generation, if we've reached a specific number of generations, etc. If this function determines to not stop the process, the algorithm advances to step 3, otherwise, it returns the best individual and its rating (fitness function) providing us with the solution for the problem.
Select Parents: If the simulation continues to this step, then we must select the parents for the next generation. Selecting parents is basically following the idea stated in the Sexual Selection theory, and we group the best individual with the 2nd best, the 3rd with 4th, etc. for all individuals in the population set. This way, we plan to create children out of the best chromosomes, with the hopes of obtaining a fitter individual out of the match.
Create Children: After grouping each individual it's time to generate their children. To create children out of each pair of parents, we'll use crossover. Crossover is analogous to reproduction. It’s a genetic operator that takes two or more parent chromosomes and produces two or more child chromosomes. So, supposing we have parents P1 and P2, we can combine them by splitting their chromosomes randomly, and obtaining four chromosome parts P1-1, P1-2, P2-1, P2-2. Children for these two individuals are the chromosomes P1-1|P2-2 and P1-2|P2-1. Repeating this crossover process for all parent pairs.
Mutate Children: Despite initializing the population to a seemingly random distribution, eventually the parents get too genetically similar to make any improvements during the crossover. In order to reach a proper solution, Genetic algorithms, also need to insert some entropy into the simulation, by provoking "mutations". Mutations to children's chromosomes should happen with a relatively low frequency, for example, a 5% probability of a mutation occurring. Mutations in Genetic Algorithms could range from shuffling individual genes', to swapping them entirely with a new value, etc. Too many mutations could add too much entropy to the algorithm, and it wouldn't get a proper result.
Start from step 2, and redo the process again, starting a new generation, until it reaches an optimal solution, triggered by the fitness function rating being high enough to reach the termination criteria.

I hope that this explanation has been enough to understand how a Genetic Algorithm works before we enter into the technical bits.

The Book's framework

When starting to read the book, after a more in-depth explanation of what I mentioned here (a lot of things in this post are from there, also), the author guides the reader through the development of a framework that we can use for almost any Genetic Algorithm problem with minimal adjustments to its base logic.

Recalling the simple flow specified above, we can see that some things can be abstracted through some code, by using a defined API for what chromosomes and genes should be. To fully grasp how we got to the next pieces of code, you'll need to read the book, but the most important parts can be summarized into the following:

The definition of the Chromosome type.
- By using :genes as an Enum we can take advantage of all the existing functions. We can use complex structs to represent a single gene, but the set of them follows the same API.
- Defining other helpful fields like fitness, size and age that will help in the evaluation of the population.

defmodule Types.Chromosome do
  @type t :: %__MODULE__{
          genes: Enum.t(),
          size: integer(),
          fitness: number(),
          age: integer()
        }

  @enforce_keys :genes
  defstruct [:genes, size: 0, fitness: 0, age: 0]
end

The evolve function. This represents the flow above, where we get a population (list of Chromosomes), and iterate recursively over selection -> crossover -> mutation -> restart until the termination criteria are reached. I'll skip that part here, but this and the other functions remain almost the same across problems, but the book also shows different approaches.

def evolve(population, problem, generation, opts \\ []) do
    population = evaluate(population, &problem.fitness_function/1)
    best = hd(population)
    best_fitness = best.fitness

    IO.write("\rCurrent Best: #{best_fitness}")

    if problem.terminate?(population, generation) do
      best
    else
      generation = generation + 1

      population
      |> select()
      |> crossover()
      |> mutation()
      |> evolve(problem, generation, opts)
    end
  end
end

This way, the few things we need to worry about are providing the framework with an initial population and defining the terminate?(population, generation) function, which evaluates the termination criteria.

Using expected returns as Fitness Function

Recalling the problem stated above, for this project we wanted to know which Assets would make it into the portfolio using Genetic Algorithms. A good first approach for this, and one that is very simple, is just optimizing for the expected returns. Why? We can avoid the complex math related to Sharpe Ratio, and just worry about a singular variable: the asset expected return. We had already selected the assets we were going to use. Some of the expected returns, at the time of writing, were:

ETH Staking	Uniswap	BTC	Maker	Filecoin	Aave
4%	-0.0054%	-0.0017%	-0.0038%	-0.011%	-0.0009%

Note that this return would be in ETH tokens, and not Euros or USD. Looking at the table, the crypto winter is clear, and the only positive return is ETH Staking (you can check more here on how it works). It's not necessary to be a great Math genius to understand that the best solution to optimize our risk-reward portfolio, a huge chunk (100% basically) of the allocation should be placed under ETH Staking, looking just at expected returns. It's the only positive one 😅. We can use this knowledge to scope out the values presented as results by our Genetic Algorithm. If the optimal solution, when the termination criteria are reached, has higher allocations on negative expected return Assets, then something is not right. Pretty easy, no?

Let's start by modelling our problem in a structure that corresponds to the framework's API.

Gene: is a value ranging from 0 to 100, and represents the percentage of portfolio allocation to an asset. Initialized randomly.
Chromossome: has a list of 10 Genes. One for each different asset we want to analyse. We made sure that the initialized sum of the 10 assets was under 100 (we can't allocate more than 100% of our portfolio).
Population: 100 different Chromossomes. After each evolution round, we classify each Chromossome with the Fitness Function.
Fitness Function: For this first approach, the fitness of a given Chromossome is just multiplying the algorithm-predicted allocation with the Expected Return for that respective asset, and summing everything at the end. That would give us the global expected return for our portfolio. Example:

@expected_returns [0.04, -0.000054, -0.000017, -0.000038, -0.00011]
genes = [80, 5, 2, 3, 10]

fitness = 
 expected_returns
 |> Enum.zip(genes)
 |> Enum.map(fn {p, g} -> p * (g / 100) end)
 |> Enum.sum()

Termination Criteria: When the population age reaches 10000. There were other possibilities for termination criteria, but you should definitely read the book to understand more about this.

Well, after running everything a couple of times the results were just random (yeah, I ran a couple of times just to make sure the results wouldn't be different). What was wrong with it, then?

The culprit was the mutation function. I didn't talk about that in the previous paragraphs because I was relying on the function from the framework to run our algorithm. Big mistake. That function simply shuffled Genes around inside the same Chromossome, meaning that every time a mutation happened to a Chromossome in the evolution process, values from the initialization would just be shuffled, and not adding any entropy to the values themselves. That's why we always saw random values allocated for each Asset, it was just the random initialization.
- The solution for this required updating the mutation function. Instead of shuffling around the genes, the new mutation function should increase or decrease the current value randomly, so that values change throughout evolution. The code below solved that issue.
```
new_genes = 
  Enum.map(chromosome.genes, fn gene -> 
    range = floor(gene * 0.5)

    updated_value = gene + Enum.random(-range..range)

    if updated_value <= 0, do: 0, else: updated_value
  end)
```
However, problems didn't stop here. Now, we were having the best solution presented with 0 at almost every gene, including the ETH Staking one, which is the go-to asset. This one was more difficult to find out the root cause because there were two different reasons.
- Penalizing too much the very good Chromossomes. What's this about? Since that in the new mutation function, we're increasing the value randomly, sometimes the sum of all genes would be higher than 100 (something that we only account for in the initialization). Our fitness function had a small clause that if the sum of all genes was higher than 100, it would return 0 for that Chromossome's fitness, because it would be an invalid individual. With this, we were penalizing too much the Chromossomes reaching close to 100% allocation, effectively killing all of them until only smaller values remain in the population. The solution was to remove that clause from the fitness function and normalise the values if the overall sum went over 100.
- Penalizing too much the very bad Chromossomes. Looking at the mutation function there is the following line if updated_value <= 0, do: 0, else: updated_value. The problem with this was that if a gene reached 0 or below (because of randomly decreasing the old value), the gene would never "recover" from that 0 value, effectively locking most of the assets' allocations to 0. Updating the line to something like: if updated_value <= 0, do: Enum.random(0..2), else: updated_value fixed this problem, because it gives a small chance for that gene to have a different allocation value, and not just 0.

After these changes, we're getting pretty acceptable results where the Genetic Algorithm's best solution, presents ETH Staking with >90% allocations, and the other assets take home the rest 10%.

Using Sharpe Ratio as the Fitness Function

I've already explained what Sharpe Ratio is and why it's the final approach for this problem. TLDR, it also takes into account the inherent variation of Assets (associated risk) and not just their expected returns.

Recalling the Sharpe Ratio function from one of the images above, we need to know the portfolio expected returns, risk-free rate and the portfolio's standard deviation. We already know how to calculate the first two, we just need to calculate the standard deviation.

According to WallStreetMojo, the following formula is how you do it.

Where:

wX: allocated "weight" to asset X.
R(kX, kY): the correlation between asset X and Y.
σX: Standard Deviation for asset X.

We pre-calculated all of the correlations between Assets, and the Standard Deviation for each one. Implemented the function to calculate the full portfolio's Standard Deviation, and replaced the fitness function above with the complete formula for Sharpe Ratio.

The results were just bad, and nonsense all over again... The problem this time was that the time until the end of the Hackathon was running out, and there were other things in the project that needed to be done, so after a couple of hours of debugging it, we decided to move over.

It's a bit of a bummer finishing things this way, but at this point, the issue was probably in the pre-calculated values and not in the Genetic Algorithm itself, as we just changed the fitness calculation. The main thing for this blog post is the algorithms, and not Hackathon rushed statistics...

Wrapping up

Looking back at how things went, I should have finished reading the book before the Hackathon 😅, to reduce some of the inherent stress that comes with participating in one, instead of going insane debugging why my population doesn't evolve properly.

Despite not reaching a perfect or even a "working" solution, I learned a lot about tweaking these algorithms (after all, evolution is not that simple). We also ended up winning a prize of 1500$ awarded by Protocol Labs for the third-best project of the hackathon using one of their products (Bacalhau - cod in Portuguese, which comes from the extended name of Computing over Data). We used Bacalhau to run our genetic algorithms simulation in a decentralized network of servers, that automatically stores its results in IPFS.

Have a look at the code written during the Hackathon in our Github repo (not just the Genetic Algorithms code), and feel free to reach out if you have any questions. A small disclaimer, remember that it is all Hackathon code, and it gets a bit messy...

Thanks for reading, and see you in the next one 👋

NFT Attribute Types: A rarity system (but with Maths)

José Diogo Viana — Fri, 18 Mar 2022 10:12:04 +0000

At this point in time, almost everyone has heard about NFTs. Even if you don't really understand how they work, you associate them with images of Apes or Cats. Check a couple of them, just for reference:

Technically, and also theoretically, NFTs are not just that, but for this blog post, we'll just stick with profile pictures that have different attributes. If you want to know a bit more in-depth about NFTs in general and other possible applications, go check Francisco's post, also for the initiates in the area.

What are these attributes, you ask? Well, they are associated with the metadata of the Asset, and in the case of the profile pics, these attributes will define what your final image looks like and create a perceived sense of scarcity (or rarity).

As an example, we will use the Bored Ape owned by our good friend Roberto from Subvisual.

Checking on the metadata, provided by its OpenSea page, this token has the following attributes, and respective rarities:

Background: blue (12%);
Clothes: tweed suit (1%);
Eyes: coins (5%);
Fur: gray (5%);
Hat: Faux Hawk (1%);
Mouth: Phoneme Oh (2%);

How were these percentages calculated? We'll try to better understand them below, and try to come up with our own way of pre-calculating a rarity for an attribute.

If this got your attention, and you want to know more about how overall rarities are calculated (how someone judges an NFT as rarer than another), go check this post from rarity.tools.

What are we doing and why?

We want to pre-generate the metadata for a set of NFTs that will have several attributes. This metadata has information on the traits of that specific NFT, in this case, each attribute type and the respective value.

For example, if we want to generate the metadata for an NFT that has different types of glasses, the plan is to calculate the total number of tokens that will have round, square, emoji glasses, or even no glasses at all. This will specify the rarity of each attribute type.

The question of how other Buidlers are doing this came up in one of the projects we are currently working at, as Finiam. We assumed right away that they should obey some kind of mathematical rule, defined by the creators, distributing attribute values, for the final image to have a "real" sense of rarity and scarcity... But, there's also the chance that each attribute value is just licking the finger and sticking it up in the wind.

This way, Resende and I started looking for references and if someone had documented their journey of calculating rarities and how they had done it. We found no evidence that they did have used mathematical distributions, or at least they weren't open about it at all.

Bored Ape Yacht Club - rarities breakdown

Resende had developed a Jupyter notebook where you can just introduce the URI of the metadata for a certain contract, and then the script creates a dictionary (map) with the overall information of each attribute type.

We decided to inspect the BAYC contract. Looking at their OpenSea collection page, you can see that they have 7 different attributes, and the number of types for each one varies from 6 to 43. For the sake of this blog post, we'll just show you two of the generated plots. If interested to get more information give us a nudge on Twitter.

The following plots represent a histogram for each attribute type of Clothes and Eyes, respectively.

We won't discuss if they used some mathematical distribution to calculate each amount or if it was eyeballing, but you can try to estimate if this really uses any math as the base.

Getting our hands dirty

After this, for us, it was very simple: "Let's do it ourselves!" The perfect excuse to finally put in practice those Calculus and Statistics lessons in college that we never got to use, until now!

This way, the goal to develop something interactive to calculate the number of NFTs that will have each type of attribute, was set.

Note: Here it's also possible to check a statistical analysis on a different type of projects (eg. generative art from Artblocks.io. So if you're into statistics give it a look also.)

Looking at mathematical distributions and functions

The first step was looking at the plots of some distributions that we already knew and seeing which ones could be adapted into defining a cool rarity system.

We looked into a bunch of them, and to be honest, we don't remember anymore which ones were. Some weren't even a distribution, like the sin function, that we also investigated, because of its malleability to have the 0's where we want them and also for the waving shape. So, as a summary, we'll only talk about the two that got selected.

Negative Exponential

After our initial brainstorm session, we knew we wanted to use the negative exponential function. Just looking at the plot, it screams rarity system. How? Let's have a look at the example below.

Imagine dividing the area of the negative exponential function (integral in the Wikipedia if you're not familiar with it) by the number of total attributes we want to calculate a rarity for, just like it is in the image. In that case, we have 25 columns, that would represent 25 different attributes. We get each rarity by calculating the approximate area of each column and dividing it by the total area, thus getting a percentage. This percentage then represents the mathematical distribution and the respective percentage of occurrences for that attribute.

I know I just described how distributions are calculated with Integrals, but the point here is that we get a clear decline of the area of each column, giving us the proportion for each attribute from the most common to the rarest.

Poisson Distribution

Then, the other option that we also went with is the Poisson distribution, because it has a cool name. Well, not only that but because it offers a very different perspective of the negative exponential and enables more possible combinations of attributes rarity.

Looking at the image above, you can see what's that difference I was mentioning. Just looking at the bars you might confuse the Poisson distribution with a Normal distribution, but they are slightly different, and that's the real reason we chose it.

The Poisson Distribution has a lower bound at 0 and no upper bound, while the Normal Distribution has no bounds at all. Also, unlike a normal distribution, which is always symmetric, the basic shape of a Poisson distribution changes. To know a bit more about these differences check this.

The same mathematical approach with the negative exponential to calculate areas and percentages, also applies here, as with other distributions.

Taking advantage of Google Sheets

We didn't want to implement these distributions ourselves, despite understanding the maths behind it. So we turn our heads to the good friend Google Sheets that already has a bunch of distributions implemented, and can be used out of the box.

Another good thing about Google Sheets, or any other famous spreadsheet, is that it's also easily programmable and interactive. We can define cells where we change the inputs, and others displaying results, change accordingly.

And finally, plotting results and conclusions is also effortless, and pretty much necessary for this use case. It's kinda cool to visualize what different attribute rarities would look like.

Making it dynamic with configurable values

As I hinted earlier, we wanted to do it configurable so we could test different combinations and options with ease.

Two things that we need to be interactive are the number of tokens to be emitted, or in other words, the amount of NFTs that will exist in this collection and the number of different possibilities an attribute will have, so we can calculate the distribution only for those values.

Also, looking at the Poisson Distribution function, we see that it's also possible to make λ vary. λ must be greater than 0, and it is equal to the expected value of X and also to its variance (variance in Wikipedia). This way, we can have even more combinations and possibilities.

Calculating rarities with Poisson

To calculate rarities with the Poisson distribution we used the function POISSON.DIST(x, mean, [cumulative]) that Google Sheets already provides. Sheets provides more information on it, but basically, in our situation:

x will be the id of the attribute ranging from 0 to total attributes.
f will be the mean of the Poisson distribution function (λ).
[cumulative] is a boolean to identify if we want to use the cumulative distribution function rather than the distribution function (the sum of all columns until the X we are using instead of just the column X, using the same analogy as above).

However, just doing this we have a slight problem... The Poisson Distributed is not bounded to the right of the X-axis, so if we just want to calculate the distribution for a small number of attributes, there's the chance that the probabilities don't add up to 100%, because we're missing some x-values.

Looking at the generated plot above using only the distribution value for 15 different attributes of 10000 tokens (NFTs), and with a mean value of 10, we see that some values are missing to the right, and if we add all of the probabilities we get just 91.65%. The rest 8.35% are for values of X above 15, which we don't want. The objective is if we have only 15 attributes, the sum of their probabilities must be 100%.

So the solution we came up with is calculating the missing percentage to 100% and dividing it by the number of attributes, and adding this part to each attribute. Using the example above, we would divide the 8.35% that is missing by 15 (equating to approximately 0.56%), and add this value to each existing column, therefore spreading this exceed evenly.

The final formula is this, using the cumulative version to actually obtain the exceeding probabilities to the right of the function.

POISSON.DIST(x, mean, false) + ((1 - POISSON.DIST(totalAttributes - 1, mean, true)) / totalAttributes)

Below, we have the updated version, and if we compare the two plots, we see that now the first attributes have a probability different than 0, because we are also taking into account the exceeding values, and the sum of all of them is equal to 100% or the total number of tokens to be created.

Calculating rarities with Negative Exponential

For the negative exponential distribution, we used the function EXPONDIST(x, lambda, [cumulative]).

x will be the id of the attribute raging from 0 to total attributes.
lambda is the parameter of the distribution, often called the rate parameter.
[cumulative] represents the same as in the Poisson one.

In this case, we are leaving the lambda fixed as 1, for simplicity. This distribution was a bit more complicated to get around so we went with the shortest path to having anything providing correct results.

The first problem arose from the idea of dividing the function into columns. This process was not that direct, as in the Poisson case. The formula we arrived at to have a proper result, is the following:

EXPONDIST((4.60517 / totalAttributes) * X, 1, false)  - EXPONDIST((4.60517 / totalAttributes) * (X + 1), 1, false)

Why the subtraction, and why that 4.60517?? First, let's have look at the "strange" value and we got it.

The value (~4.6) comes from solving the following expression of - (ln (1 - 0.99)). This is obtained from the integral of the negative exponential function e^-x and trying to obtain the X value where the area of the function is approximately 0.99. If you want to check the step-by-step solution, have a look at this Wolfram solution for it. Following the same approach, we can divide it into smaller columns and get the relative proportion. But why 0.99? As this function tends to 1 at the infinity, we chose to use 99% as the full area, otherwise, percentages for the rarest attributes would be basically 0.

Returning to the expression above (4.60517 / totalAttributes) * X we divide the area by the number of attributes and multiply it by the value of X we are calculating at the time, simulating the column divisions referenced above.

The same issue with boundaries that happened in the Poisson Distribution also happens here, so we used the same solution. Calculate the exceeding values, and add them evenly to each attribute percentage.

This way both distributions may not produce the values as they would in normal mathematical conditions, as they were adapted to fit this very specific problem with a discrete number of variables. Note: we are not mathematicians so we are sure that there is some obscure way of doing the same thing but mathematically correct.

How do we use this for tangible information?

The final question is how we retrieve the actual number of tokens that will have each type of attribute. For that, we decide to do a quick demo.

In the video above, we set the number of tokens to 1000 and 5 different attributes (where we set some fake attribute names just to help identify stuff in the long run), and then the mean value of the Poisson Distribution to 3 (basically setting the most common id). It's also possible to see how the cells and plots change when we change the different configurable values.

Red columns represent the actual number of tokens that we want for a specific attribute, based on the predictions given by the distribution we choose, as they produce decimal values that can't be applied to our use case (a token must be a full unit).

Final thoughts

Looking back at this experience, there's a slight chance that the solution was a bit of over-engineering, but it was the perfect opportunity to put this knowledge to the test.

We hope you have liked this approach to rarities in NFTs, and feel free to reach out if you want to know more about this spreadsheet.

Have a nice one and see you in the next blogpost 👋

Developing a keyboard heatmap and customizing keys with Karabiner

José Diogo Viana — Wed, 02 Jun 2021 12:58:10 +0000

For the last 5 years, I've been using an Apple Magic Keyboard. There wasn't any major reason to be using it, except having the same layout of my laptop, but it did the job pretty nicely.

After joining Finiam, I started looking more into how I used my keyboard, and how I could become more efficient with it. A thing that was bothering me for some time was the fact that my keyboard had a Portuguese layout, and every time I wanted to make {, ] characters, Finger Gymnastics 101, would be required. I know I could always change the input language, but it wasn't the same thing. This was just the excuse I needed to get a new keyboard and with a US layout. So I went by and bought a Keychron k2v2 and set myself up to start looking for more ways of having it optimized for me.

The first thing that I did was installing Karabiner-Elements. It's a keyboard customizer for macOS, where you can easily swap a key with another, use a base set of complex modifications developed by the community or create your own rules. I started investigating and trying new combinations... But then it came to my mind:

"Wouldn't it be nice if I could make a heatmap of my keystrokes and then create aliases or easier configurations for things that I identify as the most used?"

Well, of course, it would, and I can do it as a nice excuse to play with Elixir again. So let's get our hands dirty and get into some code.

Developing the Heatmapper

First, we need to collect our keystrokes. This would be a nice challenge to do, but I'll leave it on the side for now. So, I started looking into existing keyloggers in Github (Yeah, I know this sounds a bit weird because the chosen one could be malicious - I'm aware of that possibility). After a few possibilities, I found out caseyscarborough/keylogger, went through the code and it seemed simple to use and make changes to adjust it for my specific needs. Now let's dive into the heatmap and drawing part.

From the beginning I wanted the heatmapper to draw the keyboard and not use a base SVG layout (or something similar). Zamith's mogrify_draw enables me to draw keyboard keys in a simple way using mogrify as the base.

Another thing was that since the code will handle drawing a keyboard, it would be an even cooler solution to be able to draw any keyboard, given the provided structure of the keys.

The first step is to parse the log file of keystrokes provided. As I mentioned earlier, I made some changes to the keylogger and this version can be found here. You can then follow the instructions in the README on how to use it, but it outputs to the file the following on each keystroke read:

<keycode>::<keyname>

I'll probably remove keyname in the future, as I'm not using it in the application itself (just for debugging purposes), and therefore reduce the size of the log file.

Looking, at the code, we first open the file and then parse its content to generate a map of frequencies. In the parse_line/1 function, it uses a regex to get the keycode in each line.

{:ok, contents} <- File.read(path)
frequencies = parse(contents)

def parse(contents) do
  contents
  |> String.split("\n", trim: true)
  |> Enum.map(&parse_line(&1))
  |> Enum.reject(&is_nil(&1))
  |> Enum.frequencies()
  |> normalize_frequencies
end

The normalize_frequencies/1 function, calculates the maximum value of keystrokes (the max frequency), and then for each value in the frequencies map, returns a normalized value: value/max. See the example below:

# log file contents
31::o
45::n
8::c
31::o

# frequencies map after parse
frequencies = %{ 31: 2, 45: 1, 8: 1 }

# frequencies normalized
max_frequency = 2
normalized = %{ 31: 2/2, 45: 1/2, 8: 1/2 }

Now that we have normalized values, we can jump to the drawing process. The first thing to do here is to implement the logic for several keyboard layouts to be drawn, so together with the path to the log file, it's also necessary to pass which keyboard type to draw.

For organizational purposes, I decided to divide the layouts by keyboard brand first, and then by type. For example, below we have the types :keychron, :macbook and :niz.

def get_keyboard_layout(model, keyboard) do
  case model do
    :keychron -> 
      keyboard_layout(Layouts.Keychron, keyboard)
    :macbook -> 
      keyboard_layout(Layouts.Macbook, keyboard)
    :niz -> 
      keyboard_layout(Layouts.Niz, keyboard)
    _ -> {:error, :no_model}
  end
end

$> get_keyboard_layout(:keychron, :k2v2)

Calling the function with a valid model and type returns the keyboard layout. Each layout is under layouts/<model>.ex, and consists of a function with the name of the keyboard type and the following structure:

# file: layouts/keychron.ex
def k2v2 do
  [
    [
      %{size: 1, keycode: 53, name: "esc"},
      ...
    ],
    [
      %{size: 1, keycode: 50, name: "`~"},
      ...
    ],
    ...
  ]
end

It's basically a list of rows of the keyboard, where each row, the list of keyboard keys is defined. A keyboard key can be defined with keycode, name, and size (assuming that a size of 2, will draw a key with a width of 2 * DEFAULT_KEY_PX_SIZE and a fixed height of DEFAULT_KEY_PX_SIZE). It is also possible to define a height and width instead of size if we wish to draw keys with different heights.

Now let's see how everything is drawn.

The first step is using mogrify_draw to paint the base image where everything will be laid next. It'll create a png with the respective width and height of the keyboard we want to draw.

defp draw_base_image(keyboard) do
  %Mogrify.Image{path: "heatmap.png", ext: "png"}
  |> custom("size", "#{95 * keyboard_width}x#{95 * keyboard_height}")
  |> canvas("white")
end

To draw each keyboard key, we go through each row, and subsequently across all row's keys, keeping track of the xx_position and yy_position that we are at the moment. With this information, we know the exact place in pixels to start drawing the key in the base image. But before actually drawing, we get the respective frequency for that key and the RGB color calculated accordingly to the frequency passed. Then, we draw a rounded rectangle (the key) and a white text over it (the key name).

defp draw_key(key, xx_position, yy_position, image, frequencies) do
    freq = frequencies[to_string(key.keycode)] || 0
    {red, green, blue} = get_rgb_color(freq)

    image
    |> custom("fill", "rgb(#{red},#{green},#{blue})")
    |> rounded_rectangle(
      xx_position * @key_size + 10,
      yy_position * @key_size + 10,
      xx_position * @key_size + @key_size * key_width(key),
      yy_position * @key_size + @key_size * key_height(key),
      10,
      10
    )
    |> custom("fill", "white")
    |> Mogrify.Draw.text(
      xx_position * @key_size + 45,
      yy_position * @key_size + 45,
      key.name
    )
  end

Looking at the function rounded_rectangle it's derived from the ones enabled by mogrify_draw, but using the roundRectangle primitive from the -draw given by the mogrify command-line tools (check here all the primitives from mogrify if you need something more custom). For this primitive, it's necessary to pass the base image, the upper left and bottom right point, together with the border-radius information.

defp rounded_rectangle(
       image,
       upper_left_x,
       upper_left_y,
       lower_right_x,
       lower_right_y,
       border_w,
       border_h
     ) do
  image
  |> custom(
    "draw",
    "roundRectangle #{
      to_string(
        :io_lib.format("~g,~g ~g,~g ~g,~g", [
          upper_left_x / 1,
          upper_left_y / 1,
          lower_right_x / 1,
          lower_right_y / 1,
          border_w / 1,
          border_h / 1
        ])
      )
    }"
  )
end

After recursively going through all the rows and keys, the image heatmap.png is created.

Grey keys are keys that weren't used (or the keycode is not being detected by the keylogger.)
Stronger blue keys are the ones used with less frequency.
Mix of blue and red are the keys with mid-frequency.
Red keys the most used ones.

You can check the complete code and more detailed instructions on how to run it, in this Github repo.

Looking at the final result of a week of work we can identify some keys that are heavily used, such as the arrow keys, left command, left shift, left control, and even the backspace. It's somewhat a good representation of my usage as the referred keys take part in most of the keybindings that I use for work (except the backspace - not sure of what I delete so much 😅).

Creating custom key modifications with Karabiner-Elements

As I mentioned earlier, initially I made some simple modifications with Karabiner, but now we are in the condition of making some more complex ones, that are not provided by the community.

To create custom modification rules, we need to create a file under ~/.config/karabiner/assets/complex_modifications/ where you can have only one file with all the rules or group them in different files. After having valid rules they will appear as available to enable in the Complex modifications tab.

As an example, below, I'm creating a simple rule to change the caps_lock key to be a control key and just be a caps_lock when pressed together with shift.

karabiner/assets/complex_modifications/capsToControl.json

{
  "title": "Change caps key",
  "rules": [
    {
      "description": "Change caps_lock key to control. (Use shift+caps_lock as caps_lock)",
      "manipulators": [
        {
          "type": "basic",
          "from": {
            "key_code": "caps_lock",
            "modifiers": {
              "mandatory": [
                "shift"
              ],
              "optional": [
                "caps_lock"
              ]
            }
          },
          "to": [
            {
              "key_code": "caps_lock"
            }
          ]
        },
        {
          "type": "basic",
          "from": {
            "key_code": "caps_lock",
            "modifiers": {
              "optional": [
                "any"
              ]
            }
          },
          "to": [
            {
              "key_code": "right_control"
            }
          ]
        }
      ]
    },
  ]
}

To see more on how to write these rules, you can the Karabiner Configuration Reference Manual or use this online editor to generate them.

Wrapping up

There's still some stuff to do to make the heatmap more accurate and robust, such as making it work for more complex keyboard layouts, like split keyboards or with vertical gaps between keys. But I'll leave it for later when I have the time.

This was a fun and simple project to try some ideas and make something generic that could be used by other people. The experiments with Karabiner-Elements weren't all described here, but they were about trying more multiple key combinations, using only keys that I don't find necessary for my daily use, such as the home, end, page down and page up.

Hope you also liked reading it and that you give it a try.

Have a nice one and see you in the next post 👋

Quantum computing in the financial world

José Diogo Viana — Wed, 24 Feb 2021 15:16:09 +0000

Introduction to Quantum computing

When opening IBM's Quantum Computing page we see "Tomorrow’s computing, today". For me, this is a great way to show newcomers what quantum computing really means today. I'm no expert in quantum mechanics, I don't even know enough to the extent of explaining how it works to someone. But one thing I know is that we are in a similar phase as Engineers and Computer Scientists were in the '40s or '50s, developing the first classical computers. I don't believe that quantum computers can fully replace our classical ones, but they surely will have a major role in our lives in the future, and we are just at the beginning.

I won't explain how quantum computers work, but I recommend watching the video below, from Kurzgesagt – In a Nutshell. It's a 7-minute video and does a great job comparing quantum computers with classical ones, so if you are completely new to this matter, you should take the time to watch it, also.

You might have heard about NPC (NP-Complete) problems, if not, the Wikipedia page makes a good summary with some examples. Why am I talking about these problems? A NP problem is very hard to answer with our current technology and could take years if not centuries to reach an answer.

That's where the Monte Carlo algorithm comes in! It is a computation process that uses random numbers to produce an outcome. So, if there is a procedure for verifying whether the answer given by a Monte Carlo algorithm is correct, and the probability of a correct answer is bounded above zero, then with probability one running the algorithm repeatedly while testing the answers will eventually give a correct answer.

At the moment of writing this blog post, we only achieved quantum supremacy two times. Quantum supremacy is when a quantum computer outperforms a classical one. The goal of this blog post is not to enter into details, but you can read more here about the last time this has been achieved. And even more recently, Quantum Algorithms cracked nonlinear equations, by first "disguising" them as linear ones (more here).

In finance, problems like portfolio evaluation, personal finance planning, risk evaluation, and derivatives pricing are good examples of applications to Monte Carlo methods.

Nevertheless, Monte Carlo methods have a problem. If we want to obtain the most probable outcome of a wide distribution or obtain a result with a very small associated error, the required number of simulations can be stratospheric.

Quantum Amplitude Estimation (QAE) algorithm is a quantum-accelerated Monte Carlo algorithm. QAE can sample a probabilistic distribution quadratically faster than the next best classical method, allowing it to estimate expectation values extremely efficiently.

To know more about these particular methods and how to use them in financial problems, check this post from the Quantum World Association.

Financial Businesses and Quantum

According to IBM, here, "25% of small- and medium-sized financial institutions lose customers due to offerings that don’t prioritize the customer experience." and "It is estimated that financial institutions are losing between USD 10 billion and 40 billion in revenue a year due to fraud and poor data management practices."

Quantum computers can run whole-market simulations better in finding patterns, performing classifications, and making predictions that are not possible today with classical computers.

JPMorgan Chase wants to be ready for the quantum leap. The company has an unofficial leader, with the task to build a “quantum culture”, trying to discover how quantum computing can change the financial industry.

Barclays first started to explore quantum computing in 2017 and concluded that quantum computing’s potential was so great that they should commit to an initial program of research and development.

And we have many other cases to add to these ones.

Programming Quantum Computers

There are various approaches at the moment, to program a quantum computer. Microsoft has developed Q# programming language, IBM has Qiskit, a Python package, etc.

These are still in an early stage compared with what we can achieve with classical programming languages. Nevertheless, the road ahead seems promising.

Coding the first quantum circuit

IBM offers a Circuit Composer. It's a graphical quantum programming tool that lets you drag and drop operations to build quantum circuits and run them on real quantum hardware. You can check some instructions on how to use it here.

This Circuit Composer should look like the following screenshot.

In this blog post, we will follow the IBM tutorial and develop a Bell State as a "Hello World" circuit.

A Bell State is defined as a maximally entangled quantum state of two qubits. This means that if Alice and Bob held an entangled qubit, and Alice measured her qubit the outcome would be perfectly random, with either possibility having a probability of 1/2. But if Bob then measured his qubit, the outcome would be the same as the one Alice got. So, to Bob, at first sight, he would also get a random outcome, but if Alice and Bob communicated they would find out that, although the outcomes seemed random, they were correlated.

Step 1 - Add an H gate (Hadamard gate) to your circuit `q[0]`.

The Hadamard gate acts on a single qubit. It maps the basis state of the qubit to have equal probabilities to become 1 or 0 (i.e. creates a superposition).

To add a gate to your circuit, drag and drop the operation from the palette of quantum operations to the top qubit, q[0].

Step 2 - Add a controlled-NOT gate to your circuit.

Controlled gates, such as controlled-NOT gate (or CNOT or CX) act on 2 or more qubits, where one or more qubits act as a control for some operation. For example, the CX acts on 2 qubits and performs the NOT operation on the second qubit only when the first qubit is |1⟩, otherwise leaves it unchanged.

To add a CX gate to your circuit, drag and drop the operation from the palette of quantum operations to the right of the gate. This operation acts on two qubits.

Step 3 - Add a measurement operation.

To add a measurement to your circuit, drag and drop the measurement operation from the palette of quantum operations to the right of the CX operation.

The measurement result is recorded as a classical bit on the classical register. The vertical wire connecting the measurement operation to the bottom wire depicts information flowing from the qubit down to the classical register.

Run the job and analyze results

You have now built your first quantum circuit. The visualization panels below your circuit give a simulated result that updates automatically as you add and remove operations. But you can also run the job in one real quantum computer, by clicking the top right corner button. It is possible to configure some settings, like system, provider, or shots (the number of times to run the code). Running jobs can take some time, depending on the number of jobs in the queue, so be prepared for that.

Therefore, most of the time is sufficient to just observe the simulation results. By looking at Measurement Probabilities, we can conclude that the final state of our little circuit has a 50% chance to be |000> or |001>.

However, looking at the real result, the probabilities are not 50%. This is caused by noise and all sorts of technology limitations at the moment.

It is also possible to check the code version of the circuit we created.

Wrapping up

Quantum computing is a very interesting and promising topic, and we at Finiam are going to be attentive to new signs of progress that can impact the world.

Simulations with Elixir and the Actor Model

José Diogo Viana — Thu, 21 Jan 2021 14:29:52 +0000

Is there anyone else about to read this, as fascinated as I am by simulations? A simulation of a game, weather, aerodynamic behavior, or any other action, has intrinsically the same goal - take a peek into the future and see how things will perform in real life.

These simulations all require an extent of calculations and variables to take into account, but also a model that will elegantly run through them.

"Aren't you describing Machine Learning? Isn't it also predicting the future with complicated maths and a model that no one really understands?", you ask. Yeah, I guess that ML fits into this description, but that's not the kind of predictions that I will be writing here.

As an Engineer, I like to think that I build things. Some digital, some physical. Some work, some don't. So, for this problem, I'll use a carpentry workshop as my example (because, you know, carpenters build things that mostly work).

Let's say that in our imaginary carpentry we have the following working stations:

Cut wood
Polish wood
Paint wood
Varnish and protective coating
Assembly

In this workshop, not all orders have the same processing. A chair may not need a painting or protective coating, but a kitchen table might. Or it can be a supplier for that Swedish company that doesn't assemble their products cough cough. Also, a given workstation can receive orders from various different stations. Imagine a directed graph like the one below:

If we were about to score a big deal with a major client, but we don't know if we can handle the request in time, given that we have already so many orders in the shop, how can we decide on that? Simulation Time!

Assuming that we know how much time each process takes, and the types of processing each order needs, we can verify when all of our registered orders, together with the new one, will be completed. I know these calculations could be done by hand, (yeah yeah) but bear with me on this one.

Note that I'm not a Production Engineer, don't take my advice on production management. It's just for the sake of the blog post and the use for simulations in our lives.

Looking at workstations as if they were picking an order from the pile/queue, "doing their thing" and sending it to the next step, we can think of them as completely individual and independent actors from each other, inside the simulation engine. They simply pick an order from the queue and put it in the next one after the work is done. If nothing is waiting in line, the workstation stops until a new one appears. It can also be the starting or finishing point of an order.

Quick intro to Elixir and Actors

Continuing with the same thought, we identify some Actor Model programming characteristics. In response to a message it receives, an actor can:

make local decisions;
create more actors;
send more messages;
determine how to respond to the received message.

Actors may modify their private state, but can only affect each other indirectly through messaging. An actor is also a lightweight entity, unlike Operating System processes or threads.

One of the selling points of Elixir is its support for concurrency. The concurrency model relies on Actors. So, implementing the initial idea, where each workstation is an actor part of the simulation, should not be a problem.

Different types of simulations

There are two different types of simulation, Discrete Event Simulation (DES) and Continuous Simulation (CS).

DES: It models the operation of a system as a (discrete) sequence of events in time.
CES: the system state is changed continuously over time based on a set of differential equations defining the rates of change of state variables.

In our carpentry workshop, events can be treated as discrete. So in our simulation, time "hops" because events are instantaneous (jump straight to finish processing time) and do not occur in a continuous form. This way, the clock skips to the next event moment, each time an event is completed, as the simulation proceeds;

Step by step

Guess the boring talk is over and it's now time to put the ideas into work.

Here, I'll use GenServers, as workstations. With the Erlang and GenServer characteristics, a process has a FIFO "mailbox", ensuring the ordering of the messages received by each workstation. If process A sends messages x and y, by this order, to process B, we know that it'll receive them by the same order.

# Process A
send(B, x)
send(B, y)

# Process B 
receive() # => x
receive() # => y

Though, if we have a process C that also sends a message w to B in the meantime, we don't have guarantees in what order it'll be received. Only that x will be first then y.

# Process A
send(B, x)
send(B, y)

# Process C
send(B, w)

# Process B 
receive() # => w || x
receive() # => w || x || y
receive() # => w || y

Everyone that has already dealt with concurrency problems like this one, with several actors swapping messages, can smell a problem here... We need to have some kind of synchronization between processes. Order of messages from different actors/workstations matters in our simulation. That's why I'm introducing the Simulation Manager. We must ensure that the Manager only fast forwards the clock when it has all the updated values from the actors affected by some event in the past, keeping "timed events" from overlapping.

This process will ensure synchronization and message order. The existence of a centralized actor simplifies this problem and potentially increases efficiency due to less metadata and extra messages in a peer-to-peer case.

A bit of code

1. To start our simulation, we start a new process (actor). This will be the Simulation Manager and is responsible for spawning an actor for each workstation that composes our workshop's pipeline;

# Simulation Manager code
def start_link do
  actors_pids =
    CarpentryWorkshop.Workstations.get_all()
    |> Map.new(fn place -> start_place_actor(place) end)

  GenServer.start_link(
    __MODULE__,
    %{
      pids: actors_pids,
      clock: Timex.now(),
      events: [],
      awaiting: []
    },
    name: SimManager
  )
end

defp start_place_actor(place) do
  {:ok, pid} = Pipe.SimPlace.start_link(place)
  {place.id, pid}
end

In the code above, we have the Simulation Manager starting all the other actors and storing their pids to identify each one in the rest of the distributed algorithm. Note that there were other alternatives to this in Elixir, but this one is probably the easiest.

Events represent messages received from each place and awaiting are actors that the Manager should wait for before advancing the simulation time.

2. Each of these actors will collect the list of orders in their queue, and sort them by the delivery date. Announcing to the manager when the next event will occur. The next event is calculated by the processing time of the first package in the queue;

# Simulation Place code
def start_link(place_id) do
  events = get_events_list
  GenServer.start_link(
    __MODULE__,
    %{
      self_info: place_id,
      pids: %{},
      events: events,
    }
  )

  next = hd(events)
    events
    |> hd
    |> finish_processing_timestamp
  send(SimManager, {:next_event, next, place_id} 
end

def get_events_list do
  place_id
  |> CarpentryWorkshop.
      Orders.list_for_place(place_id)
   |> CarpentryWorkhop.
       Orders.sort_by_delivery_time(events)
end

def finish_processing_timestamp do
    # Based on the type of order
    # Calculate end processing time
end

3. When the manager gets all announcements, orders them, and advances clock time to the next event, warning the actor where the event will occur;

def handle_info({:next_event, event, sender}, state) do
  ordered = insert_ordered(state.events, event, sender)
  awaiting = state.awaiting -- [sender]
  {rest_events, next_step} = 
    send_next_event(ordered, awaiting)
  ...
end

# we have all answers
def send_next_event(events, []) do
  next_event = get_first_from_queue(events)
  send(next_event.place_id, {:advance, event})
end

# missing answers from actors
def send_next_event(events, awaiting) do
  {events, nil}
end

4. Then, the respective workstation Actor receives a message from the manager and completes the "event" (package processed). If the package is concluded, it will update the database value for the expected delivery time. Else, it will send a message with the package to the next workstation (actor). It also warns the manager, that it needs to wait for the next event update from itself and the next workstation;

def handle_info({:advance, event}, state) do
  # check if this is the last place for this order
  case tl(event.rest_path) do
    [] ->
      # Update DB Value
      update_prediction(event)

    path ->
      warn_next_workstation(state, event)
      warn_manager(state, event)
  end
  ...
end

5. The intervening workstation actor will recalculate the next event time and announce it to the simulation manager (nil in case that there is no package in the queue), to continue the simulation; This part is analogous to Step 2.

6. The simulation stops when the manager doesn’t have more "next events" in the queue.

if rest_events == [] && next_step == nil do
  # Finished Simulation
  stop_workstation_actors(pids)
  {:stop, :normal, []}
end

In the code snippets above, the parts for calculating timestamps and the next events are omitted, but they account for a major part of this simulation and synchronization between actors.

After glueing every missing part of this simulation, our workshop is now ready to answer when an order (newly placed or not) will be ready.

Wrapping up

We demonstrated some of the advantages of using Elixir for distributed algorithms/problems. However, use this kind of distributed approach with caution, as it can introduce all sorts of bugs. And believe me, you won't like debugging distributed stuff...

Hope I got you a little bit more interested in simulations in general and taking advantage of the actor model to solve exquisite problems.

DEV Community: José Diogo Viana

Sign in with Metamask using LiveView

Why is it relevant?

How does it work?

Why Elixir and Liveview?

A look into the desired Flow

Getting our hands dirty

How to interact with Metamask from Javascript?

Wrapping up

How to build a Concurrent & Resilient Service in Elixir

Resilience & Fault Tolerance

Concurrency

The BEAM & Elixir

Taking advantage of Supervisors

Dynamic and Partition Supervisors

A simple Framework

Going the Extra mile

Using a :dets and other disk storage mechanisms

Process Pool

Different Handlers that can be plugged and played

Further Reading

Wrapping up

Genetic Algorithms to optimize an Asset Portfolio

The base idea

Sharpe Ratio

Genetic Algorithms

The Book's framework

Using expected returns as Fitness Function

Using Sharpe Ratio as the Fitness Function

Wrapping up

NFT Attribute Types: A rarity system (but with Maths)

What are we doing and why?

Bored Ape Yacht Club - rarities breakdown

Getting our hands dirty

Looking at mathematical distributions and functions

Negative Exponential

Poisson Distribution

Taking advantage of Google Sheets

Making it dynamic with configurable values

Calculating rarities with Poisson

Calculating rarities with Negative Exponential

How do we use this for tangible information?

Final thoughts

Developing a keyboard heatmap and customizing keys with Karabiner

Developing the Heatmapper

Creating custom key modifications with Karabiner-Elements

Wrapping up

Quantum computing in the financial world

Introduction to Quantum computing

Financial Businesses and Quantum

Programming Quantum Computers

Coding the first quantum circuit

Step 1 - Add an H gate (Hadamard gate) to your circuit q[0].

Step 2 - Add a controlled-NOT gate to your circuit.

Step 3 - Add a measurement operation.

Run the job and analyze results

More complex tutorials

Wrapping up

Simulations with Elixir and the Actor Model

Quick intro to Elixir and Actors

Different types of simulations

Step by step

A bit of code

Wrapping up

Using a `:dets` and other disk storage mechanisms

Step 1 - Add an H gate (Hadamard gate) to your circuit `q[0]`.