DEV Community: Milan Pevec

Elixir, Releases, and Docker

Milan Pevec — Sun, 27 Sep 2020 08:18:42 +0000

The goal of today's article is to show how to release Elixir application, written in my previous blog Elixir Plug and JWT and then run it inside of the docker container.

I would like to clear some things already in the beginning. There are a lot of options/solutions in this field and I'm not stating what's best or what's worse, nor am I an expert, but I would like to present what I've encountered in my research. And with that maybe help someone.

You can find all the code on my Github.

Why Docker with Elixir?

Ok, lets first create a production release. Where ie. on which environment could we do it?

We need to be aware, that the result of the Elixir release is a single unit that contains the app and the runtime, but is limited in self-containment - it can only run on systems sufficiently similar to the build system.

Hmm, so If I release the Elixir app on macOS it will (might) not run on Linux. Or even if it's released on a version of Linux with a different Glibc library than targeted Linux, the application might crash or not run either.

How to tackle this issue? The answer lies in the dockerization (containers). That sets our first goal, to release Elixir application inside of a docker container that is based on the same docker image as the target container where the app will be run in production.

I have chosen the Alpine Elixir to be the base image. It contains the full installation of Erlang/Elixir environment and its built on Alpine Linux.

At this point lets start writing our Dockerfile by adding the first line:

# Dockerfile
FROM bitwalker/alpine-elixir:latest AS release_stage

where we define that our image will be based on the alpine-elixir. Because we will use multi-stage build, we name this stage as release_stage. In this way, we keep one Dockerfile for releasing and for running the app.

Release me

Releasing an Elixir app is as simple as running mix release command, which will precompile and package all the code together with runtime (Erlang VM), which can be then used for running the app in the production or similar environments. There is a lot, a lot more behind the story of Elixir releases, I would just like to emphasize two points for its usage:

code preloading, when all Elixir modules are preloaded hence spikes on first requests are removed;
runtime configuration, where we can define different configuration every time we (re)start the app, without recompiling (releasing again) the app. We are planning to use that later!

Ok, lets first try to release our app outside of the docker container. We need to go to the root of the project and run the following line:

MIX_ENV=prod mix release

First, we set the MIX_ENV variable to prod, which puts production configuration into action (file config/prod.exs). Specifically, we want our production tokens to have a shorter expiration time than development tokens, so we do the following:

import Config

config :jwt_example,
  jwt_expiration_time_minutes: 15

Good. Now a release process will be called, all artifacts are built and saved in the folder _build/prod/rel/jwt_example, so we can easily run the application by calling:

_build/prod/rel/jwt_example/bin/jwt_example start

In order to do that also inside of the container, we need to prepare an appropriate docker image. What do we need for that?

Release stage

For running Elixir release we need all the source code, dependencies, and configuration. Lets first get project dependencies to the docker image and build them by adding the following lines to the Dockerfile:

COPY mix.exs .
COPY mix.lock .
RUN mix deps.get
RUN mix deps.compile

Each instruction in the Dockerfile adds a layer to the image and layers are cached, so in case of rebuilding the image and there are no changes in a specific layer, that layer will not be recalled, e.g.:

...
Step 5/10 : COPY mix.lock .
---> Using cache
---> 9d7f3925326c
Step 6/10 : RUN mix deps.get
---> Using cache
---> 79bae808d41c
Step 7/10 : RUN mix deps.compile
---> Using cache
---> 96c1e1d63927
Step 8/10 : COPY config ./config
---> efba111358c3
Step 9/10 : COPY lib ./lib
---> 04b6c23bdefb
Step 10/10 : COPY test ./test
---> b4f648c97967

You can see in the output (while building an image) which layers have been cached.

This gives us a reason to put things that have a lower probability to change as first, and things, like copying source code, as last to the Dockerfile.

Lets now copy the configuration and sources and then define the execution of the release:

COPY config ./config
COPY lib ./lib
COPY test ./test

ENV MIX_ENV=prod
RUN mix release

JWS secrets, ver1

What about the secret used for the h256 signature? As we wrote in the previous article Elixir Plug and JWT, we were using a solution where the secret is stored in the file, named hs256-signature-key. A secret file is read at the compile-time and secret is then stored in the application environment:

#config/config.exs:

config :jwt_example,
  jwt_secret_hs256_signature: Secret.get("hs256-signature-key", "default secret string")

If we want to keep the same approach, then we need to get the production secret to the docker image itself - also at the compile time, while running the release.

So imagine that you're a release manager and you have access rights to the production secret file stored in some secure location (e.g. /etc/elixir/production/hs256-signature-key) and you're preparing a docker image:

docker build --build-arg SECRET="$(cat /etc/elixir/hs256-signature-key)" —build-arg SECRET_FILE="hs256-signature-key" -t t3 .

The above docker build command will take two so-called build arguments:

SECRET_FILE: the name of the file inside of the container that will hold the secret;
SECRET: a secret itself. We need to send its content trough argument, because of the limitation of the docker build command - it's not allowed to copy files outside of the Dockerfile scope, so we can not copy directly /etc/elixir/hs256-signature-key. We must read the file on the fly and store its content in the build argument.

Accordingly to that, we will add the following lines to Dockerfile:

ARG SECRET
ARG SECRET_FILE

RUN mkdir $HOME/secrets
RUN echo "$SECRET" > $HOME/secrets/$SECRET_FILE
RUN chown -R default $HOME/secrets

The above lines will create a folder for secrets and store our secret coming from the build argument and then we set the permission that mix release could read the file while compiling.

Disclaimer:

It's not recommended to use build-time arguments to pass secrets. Even if we delete a secret file at the end, a secret can still be visible when running the docker history command. There are a couple of options to avoid this (check Access Private Repositories from Your Dockerfile Without Leaving Behind Your SSH Keys), but we will switch later to a different approach.

Almost there

Till now we have prepared the build stage of our multi-stage image. What's left is the run stage. It will be based on the same alpine-elixir image, although we could have taken a "thinner" image with only Erlang runtime. Let's add lines to Dockerfile:

FROM bitwalker/alpine-elixir:latest AS run_stage

COPY --from=release_stage $HOME/_build .
RUN chown -R default: ./prod
USER default
CMD ["./prod/rel/jwt_example/bin/jwt_example", "start"]

We copied the released application from the previous stage and run the release under the default user. That's it! Now that the Dockerfile is finished, we can build the image:

docker build --build-arg SECRET="$(cat /etc/elixir/hs256-signature-key)" --build-arg SECRET_FILE="hs256-signature-key" -t jwt_example .

and run the container, while exposing its port 4001, on which our cowboy server is listening:

docker run --name c_jwt_example -d --publish 4001:4001 jwt_example:latest

Now if we call

docker ps -a

we will see our container up and running.

Let's run our REST calls and at the same time check the container logs (stdout, stderr of the running app) with:

docker logs c_jwt_example —follow

we can see the following output:

##### 16:13:20.485 application=jwt_example domain=elixir file=lib/jwt_example/application.ex function=start/2 line=6 mfa=JwtExample.Application.start/2 module=JwtExample.Application pid=<0.2056.0> [info] Starting application..
##### 12:34:18.531 application=plug domain=elixir file=lib/plug/logger.ex function=call/2 line=27 mfa=Plug.Logger.call/2 module=Plug.Logger pid=<0.2163.0> [debug] POST /login
##### 12:34:18.535 application=plug domain=elixir file=lib/plug/logger.ex function=call/2 line=34 mfa=Plug.Logger.call/2 module=Plug.Logger pid=<0.2163.0> [debug] Sent 200 in 4ms

Great, as you can see our application is up and serving requests.

For the meaning of the docker parameters, you can check the official docker documentation.

Just before you go away

As we mentioned before, Elixir releases are enabling us to use the runtime configuration, that could we use for setting JWS secret at runtime and not at compile time!

Let's create a config/releases.exs file and add the following lines inside:

import Config

config :jwt_example,
  cowboy_port: String.to_integer(System.fetch_env!("COWBOY_PORT")),
  jwt_secret_hs256_signature: System.fetch_env!("HS256_SIGNATURE_KEY")

As you can see we set up the cowboy port at runtime also.

Please note one little detail, the System.fetch_env/1 function always returns a string, so in case of the integer values, we need to do the conversion.

That means we don't need to recompile the application (rebuild the release) to apply changes in this config file, we just need to restart our container and the new configuration will be taken into account!

For the above case, we will set port and secret as environment variables while running the container. At the same time, we can get rid of the build arguments about JWS secret, so our final Dockerfile looks like this:

FROM bitwalker/alpine-elixir:latest AS release_stage

COPY mix.exs .
COPY mix.lock .
RUN mix deps.get
RUN mix deps.compile

COPY config ./config
COPY lib ./lib
COPY test ./test

ENV MIX_ENV=prod
RUN mix release

FROM bitwalker/alpine-elixir:latest AS run_stage

COPY --from=release_stage $HOME/_build .
RUN chown -R default: ./prod
USER default
CMD ["./prod/rel/jwt_example/bin/jwt_example", "start"]

We then call the image build like this (no secret arguments needed):

docker build -t jwt_example .

and run the container with needed environment variabbles:

docker run --name c_jwt_example -d --publish is 8002:8002 -e COWBOY_PORT="8002" -e HS256_SIGNATURE_KEY="$(cat /etc/elixir/hs256-signature-key)" jwt_example:latest

Very straight forward isn't it! Happy coding!

Elixir Plug and JWT

Milan Pevec — Thu, 10 Sep 2020 10:18:07 +0000

Coming myself from the java world (e.g. Microprofile spec) and knowing JWT implementation there, I would like to see it in action with Elixir, hoping to get all that jazz, that Elixir is giving us. And importantly, without using the Phoenix framework.

You can find the Elixir source code of this project on Github.

Not going into too many details about JWT, what is it and how it works (you can get more information here: jwt.io), I would just like to write down some points that are gonna make our lives easier.

What is JWT

JWT or JSON Web Token is a token - string mainly used for authorization of a client. After the client successfully logs into the server it receives JWT in the response and stores it in browser storage. Every subsequent request will contain this token in its header, which will be then used on the server-side to authorize (or not) the request. The client does not even need to decode it.

Quest for the three strings

In our search for JWT, we are actually doing a quest for three strings which together, separated with a dot, make a JWT. These three string are named by their nature so we can define JWT as:

"Header"."Payload"."Signature"

Header and Payload are nothing more than JSONs which are Base64Url encoded. That's all! That means if someone manages to hijack the token, then they can easily decode the Header and Payload. So that means you should never store sensitive information in the JWT. For example, we could use online encoder and encode or decode JSONs ourselves if only we could hijack the token.

Show me JSONs

Let's have a look at how these JSONs look like. The header contains the information about the algorithm used for signature, e.g.:

{"alg":"HS256","typ":"JWT"}

and its Base64Url representation is

eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9

The payload contains so-called claims. Without going into details lets just say that claims contain information about the user, like his name, ID, the expiration time of the token and in the case of Java Microprofile we usually store also a list of groups (which are then used for authorization). The example of the payload is:

{
"sub": "John Doe",
"user": "johndoe@hey.com",
"iss": "Server ms",
"exp": 1516239022,
"groups": ["guest"]
}

and its Base64Url represantation is:

eyJzdWIiOiJKb2huIERvZSIsInVzZXIiOiJqb2huZG9lQGhleS5jb20iLCJpc3MiOiJTZXJ2ZXIgbXMiLCJleHAiOjE1MTYyMzkwMjIsImdyb3VwcyI6WyJndWVzdCJdfQ

Can I trust you?

The last missing piece of the token is the signature. What does it mean signed JWT? What does the signature present inside of the token?

Verifying signature is considered to be a part of the validation of the token (other parts of validations is checking the expiration time etc). If JWT is valid, that means the JWT payload can be trusted, in other words, no one has changed the payload ie. claims during the transport of the token and the server can be trusted.

An example of maliciously changed payload is when attacker would change the "groups" claim to contains group "admin" etc in order to force server to allow admin only allowed request.

What is important here is to distinguish between signature and encryption. As we already know, header and payload are just base64url encoded.

The signature does not provide secrecy!

JWTs by default are not encrypted. Encryption is covered by JSON Web Encryption (JWE) and we are not going to cover it. Signature is covered by JSON Web Signatures (JWS) and we will show its usage with Elixir.

There are several types of signing algorithms available according to the JWS. In our example, we will use HMAC using SHA-256 or shorter HS256. HMAC algorithms combine payload with a secret using a cryptographic hash function (e.g. SHA-256) or with simple words, we need a secret in order to sign a token.

That's a secret I'll never tell

Where to store secrets in Elixir? In my research, people store secrets in many ways but basic principles stay the same:

Secrets depend on the environment.
Secrets should not be stored in the source control.
Access to the secrets must be managed.

Me personally I'm using the approach of one file per secret, where the secret file is stored in the filesystem and is loaded by "config.exs". On the downside, that means secrets are loaded at build (compile) time, but on the upside, you can have good access management on the filesystem, while still avoiding storing it by mistake in the source control. This approach is also used by https://github.com/thechangelog/changelog.com

We could also use "releases.exs" per environment - in case you use releases in general (e.g.: MIX_ENV=prod mix release). In this way, the configuration is read when the system starts.

We could also use environment variables like the twelve-factor app is suggesting https://12factor.net/config, but then again the access management can be more tricky. Once again, choose whatever suits you best.

Show me the code already

Ok, its time for the code (Github). Let's emphasize important parts.

As you can see we are only using the following dependencies:

{:plug_cowboy, "~> 2.0"},
{:jason, "~> 1.2"},
{:jose, "~> 1.10.1"}

So for JWT operations, I've chosen JOSE library which for my taste offers the perfect taste of information compression - its nor too high-level nor too low-level code.

One plug pipeline per Application

One very important thing is, that by using "Plug.Router" you can have only one pipeline defined per application. There is a workaround explained here:

https://groups.google.com/g/elixir-lang-talk/c/U_o81N1VPfg

but I did not take the above solution.

On top of this limitation, there is another catch, every module that uses "Plug.Router" needs to have at least two plugs (match, dispatch) defined.

Combining those two important pieces of information, my code looks like the following. In my main application router I have defined my pipeline and forwards:

forward("/greetings", to: Greetings)
forward("/login", to: Login)

plug(Plug.Logger, log: :debug)
plug(:match)
plug(JwtExample.Plug.Auth, public_path: "/login")
plug(Plug.Parsers,
  parsers: [:json],
  pass: ["application/json"],
  json_decoder: Jason)
plug(:dispatch)

As you can see, we are using custom plug JwtExample.Plug.Auth for checking and verifying tokens with an option "public_path". This sets the exception for which this plug will skip the JWT check, ie. the login page should not have a JWT check off course. For other resources, plug will take the JWT token from the header and verify it.

Creating and verifying JWT is simple using JOSE library, we just need to provide:

JWK - JSON Web Keys which contains application secret for signing.
JWS - JSON Web Signature which contains defined algorithm.
JWT claims of our application.

Code for creating then looks like this:

# JSON Web Keys
jwk = %{
  "kty" => "oct",
  "k" => encode_secret()
}

# JSON Web Signature (JWS)*
jws = %{
  "alg" => "HS256"
}

# JSON Web Token (JWT)*
jwt = %{
  "iss" => Application.get_env(:jwt_example, :jwt_issuer),
  "sub" => id,
  "exp" => DateTime.utc_now() |> DateTime.add(expired_in * 60, :second) |> DateTime.to_unix(),
  "groups" => groups,
  "email" => email
}

{_, token} = JOSE.JWT.sign(jwk, jws, jwt) |> JOSE.JWS.compact()

The code for JWT verification looks like this:

jwk = %{
  "kty" => "oct",
  "k" => encode_secret()
}

case JOSE.JWT.verify(jwk, token) do
  {true, claims, *_*} -> {:ok, claims}
  _ -> {:error, "Token signature verification failed!"}
end

where the token is taken from the request header and as results we have claims.

More details on how plug is grabbing token etc. you can find in the sources (Github). Enjoy coding!

Understanding Elixir Plug

Milan Pevec — Tue, 21 Jul 2020 11:32:16 +0000

The goal of today's blog is to understand what is Elixir Plug and how to use it to write REST API in Elixir without the Phoenix framework. Let's do this.

Plug

Following official doc one of the definitions of the Plug is:

"Connection adapters for different web servers in Erlang VM"

Let's try to understand this sentence.
The Connection is a data structure (struct) encapsulating data from a web server that lays beneath. It containing data about the host, method, path, scheme, port, status, etc.

For the webserver in Erlang VM we can use cowboy.

Adapting a connection with adapters means changing its data structure and returning a new version of it (immutability).

Great. All clear now. It's all about adopting/manipulating connections. Let's show with the examples how can we achieve that in the Elixir way.

Plug, as simple as possible

Let's create a new project with supervisor support:

mix new app —sup

and add the dependency named plug_cowboy. This package contains the group of dependencies for a cowboy (webserver), plug and telemetry (for measurements):

def deps do
  [
    {:plug_cowboy, "~> 2.0"}
  ]
end

If we take the Plug example from official pages:

defmodule MyPlug do
  import Plug.Conn

  def init(options) do
    # initialize options
    options
  end

  def call(conn, _opts) do
    conn
    |> put_resp_content_type("text/plain")
    |> send_resp(200, "Hello world")
  end
end

we can see the usage of the so-called Module Plug (there is also a functional version).

If we want a module to act as a plug then we need to implement init/1 and call/2 functions. With init/1 we can set options, but more important with call/2 we perform the adaption of the connection. We use Plug.Conn functions like put_resp_content_type/1 for adaptation.

As final step lets change also the generated application module and we are ready to go:

defmodule App5.Application do
  use Application

  def start(_type, _args) do
    children = [
      {Plug.Cowboy, scheme: :http, plug: MyPlug, options: [port: 4001]}
    ]

    opts = [strategy: :one_for_one, name: App5.Supervisor]
    Supervisor.start_link(children, opts)
  end
end

You can see, that the initial Plug is defined as the MyPlug from above.

If we run this with classical iex -S mix (which compiles and runs our app) you can go on localhost:4001 and see the response "Hello world".
In official docs it's written that we can/should run this without iex process, with mix run —no-halt, but for sake of this blog we did it with iex.

What's also important in the above example is that the plug adapts connection for every request, so even for a request like:

http://localhost:4001/favicon.ico

we have the same "hello world" response, which is actually not what we want.

If only I could match?

So what do we need to do for a matching ? We are still adopting a connection so we still need a plug, but we will use the connection structure for routing.
Based on attributes like request method (GET, POST,..) and request URL, the plug will route incoming requests to different adaption logics:

defmodule MyPlug do
    import Plug.Conn

    def init(options) do
        # initialize options
        options
    end

    def call(conn, _opts) do
        match(conn.method, conn.path_info, conn)
    end

    defp match("GET", ["hello"], conn) do
        send_resp(conn, 200, "world")
    end

    defp match(_, _, conn) do
        send_resp(conn, 404, "oops")
    end
end

What we can see above is that we use method and path_info for Elixir function matching. In case of GET and path hello we route to response "world" otherwise we route to 404.

In practice we don't do this although we could, we rather use provided Plug.Router which offers much more.

Plug.Router

The charm of Plug.Router is, that it provides the DSL for matching/dispatching, which makes our code cleaner and error-prone. Let's rewrite the above example using Router.

First, we will replace the configuration for Plug.Cowboy:

children = [
    {Plug.Cowboy, scheme: :http, plug: AppRouter, options: [port: 4001]}
]

where we replaced MyPlug with AppRouter.

Then we will create AppRouter module with the use of Plug.Router:

defmodule AppRouter do
  use Plug.Router

    plug :match
    plug :dispatch

    match "/hello", via: :get do
      send_resp(conn, 200, "world")
    end

    match _ do
      send_resp(conn, 404, "oops")
    end
end

As you can see, we are using Plug.Router DSL, particularly match function which does all the work.

Give me some routes, please

We can go even further and replace match function with the following route definition:

get "/hello" do
  send_resp(conn, 200, "world")
end

When I first time saw this, I had a big AHA moment, like its usually a case with learning Elixir.

Let's break the route definition into pieces. First, we define the route request method (GET, POST, PUT), then we define route request path (/hello) and the last we define the route do/end block. What happens with those pieces?

Pipeline

We haven't talked yet about the two lines above:

plug :match
plug :dispatch

Every use of Plug.Router needs to have by default those two lines of code - so-called plug pipeline.
Plug pipeline you say? Yes, that means :match and :dispatch are also plugs.
The :match plug is responsible for matching by using method and path of the route. Behind the scenes all routes are compiled to match functions, like we saw above. The do/end block is stored as a function in the connection itself and is retrieved and called then by :dispatch plug.

As a little note: we can already notice another mechanism here - in order to share something between plugs we store information inside the connection.

So all three little pieces of the route are now used, but if by default match/dispatch are always needed, why do we need to write them down? The reason lies in the definition of the pipeline itself! We can add other plugs in the pipeline too and we can add them before or after a specific plug. The typical case is a plug responsible for parsing JSON.

REST API

Now that we understand what plug is, we can show an example of GET request with path parameter, query parameter, and an example of a typical POST request and response with application/json mime type:

defmodule AppRouter do
    use Plug.Router

    if Mix.env == :dev do
      use Plug.Debugger
    end

    import Plug.Conn

    plug :match
    plug Plug.Parsers, parsers: [:json],
                      pass: ["application/json"],
                      json_decoder: Jason
    plug :dispatch

    get "/hello" do
        # query parameter is user like this:
    # http://localhost:4001/hello?name=John
    # which will create %{"name" => "John"} 
      send_resp(conn, 200, "hello #{Map.get(conn.query_params, "name")}")
    end

    get "/hello/:name" do
      send_resp(conn, 200, "hello #{name}")
    end

    post "/hello" do
      # json body of POST request {"name": "John"} is parsed to %{"name" => "John"} 
      # so it can be accesable with e.g. Map.get(conn.body_params, "name") or with pattern matching
      name = 
        case conn.body_params do
          %{"name" => a_name } -> a_name
          _ -> ""
        end

      # returning JSON: {"id":1, "name": name}
      response = %{"id" => 1, "name" => name}
      send_resp(conn |> put_resp_content_type("application/json"), 200, Jason.encode!(response))
    end

    match _ do
      send_resp(conn, 404, "oops")
    end

end

I will not go into details about parsers, you can read about them here: https://hexdocs.pm/plug/Plug.Parsers.html

But I will explain shortly the code:

We are using Plug.Debugger in the router module for cases when an error occurs, like with the error of decoding json, a server responds with a nice error page.
We put Plug.Parsers in the pipeline after :match which will parse query parameters of GET requests and put them into conn.query_params and which will parse the json body of POST requests and put them into conn.body_params.
For decoding JSON we are using Jason, which we had to add to mix deps {:jason, "~> 1.2"},
In the example of POST we are using pattern matching. We could easily change this code to validate json structure and in case of failure return 500. We also have added a response header (again, by manipulating connection) and we send json as a response as well.

Happy coding!

Elixir, Ecto, OTP? The beginner's view, Part II.

Milan Pevec — Thu, 25 Jun 2020 08:06:26 +0000

In Part I of this series, we were researching the OTP concepts for the single purpose, to understand initial steps of Ecto integration. What was missing was the understanding of the Elixir Application concept.

Elixir application ?

In the examples used in Part I, we have been actually working inside of the application the entire time!

When we run mix new app, mix creates application for us. And whenever we run iex -S mix, mix compiles, and then even starts our app by default. In our previous examples, we just haven't implemented the so-called application callback yet. A callback to run/start something, like e.g. Greetings.Supervisor, when our application is started.

Let's add one.

If we open a file mix.exs and check the following function:

def application do
  [
    extra_applications: [:logger]
  ]
end

we see that there is an option with key extra_applications defined. The value represents the list of the applications that needs to be started before our app is started. You are right if you're thinking that iex is starting :logger application too.

Now let's add the application callback:

def application do
  [
    extra_applications: [:logger],
    mod: {App, []}
  ]
end

With the mod keyword we are defining which Elixir module is our application callback module. You can not put here whatever callback you want, but only elixir module that uses Application behavior:

defmodule App do
  use Application

  def start(_type, _args) do
    Greetings.Supervisor.start_link
  end
end

Here we used start/2 callback, which will be called when the application starts and inside we are calling our Greetings.Supervisor.start_link, like we did before manually.

Let's try it. After we call iex -S mix, which compiles and runs our application with application callback, we can do directly our API calls:

iex(1)> Greetings.Server.add(:supervised_greetings_genserver, 1, "Hi")
:ok
iex(2)> Greetings.Server.add(:supervised_greetings_genserver, 2, "Hello")
:ok
iex(3)> Greetings.Server.list(:supervised_greetings_genserver)           
%{1 => "Hi", 2 => "Hello"}

Sweet. We didn't need to do any calls related to the supervisor or similar, everything was done for us in the callback.

You can see more details about our app in the so-called application definition file, which is actually an artifact created by the compilation process. The file is located in _build/dev/lib/example-app/example-app.app. There you can see the list of all applications to be run before, names, etc.

A lit a bit of a different taste

As we saw, we created a supervisor module in the previous example of the Part I and we used the start_link/3 function:

start_link(module, init_arg, options \ [])

This is so-called module-based supervisor.

But there is, again, a shorter way, where we can use start_link/2 function directly in our application callback module. That means we can remove our supervisor module file and do the following:

defmodule App do
  use Application

  def start(_type, _args) do
    children = [
        {Greetings.Server, name: :supervised_greetings_genserver}
    ]

    # name is optional
    opts = [strategy: :one_for_one, name: Greetings.Supervisor]
    Supervisor.start_link(children, opts)
  end
end

As we can see, now we can call start_link/2 directly and we made our code even cleaner!

Why are we showing all this? If you remember, our mission is to understand Ecto initial steps as a total outsider to Elixir. Lets do the recap now.

Recap

So far we saw how the Elixir syntax for Ecto config is used. We saw the basics of the OTP and communication between processes. And above we saw the concept of the Application - we saw everything in order to understand initial steps for using Ecto. Let's use this knowledge and start from scratch.

First, we use mix and create new application with the support of the supervisors (and supervision trees):

mix new greetings —sup

This will create Greetings.Application with Application behavior, will add application callback in the mix.exs and implement the application callback function with calling start_link of Supervisor.

mix.exs:

...
def application do
  [
    extra_applications: [:logger],
    mod: {Greetings.Application, []}
  ]
end
...

application.ex:

defmodule Greetings.Application do
  use Application

  def start(_type, _args) do
    children = []

    opts = [strategy: :one_for_one, name: Greetings.Supervisor]
    Supervisor.start_link(children, opts)
  end
end

The list of its children (workers) is still empty at this point, we will complete it later.

Now let's add Ecto specifics. First let's add dependencies in the mix.exs:

defp deps do
  [
    {:ecto_sql, "~> 3.0"},
    {:postgrex, ">= 0.0.0"}
  ]
end

and install them with running mix deps.get.

Now let's create Ecto configuration and repository by running the following line:

mix ecto.gen.repo -r Greetings.Repo

This will create a lot of artifacts, but what's important now is that configuration is added in the config.exs:

import Config

config :greetings, Greetings.Repo,
  database: "greetings_repo",
  username: "user",
  password: "pass",
  hostname: "localhost"

Where we can put appropriate data for our database.

Also the Ecto.Repo behavior module was created:

defmodule Greetings.Repo do
  use Ecto.Repo,
    otp_app: :greetings,
    adapter: Ecto.Adapters.Postgres
end

Because we can have more repositories and we want our application to know about them all, we need to add a newly created repository to the list in the config. So we add in the config.exs the following:

config :greetings,
  ecto_repos: [Friends.Repo]

Last step is to add our Ecto repository as a child of the application supervisor, as we saw before with our Greeting.Server:

defmodule Greetings.Application do
  use Application

  def start(_type, _args) do
    children = [Greetings.Repo]

    opts = [strategy: :one_for_one, name: Greetings.Supervisor]
    Supervisor.start_link(children, opts)
  end
end

That's it! We put a lot of effort into understanding all the concepts, but I hope now it's more clear, at least it is for me.

Now let's try it out. We run iex -S mix (as we know now, this will do all the hard work) and run a DB query:

iex(1)> Ecto.Adapters.SQL.query(Greetings.Repo, "SELECT now()", [])

09:46:51.405 [debug] QUERY OK db=9.6ms decode=1.2ms queue=19.7ms idle=1231.4ms
SELECT now() []
{:ok,
 %Postgrex.Result{
   columns: ["now"],
   command: :select,
   connection_id: 86343,
   messages: [],
   num_rows: 1,
   rows: [[~U[2020-06-25 07:46:51.371536Z]]]
 }}

That's it, we manage to get to the end of our mission.

Elixir, Ecto, OTP? The beginner's view, Part I.

Milan Pevec — Wed, 24 Jun 2020 15:44:23 +0000

In my previous post Elixir, Weird Syntax, Is It Really?, I've started to talk about Elixir syntax as a total outsider. We touched the configuration for Ecto and its syntax. Let's continue here by showing my understanding of Ecto integration with Elixir.

In Ecto getting-started guide it is written that when we use Elixir mix for generating Elixir application, we need to provide —-sup option:

mix new our_app --sup

The --sup option ensures that our application has a supervision tree, which is mandatory when using Ecto. For a second, let's forget about these two keywords.

The starting point for using Ecto is the Ecto Repository. Without going into too many details we can see it as a heart of Ecto, a proxy that communicates with DB. After we have created it, we need to do "the last step" of its setup. The guide says:

The final piece of configuration is to setup the repository as a supervisor within the application's supervision tree... This piece of configuration will start the Ecto process which receives and executes our application's queries. Without it, we wouldn't be able to query the database at all!"

Ok, it makes some sense. First, we created our application with the support of supervision tree and then we configure our Ecto repository as a supervisor in application supervision tree.

Let's take a moment to break and write down all keywords (in none particular order):

supervision tree, supervisor, process, application

What's that all about? How do I understand these keywords without any background in Elixir?

The answer is hidden in the OTP. Let's go through its basic concepts to better understand how all things work together and then to understand how Ecto is configured/integrated in our applications. That's the goal of this blog, that's the mission!

OTP

One of many benefits of Elixir is the underlying Erlang runtime, which is giving us access to OTP (Open Telecom Platform).

Wait, Telecom what? The initials are not so important, lets just say that when we talk about OTP we talk about a set of libraries like Supervisor, GenServer, Application, Agent, Task...., a lot of libraries and tools. Let's try impossible, lets use a couple of them in one meaningful sentence:

Using OTP we can organize our software into lightweight execution units - processes, which can be observed and restarted if they fail with special processes called supervisors, where also supervisors can be supervised with other supervisors, which then together represent a supervision tree.

Interesting...

Process

OTP processes are the basic game piece. They are very lightweight, so don't confuse them with OS processes. You can have thousands of them, as many as you need. If you wanna hold on the state, you spawn a process; if u wanna concurrency, you spawn a process, and so on.

Lets spawn a process with iex and give it a function to run:

pid = spawn(fn -> IO.puts("hello world") end)

hello world
#PID<0.168.0>

With the pid variable we capture the process id for tracking purposes:

Process.alive?(pid)

false

This means we give the work - function to the process. It does the work and then it dies.

Process and the message

The most intuitive way to communicate between processes is with messages. In order to read messages, each process has a single mailbox - everyone can send a message to it but only the owner can read from its mailbox.

Using pattern matching you can decide on which messages to react and, by the nature of pattern matching itself, in which order.

Let's take a look at the example. Let's say we want to design a process, that will:

hold on the state of all greetings (like "Hello" etc.);
receive a message with a new greeting that will be put into the current state;
receive a message that will make our process send its current state;

We will communicate with this process within the iex (which is a process itself with its own mailbox!).

defmodule Greetings.Process do
  def start do
    spawn(__MODULE__, :loop, [%{}])
  end

  def loop(state) do
    receive do
      {:add, greeting_id, greeting} ->
        new_state = Map.put(state, greeting_id, greeting )
        loop(new_state)
      {:greetings, destination_pid} ->
        send(destination_pid, state)
        loop(state)
    end
  end
end

The code is very self-explanatory. In the start function, we spawn a process by sending to it the loop function with an empty map parameter. Map represents our initial and empty state.

In the loop function we are grabbing two types of messages by using pattern matching:

for adding a new greeting into our state;
for sending the current state (also as a message!) to the specific process with destination_pid. We are keeping our process alive and hence the state by looping.

Let's try it! In the iex we are first going to spawn a process and store its pid:

iex(42)> pid = Greetings.Process.start
#PID<0.293.0>

iex(43)> send(pid, {:add, 1, "Hello"})
{:add, 1, "Hello"}

iex(44)> send(pid, {:add, 2, "Bon jour"})
{:add, 2, "Bon jour"}

As of last we are going to send a message, which will trigger our process to send us back the current state:

iex(45)> send(pid, {:greetings, self})           
{:greetings, #PID<0.140.0>}

self is actually a pid of iex process, so that means we are saying to our process, send the state to us, to iex mailbox.
Now that we know that, we can ask iex to flush its mailbox:

iex(47)> flush
%{1 => "Hello", 2 => "Bon jour"}
:ok

Great. Feels very natural and clear.

Apparently this is a very common pattern in Elixir, hence they have built a more generic solution called GenServer. Why is that needed? Because the above example is missing a lot of functionality and has a lot of issues like deadlocks, message ordering, incompatibility with supervisors, etc.

GenServer

If we want to tackle the incompatibility with supervisors - remember our initial goal - we need to introduce GenServer.

On Hexdocs it's written:
"A GenServer is a process like any other Elixir process and it can be used to keep state, execute code asynchronously and so on... It will also fit into a supervision tree.".

Let's implement the previous example using Genserver.

defmodule Greetings.Server do
  use GenServer

  @name __MODULE__

  # API
  def start do
    GenServer.start(__MODULE__, :empty_state, name: @name)
  end

  def add(id, greeting) do
    GenServer.cast(@name, {:add, greeting_id, greeting})
  end

  def list do
    GenServer.call(@name, {:greetings})
  end

  # Callbacks
  def init(:empty_state) do
    {:ok, %{}}
  end

  def handle_cast({:add, greeting_id, greeting}, state) do
    new_state = Map.put(state, greeting_id, greeting )
    {:noreply, new_state}
  end

  def handle_call({:greetings}, _from, state) do
    {:reply, state, state}
  end
end

GenServer client APIs

API functions are used by the client to communicate with the server.

We need three functions: start, add/2, list.

Inside of the start function we call the GenServer to start a process. We set options to say, that the name of the started process should be the module name. What have we achieved with that? On downside now we have a "single server", which is fine for this purpose; but on the upside, we don't carry a process id in every API call so we just make it more clear. We also set the :empty_state atom, which will be explained later.

In the add/2 function, the client sends cast type of the message, which is async hence the client does not expect a reply. We are using this for adding new greetings.

In the list function, the client sends a call type of the message, which is sync and the client expects a reply from the server. We are using this to ask the server to reply with the list of all greetings - the state.

GenServer callback functions - Server APIs

With the Elixir use macro we are announcing that our Greetings.Server will use GenServer and will be populated with its specific behavior. I see this approach as a template method pattern.

In practice, that means we need to provide a series of callback functions that follow naming and typing conventions and are called by the server (GenServer).

We need three callback functions: init/1, handle_cast/2, handle_call/3.

When init/1 is called we return a tuple with our empty greetings state. The function has one parameter which matches the parameter in the client API, when we called GenServer.start/3. That means that atom :empty_state exists here for function clause matching.

With function handle_cast/2 server defines a new state (by adding a new greeting into it) and then return it.

With function handle_call/3 server replies with the current state of all greetings.

The second parameter _from is not needed hence the underscore.

Let's talk about the return value. The first state parameter is actually a payload that will be sent back to the client (our greetings) and the second state parameter is the payload that will be saved.

Let's try now our server. We are going to start a server, add greetings, and list them:

iex(56)> Greetings.Server.start
{:ok, #PID<0.645.0>}

iex(57)> Greetings.Server.add(1, "Hello")
:ok

iex(58)> Greetings.Server.add(2, "Hi")   
:ok

iex(59)> Greetings.Server.list           
%{1 => "Hello", 2 => "Hi"}

Great, very clear again. What have we seen so far is how OTP is working with processes by sending messages. Elixir also did another level of abstraction on top of GenServer by introducing Agents - for storing state, and Tasks - for concurrency jobs.

So we could rewrite our example again by using Agent and make the code even clearer. But this is out of the scope of this blog.

So now we can imagine better how a library like Ecto as a process could receive a message for querying DB and reply with results.

Let's try now to monitor our server with the supervisor.

Supervisor and monitoring

As you can notice we have used functions start/spawn but not functions start_link/spawn_link while we were creating processes. That means if the created process dies, nothing will happen with its parent, nor the process will be recreated. There is no link between them. If we would use start_link/spawn_link functions, then both processes die.

What if I would tell you, there is a way that the parent process can monitor its children and restart them in the case of a crash? That's possible when we have a supervisor process that monitors its linked children. When we have supervisors monitoring other supervisors, then we talk about supervision tree.

The example is even seen in the iex - if iex spawns a linked process that crashes, also iex will crash. But because iex is in its own supervision tree, a new iex process will be spawn after the crash.

Lets now create a supervisor and configure it to monitor our previous GenServer.

First, we will make little changes in the APIs of a GenServer:

  ...
  # API
  def start_link(opts) do
    GenServer.start_link(__MODULE__, :empty_state, opts)
  end

  def add(pid, greeting_id, greeting) do
    GenServer.cast(pid, {:add, greeting_id, greeting})
  end

  def list(pid) do
    GenServer.call(pid, {:greetings})
  end
  ...

We did two changes. First is we are using start_link/1 instead of the start and second is, now we are going to use pid parameter inside of our client API functions. Why?
We will name our GenServer process inside of its supervisor and send it as an option parameter (opts) when start_link/3 is called, so we don't have a fixed name for process anymore.

The implementation of supervisor looks like this:

defmodule Greetings.Supervisor do
    use Supervisor

    # Callbacks
    def start_link() do
        Supervisor.start_link(__MODULE__, [])
    end

    def init([]) do
        children = [
            {Greetings.Server, name: :supervised_greetings_genserver}
        ]

        Supervisor.init(children, strategy: :one_for_one)
    end
end

You can see we implement two callback functions, start_link - here we spawn a linked supervisor process and init/1 - here we initialize supervisor by defining its workers (child processes). We saw before how the use macro and callbacks are working.

In our case, the worker is our GenServer (Greetings.Server) where we name the process with the :supervised_greetings_genserver. This name is then used as a parameter when Greetings.Server.start_link/1 is called. Without that, we would have to work with process pid, which is less readable.

We also define the basic one-to-one strategy which means if the child process dies, a supervisor will just restart a new one.

So now that we have a supervisor in place, we don't start Greetings.Server directly, but we let the supervisor start it.

We will also use auxiliary call Supervisor.which_children/1 which will give us a list of supervised processes.

And last, we will use our Greetings.Server API for adding and listing greetings.

iex(1)> {:ok, sup} = Greetings.Supervisor.start_link
{:ok, #PID<0.158.0>}

iex(2)> Supervisor.which_children(sup)
[{Greetings.Server, #PID<0.159.0>, :worker, [Greetings.Server]}]

iex(3)> Greetings.Server.add(:supervised_greetings_genserver, 1, "Hello")
:ok

iex(4)> Greetings.Server.add(:supervised_greetings_genserver, 2, "Hi")   
:ok

iex(5)> Greetings.Server.list(:supervised_greetings_genserver)        
%{1 => "Hello", 2 => "Hi"}

Perfect. In order to complete our mission, we need to understand also the basics of the Application concept, which we will do in the follow-up blog.

Elixir, Weird Syntax, Is It Really ?

Milan Pevec — Sat, 06 Jun 2020 21:00:33 +0000

Intro

So, Ive decided. To learn. Elixir. Ive started with watching some Youtube videos (Elixir Conference), just to get into it, to become attracted to it, cause time is a big constraint in my life, unfortunately, and without real attraction, there is no motivation.

Syntax

For me coming from Java, Javascript world, doing some Ruby flirting, the syntax seemed strange. A bit. I was reading official docs (Getting Started), diagonally, but I wanted to jump directly into the sea with writting one example using Ecto library. And there I saw the subject of this blog - I saw the following lines of the code:

import Config

config :app1, Friends.Repo,
  database: "app1_repo",
  username: "user",
  password: "pass",
  hostname: "localhost"

Huh? That was my first reaction. Where is function, where are parentheses, brackets ? Weird syntax, is it really ?

Drill down sir

Seing that syntax was confusing. First I tough that I will skip "why?" and just use this. "Its just like this" approach. But after a day or two I've returned to this lines again.
Ive created my own example, slowly moving towards realisation that syntax is actually great and not weird at all!

Step 1

I've created two elixir source files "canvas.exs":

defmodule Canvas do
    def draw(title, color \\ "Red by default", shape \\ "Circle by default" ) do
        IO.puts("Draw a #{shape} with the color #{color}. Title is: #{title}.")
    end
end

and "main.exs":

Canvas.draw("Nice Canvas")

I've compiled canvas.exs and run main.exs with the expected output:

Draw a Circle by default with the color Red by default. Title is: Nice Canvas.

So far so good. As you can see, we created a module, function with default parameters and we print the final string.
Now you might ask yourself, what has this to do with our subject of this blog ? You will see, we will get there.

Step 2

What if we use the keyword list data structure for parameters of our function ? Lets try.

Our main.exs is gonna change to:

options = [{:color, "Green"}, {:shape, "Square"}]
Canvas.draw("Nice Canvas", options)

As you can see, keyword list is a key-value list, where keys are atoms. Neat.

Our canvas.exs is gonna change then to:

defmodule Canvas do
    def draw(title, options \\ []) do
        color = Keyword.get(options, :color, "Red by default")
        shape = Keyword.get(options, :shape, "Circle by default")
        IO.puts("Draw a #{shape} with the color #{color}. Title is: #{title}.")
    end
end

Elixir is offering Keyword module for manipulating keyword lists. Again very neat. But so far, very javascript-ish.

Step 3

From now on, we will change only main.exs to bring it closer to our objective; Canvas module is going to stay the same.

Lets first change our keyword list. Because keys are atoms, we can use special syntax, where we use key values as following:

options = [color: "Green", shape: "Square"]

And now lets put it directly to our function call:

Canvas.draw("Nice Canvas", [color: "Green", shape: "Square"])

What if I tell you that there is more! Elixir official docs says the following:
In general, when the keyword list is the last argument of a function, the square brackets are optional.
That means we can do this:

Canvas.draw("Nice Canvas", color: "Green", shape: "Square")

Whaaat, thats so cool.

Step 4

And now the final changes. As you probably know, there are two things more:

for named functions parentheses are optional;
if we use Import we don't need to prefix functions with module name;

So our final code change is:

import Canvas

draw "Nice Canvas", color: "Green", shape: "Square"

or if we break lines:

import Canvas

draw "Nice Canvas",
  color: "Green",
  shape: "Square"

Which is (almost) exactly the same as our initial Ecto code:

import Config

config :app1, Friends.Repo,
  database: "app1_repo",
  username: "user",
  password: "pass",
  hostname: "localhost"

So now we can understand that above code is actually a call of the named function "config/3" ie. with three parameters:

atom :app1
alias Friends.Repo (which behind the scenes is an atom)
keyword list used as last parameter, ie. function options

Final toughts

So, is it weird ? No, I think is great, it's readable and has clarity. I like it.