DEV Community: Tjaco Oostdijk

Extending Flutter Driver with custom commands

Tjaco Oostdijk — Mon, 17 Feb 2020 00:00:00 +0000

Flutter Driver is a library to write end-to-end integration tests for Flutter apps. It’s similar to Selenium WebDriver (for web apps), Espresso (for native Android apps) and Earl Grey (for native iOS apps). It works by instrumenting the Flutter app, deploying it on a real device or emulator and then ‘driving’ the application using a suite of Dart tests.

A typical, basic Flutter Driver test looks like this:


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

import 'package:flutter_driver/flutter_driver.dart';
import 'package:test/test.dart';

void main() {
  group('Counter App', () {
    final counterTextFinder = find.byValueKey('counter');
    final buttonFinder = find.byValueKey('increment');

    FlutterDriver driver;

    // Connect to the Flutter driver before running any tests.
    setUpAll(() async {
      driver = await FlutterDriver.connect();
    });

    // Close the connection to the driver after the tests have completed.
    tearDownAll(() async {
      driver?.close();
    });

    test('starts at 0', () async {
      expect(await driver.getText(counterTextFinder), "0");
    });

    test('increments the counter', () async {
      await driver.tap(buttonFinder);

      expect(await driver.getText(counterTextFinder), "1");
    });
  });
}

In the setUpAll hook a connection between the test and the running application is setup via the Flutter Driver API. This works because the application is instrumented with a Flutter Driver extension; basically an API injected into your app that can receive requests from our tests to “drive” the application.

The instrumentation of the Flutter app works by wrapping your app’s main function like this:


1
2
3
4
5
6
7

import 'package:flutter_driver/driver_extension.dart';
import 'package:my_app/main.dart' as app;

void main() {
  enableFlutterDriverExtension(); // <-- ENABLE INSTRUMENTATION
  app.main();
}

Flutter Driver supports a handful of API’s to communicate with the running app. For example getText, tap, waitFor etc. For me, coming from Nightwatch.js, the number of things that can be done to drive the application is quite limited.

Fortunately it’s possible to extend Flutter Driver to support custom commands. These commands allow you to communicate between your tests and the application and are also the foundation for all of Flutter Driver’s own API’s like getText, tap etc.

Extending Flutter Driver

To extend Flutter Driver with a custom command we need to provide a DataHandler. As the docs say:


1
2

Optionally you can pass a [DataHandler] callback. It will be called if the
test calls [FlutterDriver.requestData].

Flutter Driver will pass whatever is sent from test with driver.requestData(...) to the DataHandler. DataHandler only supports sending and receiving Strings, so you might want to encode your messages using JSON.

To demonstrate this, let’s implement a handler to navigate back to the root route of our app. This way we can ensure that every test starts from the root page of our application.

The first step is to provide a DataHandler to Flutter Driver:


1
2
3

enableFlutterDriverExtension(handler: (payload) async {
  print(payload);
});

The handler will receive a String payload and can optionally return a String response.

For the sake of simplicity let’s use a String as payload for now:


1
2
3
4
5

enableFlutterDriverExtension(handler: (payload) async {
  if(payload == "navigate_to_root") {
    // do something smart here
  }
});

From here, we need to implement something that will allow us to navigate to the root of our app. I’m not sure if the following is necessarily the best way to do this (if you know a better way please let me know!), but it works and is relatively straightforward.

We’ll use a NavigationObserver to get a hold of the NavigatorState, which we can use to push and pop routes. We need to be able to pass in a NavigationObserver from our test entry point, so we can access it when we receive a command to navigate to root.

Change the main function of your app as follows:


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

void main() {
  _main(null);
}

void mainTest(NavigatorObserver navigatorObserver) {
  _main(navigatorObserver);
}

void _main(NavigatorObserver navigatorObserver) {
  runApp(MyApp(
    navigatorObserver: navigatorObserver,
  ));
}

class MyApp extends StatelessWidget {
  final NavigatorObserver navigatorObserver;

  const MyApp({Key key, this.navigatorObserver}) : super(key: key);

  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      // all other MaterialApp initialisation here
      navigatorObservers: navigatorObserver == null ? [] : [navigatorObserver],
    );
  }
}

This allows us to hook up a NavigationObserver from our test wrapper like so:


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

import 'package:flutter/material.dart';
import 'package:flutter_driver/driver_extension.dart';
import 'package:my_app/main.dart' as app;

void main() {
  final navigationObserver = NavigatorObserver();

  enableFlutterDriverExtension(handler: (payload) async {
    if (payload == "navigate_to_root") {
      navigationObserver.navigator.popUntil(ModalRoute.withName('/'));
    }

    return null;
  });

  app.mainTest(navigationObserver);
}

Now from our tests we can send our custom command, for example in a setUp hook:


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

void main() {
  FlutterDriver driver;

  setUpAll(() async {
    driver = await FlutterDriver.connect();
  });

  tearDownAll(() async {
    driver?.close();
  });

  setUp(() async {
    await driver.requestData("navigate_to_root");
  });   

  /* Actual tests here */
}

This will make sure that before every test, the app navigates back to the root route no matter where we navigated to in our tests.

Of course, this is just an example of how you can implement communication between your Driver tests and your app. If you’re going to send more complex commands that require arguments you might want to send JSON data, but I’ll leave that as an exercise to you, dear reader ;-)

Using Phoenix Presence in LiveView |> A Simple Example

Tjaco Oostdijk — Tue, 11 Feb 2020 00:00:00 +0000

Normally I'm not a big fan of things that seem like magic, but in the case of Phoenix Presence I actually love it. I love the way you can very simply add Phoenix Presence to your Phoenix app and easily track user's presence with it. Amazing! I will definitely dig into the internals at some point in the future, but for now I wanted to show a very simple example of using Phoenix Presence inside LiveView!

Let's see how that would work.

Who's reading?

We are going to build a very simple but powerful feature for the blog you're currently reading. We'll add an indicator at the top of the page that shows how many people are currently reading this page. So where do we want this indicator? Preferably this would reside on the same page as a blog post so that it would indicate for every individual blog post how many users are currently reading that article. To do this, we'll create a very small LiveView component inserted in the markdown of our post. Just like we did in my last post when I rendered scrambled text. At a later stage I will probably want to make it more generic, but for now let's focus on getting this to work for this blog post.

If you head over to my interactive blog, you can see this counter in action!

This is how we embed the LiveView Component into out markdown post:

<%= Phoenix.LiveView.Helpers.live_render(@conn, RealWorldPhoenixWeb.Live.ReaderCount, session: %{slug: "simple_phoenix_presence"}) %>

LiveView or LiveView Component?

LiveView is maturing quickly and currently also has a concept of Components. Components basically are small building blocks that either are stateful or stateless. I am definitely planning on doing a writeup on using LiveView components, but for our usecase we simply need the basic LiveView.

Let's create a basic LiveView boilerplate:

defmodule RealWorldPhoenixWeb.Live.ReaderCount do
  use Phoenix.LiveView

  @moduledoc """
    A small LiveView that shows the number of readers of a post using Phoenix Presence
  """

  def render(assigns) do
    ~L"""
      Readers: <\%= @reader_count %>
    """
  end

  def mount(_session, socket) do
    {:ok, assign(socket, :reader_count, 0)}
  end
end

Ok, so now we need to hook up Presence. Let's see how that would work. First we'll head over to the docs and see how it is setup. The first thing I notice is this: Provides Presence tracking to processes and channels.. We are not really using Channels, but our LiveView is a process so I guess that should work, right?

Presence Setup

Let's first setup Presence using the guide provides by the Phoenix Team:

1. Setup a Presence Module in our app:

defmodule RealWorldPhoenix.Presence do
  use Phoenix.Presence, otp_app: :real_world_phoenix,
                        pubsub_server: RealWorldPhoenix.PubSub
end

2. Add this module to our supervision tree:

# lib/real_world_phoenix/application.ex

defmodule RealWorldPhoenix.Application do
  @moduledoc false
  use Application

  def start(_type, _args) do
    children = [
      RealWorldPhoenix.Repo,
      RealWorldPhoenixWeb.Endpoint,
      RealWorldPhoenix.Presence # <= Add This!
    ]

    opts = [strategy: :one_for_one, name: RealWorldPhoenix.Supervisor]
    Supervisor.start_link(children, opts)
  end

  def config_change(changed, _new, removed) do
    RealWorldPhoenixWeb.Endpoint.config_change(changed, removed)
    :ok
  end
end

I don't have any Channels... ?

So the next and last step in the guide shows an example of adding Presence tracking to a channel implementation. But we are not using channels! So how do we go about and add this to our LiveView?

Presence uses a PubSub (Publish-Subscribe) mechanism to perform it's magic. So what we should do in our LiveView is make sure we subscribe to the topic of the current page. Once we do that we can add a callback that will track any presence_diff where we'll update the counter if someone leaves or joins the current topic.

In our mount/2 function we'll subscribe to the page topic. Basically using a topic such as blog:simple_phoenix_presence. So in essence a general category with a post slug after the colon. Here are the updates to our LiveView module:

defmodule RealWorldPhoenixWeb.Live.ReaderCount do
  ...

  def mount(%{slug: slug}, socket) do
    # before subscribing, let's get the current_reader_count
    topic = "blog:#{slug}"
    initial_count = Presence.list(topic) |> map_size

    # Subscribe to the topic
    RealWorldPhoenixWeb.Endpoint.subscribe(topic)

    # Track changes to the topic
    Presence.track(
      self(),
      topic,
      socket.id,
      %{}
    )

    {:ok, assign(socket, :reader_count, initial_count)}
  end
  # ...
end

Listening to topic events to track changes

Now the only thing left for us to do is implement a callback in our live_view to catch any diffs happening on the topic we are tracking. The following function will do the trick:

...
  def handle_info(
        %{event: "presence_diff", payload: %{joins: joins, leaves: leaves}},
        %{assigns: %{reader_count: count}} = socket
      ) do
    reader_count = count + map_size(joins) - map_size(leaves)

    {:noreply, assign(socket, :reader_count, reader_count)}
  end
...

We pattern match on the "presence_diff" event and also use pattern matching to get the current values for joins, leaves and count respectively. When someone subscribes to the topic, there will be a change in the joins and when they leave a similar change in leaves. The reader_count is something we update ourselves in the socket whenever someone joins or leaves.

That is really all we need to do to get realtime updates of the number of readers of our page. That is quite easy and really cool to be able to see that data. Currently we don't ask user for information once they enter our blog, but once we add login or even just their name, we would be able to show that data as part of the presence_diff as well. But, let's leave that for another time.

Conclusion

I hope that I showed you how easy it is to add Phoenix Presence to a LiveView component. I was pleasantly surprised as to how little effort this actually takes.

Hope you learned something new!

Until next time!

Using Phoenix Presence in LiveView |> A Simple Example

Tjaco Oostdijk — Tue, 11 Feb 2020 00:00:00 +0000

Normally I’m not a big fan of things that seem like magic, but in the case of Phoenix Presence I actually love it. I love the way you can very simply add Phoenix Presence to your Phoenix app and easily track user’s presence with it. Amazing! I will definitely dig into the internals at some point in the future, but for now I wanted to show a very simple example of using Phoenix Presence inside LiveView!

Let’s see how that would work.

Who’s reading?

We are going to build a very simple but powerful feature for the blog you’re currently reading. We’ll add an indicator at the top of the page that shows how many people are currently reading this page. So where do we want this indicator? Preferably this would reside on the same page as a blog post so that it would indicate for every individual blog post how many users are currently reading that article. To do this, we’ll create a very small LiveView component inserted in the markdown of our post. Just like we did in my last post when I rendered scrambled text. At a later stage I will probably want to make it more generic, but for now let’s focus on getting this to work for this blog post.

If you head over to my interactive blog, you can see this counter in action!

This is how we embed the LiveView Component into out markdown post:


1

<%= Phoenix.LiveView.Helpers.live_render(@conn, RealWorldPhoenixWeb.Live.ReaderCount, session: %{slug: "simple_phoenix_presence"}) %>

LiveView or LiveView Component?

Let’s create a basic LiveView boilerplate:


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

defmodule RealWorldPhoenixWeb.Live.ReaderCount do
  use Phoenix.LiveView

  @moduledoc """
    A small LiveView that shows the number of readers of a post using Phoenix Presence
  """

  def render(assigns) do
    ~L"""
      Readers: <\%= @reader_count %>
    """
  end

  def mount(_session, socket) do
    {:ok, assign(socket, :reader_count, 0)}
  end
end

Ok, so now we need to hook up Presence. Let’s see how that would work. First we’ll head over to the docs and see how it is setup. The first thing I notice is this: Provides Presence tracking to processes and channels.. We are not really using Channels, but our LiveView is a process so I guess that should work, right?

Presence Setup

Let’s first setup Presence using the guide provides by the Phoenix Team:

1. Setup a Presence Module in our app:


1
2
3
4

defmodule RealWorldPhoenix.Presence do
  use Phoenix.Presence, otp_app: :real_world_phoenix,
                        pubsub_server: RealWorldPhoenix.PubSub
end

2. Add this module to our supervision tree:


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

# lib/real_world_phoenix/application.ex

defmodule RealWorldPhoenix.Application do
  @moduledoc false
  use Application

  def start(_type, _args) do
    children = [
      RealWorldPhoenix.Repo,
      RealWorldPhoenixWeb.Endpoint,
      RealWorldPhoenix.Presence # <= Add This!
    ]

    opts = [strategy: :one_for_one, name: RealWorldPhoenix.Supervisor]
    Supervisor.start_link(children, opts)
  end

  def config_change(changed, _new, removed) do
    RealWorldPhoenixWeb.Endpoint.config_change(changed, removed)
    :ok
  end
end

I don’t have any Channels… ?

So the next and last step in the guide shows an example of adding Presence tracking to a channel implementation. But we are not using channels! So how do we go about and add this to our LiveView?

Presence uses a PubSub (Publish-Subscribe) mechanism to perform it’s magic. So what we should do in our LiveView is make sure we subscribe to the topic of the current page. Once we do that we can add a callback that will track any presence_diff where we’ll update the counter if someone leaves or joins the current topic.

In our mount/2 function we’ll subscribe to the page topic. Basically using a topic such as blog:simple_phoenix_presence. So in essence a general category with a post slug after the colon. Here are the updates to our LiveView module:


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

defmodule RealWorldPhoenixWeb.Live.ReaderCount do
  ...

  def mount(%{slug: slug}, socket) do
    # before subscribing, let's get the current_reader_count
    topic = "blog:#{slug}"
    initial_count = Presence.list(topic) |> map_size

    # Subscribe to the topic
    RealWorldPhoenixWeb.Endpoint.subscribe(topic)

    # Track changes to the topic
    Presence.track(
      self(),
      topic,
      socket.id,
      %{}
    )

    {:ok, assign(socket, :reader_count, initial_count)}
  end
  # ...
end

Listening to topic events to track changes

Now the only thing left for us to do is implement a callback in our live_view to catch any diffs happening on the topic we are tracking. The following function will do the trick:


1
2
3
4
5
6
7
8
9
10

...
  def handle_info(
        %{event: "presence_diff", payload: %{joins: joins, leaves: leaves}},
        %{assigns: %{reader_count: count}} = socket
      ) do
    reader_count = count + map_size(joins) - map_size(leaves)

    {:noreply, assign(socket, :reader_count, reader_count)}
  end
...

We pattern match on the “presence_diff” event and also use pattern matching to get the current values for joins, leaves and count respectively. When someone subscribes to the topic, there will be a change in the joins and when they leave a similar change in leaves. The reader_count is something we update ourselves in the socket whenever someone joins or leaves.

That is really all we need to do to get realtime updates of the number of readers of our page. That is quite easy and really cool to be able to see that data. Currently we don’t ask user for information once they enter our blog, but once we add login or even just their name, we would be able to show that data as part of the presence_diff as well. But, let’s leave that for another time.

Conclusion

I hope that I showed you how easy it is to add Phoenix Presence to a LiveView component. I was pleasantly surprised as to how little effort this actually takes.

Hope you learned something new!

Until next time!

Rendering Markdown in Elixir

Tjaco Oostdijk — Wed, 29 Jan 2020 00:00:00 +0000

Recently, I acquired the domain realworldphoenix.com in order to continue my blogging adventure on a separate blog dedicated to writing exclusive content on Phoenix and Elixir (and some related subjects occassionally). I could of course take an off the shelf blogging platform or just use some kind of static site generator. But as I am writing about using Phoenix in the Real World, I figured why not just create a Phoenix application to host my blog. Then I can also do some nice things like make it a bit interactive or even let people sign-up and comment without using a third-party tool for that. We'll see.

Writing content for a blog.

As most of you would probably agree, writing long-form content using html is a bit of a pain. So most people use markdown to write blogposts and I'm no exception. So let's find out how we can write markdown, put it under version control (just because that is nice to have) and then make sure it gets rendered as html in our Phoenix app.

Earmark

The go-to way to convert markdown to html in Elixir land is the great library written by Dave Thomas, Earmark. It is used by ex_doc and is a nice and well maintained library. We could of course convert our markdown in our phoenix controller directly using Earmark and output that in our view, but we can do even better. We can make sure that our Phoenix app can actually render markdown files stored in our template directly to html, just like it render's eex to html.

Phoenix Markdown Library

In fact there is even a library that handles this for us: Phoenix Markdown. Now I can just use this and be done with it (and as you are reading this... I am actually already using it ;) ), but my curious mind wants to know more! I'd like to know how this mechanism works so I have a better understanding of how rendering works in Phoenix. So, without further ado, let's crack this baby open and take a look inside. Ooohhh what is that....?

It's a Template Engine!

Ofcourse, Phoenix has this all figured out. The way that eex and exs are implemented is as Phoenix Template Engines. It is fairly straightforward to create your own Template Engine by implementing the Phoenix.Template.Engine behaviour. This is exactly what Boyd Multerer has created in his library phoenix_markdown. Does that name sound familiar? Oh, that's because boyd was also the creator of the awesome Scenic Library. Great stuff!

How do we create a Template Engine?

So how does one go about and create a template engine? Well, the best way to find out is to see if we can create one for ourselves, right? So let's create a template engine that scrambles all the text that gets rendered by the engine. We'll use the Cambridge University Scrambled Text concept. So scramble all letters in words but keep the first and last letters intact. Fun fact, this so-called research was actually never done at Cambridge University! Somehow this internet meme got morphed into being a Cambridge research subject, but was actually never the case. Here's a nice article from a Cambridge Professor about this.
This however doesn't hold us back from using the concept to build our awesome scramble engine! So let's get cakrcnig!

Our scramble Engine

If you want to follow along you can create a fresh phoenix project and put the files in there to test this out yourself.
We'll define our engine in lib/engine/scrambled.ex. The Phoenix.Template.Engine behaviour requires us to implement compile/2. In there we can add our magic formula. :)

This is what I came up with for our use case:

defmodule RealWorldPhoenix.Engines.Scrambled do
  @moduledoc false

  @behaviour Phoenix.Template.Engine

  def compile(path, _name) do
    path
    |> File.read!()
    |> scramble_words()
    |> Earmark.as_html!()
    |> EEx.compile_string(engine: Phoenix.HTML.Engine, file: path, line: 1)
  end

  defp scramble_words(content) do
    content
    |> String.split("\n", trim: true)
    |> Enum.map(&shuffle_words_in_sentence/1)
    |> Enum.join("\n")
  end

  defp shuffle_words_in_sentence(sentence) do
    sentence
    |> String.split(" ")
    |> Enum.map(&shuffle/1)
    |> Enum.join(" ")
  end

  defp shuffle(word) do
    if String.length(word) < 4 do
      word
    else
      scrambled =
        word
        |> String.split("", trim: true)
        |> Enum.slice(1..-2)
        |> Enum.shuffle()
        |> Enum.join("")

      String.first(word) <> scrambled <> String.last(word)
    end
  end
end

And to use this in our Phoenix app, we need to configure the file extension we want to invoke scrambling for. You can see this is the same as I did in my app to configure phoenix_markdown. At compile time Phoenix will match any templates with the .scrambled extension and will run that source file through our Engine by calling our compile/2 function defined in our Engine module.

# config/config.exs

config :phoenix, :template_engines,
  md: PhoenixMarkdown.Engine,
  scrambled: RealWorldPhoenix.Engines.Scrambled

So I am expecting this to scramble content written in markdown. I'm just considering paragraphs of text, just to not complicate things unnecessarily for this use case. That is why I am basically splitting sentences by splitting on the newline character and then processing each sentence and the containing words before joining them together again with newlines. Seems pretty straightforward. Let's see if this works!

Smartypants... 🤔

Phoenix Markdown has an option to render server_tags, which basically means that you can invoke Elixir inside your markdown page using the standard tags you use in eex templates to invoke some Elixir code and evaluate it. I wanted to use this to render the following bit of text to illustrate my scrambled engine, but the first pass led to this error while compiling:

== Compilation error in file lib/real_world_phoenix_web/views/post_view.ex ==
** (SyntaxError) lib/real_world_phoenix_web/templates/post/blogs/2020-01-28/rendering_markdown.html.md:60: unexpected token: "“" (column 39, code point U+201C)
    (eex) lib/eex/compiler.ex:45: EEx.Compiler.generate_buffer/4
    (phoenix) lib/phoenix/template.ex:361: Phoenix.Template.compile/3
    (phoenix) lib/phoenix/template.ex:167: anonymous fn/4 in Phoenix.Template."MACRO-__before_compile__"/2
    (elixir) lib/enum.ex:1948: Enum."-reduce/3-lists^foldl/2-0-"/3
    (phoenix) expanding macro: Phoenix.Template.__before_compile__/1
    lib/real_world_phoenix_web/views/post_view.ex:1: RealWorldPhoenixWeb.PostView (module)
    (elixir) lib/kernel/parallel_compiler.ex:229: anonymous fn/4 in Kernel.ParallelCompiler.spawn_workers/7

What? 😕🤷

Ok, so what is happening here. It is seeing an unexpected token and it seems to be a left handed double quote. After banging my head against the wall for about a day, I decided to rtfm again and luckily Boyd has us covered. See here, and I'll quote it below:

By default Earmark replaces some characters with prettier UTF-8 versions. For example, single and double quotes are replaced with left- and right-handed versions. This may break any server tag which contains a prettified character since EEx cannot interpret them as intended. To fix this, disable smartypants processing.

So it is actually a simple fix in our config:

config :phoenix_markdown, :earmark, %{
  smartypants: false
}

Interactive blog...

The example below is rendered using LiveView. You can see this if you head over to my interactive blog.

<%= Phoenix.LiveView.Helpers.live_render(@conn, RealWorldPhoenixWeb.Live.Scrambled) %>

And here is the LiveView component responsible for rendering the scrambled text:

defmodule RealWorldPhoenixWeb.Live.Scrambled do
  use Phoenix.LiveView
  @moduledoc false

  def render(assigns) do
    ~L"""
      <h2 id="scramble">Let's Scramble!
        <button phx-click="scramble" class="button">
          <%= if @scrambled, do: "un-scramble", else: "scramble" %>
        </button>
      </h2>
      <%= get_scrambled_content(@scrambled) %>
    """
  end

  def mount(_session, socket) do
    {:ok, assign(socket, :scrambled, true)}
  end

  def handle_event("scramble", _value, socket) do
    {:noreply, assign(socket, :scrambled, !socket.assigns.scrambled)}
  end

  defp get_scrambled_content(true) do
    Phoenix.View.render(RealWorldPhoenixWeb.PostView, "blogs/2020-01-28/_scrambled.html", [])
  end

  defp get_scrambled_content(false) do
    Phoenix.View.render(RealWorldPhoenixWeb.PostView, "blogs/2020-01-28/_notscrambled.html", [])
  end
end

Hold on buddy! This can't work!

The above LiveView renders scrambled text and when the button is pressed it unscrambles it.

Ah I was wondering if you would notice. You smartypants out there probably were already wondering how in the world I get this scrambling functionality working. No, I'm not fooling you, it is running though our actual template engine. But indeed Phoenix pre-compiles the templates. That is also one of the reasons it is so fast.

So it is not running through our template when you press that button. I confess. I have two versions of the same file it switches between, one with the .scrambled extension and one without that are both precompiled by Phoenix when the app gets compiled. You will notice that the scrambling is also the same if you switch back and forth. That is because, once compiled, it does not recompile necessarily.

Conclusion

Although my Scrambled Engine might not be so useful, I think it does illustrate how easy it can be to create a Template Engine that does something small and works pretty quickly out of the box.

Be sure to check out the Core Example EEx Engine if you want to dive in some more. And be sure to read the awesome docs about templating in Phoenix!

I hope this was helpful and that you learned something new.

Until next time!

Rendering Markdown in Elixir

Tjaco Oostdijk — Wed, 29 Jan 2020 00:00:00 +0000

Writing content for a blog.

As most of you would probably agree, writing long-form content using html is a bit of a pain. So most people use markdown to write blogposts and I’m no exception. So let’s find out how we can write markdown, put it under version control (just because that is nice to have) and then make sure it gets rendered as html in our Phoenix app.

Earmark

Phoenix Markdown Library

In fact there is even a library that handles this for us: Phoenix Markdown. Now I can just use this and be done with it (and as you are reading this… I am actually already using it ;) ), but my curious mind wants to know more! I’d like to know how this mechanism works so I have a better understanding of how rendering works in Phoenix. So, without further ado, let’s crack this baby open and take a look inside. Ooohhh what is that….?

It’s a Template Engine!

Ofcourse, Phoenix has this all figured out. The way that eex and exs are implemented is as Phoenix Template Engines. It is fairly straightforward to create your own Template Engine by implementing the Phoenix.Template.Engine behaviour. This is exactly what Boyd Multerer has created in his library phoenix_markdown. Does that name sound familiar? Oh, that’s because boyd was also the creator of the awesome Scenic Library. Great stuff!

How do we create a Template Engine?

So how does one go about and create a template engine? Well, the best way to find out is to see if we can create one for ourselves, right? So let’s create a template engine that scrambles all the text that gets rendered by the engine. We’ll use the Cambridge University Scrambled Text concept. So scramble all letters in words but keep the first and last letters intact. Fun fact, this so-called research was actually never done at Cambridge University! Somehow this internet meme got morphed into being a Cambridge research subject, but was actually never the case. Here’s a nice article from a Cambridge Professor about this.This however doesn’t hold us back from using the concept to build our awesome scramble engine! So let’s get cakrcnig!

Our scramble Engine

If you want to follow along you can create a fresh phoenix project and put the files in there to test this out yourself.We’ll define our engine in lib/engine/scrambled.ex. The Phoenix.Template.Engine behaviour requires us to implement compile/2. In there we can add our magic formula. :)

This is what I came up with for our use case:


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

defmodule RealWorldPhoenix.Engines.Scrambled do
  @moduledoc false

  @behaviour Phoenix.Template.Engine

  def compile(path, _name) do
    path
    |> File.read!()
    |> scramble_words()
    |> Earmark.as_html!()
    |> EEx.compile_string(engine: Phoenix.HTML.Engine, file: path, line: 1)
  end

  defp scramble_words(content) do
    content
    |> String.split("\n", trim: true)
    |> Enum.map(&shuffle_words_in_sentence/1)
    |> Enum.join("\n")
  end

  defp shuffle_words_in_sentence(sentence) do
    sentence
    |> String.split(" ")
    |> Enum.map(&shuffle/1)
    |> Enum.join(" ")
  end

  defp shuffle(word) do
    if String.length(word) < 4 do
      word
    else
      scrambled =
        word
        |> String.split("", trim: true)
        |> Enum.slice(1..-2)
        |> Enum.shuffle()
        |> Enum.join("")

      String.first(word) <> scrambled <> String.last(word)
    end
  end
end


1
2
3
4
5

# config/config.exs

config :phoenix, :template_engines,
  md: PhoenixMarkdown.Engine,
  scrambled: RealWorldPhoenix.Engines.Scrambled

So I am expecting this to scramble content written in markdown. I’m just considering paragraphs of text, just to not complicate things unnecessarily for this use case. That is why I am basically splitting sentences by splitting on the newline character and then processing each sentence and the containing words before joining them together again with newlines. Seems pretty straightforward. Let’s see if this works!

Smartypants… 🤔


1
2
3
4
5
6
7
8
9

== Compilation error in file lib/real_world_phoenix_web/views/post_view.ex ==
** (SyntaxError) lib/real_world_phoenix_web/templates/post/blogs/2020-01-28/rendering_markdown.html.md:60: unexpected token: "“" (column 39, code point U+201C)
    (eex) lib/eex/compiler.ex:45: EEx.Compiler.generate_buffer/4
    (phoenix) lib/phoenix/template.ex:361: Phoenix.Template.compile/3
    (phoenix) lib/phoenix/template.ex:167: anonymous fn/4 in Phoenix.Template."MACRO- __before_compile__"/2
    (elixir) lib/enum.ex:1948: Enum."-reduce/3-lists^foldl/2-0-"/3
    (phoenix) expanding macro: Phoenix.Template. __before_compile__ /1
    lib/real_world_phoenix_web/views/post_view.ex:1: RealWorldPhoenixWeb.PostView (module)
    (elixir) lib/kernel/parallel_compiler.ex:229: anonymous fn/4 in Kernel.ParallelCompiler.spawn_workers/7

What? 😕🤷

So it is actually a simple fix in our config:


1
2
3

config :phoenix_markdown, :earmark, %{
  smartypants: false
}

Interactive blog…

The example below is rendered using LiveView. You can see this if you head over to my interactive blog.


1

<%= Phoenix.LiveView.Helpers.live_render(@conn, RealWorldPhoenixWeb.Live.Scrambled) %>

And here is the LiveView component responsible for rendering the scrambled text:


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

defmodule RealWorldPhoenixWeb.Live.Scrambled do
  use Phoenix.LiveView
  @moduledoc false

  def render(assigns) do
    ~L"""
      <h2 id="scramble">Let's Scramble!
        <button phx-click="scramble" class="button">
          <%= if @scrambled, do: "un-scramble", else: "scramble" %>
        </button>
      </h2>
      <%= get_scrambled_content(@scrambled) %>
    """
  end

  def mount(_session, socket) do
    {:ok, assign(socket, :scrambled, true)}
  end

  def handle_event("scramble", _value, socket) do
    {:noreply, assign(socket, :scrambled, !socket.assigns.scrambled)}
  end

  defp get_scrambled_content(true) do
    Phoenix.View.render(RealWorldPhoenixWeb.PostView, "blogs/2020-01-28/_scrambled.html", [])
  end

  defp get_scrambled_content(false) do
    Phoenix.View.render(RealWorldPhoenixWeb.PostView, "blogs/2020-01-28/_notscrambled.html", [])
  end
end

Hold on buddy! This can’t work!

The above LiveView renders scrambled text and when the button is pressed it unscrambles it.

Ah I was wondering if you would notice. You smartypants out there probably were already wondering how in the world I get this scrambling functionality working. No, I’m not fooling you, it is running though our actual template engine. But indeed Phoenix pre-compiles the templates. That is also one of the reasons it is so fast.

Conclusion

Although my Scrambled Engine might not be so useful, I think it does illustrate how easy it can be to create a Template Engine that does something small and works pretty quickly out of the box.

Be sure to check out the Core Example EEx Engine if you want to dive in some more. And be sure to read the awesome docs about templating in Phoenix!

I hope this was helpful and that you learned something new.

Until next time!

Real World Phoenix |> Let's D4Y |> using K8S

Tjaco Oostdijk — Tue, 21 Jan 2020 00:00:00 +0000

As promised in my last post, here is step 3 in my deployment adventure. The onewhere I use Kubernetes!

In the last post we explored a very straightforward and simpleway to deploy a Phoenix application using the built-in Elixir releases on render.com.

Today I want to discuss a much more complicated setup, using Kubernetes. I know formost applications you will surely not need a setup using Kubernetes, but I’vebeen having an urge to try it out for a while now and wanted to see if it isreally that complicated as I seem to be hearing through the grapevine. Whoknows, maybe it turns out it’s actually not that hard at all. Also I really like the fact that I can have a cluster to spin up nodes (ie.applications) of all the experiments / ideas / apps that I’m currently playingwith, so I can actually expose those into the Real World with a reproducable andeasy deployment workflow. Let’s see if we can make that dream come true!

The Kubernetes Cluster

If you are not aware of what Kubernetes is, head over to their website, as they have great documentation to get started on the basics.

To use Kubernetes for our deployment the first thing we’ll need is a Kubernetescluster. This is the part where you see most tutorials and guides grab for alocal setup with minikube. While I like that it’s possible to do this, it stillseems a bit too far from real life for me, so I wanted to have a bit more of aproduction-like environment, like… for instance… a production environment :)

The beauty is that these days there are more and more managed Kubernetesservices that do all the hard work like setting up the cluster, managingupgrades etc etc. I’ll be using the service DigitalOcean is offering. Mainly because Ialready have a lot of content running at DigitalOcean, so it is known territoryfor me and they make it really easy to get started with Kubernetes.If you haven’t used DigitalOcean before, you can use this referrallink to create an account and get $100 creditto spend in the first 60 days. So more than enough to follow along with this tutorial.

Terraform + Helm

We are going to need to use 2 more tools to get this setup up and running. Terraform and Helm. We’ll use terraform to setup most of the infrastructure. In this way I don’t have to remember all the tweaks and install steps I took to get everything up and running. I can just automate the whole setup declaratively and put the whole thing under version control. This is often referred to these days as an infrastructure-as-code setup.I wanted to use terraform for everything, but ran into some trouble setting up a few of the tools needed, so reverted to Helm (the Kubernetes package manager) toinstall Traefik and our Gitlab Runner.

Our Terraform setup will include:

Setting up a DigitalOcean Kubernetes Cluster with 2 worker nodes
Connecting a gitlab project to the kubernetes cluster
Creating a kubernetes_secret to be able to pull images from our private gitlabregistry.
Setup a managed database cluster service @ digitalocean

And we’ll use Helm to:

Install Traefik as an ingress controller
Create a Gitlab Runner in our cluster

So let’s get crackin’!

Install doctl to talk to your DigitalOcean account

If we want to automate any of this stuff we’ll have to be able to talk toDigitalOcean programmatically, so installing doctl is step one. Please refer tothis guide to get that setup: doctl up and runningguide

After you have doctl installed, make sure you create a personal access token in DigitalOcean and add that in an environment variable and also initialize youraccount.


1
2
3

export DIGITALOCEAN_TOKEN=[token]
doctl auth init -t $DIGITALOCEAN_TOKEN
doctl account get

Install Terraform

Of course there is a handy install guide for terraform also. Actually just anexecutable, as it is written in go, so they have a handy packaged binary you can use.See: terraform install

Setup our project

If you want to follow along with this guide, create a folder that will hold yourterraform configs and create a main.tf file that will hold the terraform declarations.


1
2
3

mkdir kube-terra
cd kube-terra
touch main.tf

Setup the DigitalOcean Kubernetes cluster

Terraform works in a declaritive manner, which means that we state what we wantthe world to be like and terraform figures out how to get to that state. Interraform you describe an item you want to exist as a resource. A Kubernetescluster is a form of a resource that is provided by one of the many providersthat exist in terraform. So let’s create a DigitalOcean kubernetes cluster:

Creating a resource is always in the format: resource [kind] [reference] {}We can use the reference we provide later on to refer back to this resource.


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

resource "digitalocean_tag" "kubernetes-cl01" {
  name = "kubernetes-cl01"
}

data "digitalocean_kubernetes_versions" "versions" {}

resource "digitalocean_kubernetes_cluster" "cl01" {
  name = "cl01"
  region = "ams3"
  version = data.digitalocean_kubernetes_versions.versions.latest_version

  node_pool {
    name = "default"
    size = "s-1vcpu-2gb"
    node_count = 2
    tags = ["${digitalocean_tag.kubernetes-cl01.id}"]
  }
}

Now you can get your cluster setup by issuing the following commands in the folder youcreated:


1
2

tarraform init
terraform apply

This will take about 5 minutes and then you’ll have a fresh kubernetes clusterup and running. Now that was easy!

Install Traefik

Kubernetes needs an ingress controller to route traffic from the outside worldto the services that are running inside the cluster as these services are notexposed to the outside world, which is a good thing! Nginx is often used, but I went with Traefik, a very nice alternative that has a lot of nice functinality out of the box, like automatic ssl certificates using Let’s Encryptand auto discovery of services running in the cluster. This setup also takes advantage of a DigitalOcean load balancer, which will automatically be created by the serviceType set below to LoadBalancer.

Create a file called traefik-values.yml in the root of your kubernetes config directory and add the following content. If you manage dns settings with DigitalOcean, you can comment out those parts as well. This will provide automatic ssl certificate generation. Very cool! If not you can just leave it like this. It’ll work, but without ssl enabled. A benefit of using the dns-challenge as opposed to the more standard acme-challenge is that you can also get a valid certificate if your production system is behind a firewall.


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

image: traefik
dashboard:
  enabled: true
  domain: domain.example.com # put a (sub)domain here where you want to access the traefik dashboard
serviceType: LoadBalancer
rbac:
  enabled: true
# ssl:
# enabled: true # Enables SSL
# enforced: true # Redirects HTTP to HTTPS
# acme:
# enabled: true # Enables Let's Encrypt certificates
# staging: false # Use Lets Encrypt staging area for this example. For production purposes set this to false
# email: info@drumusician.com # Email address that Let's Encrypt uses to notify about certificate expiry etc.
# challengeType: "dns-01"
# dnsProvider:
# name: digitalocean # This is why you need your domain to be under Digital Ocean control
# digitalocean:
# DO_AUTH_TOKEN: $DIGITALOCEAN_TOKEN
# domains:
# enabled: true
# domainsList:
# - main: domain.example.com # domain that belongs to this certificate

Now we can go ahead and install traefik in our cluster:

Install helm:


1

brew install kubernetes-helm

Pull in our fresh cluster configuration locally (otherwise helm willfail):


1

doctl kubernetes cluster kubeconfig save cl01

And install traefik:


1
2
3
4

# get the helm/stable charts
helm repo add stable https://kubernetes-charts.storage.googleapis.com/
# install Traefik
helm install traefik --values traefik-values.yml stable/traefik

After Traefik is installed we have enough running to be able to deploy our applicationto the cluster. We are going to use gitlab-ci to deploy so we’ll need toinstall a gitlab-runner in our cluster first so we can use that in our project.

Creating a kubernetes secret to access our registry

If we install the Gitlab runner in the next step, we’ll need a kubernetes secretto be able to pull from our private registry in gitlab.Go to https://gitlab.com/profile/personal\_access\_tokens and create an accesstoken to access the gitlab registry. Scopes: api, read_registry.

Then create a file at ~/.docker/docker-registry.json with the credentials forthe access token:


1
2
3
4
5
6
7

{
  "registry.gitlab.com": {
    "username": "", # access-token-name
    "password": "", # token
    "email": "" # email of the gitlab account
  }
}

And then in our main.tf we can create the kubernetes secret.


1
2
3
4
5
6
7
8
9
10
11

resource "kubernetes_secret" "docker_pull_secret" {
  metadata {
    name = "gitlab.com"
  }

  data = {
    ".dockercfg" = file("~/.docker/docker-registry.json")
  }

  type = "kubernetes.io/dockercfg"
}

With that in place, we’ll have to initialize the new provider and then we can add the secret:


1
2

terraform init
terraform apply

Install Gitlab Runner

To be able to deploy into our cluster from Gitlab CI we will need our runner to run inside of our cluster and have enough access rights to actually spin-up pods and services in our cluster. So we’ll install that using helm as well. Gitlab has an install script from their interface, but that doesn’t provide you with customisation options that we’ll need for our use case.

Create the following file in your kubernetes dir: gitlab-runner-values.yml andadd this content:


1
2
3
4
5
6
7
8

gitlabUrl: https://gitlab.com/
runnerRegistrationToken: "" # copy this from your gitlab project settings: 
rbac:
  create: true
  clusterWideAccess: true
runners:
  privileged: true
imagePullSecrets: ["gitlab.com"]

Now let’s install the runner:


1

helm install --namespace default gitlab-runner -f gitlab-runner-values.yml gitlab/gitlab-runner

Connect a gitlab project to the cluster

This is something you can do through the Gitlab interface, but I like automatingit here as well. We need to provide the Gitlab project in terraform like this.If you don’t have a project yet, you should create it on gitlab.com and add theID here.


1
2
3

data "gitlab_project" "project-x" {
  id = [your-project-id]
}

Then we can use this reference to create the settings in our project:


1
2
3
4
5
6
7
8
9
10
11
12

resource "gitlab_project_cluster" "gitlab-kubernetes" {
  project = data.gitlab_project.project-x.id
  name = "my-awesome-cluster"
  domain = "[mydomain.com]"
  enabled = true
  kubernetes_api_url = digitalocean_kubernetes_cluster.cl01.endpoint
  kubernetes_token = digitalocean_kubernetes_cluster.cl01.kube_config[0].token
  kubernetes_ca_cert = base64decode(digitalocean_kubernetes_cluster.cl01.kube_config[0].cluster_ca_certificate)
  kubernetes_namespace = ""
  kubernetes_authorization_type = "rbac"
  environment_scope = "*"
}

And lastly we are not going the skip the database, as that is too easy to do.You could potentially setup a database persistent volume in kubernetes and havedb_pods spinup, but I think that it is much easier to have thedatabase as a separate service outside of kubernetes. The DigitalOcean manageddatabase service is a great option. You pay a little extra, but it takes care ofbackups etc. It is just one of those things that you don’t want to worry about,right?

Of course, we can easily add the creation of a database cluster to our terraformsetup. Note that I am using only 1 node for the cluster in this example. For actual failsafe production usage, you probably want to have at least 2 nodes.


1
2
3
4
5
6
7
8

resource "digitalocean_database_cluster" "postgres-db" {
  name = "postgres-db-cluster"
  engine = "pg"
  version = "11"
  size = "db-s-1vcpu-1gb"
  region = "ams3"
  node_count = 1
}

And while we are at it, let’s create the database and firewall settings as well.


1
2
3
4
5
6
7
8
9
10
11
12
13

resource "digitalocean_database_db" "test-prod" {
  cluster_id = digitalocean_database_cluster.postgres-db.id
  name = "test_prod"
}

resource "digitalocean_database_firewall" "db-fw" {
  cluster_id = digitalocean_database_cluster.postgres-db.id

  rule {
    type = "k8s"
    value = digitalocean_kubernetes_cluster.cl01.id
  }
}

This concludes our cluster setup. To get our app deployed we’ll have to add somesetup to our project. We’ll need to dockerize our project, add kubectl deployconfig files and add a gitlab-ci.yml that will trigger the gitlab-ci pipeline.

A project to deploy

Now we have our kubernetes cluster up-and-running it is time to see how we woulddeploy our app to the cluster.

I have a git tag prepared for you to use from my real_world_phoenix project. Youcan clone that checkout to use to deploy to the cluster. It has the necessaryfiles to trigger deployment which I’ll explain next.

Go ahead and clone the project from the tag I created:


1

git clone https://gitlab.com/drumusician/real_world_phoenix.git --branch kubernetes-deploy

Gitlab pipeline

To get our app deployed to kubernetes we’ll need to package it up in a containerand we’ll use Gitlab CI system for all of this. That means we’llprepare our app and package it using a 2-step dockerfile. The advantage of the2-step dockerfile method is that we can use a larger docker image to package ourapp that has all the packages for building and packaging our release and use a much slimmer container to just run our release. The release will be self-contained, so there is not a lot needed for it to run.

The steps we’ll need in our CI file are:

init # some compiling and pushing of artifacts
build and push # build our container and push to our registry
deploy # deploy to our kubernetes cluster

For pushing we’ll use the gitlab container registry that is available in everygitlab project. The deployment will be a collection of kubernetes yaml filesthat we can version control in our project.

Dockerfile


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54

# ---- Build Stage ----
FROM elixir:alpine AS app_builder

# Set environment variables for building the application
ENV MIX_ENV=prod \
    TEST=1 \
    LANG=C.UTF-8

RUN apk add --update git nodejs npm && \
    rm -rf /var/cache/apk/*

# Install hex and rebar
RUN mix local.hex --force && \
    mix local.rebar --force

# Create the application build directory
RUN mkdir /app
WORKDIR /app

# Copy over all the necessary application files and directories
COPY config ./config
COPY lib ./lib
COPY priv ./priv
COPY assets ./assets
COPY mix.exs .
COPY mix.lock .

# Fetch the application dependencies and build the application
RUN mix deps.get
RUN mix deps.compile
RUN npm run deploy --prefix ./assets
RUN mix phx.digest
RUN mix release

# ---- Application Stage ----
FROM alpine AS app

ENV LANG=C.UTF-8

# Install openssl
RUN apk add --update openssl ncurses-libs postgresql-client && \
    rm -rf /var/cache/apk/*

# Copy over the build artifact from the previous step and create a non root user
RUN adduser -D -h /home/app app
WORKDIR /home/app
COPY --from=app_builder /app/_build .
RUN chown -R app: ./prod
USER app

COPY entrypoint.sh .

# Run the Phoenix app
CMD ["./entrypoint.sh"]

Entrypoint

The entrypoint.sh is basically just a bash script that verifies the db is upand running before starting and running any pending migrations before startup aswell.


1
2
3
4
5
6
7
8
9
10
11
12
13

#!/bin/sh
# Docker entrypoint script.

# Wait until Postgres is ready
# while ! pg_isready -q -h $DB_HOST -p 5432 -U $DB_USER
# do
# echo "$(date) - waiting for database to start"
# sleep 2
# done

./prod/rel/real_world_phoenix/bin/real_world_phoenix eval RealWorldPhoenix.Release.migrate

./prod/rel/real_world_phoenix/bin/real_world_phoenix start

Gitlab CI Yaml file

To trigger the gitlab-ci pipeline we’ll need to add a .gitlab-ci.yml file inthe root of our project.

This is the content needed to initialize, build, push and deploy using the CIpipeline. We start off creating a few variables we’ll need and then define thestages. I have added some comments in the yaml file below that should explainthe steps in detail.

Note the use of Kaniko to build the container and push it to our registry. Kaniko doesn’t depend on a Docker daemon and executes each command within a Dockerfile completely in userspace. This enables building container images in environments that can’t easily or securely run a Docker daemon, such as a our Kubernetes cluster. Very nice!


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67

variables:
  REGISTRY: registry.gitlab.com
  CONTAINER_RELEASE_IMAGE: $REGISTRY/$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME:$CI_COMMIT_SHORT_SHA

stages:
  - init
  - build
  - deploy

# the dot syntax makes this a hidden step that we can include in other places further down.
.elixir_default: &elixir_default
  image: elixir:1.9
  before_script:
    - mix local.hex --force
    - mix local.rebar --force

.javascript_default: &javascript_default
  image: node:alpine
  before_script:
    - cd assets

# Compile our elixir artifacts
elixir_compile:
  <<: *elixir_default
  stage: init
  script:
    - mix deps.get --only test
    - mix compile
    - mix compile --warnings-as-errors
  artifacts:
    paths:
      - mix.lock
      - _build
      - deps

# compile javascript artifacts
javascript_deps:
  <<: *javascript_default
  stage: init
  script:
    - npm install --progress=false
  artifacts:
    paths:
      - assets/node_modules

# build our container image
build:
  stage: build
  image:
    name: gcr.io/kaniko-project/executor:debug
    entrypoint: [""]
  script:
    - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json
    - /kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile --destination $CONTAINER_RELEASE_IMAGE

# and deploy our kubernetes cluster
deploy:
  stage: deploy
  image:
    name: lwolf/kubectl_deployer:latest
  script:
    - cat deploy.yml | envsubst | kubectl apply -f -
    - cat service.yml | envsubst | kubectl apply -f -
    - cat ingress.yml | envsubst | kubectl apply -f -
  only:
    refs:
      - master

Yaml deployment files: service, ingress and deploy

The last thing we need is the kubernetes yaml files that are used in the cipipeline:

service.yml

The service pod running our application. This is basically our app running in a docker container.


1
2
3
4
5
6
7
8
9
10
11

apiVersion: v1
kind: Service
metadata:
  name: real-world-phoenix-service
spec:
  ports:
  - name: web
    port: 8001
    protocol: TCP
  selector:
    app: real_world_phoenix

ingress.yml

The mapping of our domain name to our service


1
2
3
4
5
6
7
8
9
10
11
12
13

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: real-world-phoenix-ingress
spec:
  rules:
  - host: $DOMAIN
    http:
      paths:
      - path: "/"
        backend:
          serviceName: real-world-phoenix-service
          servicePort: web

deploy.yml

Our deployment configuration. This provides Kubernetes the info about which container image to use and how many replicas of our app we want to have running.


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

apiVersion: apps/v1
kind: Deployment
metadata:
  name: real-world-phoenix
  labels:
    app: real_world_phoenix
spec:
  revisionHistoryLimit: 5
  replicas: 1
  selector:
    matchLabels:
      app: real_world_phoenix
  template:
    metadata:
      labels:
        app: real_world_phoenix
    spec:
      imagePullSecrets:
        - name: "gitlab.com"
      containers:
      - name: real-world-phoenix
        image: registry.gitlab.com/$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME:$CI_COMMIT_SHORT_SHA
        imagePullPolicy: Always
        name: real-world-phoenix-deployment
        resources: {}
        ports:
          - containerPort: 8001
        env:
          - name: SECRET_KEY_BASE
            value: $SECRET_KEY_BASE
          - name: DB_USER
            value: $DB_USER
          - name: DB_PASSWORD
            value: $DB_PASS
          - name: DB_NAME
            value: "test_prod"
          - name: DB_HOST
            value: $DB_CLUSTER
          - name: DB_PORT
            value: "25060"
          - name: APP_PORT
            value: "8001"
          - name: APP_HOSTNAME
            value: $DOMAIN

The above yaml file has references to a number of environment variables. The nice thing is that we can add all of these secrets inside of our Gitlab project and the deployment will use these values. In your project go to Settings -> CI/CD -> Variables.Now if we have all of this in our project all we have to do is commit and push it to our gitlab repo. Then Gitlab CI will do it’s magic and our application will be up and running after the pipeline jobs succeed. Nice!

Conclusion

While it is not necessarily the easiest setup, I really like the fact that once I have this setup I can fairly easily put some configuration in any of my gitlab projects and CI will take care of the rest, even the ssl certificate is handled for me so I don’t have to think about renewing them and all that stuff.

The reduction of manual work by using terraform is also a really nice benefit here. Terraform is a powerful tool, so use with care! If there is any interest in exploring terraform more, please leave a comment and I’ll plan in some more in-depth post exploring the world of terraform.

I hope this post was easy to follow and I hope that for those who followed along everything worked as expected? If not, do let me know, because I want to make sure everything here works as expected.

Until next time!

Real World Phoenix |> Let's D4Y |> using K8S

Tjaco Oostdijk — Tue, 21 Jan 2020 00:00:00 +0000

As promised in my last post, here is step 3 in my deployment adventure. The one
where I use Kubernetes!

In the last post we explored a very straightforward and simple
way to deploy a Phoenix application using the built-in Elixir releases on render.com.

Today I want to discuss a much more complicated setup, using Kubernetes. I know for
most applications you will surely not need a setup using Kubernetes, but I've
been having an urge to try it out for a while now and wanted to see if it is
really that complicated as I seem to be hearing through the grapevine. Who
knows, maybe it turns out it's actually not that hard at all.
Also I really like the fact that I can have a cluster to spin up nodes (ie.
applications) of all the experiments / ideas / apps that I'm currently playing
with, so I can actually expose those into the Real World with a reproducable and
easy deployment workflow. Let's see if we can make that dream come true!

The Kubernetes Cluster

If you are not aware of what Kubernetes is, head over to their website, as they have great documentation to get started on the basics.

To use Kubernetes for our deployment the first thing we'll need is a Kubernetes
cluster. This is the part where you see most tutorials and guides grab for a
local setup with minikube. While I like that it's possible to do this, it still
seems a bit too far from real life for me, so I wanted to have a bit more of a
production-like environment, like... for instance... a production environment :)

The beauty is that these days there are more and more managed Kubernetes
services that do all the hard work like setting up the cluster, managing
upgrades etc etc. I'll be using the service DigitalOcean is offering. Mainly because I
already have a lot of content running at DigitalOcean, so it is known territory
for me and they make it really easy to get started with Kubernetes.
If you haven't used DigitalOcean before, you can use this referral
link to create an account and get $100 credit
to spend in the first 60 days. So more than enough to follow along with this tutorial.

Terraform + Helm

We are going to need to use 2 more tools to get this setup up and running. Terraform and Helm. We'll use terraform to setup most of the infrastructure. In this way I don't have to remember all the tweaks and install steps I took to get everything up and running. I can just automate the whole setup declaratively and put the whole thing under version control.
This is often referred to these days as an infrastructure-as-code setup.
I wanted to use terraform for everything, but ran into some trouble setting up a
few of the tools needed, so reverted to Helm (the Kubernetes package manager) to
install Traefik and our Gitlab Runner.

Our Terraform setup will include:

Setting up a DigitalOcean Kubernetes Cluster with 2 worker nodes
Connecting a gitlab project to the kubernetes cluster
Creating a kubernetes_secret to be able to pull images from our private gitlab registry.
Setup a managed database cluster service @ digitalocean

And we'll use Helm to:

Install Traefik as an ingress controller
Create a Gitlab Runner in our cluster

So let's get crackin'!

Install doctl to talk to your DigitalOcean account

If we want to automate any of this stuff we'll have to be able to talk to
DigitalOcean programmatically, so installing doctl is step one. Please refer to
this guide to get that setup: doctl up and running
guide

After you have doctl installed, make sure you create a personal access token in DigitalOcean and add that in an environment variable and also initialize your
account.

export DIGITALOCEAN_TOKEN=[token]
doctl auth init -t $DIGITALOCEAN_TOKEN
doctl account get

Install Terraform

Of course there is a handy install guide for terraform also. Actually just an
executable, as it is written in go, so they have a handy packaged binary you can use.
See: terraform install

Setup our project

If you want to follow along with this guide, create a folder that will hold your
terraform configs and create a main.tf file that will hold the terraform declarations.

mkdir kube-terra
cd kube-terra
touch main.tf

Setup the DigitalOcean Kubernetes cluster

Terraform works in a declaritive manner, which means that we state what we want
the world to be like and terraform figures out how to get to that state. In
terraform you describe an item you want to exist as a resource. A Kubernetes
cluster is a form of a resource that is provided by one of the many providers
that exist in terraform. So let's create a DigitalOcean kubernetes cluster:

Creating a resource is always in the format: resource [kind] [reference] {}
We can use the reference we provide later on to refer back to this resource.

resource "digitalocean_tag" "kubernetes-cl01" {
  name = "kubernetes-cl01"
}

data "digitalocean_kubernetes_versions" "versions" {}

resource "digitalocean_kubernetes_cluster" "cl01" {
  name    = "cl01"
  region  = "ams3"
  version = data.digitalocean_kubernetes_versions.versions.latest_version

  node_pool {
    name       = "default"
    size       = "s-1vcpu-2gb"
    node_count = 2
    tags       = ["${digitalocean_tag.kubernetes-cl01.id}"]
  }
}

Now you can get your cluster setup by issuing the following commands in the folder you
created:

tarraform init
terraform apply

This will take about 5 minutes and then you'll have a fresh kubernetes cluster
up and running. Now that was easy!

Install Traefik

Kubernetes needs an ingress controller to route traffic from the outside world
to the services that are running inside the cluster as these services are not
exposed to the outside world, which is a good thing! Nginx is often used, but I went with Traefik, a very nice alternative that has a lot of nice functinality out of the box, like automatic ssl certificates using Let's Encrypt
and auto discovery of services running in the cluster. This setup also takes advantage of a DigitalOcean load balancer, which will automatically be created by the serviceType set below to LoadBalancer.

Create a file called traefik-values.yml in the root of your kubernetes config directory and add the following content. If you manage dns settings with DigitalOcean, you can comment out those parts as well. This will provide automatic ssl certificate generation. Very cool! If not you can just leave it like this. It'll work, but without ssl enabled. A benefit of using the dns-challenge as opposed to the more standard acme-challenge is that you can also get a valid certificate if your production system is behind a firewall.


image: traefik
dashboard:
  enabled: true
  domain: domain.example.com                     # put a (sub)domain here where you want to access the traefik dashboard
serviceType: LoadBalancer
rbac:
  enabled: true
# ssl:
#   enabled: true                                # Enables SSL
#   enforced: true                               # Redirects HTTP to HTTPS
# acme:
#   enabled: true                                # Enables Let's Encrypt certificates
#   staging: false                               # Use Lets Encrypt staging area for this example. For production purposes set this to false
#   email: info@drumusician.com                  # Email address that Let's Encrypt uses to notify about certificate expiry etc.
#   challengeType: "dns-01"
#   dnsProvider:
#     name:  digitalocean                        # This is why you need your domain to be under Digital Ocean control
#     digitalocean:
#       DO_AUTH_TOKEN: $DIGITALOCEAN_TOKEN
#   domains:
#     enabled: true
#     domainsList:
#       - main: domain.example.com               # domain that belongs to this certificate

Now we can go ahead and install traefik in our cluster:

Install helm:

brew install kubernetes-helm

Pull in our fresh cluster configuration locally (otherwise helm will
fail):

doctl kubernetes cluster kubeconfig save cl01

And install traefik:

# get the helm/stable charts
helm repo add stable https://kubernetes-charts.storage.googleapis.com/
# install Traefik
helm install traefik --values traefik-values.yml stable/traefik

After Traefik is installed we have enough running to be able to deploy our application
to the cluster. We are going to use gitlab-ci to deploy so we'll need to
install a gitlab-runner in our cluster first so we can use that in our project.

Creating a kubernetes secret to access our registry

If we install the Gitlab runner in the next step, we'll need a kubernetes secret
to be able to pull from our private registry in gitlab.
Go to https://gitlab.com/profile/personal_access_tokens and create an access
token to access the gitlab registry. Scopes: api, read_registry.

Then create a file at ~/.docker/docker-registry.json with the credentials for
the access token:

{
  "registry.gitlab.com": {
    "username": "",            # access-token-name
    "password": "",            # token
    "email": ""                # email of the gitlab account
  }
}

And then in our main.tf we can create the kubernetes secret.

resource "kubernetes_secret" "docker_pull_secret" {
  metadata {
    name = "gitlab.com"
  }

  data = {
    ".dockercfg" = file("~/.docker/docker-registry.json")
  }

  type = "kubernetes.io/dockercfg"
}

With that in place, we'll have to initialize the new provider and then we can add the secret:

terraform init
terraform apply

Install Gitlab Runner

To be able to deploy into our cluster from Gitlab CI we will need our runner to run inside of our cluster and have enough access rights to actually spin-up pods and services in our cluster. So we'll install that using helm as well. Gitlab has an install script from their interface, but that doesn't provide you with customisation options that we'll need for our use case.

Create the following file in your kubernetes dir: gitlab-runner-values.yml and
add this content:

gitlabUrl: https://gitlab.com/
runnerRegistrationToken: "" # copy this from your gitlab project settings: 
rbac:
  create: true
  clusterWideAccess: true
runners:
  privileged: true
imagePullSecrets: ["gitlab.com"]

Now let's install the runner:

helm install --namespace default gitlab-runner  -f gitlab-runner-values.yml gitlab/gitlab-runner

Connect a gitlab project to the cluster

This is something you can do through the Gitlab interface, but I like automating
it here as well. We need to provide the Gitlab project in terraform like this.
If you don't have a project yet, you should create it on gitlab.com and add the
ID here.

data "gitlab_project" "project-x" {
  id = [your-project-id]
}

Then we can use this reference to create the settings in our project:

resource "gitlab_project_cluster" "gitlab-kubernetes" {
  project                       = data.gitlab_project.project-x.id
  name                          = "my-awesome-cluster"
  domain                        = "[mydomain.com]"
  enabled                       = true
  kubernetes_api_url            = digitalocean_kubernetes_cluster.cl01.endpoint
  kubernetes_token              = digitalocean_kubernetes_cluster.cl01.kube_config[0].token
  kubernetes_ca_cert            = base64decode(digitalocean_kubernetes_cluster.cl01.kube_config[0].cluster_ca_certificate)
  kubernetes_namespace          = ""
  kubernetes_authorization_type = "rbac"
  environment_scope             = "*"
}

And lastly we are not going the skip the database, as that is too easy to do.
You could potentially setup a database persistent volume in kubernetes and have
db_pods spinup, but I think that it is much easier to have the
database as a separate service outside of kubernetes. The DigitalOcean managed
database service is a great option. You pay a little extra, but it takes care of
backups etc. It is just one of those things that you don't want to worry about,
right?

Of course, we can easily add the creation of a database cluster to our terraform
setup. Note that I am using only 1 node for the cluster in this example. For actual failsafe production usage, you probably want to have at least 2 nodes.

resource "digitalocean_database_cluster" "postgres-db" {
  name       = "postgres-db-cluster"
  engine     = "pg"
  version    = "11"
  size       = "db-s-1vcpu-1gb"
  region     = "ams3"
  node_count = 1
}

And while we are at it, let's create the database and firewall settings as well.

resource "digitalocean_database_db" "test-prod" {
  cluster_id = digitalocean_database_cluster.postgres-db.id
  name       = "test_prod"
}

resource "digitalocean_database_firewall" "db-fw" {
  cluster_id = digitalocean_database_cluster.postgres-db.id

  rule {
    type  = "k8s"
    value = digitalocean_kubernetes_cluster.cl01.id
  }
}

This concludes our cluster setup. To get our app deployed we'll have to add some
setup to our project. We'll need to dockerize our project, add kubectl deploy
config files and add a gitlab-ci.yml that will trigger the gitlab-ci pipeline.

A project to deploy

Now we have our kubernetes cluster up-and-running it is time to see how we would
deploy our app to the cluster.

I have a git tag prepared for you to use from my real_world_phoenix project. You
can clone that checkout to use to deploy to the cluster. It has the necessary
files to trigger deployment which I'll explain next.

Go ahead and clone the project from the tag I created:

git clone https://gitlab.com/drumusician/real_world_phoenix.git --branch kubernetes-deploy

Gitlab pipeline

To get our app deployed to kubernetes we'll need to package it up in a container
and we'll use Gitlab CI system for all of this. That means we'll
prepare our app and package it using a 2-step dockerfile. The advantage of the
2-step dockerfile method is that we can use a larger docker image to package our
app that has all the packages for building and packaging our release and use a much slimmer container to just run our release. The release will be self-contained, so there is not a lot needed for it to run.

The steps we'll need in our CI file are:

init # some compiling and pushing of artifacts
build and push # build our container and push to our registry
deploy # deploy to our kubernetes cluster

For pushing we'll use the gitlab container registry that is available in every
gitlab project. The deployment will be a collection of kubernetes yaml files
that we can version control in our project.

Dockerfile

# ---- Build Stage ----
FROM elixir:alpine AS app_builder

# Set environment variables for building the application
ENV MIX_ENV=prod \
    TEST=1 \
    LANG=C.UTF-8

RUN apk add --update git nodejs npm && \
    rm -rf /var/cache/apk/*

# Install hex and rebar
RUN mix local.hex --force && \
    mix local.rebar --force

# Create the application build directory
RUN mkdir /app
WORKDIR /app

# Copy over all the necessary application files and directories
COPY config ./config
COPY lib ./lib
COPY priv ./priv
COPY assets ./assets
COPY mix.exs .
COPY mix.lock .

# Fetch the application dependencies and build the application
RUN mix deps.get
RUN mix deps.compile
RUN npm run deploy --prefix ./assets
RUN mix phx.digest
RUN mix release

# ---- Application Stage ----
FROM alpine AS app

ENV LANG=C.UTF-8

# Install openssl
RUN apk add --update openssl ncurses-libs postgresql-client && \
    rm -rf /var/cache/apk/*

# Copy over the build artifact from the previous step and create a non root user
RUN adduser -D -h /home/app app
WORKDIR /home/app
COPY --from=app_builder /app/_build .
RUN chown -R app: ./prod
USER app

COPY entrypoint.sh .

# Run the Phoenix app
CMD ["./entrypoint.sh"]

Entrypoint

The entrypoint.sh is basically just a bash script that verifies the db is up
and running before starting and running any pending migrations before startup as
well.

#!/bin/sh
# Docker entrypoint script.

# Wait until Postgres is ready
# while ! pg_isready -q -h $DB_HOST -p 5432 -U $DB_USER
# do
#   echo "$(date) - waiting for database to start"
#   sleep 2
# done

./prod/rel/real_world_phoenix/bin/real_world_phoenix eval RealWorldPhoenix.Release.migrate

./prod/rel/real_world_phoenix/bin/real_world_phoenix start

Gitlab CI Yaml file

To trigger the gitlab-ci pipeline we'll need to add a .gitlab-ci.yml file in
the root of our project.

This is the content needed to initialize, build, push and deploy using the CI
pipeline. We start off creating a few variables we'll need and then define the
stages. I have added some comments in the yaml file below that should explain
the steps in detail.

Note the use of Kaniko to build the container and push it to our registry. Kaniko doesn't depend on a Docker daemon and executes each command within a Dockerfile completely in userspace. This enables building container images in environments that can't easily or securely run a Docker daemon, such as a our Kubernetes cluster. Very nice!

variables:
  REGISTRY: registry.gitlab.com
  CONTAINER_RELEASE_IMAGE: $REGISTRY/$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME:$CI_COMMIT_SHORT_SHA

stages:
  - init
  - build
  - deploy

# the dot syntax makes this a hidden step that we can include in other places further down.
.elixir_default: &elixir_default
  image: elixir:1.9
  before_script:
    - mix local.hex --force
    - mix local.rebar --force

.javascript_default: &javascript_default
  image: node:alpine
  before_script:
    - cd assets

# Compile our elixir artifacts
elixir_compile:
  <<: *elixir_default
  stage: init
  script:
    - mix deps.get --only test
    - mix compile
    - mix compile --warnings-as-errors
  artifacts:
    paths:
      - mix.lock
      - _build
      - deps

# compile javascript artifacts
javascript_deps:
  <<: *javascript_default
  stage: init
  script:
    - npm install --progress=false
  artifacts:
    paths:
      - assets/node_modules

# build our container image
build:
  stage: build
  image:
    name: gcr.io/kaniko-project/executor:debug
    entrypoint: [""]
  script:
    - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json
    - /kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile --destination $CONTAINER_RELEASE_IMAGE

# and deploy our kubernetes cluster
deploy:
  stage: deploy
  image:
    name: lwolf/kubectl_deployer:latest
  script:
    - cat deploy.yml | envsubst | kubectl apply -f -
    - cat service.yml | envsubst | kubectl apply -f -
    - cat ingress.yml | envsubst | kubectl apply -f -
  only:
    refs:
      - master

Yaml deployment files: service, ingress and deploy

The last thing we need is the kubernetes yaml files that are used in the ci
pipeline:

service.yml

The service pod running our application. This is basically our app running in a docker container.

apiVersion: v1
kind: Service
metadata:
  name: real-world-phoenix-service
spec:
  ports:
  - name: web
    port: 8001
    protocol: TCP
  selector:
    app: real_world_phoenix

ingress.yml

The mapping of our domain name to our service

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: real-world-phoenix-ingress
spec:
  rules:
  - host: $DOMAIN
    http:
      paths:
      - path: "/"
        backend:
          serviceName: real-world-phoenix-service
          servicePort: web

deploy.yml

Our deployment configuration. This provides Kubernetes the info about which container image to use and how many replicas of our app we want to have running.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: real-world-phoenix
  labels:
    app: real_world_phoenix
spec:
  revisionHistoryLimit: 5
  replicas: 1
  selector:
    matchLabels:
      app: real_world_phoenix
  template:
    metadata:
      labels:
        app: real_world_phoenix
    spec:
      imagePullSecrets:
        - name: "gitlab.com"
      containers:
      - name: real-world-phoenix
        image: registry.gitlab.com/$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME:$CI_COMMIT_SHORT_SHA
        imagePullPolicy: Always
        name: real-world-phoenix-deployment
        resources: {}
        ports:
          - containerPort: 8001
        env:
          - name: SECRET_KEY_BASE
            value: $SECRET_KEY_BASE
          - name: DB_USER
            value: $DB_USER
          - name: DB_PASSWORD
            value: $DB_PASS
          - name: DB_NAME
            value: "test_prod"
          - name: DB_HOST
            value: $DB_CLUSTER
          - name: DB_PORT
            value: "25060"
          - name: APP_PORT
            value: "8001"
          - name: APP_HOSTNAME
            value: $DOMAIN

The above yaml file has references to a number of environment variables. The nice thing is that we can add all of these secrets inside of our Gitlab project and the deployment will use these values. In your project go to Settings -> CI/CD -> Variables.
Now if we have all of this in our project all we have to do is commit and push it to our gitlab repo. Then Gitlab CI will do it's magic and our application will be up and running after the pipeline jobs succeed. Nice!

Conclusion

While it is not necessarily the easiest setup, I really like the fact that once I have this setup I can fairly easily put some configuration in any of my gitlab projects and CI will take care of the rest, even the ssl certificate is handled for me so I don't have to think about renewing them and all that stuff.

The reduction of manual work by using terraform is also a really nice benefit here. Terraform is a powerful tool, so use with care! If there is any interest in exploring terraform more, please leave a comment and I'll plan in some more in-depth post exploring the world of terraform.

I hope this post was easy to follow and I hope that for those who followed along everything worked as expected? If not, do let me know, because I want to make sure everything here works as expected.

Until next time!

When Elixir's performance becomes Rust-y

Tjaco Oostdijk — Tue, 14 Jan 2020 00:00:00 +0000

Elixir is a good language to develop fault tolerant and predictable, performant systems. However for heavy computational work it's less suitable. How can we keep using Elixir for the things it's good at, but achieve better performance for computational heavy work?

The post assumes basic Elixir knowledge. Rust knowledge is not required.

The use case

In this use case, we want to calculate the prime numbers in the range of 1 up to 1 million. I'd like to show off something a bit more computational intensive than a simple addition of numbers, and also show that Elixir data types map well into Rust.

The code passes a list of numbers instead of just an integer denoting the maximum, to show off that data structures such as lists also translate fine between Elixir and Rust. There are a lot more efficient algorithms available for determining prime numbers, but this approach is fine to show the performance difference.

Starting off with pure Elixir

So let's get coding! We'll start off with the Elixir implementation after making a new project. In this example the project is called rust_nif.

defmodule RustNif.ElixirPrimes do
  # http://erlang.org/doc/efficiency_guide/listHandling.html
  def prime_numbers(numbers) do
    prime_numbers(numbers, [])
  end

  defp prime_numbers([], result) do
    {:ok, result |> Enum.reverse() }
  end

  defp prime_numbers([number | rest], result) do
    new_result = result
    |> add_if_prime_number(number)

    prime_numbers(rest, new_result)
  end

  defp add_if_prime_number(numbers, 1), do: numbers

  defp add_if_prime_number(numbers, 2) do
    [2 | numbers]
  end

  defp add_if_prime_number(numbers, n) do
    range = 2..(n - 1)
    case Enum.any?(range, fn x -> rem(n, x) == 0 end) do
      false -> [n | numbers]
      _ -> numbers
    end
  end
end

time = Time.utc_now
result = Enum.into(1..100000, []) |> RustNif.ElixirPrimes.prime_numbers()
execution_time = Time.diff Time.utc_now, time
IO.puts "Elixir task finished after #{execution_time} seconds"
IO.inspect(result)

We prepend each number to the list, and reverse the list in the final result. See this section as to why we do this.
Running this, the results on my machine were:

{:ok,
 [2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71,
  73, 79, 83, 89, 97, 101, 103, 107, 109, 113, 127, 131, 137, 139, 149, 151,
  157, 163, 167, 173, 179, 181, 191, 193, 197, 199, 211, 223, ...]}
Elixir task finished after 33 seconds

That takes quite some time. Surely we can do better.

So what are our alternatives?

Ports & Nifs

We could resort to ports or NIFs, if you would want to improve this by writing it in another language.
There's a good summary of ports and nifs here: https://spin.atomicobject.com/2015/03/16/Elixir-native-interoperability-ports-vs-nifs/

In short:

Ports:

Safety
Error trapping
Flexible communication
No external Erlang/Elixir specific libraries required
Communication via STDIN and STDOUT

NIFS:

Fast and simple implementationn
No context switch required
Simple debugging
Not very safe (a bug in your NIF could crash the whole VM)
NIF-specific C libraries
Native implementation can leak into Elixir code

(Web) service

We could define a separate web service, exposing an API that could handle this, in a language better suited for computation heavy tasks. We'd need to add a new service to our stack though, and also account for potential network failures and sending potentially large payloads.

Background job

Depending on the case, it would be acceptable to move this computation into a background task (we don't need to return the immediate result to the end user). But in our scenario, this isn't the case

Summary

A NIF looks nice at a glance, but we'd need to eliminate the risk of the NIF being able to crash the beam VM. But what if I told you that we can do that?

Introducing Rust

This is where Rust comes in. Rust is a low level language, like C, developed by Mozilla. It's designed to provide better memory safety than C does, while also maintaining solid performance.

There's a library which allows creating Elixir/Erlang NIF's which make use of Rust, called rustler. One of its main features is that its safety should prevent the Beam from crashing if something unexpected happens. In the meanwhile, we get to leverage Rust's performance in our Elixir apps when we need it!

Writing the Rust code

Disclaimer: I don't have much prior knowledge with Rust, I'm sure this code can be improved. We have to start somewhere!

We'll go through the code piece by piece, the total solution will be posted at the end of the article.

The first thing we need to do is add rustler to the project, so let's do that!
Let's add {:rustler, "~> 0.21.0"}, to our dependencies in mix.exs and install it with mix deps.get.

That'll allow us to run mix rustler.new. Let's call our module PrimeNumbers (which will call the rust crate primenumbers by default, which is fine).

We'll also need to add rustler as compiler in the project, and define our Rust crate.

def project
  [
    compilers: [:rustler, :phoenix, :gettext] ++ Mix.compilers(),
    rustler_crates: [
      primenumbers: []
    ],
    ... # other project options
  ]

Let's start writing the Rust code! there will be a rust file at native/primenumbers/src/lib.rs, which looks like this currently:

use rustler::{Encoder, Env, Error, Term};

mod atoms {
    rustler_atoms! {
        atom ok;
        atom error;
        //atom __true__ = "true";
        //atom __false__ = "false";
    }
}

rustler::rustler_export_nifs! {
    "Elixir.PrimeNumbers",
    [
        ("add", 2, add)
    ],
    None
}

fn add<'a>(env: Env<'a>, args: &[Term<'a>]) -> Result<Term<'a>, Error> {
    let num1: i64 = args[0].decode()?;
    let num2: i64 = args[1].decode()?;

    Ok((atoms::ok(), num1 + num2).encode(env))
}

The first statement imports the Encoder, Env, Error and Term types from rustler, so we can make use of them.
The atoms section declares atoms which will map to Elixir's :ok annd :error atoms, so we can provide an interface similar to Elixir.

Methods defined in the rustler::rustler_export_nifs! statement are made available to Elixir, in the PrimeNumbers module. In the example, add is exported and defined with an arity of 2.

The add function decodes the arguments that are provided, adds them together and yields them back to Elixir as {:ok, <result>}. (Although I won't dive deeper into it, rustler makes this possible by implementing the External Term Format).

So let's modify that so we can get us some prime numbers.

use rustler::{Encoder, Env, Error, Term};

mod atoms {
    rustler::rustler_atoms! { // <- We actually prefixed this with rustler! it's a small oversight in the template generation. see https://github.com/rusterlium/rustler/issues/260
        atom ok;
        atom error;
    }
}

The first part remains the same, we'll declare the ok and error atoms for use with Elixir.

rustler::rustler_export_nifs! {
    "Elixir.RustNif.PrimeNumbers",
    [
        ("prime_numbers", 1, prime_numbers, rustler::SchedulerFlags::DirtyCpu)
    ],
    None
}

We'll register a prime_numbers method to rustler with an arity of 1 to be exported. This will be made available to Elixir.

One important addition we've made compared to the given example is to mark the function as being a 'Dirty NIF' by sending along SchedulerFlags::DirtyCpu. It's important to mark your NIF as 'Dirty' if their operation will take longer than a millisecond to complete. Not doing so could result in blocking your main operation in the Beam VM, potentially causing weird side effects.
More information about this behavior can be found in the Erlang documentation. (Thank you for this addition, Redmar Kerkhoff!)

fn is_prime_number(x: i64) -> bool {
    let end_of_range = x - 1;

    if x == 1 {
        false
    } else {
        !(2..end_of_range).any(|num| x % num == 0)
    }
}

We'll define a method to determine whether a number is a prime number. It accepts the number as argument and returns a boolean.

fn prime_numbers<'a>(env: Env<'a>, args: &[Term<'a>]) -> Result<Term<'a>, Error> {
    let is_list: bool = args[0].is_list();

    if !is_list {
        Ok((atoms::error(), "No list supplied").encode(env))
    } else {
        let numbers: Vec<i64> = args[0].decode()?;
        let mut vec = Vec::new();

        for number in numbers {
            if is_prime_number(number) {
                vec.push(number)
            }
        }

        Ok((atoms::ok(), &*vec).encode(env))
    }
}

We can use the same function signature that the example add function used (note the <'a> are lifetimes, not generics or similar).

In this case we will verify whether the first argument is a list, and stop the function and yield {:error, "No list suppled"} if it's not.

Afterwards, we use rustler's decode() method to convert the data type into a Vec of i64. We'll make a mutable Vec (you need to specify mutability explicitely in Rust), and loop through the numbers to determine the prime number. If a number is determined to be a prime number, we push it into the Vec.

Glueing Elixir and Rust together

We'll need a module in Elixir that will be the interface to the Rust code. The module name corresponds to the name known in the Rust code.

# prime_numbers.ex
defmodule RustNif.PrimeNumbers do
  use Rustler, otp_app: :rust_nif, crate: :primenumbers

  # Overriden when module is loaded
  def prime_numbers(_nums), do: :erlang.nif_error(:nif_not_loaded)
end

We define the OTP app we use, and the Rust crate we're loading, and that's it.

Comparison of performance

Now that we have both the Elixir and Rust(Elixir) versions of the code, let's compare them on their execution speed!

defmodule Benchmark do
  def benchmark do
    Process.spawn(fn -> benchmark_elixir() end, [:link])
    Process.spawn(fn -> benchmark_rust() end, [:link])
    nil
  end

  def benchmark_elixir do
    time = Time.utc_now
    result = Enum.into(1..100000, []) |> RustNif.ElixirPrimes.prime_numbers()
    execution_time = Time.diff Time.utc_now, time

    IO.inspect result
    IO.puts "Elixir task finished after #{execution_time} seconds"
  end

  def benchmark_rust do
    time = Time.utc_now
    result = Enum.into(1..100000, []) |> RustNif.PrimeNumbers.prime_numbers()
    execution_time = Time.diff Time.utc_now, time

    IO.inspect result
    IO.puts "Rust task finished after #{execution_time} seconds"
  end
end

iex(1)> Benchmark.benchmark
nil
{:ok,
 [2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71,
  73, 79, 83, 89, 97, 101, 103, 107, 109, 113, 127, 131, 137, 139, 149, 151,
  157, 163, 167, 173, 179, 181, 191, 193, 197, 199, 211, 223, ...]}
Rust task finished after 4 seconds
{:ok,
 [2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71,
  73, 79, 83, 89, 97, 101, 103, 107, 109, 113, 127, 131, 137, 139, 149, 151,
  157, 163, 167, 173, 179, 181, 191, 193, 197, 199, 211, 223, ...]}
Elixir task finished after 33 seconds

Well that's quite a lot better!

Do other people do this?

Yes! An example of a company having this setup is Discord, see Using Rust to Scale Elixir for 11 Million Concurrent Users

Conclusion

Let me preface the conclusion by stating you shouldn't just introduce Rust in your Elixir stack like this because you think you'll need it.

It's great to see the ease in which a low level, but safe language such as Rust, can be added to our stack to overcome one of Elixir's weaker points with relative ease.

I'm sure both implementations can be further refined. The point that I want to make is that it's easy to resort to a more performant language when need does arise.

The full code samples

# Elixir_primes.ex
defmodule RustNif.ElixirPrimes do
  # http://erlang.org/doc/efficiency_guide/listHandling.html
  def prime_numbers(numbers) do
    prime_numbers(numbers, [])
  end

  defp prime_numbers([], result) do
    {:ok, result |> Enum.reverse() }
  end

  defp prime_numbers([number | rest], result) do
    new_result = result
    |> add_if_prime_number(number)

    prime_numbers(rest, new_result)
  end

  defp add_if_prime_number(numbers, 1), do: numbers

  defp add_if_prime_number(numbers, 2) do
    [2 | numbers]
  end

  defp add_if_prime_number(numbers, n) do
    range = 2..(n - 1)
    case Enum.any?(range, fn x -> rem(n, x) == 0 end) do
      false -> [n | numbers]
      _ -> numbers
    end
  end
end

# prime_numbers.ex
defmodule RustNif.PrimeNumbers do
  use Rustler, otp_app: :rust_nif, crate: :primenumbers

  # Overriden when module is loaded
  def prime_numbers(_nums), do: :erlang.nif_error(:nif_not_loaded)
end

// lib.rs
use rustler::{Encoder, Env, Error, Term};

mod atoms {
    rustler::rustler_atoms! {
        atom ok;
        atom error;
    }
}

rustler::rustler_export_nifs! {
    "Elixir.RustNif.PrimeNumbers",
    [
        ("prime_numbers", 1, prime_numbers, rustler::SchedulerFlags::DirtyCpu)
    ],
    None
}

fn is_prime_number(x: i64) -> bool {
    let end_of_range = x - 1;

    if x == 1 {
        false
    } else {
        !(2..end_of_range).any(|num| x % num == 0)
    }
}

fn prime_numbers<'a>(env: Env<'a>, args: &[Term<'a>]) -> Result<Term<'a>, Error> {
    let is_list: bool = args[0].is_list();

    if !is_list {
        Ok((atoms::error(), "No list supplied").encode(env))
    } else {
        let numbers: Vec<i64> = args[0].decode()?;
        let mut vec = Vec::new();

        for number in numbers {
            if is_prime_number(number) {
                vec.push(number)
            }
        }

        Ok((atoms::ok(), &*vec).encode(env))
    }
}

When Elixir's performance becomes Rust-y

Tjaco Oostdijk — Tue, 14 Jan 2020 00:00:00 +0000

Elixir is a good language to develop fault tolerant and predictable, performant systems. However for heavy computational work it’s less suitable. How can we keep using Elixir for the things it’s good at, but achieve better performance for computational heavy work?

The post assumes basic Elixir knowledge. Rust knowledge is not required.

The use case

In this use case, we want to calculate the prime numbers in the range of 1 up to 1 million. I’d like to show off something a bit more computational intensive than a simple addition of numbers, and also show that Elixir data types map well into Rust.

Starting off with pure Elixir

So let’s get coding! We’ll start off with the Elixir implementation after making a new project. In this example the project is called rust_nif.


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

defmodule RustNif.ElixirPrimes do
  # http://erlang.org/doc/efficiency_guide/listHandling.html
  def prime_numbers(numbers) do
    prime_numbers(numbers, [])
  end

  defp prime_numbers([], result) do
    {:ok, result |> Enum.reverse() }
  end

  defp prime_numbers([number | rest], result) do
    new_result = result
    |> add_if_prime_number(number)

    prime_numbers(rest, new_result)
  end

  defp add_if_prime_number(numbers, 1), do: numbers

  defp add_if_prime_number(numbers, 2) do
    [2 | numbers]
  end

  defp add_if_prime_number(numbers, n) do
    range = 2..(n - 1)
    case Enum.any?(range, fn x -> rem(n, x) == 0 end) do
      false -> [n | numbers]
      _ -> numbers
    end
  end
end

time = Time.utc_now
result = Enum.into(1..100000, []) |> RustNif.ElixirPrimes.prime_numbers()
execution_time = Time.diff Time.utc_now, time
IO.puts "Elixir task finished after #{execution_time} seconds"
IO.inspect(result)

We prepend each number to the list, and reverse the list in the final result. See this section as to why we do this.Running this, the results on my machine were:


1
2
3
4
5

{:ok,
 [2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71,
  73, 79, 83, 89, 97, 101, 103, 107, 109, 113, 127, 131, 137, 139, 149, 151,
  157, 163, 167, 173, 179, 181, 191, 193, 197, 199, 211, 223, ...]}
Elixir task finished after 33 seconds

That takes quite some time. Surely we can do better.

So what are our alternatives?

Ports & Nifs

We could resort to ports or NIFs, if you would want to improve this by writing it in another language.There’s a good summary of ports and nifs here: https://spin.atomicobject.com/2015/03/16/Elixir-native-interoperability-ports-vs-nifs/

In short:

Ports:

Safety
Error trapping
Flexible communication
No external Erlang/Elixir specific libraries required
Communication via STDIN and STDOUT

NIFS:

Fast and simple implementationn
No context switch required
Simple debugging
Not very safe (a bug in your NIF could crash the whole VM)
NIF-specific C libraries
Native implementation can leak into Elixir code

(Web) service

We could define a separate web service, exposing an API that could handle this, in a language better suited for computation heavy tasks. We’d need to add a new service to our stack though, and also account for potential network failures and sending potentially large payloads.

Background job

Depending on the case, it would be acceptable to move this computation into a background task (we don’t need to return the immediate result to the end user). But in our scenario, this isn’t the case

Summary

A NIF looks nice at a glance, but we’d need to eliminate the risk of the NIF being able to crash the beam VM. But what if I told you that we can do that?

Introducing Rust

This is where Rust comes in. Rust is a low level language, like C, developed by Mozilla. It’s designed to provide better memory safety than C does, while also maintaining solid performance.

There’s a library which allows creating Elixir/Erlang NIF’s which make use of Rust, called rustler. One of its main features is that its safety should prevent the Beam from crashing if something unexpected happens. In the meanwhile, we get to leverage Rust’s performance in our Elixir apps when we need it!

Writing the Rust code

Disclaimer: I don’t have much prior knowledge with Rust, I’m sure this code can be improved. We have to start somewhere!

We’ll go through the code piece by piece, the total solution will be posted at the end of the article.

The first thing we need to do is add rustler to the project, so let’s do that!Let’s add {:rustler, "~> 0.21.0"}, to our dependencies in mix.exs and install it with mix deps.get.

That’ll allow us to run mix rustler.new. Let’s call our module PrimeNumbers (which will call the rust crate primenumbers by default, which is fine).

We’ll also need to add rustler as compiler in the project, and define our Rust crate.


1
2
3
4
5
6
7
8

def project
  [
    compilers: [:rustler, :phoenix, :gettext] ++ Mix.compilers(),
    rustler_crates: [
      primenumbers: []
    ],
    ... # other project options
  ]

Let’s start writing the Rust code! there will be a rust file at native/primenumbers/src/lib.rs, which looks like this currently:


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

use rustler::{Encoder, Env, Error, Term};

mod atoms {
    rustler_atoms! {
        atom ok;
        atom error;
        //atom __true__ = "true";
        //atom __false__ = "false";
    }
}

rustler::rustler_export_nifs! {
    "Elixir.PrimeNumbers",
    [
        ("add", 2, add)
    ],
    None
}

fn add<'a>(env: Env<'a>, args: &[Term<'a>]) -> Result<Term<'a>, Error> {
    let num1: i64 = args[0].decode()?;
    let num2: i64 = args[1].decode()?;

    Ok((atoms::ok(), num1 + num2).encode(env))
}

The first statement imports the Encoder, Env, Error and Term types from rustler, so we can make use of them.The atoms section declares atoms which will map to Elixir’s :ok annd :error atoms, so we can provide an interface similar to Elixir.

Methods defined in the rustler::rustler_export_nifs! statement are made available to Elixir, in the PrimeNumbers module. In the example, add is exported and defined with an arity of 2.

The add function decodes the arguments that are provided, adds them together and yields them back to Elixir as {:ok, <result>}. (Although I won’t dive deeper into it, rustler makes this possible by implementing the External Term Format).

So let’s modify that so we can get us some prime numbers.


1
2
3
4
5
6
7
8

use rustler::{Encoder, Env, Error, Term};

mod atoms {
    rustler::rustler_atoms! { // <- We actually prefixed this with rustler! it's a small oversight in the template generation. see https://github.com/rusterlium/rustler/issues/260
        atom ok;
        atom error;
    }
}

The first part remains the same, we’ll declare the ok and error atoms for use with Elixir.


1
2
3
4
5
6
7

rustler::rustler_export_nifs! {
    "Elixir.RustNif.PrimeNumbers",
    [
        ("prime_numbers", 1, prime_numbers, rustler::SchedulerFlags::DirtyCpu)
    ],
    None
}

We’ll register a prime_numbers method to rustler with an arity of 1 to be exported. This will be made available to Elixir.

One important addition we’ve made compared to the given example is to mark the function as being a ‘Dirty NIF’ by sending along SchedulerFlags::DirtyCpu. It’s important to mark your NIF as ‘Dirty’ if their operation will take longer than a millisecond to complete. Not doing so could result in blocking your main operation in the Beam VM, potentially causing weird side effects.More information about this behavior can be found in the Erlang documentation. (Thank you for this addition, Redmar Kerkhoff!)


1
2
3
4
5
6
7
8
9

fn is_prime_number(x: i64) -> bool {
    let end_of_range = x - 1;

    if x == 1 {
        false
    } else {
        !(2..end_of_range).any(|num| x % num == 0)
    }
}

We’ll define a method to determine whether a number is a prime number. It accepts the number as argument and returns a boolean.


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

fn prime_numbers<'a>(env: Env<'a>, args: &[Term<'a>]) -> Result<Term<'a>, Error> {
    let is_list: bool = args[0].is_list();

    if !is_list {
        Ok((atoms::error(), "No list supplied").encode(env))
    } else {
        let numbers: Vec<i64> = args[0].decode()?;
        let mut vec = Vec::new();

        for number in numbers {
            if is_prime_number(number) {
                vec.push(number)
            }
        }

        Ok((atoms::ok(), &*vec).encode(env))
    }
}

We can use the same function signature that the example add function used (note the <’a> are lifetimes, not generics or similar).

In this case we will verify whether the first argument is a list, and stop the function and yield {:error, "No list suppled"} if it’s not.

Afterwards, we use rustler’s decode() method to convert the data type into a Vec of i64. We’ll make a mutable Vec (you need to specify mutability explicitely in Rust), and loop through the numbers to determine the prime number. If a number is determined to be a prime number, we push it into the Vec.

Glueing Elixir and Rust together

We’ll need a module in Elixir that will be the interface to the Rust code. The module name corresponds to the name known in the Rust code.


1
2
3
4
5
6
7

# prime_numbers.ex
defmodule RustNif.PrimeNumbers do
  use Rustler, otp_app: :rust_nif, crate: :primenumbers

  # Overriden when module is loaded
  def prime_numbers(_nums), do: :erlang.nif_error(:nif_not_loaded)
end

We define the OTP app we use, and the Rust crate we’re loading, and that’s it.

Comparison of performance

Now that we have both the Elixir and Rust(Elixir) versions of the code, let’s compare them on their execution speed!


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

defmodule Benchmark do
  def benchmark do
    Process.spawn(fn -> benchmark_elixir() end, [:link])
    Process.spawn(fn -> benchmark_rust() end, [:link])
    nil
  end

  def benchmark_elixir do
    time = Time.utc_now
    result = Enum.into(1..100000, []) |> RustNif.ElixirPrimes.prime_numbers()
    execution_time = Time.diff Time.utc_now, time

    IO.inspect result
    IO.puts "Elixir task finished after #{execution_time} seconds"
  end

  def benchmark_rust do
    time = Time.utc_now
    result = Enum.into(1..100000, []) |> RustNif.PrimeNumbers.prime_numbers()
    execution_time = Time.diff Time.utc_now, time

    IO.inspect result
    IO.puts "Rust task finished after #{execution_time} seconds"
  end
end

iex(1)> Benchmark.benchmark
nil
{:ok,
 [2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71,
  73, 79, 83, 89, 97, 101, 103, 107, 109, 113, 127, 131, 137, 139, 149, 151,
  157, 163, 167, 173, 179, 181, 191, 193, 197, 199, 211, 223, ...]}
Rust task finished after 4 seconds
{:ok,
 [2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71,
  73, 79, 83, 89, 97, 101, 103, 107, 109, 113, 127, 131, 137, 139, 149, 151,
  157, 163, 167, 173, 179, 181, 191, 193, 197, 199, 211, 223, ...]}
Elixir task finished after 33 seconds

Well that’s quite a lot better!

Do other people do this?

Yes! An example of a company having this setup is Discord, see Using Rust to Scale Elixir for 11 Million Concurrent Users

Conclusion

Let me preface the conclusion by stating you shouldn’t just introduce Rust in your Elixir stack like this because you think you’ll need it.

It’s great to see the ease in which a low level, but safe language such as Rust, can be added to our stack to overcome one of Elixir’s weaker points with relative ease.

I’m sure both implementations can be further refined. The point that I want to make is that it’s easy to resort to a more performant language when need does arise.

The full code samples


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

# Elixir_primes.ex
defmodule RustNif.ElixirPrimes do
  # http://erlang.org/doc/efficiency_guide/listHandling.html
  def prime_numbers(numbers) do
    prime_numbers(numbers, [])
  end

  defp prime_numbers([], result) do
    {:ok, result |> Enum.reverse() }
  end

  defp prime_numbers([number | rest], result) do
    new_result = result
    |> add_if_prime_number(number)

    prime_numbers(rest, new_result)
  end

  defp add_if_prime_number(numbers, 1), do: numbers

  defp add_if_prime_number(numbers, 2) do
    [2 | numbers]
  end

  defp add_if_prime_number(numbers, n) do
    range = 2..(n - 1)
    case Enum.any?(range, fn x -> rem(n, x) == 0 end) do
      false -> [n | numbers]
      _ -> numbers
    end
  end
end



1
2
3
4
5
6
7

# prime_numbers.ex
defmodule RustNif.PrimeNumbers do
  use Rustler, otp_app: :rust_nif, crate: :primenumbers

  # Overriden when module is loaded
  def prime_numbers(_nums), do: :erlang.nif_error(:nif_not_loaded)
end



1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46

// lib.rs
use rustler::{Encoder, Env, Error, Term};

mod atoms {
    rustler::rustler_atoms! {
        atom ok;
        atom error;
    }
}

rustler::rustler_export_nifs! {
    "Elixir.RustNif.PrimeNumbers",
    [
        ("prime_numbers", 1, prime_numbers, rustler::SchedulerFlags::DirtyCpu)
    ],
    None
}

fn is_prime_number(x: i64) -> bool {
    let end_of_range = x - 1;

    if x == 1 {
        false
    } else {
        !(2..end_of_range).any(|num| x % num == 0)
    }
}

fn prime_numbers<'a>(env: Env<'a>, args: &[Term<'a>]) -> Result<Term<'a>, Error> {
    let is_list: bool = args[0].is_list();

    if !is_list {
        Ok((atoms::error(), "No list supplied").encode(env))
    } else {
        let numbers: Vec<i64> = args[0].decode()?;
        let mut vec = Vec::new();

        for number in numbers {
            if is_prime_number(number) {
                vec.push(number)
            }
        }

        Ok((atoms::ok(), &*vec).encode(env))
    }
}

Real World Phoenix |> Let's d4y

Tjaco Oostdijk — Tue, 31 Dec 2019 00:00:00 +0000

I believe there is a strong case to be made for shipping your app / idea as soon as possible as this pushes you to think about creating something small that is functional and finished.

So today I want to cover a topic that has been discussed a lot in the community and there seems to be a tendency in Elixir land that they are hard. Let’s discuss deployments! The fact that it is often categorised as difficult or hard seems to scare people away and confuses a lot of people. I’m here to tell you that deployments are not hard, once you know your options.

Releases - yes you should use them!

There are basically two ways to run your elixir app in production. One is using mix and the other is using releases. I want to focus today on using releases as using a build tool to deploy and run your app seems silly… And I think demistifying all the fuss around releases will make sure we get more comfortable using them.

We’ll take 3 steps in learning about deployment. First we’ll package and run our release locally to test out how releases work, next we’ll take one of the easiest platforms I’ve seen to deploy our app, render.com, and lastly we’ll go the other deep end and run our app in a kubernetes cluster!

In this post we’ll cover steps 1 and 2 and I’m going to leave the kubernetes way for the next blog post as that needs some more indepth explaination. You might think that I’m nuts going the kubernetes route, but I wanted to play around with it and see if it is really that hard to grasp… tl;dr => It really isn’t nowadays due to all the managed kubernetes services that handle a lot of the hard stuff behind the scenes.

Baking and running a release locally

After all the hard work that has been done on distillery by Paul Schoenfelder (a.k.a. bitwalker), Elixir has embraced releases as part of core. So no more excuses not to use releases! And ofcourse Phoenix has gone along with this trend and is enabling deployment with releases out of the box.

So let’s see how easy it is to get this up and running. A lot of guides out there omit the database part, but I’ll add that here so we know how to enable that as well. For production we’ll set a DATABASE_URL environment variable as that is most convenient to use when deploying. We will also set the SECRET_KEY_BASE as an environment variable. Locally we’ll test our release in development mode, because setting a local release to production mode requires all kinds of extra settings that we really don’t need to test this out.

If you want to try this out while reading this, you can create a fresh Phoenix app and follow along:


1
2

mix archive.install hex phx_new 1.4.11
mix phx.new test_deployment

Now let’s take this step by step:

setup a releases.exs file


1

mv config/prod.secret.exs config/releases.exs

change use Mix.Config to import Config in releases.exs


1

sed -i'' -e 's/use Mix.Config/import Config/g' config/releases.exs

set server: true in config/releases.exs


1

sed -i'' -e 's/^#.*config :test_deployment, TestDeploymentWeb.Endpoint, server: true/config :test_deployment, TestDeploymentWeb.Endpoint, server: true/g' config/releases.exs

remove import_config prod.secret.exs from config/prod.exs


1

sed -i'' -e 's/import_config "prod.secret.exs"//g' config/prod.exs

create our database locally (so we can actually test it with a database). Note that we create the development db with this command


1

mix ecto.create

set environment variables


1
2

export SECRET_KEY_BASE=$(mix phx.gen.secret)
export DATABASE_URL=ecto://postgres:postgres@localhost/test_deployment_dev

prepare our app


1
2
3
4
5
6
7

# fetch dependencies
mix deps.get --only prod
MIX_ENV=prod mix compile

# compile assets
npm run deploy --prefix ./assets
mix phx.digest

build our release


1

MIX_ENV=prod mix release

run it!


1

_build/prod/rel/test_deployment/bin/test_deployment start

After that visit http://localhost:4000 and see your fresh release shine!

Deploying a release to render.com

Now let’s see what it takes to put this live on render.com. If you are following along, make sure you have the repo you want to use on gitlab or github so you can pull that in easily.

Creating an account on render.com is easy and deploying and running your app is charged by the second, so you can safely follow along this turorial and shut the app down once done so you have an idea of the workflow involved in getting your app live on render.com. Note that signing up with github or gitlab gives you the extra benefit of being able to select the repo to deploy right from render, Which makes it even easier!

So go to https://dashboard.render.com/select-repo?type=web and select the repository you want to deploy. After that fill in the neccessary details. So choose a descriptive name and set the environment to Elixir. You’ll notice that the default settings to deploy here are mix phx.digest and mix phx.server.In our case we don’t want to use that, because we are using releases. To use releases in render we have to change these two settings.

We’ll set the build command to: build.sh, which we will create in our repo shortly. The start command will be the command we used when testing the release locally. In our case this would be: _build/prod/rel/test_deployment/bin/test_deployment start. There are two more settings needed in the advanced setting. First one is the SECRET_KEY_BASE environment variable(you can just generate one with mix phx.gen.secret and paste that in here) and the other one is the database url. We’ll need to create the database first for this.

Setting up the database

To set up a database for our app go to https://dashboard.render.com/new/database and create a database with the settings you prefer. After creating the databaseyou can go into the settings page and copy the External connection string. This string should go into the advanced setting of your app as DATABASE_URL (exactly as we did when testing it locally).

Build script

The build script gives us all the power to customise building our app when deploying. In our case this means entering the commands that will be run to prepare our app for deployment.

This is the content of our build.sh file. As you can see we are basically addingthe steps we used when creating the release on our local machine.


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

#!/usr/bin/env elixir

# Initial setup
mix deps.get --only prod
MIX_ENV=prod mix compile

# Compile assets
npm install --prefix ./assets
npm run deploy --prefix ./assets
mix phx.digest

# Remove the existing release directory and build the release
rm -rf "_build"
MIX_ENV=prod mix release

# for auto DB migration upon deploy
MIX_ENV=prod mix ecto.migrate

After creating the build.sh file, make sure to make it executable:


1

chmod a+x build.sh

And….. deploy!

Once we have the database up and running and the environment variables setup we can proceed to deploy. And the beauty here is that once you push to master with the build.sh included it will automagically trigger a deploy on render.com

Custom domain

Once you have your app deployed, you’ll get a url generated by render to access your app. However it is fairly easy to add your custom domain in your app settings.

Alright, hope this was helpful and I certainly hope that this illustrates how straightforward it is to use releases to deploy Elixir / Phoenix apps. As promised in the next post I’ll cover a deployment setup that is a bit more complex using kubernetes. That is actually also the setup that I’ve used to deploy realworldphoenix.com, so more good stuff is coming in 2020!

Happy New Year to all of you and may 2020 be the year that Elixir shines for us all!

Until next time!

Real World Phoenix |> Let's Send Some Emails

Tjaco Oostdijk — Tue, 24 Dec 2019 00:00:00 +0000

Sending emails is something probably almost all apps will have to be able to do at some point in time. So how do we go about doing such a thing with Phoenix and what libraries can we use for this, if any.

The mindset in the Elixir ecosystem is not to make use of a library right off the bat. Since setting up and sending emails is a common task that takes quite some effort to take from scratch, a library is very welcome. Luckily we have a few very good options in Elixir land. As a side note, if you want to see what options you have for libraries in certain categories, Elixir libhunt is a very nice place to investigate the stats for some similar libraries. It is like the Ruby Toolbox, if you are coming from ruby. You can check the comparison of some popular email sending libraries here.

We are not going for the number one in this case(gen_smtp) as, although it looks really nice, it’s a bit too low level OTP for our use case. The two main ones that work well with Phoenix are Swoosh and Bamboo. They are very similar and today I’m going with the one created by Thoughtbot, Bamboo, mainly because I’ve used that before. However there is little difference between the two and both actually support most transactional email providers out of the box. For the sake of this tutorial I’ll use a free account setup with sendgrid. Sendgrid provides a generous trial period and also have a pretty extensive free number of emails you can send every month. So more than enough for some simple experiments.

Ok, so let’s get this show on the road. My main goal for this post is to let my app send a welcome mail for new signups of my app. The app is a student management app for music teachers, and there is a signup form that is built using LiveView that is used to sign up either as a teacher or as a student. The authentication mechanism is built using Pow.

Let’s start by adding Bamboo to our mix project.


1
2
3
4
5
6

# mix.exs
...
  def deps do
    [{:bamboo, "~> 1.3"}]
  end
...

And make sure we pull it in:


1

$ mix deps.get

We’ll need a mailer module for sending the actual emails and one or more email modules for creating emails to be sent. We’ll start with the mailer module:


1
2
3

defmodule StudentManager.Mailer do
  use Bamboo.Mailer, otp_app: :student_manager
end

And here is an example email module for crafting your emails. We create a base_email/0 function for some general options and two functions for a teacher and student welcome email respectively. This should all be fairly self-explanatory.


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

defmodule StudentManager.Email do
  use Bamboo.Phoenix, view: StudentManagerWeb.EmailView

  def welcome_email(%{roles: ["teacher"]} = user) do
    base_email()
    |> to(user.email)
    |> subject("Welcome to StudMan App")
    |> assign(:user, user)
    |> render("welcome_teacher.html")
  end

  def welcome_email(%{roles: ["student"]} = user) do
    base_email()
    |> to(user.email)
    |> subject("Welcome to StudMan App")
    |> assign(:user, user)
    |> render("welcome_student.html")
  end

  defp base_email() do
    new_email()
    |> from("support@studman.nl")
  end
end

If you look closely you’ll notice I’m using pattern matching to send out a student or teacher mail based on the role that gets assigned when they sign up. I’m also using Bamboo.Phoenix here in order to take advantage of the template engine included in Phoenix. This really makes it a breeze to create html emails. Normally you would configure this to also send out a plain text version, but for now we’ll just create the html version.

One last thing we need to do is add some configuration for our mailer in config/config.exs.


1
2
3

config :student_manager, StudentManager.Mailer,
  adapter: Bamboo.SendGridAdapter,
  api_key: "my_sendgrid_api_key"

So all very straightforward stuff. Cool! Just following the guides from Bamboo should get you up and running quickly. We can now add sending of the welcome email in the signup process we created in the last blogpost. Remember from last post that we used LiveView for our registration form? Because of that, it’s easy to add our email in the signup flow like this:


1
2
3
4
5
6
7
8
9
10
11
12
13
14

  def handle_event("save", %{"user" => user_params}, socket) do
    case create_user(socket, user_params) do
      {:ok, user} ->
        Email.welcome_email(user) |> Mailer.deliver_later()

        {:stop,
         socket
         |> put_flash(:info, "user created")
         |> redirect(to: "/")}

      {:error, %Ecto.Changeset{} = changeset} ->
        {:noreply, assign(socket, changeset: changeset)}
    end
  end

The fact that I’m using LiveView here makes it pretty easy to hook into the registration process. If we were using the regular sign-up process of Pow we would need to do a little more work to get this email delivered. Let’s see how that would work. Luckily the author of Pow also has us covered. The library provides callback hooks which you can use to hook into the sign-up process in certain spots. In our case, the after_create hook for the registration_controller is the hook we want to grab.

Lets’s create a file for our controller_callbacks: student_manager_web/pow/callbacks_controller.ex with the following contents:


1
2
3
4
5
6
7
8
9
10
11
12
13

defmodule StudentManagerWeb.Pow.CallbacksController do
  @moduledoc false

  use Pow.Extension.Phoenix.ControllerCallbacks.Base

  def before_respond(Pow.Phoenix.RegistrationController, :create, {:ok, user, conn}, _config) do
    user
    |> StudentManager.Email.welcome_email()
    |> StudentManager.Mailer.deliver_later()

    {:ok, user, conn}
  end
end

Now to be able to trigger these callbacks, we’ll need to add this in our pow config in config/config.exs


1
2
3
4
5
6

# Pow Configuration
config :student_manager, :pow,
  user: StudentManager.Accounts.User,
  repo: StudentManager.Repo,
  web_module: StudentManagerWeb,
  controller_callbacks: StudentManagerWeb.Pow.ControllerCallbacks

That should do it for sending out a welcoming email on signup. Hope this was helpful!

Until next time!

Real World Phoenix |> Let's Send Some Emails

Tjaco Oostdijk — Tue, 24 Dec 2019 00:00:00 +0000

We are not going for the number one in this case(gen_smtp) as, although it looks really nice, it's a bit too low level OTP for our use case. The two main ones that work well with Phoenix are Swoosh and Bamboo. They are very similar and today I'm going with the one created by Thoughtbot, Bamboo, mainly because I've used that before. However there is little difference between the two and both actually support most transactional email providers out of the box. For the sake of this tutorial I'll use a free account setup with sendgrid. Sendgrid provides a generous trial period and also have a pretty extensive free number of emails you can send every month. So more than enough for some simple experiments.

Ok, so let's get this show on the road. My main goal for this post is to let my app send a welcome mail for new signups of my app. The app is a student management app for music teachers, and there is a signup form that is built using LiveView that is used to sign up either as a teacher or as a student. The authentication mechanism is built using Pow.

Let's start by adding Bamboo to our mix project.

# mix.exs
...
  def deps do
    [{:bamboo, "~> 1.3"}]
  end
...

And make sure we pull it in:

$ mix deps.get

We'll need a mailer module for sending the actual emails and one or more email modules for creating emails to be sent. We'll start with the mailer module:

defmodule StudentManager.Mailer do
  use Bamboo.Mailer, otp_app: :student_manager
end

defmodule StudentManager.Email do
  use Bamboo.Phoenix, view: StudentManagerWeb.EmailView

  def welcome_email(%{roles: ["teacher"]} = user) do
    base_email()
    |> to(user.email)
    |> subject("Welcome to StudMan App")
    |> assign(:user, user)
    |> render("welcome_teacher.html")
  end

  def welcome_email(%{roles: ["student"]} = user) do
    base_email()
    |> to(user.email)
    |> subject("Welcome to StudMan App")
    |> assign(:user, user)
    |> render("welcome_student.html")
  end

  defp base_email() do
    new_email()
    |> from("support@studman.nl")
  end
end

If you look closely you'll notice I'm using pattern matching to send out a student or teacher mail based on the role that gets assigned when they sign up. I'm also using Bamboo.Phoenix here in order to take advantage of the template engine included in Phoenix. This really makes it a breeze to create html emails. Normally you would configure this to also send out a plain text version, but for now we'll just create the html version.

One last thing we need to do is add some configuration for our mailer in config/config.exs.

config :student_manager, StudentManager.Mailer,
  adapter: Bamboo.SendGridAdapter,
  api_key: "my_sendgrid_api_key"

So all very straightforward stuff. Cool! Just following the guides from Bamboo should get you up and running quickly. We can now add sending of the welcome email in the signup process we created in the last blogpost. Remember from last post that we used LiveView for our registration form? Because of that, it's easy to add our email in the signup flow like this:

  def handle_event("save", %{"user" => user_params}, socket) do
    case create_user(socket, user_params) do
      {:ok, user} ->
        Email.welcome_email(user) |> Mailer.deliver_later()

        {:stop,
         socket
         |> put_flash(:info, "user created")
         |> redirect(to: "/")}

      {:error, %Ecto.Changeset{} = changeset} ->
        {:noreply, assign(socket, changeset: changeset)}
    end
  end

The fact that I'm using LiveView here makes it pretty easy to hook into the registration process. If we were using the regular sign-up process of Pow we would need to do a little more work to get this email delivered. Let's see how that would work. Luckily the author of Pow also has us covered. The library provides callback hooks which you can use to hook into the sign-up process in certain spots. In our case, the after_create hook for the registration_controller is the hook we want to grab.

Lets's create a file for our controller_callbacks: student_manager_web/pow/callbacks_controller.ex with the following contents:

defmodule StudentManagerWeb.Pow.CallbacksController do
  @moduledoc false

  use Pow.Extension.Phoenix.ControllerCallbacks.Base

  def before_respond(Pow.Phoenix.RegistrationController, :create, {:ok, user, conn}, _config) do
    user
    |> StudentManager.Email.welcome_email()
    |> StudentManager.Mailer.deliver_later()

    {:ok, user, conn}
  end
end

Now to be able to trigger these callbacks, we'll need to add this in our pow config in config/config.exs

# Pow Configuration
config :student_manager, :pow,
  user: StudentManager.Accounts.User,
  repo: StudentManager.Repo,
  web_module: StudentManagerWeb,
  controller_callbacks: StudentManagerWeb.Pow.ControllerCallbacks

That should do it for sending out a welcoming email on signup. Hope this was helpful!

Until next time!

DEV Community: Tjaco Oostdijk

Extending Flutter Driver with custom commands

Extending Flutter Driver

Using Phoenix Presence in LiveView |> A Simple Example

Who's reading?

LiveView or LiveView Component?

Presence Setup

1. Setup a Presence Module in our app:

2. Add this module to our supervision tree:

I don't have any Channels... ?

Listening to topic events to track changes

Conclusion

Using Phoenix Presence in LiveView |> A Simple Example

Who’s reading?

LiveView or LiveView Component?

Presence Setup

1. Setup a Presence Module in our app:

2. Add this module to our supervision tree:

I don’t have any Channels… ?

Listening to topic events to track changes

Conclusion

Rendering Markdown in Elixir

Writing content for a blog.

Earmark

Phoenix Markdown Library

It's a Template Engine!

How do we create a Template Engine?

Our scramble Engine

Smartypants... 🤔

Interactive blog...

Hold on buddy! This can't work!

Conclusion

Rendering Markdown in Elixir

Writing content for a blog.

Earmark

Phoenix Markdown Library

It’s a Template Engine!

How do we create a Template Engine?

Our scramble Engine

Smartypants… 🤔

Interactive blog…

Hold on buddy! This can’t work!

Conclusion

Real World Phoenix |> Let's D4Y |> __using__ K8S

The Kubernetes Cluster

Terraform + Helm

Install doctl to talk to your DigitalOcean account

Install Terraform

Setup our project

Setup the DigitalOcean Kubernetes cluster

Install Traefik

Creating a kubernetes secret to access our registry

Install Gitlab Runner

Connect a gitlab project to the cluster

A project to deploy

Gitlab pipeline

Dockerfile

Entrypoint

Gitlab CI Yaml file

Yaml deployment files: service, ingress and deploy

service.yml

ingress.yml

deploy.yml

Conclusion

Real World Phoenix |> Let's D4Y |> using K8S

The Kubernetes Cluster

Terraform + Helm

Install doctl to talk to your DigitalOcean account

Install Terraform

Setup our project

Setup the DigitalOcean Kubernetes cluster

Install Traefik

Creating a kubernetes secret to access our registry

Install Gitlab Runner

Connect a gitlab project to the cluster

A project to deploy

Gitlab pipeline

Dockerfile

Entrypoint

Gitlab CI Yaml file

Real World Phoenix |> Let's D4Y |> using K8S