DEV Community: Dimitris Zorbas

A Livebook Smart-Cell to Render Diagrams

Dimitris Zorbas — Sun, 20 Nov 2022 00:00:00 +0000

I wrote my first Livebook smart-cell which renders diagrams from a textual description.

I recently discovered Kroki, an opensource tool which generates
images from the text notation of popular diagramming languages. I then
thought, Livebook makes such an excellent application companion for
runnable documentation. What if we could easily embed architectural (or any kind) of diagrams
without having to go through some complicated pipeline to convert the
diagram definition into an image first?

Yes, we can!

An old Chinese proverb goes "Up-to-date runnable documentation with diagrams is worth a thousand Confluence pages".
Elixir's standard documentation generator, ex_doc makes it
trivial to include Livebook notebooks in your documentation and run them.
It even supports Mermaid out of the box, you just need to wrap the source
in mermaid backticks.

```mermaid
graph TD;
  A-->B;
  A-->C;
  B-->D;
  C-->D;
```

What if your diagrams are not part-fish? No problem, there's kino_kroki.

You quickly can give it a go by following the link below:

Note: Please mind that you need to be running Livebook > 0.6

To try it from an existing notebook, throw kino_kroki to the mix with:

Mix.install([:kino_kroki])

Then you'll notice a new type of smart-cell titled diagram.

An editor will appear where you can paste and edit the diagram source
and when you evaluate the cell, the diagram will be rendered.

Prior Art

Smart-cells are a powerful way to extend the functionality of Livebook.
There's a fantastic guide bundled with Livebook, to help you write your
own smart-cell, which you can read here.

I took inspiration from github_graphql_smartcell which is somewhat similar to Kino.Kroki in the sense that they both include a source editor.

How it Works

As mentioned in the intro, this smart-cell is powered by Kroki.
It's API takes the diagram source as a URL encoded and compressed string
and responds with an image.

The heart of the smart-cell is this tiny module:

defmodule Kino.Kroki do
  @moduledoc """
  A simple encoder for the online diagram renderer https://kroki.io/
  """

  @doc """
  Returns a `Kino.Markdown` image to render the diagram.

  ### Examples

      iex> Kino.Kroki.new(Kino.Kroki.Sample.get(:graphviz), :graphviz)
      %Kino.Markdown{
        content: "![svg](https://kroki.io/graphviz/svg/eJx9kM"
      }
  """
  @spec new(graph :: String.t(), type :: Kino.Kroki.Samples.type()) :: Kino.Markdown.t()
  def new(graph, type) do
    graph
    |> :zlib.compress()
    |> Base.url_encode64()
    |> then(&"https://kroki.io/#{type}/svg/#{&1}")
    |> then(&"![svg](#{&1})")
    |> Kino.Markdown.new()
  end
end

Compressing and encoding in Elixir and OTP as you can see, is particularly simple, compared to
some other platforms (see examples). With the time left I contributed the Elixir example.

Making it Smart

So far, there's nothing particularly dynamic about Kino.Kroki, it's a
regular Elixir module which can return a Markdown image.

To better understand the concept of a smart-cell, consider it a code
template parameterized through UI interactions. At any point you can
convert a smart-cell into a code cell for further modifications.

We'll dissect each part of the module below:

defmodule Kino.KrokiSmartcell do
  use Kino.JS
  use Kino.JS.Live
  use Kino.SmartCell, name: "Diagram"

  # omitted
end

We're defining a module for the smart-cell, which adds Kino.JS making
the module asset-aware allowing us to inject JavaScript for our cell
through the asset "main.js" macro as well as CSS through asset "main.css".

We also use Kino.JS.Live since we want the cell to be dynamic,
listening to the update_type event then setting a diagram sample
according to the selected type.

With the init/2 callback, that receives the argument and the "server" context, we initialise the type, setting it to "graphviz" and we also fetch and assign a sample textual definition of a GraphViz diagram. More details about the module returning samples will be available further below.

@default_type "graphviz"

@impl true
def init(_attrs, ctx) do
  ctx =
    ctx
    |> assign(type: @default_type)
    |> assign(diagram: Kino.Kroki.Samples.get(@default_type))

  {:ok, ctx}
end

With the handle_connect/1, which is invoked whenever a new client connects,
we set up the initial state of the new client.

@impl true
def handle_connect(ctx) do
  {:ok, %{type: ctx.assigns.type, diagram: ctx.assigns.diagram}, ctx}
end

The handle_event/3 callback is responsible for handling any messages sent by the client. We fetch a sample diagram source for the selected type, update the context and broadcast an event which will be handled by JavaScript side of this cell and update the editor element.

@impl true
def handle_event("update_type", type, ctx) do
  ctx = assign(ctx, type: type)

  diagram = Kino.Kroki.Samples.get(type)

  ctx = update(ctx, :diagram, fn _ -> diagram end)
  broadcast_event(ctx, "update_type", %{type: ctx.assigns.type, diagram: diagram})

  {:noreply, ctx}
end

With the to_attrs/1 callback we compute the arguments for to_source/1. We set the type
and diagram source.

@impl true
def to_attrs(ctx) do
  %{"type" => ctx.assigns.type, "diagram" => ctx.assigns.diagram}
end

We implement to_source/1 which returns the Elixir code for our cell.
This is what will replace our smart-cell if we convert it to a "dumb" cell.
We return the AST of the Elixir source which by calling Kino.Kroki.new/2 renders a markdown image
generated by the Kroki server.
Kino.SmartCell.quoted_to_string/1 converts the AST to a readable, formatted code string.

@impl true
def to_source(attrs) do
  quote do
    Kino.Kroki.new(unquote(attrs["diagram"]), unquote(attrs["type"]))
  end
  |> Kino.SmartCell.quoted_to_string()
end

The JavaScript source of Kino.Kroki is:

  asset "main.js" do
    """
    export function init(ctx, payload) {
      ctx.importCSS("main.css");
      ctx.importCSS("https://fonts.googleapis.com/css2?family=Inter:wght@400;500&display=swap");

      root.innerHTML = `
        <div class="app">
          <form>
            <div class="container">
              <div class="row header">
                <div class="inline-field">
                  <label class="inline-input-label label">Diagram type</label>
                  <select class="input" name="type"/>
                    <option value="blockdiag">BlockDiag</option>
                    <!-- rest of the options omitted -->
                  </select>
                </div>

                <div class="logo">
                  <span>Powered by: </span>
                  <a href="https://kroki.io">
                    <img alt="kroki" src="https://kroki.io/assets/logo.svg"/>
                  </a>
                </div>
              </div>

              <div class="row">
                <div class="field grow">
                  <label class="input-label">Diagram Source</label>
                  <textarea
                    id="diagram-source"
                    name="diagram"
                    class="input textarea code"
                    placeholder=""
                    rows="25">#{Kino.Kroki.Samples.get(@default_type)}</textarea>
                </div>
              </div>
            </container>
          </form>
        </div>
      `;

      const typeEl = ctx.root.querySelector(`[name="type"]`);
      const diagramEl = ctx.root.querySelector(`#diagram-source`);

      typeEl.addEventListener("change", (event) => {
        ctx.pushEvent("update_type", event.target.value);
      });

      ctx.handleEvent("update_type", (event) => {
        typeEl.value = event.type;
        diagramEl.value = event.diagram;
      });

      ctx.handleSync(() => {
        // Synchronously invokes change listeners
        document.activeElement &&
          document.activeElement.dispatchEvent(new Event("change"));
      });
    }
    """
  end

We use the asset/2 macro to define the necessary JS code inline, but
there's also the option to set a path to load assets with:

use Kino.JS, assets_path: "lib/assets/kino_kroki"

We need to export an init function for our JavaScript module. The
first parameter ctx is the client-side context which provides a
variety of useful functions such as importCSS which we use to load CSS
from a URL.

We initialise the root element of the smart-cell with root.innerHTML
and set up event handlers. With ctx.pushEvent we push a client-side
change to the cell server and with ctx.handleEvent we apply changes
from the server.

typeEl.addEventListener("change", (event) => {
  ctx.pushEvent("update_type", event.target.value);
});

ctx.handleEvent("update_type", (event) => {
  typeEl.value = event.type;
  diagramEl.value = event.diagram;
});

ctx.handleSync(() => {
  // Synchronously invokes change listeners
  document.activeElement &&
    document.activeElement.dispatchEvent(new Event("change"));
});

The complete source of Kino.Kroki can be found here.

Finishing Touches

We need to make sure Livebook knows about this new smart-cell to list it
as one of the available options.

In the mix.exs of Kino.Kroki there is:

def application do
  [
    mod: {Kino.Kroki.Application, []}
  ]
end

and in the application module:

defmodule Kino.Kroki.Application do
  @moduledoc false

  use Application

  @impl true
  def start(_type, _args) do
    # 👇 This tells Kino we implemented a smart-cell
    Kino.SmartCell.register(Kino.KrokiSmartcell)

    Supervisor.start_link([], strategy: :one_for_one, name: KinoDB.Supervisor)
  end
end

Diagram Samples

Kino.Kroki is the last module of the smart-cell. It reads and parses a text file
containing samples for the diagram types Kroki can generate images from.
Let's take it apart to see if there are any useful patterns in there (hint: there are).

The text file containing the samples has the following format:

<--- sample:blockdiag --->
blockdiag {
  Kroki -> generates -> "Block diagrams";
  Kroki -> is -> "very easy!";

  Kroki [color = "greenyellow"];
  "Block diagrams" [color = "pink"];
  "very easy!" [color = "orange"];
}
<--- sample:wavedrom --->
{ signal: [
  { name: "clk",         wave: "p.....|..." },
  { name: "Data",        wave: "x.345x|=.x", data: ["head", "body", "tail", "data"] },
  { name: "Request",     wave: "0.1..0|1.0" },
  {},
  { name: "Acknowledge", wave: "1.....|01." }
]}

We start by assigning the location of the file to a module attribute:

defmodule Kino.Kroki.Samples do
  @samples_file :code.priv_dir(:kino_kroki)
                |> Path.join("samples.txt")

Then with:

@external_resource @samples_file

We instruct the compiler to recompile Kiko.Kroki.Samples each time there's a change to the samples file.

We then parse the file into a map where the keys are diagram types and the values are samples.

@samples_separator ~r/<--- sample:(?<type>.+) --->\n/m
@samples @samples_file
         |> File.read!()
         |> String.split(@samples_separator,
           include_captures: true,
           trim: true
         )
         |> Enum.chunk_every(2)
         |> Enum.map(fn
           [separator, diagram] ->
             {String.to_atom(Regex.named_captures(@samples_separator, separator)["type"]),
              String.trim(diagram)}
         end)
         |> Map.new()

We also define a type type, which is a union of all the supported diagram types.

@type type :: unquote(Enum.reduce(Map.keys(@samples), &{:|, [], [&1, &2]}))

This is rendered nicely in the docs and helps the user of this library
provide a valid argument when used programmatically.

Outro

There's a lot of ground to cover with the recent extensibility
improvements the amazing developers, community and sponsors a bringing
to Livebook. I recommend checking out the docs for:

If you are seeking inspiration for your next notebook or smart-cell,
check out notes.club which is an Elixir app to
help you discover notebooks contributed by the community.

Coming soom to a notebook near you "an open_api_spex smart-cell", stay tuned!

Spotted a Mistake?

Please contact me on twitter, or in the comments, or submit a PR for corrections.

A Guide to Secure Elixir Package Updates

Dimitris Zorbas — Tue, 29 Mar 2022 11:46:06 +0000

Keeping your dependencies up-to-date is essential to ensure that your applications stay healthy, secure, and performant. Thankfully, the BEAM ecosystem has its own package manager, Hex, which is fast, mature, and simple to use.

This article explores the available tools and commands to manage Hex dependencies and some tips to make the process more enjoyable.

Let's dive in!

List Updatable Dependencies in Your Elixir App

You can use the commands below to understand the relationships between dependencies before you attempt to update any of them.

List all your application's dependencies with:

mix deps --all

* bunt 0.2.0 (Hex package) (mix)
  locked at 0.2.0 (bunt) 7af5c7e0
  ok
* castore 0.1.15 (Hex package) (mix)
  locked at 0.1.15 (castore) c69379b9
  ok
* connection 1.1.0 (Hex package) (mix)
  locked at 1.1.0 (connection) 722c1eb0
  ok
* cowboy 2.9.0 (Hex package) (rebar3)
  locked at 2.9.0 (cowboy) 2c729f93
  ok
* cowboy_telemetry 0.4.0 (Hex package) (rebar3)
  locked at 0.4.0 (cowboy_telemetry) 7d98bac1
  ok

Or you can choose to print them in a tree format with:

mix deps.tree

This produces output like the following:

short
├── credo ~> 1.6 (Hex package)
│   ├── bunt ~> 0.2.0 (Hex package)
│   ├── file_system ~> 0.2.8 (Hex package)
│   └── jason ~> 1.0 (Hex package)
├── ecto_psql_extras ~> 0.6 (Hex package)
│   ├── ecto_sql ~> 3.4 (Hex package)
│   ├── postgrex >= 0.15.7 (Hex package)
│   └── table_rex ~> 3.1.1 (Hex package)

To produce an image output, run:

mix deps.tree --format dot && dot -Tpng deps_tree.dot -o deps_tree.png

Note: This option requires Graphviz. Read these Graphviz instructions to install it on your system.

Then open the created deps_tree.png file with a viewer of your choice. Being able to quickly visualize dependencies can help you decide whether a package is worth keeping in your mix.lock. A package could be pulling too many sub-dependencies or might not even be used in your app. Remember, the fewer dependencies, the easier it is to keep things up-to-date.

When you remove a dependency from mix.exs, it will remain in mix.lock. To remove unused dependencies, run:

mix deps.clean --unlock --unused

Check for Outdated Dependencies with Hex

After making sense of your dependencies tree and performing any necessary cleanups, check for outdated packages with:

mix hex.outdated --all

The output will resemble:

Dependency              Current  Latest  Status
bunt                    0.2.0    0.2.0   Up-to-date
cowlib                  2.11.0   2.11.0  Up-to-date
credo                   1.6.1    1.6.3   Update possible
db_connection           2.4.1    2.4.1   Up-to-date
decimal                 2.0.0    2.0.0   Up-to-date
earmark_parser          1.4.19   1.4.20  Update possible
postgrex                0.15.13  0.16.2  Update not possible

To view the diffs in each available update, visit:
https://hex.pm/l/AsY7q

Note: The --all flag shows all outdated packages, including the children of packages defined in mix.exs.

Notice the link in the output above. Hex prepares a nice page for us to inspect the diffs:

Pro-Tip

Use the --within-requirements flag in your CI to notify you of available updates.

mix hex.outdated --within-requirements 1>/dev/null || echo 'Updates available!'

The output of Elixir's package management tasks tends to be concise, well-documented and precisely guides you towards actions.

Inspecting Changes with Hex

To see changes between two package versions in the terminal, run:

# This will display the diff for the package opus
mix hex.package diff opus 0.7.0 0.8.1

diff --git a/var/folders/nc/t881hgn/T/opus-0.7.0-9F0FC44B/mix.exs b/var/folders/nc/t881/T/opus-0.8.1-2BEF6AED/mix.exs
index 0aba420..0c1ad3e 100644
--- a/var/folders/nc/t881/T/opus-0.7.0-9F0FC44B/mix.exs
+++ b/var/folders/nc/t881/T/opus-0.8.1-2BEF6AED/mix.exs
@@ -4,7 +4,7 @@ defmodule Opus.Mixfile do
   def project do
     [
       app: :opus,
-      version: "0.7.0",
+      version: "0.8.1",
       elixir: "~> 1.6",
       elixirc_paths: elixirc_paths(Mix.env()),
       build_embedded: Mix.env() == :prod,
@@ -39,7 +39,7 @@ defmodule Opus.Mixfile do
   defp deps do
     [
       {:retry, "~> 0.8"},
-      {:telemetry, "~> 0.4", optional: true},
+      {:telemetry, "~> 0.4 or ~> 1.0", optional: true},
       {:credo, "~> 0.8.10", only: [:dev, :test], runtime: false},
       {:ex_doc, "~> 0.24.2", only: :dev, runtime: false},
       {:dialyxir, "~> 1.0.0-rc.3", only: [:dev, :test], runtime: false},

You can view the diff in the browser by navigating to:

https://diff.hex.pm/diff/<package_name>/<version1>..<version2>

For example: https://diff.hex.pm/diff/opus/0.7.0..0.8.1

Hex Diff generates a highlighted git diff which you can view in the browser. You can share the link or even highlight a specific row.

Third-party dependencies are essentially somebody's code downloaded from the internet, which ends up in your application. There is no shortage of examples where packages have been hijacked and malicious versions uploaded.

Ideally, you should inspect the diff of every update. Hex seems to be the only package manager with this built-in feature at the moment.

Browsing Changelogs

Ultimately, an update might be available, but is it safe to apply it? Are there any code or configuration changes required for the update to work without issues? The diff between two package versions may contain thousands of lines of templates, tests, and docs that might not seem relevant to you.

Furthermore, a package might not even follow Semver (semver indicates whether the update is safe in compatibility terms).

Commonly, package maintainers keep a changelog to communicate notable changes and upgrade paths concisely. Read more about the benefits of keeping a changelog.

Now the bad news: not all packages have a changelog. So let's go changelog hunting!

The following task will fetch information for the credo package:

mix hex.info credo

A static code analysis tool with a focus on code consistency and teaching.

Config: {:credo, "~> 1.6"}
Locked version: 1.6.3
Releases: 1.6.3, 1.6.2, 1.6.1, 1.6.0, 1.6.0-rc.1, 1.6.0-rc.0, 1.5.6, 1.5.5, ...

Licenses: MIT
Links:
  Changelog: https://github.com/rrrene/credo/blob/master/CHANGELOG.md <--- Here
  GitHub: https://github.com/rrrene/credo

As you can see, the maintainer has added a link to the changelog, so that's nice of them.

There is even a link to the changelog in the HexDocs, which some developers may find really handy:

Tips:

Ensure there's a link to your changelog in mix.exs
Include the changelog in the hexdocs

Automated Changelog Fetching

Hunting for changelogs can get tedious after a while, especially if you want to update many packages. Thankfully, there is now an experimental package for that.

You can add it in your dependencies with:

defp deps do
  [
    {:changelog, "~> 0.1", only: [:dev, :test], runtime: false}
  ]
end

Ensure it is fetched:

mix deps.get

Invoke it for all updatable packages:

mix changelog

Or for a number of packages:

mix changelog tailwind jason

Package: tailwind
Current version: 0.1.4
Latest version:  0.1.5
Hexdiff: https://diff.hex.pm/diff/tailwind/0.1.4..0.1.5

## v0.1.5 (2022-01-18)
  * Prune app.js css import to remove required manual step on first install

Note: This will only print the version to update to, a link to the diff, and the changelog when there is a more recent version.

I wrote this open-source task. It uses some heuristics to locate the changelog by retrieving the Hex package metadata from the API, falling back to common locations in the repo.

My wish is that Hex will standardize including a link to a changelog in mix.exs and mix hex.info, and that Hex Diff will be enhanced to include changelog information.

Updating Dependencies in Elixir

To update all dependencies, run:

mix deps.update --all

In the output, you will see version updates in the following format:

Upgraded:
  credo 1.6.1 => 1.6.3
  earmark_parser 1.4.19 => 1.4.20
  ecto_sql 3.7.1 => 3.7.2
  ex_doc 0.27.3 => 0.28.2 (minor)
  makeup 1.0.5 => 1.1.0
  mint 1.4.0 => 1.4.1
  nimble_parsec 1.2.0 => 1.2.2
  nimble_pool 0.2.5 => 0.2.6
  phoenix_live_dashboard 0.6.2 => 0.6.5
  phoenix_live_view 0.17.5 => 0.17.7
  phoenix_view 1.1.0 => 1.1.2
  plug 1.12.1 => 1.13.3
  postgrex 0.15.13 => 0.16.2 (minor)
  tailwind 0.1.4 => 0.1.5
New:
  hpax 0.1.1

This is the same as running:

mix deps.unlock --all && mix deps.get

Keep in mind that this task will try to upgrade to versions that match the specifications in your mix.exs.

For example:

def deps do
  [
    {:some_package, "~> 0.9"}
  ]
end

Even if there is a more recent 1.0 version for some_package, it does not match the specification above, and it won't be upgraded. You will have to change your mix.exs and try again.

As we saw in the first section of this article, there is a task you can run to check if an update is possible for a package.

mix hex.outdated some_package

Dependency              Current  Latest  Status
some_package            0.9.0    1.0.0   Update not possible

There might be a conflict in some cases where the package resolution cannot find a version to satisfy the dependencies in mix.exs.

A workaround (that you should use with caution) is override.

def deps do
  [
    {:some_package, "~> 1.0"},
    {:other_package, "~> 2.0", override: true}
  ]
end

In this case, the dependency will override any other definitions by other dependencies.

A personal Brain with Nerves and LiveBook (podcast)

Dimitris Zorbas — Mon, 27 Dec 2021 10:11:23 +0000

In the most recent episode of the Elixir Mix podcast, I talked about how I built Brain a device to display my Kindle highlights and notes on an e-ink screen using Nerves and Livebook.

Enjoy!
https://elixirmix.com/personal-brain-with-nerves-and-livebook-dimitri-zorbas

Livebook Animations

Dimitris Zorbas — Sun, 26 Dec 2021 17:43:38 +0000

An exciting new feature landed in Livebook (through Kino) which gives
the ability to animate any output.

In the process of experimenting with Brain and its camera, I needed to
quickly sketch out some code and output video in a Livebook notebook.

I thought the following would do the trick:

Kino.Image.new(Picam.next_frame, :jpeg)
|> Kino.render

but it creates a new output cell every time Kino.render/1 is called.

So I posted this issue (kino#48) and implemented a new
widget Kino.ImageDynamic which can be updated with
Kino.ImageDynamic.push/2.

Then I also implemented a Kino.clear/0
function to dynamically clear any output cell, so that its contents can
be replaced by calling render again.

Thankfully the fruits of this conversation on the issue gave us a more
robust API for animation.

Kino.animate/3

This PR kino#49 and version 0.3.1 of Kino
bring Kino.Frame and the Kino.animate/3 function.

Watch a showcase of the feature below:

`Kino.Frame`

With Kino.Frame.new/0 you can start a new
widget which can be updated with Kino.Frame.render/2.

widget = Kino.Frame.new() |> tap(&Kino.render/1)

Kino.Frame.render(widget, 1)
Kino.Frame.render(widget, 2)
Kino.Frame.render(widget, 3)
Kino.Frame.render(widget, 4)

You'll notice that with the code above we only get a single output
cell which gets updated four times.

With Kino.animate/3 the above can be expressed more concisely:

Kino.animate(50, 1, fn
  i when i in 1..4 -> {:cont, i, i + 1}
  _ -> :halt
end)

Life

Let's put this new API to the test by implementing Life.
To try this on your Livebook instance by importing this notebook.

The implemenation is based on this gist.

defmodule Life.Grid do
  defstruct data: nil

  def new(data) when is_list(data) do
    %Life.Grid{data: list_to_data(data)}
  end

  def size(%Life.Grid{data: data}), do: tuple_size(data)

  def cell_status(grid, x, y) do
    grid.data
    |> elem(y)
    |> elem(x)
  end

  def next(grid) do
    %Life.Grid{grid | data: new_data(size(grid), &next_cell_status(grid, &1, &2))}
  end

  defp new_data(size, fun) do
    for y <- 0..(size - 1) do
      for x <- 0..(size - 1) do
        fun.(x, y)
      end
    end
    |> list_to_data
  end

  defp list_to_data(data) do
    data
    |> Enum.map(&List.to_tuple/1)
    |> List.to_tuple()
  end

  def next_cell_status(grid, x, y) do
    case {cell_status(grid, x, y), alive_neighbours(grid, x, y)} do
      {1, 2} -> 1
      {1, 3} -> 1
      {0, 3} -> 1
      {_, _} -> 0
    end
  end

  defp alive_neighbours(grid, cell_x, cell_y) do
    for x <- (cell_x - 1)..(cell_x + 1),
        y <- (cell_y - 1)..(cell_y + 1),
        x in 0..(size(grid) - 1) and
          y in 0..(size(grid) - 1) and
          (x != cell_x or y != cell_y) and
          cell_status(grid, x, y) == 1 do
      1
    end
    |> Enum.sum()
  end
end

Then we need a function which returns an SVG string to visualise the
grid.

defmodule Life.Svg do
  @cell_size 10

  def render(grid) do
    size = Life.Grid.size(grid)

    cells =
      for y <- 0..(size - 1), x <- 0..(size - 1), into: "" do
        status = Life.Grid.cell_status(grid, x, y)
        fill = if status == 0, do: "#EEE", else: "purple"

        "<rect x=\"#{x * @cell_size}\" y=\"#{y * @cell_size}\" width=\"10\" height=\"10\" fill=\"#{fill}\" />\n"
      end

    """
    <svg viewBox="0 0 #{@cell_size * size} #{@cell_size * size}" xmlns="http://www.w3.org/2000/svg">
      #{cells}
    </svg>
    """
    |> Kino.Image.new(:svg)
  end
end

Now we'll add a function to generate random starting configurations.

randomize = fn size ->
  for _ <- 1..size, do: Enum.map 1..size, fn _ -> Enum.random([0,1]) end
end

Next, we'll add a button to generate a few configurations and preview them.

button = Kino.Control.button("randomize")
Kino.Control.subscribe(button, :randomize)

button

When a button is pressed it sends events as messages. We handle them by rendering an SVG.

widget = Kino.Frame.new() |> Kino.render()
loop = fn f ->
  receive do
    {:randomize, _} ->
      # Preview the configuration when the button is pressed
      Kino.Frame.render(widget, Life.Svg.render(Life.Grid.new(randomize.(22))))
      f.(f)
    _ -> :ok
  end
end

loop.(loop)

Buttons are a new addition to Kino and Livebook released in version 0.4.0.
You can find their docs here.

Having made sure that we can correctly render a grid, we can finally animate it.

Kino.animate(100, Life.Grid.new(randomize.(25)), fn grid ->
  {:cont, Life.Svg.render(grid), Life.Grid.next(grid)}
end)

Thanks for reading this post, hope you'll find it useful and make
your notebooks pop with captivating animations.

An Unusual Pomodoro Timer on Elixir and Nerves

Dimitris Zorbas — Tue, 12 Oct 2021 00:00:00 +0000

In my previous post about "Organising Book Highlights and Notes",
I wrote:

"Some day I may build a gadget for my desk to display a daily quote".

A few days later, it's on my desk and it couldn't have been a better
pretext to give Livebook on Nerves a try.

Finished Product

It's alive! 🧟

The case is not made of flesh, but working with clay is not my forte. I should buy a 3D printer at some point..

Goal

The scope of this weekend-project was to build some sort of smart desktop
ornament. It should periodically display a random quote from my Kindle
highlights and notes on an E Ink screen.
The quote is fetched via bookworm 🪱📚.
The display should refresh every 25 minutes and flash, marking the end of a time
block.

As the title hints, this is an unusual pomodoro timer. It deviates
from the standard pomodoro technique, but it suits me.

Name

A project needs a name. The obvious choice for something built on Nerves is
no other than Brain 🧠!

And here's a sample quote Brain would display:

The function of the brain and nervous system is to protect us from
being overwhelmed and confused by this mass of largely useless and irrelevant
knowledge, by shutting out most of what we should otherwise perceive or
remember at any moment, and leaving only that very small and special selection
which is likely to be practically useful.

Aldous Huxley, The Doors of Perception

The combination this name and Elixir's fault tolerance reminds me this excellent cult horror film:

Optional Features

Display the week number

I tend to make daily plans and thinking in terms of numbered weeks, it
drives me to make them memorable.

Display the temperature outside

I tend to hit the gym after work, so this is mostly to inform me whether to wear shorts.
(plot twist: I'll wear shorts anyway).

Why E Ink?

The slow refresh rate of E Ink devices is ideal for something I'll keep
right in from of me. It can be useful even when powered off.
It's not even backlit and it doesn't distract me at all.
I was sceptical of having yet another "screen" to look at. I've
even ditched my multi-monitor setup for a single monitor with the
laptop's monitor solely for messaging apps like Slack.
Oh and the energy ⚡️ consumption is minimal, Pisugar could power it for a couple of days (untested
assumption).

What you'll need

Raspberry Pi 2 model B (that's what I had lying around)
SD card
WiFi USB dongle (Raspberry Pi 2 doesn't have on-board WiFi)
Pimoroni Inky wHat E Ink Display
A JSON file with notes to display (see bookwork)

You might notice there's a camera on my device, it's not required for
the features discussed in this post. A future next one will be about it
though, stay tuned!

Instructions

Follow the instructions below to build one yourself.

1. Burn the Nerves Livebook Firmware

Install prerequisites for packaging firmware images

MacOS

brew update
brew install fwup squashfs coreutils xz pkg-config

Linux (Debian)

sudo apt install build-essential automake autoconf git squashfs-tools ssh-askpass pkg-config curl

Then install nerves_bootstrap with:

mix archive.install hex nerves_bootstrap

You can find more detailed information about getting started with Nerves
here.

Burn the Image

Follow the instructions on the nerves_livebook repo
to burn the firmware using fwup or run the code below in your shell:

wget https://github.com/livebook-dev/nerves_livebook/releases/download/v0.2.26/nerves_livebook_rpi2.fw

# Mind to replace with your SSID and passphrase
sudo NERVES_WIFI_SSID='access_point' NERVES_WIFI_PASSPHRASE='passphrase' fwup nerves_livebook_rpi2.fw

2. Mount the E Ink Screen Hat on the Pi

At this point you can already power on your Pi.

Point your browser to http://nerves.local ..and voila!

The password is "nerves".

🙌 You have an easily accessible instance of Livebook running on a tiny computer.
It runs in embedded mode by default which means that code evaluated in
your notebooks runs in the context of the Livebook node.

This is ideal, as it allows us to redefine Scene modules from within a
notebook and trigger a screen refresh without uploading firmware changes and rebooting the Pi.
More about that further below.

3. Clone the Brain Repo

Hang tight, we're halfway there, run:

git clone git@github.com:zorbash/brain.git

4. Add Some Notes

You can use bookwork to download your Kindle highlights in a
format Brain understands or simply copy mine from https://github.com/zorbash/notes/blob/main/books.json
and place them in priv/notes/.

5. Deploy! 🚀

Then run:

export MIX_TARGET=rpi2

mix deps.get
mix firmware
mix firmare.gen.script
./upload.sh livebook@nerves.local

When it asks for a password, type in nerves.

That's it, a few moments later the screen should flash and your Brain
will have come to life.

Making Changes

Brain is open-source and it's easy to tweak it according to your needs.
The UI is built using Scenic which makes it trivial to
preview changes locally without re-uploading firmware.

Start Scenic to preview your changes:

MIX_ENV=dev MIX_TARGET=host iex -S mix scenic.run

You should see a window like this:

The Scene to modify is NervesLivebook.Scenes.Main and you can trigger
an update with:

send :main_scene, :update

To trigger a screen refresh remotely run:

ssh livebook@nerves.local 'send :main_scene, :update'

💡 Since Brain runs Livebook (http://nerves.local), you can trigger an update by evaluating an
Elixir cell with send :main_scene, :update in a notebook.

Debugging

You can connect to your Brain with:

ssh livebook@nerves.local

Then you can inspect the logs with:

RingLogger.next

For further debugging, check out toolshed which bundles a
variety of helpful utility helpers.

use Toolshed
import IEx.Helpers

h(Toolshed)

Caveats

The Pimoroni E Ink screen afaik does not support partial refresh, it'd
be cool to partially refresh the header (week, temperature, time) every
minute.

Scenic's text wrapping could support break-word, see scenic#118.

Future Enhancements

Get it to show notes per bookshelf, for example it could be configured to show only notes from the computer science bookshelf between 10 and 11 AM and from philosophy between 6 and 7 PM.

Livebook

Livebook is an incredible piece of technology, improving at a mind-blowing
rate.

A couple of months ago I started work on an enhancement to be
able to show the documentation for any module / function in a notebook.
My intention was to expose a Livedoc.render(:plug) to fetch and
render the documentation of the given hex package.

Thankfully, I was surprised to find out that this has already been
implemented in livebook#453 🙌.

If you haven't tried Livebook yet, please check out livebook.dev.
You can even configure the default Livebook location where notebooks
will open. I've set mine to http://nerves.local where Brain lives.

With your notes loaded in Brain, you can quickly browse and search them with the following notebook:

# Notes

<!-- livebook:{"livebook_object":"cell_input","name":"search","reactive":true,"type":"text","value":""} -->

query = IO.gets("search: ") |> String.trim()
notes = NervesLivebook.Notes.all() |> Enum.filter(&String.contains?(&1.text, query))

Kino.DataTable.new(notes)

Reactive inputs make this possible.

Watch a video if it in action here

Acknowledgements

This project wouldn't be as enjoyable and painless without nerves,
nerves_livebook, scenic, inky.

Thank you @mobileoverlord, @fhunleth, @boydm,
and @lawik and all of the contributors to this amazing part of the Elixir ecosystem.

Organising Book Highlights and Notes

Dimitris Zorbas — Sat, 02 Oct 2021 00:00:00 +0000

I've used a variety of tools to organise my reading and notes. Given I spend a significant amount of my time studying,
depending on 3rd parties gives me anxiety. Any of the tools I use, even the open-source offline-first ones,
can become unmaintained, ridden with security vulnerabilities, slow or they may change in way which makes
me reluctant to use them.

To some extent, this post is a sequel to "knowledge mapping".

Why care about my notes and Highlights?

A friend asked this question. She said, are you going to use this
mid-conversation to correct someone? Do you want to be the "Well, Ackchyually.." guy?

Most definitely not. I've noticed the rate of information
I consume keeps increasing disproportionately to the rate I digest it
into knowledge. I repeat it's about building knowledge not
memorising.

This project in particular, is about using tech for
"good", doing your thing, or escaping from servitude to
the capitalistic delusion of perpetual exponential growth (too far eh? 😅).

Initially I became reasonably frustrated of not being able to revisit my Kindle highlights.
Half of the books I read, I do so on my Kindle device, I highlight sentences, then said
highlights appear on goodreads.com and read.amazon.co.uk.

I felt locked-in using Goodreads (owned by Amazon) and read.amazon.
What if my highlights become unavailable due to some licensing issue?

Having all my highlights in one place, open up some interesting possibilities. Similar
to how I fancy creating playlists, I might for example, some day curate my favourite
quotes by an author. Some day I may build a gadget for my desk to display a daily quote
(raspberrypi + e-ink).

So.. I built a thing

I started off with a simple script which downloads all my Kindle highlights.
It formats and builds a readable, pretty-printed JSON file which I sync
in Git.

Then I enhanced it so that I can write my own notes and highlights from
non-Kindle physical books.

Then I made it create and incrementally import highlights
into a Notion database. Why? I was too lazy to write a frontend and I'm
starting to like Notion.
What's great about this database is that it can be embedded as a view in
any page.

The Notion database can then be shared. He's mine.

Demo Time

The script can be found here and accepts the following commands:

`sync_local`

./notes sync_local

It downloads Kindle highlights and imports local notes into a books.json file.

The location of the flat-file "database" can be configured through a
DB_FILEPATH env var, which can also be set in a .env file.

The "database" file is intended to be version-controlled with Git.

Local Notes

Highlights and notes can also be imported into the "database" books.json file.
Such notes are written as YAML, yes YAML. I contemplated
creating my own tiny markup language for this (ideally using
nimble_parsec), this non-feature ended up in the "backlog".

The location of the notes file can be configure through the NOTES_FILEPATH env var, which can also be set in a .env file.

-
  # The asin key may also hold an ISBN
  asin: 0-679-76288-4
  title: High Output Management
  author: Andy Grove
  highlights:
    -
      location: 17
      text: >
        A genuinely effective indicator will cover the output of the work unit and not simply
        the activity involved.

search

./notes search <keyword>

Prints any highlights which match the given keyword.

Example:

./notes search work

Found 179 results for "work"

╔════════════════════════════════════════════════════════════════════╗
║  Book:   High Output Management                                    ║
║  Author: Andy Grove                                                ║
╚════════════════════════════════════════════════════════════════════╝

 A genuinely effective indicator will cover the output of the work unit
and not simply the activity involved.

random

./notes random

Returns a random highlight.

Example:

╔════════════════════════════════════════════════════════════════════╗
║  Book:   The Genealogy of Morals                                   ║
║  Author: Friedrich Nietzsche                                       ║
╚════════════════════════════════════════════════════════════════════╝

 All sick and diseased people strive instinctively after a herd-organisation,
out of a desire to shake off their sense of oppressive discomfort and weakness;
the ascetic priest divines this instinct and promotes it;

update_notion

./notes update_notion

Syncs the local "database" file into a Notion database.

It supports the --since <date> flag to only sync the database entries
which have been updated since the given date (ISO-8601 formatted). This
option is particularly useful since the Notion API is rate-limited and
for more than 1000 highlights syncing can take significant time (more
than 10 minutes).

Recap

Features

Offline-first - The "database" is a single JSON file
Version control - The JSON database file is prettified making it easy to review and commit changes to git
Both notes and the JSON database are readable and searchable using a text editor or tools like jq
Fast search - Both the CLI search and Notion's search are fast

Non-Features

Removing highlights from the local database - I never delete highlights
Full-text search - Simple regex case-insensitive does the trick for now

Contributing

Feel free to fork either the bookworm script or my notes.
As stated at the top of this post, this is a fun project for me, so
expect no support. However, I'd be glad to discuss ideas on this domain.

Phoenix Telemetry

Dimitris Zorbas — Wed, 07 Apr 2021 00:00:00 +0000

Telemetry is becoming the defacto library to instrument and publish
metrics in Elixir apps. This post is a step-by-step guide to integrate
Telemetry in a Phoenix app which leverages Opus.

What is Telemetry

Telemetry is a dynamic dispatching library for metrics and instrumentations.
It is lightweight, small and can be used in any Erlang or Elixir project.

Project page: https://github.com/beam-telemetry/telemetry

Instead of every library / framework having its own way of publishing
metrics, the ecosystem is gradually converging on using Telemetry.

How it Works

You may attach event handlers which are stored in an ETS table. When an
event is emitted, all matching event handlers are called. Due to the
fact that handlers are called synchronously, you should make sure they
are fast, so not to cause bottlenecks.

Example

Set up a handler which logs to the console

Start an IEx session with iex, then run:

# Install telemetry (Mix.install requires Elixir > v1.12)
Mix.install([:telemetry_metrics])

# Attach a uniquely named handler which prints measurements
:telemetry.attach("example-handler",
                  [:example, :metric, :done],
                  (fn _metric, measurements, _meta, _config ->
                    IO.inspect measurements
                  end),
                  nil)

Emit an event

:telemetry.execute [:example, :metric, :done], %{latency: 1337}, %{}

See the output

You should see %{latency: 1337} in the output.

How it Works

:telemetry.attach/4 inserts an object in an ETS table named telemetry_handler_table.

We can inspect the table to verify this:

:ets.tab2list(:telemetry_handler_table)

#=> [
#=>   {:handler, "example-handler", [:example, :metric, :done],
#=>   #Function<5.126501267/4 in :erl_eval.expr/5>, nil}
#=> ]

When an event is emitted, the process emitting the event will call all
the event handlers matching the given event name. To list the matching
handlers, you can call:

:ets.lookup :telemetry_handler_table, [:example, :metric, :done]

#=> [
#=>   {:handler, "example-handler", [:example, :metric, :done],
#=>   #Function<5.126501267/4 in :erl_eval.expr/5>, nil}
#=> ]

Phoenix Telemetry

If you're building a Phoenix application, you're in luck. Since v1.5, new Phoenix applications are generated
with a Telemetry supervisor. Many popular libraries emit telemetry
events and you can easily visualize events in LiveDashboard.

Opus Telemetry

Opus is a library to build "service modules".

Consider the following pipeline module:

defmodule Example.GenerateSession do
  use Opus.Pipeline

  check :valid?, with: &match?(%{user_id: uid} when is_integer(uid), &1)
  step :generate, with: &"session-#{&1.user_id}"
end

You can call this module with:

Example.GenerateSession.call %{user_id: 1337}

# => {:ok, "session-123"}

Wouldn't it be ideal to publish metrics for the duration of each step as
well as the total duration?

Thankfully, Opus supports defining instrumentation modules, but you can
also define instrumentation functions inline.

To emit telemetry events, add the following module in your codebase:

defmodule Example.OpusTelemetry do
  require Logger

  def instrument(:pipeline_started, %{pipeline: pipeline}, %{input: _input}) do
    :telemetry.execute(
      [:opus, :pipeline, :start],
      %{time: System.system_time()},
      %{pipeline: inspect(pipeline)}
    )
  end

  def instrument(:stage_completed, %{stage: %{name: name, pipeline: pipeline}}, %{time: time}) do
    :telemetry.execute(
      [:opus, :pipeline, :stage, :stop],
      %{duration: time},
      %{pipeline: inspect(pipeline), stage: name}
    )
  end

  def instrument(:pipeline_completed, %{pipeline: pipeline}, %{result: {:ok, _}, time: time}) do
    emit_stop(%{pipeline: pipeline, success?: true, duration: time})
  end

  def instrument(:pipeline_completed, %{pipeline: pipeline}, %{result: {:error, _}, time: time}) do
    emit_stop(%{pipeline: pipeline, success?: false, duration: time})
  end

  def instrument(_event, _, _) do
    :ok
  end

  defp emit_stop(%{pipeline: pipeline, success?: success?, duration: duration}) do
    :telemetry.execute(
      [:opus, :pipeline, :stop],
      %{duration: duration, success: success?},
      %{pipeline: inspect(pipeline)}
    )
  end
end

📣 The next version of Opus, will provide an Opus.Telemetry module, and
you won't need to copy-paste code into your app.

Next, update your config/config.exs with:

config :opus, :instrumentation, [Example.OpusTelemetry]

We can first set up a very simple handler to verify that instrumentation
is configured correctly. In an IEx session run:

require Logger

:telemetry.attach "opus-test",
                  [:opus, :pipeline, :stop],
                  (fn _, m, _, _ -> Logger.info inspect(m) end),
                  nil

Then call the sample pipeline a few times:

iex(16)> Example.GenerateSession.call %{user_id: 1337}
# 20:40:43.164 [info] %{duration: 15000, success: true} 👈 Printed by the handler
{:ok, "session-1337"}

iex(21)> Example.GenerateSession.call %{user_id: 42}
# 20:40:43.164 [info] %{duration: 17000, success: true}
{:ok, "session-42"}

iex(22)> Example.GenerateSession.call %{user_id: "invalid"}
# 20:40:43.164 [info] %{duration: 19000, success: false}
{:error,
 %Opus.PipelineError{
   error: :failed_check_valid?,
   input: %{user_id: "invalid"},
   pipeline: Example.GenerateSession,
   stacktrace: nil,
   stage: :valid?
 }}

It works! 🙌

Moving on to configure our metrics in our Phoenix app so that we can graph then in LiveDashboard.

# TODO: Add the code in ExampleWeb.Telemetry module

Any recently generated Phoenix application (after v1.5) will already
have a Telemetry supervisor, which looks like this:

defmodule ExammpleWeb.Telemetry do
  use Supervisor
  import Telemetry.Metrics

  def start_link(arg) do
    Supervisor.start_link(__MODULE__, arg, name: __MODULE__)
  end

  @impl true
  def init(_arg) do
    children = [{Telemetry.Metrics.ConsoleReporter, metrics: metrics()}]

    Supervisor.init(children, strategy: :one_for_one)
  end

  def metrics do
    [
      # Phoenix Metrics
      summary("phoenix.endpoint.stop.duration",
        unit: {:native, :millisecond}
      ),
      summary("phoenix.router_dispatch.stop.duration",
        tags: [:route],
        unit: {:native, :millisecond}
      )
    ]
  end

  defp periodic_measurements do
    []
  end
end

What you need to do is to change the metrics method above to include
the Opus metrics you wish reported:

def metrics do
  [
    # Phoenix Metrics
    summary("phoenix.endpoint.stop.duration",
      unit: {:native, :millisecond}
    ),
    summary("phoenix.router_dispatch.stop.duration",
      tags: [:route],
      unit: {:native, :millisecond}
    ),

    # 👇 Add these two
    counter("opus.pipeline.start.time",
      description: "Pipeline Started",
      tags: [:pipeline]
    ),
    summary("opus.pipeline.stop.duration",
      description: "Pipeline Duration",
      tags: [:pipeline],
      unit: {:native, :millisecond}
    )
  ]
end

One thing to notice is how the first argument to metric functions is constructed:

"opus.pipeline.stop.duration"

[:opus, :pipeline, :stop]      :duration
<----- event name ------>  <-- measurement -->

Live Dashboard

Time to fire up the Phoenix server and point the browser to the dashboard:

http://localhost:4000/dashboard/metrics?nav=opus

Upcoming Blog post

There's so much to cover in a single post.
In the next post, I'll to write about:

Creating a custom LiveDashboard page to visualize pipelines (think opus_graph)
Reporting Telemetry events to DataDog

sidekiq-dry

Dimitris Zorbas — Sun, 17 Jan 2021 20:15:53 +0000

I published a new gem, sidekiq-dry aiming to tackle a variety of
common frustrations when it comes to Sidekiq jobs and their arguments.

Rationale

Sidekiq is among the most popular background job solutions. It's my
first choice for Ruby apps. The dry-rb family of gems is also
indispensable in non-trivial applications. What if we combined the two..

With sidekiq-dry you may pass instances of Dry::Struct as arguments
to your Sidekiq jobs. But why?

Prevent Type Ambiguity

Numerous times I've had to debug jobs which where failing due to being
enqueued with invalid arguments.

Example:

class SendInvitationEmailJob
  include Sidekiq::Worker

  def perform(user_id, invitee_email)
    # code
  end
end

SendInvitationEmailJob.perform_async(user.id, params[:invitee_email])

The problem with the above code is that if the user_id is not an
Integer id or the invitee_email is not a valid email String then
there's absolutely no chance that the enqueued job will complete
successfully. Of course Dry::Struct is not to be used for validations,
there's dry-validate for that, or ActiveModel / ActiveRecord
validations if you prefer. Giving more structure to your
background job arguments improves the system's robustness. Your objects
in transport through Redis, as long as the job is enqueued, they are
guaranteed to have the expected structure when the job is performed.

The above example would be refactored to:

class SendInvitationEmailJob
  include Sidekiq::Worker

  def perform(params)
    # code
  end
end

class SendInvitationEmailJob::Params < Dry::Struct
  attribute :user_id, Types::Strict::Integer
  attribute :invitee_email, Types::Strict::String.constrained(format: /\A[\w+\-.]+@[a-z\d\-]+(\.[a-z]+)*\.[a-z]+\z/i)
end

job_params = SendInvitationEmailJob::Params.new(user_id: user.id, invitee_email: params[:invitee_email])

SendInvitationEmailJob.perform_async(job_params)

At this point you might ask, what if we passed in a Hash, instead of a Dry::Struct?
Well, Hash arguments are deserialised with String keys which can lead to surprises.

Eliminate Positional Arguments

When your background job takes two or more positional arguments, it's
better to refactor it to take a single struct object with a
comprehensible name.

In the Rails world it's common to enqueue jobs with a record's id.
There's nothing wrong with this pattern. However, in some cases, developers may define a
model blindly following the convention.

Documentation

By using Dry::Struct arguments you'll be able to express constraints
straight in your code. Instead of documenting the types of each job argument,
which can easily become outdated, you can refer to the types of the attributes of the struct.

class Post < Dry::Struct
  attribute :title,  Types::Strict::String
  attribute :tags,   Types::Array.of(Types::Coercible::String).optional
  attribute :status, Types::String.enum('draft', 'published', 'archived')
  attribute :body,   Types::String.constrained(min_size: 10, max_size: 10_000)
end

Arguably, in the example above, both types and constraints improve readability.

Versioning

Adding this gem does not break any existing jobs in your app.
It only works on jobs enqueued with Dry::Struct objects.

Adding a new attribute to a parameter struct won't break already enqueued jobs.

It's trivial to version your structs using either a version attribute:

class Coupons::ApplyCouponJob::Params < Dry::Struct
  attribute :user_id,     Types::Strict::Integer
  attribute :coupon_code, Types::Strict::String
  attribute :version,     Types::Strict::String.default('1')
end

or versioned classes:

class Coupons::ApplyCouponJob::Params::V1 < Dry::Struct
  attribute :user_id,     Types::Strict::Integer
  attribute :coupon_code, Types::Strict::String
end

Caveats

Job processing libraries compatible with Sidekiq, for example
exq, won't deserialise your Dry::Struct arguments. This is most likely an acceptable tradeoff.

The Gem

The gem is hosted on rubygems (link). It provides two Sidekiq
middlewares which serialise and deserialise instances of Dry::Struct
arguments in your jobs.

Installation

Add the gem in your Gemfile:

gem 'sidekiq-dry'

Configure Sidekiq to use the middlewares of the gem:

# File: config/initializers/sidekiq.rb

Sidekiq.configure_client do |config|
  config.client_middleware do |chain|
    chain.add Sidekiq::Dry::Client::SerializationMiddleware
  end
end

Sidekiq.configure_server do |config|
  config.server_middleware do |chain|
    chain.add Sidekiq::Dry::Server::DeserializationMiddleware
  end
end

Knowledge Mapping

Dimitris Zorbas — Sun, 23 Aug 2020 00:00:00 +0000

What do I know? What do we know?

How do we know what we know and what is there that we should know?

Video games like Age of Empires, Civilisation, both favourites of mine and others have the concept of technology trees.

I study a lot, where should I spend my time? What should I read next? Is my knowledge broad enough?
I’ve used goodreads quite extensively, it’s great to be able to discover new books to read, mark your progress, make friends and such.

Nowadays there’s so much information readily available but not that many tools to organise and allocate your “research points”.

Learning Together

The recent COVID-19 pandemic surfaced the shortcomings of humanity to work together to solve fundamental healthcare issues. Some “experts” rushed to call it a black swan despite it clearly not being one (read more).

Thousands of Americans are victims of the opioid crisis and countless ones
are losing their money in the wild-west of unregulated fintech companies with mottos like “anyone can be a trader” or in cryptocurrencies 🙃.

The media promote optimising for the short-term making us think we’re the last generation to walk the planet. Our countries are involved in wars, we fund wars with our taxes, yet the disasters that war brings seem so remote.

How do we respond? We educate ourselves, we talk, we act. A university degree or a code bootcamp might help you land a job, but securing your freedom of thought it’s a never-ending struggle.

Learning to Learn

Be more systematic, build your knowledge map and start exploring branch
by branch.
You don’t need a degree as a reward, nor any imaginary internet points.
The real reward is that you’ll be able to better understand the natural world and society.
Think of #BlackLivesMatter how much do you know about it? Do you want to be the person who forms
an opinion based on a couple of tweets or headlines?
Create a list of books, films, articles to go through and keep track of your progress.

Reading, has this incredible effect that it minimises your chances of becoming a racist,
an anti-vaxxer, a climate-change denier or a flat earther. Quite a reward ain’t it?

Universities

Universities play the role of gatekeepers of knowledge and the perpetuation of class segregation.
Your studies have a certain duration and you get a degree. While such a concept is valuable to maintain
the status quo and keep the economy going it doesn't really seem to be about knowledge.
Learning is a continuous process and universities should not extinguish your passion for knowledge in exchange for a degree.
For me, receiving my degree wasn't something to celebrate, more like a warrant to stay away from the
supposed mecca of learning.

Fortunately in the digital age there are plenty of choices when it comes to expanding your knowledge.
Khanacademy, Coursera, Udacity to name a few.
Most of them seem heavily leaning towards STEM studies though. Where do we learn about the world?

First things first, by “the world” I mean outside the tech bubble.

Yes, I can fix your computer, but how do I fix racism, how do I get to know my body, improve my health,
manage my personal finances and understand democracy and political science, so to meaningfully
participate in the commons?

There's Hope

The open-source community is a fantastic example of people working asynchronously together and
having colossal impact in our lives.

A laughably simple algo to keep learning could be:

Map your knowledge in a graph format
Keep notes / annotate your readings publicly
Pick another branch from the tree
Go to step 2

Remember, there's nothing embarrassing in admitting you don't know
something, it's empowering and the first step to master any subject.

What about the cover image of this post?
Well, I couldn't help it and wrote some code to support it.

Visualising Knowledge Maps

Khanacademy used to have a knowledge map feature which I found
inspiring. Browsing a universe of infinite topics, continuing education ftw.

Building my Knowledge Map

So, I decided to build my own map and needed to bootstrap it somehow. I
quickly extracted, cleaned and analysed the tags of my Tefter bookmarks.
For the uninitiated, Tefter is a social bookmarking app I use heavily and I also develop 🤠.

On the graph, tags are nodes and edges connect topics when they appear as tags on the same bookmark.

Is this a real knowledge map? No, it's the minimum viable hack, a
compass to aid me steer towards building the real one.

Feel free to have a look and remix the code of the visualisation on Glitch.

To make progress I'll attempt to bring order to that
chaotic graph by grouping some topics together. I'm also starting this
book "Consilence: The Unity of Knowledge" which might help me with that.

I'll also try to evaluate some apps and methodologies below:

You've reached the end of this post. Please post your thoughts,
feedback and ideas in the comments.

Omni - A tabs, history and bookmarks search companion

Dimitris Zorbas — Mon, 25 May 2020 15:11:50 +0000

We've launched Omni, the new Tefter browser extension that lets you search and navigate your open tabs, history and Tefter bookmarks. It also automatically groups together similar links.

We usually have many tabs open, it hurts productivity and increases stress. After a certain number of tabs, their titles are not even visible. Most of us switch between tabs, helplessly guessing by the favicons, hundreds of times a day. When it comes to history, trying to find a page from let's say last week, the browsers make it almost impossible. Think about it, Google with their Chrome browser has interest in you blindly "googling" stuff instead of really helping you. Firefox would want you to bookmark everything on Pocket, which shows ads and you may even end up buying a subscription there.

Omni is the fastest tab and history search tool out there. It's free, never shows ads and won't sell your data. All your data stays local and is optimally indexed. It's tailored for users who prefer keyboard-driven interactions. It analyzes your history and tab switching behavior and the more you use it the better the search results you see.

We truly believe it can change the way we surf the web and this is only the beginning.

To install:

We'd love to get some feedback from you. Please comment and upvote!

https://www.producthunt.com/posts/omni-tabs-and-history-search-by-tefter

Writing a Command-Line Application in Elixir

Dimitris Zorbas — Sun, 01 Mar 2020 00:00:00 +0000

I've always been fascinated by well-made applications for the terminal. Who doesn't install htop on a new machine, am I right?

My plan was to build something that I'd use daily and other people would potentially
find useful. Therefore I decided to build a cli app for Tefter.

It's built on Elixir and Ratatouille and it's open-source.

Check out the source or download and try it or install via brew.

brew tap tefter/homebrew-cli
brew install tefter

Why Elixir

Elixir is getting popular for web and distributed applications, but these
days devs tend to write cli apps in Rust / Go / C. We use Elixir at
Tefter, so there was a case for code
reuse and at some point I stumbled upon Ratatouille.
It's an Elm inspired framework, which leverages termbox,
a C library for text-based user interfaces. Being charmed by the beautiful
API of Ratatouille and eager to overcome potential hurdles, once again I chose Elixir.

Pros

High-level language
Fault-tolerance
Optimal offline storage (ETS / DETS / Mnesia)
Some portability with Releases
A decent framework (Ratatouille)
Fantastic Concurrency (see: commands)
Code-reloading for quick debugging
It's so much fun :-)

Cons

Releases bundle the VM and could be smaller
Releases are not truly portable (a release built on a Linux machine won't work on a Mac, etc)
No trivial way to fork-exec

Demo Time!

Take a glimpse of how the app behaves:

About Tefter

Before going into more detail about the specifics of the cli, let's talk about Tefter first.
It's a tool aiming to optimise your web surfing routine, a combination of personal search-engine,
a social bookmarking tool and a place to archive stuff to read later and write notes. One would interact
with Tefter through the Web app, the browser extension, the mobile and
the desktop apps or Slack!

The App

At the moment of writing, it features three main tabs for Search, Aliases and Bookmarks.
Some of the advantages of the cli app to the rest of the available are:

You don't have to leave your terminal and keyboard
vim-style keybindings with mouse support 😎
Works offline

Shortcuts

Key	Action
Ctrl+s	Jump to Search tab
Ctrl+a	Jump to Aliases tab
Ctrl+b	Jump to Bookmarks tab
Ctrl+h	Jump to Help tab
Tab	Jump to the next tab
Home	Jump to the first tab
↑	Move up
Ctrl+k	Move up
↓	Move down
Ctrl+j	Move down
Ctrl+d	Scroll down
Ctrl+u	Scroll up
Enter	Open browser window with item under cursor
Esc	Cancel command / Quit modal
F5	Force refresh resources
Ctrl+q	Quit
/	Enter filtering mode
:	Enter command mode

Authentication

The authentication is implemented to be as seamless as possible. It
won't ask the user to type their username and password.

The first time the application is started, it looks for an authentication
token in a ~/.tefter file. This file holds a plain JSON config. If not
found it'll start a tiny web server listening on a random port. It'll
then open a browser window to a special Tefter endpoint which redirects
to the address of the local web server with the authentication token
encoded in the query params. The app then proceeds to create the
~/.tefter file.

See it in action:

Search

This work in a similar manner to the auto-complete of the Web app. The
user types and results appear in the panel below. Results can be
bookmarks, lists, domains, tags or aliases.

App Architecture

Let's have a look at how this works. At the moment, the app is structured as follows:

lib/tefter_cli
├── app
│   └── state.ex
├── app.ex
├── application.ex
├── auth_server.ex
├── authentication.ex
├── bookmarks.ex
├── cache.ex
├── command.ex
├── config.ex
├── system.ex
└── views
    ├── aliases
    │   ├── actions.ex
    │   └── state.ex
    ├── aliases.ex
    ├── authentication.ex
    ├── bookmarks
    │   ├── actions.ex
    │   └── state.ex
    ├── bookmarks.ex
    ├── components
    │   ├── bottom_bar.ex
    │   ├── cursor.ex
    │   ├── info_panel.ex
    │   ├── pagination.ex
    │   └── top_bar.ex
    ├── help.ex
    ├── helpers
    │   └── text.ex
    ├── lists.ex
    ├── search
    │   └── state.ex
    └── search.ex

In the views/ directory there is a module per tab, so we have
search.ex, aliases.ex, bookmarks.ex and help.ex. Each view has a
state management module, eg views/bookmarks/state.ex and where
applicable a module for actions. The actions handle side-effects such as
the interaction with the server and the cache.

The main entrypoint for the application is app.ex. It's rather brief
so it fits in the snippet below:

defmodule TefterCli.App do
  @behaviour Ratatouille.App

  alias Ratatouille.Runtime.{Subscription}
  alias TefterCli.App.State
  alias TefterCli.Views.{Search, Bookmarks, Lists, Aliases, Authentication, Help}

  @tabs [:search, :aliases, :bookmarks, :help]

  @impl true
  def init(_), do: State.init()

  @impl true
  def update(state, msg), do: State.update(state, msg)

  @impl true
  def render(%{token: nil} = state), do: Authentication.render(state)
  def render(%{tab: :search} = state), do: Search.render(state)
  def render(%{tab: :bookmarks} = state), do: Bookmarks.render(state)
  def render(%{tab: :aliases} = state), do: Aliases.render(state)
  def render(%{tab: :lists} = state), do: Lists.render(state)
  def render(%{tab: :help} = state), do: Help.render(state)

  @doc "Returns the available application tabs"
  def tabs, do: @tabs

  @impl true
  def subscribe(%{token: nil}), do: Subscription.interval(500, :check_token)
  def subscribe(_), do: Subscription.interval(100_000, :check_token)
end

It's placed under the supervision tree with:

{
  Ratatouille.Runtime.Supervisor,
  runtime: [app: TefterCli.App, quit_events: [{:key, Ratatouille.Constants.key(:ctrl_q)}]]
}

where we declare the "main" module of the app and that ctrl + q quits.

The most important functions of TefterCli.App are update/2 and render/1.

The update/2 receives the model - the current state of the app, as
the first argument and a message as the seconds one. The message is
usually a tuple {:event, event} where event is a termbox mouse or keyboard event like the following:

%ExTermbox.Event{
  ch: 0,
  h: 0,
  key: 27,
  mod: 0,
  type: 1,
  w: 0,
  x: 0,
  y: 0
}

The update/2 should return the updated state, but in some cases you
may have it return {model(), Command.t()}.
More about commands later.
The render/1 receives the model and must return a %Ratatouille.Element{}.
Thankfully you don't have to assemble the element structs manually and
there are macros for that. Example:

def render(model) do
  view do
    label(content: "Hello, #{model.name}!")
  end
end

In TefterCli view modules like TefterCli.Views.Bookmarks define the render/1 function and
delegate the update/2 to their state management modules like TefterCli.Views.Bookmarks.State.
At the moment, the model in TefterCli is a plain map, but will be refactored to be a struct in the future.

Anatomy of the view

Most views share the TopBar and BottomBar. Views with paginated resources have the InfoPanel which displays pagination info the permits typing commands.

Aliases

What is an alias?

Think of an alias as a dynamic shortened link. You can create a maps
alias pointing to https://www.google.com/maps/search/{{*}}?hl=en&source=opensearch and then with the browser extension installed,
type go/maps/london in the address bar to be redirected to
https://www.google.com/maps/search/london?hl=en&source=opensearch.

So with {{*}} you can set dynamic segments in your shortened links.
Dynamic segments are optional though.

In the command-line app you can:

View all the aliases you've created
Create an alias with the :c <alias> <url> command
Delete an alias with the :d command
Search for an alias by typing /
Open a browser window with the link of an alias by pressing enter

Example:

Bookmarks

The bookmarks tab is very similar to aliases. A user is likely to have way more bookmarks than aliases,
since on average a user has ~1000 bookmarks but fewer than 10 aliases.
This calls for a different pagination strategy. In aliases, there's a
sliding viewport with an offset controlled by the cursor. This doesn't
work well with bookmarks. I tested it initially with my bookmarks (I
have more that 9K bookmarks) and it was sluggish. The reason is, that on
every keyboard / mouse event, Ratatouille tries to re-render everything.
In the case of thousands of bookmarks within a viewport, it renders each
and every bookmark despite most of them being off-screen. The solution
is to only feed a slice of the bookmarks list to the viewport.

On a similar note, Ratatouille will re-render everything every
500ms, see here.

Adding Bookmarks

One can add a bookmark by typing :c <url>.

Deleting Bookmarks

To delete a bookmark, type :d and the currently selected bookmark will
be deleted.

Filtering

To filter, type /. To highlight a result, TefterCli.Views.Helpers.Text.highlight/2 is used.
Unfortunately, ExTermbox seems to drop diacritical marks from strings which would let me
render highlighted text like this "z̲o̲r̲b̲a̲s̲h̲" (see: TefterCli.Views.Helpers.Text.underline/1)
and I resorted in surrounding matching text with [ and ].

Development

Clone the repo, run it with iex -S mix, make changes and
you're welcome to submit a pull-request!

Since the app takes over your IEx session, to simplify your debugging,
most events are logged to a file in log/dev.log in development.

To drop to the IEx console, you can go to the search
tab and hit ctrl + y.

To reload the source without restarting the app hit f5.

Packaging

There are two simple bash scripts to prepare releases for Linux and MacOS
in ./bin/release_linux and ./bin/release_macos respectively.
They both use mix release to prepare a tarball which bundles the Erlang VM.
The Linux script leverages Docker to ensure that a release can be built
even on a non-Linux machine.

The framework - Ratatouille

Ratatouille is impressive. It makes you want to write something in it
and it's well documented and it's also rather simple. One can read its
source in one go.

Architecture

There's the view, which is better explained quoting the documentation.

In Ratatouille, a view is simply a tree of elements. Each element in the tree
holds an attributes map and a list of zero or more child nodes. Visually, it
looks like something this:

%Element{
  tag: :view,
  attributes: %{},
  children: [
    %Element{
      tag: :row,
      attributes: %{},
      children: [
        %Element{tag: :column, attributes: %{size: 4}, children: []},
        %Element{tag: :column, attributes: %{size: 4}, children: []},
        %Element{tag: :column, attributes: %{size: 4}, children: []}
      ]
    }
  ]
}

Then there's the runtime, which is basically this:

defp loop(state) do
  :ok = Window.update(state.window, state.app.render(state.model))

  receive do
    {:event, %{type: @resize_event} = event} ->
      state
      |> process_update({:resize, event})
      |> loop()

    {:event, event} ->
      if quit_event?(state.quit_events, event) do
        shutdown(state)
      else
        state
        |> process_update({:event, event})
        |> loop()
      end

    {:command_result, message} ->
      state
      |> process_update(message)
      |> loop()
  after
    state.interval ->
      state
      |> process_subscriptions()
      |> loop()
  end
end

I'd change Ratatouille.Runtime to be a GenServer for
plenty of reasons, introspection with :sys being one of them.

Caveats

Ratatouille is fantastic, but there are a few things that could be improved:

Performs unnecessary re-renderings
Lack of form controls

What's Next

Debian and Homebrew packages
Windows support (mention the desktop app)
Lists
Organisations
Create and delete aliases
Display notes in an overlay
Edit a bookmark Trigger the edit mode with the :e command. Then open $EDITOR with a tempfile containing the JSON representation of the bookmark. When the editor is closed update the bookmark
Import chrome / firefox bookmarks

Thank you

I want to thank ndreynolds for creating Ratatouille and I hope people will gain something
by reading this post and the source of the app and provide feedback!

A Slack bookmarking application in Elixir with Opus

Dimitris Zorbas — Wed, 23 Oct 2019 00:00:00 +0000

This post describes how we used Elixir and Opus in one of our services at
Tefter, which implements bookmarking collaboration in Slack.

My relationship with Slack

I remember, when Slack started getting viral and it was set as the main
chat app at work, I was very reluctant to use it. I was quite happy with IRC and always in favour of open protocols. Since it supported an IRC / XMPP gateway, tweaking my irssi config and later finch was trivial and my overall experience was good. Later I developed my first Slack apps to experiment, accomplish trivial tasks and participate in company hackathons.

Tefter

Recently at Tefter, we released a new organisations feature. This feature, gives users the ability to collaborate within a Slack workspace.

Essential commands of the Slack app:

Create a bookmark

/tefter <url>

Create an alias

/tefter alias <alias> <url>

Search

/tefter search <query>

# Alternatively
/tefter s <query>

This is what a search looks like:

Retrieve a link by alias

/tefter <alias>

Aliases are especially useful for recurring questions concerning a link. For example
"What is the API documentation page for service x?". There you can
create a "docs" alias and point people to the link by calling /tefter docs.

Microlith

The microservice dealing with that side of our system is named Microlith. That is to contradict
its tendency to become a monolith 🙈. It is written in Elixir and it
leverages a library for railway-oriented programming called Opus.
Surprisingly I've never blogged about this tiny library of mine before, but a few others have. It incorporates some software design principles I keep close to my heart.

The main principles of Opus are:

Each Opus pipeline module has a single entry point and returns tagged tuples {:ok, value} | {:error, error}
A pipeline is a composition of stateless stages
A stage returning {:error, _} halts the pipeline
A stage may be skipped based on a condition function (:if and :unless options)
Exceptions are converted to {:error, error} tuples by default
An exception may be left to raise using the :raise option
Each stage of the pipeline is instrumented. Metrics are captured automatically (but can be disabled).
Errors are meaningful and predictable

In this post, I'll show you some code examples from Microlith where Opus is used.

The great thing about Opus is that a use-case can be described as a
series of stages. Similar to your grandma's beef stew recipe.

Creating a bookmark with Opus

So the "recipe" to create a bookmark from Slack is:

check that the payload has the correct format
check that the payload contains a URL to bookmark
normalise the URL
retrieve the Tefter account by its Slack identifier
check that the account can create a bookmark
create the bookmark
respond to the user

Our next move will be to translate this to pseudo-code in Opus terms.

check :valid_payload?
check :payload_contains_url?
step :normalize_url
step :fetch_user
check :can_create_bookmark?
step :create_bookmark
step :respond

A quick rundown of the available stages of Opus.

step: This stage processes the input value and with a success value the next stage is called with that value.
With an error value the pipeline is halted and an {:error, any} is returned.
check: This stage is intended for validations. It calls the stage function and unless it returns true, it halts the pipeline.
tee: This stage is intended for side effects, such as a notification or a call to an external system where the return value is not meaningful. It never halts the pipeline.
link: This stage is to link with another Opus.Pipeline module. It calls call/1 for the provided module. If the module is not an Opus.Pipeline it is ignored.
skip: The skip macro can be used for linked pipelines. A linked pipeline may act as a true bypass, based on a condition,
expressed as either :if or :unless. When skipped, none of the stages are executed and it returns the input,
to be used by any next stages of the caller pipeline.

Now we can define our Opus.Pipeline module.

defmodule Microlith.Commands.CreateBookmark do
  @moduledoc "Pipeline which handles the bookmark command to create a bookmark"

  use Opus.Pipeline

  alias Microlith.Pipelines.FetchUser

  check :valid_payload?, error_message: :invalid_payload
  check :contains_url?, error_message: "Command called without a URL"
  step :trim_url, with: &%{&1 | url: String.trim(&1[:url])}
  step :fetch_user
  check :can_create_bookmark?
  step :create_bookmark
  step :respond
end

The module we just defined, can be used as follows:

payload = %{
  input: "https://zorbash.com",
  slack_user_id: "a Slack user identifier",
  team_id: "a Slack team identifier",
  team_domain: "whitehouse",
  response_url: "https://hooks.api.slack.com/deadbeef"
}

case Microlith.Commands.CreateBookmark.call(payload) do
  {:ok, response} ->
    json(response)
  {:error, error} ->
    Logger.warn(inspect(error))
    send_resp(conn, 422, "The request could not be accepted")
end

Concurrency

Like any decent cooking recipe, the implementation is left to the chef.
You may want to crack 6 eggs with one hand while stirring some sauce
with the other, but the end-result should be the same.

Thankfully the Elixir toolset is very well equipped with facilities to make operations in a pipeline concurrent. In most cases all we have to do is to start Tasks and pass them down to next stages. When a stage requires the result of a Task, Task.await/1 can be used.

Visualising Pipelines

I kept my favourite part for last.

With Opus you can visualise your pipelines using Opus.Graph using Opus.Graph.generate/1.

So at some point in the development of Microlith, it looked like this:

Protip: You should prefer the SVG output, where you can hover on stages and pipelines to read their documentation.

Closing Thoughts

I hope that this post gives an idea of the features of Opus and I promise to cover the next of them in a following post.
If you're using Opus, I'd be glad to hear your feedback.

Do you represent an open-source community and you're interested to try
out Tefter Organizations? Let me know and we'll add you to an unlimited
plan without any cost.

DEV Community: Dimitris Zorbas

A Livebook Smart-Cell to Render Diagrams

Yes, we can!

Prior Art

How it Works

Making it Smart

Finishing Touches

Diagram Samples

Outro

A Guide to Secure Elixir Package Updates

List Updatable Dependencies in Your Elixir App

Check for Outdated Dependencies with Hex

Inspecting Changes with Hex

Browsing Changelogs

Automated Changelog Fetching

Updating Dependencies in Elixir

Further Reading: Keep Your Package Updates Safe in Elixir

A personal Brain with Nerves and LiveBook (podcast)

Livebook Animations

Kino.animate/3

Kino.Frame

Life

An Unusual Pomodoro Timer on Elixir and Nerves

Finished Product

Goal

Name

Optional Features

Why E Ink?

What you'll need

Instructions

1. Burn the Nerves Livebook Firmware

Install prerequisites for packaging firmware images

Burn the Image

2. Mount the E Ink Screen Hat on the Pi

3. Clone the Brain Repo

4. Add Some Notes

5. Deploy! 🚀

Making Changes

Debugging

Caveats

Future Enhancements

Livebook

Acknowledgements

Organising Book Highlights and Notes

Why care about my notes and Highlights?

So.. I built a thing

Demo Time

sync_local

search

random

update_notion

Recap

Features

Non-Features

Contributing

Further Reading

Phoenix Telemetry

What is Telemetry

How it Works

Example

How it Works

Phoenix Telemetry

Opus Telemetry

Live Dashboard

Upcoming Blog post

Further Reading

sidekiq-dry

Rationale

Prevent Type Ambiguity

Eliminate Positional Arguments

Documentation

Versioning

Caveats

The Gem

Installation

Further Reading

Knowledge Mapping

Learning Together

Learning to Learn

Universities

`Kino.Frame`

`sync_local`