DEV Community: Sergey Fedorov

Where is my ETS table!?

Sergey Fedorov — Tue, 28 Jan 2020 22:08:01 +0000

Foreword

In a perfect world, everyone will read the documentation beforehand.

TL;DR

While working on the Elixir project Avrora (convenient encoding and decoding of Avro messages) I've received an issue with testing environment.

Which was odd to me because it turns out that some ETS references become invalid during a simple test case.

A very straightforward Phoenix controller

defmodule AvroraIssueWeb.PageController do
  use AvroraIssueWeb, :controller

  def index(conn, _params) do
    {:ok, _} = Avrora.encode(%{"login" => "fxn"}, schema_name: "com.example.User") |> IO.inspect()
    json(conn, %{})
  end
end

fail to pass dead simple test cases

defmodule AvroraIssueWeb.PageControllerTest do
  use AvroraIssueWeb.ConnCase

  test "1", %{conn: conn} do
    get(conn, "/")
    assert true
  end

  test "2", %{conn: conn} do
    get(conn, "/")
    assert true
  end
end

If we look inside the method Avrora.encode/2, we will find that it triggers JSON string parsing, which will create Elixir struct with a reference to the created ETS table. The result of the Avrora.encode/2 will be cached.

But how my ETS reference in the cache becomes invalid? And why?

Let's take a look at what ETS is?

It's an Erlang Built-In Term Storage which provides the ability to store very large quantities of data in an Erlang runtime system and to have constant access time to the data (depends on the table type).

In Elixir you can access it by using the :ets module.

If we continue reading official Erlang documentation, we will find one important sentence:

Notice that there is no automatic garbage collection for tables. Even if there
are no references to a table from any process, it is not automatically destroyed
unless the owner process terminates

ETS must have an owner process

To illustrate that ETS behavior let's take a look at this example:

# file: where_is_my_ets.ex

# This is a wrapper around the :ets module.
# We will use named tables to visualize it later.
defmodule Table do
  def new(name) do
    :ets.new(name, [:named_table])
  end
end

# This is a GenServer which can create named tables
# by calling Table.new/1
defmodule Owner do
  use GenServer

  @impl true
  def init(state) do
    {:ok, state}
  end

  @impl true
  def handle_call({:new, name}, _from, state) do
    {:reply, Table.new(name), state}
  end
end

Now let's start IEx session and try this code via iex where_is_my_ets.ex. Notice that IEx session is also a process, hence all created tables right in the REPL will belong to that process, let's give it a try:

# Let's remember the PID of a REPL session
iex(1)> self
#PID<0.113.0>

# Now let's check what kind of ETS tables already exist
iex(2)> :ets.all
[:logger, :ac_tab, #Reference<0.1096493602.1531314178.170593>, ...,
 :elixir_config, :elixir_modules, IEx.Config, IEx.Pry]

# Creating new table with name :repl
iex(3)> Table.new(:repl)
:repl

# Who is the owner?
iex(4)> :ets.info(:repl)
[
  id: #Reference<0.1096493602.1531314178.171287>,
  read_concurrency: false,
  write_concurrency: false,
  compressed: false,
  memory: 311,
  owner: #PID<0.113.0>, # <--- This is our REPL session!
  heir: :none,
  name: :repl,
  size: 0,
  node: :nonode@nohost,
  named_table: true,
  type: :set,
  keypos: 1,
  protection: :protected
]

# Now we are able to see :repl table in the list of all tables
iex(5)> :ets.all
[:logger, :ac_tab, #Reference<0.1096493602.1531314178.170593>, ...,
 :elixir_config, :elixir_modules, IEx.Config, IEx.Pry, :repl]

# Starting Owner process without linking to REPL process
iex(6)> {:ok, pid} = GenServer.start(Owner, [])
{:ok, #PID<0.123.0>}

# And create a new table in REPL, but by different process
iex(7)> GenServer.call(pid, {:new, :hello})
:hello

# Who is the owner of that table?
iex(8)> :ets.info(:hello)
[
  id: #Reference<0.1096493602.1531314178.171427>,
  read_concurrency: false,
  write_concurrency: false,
  compressed: false,
  memory: 311,
  owner: #PID<0.123.0>, # <--- This is our Owner process
  heir: :none,
  name: :hello,
  size: 0,
  node: :nonode@nohost,
  named_table: true,
  type: :set,
  keypos: 1,
  protection: :protected
]

# We can see :hello table too
iex(9)> :ets.all
[:logger, :ac_tab, #Reference<0.1096493602.1531314178.170593>, ...,
 :elixir_config, :elixir_modules, IEx.Config, IEx.Pry, :repl, :hello]

# What will happen if process Owner will die?
iex(10)> Process.exit(pid, [])
true

# Table :hello gone ...
iex(11)> :ets.info(:hello)
:undefined

# Also from the "all" list
iex(12)> :ets.all
[:logger, :ac_tab, #Reference<0.1096493602.1531314178.170593>, ...,
 :elixir_config, :elixir_modules, IEx.Config, IEx.Pry, :repl]

So what was the problem with tests?

In the Phoenix framework, every action in the controller will spawn a process to handle the request. Inside that process, an ETS table was created and its reference stored in a shared cache.

But once an action is done, the ETS table is wiped and the reference becomes invalid. To solve this issue I delegate ETS creation to a separate process that becomes their owner instead of short-living Phoenix controller.

Some useful links

What I miss in Github Actions pipeline?

Sergey Fedorov — Thu, 17 Oct 2019 11:55:49 +0000

You might have heard that Github launched their awesome Github Actions now for everyone?

Nat Friedman

@natfriedman

Good news: anyone who signs up for the GitHub Actions beta will get admitted instantly. Give it a try! github.com/features/actio…

03:10 AM - 18 Sep 2019

247 674

Github Actions brings a huge potential for automation of almost any kind of routines.

And the first thing I tried was a bot. I like the idea of interactive automation via commands, be it in Slack or Github. But if you can't run your server with a bot, Github Actions comes to the rescue!

The idea

Let's say you want to have a great GIPHY bot which can handle commands from the issue or pull request comments and its syntax could look like this:

~giphy cat

The workflow of the Github Action can be described as:

Trigger when the issue gets a new comment
Extract the command and its arguments from the comment's body
Query api.giphy.com and fetch the first matching picture
Post a comment back with the gif

The action

As an entry point, the official documentation is a good place to start.

Let's say we know how to define a skeleton of the action which should trigger on a new comment

---
name: A GIPHY bot
on: issue_comment
jobs:
  post_image:
    name: Post an image from the search
    runs-on: ubuntu-latest
    steps:
      # ... our steps ...

After our skeleton is defined we need to set the steps that will be executed. According to the workflow we describe, after a new issue comment appears, we need to parse the message body and then pass the result to the next step. Sounds familiar, right?

It sounds like a Linux pipe operator and if I needed to do this in a console, I would do something like this

$ echo "~giphy cat" | \
  grep --color=never -oP "(?<=~giphy\s)[\w\d\s]+" | \
  xargs -I{} curl "https://api.giphy.com/v1/gifs/search?api_key=<API Key>&limit=1&rating=g&q={}"

Ok, let's try it ...

The fail

Execute a command, take its result and pass to the next step, sounds easy, right? According to the documentation of the spec context, you should identify your step and then you can use its output like this

---
# ... omitted ...

steps:
  - name: Step 1
    id: step_1
    run: echo "Hello World"
    shell: bash

  - name: Step 2
    id: step_2
    run: echo "${{ steps.step_1.outputs.<output name> }}"
    shell: bash

What to put in as the "output name" you ask? I asked myself the same question. At first glance, I would expect to have like the Linux standards streams here

Name	File descriptor	Abbreviation
Standard input	0	stdin
Standard output	1	stdout
Standard error	2	stderr

But if you try to use it, you will get an error:

---
# ... omitted ...

steps:
  - name: Step 1
    id: step_1
    run: echo "Hello World"
    shell: bash

  - name: Step 2
    id: step_2
    run: echo "Simon says - ${{ steps.step_1.outputs.0 }}"
    shell: bash

### ERRORED 09:27:44Z

- Your workflow file was invalid: The pipeline is not valid.
  .github/workflows/giphy.yml (Line: 16, Col: 14): Unexpected symbol: '0'. 
  Located at position 29 within expression: steps.parse.outputs.0

Seems it expects letters for the output name, which I can understand because your command can set more than one value for its execution result.

Surely we can write a custom action and use the action API to set the output, but I just want to have a dead-simple pipelining and less code to maintain.

The help comes from Google. Looks like I was not the only one who fails to get step output. Same question had been asked in the Github Community and some other blog posts

https://jasonet.co/posts/new-features-of-github-actions/
These methods use functionality of the runtime that isn’t documented (I had to read through some code to figure out how it was working), so it may feel a little magical. Now, if you’re like me, you’re thinking “Can this be done without the toolkit or JavaScript?” Turns out, yes!

Turns out, that to set the result you should use the yet undocumented functionality of the commands which looks like this

/**
 * Commands
 *
 * Command Format:
 *   ##[name key=value;key=value]message
 *
 * Examples:
 *   ##[warning]This is the user warning message
 *   ##[set-secret name=mypassword]definitelyNotAPassword!
 */

Let's give it a try:

---
# ... omitted ...

steps:
  - name: Step 1
    id: step_1
    run: echo "##[set-output name=hello;]Hello World"
    shell: bash

  - name: Step 2
    id: step_2
    run: echo "Simon says - ${{ steps.step_1.outputs.hello }}"
    shell: bash

2019-10-16T17:27:03.2937669Z Current runner version: '2.159.0'
2019-10-16T17:27:03.2956102Z Prepare workflow directory
2019-10-16T17:27:03.4191005Z Prepare all required actions
2019-10-16T17:27:03.5977738Z ##[group]Run echo "##[set-output name=hello;]Hello World"
2019-10-16T17:27:03.5978178Z echo "##[set-output name=hello;]Hello World"
2019-10-16T17:27:04.3506195Z shell: /bin/bash --noprofile --norc -e -o pipefail {0}
2019-10-16T17:27:04.3506360Z ##[endgroup]
2019-10-16T17:27:04.4259015Z ##[set-output name=hello;]Hello World
2019-10-16T17:27:04.4740495Z ##[group]Run echo "Simon says - Hello World"
2019-10-16T17:27:04.4740667Z echo "Simon says - Hello World"
2019-10-16T17:27:04.4778909Z shell: /bin/bash --noprofile --norc -e -o pipefail {0}
2019-10-16T17:27:04.4779052Z ##[endgroup]
2019-10-16T17:27:04.4858165Z Simon says - Hello World
2019-10-16T17:27:04.4936789Z Cleaning up orphan processes

Ok, now we know how to set output! I will show the result of the workflow with the steps pipelining and a use case of extra actions, like actions/github-script.

---
name: A GIPHY reply bot
on: issue_comment
jobs:
  post_image:
    name: Post an image from the GIPHY search
    runs-on: ubuntu-latest
    steps:
      - name: Parse command
        id: parse
        shell: bash
        run: |
          echo "##[set-output name=query;]$(echo "${{ github.event.comment.body }}" | grep --color=never -oP "(?<=~giphy\s)[\w\d\s]+")"

      - name: Search picture
        id: search
        shell: python
        run: |
          import json
          from urllib import urlopen

          url = 'https://api.giphy.com/v1/gifs/search?limit=1&rating=g&api_key=${{ secrets.API_KEY }}&q=${{ steps.parse.outputs.query }}'
          body = json.loads(urlopen(url).read())

          print('##[set-output name=url;]' + body['data'][0]['images']['fixed_width']['url'])

      - name: Reply with a picture
        uses: actions/github-script@0.2.0
        with:
          github-token: ${{ github.token }}
          script: |
            return github.issues.createComment({
              ...context.issue, body: "![giphy](${{ steps.search.outputs.url }})"
            });

NOTE: Don't forget to set the repository secret API_KEY with your giphy application key if you want to try this workflow

Thoughts

I like the new YAML syntax. It feels more logical and simple. I see the constant improvements over the GUI of the Actions page.

I wish that Github makes steps more like a Linux pipeline, which will allow you to easily transfer your knowledge of working in the console to the Github Actions.

At the same time, I understand the need to create custom Actions to satisfy different needs, but listen, somehow GNU manages to have an awesome set of composable commands and so can we, too!

Here is the repository with the code samples

Strech / giphy-bot-example

An example of pipelining in Github actions

A Github Actions GIPHY bot

If you would like to see how bot is reacting on your messages, go to this issue and leave a comment like

~giphy grumpy cat

View on GitHub