DEV Community: Contact Stack

Adding a custom watcher to Phoenix

Michael Jones — Wed, 29 Apr 2020 17:47:48 +0000

At Contact-Stack, we've recently added Elm to one of our Phoenix projects. Elm is a fantastic language with a clear syntax, functional design, immutable data and a helpful compiler.

For reasons of mild personal preference, we opted to not use elm-webpack-loader when integrating the Elm code with the current Javascript setup that we already have from Phoenix. Though ideally, we would still like the experience we have when we edit the Javascript, ie. it is rebuilt and Phoenix reloads the current browser with the newly built assets.

Unfortunately for us the Elm compiler doesn't have a 'watch' mode so we can't rely on that. We need a separate process to run the Elm compiler whenever there is a change. I cannot find it now but I've seen comments from Richard Feldman, a prominent member of the Elm community, suggesting that the Chokidar project can be used to set up a simple watcher which runs the Elm compiler. Chokidar is a node project that does a great job of wrapping some of the node standard library functionality to provide a robust watcher. It is used by a number of high profile node projects, including Webpack, to provide file watching functionality.

For reference, the exact build command that I would like run is:

elm make src/Main.elm --output=../priv/static/js/elm.js

From within the assets directory in the standard Phoenix project layout.

Now to start, we might consider adding the chokidar-cli which allows you to set up watchers with a simple command. We can add chokidar-cli with yarn add -D chokidar and then run:

chokidar "**/*.elm" -i node_modules -c "elm make src/Main.elm --output=../priv/static/js/elm.js"

If we run that in the assets directory it works great so maybe that's a good start. We add -i node_modules as for some reason chokidar starts tracking some files in node_modules too and we don't need it to.

How do we go about adding it to Phoenix? If we look in the config/dev.exs we see a block that looks like this:

config :contact_stack, ContactStackWeb.Endpoint,                                                                                                           
  http: [port: 4000],                                                                                                                                      
  debug_errors: true,                                                                                                                                      
  code_reloader: true,                                                                                                                                     
  check_origin: false,                                                                                                                                     
  watchers: [                                                                                                                                              
    node: [                                                                                                                                                
      "node_modules/webpack/bin/webpack.js",                                                                                                               
      "--mode",                                                                                                                                            
      "development",                                                                                                                                       
      "--watch-stdin",                                                                                                                                     
      cd: Path.expand("../assets", __DIR__)                                                                                                                
    ]                                                                                                                                                      
  ]

The relevant entry, as you might guess, is the watchers list. This is a list of key-value pairs that each provide a program and a set of arguments for Phoenix to run as part of its watcher functionality. So in this case, it is going to run node with that list of arguments which will result in it running webpack in development mode. The last part is to ask Phoenix to run it in the assets directory.

So we could try to extend this way:

   watchers: [
     node: [
       "node_modules/webpack/bin/webpack.js",
       "--mode",
       "development",
       "--watch-stdin",
       cd: Path.expand("../assets", __DIR__)
+    ],
+    node: [
+      "node_modules/.bin/chokidar",
+      "**/*.elm",
+      "-i",
+      "node_modules",
+      "-c",
+      "elm make src/Main.elm --output=../priv/static/js/elm.js",
+      cd: Path.expand("../assets", __DIR__)
     ]
   ]

And actually, this seems to work great. We run this and, every time we save an Elm file, Phoenix runs the Elm compiler with the command we've provided.

Unfortunately, if we kill the Phoenix server and check the running processes on our machine:

^C
BREAK: (a)bort (A)bort with dump (c)ontinue (p)roc info (i)nfo
       (l)oaded (v)ersion (k)ill (D)b-tables (d)istribution
$ ps -ef | grep chokidar
michael  17499     1  0 16:16 ?        00:00:00 /home/michael/.nave/installed/12.14.0/bin/node node_modules/.bin/chokidar "**/*.elm" -c "elm make src/Main.elm --output=../priv/static/js/elm.js"

We can see that the Chokidar process is still running. This is not great. We want to be able to freely restart our Phoenix dev server as often as we like without building up a back log of Chokidar processes which are all watching our Elm files and each running the Elm compiler on every change.

So why is this happening? Well, I'm guessing that Phoenix uses Elixir's Port functionality to run the watcher subprocesses and Elixir's Ports functionality as a big warning about zombie processes that are left over when the main Elixir system process has stopped. It seems that Elixir doesn't, or perhaps can't, proactivity kill subprocesses that it has started when closing down. Rather it relies on those subprocesses noticing that their standard input has been closed and exiting themselves. I haven't come across this mechanism outside of Elixir but it might well be common. And if we glance up at the webpack watcher config again we'll see that they're using a --watch-stdin flag. Coincidence? Probably not.

Unfortunately, chokidar-cli doesn't have a --watch-stdin flag nor any search results for stdin in the code so it seems like we can't rely on that.

But webpack is written in Javascript running on node so it must be possible and the main chokidar package is a library that allows you to easily access the file-watching functionality.

If we go splunking through the webpack code looking for references to standard input then we come across these lines in the webpack-cli project. Now I don't perfectly understand what is going on here but it seems that it is listening for a 'end' event on the standard input stream and using process.exit() to close the whole program if it happens. That seems to fit with what Elixir's Ports expect.

If we combine that with some relatively basic chokidar library usage as they outline in the README then we get something like this:

const chokidar = require("chokidar");
const { execSync } = require("child_process");

// Exit the process when standard input closes due to:
//   https://hexdocs.pm/elixir/1.10.2/Port.html#module-zombie-operating-system-processes
//
process.stdin.on("end", function() {
    console.log("standard input end");
    process.exit();
});

process.stdin.resume();

// Set up chokidar to watch all elm files and rebuild the elm app ignoring process errors
chokidar.watch("**/*.elm", { ignored: "node_modules" }).on("all", (event, path) => {
    console.log(event, path);
    try {
        execSync("./node_modules/.bin/elm make src/Main.elm --output=../priv/static/js/elm.js");
    } catch (error) {}
});

And if we save it in a file called assets/watch-elm.js. And then we change our config/dev.exs config to read:

    node: [
      "./watch-elm.js",
       cd: Path.expand("../assets", __DIR__)
    ]

Then we can run mix phx.server and see that not only does the Elm compiler get run correctly on changes but when we kill our dev server the watcher process dies too. Success!

Conclusion

Adding new watcher processes to Phoenix is relatively easy in some ways but this matter of watching standard-input is a bit confusing and is probably handled quite differently in different languages.

Notes

The Phoenix documentation does provide a helper bash script that you can use to wrap an executable and which does the 'listening for standard input to close' for you. I have used that successfully when having to run a subprocess during Elixir tests but I wasn't unable to get it working on first try in this situation. Possibly the complexities of the extra escaping of the command line arguments got the better of me. I'm not sure.

Mixing Gleam & Elixir

Michael Jones — Fri, 20 Mar 2020 12:33:26 +0000

Update: This was very much an initial exploration. There is now an official project with a mix_gleam plugin. Please check that out!

At Contact Stack, we've chosen Elixir & Phoenix as the core of our tech stack and we're happy with the choice. But... we do miss having a really good type system and a helpful compiler (like with Elm) to guide us through development.

Elixir targets the BEAM virtual machine and there are few statically typed languages that target the BEAM as well. Here we're going to look into a simple set up for integrating Gleam into your Elixir project using a Mix Compiler Task.

Gleam is a statically typed functional programming language which uses a syntax that is more approachable to, say, Javascript, C or Java programmers than some ML flavoured languages. The Gleam compiler is written in Rust and outputs Erlang source files which can then be compiled to BEAM by the Erlang compiler.

Mix is a multi-functional build tool for Elixir projects. It can manage dependencies, run tests, execute tasks and compile your Elixir code. It also supports compiling erlang files by default and can be extended with Mix Compiler Tasks to compile other languages. We're going to create a Mix Compiler Task for Gleam so that Mix can use the Gleam compiler to build Gleam files.

You can try the steps below yourself and check out the demonstration repository: https://github.com/michaeljones/gleam-phoenix-mix.

Here we go!

We're assuming you have the gleam compiler available in your environment to execute. Run this to check.

gleam --version

If you don't, then you can follow the installation instructions over in the Gleam docs.

Now we're going to go through setting up a Phoenix project from scratch. We won't go into the details but just so we have a clear base to work with.

Run the following commands to set up an empty phoenix project:

mkdir gleam-phoenix-mix
cd gleam-phoenix-mix
mix phx.new . --app my_app
mix ecto.create
cd assets 
npm install
cd ..

Now we need to create our mix compiler task. We put this in a lib/mix/tasks/compile directory to reflect the module name that we need to give it which is Mix.Tasks.Compile.Gleam. So we create our new directory:

mkdir -p lib/mix/tasks/compile/

And add the follow code in a file called lib/mix/tasks/compile/gleam.ex.

defmodule Mix.Tasks.Compile.Gleam do
  use Mix.Task.Compiler

  def run(_args) do
    System.cmd("gleam", ["build"])
    :ok
  end
end

All this is really doing is running the gleam compiler directly with the 'build' argument to get it to build all the gleam files it can find.

Run the elixir compiler so that we compile that mix task before we try to use it.

mix compile.elixir

Now to make gleam happy we need to set up a .toml config file for it. So we create a file called gleam.toml in the root of the project with the follow content.

name = "my_app"

Then we create a src directory for the gleam files. The Gleam compiler expects the gleam files to be in a src directory. This is different to elixir/mix which expects them in the lib directory by default. Elixir/mix can be configured but Gleam can't at the time of writing.

mkdir src

We then create our Gleam module by making a file called src/hello_world.gleam with the following code in it:

pub fn hello() {
  "Hello, from gleam!"
}

Next we make the following change to the mix.exs file to add our compiler task to the list of compilers being run when you do mix compile and to make sure mix's erlang compiler looks in the gen folder
for erlang files. The gleam compiler compiles .gleam files from the src folder into .erl files in the gen folder so we need the erlang compiler to find them there.

       elixir: "~> 1.5",
       elixirc_paths: elixirc_paths(Mix.env()),
+      erlc_paths: ["src", "gen"],
-      compilers: [:phoenix, :gettext] ++ Mix.compilers(),
+      compilers: [:phoenix, :gettext, :gleam] ++ Mix.compilers(),
       start_permanent: Mix.env() == :prod,
       aliases: aliases(),

I assume that Mix runs the compilers from the start to the end of the list. So we want the :gleam entry to be before the :erlang entry that's in Mix.compilers() so that the Erlang compiler will run after the Gleam compile and compile the Erlang files that the Gleam compiler creates.

As a simple demonstration of interop between Elixir & the compiled gleam files, make the following change to lib/my_app_web/controllers/page_controller.ex:

 defmodule MyAppWeb.PageController do
   use MyAppWeb, :controller

   def index(conn, _params) do
-    render(conn, "index.html")
+    render(conn, "index.html", title: :hello_world.hello())
   end
 end

This works because Gleam modules are compiled to Erlang modules and, in Elixir, Erlang modules are accessible via the atom that is their name.

And then we make the following change to lib/my_app_web/templates/page/index.html.eex:

 <section class="phx-hero">
-  <h1><%= gettext "Welcome to %{name}!", name: "Phoenix" %></h1>
+  <h1><%= @title %></h1>
   <p>A productive web framework that<br/>does not compromise speed or maintainability.</p>
 </section>

Finally we can run:

mix compile
mix phx.server

And load http://localhost:4000 in our browser to see "Hello, from gleam!" in the centre of the standard Phoenix welcome page.

Remember, if you're curious you can check out the final result at this demonstration repo on Github: https://github.com/michaeljones/gleam-phoenix-mix

Final Thoughts

It is exciting to see that it isn't too hard to incorporate Gleam code into an Elixir project using Mix. It is great that Mix is built with this kind of extensibility in mind. There are definitely more questions to answer and rough edges to smooth out but it is exciting to have an avenue to using a strongly typed language on the BEAM.

Introducing Dialyzer & type-specs to an Elixir Project

Michael Jones — Tue, 17 Mar 2020 14:35:45 +0000

At Contact Stack we use Elixir & Phoenix as the core of the backend. I am fascinated by the design & power of the BEAM, the virtual machine that Elixir targets, and I want to learn more about it. I also love strongly & statically typed languages, particularly due to my recent history working with Elm which has a great type system and a helpful compiler.

Elixir is dynamically typed language which leaves me a little nervous about the long term future of the project. I really like types and I find them incredibly useful when the code base grows larger and you're returning to parts of it that you haven't looked at in a while. Often I can't remember exactly the kind of data that I'm passing around and strong types really help with that understanding. Plus I've grown to love having a compiler guide you through the code base when doing refactoring as it points out all the places you need to change.

One way to bring a little more type safety to Elixir projects is to use Dialyzer. Dialyzer performs static analysis on your code and tries to warn you about anything that is almost certainly going to fail. It can be helped by providing type annotations and it also derives its own from inspecting the code. It isn't as good as a solid type system but it is better than nothing.

So, how do we go about introducing Dialzyer and type annotations through out our codebase? Here are the steps that I've followed for Contact Stack.

Install Dialyzer

Maybe a bit of an obvious first step! We can use the dialxyir project to add a dialyzer task to mix. At the time of writing that involves adding this line to your mix.exs deps:

{:dialyxir, "~> 1.0", only: [:dev], runtime: false},

Run Dialyzer & Fix issues

We can now run dialyzer with:

mix dialyzer

On the command line. This can take a long time as dialyzer analyses and creates PLT caches for all the Elixir & Erlang standard libraries. Fortunately it is much quicker on that next run though still not as fast a many compilers out there.

The errors can honestly be quite hard to understand. I would recommend googling as much as possible. I've learned a few of things though that I'll share.

Firstly, in my projects I've had to config dialyzer to make it away of ex_unit and mix otherwise it issues warnings. I suspect I only needed to add mix because I have a custom mix task. You can configure dialyzer by adding the following to your the project config in mix.exs:

dialyzer: [plt_add_apps: [:ex_unit, :mix], ignore_warnings: "config/dialyzer.ignore"]

Here we add ex_unit & mix and also indicate an ignore file.

Secondly, some errors don't seem to be clearly fixable by code that is under our control. In which case we want to add those errors to the ignore file. If you run dialyzer with:

mix dialyzer --format dialyzer

Then you can copy the start or all of the error that you want to ignore onto a new line in the ignore file we've just set up and dialyzer will skip it on future runs.

Thirdly, if you get a warning that says that some function has no local return then it normally means there is another warning within the function itself that makes dialzyer think that the function will never work. You can skip the 'no local return' warning at look at the next one in the output so see what the issue might be.

Introduce Credo

Credo is a linting tool for Elixir that can warn if you don't provide a type annotation on a function. Part of our goal here is to make sure that we provide type annotations for all the functions in our code base. This helps Dialyzer understand our intent and is self-documenting which helps ourselves and others on the code base.

So we add:

{:credo, "~> 1.2", only: [:dev, :test], runtime: false}

To our mix.exs deps.

Configure & Fix Credo Warnings

Credo checks for a lot of different things. It does not check for type annotations by default and we're going to enable that but first, as long as they are manageable, fix any current warnings that Credo has in place. You can do this by changing your code or suppressing some of Credo's warnings. I have the following config for my preferences:

%{
  configs: [
    %{
      name: "default",
      files: %{
        included: ["lib/"],
        excluded: []
      },
      checks: [
        {Credo.Check.Design.AliasUsage, false},
        {Credo.Check.Readability.AliasOrder, false}
      ]
    }
  ]
}

Extend Credo Rules to Include Type Annotations

We extend our config to include the Specs check by adding this line:

      checks: [
        {Credo.Check.Design.AliasUsage, false},
        {Credo.Check.Readability.AliasOrder, false}
+       {Credo.Check.Readability.Specs, []}
      ]

Silence the Huge Number of Credo Warnings

Now we can run credo again to get a sense of the number of missing type annotations. If you have a small number then go ahead and try to add them now. If, like me, you have something like 200 warnings or more then you might prefer to silence them for now and attend to them gradually.

To do this we can run the following command:

mix credo --strict | grep lib | cut -d " " -f 8 | cut -d ":" -f 1 | sort -u | xargs sed -i '1s/^/# credo:disable-for-this-file Credo.Check.Readability.Specs\n/'

This is going to add # credo:disable-for-this-file Credo.Check.Readability.Specs to the top of every file in your codebase that has missing type annotations.

When we run mix credo --strict again we should see a clean sheet. Now you can go into the files at your leisure and remove the comment at the top and add type annotations. Credo will warn about any that are missing and warn by default for any new files we add to the system.

Final Thoughts

I worry that dialyzer is not very useful in comparison to the type systems & compilers that I have used before but it feels good to be trying to make the code as tight and error free as possible. I am at the beginning of this journey so I can't really report on whether it is worth it long term but we'll see.