DEV Community: Michael Jones

Adding a custom watcher to Phoenix

Michael Jones — Wed, 29 Apr 2020 17:47:48 +0000

At Contact-Stack, we've recently added Elm to one of our Phoenix projects. Elm is a fantastic language with a clear syntax, functional design, immutable data and a helpful compiler.

For reasons of mild personal preference, we opted to not use elm-webpack-loader when integrating the Elm code with the current Javascript setup that we already have from Phoenix. Though ideally, we would still like the experience we have when we edit the Javascript, ie. it is rebuilt and Phoenix reloads the current browser with the newly built assets.

Unfortunately for us the Elm compiler doesn't have a 'watch' mode so we can't rely on that. We need a separate process to run the Elm compiler whenever there is a change. I cannot find it now but I've seen comments from Richard Feldman, a prominent member of the Elm community, suggesting that the Chokidar project can be used to set up a simple watcher which runs the Elm compiler. Chokidar is a node project that does a great job of wrapping some of the node standard library functionality to provide a robust watcher. It is used by a number of high profile node projects, including Webpack, to provide file watching functionality.

For reference, the exact build command that I would like run is:

elm make src/Main.elm --output=../priv/static/js/elm.js

From within the assets directory in the standard Phoenix project layout.

Now to start, we might consider adding the chokidar-cli which allows you to set up watchers with a simple command. We can add chokidar-cli with yarn add -D chokidar and then run:

chokidar "**/*.elm" -i node_modules -c "elm make src/Main.elm --output=../priv/static/js/elm.js"

If we run that in the assets directory it works great so maybe that's a good start. We add -i node_modules as for some reason chokidar starts tracking some files in node_modules too and we don't need it to.

How do we go about adding it to Phoenix? If we look in the config/dev.exs we see a block that looks like this:

config :contact_stack, ContactStackWeb.Endpoint,                                                                                                           
  http: [port: 4000],                                                                                                                                      
  debug_errors: true,                                                                                                                                      
  code_reloader: true,                                                                                                                                     
  check_origin: false,                                                                                                                                     
  watchers: [                                                                                                                                              
    node: [                                                                                                                                                
      "node_modules/webpack/bin/webpack.js",                                                                                                               
      "--mode",                                                                                                                                            
      "development",                                                                                                                                       
      "--watch-stdin",                                                                                                                                     
      cd: Path.expand("../assets", __DIR__)                                                                                                                
    ]                                                                                                                                                      
  ]

The relevant entry, as you might guess, is the watchers list. This is a list of key-value pairs that each provide a program and a set of arguments for Phoenix to run as part of its watcher functionality. So in this case, it is going to run node with that list of arguments which will result in it running webpack in development mode. The last part is to ask Phoenix to run it in the assets directory.

So we could try to extend this way:

   watchers: [
     node: [
       "node_modules/webpack/bin/webpack.js",
       "--mode",
       "development",
       "--watch-stdin",
       cd: Path.expand("../assets", __DIR__)
+    ],
+    node: [
+      "node_modules/.bin/chokidar",
+      "**/*.elm",
+      "-i",
+      "node_modules",
+      "-c",
+      "elm make src/Main.elm --output=../priv/static/js/elm.js",
+      cd: Path.expand("../assets", __DIR__)
     ]
   ]

And actually, this seems to work great. We run this and, every time we save an Elm file, Phoenix runs the Elm compiler with the command we've provided.

Unfortunately, if we kill the Phoenix server and check the running processes on our machine:

^C
BREAK: (a)bort (A)bort with dump (c)ontinue (p)roc info (i)nfo
       (l)oaded (v)ersion (k)ill (D)b-tables (d)istribution
$ ps -ef | grep chokidar
michael  17499     1  0 16:16 ?        00:00:00 /home/michael/.nave/installed/12.14.0/bin/node node_modules/.bin/chokidar "**/*.elm" -c "elm make src/Main.elm --output=../priv/static/js/elm.js"

We can see that the Chokidar process is still running. This is not great. We want to be able to freely restart our Phoenix dev server as often as we like without building up a back log of Chokidar processes which are all watching our Elm files and each running the Elm compiler on every change.

So why is this happening? Well, I'm guessing that Phoenix uses Elixir's Port functionality to run the watcher subprocesses and Elixir's Ports functionality as a big warning about zombie processes that are left over when the main Elixir system process has stopped. It seems that Elixir doesn't, or perhaps can't, proactivity kill subprocesses that it has started when closing down. Rather it relies on those subprocesses noticing that their standard input has been closed and exiting themselves. I haven't come across this mechanism outside of Elixir but it might well be common. And if we glance up at the webpack watcher config again we'll see that they're using a --watch-stdin flag. Coincidence? Probably not.

Unfortunately, chokidar-cli doesn't have a --watch-stdin flag nor any search results for stdin in the code so it seems like we can't rely on that.

But webpack is written in Javascript running on node so it must be possible and the main chokidar package is a library that allows you to easily access the file-watching functionality.

If we go splunking through the webpack code looking for references to standard input then we come across these lines in the webpack-cli project. Now I don't perfectly understand what is going on here but it seems that it is listening for a 'end' event on the standard input stream and using process.exit() to close the whole program if it happens. That seems to fit with what Elixir's Ports expect.

If we combine that with some relatively basic chokidar library usage as they outline in the README then we get something like this:

const chokidar = require("chokidar");
const { execSync } = require("child_process");

// Exit the process when standard input closes due to:
//   https://hexdocs.pm/elixir/1.10.2/Port.html#module-zombie-operating-system-processes
//
process.stdin.on("end", function() {
    console.log("standard input end");
    process.exit();
});

process.stdin.resume();

// Set up chokidar to watch all elm files and rebuild the elm app ignoring process errors
chokidar.watch("**/*.elm", { ignored: "node_modules" }).on("all", (event, path) => {
    console.log(event, path);
    try {
        execSync("./node_modules/.bin/elm make src/Main.elm --output=../priv/static/js/elm.js");
    } catch (error) {}
});

And if we save it in a file called assets/watch-elm.js. And then we change our config/dev.exs config to read:

    node: [
      "./watch-elm.js",
       cd: Path.expand("../assets", __DIR__)
    ]

Then we can run mix phx.server and see that not only does the Elm compiler get run correctly on changes but when we kill our dev server the watcher process dies too. Success!

Conclusion

Adding new watcher processes to Phoenix is relatively easy in some ways but this matter of watching standard-input is a bit confusing and is probably handled quite differently in different languages.

Notes

The Phoenix documentation does provide a helper bash script that you can use to wrap an executable and which does the 'listening for standard input to close' for you. I have used that successfully when having to run a subprocess during Elixir tests but I wasn't unable to get it working on first try in this situation. Possibly the complexities of the extra escaping of the command line arguments got the better of me. I'm not sure.

Mixing Gleam & Elixir

Michael Jones — Fri, 20 Mar 2020 12:33:26 +0000

Update: This was very much an initial exploration. There is now an official project with a mix_gleam plugin. Please check that out!

At Contact Stack, we've chosen Elixir & Phoenix as the core of our tech stack and we're happy with the choice. But... we do miss having a really good type system and a helpful compiler (like with Elm) to guide us through development.

Elixir targets the BEAM virtual machine and there are few statically typed languages that target the BEAM as well. Here we're going to look into a simple set up for integrating Gleam into your Elixir project using a Mix Compiler Task.

Gleam is a statically typed functional programming language which uses a syntax that is more approachable to, say, Javascript, C or Java programmers than some ML flavoured languages. The Gleam compiler is written in Rust and outputs Erlang source files which can then be compiled to BEAM by the Erlang compiler.

Mix is a multi-functional build tool for Elixir projects. It can manage dependencies, run tests, execute tasks and compile your Elixir code. It also supports compiling erlang files by default and can be extended with Mix Compiler Tasks to compile other languages. We're going to create a Mix Compiler Task for Gleam so that Mix can use the Gleam compiler to build Gleam files.

You can try the steps below yourself and check out the demonstration repository: https://github.com/michaeljones/gleam-phoenix-mix.

Here we go!

We're assuming you have the gleam compiler available in your environment to execute. Run this to check.

gleam --version

If you don't, then you can follow the installation instructions over in the Gleam docs.

Now we're going to go through setting up a Phoenix project from scratch. We won't go into the details but just so we have a clear base to work with.

Run the following commands to set up an empty phoenix project:

mkdir gleam-phoenix-mix
cd gleam-phoenix-mix
mix phx.new . --app my_app
mix ecto.create
cd assets 
npm install
cd ..

Now we need to create our mix compiler task. We put this in a lib/mix/tasks/compile directory to reflect the module name that we need to give it which is Mix.Tasks.Compile.Gleam. So we create our new directory:

mkdir -p lib/mix/tasks/compile/

And add the follow code in a file called lib/mix/tasks/compile/gleam.ex.

defmodule Mix.Tasks.Compile.Gleam do
  use Mix.Task.Compiler

  def run(_args) do
    System.cmd("gleam", ["build"])
    :ok
  end
end

All this is really doing is running the gleam compiler directly with the 'build' argument to get it to build all the gleam files it can find.

Run the elixir compiler so that we compile that mix task before we try to use it.

mix compile.elixir

Now to make gleam happy we need to set up a .toml config file for it. So we create a file called gleam.toml in the root of the project with the follow content.

name = "my_app"

Then we create a src directory for the gleam files. The Gleam compiler expects the gleam files to be in a src directory. This is different to elixir/mix which expects them in the lib directory by default. Elixir/mix can be configured but Gleam can't at the time of writing.

mkdir src

We then create our Gleam module by making a file called src/hello_world.gleam with the following code in it:

pub fn hello() {
  "Hello, from gleam!"
}

Next we make the following change to the mix.exs file to add our compiler task to the list of compilers being run when you do mix compile and to make sure mix's erlang compiler looks in the gen folder
for erlang files. The gleam compiler compiles .gleam files from the src folder into .erl files in the gen folder so we need the erlang compiler to find them there.

       elixir: "~> 1.5",
       elixirc_paths: elixirc_paths(Mix.env()),
+      erlc_paths: ["src", "gen"],
-      compilers: [:phoenix, :gettext] ++ Mix.compilers(),
+      compilers: [:phoenix, :gettext, :gleam] ++ Mix.compilers(),
       start_permanent: Mix.env() == :prod,
       aliases: aliases(),

I assume that Mix runs the compilers from the start to the end of the list. So we want the :gleam entry to be before the :erlang entry that's in Mix.compilers() so that the Erlang compiler will run after the Gleam compile and compile the Erlang files that the Gleam compiler creates.

As a simple demonstration of interop between Elixir & the compiled gleam files, make the following change to lib/my_app_web/controllers/page_controller.ex:

 defmodule MyAppWeb.PageController do
   use MyAppWeb, :controller

   def index(conn, _params) do
-    render(conn, "index.html")
+    render(conn, "index.html", title: :hello_world.hello())
   end
 end

This works because Gleam modules are compiled to Erlang modules and, in Elixir, Erlang modules are accessible via the atom that is their name.

And then we make the following change to lib/my_app_web/templates/page/index.html.eex:

 <section class="phx-hero">
-  <h1><%= gettext "Welcome to %{name}!", name: "Phoenix" %></h1>
+  <h1><%= @title %></h1>
   <p>A productive web framework that<br/>does not compromise speed or maintainability.</p>
 </section>

Finally we can run:

mix compile
mix phx.server

And load http://localhost:4000 in our browser to see "Hello, from gleam!" in the centre of the standard Phoenix welcome page.

Remember, if you're curious you can check out the final result at this demonstration repo on Github: https://github.com/michaeljones/gleam-phoenix-mix

Final Thoughts

It is exciting to see that it isn't too hard to incorporate Gleam code into an Elixir project using Mix. It is great that Mix is built with this kind of extensibility in mind. There are definitely more questions to answer and rough edges to smooth out but it is exciting to have an avenue to using a strongly typed language on the BEAM.

Introducing Dialyzer & type-specs to an Elixir Project

Michael Jones — Tue, 17 Mar 2020 14:35:45 +0000

At Contact Stack we use Elixir & Phoenix as the core of the backend. I am fascinated by the design & power of the BEAM, the virtual machine that Elixir targets, and I want to learn more about it. I also love strongly & statically typed languages, particularly due to my recent history working with Elm which has a great type system and a helpful compiler.

Elixir is dynamically typed language which leaves me a little nervous about the long term future of the project. I really like types and I find them incredibly useful when the code base grows larger and you're returning to parts of it that you haven't looked at in a while. Often I can't remember exactly the kind of data that I'm passing around and strong types really help with that understanding. Plus I've grown to love having a compiler guide you through the code base when doing refactoring as it points out all the places you need to change.

One way to bring a little more type safety to Elixir projects is to use Dialyzer. Dialyzer performs static analysis on your code and tries to warn you about anything that is almost certainly going to fail. It can be helped by providing type annotations and it also derives its own from inspecting the code. It isn't as good as a solid type system but it is better than nothing.

So, how do we go about introducing Dialzyer and type annotations through out our codebase? Here are the steps that I've followed for Contact Stack.

Install Dialyzer

Maybe a bit of an obvious first step! We can use the dialxyir project to add a dialyzer task to mix. At the time of writing that involves adding this line to your mix.exs deps:

{:dialyxir, "~> 1.0", only: [:dev], runtime: false},

Run Dialyzer & Fix issues

We can now run dialyzer with:

mix dialyzer

On the command line. This can take a long time as dialyzer analyses and creates PLT caches for all the Elixir & Erlang standard libraries. Fortunately it is much quicker on that next run though still not as fast a many compilers out there.

The errors can honestly be quite hard to understand. I would recommend googling as much as possible. I've learned a few of things though that I'll share.

Firstly, in my projects I've had to config dialyzer to make it away of ex_unit and mix otherwise it issues warnings. I suspect I only needed to add mix because I have a custom mix task. You can configure dialyzer by adding the following to your the project config in mix.exs:

dialyzer: [plt_add_apps: [:ex_unit, :mix], ignore_warnings: "config/dialyzer.ignore"]

Here we add ex_unit & mix and also indicate an ignore file.

Secondly, some errors don't seem to be clearly fixable by code that is under our control. In which case we want to add those errors to the ignore file. If you run dialyzer with:

mix dialyzer --format dialyzer

Then you can copy the start or all of the error that you want to ignore onto a new line in the ignore file we've just set up and dialyzer will skip it on future runs.

Thirdly, if you get a warning that says that some function has no local return then it normally means there is another warning within the function itself that makes dialzyer think that the function will never work. You can skip the 'no local return' warning at look at the next one in the output so see what the issue might be.

Introduce Credo

Credo is a linting tool for Elixir that can warn if you don't provide a type annotation on a function. Part of our goal here is to make sure that we provide type annotations for all the functions in our code base. This helps Dialyzer understand our intent and is self-documenting which helps ourselves and others on the code base.

So we add:

{:credo, "~> 1.2", only: [:dev, :test], runtime: false}

To our mix.exs deps.

Configure & Fix Credo Warnings

Credo checks for a lot of different things. It does not check for type annotations by default and we're going to enable that but first, as long as they are manageable, fix any current warnings that Credo has in place. You can do this by changing your code or suppressing some of Credo's warnings. I have the following config for my preferences:

%{
  configs: [
    %{
      name: "default",
      files: %{
        included: ["lib/"],
        excluded: []
      },
      checks: [
        {Credo.Check.Design.AliasUsage, false},
        {Credo.Check.Readability.AliasOrder, false}
      ]
    }
  ]
}

Extend Credo Rules to Include Type Annotations

We extend our config to include the Specs check by adding this line:

      checks: [
        {Credo.Check.Design.AliasUsage, false},
        {Credo.Check.Readability.AliasOrder, false}
+       {Credo.Check.Readability.Specs, []}
      ]

Silence the Huge Number of Credo Warnings

Now we can run credo again to get a sense of the number of missing type annotations. If you have a small number then go ahead and try to add them now. If, like me, you have something like 200 warnings or more then you might prefer to silence them for now and attend to them gradually.

To do this we can run the following command:

mix credo --strict | grep lib | cut -d " " -f 8 | cut -d ":" -f 1 | sort -u | xargs sed -i '1s/^/# credo:disable-for-this-file Credo.Check.Readability.Specs\n/'

This is going to add # credo:disable-for-this-file Credo.Check.Readability.Specs to the top of every file in your codebase that has missing type annotations.

When we run mix credo --strict again we should see a clean sheet. Now you can go into the files at your leisure and remove the comment at the top and add type annotations. Credo will warn about any that are missing and warn by default for any new files we add to the system.

Final Thoughts

I worry that dialyzer is not very useful in comparison to the type systems & compilers that I have used before but it feels good to be trying to make the code as tight and error free as possible. I am at the beginning of this journey so I can't really report on whether it is worth it long term but we'll see.

Moving from elm-validate to elm-verify

Michael Jones — Fri, 16 Feb 2018 14:50:06 +0000

At Zaptic, we use Elm for our business administration website to allow customers to configure the processes that they want to carry out. The site contains a number of different forms for adding & editing various entities in our system. All of these forms benefit from client side validation before the data is submitted to our servers.

When we started writing our Elm code, the recommended approach to form validation was the rtfeldman/elm-validate library. For a form backed by a data structure like this:

type alias ProductGroup =
    { name : String
    , mode : Maybe Mode
    , selectedProducts : Dict Int Product

    -- Used for storing validation errors
    , errors : List ( Field, String )
    }

We might have a validation function like this:

validateProductGroup : Validator ( Field, String ) ProductGroup
validateProductGroup =
    Validate.all
        [ .name >> ifBlank ( NameField, "Please provide a name" )
        , .mode >> ifNothing ( ModeField, "Please select a mode" )
        , .selectedProducts >> ifEmptyDict ( ProductsField, "Please select at least one product" )
        ]

This checks that the name field isn't blank, that a mode has been selected and that some products have been selected. The result is a list of errors corresponding to the criteria that aren't met:

validateProductGroup { name = "", mode = Just MultiMode, selectedProducts = Dict.empty }
-- [(NameField, "Please provide a name"), (ProductsField, "Please select at least one product")]

The list can be saved in the model and then the view searches the list for relevant entries when displaying the various input fields in order to display the error state to the user if there is one.

With this in mind, we can create our form to send a SaveProductGroup message when the form is submitted and we can handle this message in our update function with:

        SaveProductGroup ->
            case validateProductGroup model.productGroupData of
                [] ->
                    { model | productGroupData = { productGroupData | errors = Nothing } }
                        ! [ postProductGroup model.productGroupData
                          ]

                results ->
                    { model | productGroupData = { productGroupData | errors = Just results } } ! []

Here, we run the validateProductGroup function and if we get an empty list then we know that there are no errors and we can post our data to the server using postProductGroup that will create a command for us to carry out the necessary HTTP request.

The problem we encounter is that the postProductGroup needs to encode the ProductGroup structure to JSON and when it does that it has to navigate the data that we've given it. We know that the mode value is a Just because we've validated it but the convert to JSON function cannot take any shortcuts because Elm doesn't let us (which is a good thing!) We can try to write the encoding function like:

encodeProductGroup : ProductGroup -> Json.Encode.Value
encodeProductGroup { name, mode, selectedProducts } =
    Json.Encode.object
        [ ( "name", Json.Encode.string name )
        , ( "mode"
          , case mode of
                Just value ->
                    encodeMode value

                Nothing ->
                    -- Help! What do we put here?
          )
        , ( "products", Json.Encode.list <| List.map encodeProduct <| Dict.values selectedProducts )
        ]

As you can see from the comment, it is hard to figure out what to write there. Elm gives us some escape hatches like Debug.crash but that feels wrong. We could not include the mode key if we don't have a value but then we're knowingly allowing a code path that sends incorrect data to the server.

So what do we do? Well, we need to recognise that this function shouldn't be making this decision. It shouldn't have to deal with the Maybe, especially when we have already validated that it has a Just value.

So we create a new type:

type alias ProductGroupUploadData =
    { name : String
    , mode : Mode
    , selectedProducts : List Product
    }

Which is the same data that we're interested in but with the Maybe resolved to an actual value and the Dict.values transformation already applied to selectedProducts. If we change our encodeProductGroup function to expect this type then the implementation becomes trivial:

encodeProductGroup : ProductGroupUploadData -> Json.Encode.Value
encodeProductGroup { name, mode, selectedProducts } =
    Json.Encode.object
        [ ( "name", Json.Encode.string name )
        , ( "mode", encodeMode mode )
        , ( "products", Json.Encode.list <| List.map encodeProduct selectedProducts )
        ]

But how do we convert our ProductGroup to ProductGroupUploadData? This is where the stoeffel/elm-verify library comes in. It allows us to both validate our data and transform it to another structure in the same operation. It does this by using the Result type to allow it to report validation errors, if any are encountered, or the new data structure for us to use. And it does this with a Json.Decode.Pipeline-like interface:

validateProductGroup : ProductGroup -> Result (List ( Field, String )) ProductGroupUploadData
validateProductGroup =
    let
        validateProducts productGroup =
            if Dict.isEmpty productGroup.selectedProducts then
                Err [ ( ProductsField, "Please select at least one product" ) ]
            else
                Ok (Dict.values productGroup.selectedProducts)
    in
    V.validate ProductGroupUploadData
        |> V.verify .name (String.Verify.notBlank ( NameField, "Please provide a name" ))
        |> V.verify .mode (Maybe.Verify.isJust ( ConfigField, "Please select a mode" ))
        |> V.custom validateProducts

Where V is the result of import Verify as V. You can see the "pipeline" approach that might be familiar from Json.Decode.Pipeline. That means we're using ProductGroupUploadData as a constructor and each step of the pipeline is providing an argument to complete the data. We use String.Verify to check that the name isn't blank and Maybe.Verify to check that the mode is specified. Then we use Verify.custom to provide a slight more complex check for the selectedProducts. Verify.custom allows us to write a function that takes the incoming data and returns a Result with either an Err with an array of errors or an Ok with the valid value. We use it to not only check that the dictionary is empty but also extract just the values from the dictionary. We don't have to run Dict.values here, we could also do that in the encodeProductGroup function when generating the JSON but I have a personal preference for the UploadData to match the JSON closely if possible.

With that in place, we can change our SaveProductGroup case in our update function to look like this:

        SaveProductGroup ->
            case validateProductGroup model.productGroupData of
                Ok uploadData ->
                    { model | productGroupData = { productGroupData | errors = Nothing } }
                        ! [ postProductGroup uploadData
                          ]

                Err errors ->
                    { model | productGroupData = { productGroupData | errors = Just errors } } ! []

Which means that the postProductGroup is given a nice ProductGroupUploadData record and no longer has to worry about the Maybe type.

Prior to using the elm-verify library, we used a separate function to convert between the types and we only called postProductGroup if both the validation & the conversion functions were successful. The conversion function always felt a little strange though and switching to elm-verify cleans that up nicely.

Further note: Whilst the elm-verify interface is similar to Json.Decode.Pipeline it isn't quite the same. It has a version of andThen but it doesn't provide some ways to combine operations like Json.Decode.oneOf or helper functions like Json.Decode.map. This is partly to keep the interface simple and partly because it is always manipulating Result values so with a bit of thought you can use helper functions like Result.map quite easily.

I only include this as originally sought to use elm-verify exactly as I would Json.Decode.Pipeline and ended up writing a bunch of unnecessary functions in order to allow me to do just that. I should have spent more time understanding the types in the interface and working with those.

Right clicks & Elm

Michael Jones — Fri, 08 Dec 2017 14:04:08 +0000

At Zaptic, we allow our customers to create a flow chart describing the business processes that they would like to complete. We draw the flow chart in SVG using a pan & zoom interface to make it easy to look around. We track mouse down, mouse drag & mouse up events to manage this UI.

Unfortunately, in development, the experience was a little annoying as every time we loaded the browser's inspector on an element the mousedown of the right-click would be tracked by the UI but the mouseup would be over the context menu and so would not register. The result? The UI gets left in "pan" mode and attempts to follow your mouse everywhere even though you don't have any mouse buttons pressed any more.

To overcome this, we needed to look into tracking which mouse button was being used so that we could ignore the right-clicks. We are using the elm-lang/mouse package which includes a position decoder that can be used for mousedown events. That decoder only provides the mouse location as pageX and pageY, it doesn't provide any information about which button was pressed so we need to go a little deeper.

If we take a look at the source code for the module we can see that the position decoder is doing the following:

position : Json.Decoder Position
position =
  Json.map2 Position
    (Json.field "pageX" Json.int)
    (Json.field "pageY" Json.int)

It is looking for the pageX and pageY properties, as we suspected, and it expects them to be integers. If we look at the lovely MDN documentation for a mousedown event we can see that it has lots of other properties including a integer button property which contains some indication of which button is being used. We can set up our own type & decoder as:

type alias MouseClick =
    { pageX : Int
    , pageY : Int
    , button : Int
    }


mouseClickDecoder : Json.Decode.Decoder PFM.MouseClick
mouseClickDecoder =
    Json.Decode.map3 PFM.MouseClick
        (Json.Decode.field "pageX" Json.Decode.int)
        (Json.Decode.field "pageY" Json.Decode.int)
        (Json.Decode.field "button" Json.Decode.int)

And adapt our event handler to use it:

type Msg
    = DragStart MouseClick
    | ...

onMouseDown : Attribute Msg
onMouseDown =
    on "mousedown" (Json.Decode.map DragStart mouseClickDecoder)

Then in our update function that is going to handle the DragStart message we can check the value of button and ignore right-clicks. There are some inconsistencies between browsers on how different buttons are represented as integers which you can read about on quirksmode. Fortunately they all align for the 'right-click' which is represented as 2. So we can ignore that:

DragStart click ->
    -- Ignore right-clicks which are given value '2' by browsers
    if click.button /= 2 then
        let
            pos =
                { x = click.pageX, y = click.pageY }
        in
        ({ model | = drag = Just { start = pos, current = pos } }, Cmd.none)
    else
        (model, Cmd.none)

And there we have it! We're free of the right-clicks and can happily jump into the browser dev-tools whenever we like.

If you have any thoughts or advice, please let me know! Always happy to learn.

Update: Checkout Ilias' comment below. It really cleans up the message handling in the update:

-        DragStart click ->
-            -- Ignore right-clicks which are given value '2' by browsers
-            if click.button /= 2 then
-                let
-                    pos =
-                        { x = click.pageX, y = click.pageY }
-                in
-                { model | drag = Just { start = pos, current = pos } } ! []
-            else
-                model ! []
+        DragStart pos ->
+            { model | drag = Just { start = pos, current = pos } } ! []

Django to Phoenix - Part 4: Sessions

Michael Jones — Sun, 12 Nov 2017 16:24:22 +0000

Background

I have a website built with Django and I would like to start writing parts of it in Phoenix.

Disclaimer

I am a beginner at Phoenix. I would like help & advice if I am doing something wrong.

Starting Point

In the last post we set up Phoenix inside our Docker container. We have nginx splitting traffic between Phoenix & Django. All requests go to Django except for the temporary /elixir path which we direct to Phoenix for experimenting.

In this post, we're going to look at accessing the Django sessions information so that our Phoenix app can know if the user is logged in or not. We're going to have to:

Check the session cookie
Check the sessions table in the database
Add some information to the request data to indicate if there is a current user session or not.
Displaying some indication of the session on the page.

Checking the Session Cookie

The official documentation tells us that Django uses cookies for tracking the session IDs. If we log in to our Django app in Chrome, open up the inspector, go to the Application tab and then click on the localhost entry under Cookies, then we can see a cookie called sessionid which contains a string.

I'm not cookie expert but my understanding is that any available cookies for the domain are automatically sent along with each standard web request. So when we navigate to /elixir the browser will be sending the same cookies over.

Phoenix is built on top of a layer called Plug. Looking at the documentation for Plug we can see that the cookies are available as an Elixir Map on the conn data. This is the standard connection data that is fed to functions that are dealing with requests.

In order to access that cookie we're going to set up a new Phoenix controller called TimetableWeb.Session. To do this we create a file called session.ex in the directory lib/timetable_web/controllers/ in our Phoenix project. It will have the following contents:

defmodule TimetableWeb.Session do
  import Plug.Conn

  def init(_) do
  end

  def call(conn, _) do
    case conn.cookies["sessionid"] do
      nil ->
        IO.puts "No sessionid found"
      session_id ->
        IO.puts "Session ID: #{session_id}"
    end

    # Return conn
    conn
  end
end

This is standard Plug format. For the moment, we don't need to do anything in the init function but it still required. The call function requires two arguments in order to receive the data from init but we don't need it so we use a simple _ as the argument name.

As for the rest of the Plug, we look into the conn.cookies Map for the sessionid entry and pattern match on the result. If we get nil then we know we don't have an active session at the moment. If we get something other than nil then we know we must have the session ID. For the moment, we print an appropriate message and return the conn at the end of the function. In Elixir the result of the last expression is returned from the function. There is no need for an explicit return keyword in that situation.

Now we need to add our plug into the Phoenix request pipeline so that it can process incoming requests. To do this we edit our lib/timetable_web/router.ex file with the following change:

   pipeline :browser do
     plug :accepts, ["html"]
     plug :fetch_session
     plug :fetch_flash
     plug :protect_from_forgery
     plug :put_secure_browser_headers
+    plug TimetableWeb.Session, repo: Timetable.Repo
   end

   pipeline :api do
     plug :accepts, ["json"]
   end

Here we are adding our plug to the end of the browser pipeline. We're not using the api pipeline at the moment so we ignore that.

Now, if we go to our Django app, log in and visit /elixir then we see:

[info] GET /elixir
Session ID: boj26dqs1v0okevigm2vnsu11gephx21

In our console output from the Phoenix app inside our Docker container.

If we now go to our Django app, log out and visit /elixir, then we see:

[info] GET /elixir
No sessionid found

So we seem to be successfully getting the session cookie.

Checking the Sessions Database Table

Django doesn't store meaningful data in the session cookie. It is just an ID to access the data from the sessions table in the database. The documentation tells us that the session data is stored in a table called django_session. If we run python manage.py dbshell and in the Postgresql prompt run: \d django_session then we get a description of the table:

            Table "public.django_session"
    Column    |           Type           | Modifiers 
-------------------+--------------------------+-----------
 session_key  | character varying(40)    | not null
 session_data | text                     | not null
 expire_date  | timestamp with time zone | not null
Indexes:
    "django_session_pkey" PRIMARY KEY, btree (session_key)
    "django_session_expire_date" btree (expire_date)
    "django_session_session_key_like" btree (session_key varchar_pattern_ops)

We can see that there are 3 columns. The session_key most likely maps to our session_id. The session_data most likely includes the useful information about the session. Finally, there is an expire_date to indicate when the session should be considered 'stale' and no longer honoured. We can also see that the session_key field is the primary key for this table.

We want to get to a position in which our Phoenix app can check this database table to retrieve the session information. Phoenix uses a layer called Ecto to model database tables and talk to databases. We are going to use a mix task to help generate Ecto schemas. The command we're going to run is:

mix phx.gen.schema Session sessions session_key:string session_data:string expire_date:date

This generates an Ecto schema that is almost correct for what we need. The problem is that it assumes that we're going to want to have an extra id column to use as the primary key. This makes sense for a lot of tables but not our django_session table where the primary key is the session_key column. So we have to adjust the final schema to inform Ecto of our preferred primary key. Here is what it is going to look like:

defmodule Timetable.Session do
  use Ecto.Schema
  import Ecto.Changeset
  alias Timetable.Session

  @primary_key {:session_key, :string, autogenerate: false}

  schema "django_session" do
    field :session_data, :string
    field :expire_date, :utc_datetime
  end

  @doc false
  def changeset(%Session{} = session, attrs) do
    session
    |> cast(attrs, [:session_key, :session_data, :expire_date])
    |> validate_required([:session_key, :session_data, :expire_date])
  end
end

To check if this works, we can try it in the Elixir interactive shell: iex. If we run iex -S mix from Phoenix project root directory, then we can do:

iex(1)> Timetable.Session |> Timetable.Repo.get_by(session_key: "boj26dqs1v0okevigm2vnsu11gephx21")                                                       
[debug] QUERY OK source="django_session" db=131.4ms                          
SELECT d0."session_key", d0."session_data", d0."expire_date" FROM "django_session" AS d0 WHERE (d0."session_key" = $1) ["boj26dqs1v0okevigm2vnsu11gephx21"]                                     
%Timetable.Session{__meta__: #Ecto.Schema.Metadata<:loaded, "django_session">,                                                                            
 expire_date: #DateTime<2017-11-21 08:31:06.774119Z>,                        
 session_data: "ODNjNWFkMDJh....cl9pZCI6IjEifQ==",  
 session_key: "boj26dqs1v0okevigm2vnsu11gephx21"

We can see that we received a record with the session data that we're interested in.

If we want to check the expire_date as well we can do this using the Ecto query syntax:

iex(1)> import Ecto.Query, only: [from: 2]
Ecto.Query
iex(2)> now = DateTime.utc_now()
#DateTime<2017-11-12 10:31:16.153448Z>
iex(3)> (from s in Timetable.Session, where: s.session_key == "boj26dqs1v0okevigm2vnsu11gephx21" and s.expire_date >= ^now) |> Timetable.Repo.one
[debug] QUERY OK source="django_session" db=24.0ms
SELECT d0."session_key", d0."session_data", d0."expire_date" FROM "django_session" AS d0 WHERE ((d0."session_key" = 'boj26dqs1v0okevigm2vnsu11gephx21') AND (d0."expire_date" >= $1)) [{{2017, 11, 12}, {10, 31, 16, 153448}}]
%Timetable.Session{__meta__: #Ecto.Schema.Metadata<:loaded, "django_session">,
  expire_date: #DateTime<2017-11-21 08:31:06.774119Z>,
  session_data: "ODNjNWFkMDJh....cl9pZCI6IjEifQ==",
  session_key: "boj26dqs1v0okevigm2vnsu11gephx21"}

And to check that it returns nothing when the session has expired we can do:

iex(1)> import Ecto.Query, only: [from: 2]
Ecto.Query
iex(2)> {:ok, later, 0} = DateTime.from_iso8601("2017-11-22T00:00:00Z")                                                                            
{:ok, #DateTime<2017-11-22 00:00:00Z>, 0}
iex(3)> (from s in Timetable.Session, where: s.session_key == "boj26dqs1v0okevigm2vnsu11gephx21" and s.expire_date >= ^later) |> Timetable.Repo.one
[debug] QUERY OK source="django_session" db=2.9ms
SELECT d0."session_key", d0."session_data", d0."expire_date" FROM "django_session" AS d0 WHERE ((d0."session_key" = 'boj26dqs1v0okevigm2vnsu11gephx21') AND (d0."expire_date" >= $1)) [{{2017, 11, 22}, {0, 0, 0, 0}}]
nil

We get back nil when providing a date after the session expire_date.

Adding Session Information to the Request

Now that we have a way to extract information from the session table, we can update our Session plug to check for a valid session. We're going to update it to look like this:

defmodule TimetableWeb.Session do
  import Plug.Conn
  import Ecto.Query, only: [from: 2]

  def init(opts) do
    Keyword.fetch!(opts, :repo)
  end

  def call(conn, repo) do
    now = DateTime.utc_now()
    case conn.cookies["sessionid"] do
      nil ->
        assign(conn, :logged_in, false)
      session_key ->
        case repo.one(from s in Timetable.Session, where: s.session_key == ^session_key and s.expire_date >= ^now) do
          nil ->
            assign(conn, :logged_in, false)
          _ ->
            assign(conn, :logged_in, true)
        end
    end
  end
end

We have done two main adjustments:

The init function has been updated to look for and return the :repo reference from the opts. The call function has been updated to receive the repo as the second argument.
If the call function has found a session key in the cookie, we check the sessions table for that session. If the session is valid, then we return the conn data with a new logged_in field set to true. In both of the other cases where no valid cookie or session is found, we return the conn with a logged_in field set to false.

This isn't very advanced yet. A normal system would want to access the session_data from the session table as well but we'll leave that for the next post.

Displaying Session Information on the Page

For the final step, we want to display whether or not the user is logged in. We're going to keep it very basic for now and update the template for the Phoenix landing page which is lib/timetable_web/templates/page/index.html.eex. We're going to make the follow change:

 <div class="row marketing">
+
+  <div>
+    <%= if @logged_in do %>
+      Logged In!
+    <% else %>
+      Not Logged In
+    <% end %>
+  </div>
+
   <div class="col-lg-6">
     <h4>Resources</h4>

This uses EEx - Embedded Elixir - to display different content depending on the value of logged_in. As a field on conn, the logged_in value is accessible in our templates.

Now when we log in to our Django app and navigate to our /elixir page, we're going to see Logged In! displayed to us on the page. And if we log out of our Django app and come back to this page we see Not Logged In instead.

Conclusion

We have our Phoenix app reading the session information from the Django sessions table. We are able to update the web page in a basic manner to reflect the logged-in status of the current user.

In the next post, we'll extend this approach to also check the user table for the user name.

Django to Phoenix - Part 3: Initial Phoenix Setup

Michael Jones — Tue, 31 Oct 2017 20:41:03 +0000

Background

I have a website built with Django and I would like to start writing parts of it in Phoenix.

Starting Point

In the last section we added nginx to our Dockerfile so that we could introduce Phoenix into our container and use nginx to split incoming requests between Phoenix & Django depending on the URL path.

Our next step is to get a basic Phoenix set up working and have it accessible via the Docker container. There are a few steps to this:

Install Elixir & Phoenix
Create a Phoenix project
Add Elixir & Phoenix to our Docker image
Add a nginx config section for our Phoenix app
Get our Phoenix app running in the container

Install Elixir & Phoenix

In order to create a new Phoenix project and interact with it on our development machine, we need to install Elixir & Phoenix. We're not going to cover it in depth here but following the relevant install guides (Elixir & Phoenix) should get you set up.

Create a Phoenix Project

Phoenix comes with a mix task for creating a new Phoenix project. Mix is a build tool & task runner for the Elixir ecosystem. So common jobs like creating a project, adding modules or running tests often have mix tasks to help you do them.

In this case we're interested in mix phx.new which creates a new project. We'd like the Phoenix project to live in a directory alongside the Django project code. Fortunately for me, the Django code is already in a dancetimetable folder rather than directly in the project root, so we can create the Phoenix project in a folder called timetable (naming is hard) in the project root as well.

In order to do that we're going to run the following command:

mix phx.new timetable --no-brunch

We use the --no-brunch option because our Django project already uses Webpack for compiling the client side applications so we don't need Brunch.

The command asks us if we want to 'fetch and install dependencies' and we answer 'yes' because we do. Once it has finished installing & compiling, it prints the following helpful message:

We are all set! Go into your application by running:

    $ cd timetable

Then configure your database in config/dev.exs and run:

    $ mix ecto.create

Start your Phoenix app with:

    $ mix phx.server

You can also run your app inside IEx (Interactive Elixir) as:

    $ iex -S mix phx.server

But we need to change these steps for our set up. The mix ecto.create command is responsible for creating the Postgresql database for the Phoenix project but we want Phoenix to use the database that already exists for our Django app. So we're not going to run mix ecto.create, instead we're going to edit config/dev.exs to reference our development database. The change looks like this:

+config :timetable, Timetable.Repo,
   adapter: Ecto.Adapters.Postgres,
   username: "postgres",
   password: "postgres",
-  database: "testing_dev",
-  hostname: "localhost",
+  database: "tangotimetable",
+  hostname: System.get_env("DOCKER_HOST") || "localhost",
   pool_size: 10

We prepare ourselves by adding the same DOCKER_HOST lookup that we have discussed in a previous post for getting our Django server to talk to our Postgres database from inside the Docker container.

With that in place, we can run mix phx.server. Once that is running we can visit http://localhost:4000 in our browser and see the default Phoenix new project landing page.

The rest of this post is about getting to see that page again, but having it served from inside the Docker container.

Add Elixir & Phoenix to our Docker image

To install Elixir & Phoenix into our Docker image, we use the marcelocg/phoenix Dockerfile as reference. After some experimentation, the start of our Dockerfile now looks like this:


from ubuntu:14.04

run apt-get update -y
run apt-get upgrade -y

# (1) Updated - Add locales & inotify-tools
run apt-get install -y git python python-pip make g++ wget curl libxml2-dev \
        libxslt1-dev libjpeg-dev libenchant-dev libffi-dev supervisor \
        libpq-dev python-dev realpath nginx apt-transport-https locales \
        inotify-tools

# (2) Set locale information
run locale-gen en_US.UTF-8
env LANG en_US.UTF-8
env LANGUAGE en_US:en
env LC_ALL en_US.UTF-8

# Install yarn as before (unchanged)
run curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add - \
    && echo "deb https://dl.yarnpkg.com/debian/ stable main" | tee /etc/apt/sources.list.d/yarn.list

# (3) Install erlang apt repository
run wget https://packages.erlang-solutions.com/erlang-solutions_1.0_all.deb \
    && dpkg -i erlang-solutions_1.0_all.deb

# (4) Update apt information & install erlang & elixir
run apt-get update -y
run apt-get install -y yarn esl-erlang elixir

# (5) Use mix to install Phoenix
env PHOENIX_VERSION 1.3.0
run mix archive.install --force https://github.com/phoenixframework/archives/raw/master/phx_new-$PHOENIX_VERSION.ez

# (6) Install hex & rebar locally
run mix local.hex --force \
    && mix local.rebar --force

# Same as before from here on down
add https://raw.githubusercontent.com/isaacs/nave/master/nave.sh ./nave.sh

run bash nave.sh usemain 6.9.1

# ...

Going through those changes one at a time:

We add inotify-tools and locales to the install apt-get install. Phoenix uses inotify-tools to check for file changes in Linux which allows it to reload the development server when the code has changed. As for locales, it seems that Erlang & Elixir like the Unix locale to be properly set before they are installed. I don't know why, but it complains if we don't do it.
We set the locale information as shown in the marcelocg/phoenix Dockerfile.
Install the erlang-solutions apt repository package. This looks like we're installing a package, and we are, but the result of the package is to make our system aware of the erlang-solutions apt repository. This allows apt to install the latest packages from the erlang-solutions repository instead of the potentially-less-up-to-date Ubuntu repositories.
Refresh apt's cache of information and install Erlang & Elixir. If you're unfamiliar with Erlang, it is the base environment upon with Elixir runs.
Install Phoenix via the mix command. This installs Phoenix from the referenced archive file on Github. This is the recommended way to install Phoenix.
We install hex & rebar locally. Hex is a package manager tool for Erlang & Elixir and rebar is a build tool. I do not know if these are necessary in our image, I haven't experimented with removing them yet.

Now we can rebuild our Docker image and it is ready to run our Phoenix development server. The rebuild may take some time due to the nature of the changes.

Add a nginx config section for our Phoenix app

The change to the nginx config is quite simple. We already have a section for the Django app. We just want another one for the Phoenix app. At the moment all requests go to the Django app. As an experimental step, we're going to redirect requests for /elixir to our Phoenix app. This is never going to make it to production, but we're still getting set up.

So our change looks like this:

    # server_name www.tangotimetable.com;
    server_name localhost;

+   location /elixir {
+
+       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+
+       proxy_set_header Host $http_host;
+
+       # we don't want nginx trying to do something clever with
+       # redirects, we set the Host: header above already.
+       proxy_redirect off;
+
+       proxy_pass http://localhost:4000;
+   }
+
    location / {

        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

It is identical to the Django section below except that we're redirecting to port 4000 instead of port 7000. I am not sure, but I believe it is important for this section to appear above the Django section in the file. This means that the Django section acts as a catch-all for any requests not sent to the Phoenix app.

Get our Phoenix app running in the container

Now we're ready to get the Phoenix development server up and running in our container. We're going to follow a similar pattern to our Django app. We're going to add this section to our supervisord configuration:

 stderr_logfile=/dev/stderr
 stderr_logfile_maxbytes=0
+
+[program:phoenix]
+command=/src/dev/run-phoenix
+stdout_logfile=/dev/stdout
+stdout_logfile_maxbytes=0
+stderr_logfile=/dev/stderr
+stderr_logfile_maxbytes=0

Then we're going to add a run-phoenix script which looks like this:

#!/bin/bash

cd /src/timetable

export DOCKER_HOST=`netstat -nr | grep '^0\.0\.0\.0' | awk '{print $2}'`

mix phx.server

See a previous post for discussion of the DOCKER_HOST environment variable. It allows our Phoenix app to find the Postgres database which is on the host machine, not inside the container.

I leave it as an exercise to make sure that this run-phoenix file is accounted for in the Dockerfile and makes it in the Docker image in the correct place. It can be very confusing when a file isn't included in the Docker image when you expect it to be.

The final change to make is to mount the Phoenix app source folder into the Docker container at run time. This is because we haven't added the source code to the image. If we did, we'd have to rebuild the image on every change which would be painful.

So we're going to change our run-docker-dev script, from the first post, like this:

    -v `pwd`/logs/nginx:/var/log/nginx:rw \
    -v `pwd`/logs/cron:/var/log/cron:rw \
    -v `pwd`/dancetimetable:/src/dancetimetable:rw \
+   -v `pwd`/timetable:/src/timetable:rw \
    -e DANCETIMETABLE_SETTINGS=development \
    --rm \
    --interactive \

Now when we run our Docker container and visit http://localhost:8000 we see the front page of our Django app but if we visit http://localhost:8000/elixir we see the Phoenix new project landing page.

Update 2017-11-12: I forgot to include a necessary change to the Phoenix router. We have to change the timetable_web/router.ex file with the following diff:

   scope "/", TimetableWeb do
     pipe_through :browser # Use the default browser stack

-    get "/", PageController, :index
+    get "/elixir", PageController, :index
   end

   # Other scopes may use custom stacks.

So that Phoenix is expecting to be asked about /elixir and not /. We could instead change nginx to strip the /elixir from the route before it reaches Phoenix but that is not the eventual goal I have for the routes that Phoenix should handle.

Conclusion

We now have our Django app & a new Phoenix app running side by side in the Docker container. We can make the Django app responsible for some URL paths and the Phoenix app responsible for others. We have a way to go before getting this into production but it is an important step!

If you have any questions or advice, please comment below.

Django to Phoenix - Part 2: Nginx

Michael Jones — Sat, 28 Oct 2017 14:09:20 +0000

Background

I have a website built with Django and I would like to start writing parts of it in Phoenix.

Starting Point

In the last section we set up a development Dockerfile & image so that we could run our Django runserver from inside a Docker container. We did this so that we could add nginx to the container in order to split requests between our Django development server and our future Phoenix development server.

Adding Nginx to our Dockerfile

In our last post we covered the Dockerfile copy command that allows us to copy files from our working directory into the Docker image. So we're going to use that to add nginx configuration to our image. There are 4 steps to this.

Step 1: Install nginx with apt-get

Near the top of our Dockerfile, we have a run command that runs apt-get to install the various Ubuntu packages that we need. We need to update the line to add the 'nginx' package. So it is going to look something like:

# Install dependencies for builds
run apt-get install -y git python python-pip make g++ wget curl libxml2-dev \
        libxslt1-dev libjpeg-dev libenchant-dev libffi-dev supervisor \
        libpq-dev python-dev realpath apt-transport-https nginx

With nginx at the end of the install line. Note that changing such an early and important line like this requires quite a long rebuild for the Docker image so bear that in mind.

Step 2: Add `nginx.conf`

This is the nginx base configuration. Like Apache, nginx has a pattern of having a base configuration and then site specific config files. I'll include the base config here for completeness but I'm pretty sure I haven't made any changes to it from the standard nginx config (though I have stripped comments for conciseness):

user www-data;
worker_processes 4;
pid /run/nginx.pid;

daemon off;

events {
        worker_connections 768;
}

http {

        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 65;
        types_hash_max_size 2048;

        include /etc/nginx/mime.types;
        default_type application/octet-stream;

        access_log /var/log/nginx/access.log;
        error_log /var/log/nginx/error.log;

        gzip on;
        gzip_disable "msie6";

        include /etc/nginx/conf.d/*.conf;
        include /etc/nginx/sites-enabled/*;
}

So we add this to our Dockefile file with a line like this:

copy ./dev/nginx.conf /etc/nginx/

As we're building up an Ubuntu image inside Docker, we copy the config into the standard place that Ubuntu expects to see the nginx config. I've added that line quite close to the bottom of my Dockerfile but it doesn't really matter where it goes as long as it is after the apt-get command that installs nginx.

Note: As I write this I realise that this config is probably the same as one that comes with the nginx install in the container so this is probably redundant. Yay, for writing!

Step 3: Add our site config

At the bottom of the above config you can see the line: include /etc/nginx/sites-enabled/*;. This means that any additional config files in that system directory will be read by nginx when it starts up. So we add our site config to that directory:

copy ./dev/site.conf /etc/nginx/sites-enabled/

Where our site.conf looks like this:


server {
    listen 8000;

    server_name localhost;

    location / {

        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

        proxy_set_header Host $http_host;

        # we don't want nginx trying to do something clever with
        # redirects, we set the Host: header above already.
        proxy_redirect off;

        proxy_pass http://localhost:7000;
    }
}

The first important part is: listen 8000 which tells nginx to listen on that port for incoming requests. Note, that is the port that we're running our Django runserver on so we're going to have move that to another port as you can't have two programs listening on the same port.

The next important part is: location /. This creates a config block that only applies to requests that match the specified location. The matching is done such that requests to any paths starting with / are matched. As all paths start with /, this will match all of them.

Note, when I say "path", what I mean is part of the URL after the host. So the path for https://www.tangotimetable.com/events would be /events. As this starts with /, it will match our location block.

Inside the location block, the most important part is proxy_pass http://localhost:7000;. This tells nginx to forward all the requests through to the process listening on port 7000. This will be our Django runserver as soon as we change it to listen on that port.

The config is based on my production config where the other proxy_ options are required. I've honestly forgotten what they do.

Step 4: Remove the default site config

When you install nginx on an Ubuntu system, it comes with a default site config that listens on port 80 and displays an nginx landing page. This isn't a problem for our development set up as we're not using port 80 but it is a problem in production when we might want to use that. It is also something that caught me out so I wanted to highlight it. To get rid of it, we add the following lines to our Dockerfile:

# Remove default nginx config which attempts to bind to port 80
run rm /etc/nginx/sites-enabled/default

Changing our Django runserver port

As noted above, we're switching from the Django runserver listening on port 8000 to listening on port 7000. This is so that we can have nginx listening on port 8000. So we need to update the run-django script that launches the server. This script was introduced in the last post.

Here we change it like so:

- python manage.py z runserver 0.0.0.0:8000
+ python manage.py z runserver 0.0.0.0:7000

Launching nginx in the container

Finally, we need to make sure that our nginx process is launched when the container is started. If you remember, we're using a program called supervisord to manage the processes that run in the container so we have to add a config entry for nginx to our supervisord config.

This looks like:

[program:nginx]
command=/usr/sbin/nginx
stdout_events_enabled=true
stderr_events_enabled=true

I put this above the [program:django] block but I doubt it matters.

Conclusion

That's it! If I run our docker run script from the last post I see this output:

2017-10-28 09:18:42,008 CRIT Supervisor running as root (no user in config file)
2017-10-28 09:18:42,012 INFO supervisord started with pid 7
2017-10-28 09:18:43,015 INFO spawned: 'nginx' with pid 10
2017-10-28 09:18:43,017 INFO spawned: 'django' with pid 11
2017-10-28 09:18:44,034 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-10-28 09:18:44,034 INFO success: django entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

Which shows supervisord starting both nginx & Django and being happy that they stay up and running for more than 1 sec. If I visit http://localhost:8000 I see my Django apps home page as I would expect.

If you have any questions or advice, please comment below.

Django to Phoenix - Part 1: Docker for Dev

Michael Jones — Thu, 26 Oct 2017 19:50:28 +0000

Background

I have a website built with Django and I would like to start writing parts of it in Phoenix.

Starting Point

I deploy the website using Docker but when it comes to development, I run Django's local server directly. I have been advised that the best strategy for introducing a new language & framework into a website is to use nginx to split the incoming requests based on URLs.

This means that we could run & configure nginx on the local machine, or as we are already using Docker for deployment, we could start doing development using a Docker container as well and then add the nginx layer into that. That feels a lot more contained & less messy.

Setting up Django in Docker

In order to take that step, we're going to want to transition to getting the current Django set up to work in Docker.

I think that ideally I'd be able to use the same Dockerfile for both development & production in order to have the environments match as closely as possible. However when I look at my production Dockerfile, I'm not sure how to make that happen.

So we're going to start with a new development Dockerfile which we're going to call Dockerfile.dev.

from ubuntu:14.04

run apt-get update -y

# Install dependencies for builds
run apt-get install -y git python python-pip make g++ wget curl libxml2-dev \
        libxslt1-dev libjpeg-dev libenchant-dev libffi-dev supervisor \
        libpq-dev python-dev realpath apt-transport-https

# Install yarn from website
run curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add -
run echo "deb https://dl.yarnpkg.com/debian/ stable main" | tee /etc/apt/sources.list.d/yarn.list

run apt-get update -y
run apt-get install -y yarn

# Install node-js via the nave.sh helper script so that we 
# can build our client side applications 
add https://raw.githubusercontent.com/isaacs/nave/master/nave.sh ./nave.sh

run bash nave.sh usemain 6.9.1

# Install python packages
workdir /src
copy ./requirements.txt .
run pip install --upgrade pip
run pip install -r requirements.txt

# Create source code directory
run mkdir dancetimetable

# Copy over various config files & scripts
copy ./dev/supervisord.conf /etc/supervisor/conf.d/supervisord.conf

run mkdir /src/dev

copy ./dev/run-* /src/dev/
copy ./meta.json /src/meta.json

expose 8000

# Run supervisord on launch
cmd /usr/bin/supervisord -c /etc/supervisor/conf.d/supervisord.conf

This is a fairly standard Dockerfile to my understanding. For those unfamiliar with Docker, the file starts with from ubuntu:14.04 which says 'give me a blank ubuntu 14.04' image to begin with and then change it with the commands in the rest of the file. The run command runs a particular command within that image, saving any effects on the system. The copy command allows you to copy files from your working directory on your machine into the Docker image.

Interestingly whilst we copy in dependencies & configuration files, we don't copy in any of the website source code here. This is because when we run the Docker container we 'mount' folders from our local machine into the container. This is important so that when we edit & save a file on our machine, the processes inside the Docker container see it straight away and can use it. Without this we would have to rebuild the Docker image each time we make a change.

Update 28-10-2017: Someone noted that I'm still using Ubuntu 14.04 which is quite old at this point. I'm doing this as my production environment still uses 14.04. I do plan to update but for the moment that version is still fully maintained and I don't need anything more from it.

So how do we mount these directories when running the Docker container? With a command like this:

#!/bin/bash

docker run \
    -p 127.0.0.1:8000:8000 \
    -v `pwd`/logs/gunicorn:/var/log/gunicorn:rw \
    -v `pwd`/logs/nginx:/var/log/nginx:rw \
    -v `pwd`/logs/cron:/var/log/cron:rw \
    -v `pwd`/dancetimetable:/src/dancetimetable:rw \
    -e DANCETIMETABLE_SETTINGS=development \
    --rm \
    --interactive \
    --tty \
    --name dance \
    michaeljones/dancetimetable:dev

The important line is -vpwd/dancetimetable:/src/dancetimetable:rw which is where we say 'please make the /src/dancetimetable directory in the running container, reflect the contents of the local dancetimetable directory, in a read & write manner.'

The other important line is -p 127.0.0.1:8000:8000 which says 'please expose port 8000 from inside the container as the port 8000 on our machine.' This means that we can visit http://localhost:8000 and view our app. If we don't tell Docker to do this then we can't access our website.

You might notice a couple of details are missing here. For example, how are we running the Django development server inside the container? The answer to that lies in the last line of the Dockerfile. This specifies the command we want to run when launching the Docker container. In our case it is:

cmd /usr/bin/supervisord -c /etc/supervisor/conf.d/supervisord.conf

This runs the supervisord program with our specific configuration. Supervisord is a program that can be instructed to launch other programs and monitor them, making sure that they are restarted if they fail. This seems to be the recommended way to run multiple programs in a Docker container as the Dockerfile only allows you to specify one cmd entry at the end.

Whilst we don't strictly need supervisord at the moment as we've only got one command to run, I'm already familiar with supervisord from my production setup and we're going to need to run another command soon anyway.

So what does the supervisord config look like? Something like this:

[supervisord]
nodaemon=true

[program:django]
command=/src/dev/run-django
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0

The first block under supervisord is for general configuration and the nodaemon option tells supervisord to run in the foreground so that that Docker container doesn't lose track of it and stop prematurely.

The second block indicates how we want to run the Django development server. The command entry points to a script and the other settings it supervisord to redirect all standard out & standard error from Django to the container's standard out & standard error so that we get to see it in the console.

So what's in the run-django script? That looks like this:

#!/bin/bash

cd /src/dancetimetable

export DOCKER_HOST=`netstat -nr | grep '^0\.0\.0\.0' | awk '{print $2}'`

python manage.py z runserver 0.0.0.0:8000

This changes directory to where the Django manage.py file is and runs the regular runserver sub-command to get our development server running.

The two extra details are, firstly, that we need to specify 0.0.0.0 as the address rather than localhost or 127.0.0.1. I do not know why. Secondly that find the address of our host machine from the perspective of the Docker container and make that available as DOCKER_HOST. We pick this up in our Django config files when specifying the HOST in the database settings. This is because I'm not running my Postgres database inside the Docker container. It is running on my local machine outside of the container and so Django needs to know how to connect to it. This is the only way I've found to do it.

Conclusion

With all this in place, we can launch our Docker container and visit https://localhost:8000 to see our website. Now we have Django running inside a Docker container we're in a good place to add Nginx and then start introducing Phoenix.

If you have any questions or advice, please comment below.

Update 28-10-2017: I have edited the Dockerfile code block above to remove nginx elements that will be introduced and discussed in the next post.

Django to Phoenix - Part 0: Background

Michael Jones — Tue, 24 Oct 2017 18:37:24 +0000

Hey there, I wrote a website! I was pretty new to web development and I did my best. I wrote it in Django & Python as I was already familiar with Python, and Django is a friendly framework with a good ecosystem. It was a great experience. As a beginner, I was happy to have Django there to hold my hand along the way.

However time has passed and I am getting into functional programming. Whilst functional techniques are possible in Python, it doesn't feel like a good fit. I've also been reading and listening to people talk about Phoenix, Elixir & the Erlang Virtual Machine and the combination sounds kind of great.

So, I'm going to attempt to move at least some of my website, Tango Timetable, from Django to Phoenix.

Starting Point

The site is entirely in Django, with a couple of cron jobs for weekly emails. It runs using Gunicorn behind Nginx in a Docker container. I use Docker to simplify deployments as I was finding a 'git-pull & restart' approach to be too fragile.

Disclaimer

I am comfortable with Python, experienced-but-no-expert in Django and a beginner at Phoenix & Elixir. I'm going to try to document my efforts here in future posts to share my findings but also to hopefully get guidance from others who might know better.

DEV Community: Michael Jones

Adding a custom watcher to Phoenix

Conclusion

Notes

Mixing Gleam & Elixir

Final Thoughts

Introducing Dialyzer & type-specs to an Elixir Project

Install Dialyzer

Run Dialyzer & Fix issues

Introduce Credo

Configure & Fix Credo Warnings

Extend Credo Rules to Include Type Annotations

Silence the Huge Number of Credo Warnings

Final Thoughts

Moving from elm-validate to elm-verify

Right clicks & Elm

Django to Phoenix - Part 4: Sessions

Background

Disclaimer

Starting Point

Checking the Session Cookie

Checking the Sessions Database Table

Adding Session Information to the Request

Displaying Session Information on the Page

Conclusion

Django to Phoenix - Part 3: Initial Phoenix Setup

Background

Starting Point

Install Elixir & Phoenix

Create a Phoenix Project

Add Elixir & Phoenix to our Docker image

Add a nginx config section for our Phoenix app

Get our Phoenix app running in the container

Conclusion

Django to Phoenix - Part 2: Nginx

Background

Starting Point

Adding Nginx to our Dockerfile

Step 1: Install nginx with apt-get

Step 2: Add nginx.conf

Step 3: Add our site config

Step 4: Remove the default site config

Changing our Django runserver port

Launching nginx in the container

Conclusion

Django to Phoenix - Part 1: Docker for Dev

Background

Starting Point

Setting up Django in Docker

Conclusion

Django to Phoenix - Part 0: Background

Starting Point

Disclaimer

Step 2: Add `nginx.conf`