DEV Community: Ilija Eftimov

Deep dive in CORS: History, how it works, and best practices

Ilija Eftimov — Mon, 12 Apr 2021 00:00:00 +0000

The error in your browser's console

No 'Access-Control-Allow-Origin' header is present on the requested resource.

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the
remote resource at https://example.com/

Access to fetch at 'https://example.com' from origin 'http://localhost:3000'
has been blocked by CORS policy.

I am sure you've seen one of these errors, or a variation, in your browser's console. If you have not – don't fret, you soon will. There are enough CORS errors for all developers out there.

These popping-up during development can be annoying. But in fact, CORS is an incredibly useful mechanism in a world of misconfigured web servers, hostile actors on the web and organizations pushing the web standards ahead.

But let's go back the beginning...

In the beginning was the first subresource

A subresource is an HTML element that is requested to be embedded into the document, or executed in its context. In the year of 1993, the first subresource <img> was introduced. By introducing <img>, the web got prettier. And more complex.

You see, if your browser would render a page with an <img> on it, it would actually have to go fetch that sub*resource* from an origin. When a browser fetches said subresource from an origin that does not reside on the same scheme, fully qualified hostname or port – that's a cross-origin request.

Origins & cross-origin

An origin is identified by a triple: scheme, fully qualified hostname and port. For example, http://example.com and https://example.com are different origins – the first uses http scheme and the second https. Also, the default http port is 80, while the https is 443. Therefore, in this example, the two origins differ by scheme and port, although the host is the same (example.com).

You get the idea – if any of the three items in the triple are different, then the origin is different.

As an exercise if we run a comparison of the https://blog.example.com/posts/foo.html origin against other origins, we would get the following results:

URL	Result	Reason
`https://blog.example.com/posts/bar.html`	Same	Only the path differs
`https://blog.example.com/contact.html`	Same	Only the path differs
`http://blog.example.com/posts/bar.html`	Different	Different protocol
`https://blog.example.com:8080/posts/bar.html`	Different	Different port (`https://` is port 443 by default)
`https://example.com/posts/bar.html`	Different	Different host

A cross-origin request means, for example, a resource (i.e. page) such as http://example.com/posts/bar.html that would try to render a subresource from the https://example.com origin (note the scheme change!).

The many dangers of cross-origin requests

Now that we defined what same- and cross-origin is, let's see what is the big deal.

When we introduced <img> to the web, we opened the floodgates. Soon after the web got <script>, <frame>, <video>, <audio>, <iframe>, <link>, <form> and so on. These subresources can be fetched by the browser after loading the page, therefore they can all be same- or cross-origin requests.

Let's travel to an imaginary world where CORS does not exist and web browsers allow all sorts of cross-origin requests.

Imagine I got a page on my website evil.com with a <script>. On the surface it looks like a simple page, where you read some useful information. But in the <script>, I have specially crafted code that will send a specially-crafted request to bank's DELETE /account endpoint. Once you load the page, the JavaScript is executed and an AJAX call hits the bank's API.

Mind-blowing – imagine while reading some information on a web page, you get an email from your bank that you've successfully deleted your account. I know I know... if it was THAT easy to do anything with a bank's. I digress.

For my evil <script> to work, as part of the request your browser would also have to send your credentials (cookies) from the bank's website. That's how the bank's servers would identify you and know which account to delete.

Let's look at a different, not-so-evil scenario.

I want to detect folks that work for Awesome Corp, whose internal website is on intra.awesome-corp.com. On my website, dangerous.com I got an <img src="https://intra.awesome-corp.com/avatars/john-doe.png">.

For users that do not have a session active with intra.awesome-corp.com, the avatar won't render – it will produce an error. But, if you're logged in the intranet of Awesome Corp., once you open my dangerous.com website I'll know that you have access.

That means that I will be able to derive some information about you. While it's definitely harder for me to craft an attack, the knowledge that you have access to Awesome Corp. is still a potential attack vector.

While these two are overly-simplistic examples, it is this kind of threats that have made the same-origin policy & CORS necessary. These are all different dangers of cross-origin requests. Some have been mitigated, others can't be
mitigated – they're rooted in the nature of the web. But for the plethora of attack vectors that have been squashed – it's because of CORS.

But before CORS, there was the same-origin policy.

Same-origin policy

The same-origin policy prevents cross-origin attacks by blocking read access to resources loaded from a different origin. This policy still allows some tags, like <img>, to embeds resources from a different origin.

The same-origin policy was introduced by Netscape Navigator 2.02 in 1995, originally intended to protect cross-origin access to the DOM.

Even though same-origin policy implementations are not required to follow an exact specification, all modern browsers implement some form of it. The principles of the policy are described in RFC6454 of the Internet Engineering Task Force (IETF).

The implementation of the same-origin policy is defined with this ruleset:

Tags	Cross-origin	Note
`<iframe>`	Embedding permitted	Depends on `X-Frame-Options`
`<link>`	Embedding permitted	Proper `Content-Type` might be required
`<form>`	Writing permitted	Cross-origin writes are common
`<img>`	Embedding permitted	Cross-origin reading via JavaScript and loading it in a `<canvas>` is forbidden
`<audio>` / `<video>`	Embedding permitted
`<script>`	Embedding permitted	Access to certain APIs might be forbidden

Same-origin policy solves many challenges, but it is pretty restrictive. In the age of single-page applications and media-heavy websites, same-origin does not leave a lot of room for relaxation of or fine-tuning of these rules.

CORS was born with the goals to relax the same-origin policy and to fine-tune cross-origin access.

Enter CORS

So far we covered what is an origin, how it's defined, what the drawbacks of cross-origin requests are and the same-origin policy that browsers implement.

Now it's time to familiarize ourselves with Cross Origin Resource Sharing (CORS). CORS is a mechanism that allows control of access to subresources on a web page over a network. The mechanism classifies three different categories of subresource access:

Cross-origin writes
Cross-origin embeds
Cross-origin reads

Before we go on to explain each of these categories, it's important to realize that although your browser (by default) might allow a certain type of cross-origin request, that does not mean that said request will be accepted by the server.

Cross-origin writes are links, redirects, and form submissions. With CORS active in your browser, these are all allowed. There is also a thing called preflight request that fine-tunes cross-origin writes, so while some writes might be permitted by default it doesn't mean they can go through in practice. We'll look into that a bit later.

Cross-origin embeds are subresources loaded via: <script>, <link>, <img>, <video>, <audio>, <object>, <embed>, <iframe> and more. These are all allowed by default. <iframe> is a special one – as it's
purpose is to literally load a different page inside the frame, its cross-origin framing can be controlled by using the X-Frame-options header.

When it comes to <img> and the other embeddable subresources – it's in their nature to trigger cross-origin requests. That's why in CORS differentiates between cross-origin embeds and cross-origin reads, and treats them differently.

Cross-origin reads are subresources loaded via AJAX / fetch calls. These are by default blocked in your browser. There's the workaround of embedding such subresources in a page, but such tricks are handled by another policy present in modern browsers.

If your browser is up to date, all of these heuristics are already implemented in it.

Cross-origin writes

Cross-origin writes can be the very problematic. Let's look into an example and see CORS in action.

First, we'll have a simple Crystal (using Kemal) HTTP server:

require "kemal"

port = ENV["PORT"].to_i || 4000

get "/" do
  "Hello world!"
end

get "/greet" do
  "Hey!"
end

post "/greet" do |env|
  name = env.params.json["name"].as(String)
  "Hello, #{name}!"
end

Kemal.config.port = port
Kemal.run

It simply takes a request at the /greet path, with a name in the request body, and returns a Hello #{name}!. To run this tiny Crystal server, we can boot it with:

$ crystal run server.cr

This will boot the server and listen on localhost:4000. If we navigate to localhost:4000 in our browser, we will be presented a simple "Hello World" page:

Now that we know our server is running, let's execute a POST /greet to the server listening on localhost:4000, from the console of our browser page. We can do that by using fetch:

fetch(
  'http://localhost:4000/greet',
  {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ name: 'Ilija'})
  }
).then(resp => resp.text()).then(console.log)

Once we run it, we will see the greeting come back from the server:

This was a POST request, but it was not cross-origin. We sent the request from the browser where http://localhost:4000 (the origin) was rendered, to that same origin.

Now, let's try the same request, but cross-origin. We will open https://google.com and try to send that same request from that tab in our browser:

We managed to get the famous CORS error. Although our Crystal server can fulfil the request, our browser is protecting us from ourselves. It is basically telling us that a website that we have opened wants to make changes to another website as ourselves.

In the first example, where we sent the request to
http://localhost:4000/greet from the tab that rendered
http://localhost:4000, our browser looks at that request and lets it through because it appears that our website is calling our server (which is fine). But in the second example where our website (https://google.com) wants to write to http://localhost:4000, then our browser flags that request and does not let it go through.

Preflight requests

If we look deeper in our developer console, in the Network tab in particular, we will in fact notice two requests in place of the one that we sent:

What is interesting to notice is that the first request has a HTTP method of OPTIONS, while the second has POST.

If we explore the OPTIONS request we will see that this is a request that has been sent by our browser prior to sending our POST request:

What is interesting is that even though the response to the OPTIONS request was a HTTP 200, it was still marked as red in the request list. Why?

This is the preflight request that modern browsers do. A preflight request is performed for requests which CORS deems as complex. The criteria for complex request is:

A request that uses methods other than GET, POST, or HEAD
A request that includes headers other than Accept, Accept-Language or Content-Language
A request that has a Content-Type header value other than application/x-www-form-urlencoded, multipart/form-data, or text/plain

Therefore in the above example, although we send a POST request, the browser considers our request complex due to the Content-Type: application/json header.

If we would change our server to handle text/plain content (instead of JSON), we can work around the need for a preflight request:

require "kemal"

get "/" do
  "Hello world!"
end

get "/greet" do
  "Hey!"
end

post "/greet" do |env|
  body = env.request.body

  name = "there"
  name = body.gets.as(String) if !body.nil?

  "Hello, #{name}!"
end

Kemal.config.port = 4000
Kemal.run

Now, when we can send our request with the Content-type: text/plain header:

fetch(
  'http://localhost:4000/greet',
  {
    method: 'POST',
    headers: {
      'Content-Type': 'text/plain'
    },
    body: 'Ilija'
  }
)
.then(resp => resp.text())
.then(console.log)

Now, while the preflight request will not be sent, the CORS policy of the browser will keep on blocking:

But because we have crafted a request which does not classify as complex, our browser actually won't block the request:

Simply put: our server is misconfigured to accept text/plain cross-origin requests, without any other protection in place, and our browser can't do much about that. But still, it does the next best thing – it does not expose our opened page / tab to the response of that request. Therefore in this case, CORS does not block the request - it blocks the response.

The CORS policy of our browser considers this effectively a cross-origin read, because although the request is sent as POST, the Content-type header value makes it essentially the same as a GET. And cross-origin reads are blocked by default, hence the blocked response we are seeing in our network tab.

Working around preflight requests like in the example above is not recommended. In fact, if you expect that your server will have to gracefully handle preflight requests, it should implement the OPTIONS endpoints and return the correct headers.

When implementing the OPTIONS endpoint, you need to know that the preflight request of the browser looks for three headers in particular that can be present on the response:

Access-Control-Allow-Methods – it indicates which methods are supported by the response’s URL for the purposes of the CORS protocol.
Access-Control-Allow-Headers - it indicates which headers are supported by the response’s URL for the purposes of the CORS protocol.
Access-Control-Max-Age - it indicates the number of seconds (5 by default) the information provided by the Access-Control-Allow-Methods and Access-Control-Allow-Headers headers can be cached.

Let's go back to our previous example where we sent a complex request:

fetch(
  'http://localhost:4000/greet',
  {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ name: 'Ilija'})
  }
).then(resp => resp.text()).then(console.log)

We already confirmed that when we send this request, our browser will check with the server if it can perform the cross-origin request. To get this request working in a cross-origin environment, we have to first add the OPTIONS /greet endpoint to our server. In its response header, the new endpoint will have to inform the browser that the request to POST /greet, with Content-type: application/json header, from the origin
https://www.google.com, can be accepted.

We'll do this by using the Access-Control-Allow-* headers:

options "/greet" do |env|
  # Allow `POST /greet`...
  env.response.headers["Access-Control-Allow-Methods"] = "POST"
  # ...with `Content-type` header in the request...
  env.response.headers["Access-Control-Allow-Headers"] = "Content-type"
  # ...from https://www.google.com origin.
  env.response.headers["Access-Control-Allow-Origin"] = "https://www.google.com"
end

If we boot our server and send the request:

Our request remains blocked. Even though our OPTIONS /greet endpoint did allow the request, we are still seeing the error message. In our network tab there's something interesting going on:

The request to the OPTIONS /greet endpoint was a success! But the POST /greet call still failed. If we take a peek in the internals of the POST /greet request we will see a familiar sight:

In fact, the request did succeed – the server returned a HTTP 200. The preflight request did work – the browser did make the POST request instead of blocking it. But the response of the POST request did not contain any CORS headers, so even though the browser did make the request, it blocked any response processing.

To allow the browser also process the response from the POST /greet request, we need to add a CORS header to the POST endpoint as well:

post "/greet" do |env|
  name = env.params.json["name"].as(String)

  env.response.headers["Access-Control-Allow-Origin"] = "https://www.google.com"

  "Hello, #{name}!"
end

By adding the Access-Control-Allow-Origin header response header, we tell the browser that a tab that has https://www.google.com open can also access the response payload.

If we give this another shot:

We will see that POST /greet did get us a response, without any errors. If we take a peek in the Network tab, we'll see that both requests are green:

By using proper response headers on our preflight endpoint OPTIONS /greet, we unlocked our server's POST /greet endpoint to be accessed across different origin. On top of that, by providing a correct CORS response header on the response of the POST /greet endpoint, we freed the browser to process the response without any blocking.

Cross-origin reads

As we mentioned before, cross-origin reads are blocked by default. That's on purpose - we wouldn't want to load other resources from other origins in the scope of our origin.

Say, we have a GET /greet action in our Crystal server:

get "/greet" do
  "Hey!"
end

From our tab that has www.google.com rendered, if we try to fetch the GET /greet endpoint we will get blocked by CORS:

If we look deeper in the request, we will found out something interesting:

In fact, just like before, our browser did let the request through – we got a HTTP 200 back. But it did not expose our opened page / tab to the response of that request. Again, in this case CORS does not block the request - it blocks the response.

Just like with cross-origin writes, we can relax CORS and make it available for cross-origin reading - by adding the Access-Control-Allow Origin header:

get "/greet" do |env|
  env.response.headers["Access-Control-Allow-Origin"] = "https://www.google.com"
  "Hey!"
end

When the browser gets the response back from the server, it will look at the Access-Control-Allow-Origin header and will decide based on its value if it can let the page read the response. Given that the value in this case is https://www.google.com which is the page that we use in our example the outcome will be a success:

This is how the browser shields us from cross-origin reads and respects the server directives that are sent via the headers.

Fine-tuning CORS

As we already saw in previous examples, to relax the CORS policy of our website, we can set the Access-Control-Allow-Origin of our /greet action to the https://www.google.com value:

post "/greet" do |env|
  body = env.request.body

  name = "there"
  name = body.gets.as(String) if !body.nil?

  env.response.headers["Access-Control-Allow-Origin"] = "https://www.google.com"
  "Hello, #{name}!"
end

This will allow the https://www.google.com origin to call our server, and our browser will feel fine about that. Having the Access-Control-Allow-Origin in place, we can try to execute the fetch call again:

This made it work! With the new CORS policy, we can call our /greet action from our tab that has https://www.google.com rendered. Alternatively, we could also set the header value to *, which would tell the browser that the server can be called from any origin.

Such a configuration has to be carefully considered. Yet, putting relaxed CORS headers is almost always safe. One rule of thumb is: if you open the URL in an incognito tab, and you are happy with the information you are exposing, then you can set a permissive (*) CORS policy on said URL.

Another way to fine-tune CORS on our website is to use the
Access-Control-Allow-Credentials response header. Access-Control-Allow-Credentials instructs browsers whether to expose the response to the frontend JavaScript code when the request's credentials mode is include.

The request's credentials mode comes from the introduction of the Fetch API, which has its roots back the original XMLHttpRequest objects:

var client = new XMLHttpRequest()
client.open("GET", "./")
client.withCredentials = true

With the introduction of fetch, the withCredentials option was transformed into an optional argument to the fetch call:

fetch("./", { credentials: "include" }).then(/* ... */)

The available options for the credentials options are omit, same-origin and include. The different modes are available so developers can fine-tune the outbound request, whereas the response from the server will inform the browser how to behave when credentials are sent with the request (via the Access-Control-Allow-Credentials header).

The Fetch API spec contains a well-written and thorough
breakdown of the interplay of CORS and the fetch Web API, and the security mechanisms put in place by browsers.

Some best practices

Before we wrap it up, let's cover some best practices when it comes to Cross Origin Resource Sharing (CORS).

Free for all

A common example is if you own a website that displays content for the public, that is not behind paywalls, or requiring authentication or authorization – you should be able to set Access-Control-Allow-Origin: * to its resources.

The * value is a good choice in cases when:

No authentication or authorization is required
The resource should be accessible to a wide range of users without restrictions
The origins & clients that will access the resource is of great variety, you don't have knowledge of it or you simply don't care

A dangerous prospect of such configuration is when it comes to content served on private networks (i.e. behind firewall or VPN). When you are connected via a VPN, you have access to the files on the company's network:

Now, if an attacker hosts as website dangerous.com, which contains a link to a file within the VPN, they can (in theory) create a script on their website that can access that file:

While such an attack is hard and requires a lot of knowledge about the VPN and the files stored within it, it is a potential attack vector that we must be aware of.

Keeping it in the family

Continuing with the example from above, imagine we want to implement analytics for our website. We would like our users' browsers to send us data about the experience and behavior of our users on our website.

A common way to do this is to send that data periodically using asynchronous requests using JavaScript in the browser. On the backend we have a simple API that takes these requests from our users' browsers and stores the data on the backend for further processing.

In such cases, our API is public, but we don't want any website to send data to our analytics API. In fact, we are interested only in requests that originate from browsers that have our website rendered – that is all.

In such cases, we want our API to set the Access-Control-Allow-Origin header to our website's URL. That will make sure browsers never send requests to our API from other pages.

If users or other websites try to cram data in our analytics API, the Access-Control-Allow-Origin headers set on the resources of our API won't let the request to go through:

NULL origins

Another interesting case are null origins. They occur when a resource is accessed by a browser that renders a local file. For example, requests coming from some JavaScript running in a static file on your local machine have the Origin header set to null.

In such cases, if our servers do now allow access to resources for the null origin, then it can be a hindrance to the developer productivity. Allowing the null origin within your CORS policy has to be deliberately done, and only if the users of your website / product are developers.

Skip cookies, if you can

As we saw before with the Access-Control-Allow-Credentials, cookies are not enabled by default. To allow cross-origin sending cookies, it as easy as returning Access-Control-Allow-Credentials: true. This header will tell browsers that they are allowed to send credentials (i.e. cookies) in cross-origin requests.

Allowing and accepting cross-origin cookies can be tricky. You could expose yourself to potential attack vectors, so enable them only when absolutely necessary.

Cross-origin cookies work best in situations when you know exactly which clients will be accessing your server. That is why the CORS semantics do not allow us to set Access-Control-Allow-Origin: * when cross-origin credentials are allowed.

While the Access-Control-Allow-Origin: * and Access-Control-Allow-Credentials: true combination is technically allowed, it's an anti-pattern and should absolutely be avoided.

If you would like your servers to be accessed by different clients and origins, you should probably look into building an API (with token-based authentication) instead of using cookies. But if going down the API path is not an option, then make sure you implement cross-site request forgery (CSRF) protection.

Additional reading

I hope this (long) read gave you a good idea about CORS, how it came to be, and why it's necessary. Here are a few more links that I used while writing this article, or that I believe are a good read on the topic:

Cross-Origin Resource Sharing (CORS)
Access-Control-Allow-Credentials header on MDN Web Docs
Authoritative guide to CORS (Cross-Origin Resource Sharing) for REST APIs
The "CORS protocol" section of the Fetch API spec
Same-origin policy on MDN Web Docs
Quentin's great summary of CORS on StackOverflow

Testing in Go: Failing Tests

Ilija Eftimov — Thu, 30 May 2019 00:00:00 +0000

In the previous article on testing in Golang, titled Testing in Go: First Principles, we looked at what testing looks like in Golang. In this article, we will do a deeper dive in the testingpackage. More specifically, we will take a look at the different ways we can fail tests in our test suites and when and how we should use these techniques.

Without further ado, let’s take a look under the covers in the testing package.

A short overview of the `testing` package

The testing package from Golang’s standard library provides support for automated testing of Go packages. It is intended to be used in combination with the go test command. This package in combination with go test expect certain naming conventions that we will cover in another article, but for the purpose of this article we have to know that:

Every test file ends with *_test.go
Every test function has the format TestXxx, where Xxx must not start with a lowercase letter

The testing package exposes two different modes: testing and benchmarking. While what we will be discussing here is transferable to benchmarking as well, we will focus on how to fail our tests in a good way and how to provide meaningful error messages.

Logging with `Log` and `Logf`

If we look at the index of T type in the testing package, there’s a list of functions that represent errors or failures:

Quite some options, right? But before we dive into these different functions, let’s look at another one: Logf (docs).

You might be thinking: “I came here to read about signaling test failures, and for some reason, I am reading about logging”. Trust me, there’s a connection here.

Log formats its arguments using default formatting, analogous to Println, while Logf takes one or more arguments: a format string and additional arguments. It formats its arguments according to the format (analogous to Printf). Both these functions save the text in the error log. Compared to Println and Printf, the difference is that Log and Logf save the output to the error log (instead of os.Stdout) and they also add a final newline.

When tests fail or when the -test.v flag is present, the text is printed out. Basically, Log & Logf are the testing package’s Println and Printf with a twist - prints only in verbose mode or when a test fails. A simple example:

package main

import "testing"

func TestFoo(t *testing.T) {
    t.Logf("Testing Foo")
}

If we would run this using go test there won’t be any output. But, if we run it using the -test.v flag, we’ll see something like this:

$ go test -test.v
=== RUN TestFoo
-------- PASS: TestFoo (0.00s)
    foo_test.go Testing Foo
PASS
ok github.com/fteem/testing_in_go   0.006s

As you can see above, the test returns PASS because there was no test that got marked as failed. If we would like to mark a test as failed, we would have to invoke t.Fail() inside the test. Let’s do that:

package main

import "testing"

func TestMax(t *testing.T) {
    t.Logf("Testing Foo")
    t.Fail()
}

Now, when we run go test we should see the test marked as a failure:

$ go test
-------- FAIL: TestMax (0.00s)
    max_test.go Testing Foo
FAIL
exit status 1
FAIL    github.com/fteem/testing_in_go  0.006s

What’s great about this type of logging is that although the test is marked as a failure on line 7, the t.Logf call is on line 6 and the output states that clearly. If we would add any other logging below we would see a separate line for each of the Logf calls:

package main

import "testing"

func TestMax(t *testing.T) {
    t.Logf("Testing Foo")
    t.Fail()
    t.Logf("Another log from Foo")
}

And the output of the go test invocation:

$ go test
-------- FAIL: TestMax (0.00s)
    max_test.go Testing Foo
    max_test.go Another log from Foo
FAIL
exit status 1
FAIL    github.com/fteem/testing_in_go  0.006s

This behaviour is due to the way Logf functions. With every invocation, it adds the log lines to the error log, but it dumps the whole log to STDOUT only when the tests actually fail, by running all of them and seeing if any were marked as failed. That’s why in the above example we see that the failure is marked on line 7, yet all the logs are present (from line 6 and line 8).

So why are Log and Logf important? Because of all of the next functions that we will explore rely on them to write their outputs to the error log. And log output is essential to showing the failures of the failing tests.

Signaling test Failure

To be able to illustrate the different type of failing a test in Golang, we will first need to write a small function that we can test. Let’s use a simple function Max that takes a slice of ints and returns the largest integer of them all:

package main

func Max(numbers []int) int {
    var max int

    for _, number := range numbers {
        if number > max {
            max = number
        }
    }

    return max
}

Let’s write a small test function:

package main

import "testing"

func TestMax(t *testing.T) {
    input := []int{1, 2, 3, 4, 5}
    actual := Max(input)
    expected := 6

    if actual != expected {
        t.Logf("Expected %d, got %d", expected, actual)
    }
}

The TestMax function defines an input slice of ints, the result of the invocation of Max with the input as argument (called actual) and the expected value of the invocation.

If the expected value does not match the actual result, we want to tell the person (or program) running this test that we received something that we weren’t expecting. We do this by invoking t.Logf with providing a nice explanation for the human.

If you missed it – the expected value here is wrong. It expects 6 while the maximum element of the input slice is 5. That means that this test should fail when we run it:

$ go test
PASS
ok github.com/fteem/testing_in_go   0.006s

Huh? Here’s the first thing we have to understand about signaling failures in tests: we actually have to tell the testing package that this is a failed test. And we are the ones to define what failure means - usually, it’s actual != expected, but other times it can be something else. What testing only cares about is recording if there was a test failure or not. There are multiple ways to do this, t.Fail being the easiest one.

Let’s add a call to t.Fail after the t.Logf call:

package main

import "testing"

func TestMax(t *testing.T) {
    input := []int{1, 2, 3, 4, 5}
    actual := Max(input)
    expected := 6

    if actual != expected {
        t.Fail()
        t.Logf("Expected %d, got %d", expected, actual)
    }
}

Let’s run it:

$ go test -v
-------- FAIL: TestMax (0.00s)
    max_test.go Expected 6, got 5
FAIL
exit status 1
FAIL    github.com/fteem/testing_in_go  0.006s

Boom! Our test failed, as expected. You can notice that the log is now visible, including our call to Logf with our informative error message Expected 6, got 5.

Now that we have our first test failure signaled, let’s look at different ways we can signal test failures.

Delayed vs. immediate failure

Golang’s testing package promotes two types of failures: ones that immediately stop the tests from running and others that register the failure but report it after all tests finish running. To reflect this behaviour, these functions are aptly named: Fail and FailNow.

Let’s explore their behaviour and usage by continuing testing the Max function we already built.

One other test case that we could write is checking that the slice passed as argument is not empty. If it is, we will return -1, otherwise the maximum item we find. Let’s write the new test first:

package main

import "testing"

func TestMax(t *testing.T) {
    input := []int{1, 2, 3, 4, 5}
    actual := Max(input)
    expected := 5

    if actual != expected {
        t.Fail()
        t.Logf("Expected %d, got %d", expected, actual)
    }
}

func TestMaxEmptySlice(t *testing.T) {
    input := []int{}
    actual := Max(input)
    expected := -1

    if actual != expected {
        t.Fail()
        t.Logf("Expected %d, got %d", expected, actual)
    }
}

If we run it, without adding any new functionality to the Max function, it will fail:

› go test
-------- FAIL: TestMaxEmptySlice (0.00s)
    max_test.go Expected -1, got 0
FAIL
exit status 1
FAIL    _/users/Ilija/Documents/testing 0.005s

What we can see here is that the TestMax function is run and it passes, while the TestMaxEmptySlice fails. We achieve that by marking the test as failed using t.Fail(). Let’s add the new functionality to Max which would make the test pass:

func Max(numbers []int) int {
    if len(numbers) == 0 {
        return -1
    }

    var max int

    for _, number := range numbers {
        if number > max {
            max = number
        }
    }

    return max
}

While the change does not make much sense, for our purpose it will do the trick. Let’s run the test:

› go test
PASS
ok _/Users/Ilija/Documents/testing  0.004s

Now, if we look into the two things that the Max function does, we could theoretically attach an importance level to both of them. For example, the guard clause which returns -1 when the slice argument is empty has lower importance than the actual algorithm that detects the maximum element of the slice. That means that if someone is expanding that algorithm and it goes wrong we want to be very loud and “stop the world”, as none of the other functionalities is more relevant than that one.

In cases like these, we can use the FailNow() function to mark the failed test and to immediately stop any further execution of the particular test that tests the max detection. Let’s use it in our tests:

package main

import "testing"

func TestMax(t *testing.T) {
    input := []int{1, 2, 3, 4, 5}
    actual := Max(input)
    expected := 5

    if actual != expected {
        t.Logf("Expected %d, got %d", expected, actual)
        t.FailNow()
    }
}

func TestMaxEmptySlice(t *testing.T) {
    input := []int{}
    actual := Max(input)
    expected := -1

    if actual != expected {
        t.Logf("Expected %d, got %d", expected, actual)
        t.Fail()
    }
}

Also, we need to make the Max function misbehave. To short-circuit this quickly, let’s make the Max function return -100 so we can see the difference.

If we would run this now, there would not be a difference in how the tests will run, although we use FailNow:

› go test -v
=== RUN TestMax
-------- FAIL: TestMax (0.00s)
    max_test.go Expected 5, got -100
=== RUN TestMaxEmptSlice 
-------- FAIL: TestMaxEmptySlice (0.00s)
    max_test.go Expected -1, got -100
FAIL
exit status 1
FAIL    _/Users/Ilija/Documents/testing 0.005s

That is because the TestMax function only does one assertion. Let’s rework our tests a bit to use table-driven tests:

type TestCase struct {
    input []int
    expected int
}

func TestMax(t *testing.T) {
    cases := []TestCase{
        TestCase{
            input: []int{1, 2, 3, 4, 5},
            expected: 5,
        },
        TestCase{
            input: []int{-1, -2, -3, -4, -5},
            expected: -1,
        },
        TestCase{
            input: []int{0},
            expected: 0,
        },
    }

    for _, c := range cases {
        actual := Max(c.input)
        expected := c.expected

        if actual != expected {
            t.Logf("Expected %d, got %d", expected, actual)
            t.Fail()
        }
    }
}

What the table-driven tests do is they loop over multiple test cases (often structs with the expected input and output), and they set assertions on them. If we would run the tests now, with using t.Fail() instead of t.FailNow() we should see the following output:

› go test
-------- FAIL: TestMax (0.00s)
    max_test.go Expected 5, got -100
    max_test.go Expected -1, got -100
    max_test.go Expected 0, got -100
-------- FAIL: TestMaxEmptySlice (0.00s)
    max_test.go Expected -1, got -100
FAIL
exit status 1
FAIL    _/Users/Ilija/Documents/testing 0.005s

We actually get all failing cases for both tests. Let’s replace the t.Fail() invocation to t.FailNow() in the TestMax function and run the test:

› go test -v
=== RUN TestMax
-------- FAIL: TestMax (0.00s)
    max_test.go Expected 5, got -100
=== RUN TestMaxEmptSlice 
-------- FAIL: TestMaxEmptySlice (0.00s)
    max_test.go Expected -1, got -100
FAIL
exit status 1
FAIL    _/Users/Ilija/Documents/testing 0.004s

You can see that instead of running each of the test cases and marking them as failures, the test function stopped right after hitting the first case which failed, instead of going through each case before reporting all of the errors.

This is the key difference between the two, t.Fail() and t.FailNow(). By using the latter we are able to stop the tests and communicate with the developer that there’s something really wrong happening quickly. In comparison, imagine if we had to wait for all test cases to complete before we have some sort of output – it would take much longer to get any meaningful output.

Having a clear difference and use cases for Fail and FailNow in mind, let’s look into how we can combine failure reporting (Log and Logf) with failure marking (Fail and FailNow) in a more structured approach.

Signal & report failures in one go

Understanding the basics and variations of logging and signaling errors is crucial to understand what we will talk about now. If you look at all the tests we wrote so far, although we switched up the approaches by using table-driven tests, two steps were constant:

In all tests we had to check for a mismatch between the expected and the actual values, and
We had to call report (Log) and signal failure (Fail) if there was a mismatch

Luckily, the testing package makes our life a bit easier by implementing functions that combine both, the logging and the test failure. And because there are two ways to signal test failures, there are two functions to do this: Error and Fatal, with their own variants.

Let’s reuse the tests that we wrote above, namely the TestMax and TestMaxEmptySlice functions. If we would like to quickly collapse the
invocation of t.FailNow() and t.Log() we can simply use t.Fatal(). We use t.Logf() instead of t.Log() in our test functions but conveniently for us, the testing package also implements a t.Fatalf() function, which is a combination of t.FailNow() + t.Logf().

Let’s see how we can use it in our TestMax test function:

func TestMax(t *testing.T) {
    cases := []TestCase{
        TestCase{
            input: []int{1, 2, 3, 4, 5},
            expected: 5,
        },
        TestCase{
            input: []int{-1, -2, -3, -4, -5},
            expected: -1,
        },
        TestCase{
            input: []int{0},
            expected: 0,
        },
    }

    for _, c := range cases {
        actual := Max(c.input)
        expected := c.expected

        if actual != expected {
            t.Fatalf("Expected %d, got %d", expected, actual)
        }
    }
}

The change is pretty simple. Basically, we keep the log format and content, but instead of passing it to t.Logf() we pass it to t.Fatalf() as an argument, while t.FailNow() can be completely removed. If we run our specs again we will not see any difference in the behaviour or the output in comparison to the previous implementation:

› go test
-------- FAIL: TestMax (0.00s)
    max_test.go Expected 5, got -100
-------- FAIL: TestMaxEmptySlice (0.00s)
    max_test.go Expected -1, got -100
FAIL
exit status 1
FAIL    _/users/Ilija/Documents/testing 0.005s

The style of invocation of t.Fatalf() we see here also applies to t.Error() and t.Errorf(). We can make a small change to our TestMaxEmptySlice function and use t.Errorf() instead of a combination of t.Logf() and t.Fail():

func TestMaxEmptySlice t *testing.T) {
    input := []int{}
    actual := Max(input)
    expected := -1

    if actual != expected {
        t.Errorf("Expected %d, got %d", expected, actual)
    }
}

The usage of t.Errorf() is the same as the one of t.Fatalf() in the previous example. As expected, the output and the behaviour of the test function also will not change:

› go test
-------- FAIL: TestMax (0.00s)
    max_test.go Expected 5, got -100
-------- FAIL: TestMaxEmptySlice (0.00s)
    max_test.go Expected -1, got -100
FAIL
exit status 1
FAIL    _/users/Ilija/Documents/testing 0.005s

When to use `Error(f)` or `Fatal(f)`

So, we took a look at all the variations of signaling test failures and logging such failures and the behaviour of the respective functions. For those of you wondering when you should use Error or Fatal, I think that’s a very important question that we should explore and find a good answer for.

Now, there’s no hard and fast answer here. In cases like these, we need to look at certain rules or guidelines on when to use these functions. If we look at the composition of these two functions, it would look something like this:

* Error() = Fail() + Log()
* Errorf() = Fail() + Logf()
* Fatal() = FailNow() + Log()
* Fatalf() = FailNow() + Logf()

By looking at the composition of these functions, getting a clearer indication of the intended usage is easier:

If marking the test function as failed, but continuing its execution can/will provide the user (the programmer) more information that can aid them in fixing the failure, it’s better to use Error(f) instead of Fatal(f). If unsure, start with Error(f) and work your way towards a definitive approach. There are multiple examples to this scenario: your test function is to execute multiple queries against a test database and not failing on the error from the first query might give you more debugging information.

In cases, where a test function cannot recover from the failure and continuing its execution is useless/doesn’t make sense, it’s better to use Fatal(f) and stop the further execution. Such cases are errors while loading fixtures or filesystem permission errors.

These are two guidelines you can use to choose what functions to use. Of course, there are exceptions to this so it’s up to us to use good judgement when writing our test functions.

In closing

In this article, we explored a lot of the details around failing tests. We went deep in to understand how failure logging is done using Go’s testing package - although it was unclear at that moment how it is related to failing tests. Then we went on to explore signalling test failures and the different approaches we can take when marking our tests as failed.

We looked in depth what is the difference between Fail and FailNow, and how to apply them better in our tests. In the end, we looked at some neat functions that the testing package has that make our lives a bit simpler. We discussed two different strategies of using the Error and the Fatal functions in our tests.

If you would like to learn more about marking tests as failed in Go, you can check the documentation of the testing package.

Testing in Go: First Principles

Ilija Eftimov — Wed, 08 May 2019 00:00:00 +0000

If you have any programming experience, whether that’s as a student or a professional, there’s a good chance you have heard about testing. It’s an omnipresent topic, be it on conferences, books or articles. (See what I did there?)

Also, it seems like a topic that everyone agrees on - yes, testing is good and we should do it. There are many reasons why folks consider testing good for you code. But, before we go down the rabbit hole and discuss the pros and cons of testing, let’s learn how we can test our Go code.

Of course, through actual examples that you can follow along.

What is testing?

Now, before we go on, I’ll take you on a short trip down memory lane. At this moment of my carreer I have probably written thousands (if not tens of thousands) of test cases. And I have failed quite a bit at testing, especially as a novice. So if you are new to test or just haven’t gotten it under your belt yet – worry not, I got you covered.

Before you start with testing there’s one idea I would like you to know: just like your programs are consisted of code, also your tests are code. And really, that’s something you have to remember. It’s simple - test are code.

What this testing code does is it invokes the code that powers your programs and checks if what it returns is what is expected. Quite simple, innit?

It all revolves around setting expectations and then making your program meet these expectations. Usually languages provide you with packages or libraries to test your code, with each language or library having its own conventions on what testing looks like. But really, it’s just that - meeting expectations.

That’s testing in a nutshell. Let’s move on.

What is a test?

So, what is a test then?

It’s quite simple - repeatable steps by which one can verify if something is working as it is supposed to.

What is a test in Go terms?

Quite similarly, it’s a piece of code that you can run many times and it will check if your code is working as intended.

That’s it. Shall we write one?

Testing our Go code

To write a test, we first have to have a program to test. Let’s implement a simple function that will take a slice of int and return its biggest number:

func Max(numbers []int) int {
    var max int

    for _, number := range numbers {
        if number > max {
            max = number
        }
    }

    return max
}

Pretty simple. We take a slice of ints and return the largest. So, how can we test that our code works as expected?

Let’s write a function that will take two arguments: a slice of ints and the maximum of the ints in that slice. We will call it TestMax:

func TestMax(numbers []int, expected int) string {
    output := "Pass"
    actual := Max(numbers)
    if actual != expected {
        output = fmt.Sprintf("Expected %v, but instead got %v!", expected, actual)
    }
    return output
}

The TestMax function will check if the result of the Max function call matches the expected result. If it does, it will simply return "Pass", otherwise it will return an informative string with what it was expecting and what it got.

Let’s use it in our main function:

func main() {
    fmt.Println(TestMax([]int{1, 2, 3, 4}, 4))
}

The main function will invoke the TestMax once here, with a slice that contains 1,2,3,4 as argument and 4 as the expected maximum.

You might already be thinking that the example will pass. Let’s run it:

$ go run max.go
Pass

As expected - the example passed. Let’s add two more:

func main() {
    fmt.Println(TestMax([]int{1, 2, 3, 4}, 4))
    fmt.Println(TestMax([]int{4, 2, 1, 4}, 3))
    fmt.Println(TestMax([]int{0, 0, 0, 0}, 1))
}

Here we add two more examples, with two different pairs of arguments:

A slice containing 4,2,1,4 and 3, the expected maximum
A slice containing four zeroes, and 1 as the expected maximum

If we would run it, both of these examples would fail. The reason for the failures would be that none of the two slices of ints that we added do not match the expected maximum that we pass as a second argument to both of the calls. In the second example the maximum is 4, while we expect a 3. In the third example, the maximum is 0 while we expect 1.

Testing our Max function can be done with just one function (TestMax). As long as we can / an input for the function and an expected output we can test our functions.

What is important to understand here is that testing can be very simple. Here, we are able to check if our function is working as expected without any fancy frameworks and libraries - just plain Go code. If you have any experience with testing you already know that this approach does not scale too far, but it’s very good to understand the idea that tests are just code.

Using this approach, we could technically even write our own testing framework/library. The good thing is that Go already has a testing package included in its standard library, so we can avoid that.

Testing with Go’s `testing` package

Golang’s testing package provides support for automated testing of Go packages. It exposes a set of useful functions that we can use to get couple of benefits: a better looking code, standardised approach to testing and nicer looking output. Also, we eliminate the need to create our own reporting of failed/passed tests.

Let’s see how we could test our Max function using the testing package.

First, we need to create a max_test.go file, which will be the test counterpart to our max.go (where our Max function is defined). Most importantly, both of the files have to be part of the same package (in our example main):

// max_test.go
package main

import "testing"

func TestMax(t *testing.T) {
    actual := Max([]int{1, 2, 3, 4})
    if actual != 4 {
        t.Errorf("Expected %d, got %d", 4, actual)
    }
}

The testing package that is imported allows for benchmarking and testing. In our test we use the testing.T type, which when passed to Test functions manages the test state and formats the test logs.

Within the TestMax function, we get the result of the Max function and we assign it to actual. Then, we compare actual to the expected result (4). If the comparison fails, then we make the test fail by using the t.Errorf function and we supply the error message.

What happens when we run `go test`

Golang’s testing package also comes with its buddy – the go test command. This command automates testing the packages named by the import paths. go test recompiles each package along with any files with names matching the file pattern *_test.go.

To run this file, we have to use the go test command:

› go test
PASS
ok github.com/fteem/testing_in_go   0.007s

As you can see, we did not need to tell Golang which tests to run - it figured this out on its own. This is because go test is smartly done, with two different running modes.

The first mode, is called local directory mode. This mode is active whe the command invoked with no arguments. In local directory mode, go test will compile the package sources and the tests found in the current directory and then will run the resulting test binary.

After the package test finishes, go test prints a summary line showing the test status (ok or FAIL), package name, and elapsed time. This is what we actually see in our output - our test have passed. Looking at the output above we can also see that they passed in 0.007s. Pretty fast!

The second mode is called package list mode. This mode is activated when the command is invoked with explicit arguments. In this mode, go test will compile and test each of the packages listed as arguments. If a package test passes, go test prints only the final ‘ok’ summary line. If a package test fails, go test prints the full test output.

For now, we can stick with using the first mode. We will see when and how to use the second mode in one of the next articles.

Dealing with test failures

Now that we have a passing example, let’s see what tests failures look like. Let’s add another example, which we will purposely fail:

func TestMaxInvalid(t *testing.T) {
    actual := Max([]int{1, 2, 3, 4})
    if actual != 5 {
        t.Errorf("Expected %d, got %d", 5, actual)
    }
}

The TestMaxInvalid is very similar to the test function we had before, with having the wrong expectations being the only difference. Simply put - we know that Max will return 4 here, but we are expecting a 5.

While we are here, let’s add one more example where we would pass an empty slice as an argument to Max and expect -1 as a result:

func TestMaxEmpty(t *testing.T) {
    actual := Max([]int{})
    if actual != -1 {
        t.Errorf("Expected %v, got %d", -1, actual)
    }
}

Let’s run go test again and see the output:

$ go test
-------- FAIL: TestMaxInvalid (0.00s)
    max_test.go Expected 5, got 4
-------- FAIL: TestMaxEmpty (0.00s)
    max_test.go Expected -1, got 0
FAIL
exit status 1
FAIL    github.com/fteem/testing_in_go  0.009s

The two new tests failed unsurprisingly. If we inspect the output here we will notice that there are two lines per failed tests. Both of these lines, start with --- FAIL: and have the test function name after. At the end of the line there’s also the time it took for the test function to run.

In the second lines, we see the test file name with the line number of where the failure occurred. More specifically, this is where in both of our test files we invoke t.Errorf.

Let’s make our tests pass. First, we need to fix the expectation in the TestMaxInvalid test function:

func TestMaxInvalid(t *testing.T) {
    actual := Max([]int{1, 2, 3, 4})
    if actual != 4 {
        t.Errorf("Expected %d, got %d", 4, actual)
    }
}

Now, when we run it we should see one less failure:

› go test
-------- FAIL: TestMaxEmpty (0.00s)
    max_test.go Expected -1, got 0
FAIL
exit status 1
FAIL    github.com/fteem/testing_in_go  0.006s

Good. We could technically remove the TestMaxInvalid as it is the same as the TestMax function. To make the other test pass, we need to return -1 when the slice received as argument in Max is empty:

package main

func Max(numbers []int) int {
    if len(numbers) == 0 {
        return -1
    }

    var max int

    for _, number := range numbers {
        if number > max {
            max = number
        }
    }

    return max
}

The len function will check the length of the numbers slice. If it’s 0, it will return -1. Let’s run the tests again:

› go test
PASS
ok github.com/fteem/testing_in_go   0.006s

Back to passing tests. With our new change the Max function will return -1 when the slice in arguments is empty.

In closing

What we talked about in this article is about what tests are in fact. We understood that testing is useful and that test are just code - nothing more. We saw how we could test our own code without any libraries or frameworks, with just simple Golang code.

Then, we went on to explore Golang’s testing package. We saw how an actual test function looks like. We talked about function definitions, the testing.T argument that we have to pass in and how to fail a test. Then we added some more tests for our Max function and made its tests pass.

As you can see, testing in a nutshell is a very simple but powerful technique. With a little bit of simple code we can assure that our code functions in an expected matter, that we can control. And with any new functionality added to our code, we can easily throw in another test to make sure it is covered.

Obviously, there is much more to testing that we will explore in other articles, but now that we are confident with these basic ideas and approaches we can build our knowledge on top of.

Before we stop here, please let me know in the comments what you like and dislike about testing your code? Also, what topics in testing you find confusing or challenging?

Golang Datastructures: Trees

Ilija Eftimov — Wed, 13 Feb 2019 00:00:00 +0000

You can spend quite a bit of your programming career without working with trees, or just by simply avoiding them if you don’t understand them (which is what I had been doing for a while).

Now, don't get me wrong - arrays, lists, stacks and queues are quite powerful data structures and can take you pretty far, but there is a limit to their capabilities, how you can use them and how efficient that usage can be. When you throw in hash tables to that mix, you can solve quite some problems, but for many of the problems out there trees are a powerful (and maybe the only) tool if you have them under your belt.

Photo by Sebastian Engler on Unsplash

So, let's look at trees and then we can try to use them in a small exercise.

A touch of theory

Arrays, lists, queues, stacks store data in a collection that has a start and an end, hence they are called "linear". But when it comes to trees and graphs, things can get confusing since the data is not stored in a linear fashion.

Trees are called nonlinear data structures. In fact, you can also say that trees are hierarchical data structures since the data is stored in a hierarchical way.

For your reading pleasure, Wikipedia’s definition of trees:

A tree is a data structure made up of nodes or vertices and edges without
having any cycle. The tree with no nodes is called the null or empty tree. A
tree that is not empty consists of a root node and potentially many levels of
additional nodes that form a hierarchy.

What the definition states are that a tree is just a combination of nodes (or vertices) and edges (or links between the nodes) without having a cycle.

For example, the data structure represented on the diagram is a combination of nodes, named from A to F, with six edges. Although all of its elements look like they construct a tree, the nodes A, D, E and F have a cycle, therefore this structure is not a tree.

If we would break the edge between nodes F and E and add a new node called G with an edge between F and G, we would end up with something like this:

Now, since we eliminated the cycle in this graph, we can say that we have a valid tree. It has a root with the name A, with a total of 7 nodes. Node A has 3 children (B, D & F) and those have 3 children (C, E & G respectively). Therefore, node A has 6 descendants. Also, this tree has 3 leaf nodes (C, E
& G) or nodes that have no children.

What do B, D & F have in common? They are siblings because they have the same parent (node A). They all reside on level 1 because to get from each of them to the root we need to take only one step. For example, node G has level 2, because the path from G to A is: G -> F -> A, hence we need to follow two edges to get to A.

Now that we know a bit of theory about trees, let’s see how we can solve some problems.

Modelling an HTML document

If you are a software developer that has never written any HTML, I will just assume that you have seen (or have an idea) what HTML looks like. If you have not, then I encourage you to right click on the page that you are reading this and click on 'View Source'.

Seriously, go for it, I'll wait...

Browsers have this thing baked in, called the DOM - a cross-platform and language-independent application programming interface, which treats internet documents as a tree structure wherein each node is an object representing a part of the document. This means that when the browser reads your document's HTML code it will load it and create a DOM out of it.

So, let’s imagine for a second we are developers working on a browser, like Chrome or Firefox and we need to model the DOM. Well, to make this exercise easier, let’s see a tiny HTML document:

<html>
  <h1>Hello, World!</h1>
  <p>This is a simple HTML document.</p>
</html>

So, if we would model this document as a tree, it would look something like this:

Now, we could treat the text nodes as separate Nodes, but we can make our lives simpler by assuming that any HTML element can have text in it.

The html node will have two children, h1 and p, which will have tag, text and children as fields. Let’s put this into code:

type Node struct {
    tag      string
    text     string
    children []*Node
}

A Node will have only the tag name and children optionally. Let’s try to create the HTML document we saw above as a tree of Nodes by hand:

func main() {
        p := Node{
                tag:  "p",
                text: "This is a simple HTML document.",
                id:   "foo",
        }

        h1 := Node{
                tag:  "h1",
                text: "Hello, World!",
        }

        html := Node{
                tag:      "html",
                children: []*Node{&p, &h1},
        }
}

That looks okay, we have a basic tree up and running now.

Building MyDOM - a drop-in replacement for the DOM 😂

Now that we have some tree structure in place, let's take a step back and see what kind of functionality would a DOM have. For example, if MyDOM ^(TM) would be a drop-in replacement of a real DOM, then with JavaScript we should be able to access nodes and modify them.

The simplest way to do this with JavaScript would be to use

document.getElementById('foo')

This function would lookup in the document tree to find the node whose ID is foo. Let's update our Node struct to have more attributes and then work on writing a lookup function for our tree:

type Node struct {
  tag      string
  id       string
  class    string
  children []*Node
}

Now, each of our Node structs will have a tag, children which is a slice of pointers to the children of that Node, id which is the ID of that DOM node and class which is the classes that can be applied to this DOM node.

Now, back to our getElementById lookup function. Let's see how we could implement it. First, let's build an example tree that we can use for our lookup algorithm:

<html>
  <body>
    <h1>This is a H1</h1>
    <p>
      And this is some text in a paragraph. And next to it there's an image.
      <img src="http://example.com/logo.svg" alt="Example's Logo"/>
    </p>
    <div class='footer'>
      This is the footer of the page.
      <span id='copyright'>2019 &copy; Ilija Eftimov</span>
    </div>
  </body>
</html>

This is a quite complicated HTML document. Let's sketch out its structure in Go using the Node struct as a building block:

image := Node{
        tag: "img",
        src: "http://example.com/logo.svg",
        alt: "Example's Logo",
}

p := Node{
        tag:      "p",
        text:     "And this is some text in a paragraph. And next to it there's an image.",
        children: []*Node{&image},
}

span := Node{
        tag:  "span",
        id:   "copyright",
        text: "2019 &copy; Ilija Eftimov",
}

div := Node{
        tag:      "div",
        class:    "footer",
        text:     "This is the footer of the page.",
        children: []*Node{&span},
}

h1 := Node{
        tag:  "h1",
        text: "This is a H1",
}

body := Node{
        tag:      "body",
        children: []*Node{&h1, &p, &div},
}

html := Node{
        tag:      "html",
        children: []*Node{&body},
}

We start building this tree bottom - up. That means we create structs from the most deeply nested structs and working up towards body and html. Let's look at a graphic of our tree:

Implementing Node Lookup 🔎

So, let's continue with what we were up to - allow JavaScript to call getElementById on our document and find the Node that it's looking for.

To do this, we have to implement a tree searching algorithm. The most popular approaches to searching (or traversal) of graphs and trees are Breadth First Search (BFS) and Depth First Search (DFS).

Breadth-first search ⬅➡

BFS, as its name suggests, takes an approach to traversal where it explores nodes in "width" first before it goes in "depth". Here's a visualisation of the steps a BFS algorithm would take to traverse the whole tree:

As you can see, the algorithm will take two steps in depth (over html and body), but then it will visit all of the body's children nodes before it proceeds to explore in depth and visit the span and img nodes.

If you would like to have a step-by-step playbook, it would be:

We start at the root, the html node
We push it on the queue
We kick off a loop where we loop while the queue is not empty
We check the next element in the queue for a match. If a match is found, we return the match and we're done.
When a match is not found, we take all of the children of the node-under-inspection and we add them to the queue, so they can be inspected
GOTO 4

Let's see a simple implementation of the algorithm in Go and I'll share some tips on how you can remember the algorithm easily.

func findById(root *Node, id string) *Node {
        queue := make([]*Node, 0)
        queue = append(queue, root)
        for len(queue) > 0 {
                nextUp := queue[0]
                queue = queue[1:]
                if nextUp.id == id {
                        return nextUp
                }
                if len(nextUp.children) > 0 {
                        for _, child := range nextUp.children {
                                queue = append(queue, child)
                        }
                }
        }
        return nil
}

The algorithm has three key points:

The queue - it will contain all of the nodes that the algorithm visits
Taking the first element of the queue, checking it for a match, and proceeding with the next nodes if no match is found
Queueing up all of the children nodes for a node before moving on in the queue

Essentially, the whole algorithm revolves around pushing children nodes on a queue and inspecting the nodes that are queued up. Of course, if a match is not found at the end we return nil instead of a pointer to a Node.

Depth-first search ⬇

For completeness sake, let's also see how DFS would work.

As we stated earlier, the depth-first search will go first in depth by visiting as many nodes as possible until it reaches a leaf. When then happens, it will backtrack and find another branch on the tree to drill down on.

Let's see what that means visually:

If this is confusing to you, worry not - I've added a bit more granularity in the steps to aid my explanation.

The algorithm starts off just like BFS - it walks down from html to body and to div. Then, instead of continuing to h1, it takes another step to the leaf span. Once it figures out that span is a leaf, it will move back up to div to find other branches to explore. Since it won't find any, it will move back to body to find new branches proceeding to visit h1. Then, it will do the same exercise again - go back to body and find that there's another branch to explore - ultimately visiting p and the img nodes.

If you're wondering something along the lines of "how can we go back up to the parent without having a pointer to it", then you're forgetting one of the oldest tricks in the book - recursion. Let's see a simple recursive Go implementation of the algorithm:

func findByIdDFS(node *Node, id string) *Node {
        if node.id == id {
                return node
        }

        if len(node.children) > 0 {
                for _, child := range node.children {
                        findByIdDFS(child, id)
                }
        }
        return nil
}

Finding by class name 🔎

Another functionality MyDOM ^(TM) should have is the ability to find nodes by a class name. Essentially, when a JavaScript script executes getElementsByClassName, MyDOM should know how to collect all nodes with a certain class.

As you can imagine, this is also an algorithm that would have to explore the whole MyDOM ^(TM) tree and pick up the nodes that satisfy certain conditions.

To make our lives easier, let's first implement a function that a Node can receive, called hasClass:

func (n *Node) hasClass(className string) bool {
        classes := strings.Fields(n.classes)
        for _, class := range classes {
                if class == className {
                        return true
                }
        }
        return false
}

hasClass takes a Node's classes field, splits them on each space character and then loops the slice of classes and tries to find the class name that we are interested in. Let's write a couple of tests that will test this function:

type testcase struct {
        className      string
        node           Node
        expectedResult bool
}

func TestHasClass(t *testing.T) {
        cases := []testcase{
                testcase{
                        className:      "foo",
                        node:           Node{classes: "foo bar"},
                        expectedResult: true,
                },
                testcase{
                        className:      "foo",
                        node:           Node{classes: "bar baz qux"},
                        expectedResult: false,
                },
                testcase{
                        className:      "bar",
                        node:           Node{classes: ""},
                        expectedResult: false,
                },
        }

        for _, case := range cases {
                result := case.node.hasClass(test.className)
                if result != case.expectedResult {
                        t.Error(
                                "For node", case.node,
                                "and class", case.className,
                                "expected", case.expectedResult,
                                "got", result,
                        )
                }
        }
}

As you can see, the hasClass function will detect if a class name is in the list of classes on a Node. Now, let's move on to implementing MyDOM's implementation of finding all Nodes by class name:

func findAllByClassName(root *Node, className string) []*Node {
        result := make([]*Node, 0)
        queue := make([]*Node, 0)
        queue = append(queue, root)
        for len(queue) > 0 {
                nextUp := queue[0]
                queue = queue[1:]
                if nextUp.hasClass(className) {
                        result = append(result, nextUp)
                }
                if len(nextUp.children) > 0 {
                        for _, child := range nextUp.children {
                                queue = append(queue, child)
                        }
                }
        }
        return result
}

If the algorithm seems familiar, that's because you're looking at a modified findById function. findAllByClassName works just like findById, but instead of returning the moment it finds a match, it will just append the matched Node to the result slice. It will continue doing that until all of the Nodes have been visited.

If there are no matches, the result slice will be empty. If there are any matches, they will be returned as part of the result slice.

Last thing worth mentioning is that to traverse the tree we used a Breadth-first approach here - the algorithm uses a queue for each of the Nodes and loops over them while appending to the result slice if a match is found.

Deleting nodes 🗑

Another functionality that is often used in the DOM is the ability to remove nodes. Just like the DOM can do it, also our MyDOM ^(TM) should be able to handle such operations.

The simplest way to do this operation in JavaScript is:

var el = document.getElementById('foo');
el.remove();

While our document knows how to handle getElementById (by calling findById under the hood), our Nodes do not know how to handle a remove function. Removing a Node from the MyDOM ^(TM) tree would be a two-step process:

We have to look up to the parent of the Node and remove it from its parent's children collection;
If the to-be-removed Node has any children, we have to remove those from the DOM. This means we have to remove all pointers to each of the children and its parent (the node to-be-removed) so Go's garbage collector can free up that memory.

And here's a simple way to achieve that:

func (node *Node) remove() {
        // Remove the node from it's parents children collection
        for idx, sibling := range n.parent.children {
                if sibling == node {
                        node.parent.children = append(
                                node.parent.children[:idx],
                                node.parent.children[idx+1:]...,
                        )
                }
        }

        // If the node has any children, set their parent to nil and set the node's children collection to nil
        if len(node.children) != 0 {
                for _, child := range node.children {
                        child.parent = nil
                }
                node.children = nil
        }
}

A *Node would have a remove function, which does the two-step process of the Node's removal.

In the first step, we take the node out of the parent's children list, by looping over them and removing the node by appending the elements before the node in the list, and the elements after the node.

In the second step, after checking for the presence of any children on the node, we remove the reference to the parent from all the children and then we set the Node's children to nil.

Where to next?

Obviously, our MyDOM ^(TM) implementation is never going to become a replacement for the DOM. But, I believe that it's an interesting example that can help you learn and it's pretty interesting problem to think about. We interact with browsers every day, so thinking how they could function under the hood is an interesting exercise.

If you would like to play with our tree structure and write more functionality, you can head over to WC3's JavaScript HTML DOM Document documentation and think about adding more functionality to MyDOM.

Obviously, the idea behind this article was to learn more about trees (graphs) and learn about the popular searching/traversal algorithms that are used out there. But, by all means, please keep on exploring and experimenting and drop me a comment about what improvements you did to your MyDOM implementation.

OTP in Elixir: Learn GenServer by Building Your Own URL Shortener

Ilija Eftimov — Sat, 26 Jan 2019 00:00:00 +0000

Looking at any programming language you will (hopefully!) find a rich and useful standard library. I started my professional career as a software developer with Ruby, which has quite an easy-to-use and well-documented standard library with a plethora of modules and classes to use. Personally, I find the Enumerable module in Ruby with all its nice methods simply brilliant.

You might be coming from a different language, but I am sure that any serious programming language out there has a set of classes, modules, methods and functions to make your life (and work) easy.

So, what about Elixirs stdlib?

No surprise there too – Elixir has a well-documented and easy-to-use standard library. But, because it works on top of the BEAM virtual machine and inherts plenty from Erlang’s rich history, it also has a bit more – something called OTP.

Photo by Mario Caruso on Unsplash

Meet OTP 👋

From Wikipedia’s article on OTP:

OTP is a collection of useful middleware, libraries, and tools written in the Erlang programming language. It is an integral part of the open-source distribution of Erlang. The name OTP was originally an acronym for Open TelecomPlatform, which was a branding attempt before Ericsson released Erlang/OTP as open source. However, neither Erlang nor OTP is specific to telecom applications.

In continuation, it states:

It (OTP) contains:

an Erlang interpreter (called BEAM);

an Erlang compiler;

a protocol for communication between servers (nodes);

a CORBA Object Request Broker;

a static analysis tool called Dialyzer;

a distributed database server (Mnesia); and

many other libraries.

While I do not consider myself an expert in Elixir, Erlang, the BEAM or OTP by any stretch of the imagination, I would like to take you on a journey to one of the most useful and well-known behaviours of OTP – GenServer.

I am an OTP expert as much as I can find my way around here 👆 – I know
how to open the door. Photo by Jon Flobrant on Unsplash

In continuation, we will use BEAM processes, so if you’re not familiar with spawning new processes, sending and receiving messages to/from them, then it’s best to head over to“Understanding the basics of Elixir’s concurrency model”and give it a quick read. It will help you understand processes and concurrency in Elixir so you can apply the knowledge in the project that we work on in this article. I promise.

Shortening a link ✂️

Let’s write a URL shortener module that will run in a BEAM process and can receive multiple commands:

shorten – takes a link, shortens it and returns the short link as a response
get – take a short link and return the original one
flush – erase the URL shortener memory
stop – stop the process

defmodule URLShortener do
  def start do
    spawn( __MODULE__ , :loop, [%{}])
  end

  def loop(state) do
    receive do
      {:stop, caller} ->
        send caller, "Shutting down."
      {:shorten, url, caller} ->
        url_md5 = md5(url)
        new_state = Map.put(state, url_md5, url)
        send caller, url_md5
        loop(new_state)
      {:get, md5, caller} ->
        send caller, Map.fetch(state, md5)
        loop(state)
      :flush ->
        loop(%{})
      _ ->
        loop(state)
    end
  end

  defp md5(url) do
    :crypto.hash(:md5, url)
    |> Base.encode16(case: :lower)
  end
end

What the module does is when the process starts it will recursively call the URLShortener.loop/1 function, until it receives the {:stop, caller} message.

If we zoom into the {:shorten, url, caller} case we notice that we generate aMD5 digest from the URL and then we update the state map which creates a new map (called new_state). Once we get the digest we store it in a map with the key being the MD5 and the value is the actual URL. The state map will look like:

%{
  "99999ebcfdb78df077ad2727fd00969f" => "https://google.com",
  "76100d6f27db53fddb6c8fce320f5d21" => "https://elixir-lang.org",
  "3097fca9b1ec8942c4305e550ef1b50a" => "https://github.com",
  ...
}

Then, we send the MD5 value back to the caller. Obviously, this is not howbit.ly or the likes work, as their links are much shorter. (For those interested,here’san interesting discussion on the topic). However, for the purpose of this article, we’ll stick to simple MD5 digest of the URL.

The other two commands, get and flush, are pretty simple. get returns only a single value from the state map, while flush invokes loop/1 with an empty map, effectively removing all the shortened links from the process’ state(memory).

Let’s run our shortener in an IEx session:

iex(22)> shortener = URLShortener.start
#PID<0.141.0>

iex(23)> send shortener, {:shorten, "https://ieftimov.com", self()}
{:shorten, "https://ieftimov.com", #PID<0.102.0>}

iex(24)> send shortener, {:shorten, "https://google.com", self()}
{:shorten, "https://google.com", #PID<0.102.0>}

iex(25)> send shortener, {:shorten, "https://github.com", self()}
{:shorten, "https://github.com", #PID<0.102.0>}

iex(26)> flush
"8c4c7fbc57b08d379da5b1312690be04"
"99999ebcfdb78df077ad2727fd00969f"
"3097fca9b1ec8942c4305e550ef1b50a"
:ok

iex(27)> send shortener, {:get, "99999ebcfdb78df077ad2727fd00969f", self()}
{:get, "99999ebcfdb78df077ad2727fd00969f", #PID<0.102.0>}

iex(28)> flush
"https://google.com"
:ok

iex(29)> send shortener, {:get, "8c4c7fbc57b08d379da5b1312690be04", self()}
{:get, "8c4c7fbc57b08d379da5b1312690be04", #PID<0.102.0>}

iex(30)> flush
"https://ieftimov.com"
:ok

iex(31)> send shortener, {:get, "3097fca9b1ec8942c4305e550ef1b50a", self()}
{:get, "3097fca9b1ec8942c4305e550ef1b50a", #PID<0.102.0>}

iex(32)> flush
"https://github.com"
:ok

Working as expected – we send three different URLs for shortening, we receive their MD5 digests back in the process mailbox and when we query for them we get each of them back.

Although our URLShortener module works pretty neatly now, it actually lacks quite a bit of functionality. Sure, it does handle the happy path really well, but when it comes to error handling, tracing or error reporting it falls really short. Additionally, it does not have a standard interface to add more functions to the process – we sort of came up with it as we went on.

After reading all of that you’re probably thinking there is a better way to do this. And you’d be right to think so – let’s learn more about GenServers.

Enter GenServer 🚪

GenServer is an OTP behaviour. Behaviour in this context refers to three things:

an interface, which is a set of functions;
an implementation, which is the application-specific code, and
the container, which is a BEAM process

This means that a module can implement a certain group of functions (interface or signatures), that under the hood implement some callback functions (which are specific to the behaviour you work on), that are run within a BEAM process.

For example, GenServer is a gen eric server behaviour – it expects for each of the functions defined in it’s interface a set of callbacks which will handle the requests to the server. This means that the interface functions will be used by the clients of the generic server, a.k.a. the client API, while the callbacks defined will essentially be the server internals (“the backend”).

So, how does a GenServer work? Well, as you can imagine we cannot go too deep on the hows of GenServer, but we need to get a good grasp on some basics:

Server start & state
Asynchronous messages
Synchronous messages

Server start & state

Just like with our URLShortener we implemented, every GenServer is capable of holding state. In fact, GenServers must implement a init/1 function, which will set the initial state of the server (see the init/1 documentationhere for more details).

To start the server we can run:

GenServer.start_link( __MODULE__ , :ok, [])

GenServer.start_link/3 will invoke the init/1 function of the __MODULE__,passing in :ok as an argument to init/1. This function call will block until init/1 returns, so usually in this function, we do any required setup of the server process (that might be needed). For example, in our case, to rebuild URLShortener using a GenServer behaviour, we will need an init/1 function to set the initial state (empty map) of the server:

def init(:ok) do
  {:ok, %{}}
end

That’s all. start_link/3 will call init/1 with the :ok argument, which will return an :ok and set the state of the process to an empty map.

Sync & async messages 📨

As most servers out there, GenServers can also receive and reply to requests(if needed). As the heading suggests, there are two types of requests that GenServers handle – the ones expect a response (call) and the others that don’t (cast). Therefore, GenServers define two callback functions -handle_call/3 and handle_cast/2.

We will look at these functions in more depth a bit later.

Reimplementing `URLShortener`, using `GenServer` ♻️

Let’s look at how we can flip the implementation to use GenServer.

First, let’s add the shell of the module, the start_link/1 function and the init/1 function that start_link/1 will invoke:

defmodule URLShortener do
  use GenServer

  def start_link(opts \\ []) do
    GenServer.start_link( __MODULE__ , :ok, opts)
  end

  def init(:ok) do
    {:ok, %{}}
  end
end

The notable changes here are the use of the GenServer behaviour in the module, the start_link/1 function which invokes GenServer.start_link/3 which would, in fact, call the init/1 function with the :ok atom as an argument.Also, it’s worth noting that the empty map that the init/1 function returns in the tuple is the actual initial state of the URLShortener process.

Let’s give it a spin in IEx:

iex(1)> {:ok, pid} = URLShortener.start_link
{:ok, #PID<0.108.0>}

That’s all we can do at this moment. The difference here is that the GenServer.start_link/3 function will return a tuple with an atom (:ok) and the PID of the server.

Stopping the server ✋

Let’s add the stop command:

defmodule URLShortener do
  use GenServer

  # Client API
  def start_link(opts \\ []), do: GenServer.start_link( __MODULE__ , :ok, opts)

  def stop(pid) do
    GenServer.cast(pid, :stop)
  end

  # GenServer callbacks
  def init(:ok), do: {:ok, %{}}

  def handle_cast(:stop, state) do
    {:stop, :normal, state}
  end
end

Yes, I know I said we’ll add one command but ended up adding two functions:stop/1 and handle_cast/2. Bear with me now:

Because we do not want to get a response back on the stop command, we will use GenServer.cast/2 in the stop/1 function. This means that when that command is called by the client (user) of the server, the handle_cast/2 callback will be triggered on the server. In our case, the handle_cast/2 function will return a tuple of three items – {:stop, :normal, state}.

Returning this tuple stops the loop and another callback called terminate/2 is called (which is defined in the behaviour but not implemented by URLShortener)with the reason :normal and state state. The process will exit with reason:normal.

This way of working with GenServer allows us to only define callbacks and the GenServer behaviour will know how to handle the rest. The only complexity resides in the fact that we need to understand and know most types of returns that the callback functions can have.

Another thing worth pointing out is that each function that will be used by the client will take a PID as a first argument. This will allow us to send messages to the correct GenServer process. Going forward we will not acknowledge PIDs presence – we accept that it’s mandatory for our URLShortener to work. Later we will look at ways we can skip passing the PIDs as arguments.

Let’s jump back in IEx and start and stop a URLShortener server:

iex(1)> {:ok, pid} = URLShortener.start_link
{:ok, #PID<0.109.0>}

iex(2)> Process.alive?(pid)
true

iex(3)> URLShortener.stop(pid)
:ok

iex(4)> Process.alive?(pid)
false

That’s starting and stopping in all of it’s glory.

Shortening a URL

Another thing we wanted our server to have is the ability to shorten URLs, by using their MD5 digest as the short variant of the URL. Let’s do that using GenServer:

defmodule URLShortener do
  use GenServer

  # Client API
  def start_link(opts \\ []), do: GenServer.start_link( __MODULE__ , :ok, opts)
  def stop(pid), do: GenServer.cast(pid, :stop)

  def shorten(pid, url) do
    GenServer.call(pid, {:shorten, url})
  end

  # GenServer callbacks
  def init(:ok), do: {:ok, %{}}
  def handle_cast(:stop, state), do: {:stop, :normal, state}

  def handle_call({:shorten, url}, _from, state) do
    short = md5(url)
    {:reply, short, Map.put(state, short, url)}
  end

  defp md5(url) do
    :crypto.hash(:md5, url)
    |> Base.encode16(case: :lower)
  end
end

Three functions this time, but at least the md5/1 is a replica of the one we had previously. So, let’s look at the other two.

You might be seeing a pattern - we have a function that will be used by the client (shorten/2) and a callback that will be invoked on the server(handle_call/3). This time, there’s a slight difference in the functions used and naming: in shorten/2 we call GenServer.call/2 instead of cast/2, and the callback name is handle_call/3 instead of handle_cast/2.

Why? Well, the difference lies in the response - handle_call/3 will send a reply back to the client (hence the :reply atom in the response tuple), while handle_cast/2 does not do that. Basically casting is an async call where the client does not expect a response, while calling is a sync call where the response is expected.

So, let’s look at the structure of the handle_call/3 callback.

It takes three arguments: the request from the client (in our case a tuple), a tuple describing the client of the request (which we ignore), and the state of the server (in our case a map).

As a response, it returns a tuple with :reply, stating that there will be a reply to the request, the reply itself (in our case the shortened link) and the state which is the state carried over to the next loop of the server.

Of course, handle_call/3 has a bit more intricacies that we will look into later, but you can always check it’sdocumentation to learn more.

Fetching a shortened URL 🔗

Let’s implement the get command, which when provided with a short version of the link it will return the full URL:

defmodule URLShortener do
  use GenServer

  # Client API
  # ...

  def get(pid, short_link) do
    GenServer.call(pid, {:get, short_link})
  end

  # GenServer callbacks
  # ...

  def handle_call({:get, short_link}, _from, state) do
    {:reply, Map.get(state, short), state}
  end
end

The double-function entry pattern again - we add URLShortener.get/2 and another head of the URLShortener.handle_call/3 function.

The URLShortener.get/2 will call GenServer.call/2 under the hood, which when executed will cause the handle_call/3 callback to fire.

The URLShortener.handle_call/3 this time will take the command (:get) and the short_link as the first argument. Looking inside we see that, again, it’s a short function - it only returns a tuple with :reply (which states that the call will have a reply), a call to Map.get/2, whose return will be the actual response of the call, and the state, so the GenServer process maintains the state in the next loop.

At this moment, we can safely say that we have a good idea of the basics on writing functionality for a module that implements the GenServer behaviour.As you might be thinking, there’s much to explore, but these basics will allow you to create GenServers and experiment.

Before you go on, try to implement two more commands:

flush – an async call which will erase the state of the server
count – a sync call returning the number of links in the server’s state

More configurations 🎛

If we circle back to URLShortener.start_link/1 and it’s internals (namely the invocation of GenServer.start_link/3), we will also notice that we can pass options (opts) to the GenServer.start_link/3 function, that are defaulted to an empty list ([]).

What are the options that we can add here? By looking atthe documentation of GenServer.start_link/3 you’ll notice multiple interesting options:

:name - used for name registration. This means that instead of identifying a GenServer by PID, we can put a name on it.
:timeout - sets the server startup timeout (in milliseconds)
:debug - enables debugging by invoking the corresponding function in the:sys module
:hibernate_after - sets the time (in milliseconds) after which the serverprocess will go into hibernation automatically until a new request comes in.This is done by utilising :proc_lib.hibernate/3
:spawn_opt - enables passing more options to the underlying process

Most of these are advanced and are beyond our use-case here. However, there’s one configuration we could use right now: :name.

Naming the server 📢

Let’s modify our URLShortener to take a name in it’s start_link/1 function and test it in IEx. Additionally, since every URLShortener process will have a name, we can refer to the process by name instead of PID - let’s see how that would work in code:

defmodule URLShortener do
  use GenServer

  # Client API
  def start_link(name, opts \\ []) do
    GenServer.start_link( __MODULE__ , :ok, opts ++ [name: name])
  end

  def stop(name), do: GenServer.cast(name, :stop)
  def shorten(name, url), do: GenServer.call(name, {:shorten, url})

  def get(name, short_link) do
    GenServer.call(name, {:get, short_link})
  end

  # GenServer callbacks
  def init(:ok), do: {:ok, %{}}
  def handle_cast(:stop, state), do: {:stop, :normal, state}
  def handle_call({:shorten, url}, _from, state), do: {:reply, md5(url), Map.put(state, md5(url), url)}

  def handle_call({:get, short_link}, _from, state) do
    {:reply, Map.get(state, short_link), state}
  end

  defp md5(url), do: :crypto.hash(:md5, url) |> Base.encode16(case: :lower)
end

That’s all. We added a new argument to URLShortener.start_link/2 and we dropped all usage of PID and replaced it with name.

Let’s take it for a spin in IEx:

iex(1)> {:ok, pid} = URLShortener.start_link(:foo)
{:ok, #PID<0.109.0>}

iex(2)> URLShortener.shorten(:foo, "https://google.com")
"99999ebcfdb78df077ad2727fd00969f"

iex(3)> URLShortener.get(:foo, "99999ebcfdb78df077ad2727fd00969f")
"https://google.com"

iex(4)> URLShortener.stop(:foo)
:ok

iex(5)> Process.alive?(pid)
false

You can see that this is pretty cool - instead of using PID we added a name:foo to the process which allowed us to refer to it using the name instead ofthe PID. Obviously, you can see that to inspect the BEAM process in any fashion we will still need the PID, but for the client the name does the trick.

This combination of name and PID allows us to have reference to the BEAM process while improving the ease of use for the client.

If we would like to simplify things even more, we can turn the URLShortener into a “singleton” server. Before you freak out - it has none of the drawbacks that the singleton pattern that’s infamous in OO programming has. We’re merely stating that we could change the URLShortener to have one and only one process running at a certain time, by setting a static name to it:

defmodule URLShortener do
  use GenServer

  @name :url_shortener_server

  # Client API
  def start_link(opts \\ []) do
    GenServer.start_link( __MODULE__ , :ok, opts ++ [name: @name])
  end

  def stop, do: GenServer.cast(@name, :stop)
  def shorten(url), do: GenServer.call(@name, {:shorten, url})

  def get(short_link) do
    GenServer.call(@name, {:get, short_link})
  end

  # GenServer callbacks
  # ...
end

You can notice that we added a module attribute @name that holds the name of the process. In all the functions from the client API, we dropped the name from the arguments lists and we simply use @name as a reference to the process. This means that there’s going to be only one process for URLShortener with the name :url_shortener_server.

Let’s take it for a spin in IEx:

iex(1)> {:ok, pid} = URLShortener.start_link
{:ok, #PID<0.108.0>}

iex(2)> URLShortener.shorten("https://google.com")
"99999ebcfdb78df077ad2727fd00969f"

iex(3)> URLShortener.shorten("https://yahoo.com")
"c88f320dec138ba5ab0a5f990ff082ba"

iex(4)> URLShortener.get("99999ebcfdb78df077ad2727fd00969f")
"https://google.com"

iex(5)> URLShortener.stop
:ok

iex(6)> Process.alive?(pid)
false

You can notice that although we captured the PID on the first line, we do not need it at all - all of the work is done for us by URLShortener.

In this section, you saw how you can utilise names to more easily work with processes. Let’s review our full implementation of the URLShortener module.

Outro 🖖

Before we wrap up this long tutorial, let’s have a final look at our new URLShortener module, including the count/1 and flush/1 functions:

defmodule URLShortener do
  use GenServer

  # Client API
  def start_link(name, opts \\ []) do
    GenServer.start_link( __MODULE__ , :ok, opts ++ [name: name])
  end

  def shorten(name, url) do
    GenServer.call(name, {:shorten, url})
  end

  def get(name, short) do
    GenServer.call(name, {:get, short})
  end

  def flush(name) do
    GenServer.cast(name, :flush)
  end

  def stop(name) do
    GenServer.cast(name, :stop)
  end

  def count(name) do
    GenServer.call(name, :count)
  end

  # Callbacks
  def init(:ok) do
    {:ok, %{}}
  end

  def handle_cast(:flush, _state) do
    {:noreply, %{}}
  end

  def handle_cast(:stop, state) do
    {:stop, :normal, state}
  end

  def handle_call({:shorten, url}, _from, state) do
    shortened = md5(url)
    new_state = Map.put(state, shortened, url)
    {:reply, shortened, new_state}
  end

  def handle_call({:get, short}, _from, state) do
    {:reply, Map.get(state, short), state}
  end

  def handle_call(:count, _from, state) do
    count = Map.keys(state) |> Enum.count
    {:reply, count, state}
  end

  defp md5(url) do
    :crypto.hash(:md5, url)
    |> Base.encode16(case: :lower)
  end
end

The two callbacks are fairly simple - flush will just send a noreply and set the state to an empty map. count on the other hand will have a reply with the count of the items of the map, which is simply the numbers of keys there are in the state map. That’s all.

While you got to the end of the article your journey with GenServer does not end here. In fact, it just started. GenServers and OTP are very powerful tools that you can use to build generic servers that can live in small BEAM processes and have a very generic approach to building functionality (calls and callbacks).

While we did cover a lot of ground here we didn’t touch on why we named the starting function start_link instead of just start (hint: supervisors convention), or how we would approach testing such a GenServer like URLShortener.

In what kind of scenarios have you used GenServers? Or, if you do not have experience with it, where do you see yourself using it in the future?

Understanding the basics of Elixir’s concurrency model

Ilija Eftimov — Tue, 15 Jan 2019 00:00:00 +0000

This article was originally published on my blog at ieftimov.com.

If you come from an object-oriented background, you might have tried concurrencyin your favourite OO language before. Your mileage will vary, but in generalOO languages are harder to work with when it comes to concurrency. This is dueto their nature - they are designed to keep state in memory and require moreexpertise and experience to be successful with.

How does Elixir stand up to other languages when it comes to concurrency? Well,for starters just being functional and immutable is a big win - no state tomanage. But also, Elixir has more goodies packing under the hood and in itsstandard library.

Being based on Erlang’s virtual machine (a.k.a. the BEAM), Elixir uses processesto run any and all code. Note that from here on any time we mention processes,unless specified otherwise, we are referring to BEAM processes, not OS processes.

Elixir’s processes are isolated from one another, they do not share any memoryand run concurrently. They are very lightweight and the BEAM is capable ofrunning many thousands of them at the same time. That’s why Elixir exposesprimitives for creating processes, communicating between them and variousmodules on process management.

Let’s see how we can create processes and send messages between them.

Creating processes

One of the simplest (yet powerful) tools in Elixir’s toolbelt is IEx - Elixir’sREPL, short for Interactive Elixir. If we jump into IEx and run h self, wewill get the following output:

iex(1)> h self

                      def self()

  @spec self() :: pid()

Returns the PID (process identifier) of the calling process.

Allowed in guard clauses. Inlined by the compiler.

As you can see, self() is a built-in function in Elixir which returns the PID(process identifier) of the calling process. If you run it in IEx it willreturn the PID of IEx:

iex(2)> self
#PID<0.102.0>

Now, just like IEx is a BEAM process, with its own PID, it can also createnew processes. In fact, any BEAM process can spawn BEAM processes. This isdone using the Kernel.spawn/1 function:

iex(3)> spawn(fn -> 1 + 1 end)
#PID<0.108.0>

spawn/1 creates a new process which invokes the function provided as argument,fn -> 1 + 1 end. What you might notice is that we do not see the return valueof the anonymous function because the function ran in a different process. Whatwe get instead is the PID of the spawned process.

Another thing worth noticing is that once we spawn a process it will run rightaway, which means that the function will be immediately executed. We can checkthat using the Process.alive?/1 function:

iex(4)> pid = spawn(fn -> 1 + 1 end)
#PID<0.110.0>

iex(5)> Process.alive?(pid)
false

While we don’t need to look at the Process module in depth right now, it hasquite a bit more functions available for working with processes. You can exploreits documentation here.

Now, let’s look at receiving messages.

Receiving messages in processes

For the purpose of our example, let’s imagine we open a new IEx sessionand we tell the IEx process (a.k.a. self) that it might receive a messagethat it should react to. If this makes you scratch your head a bit, rememberthat since IEx is a BEAM process it can receive messages just like any otherBEAM process.

The messages that we would like our IEx process to receive will be:

tuple containing :hello atom and a string with a name (e.g. {:hello, "Jane"})
tuple containing the :bye atom and a string with a name (e.g. {:bye, "John"})

When the process receives the message in the first case it should reply with“Hello Jane”, while in the second case it should reply with “Bye John”. If wecould use the cond macro in Elixir, it would look something like:

cond message do
  {:hello, name} -> "Hello #{name}"
  {:bye, name} -> "Bye #{name}"
end

To receive messages in a process we cannot use cond, but Elixir provides uswith a function receive/1 which takes a block as an argument. Althoughit’s not cond, it looks very similar to the example above because it lets ususe pattern-matching:

receive do
  {:hello, name} -> "Hello #{name}"
  {:bye, name} -> "Bye #{name}"
end

What you’re seeing here usage of BEAM’s actor model for concurrency in Elixir.If you’re not familiar with the model worry not - you can read an interestingELI5 about ithere on Dev.to, or you can just keep on reading and by the end of this article you should have a good idea about it.

The receive function takes the received message and tries to pattern-match it to one of the statements in the block. Obviously, it accepts not just a value to return but also a function call. As you can imagine, receive/1 is what is called when the mailbox of the process gets a new message in.

Now that we fixed up the actions on the mailbox of our process, how can we send a message to our IEx process?

This is done via the send/2 function. The function takes the PID and the message itself as arguments. For our example, let’s send the following message to our IEx process:

iex(1)> send self(), {:hello, "John"}
{:hello, "John"}

What you see here is the message being dropped in IEx’s mailbox. This means that we need to invoke the receive function in our IEx session so we can process the messages in the mailbox:

iex(2)> receive do
...(2)> {:hello, name} -> "Hello #{name}"
...(2)> {:bye, name} -> "Bye #{name}"
...(2)> end
"Hello John"

Right after executing the receive block the message will be immediately processed and we will see "Hello John" returned.

What if we never receive a message?

One thing to notice here is that if we would just write the receive block in our IEx session it would block until the process receives a message.That’s expected - if there is no message in the mailbox matching any of the patterns, the current process will wait until a matching message arrives. This is the default behaviour of receive/1.

Obviously, if we get ourselves stuck in such way we can always stop the IEx session by using Ctrl+C. But, what if a process in our application gets stuck?How can we tell it that it should stop waiting after a certain amount of time if it does not receive a message?

One nicety that Elixir provides us with is setting a timeout using after:

iex(6)> receive do
...(6)> {:hello, name} -> "Hello #{name}"
...(6)> {:bye, name} -> "Bye #{name}"
...(6)> after
...(6)> 1_000 -> "Nothing after 1s"
...(6)> end
"Nothing after 1s"

What happens here is that the timeout function is executed after 1000milliseconds pass and the receive/1 function exits (hence stops blocking).This prevents processes hanging around waiting forever for a matching message toarrive in their mailboxes.

Long-running processes

So far we were looking at sending and receiving messages to the same process -the IEx process itself. This is quite a simple example and won’t help when we would like to put processes into action in a production application.

At this point, you might be wondering how to actually spawn a process that would react to multiple incoming messages, instead of just (ab)using the IEx process with the receive/1 function.

Well, to do this we have to make a process run infinitely (or until we ask it to die). How? By creating an infinite loop in the process itself.

WTF you mean by “an infinite loop”?!

Yeah, I know, it feels weird, doesn’t it? Here’s the thing - the BEAM has an optimisation in it, so-called a last-call optimisation (read Joe Armstrong’s explanation of this optimisationhere),where if the last call in a function is a call to the same function, Elixir will not allocate a new stack frame on the call stack. In fact, it will just jump to the beginning of the same function (instead of running another instance of it), which will prevent a stack overflow happening to our program.

This means that it’s virtually impossible to smash the stack in the BEAM languages (Erlang & Elixir), if we are just a bit careful when composing these self-invoking functions.

Long-running processes, continued

Let’s look at a small module with a single function:

defmodule MyProcess do
  def start do
    receive do
      {:hello, name} ->
        IO.puts "Hello #{name}!"
        start()
      {:bye, name} ->
        IO.puts "Bye #{name}. Shutdown in 3, 2, 1..."
    end
  end
end

The MyProcess.start function when run will wait for a message to arrive in the process’ mailbox. Then, it will try to pattern match on the arrived message and execute the associated code. One trick is that at the end of the first case we execute the start function again, which will create an infinite loop in the process, therefore having the process waiting for messages forever.

Let’s look how this will work in IEx:

iex(1)> pid = spawn(MyProcess, :start, [])
#PID<0.120.0>

iex(2)> send pid, {:hello, "Ilija"}
Hello Ilija!
{:hello, "Ilija"}

iex(3)> send pid, {:hello, "Jane"}
Hello Jane!
{:hello, "Jane"}

iex(4)> send pid, {:bye, "Jane"}
Bye Jane. Shutdown in 3, 2, 1...
{:bye, "Jane"}

First, we use spawn/3 to create a process that will run the MyProcess.run function. spawn/3 is a flavour of spawn/1 - the only difference is that spawn/3 knows how to run a named function in a process, while spawn/1 takes only anonymous functions as arguments.

Then, you can see that every time we send a {:hello, "Name"} message to the process (using send/2), we see the process printing back the greeting. Once we send the {:bye, "Jane"} message the process prints that it’s shutting down.

How so? Well, if you look at the MyProcess.start function you will notice that it does not invoke itself after it prints out the shutdown message. This means that once it handles that message the MyProcess.start function will finish and the process will die.

Let’s test that in IEx, using the Process.alive?/1 function:

iex(1)> pid = spawn MyProcess, :start, []
#PID<0.109.0>

iex(2)> Process.alive? pid
true

iex(3)> send pid, {:hello, "Ilija"}
Hello Ilija!
{:hello, "Ilija"}

iex(4)> Process.alive? pid
true

iex(5)> send pid, {:bye, "Ilija"}
Bye Ilija. Shutdown in 3, 2, 1...
{:bye, "Ilija"}

iex(6)> Process.alive? pid
false

Now that we know how to send and receive multiple messages to a process inElixir, you might be wondering what is a good use-case to use processes for?

The answer is: keeping state.

Keeping state using processes

You might find this weird, but this is a classic example to keeping state in Elixir. While our example here will not take care of storing state on disk, cover all edge-cases that might occur or be a bullet-proof solution, I hope it will get your brain juices flowing on processes and how to use them.

Let’s write a Store module that will have a start function. It should spawn a process which will invoke a function of the same module, for now, called foo:

# store.exs
defmodule Store do
  def start do
    spawn( __MODULE__ , :foo, [%{}])
  end

  def foo(map) do
    IO.puts "Nothing so far"
  end
end

Let’s briefly dissect the Store.start function: what it does is it spawns anew process calling the foo/1 function and it passes an empty map (%{}) as an argument to the function. If the special keyword __MODULE__ throws you off, it’s just an alias to the module name:

iex(1)> defmodule Test do
...(1)> def eql?, do: __MODULE__ == Test
...(1)> end

iex(2)> Test.eql?
true

This means that when we call Store.start we will immediately see the output of the function in the process. Right after, the process will die:

iex(4)> pid = Store.start
Nothing so far...
#PID<0.117.0>

iex(5)> Process.alive? pid
false

This means that we need to make foo/1 a function that will loop forever. Or at least until we tell it to stop.

Let’s rename foo/1 to loop/1 and make it loop forever:

defmodule Store do
  def start do
    spawn( __MODULE__ , :loop, [%{}])
  end

  def loop(state) do
    receive do
      loop(state)
    end
  end
end

If we run this module now in an IEx session the process will work forever. Instead of doing that, let’s add a special “system” message that we can send to the process so we can force it to shut down:

defmodule Store do
  def start do
    spawn( __MODULE__ , :loop, [%{}])
  end

  def loop(state) do
    receive do
      {:stop, caller_pid} ->
        send caller_pid, "Shutting down"
      _ ->
        loop(state)
    end
  end
end

So now, you see that when there’s no match, the Store.loop/1 function will just recurse, but when the :stop message is received it will just send a"Shutting down" message to the calling PID.

iex(1)> pid = Store.start
#PID<0.125.0>

iex(2)> send pid, {:stop, self()}
{:stop, #PID<0.102.0>}

iex(3)> flush()
"Shutting down."
:ok

What you’re seeing here is a very simple example of sending messages between two processes - the Store process and our IEx session process. When we send the :stop message we also send the PID of the IEx session (self), which is then used by Store.loop/1 to send the reply back. At the end, instead of writing a whole receive block for the IEx session we just invoke flush,which flushes the IEx process mailbox and returns all of the messages in the mailbox at that time.

If you’re feeling deep in the rabbit hole now worry not - we are going to address keeping state in a process right away!

Let’s say that our Store will accept four commands:

stop - the one we already implemented, which stops the Store process
put - adds a new key-value pair to the state map
get - fetches a value for a given key from the state map
get_all - fetches all of the key-value pairs that are stored in the state map

Putting a value

Let’s implement the put command:

defmodule Store do
  def start do
    spawn( __MODULE__ , :loop, [%{}])
  end

  def loop(state) do
    receive do
      {:put, key, value} ->
        new_state = Map.put(state, key, value)
        loop(new_state)
      {:stop, caller} ->
        send caller, "Shutting down."
      _ ->
        loop(state)
    end
  end
end

When we send a tuple containing {:put, :foo, :bar} to the process, it will add the :foo => :bar pair to the state map. The key here is that it will invoke State.loop/1 again with the updated state (new_state). This will make sure that the key-value pair we added will be included in the new state on the next recursion of the function.

Getting values

Let’s implement get so we can test get and put together via an IEx session:

defmodule Store do
  def start do
    spawn( __MODULE__ , :loop, [%{}])
  end

  def loop(state) do
    receive do
      {:stop, caller} ->
        send caller, "Shutting down."
      {:put, key, value} ->
        new_state = Map.put(state, key, value)
        loop(new_state)
      {:get, key, caller} ->
        send caller, Map.fetch(state, key)
        loop(state)
      _ ->
        loop(state)
    end
  end
end

Just like with the other commands, there’s no magic around get. We use Map.fetch/2 to get the value for the key passed. Also, we take the PID of the caller so we can send back to the caller the value found in the map:

iex(1)> pid = Store.start
#PID<0.119.0>

iex(2)> send pid, {:put, :name, "Ilija"}
{:put, :name, "Ilija"}

iex(3)> send pid, {:get, :name, self()}
{:get, :name, #PID<0.102.0>}

iex(4)> flush
{:ok, "Ilija"}
:ok

iex(5)> send pid, {:put, :surname, "Eftimov"}
{:put, :surname, "Eftimov"}

iex(6)> send pid, {:get, :surname, self()}
{:get, :surname, #PID<0.102.0>}

iex(7)> flush
{:ok, "Eftimov"}
:ok

If we look at the “conversation” we have with the Store process, at the beginning, we set a :name key with the value "Ilija" and we retrieve it after (and we see the reply using flush). Then we do the same exercise by adding a new key to the map in the Store, this time :surname with the value"Eftimov".

From the “conversation” perspective, the key piece here is us sending self() -the PID of our current process (the IEx session) - so the Store process knows where to send the reply to.

Getting a dump of the store

Right before we started writing the Store module we mentioned that we will also implement a get_all command, which will return all of the contents of the Store. Let’s do that:

defmodule Store do
  def start do
    spawn( __MODULE__ , :loop, [%{}])
  end

  def loop(state) do
    receive do
      {:stop, caller} ->
        send caller, "Shutting down."
      {:put, key, value} ->
        new_state = Map.put(state, key, value)
        loop(new_state)
      {:get, key, caller} ->
        send caller, Map.fetch(state, key)
        loop(state)
      {:get_all, caller } ->
        send caller, state
        loop(state)
      _ ->
        loop(state)
    end
  end
end

If you expected something special here, I am very sorry to disappoint you. The implementation of the get_all command is to return the whole state map of the process to the sender.

Let’s test it out:

iex(1)> pid = Store.start
#PID<0.136.0>

iex(2)> send pid, {:put, :name, "Jane"}
{:put, :name, "Jane"}

iex(3)> send pid, {:put, :surname, "Doe"}
{:put, :surname, "Doe"}

iex(4)> send pid, {:get_all, self()}
{:get_all, #PID<0.102.0>}

iex(5)> flush
%{name: "Jane", surname: "Doe"}
:ok

As expected, once we add two key-value pairs to the Store, when we invoke the get_all command the Store process sends back the whole state map.

While this is a very small and contrived example, the skeleton we followed hereby keeping state by using recursion, sending commands and replies back and forth to the calling process is actually used quite a bit in Erlang and Elixir.

A small disappointment

First, I am quite happy you managed to get to the end of this article.I believe that once I understood the basics of the concurrency model of Elixir,by going through these exercises were eye-opening for me, and I hope they werefor you as well.

Understanding how to use processes by sending and receiving messages is paramount knowledge that you can use going forward on your Elixir journey.

Now, as promised - a small disappointment.

For more than 90% of the cases when you want to write concurrent code in Elixir, you will not use processes like here. In fact, you won’t (almost) ever use the send/2 and receive/1 functions. Seriously.

Why? Well, that’s because Elixir comes with this thingy called OTP, that will help you do much cooler things with concurrency, without writing any of this low-level process management code. Of course, this should not stop you from employing processes when you feel that you strongly need them, or when you want to experiment and learn.

But, that’s a topic for the next blog post, where we’ll dive more in OTP and some of its behaviours.

Until then, where do you see processes as a potential use case in your projects/work?

Some more reading

If you would like to read a bit more, here are a couple of links that are worth checking out:

Processes on Elixir’s“Getting started” guide
Long-lived processes in Elixir by German Velasco on Thoughtbot’s blog
Process on Hexdocs.pm
Endless recursion on theElixir Forum

A deeper dive in Elixir's Plug

Ilija Eftimov — Sun, 06 Jan 2019 14:36:08 +0000

This post was originally published on my blog on December 31st, 2018.

Being new to Elixir and Phoenix, I spend quite some time in the projects' documentation. One thing that stood out for me recently is the first sentence of Phoenix's Plug documentation:

Plug lives at the heart of Phoenix’s HTTP layer and Phoenix puts Plug front and center.

So naturally, I felt compelled to take a deeper dive into Plug and understand it better. I hope the following article will help you out in understanding Plug.

What's Plug?

As the readme puts it, Plug is:

A specification for composable modules between web applications
Connection adapters for different web servers in the Erlang VM

But, what does this mean? Well, it basically states that Plug 1) defines the way you build web apps in Elixir and 2) it provides you with tools to write apps that are understood by web servers.

Let's take a dive and see what that means.

Web servers, yeehaw!

One of the most popular HTTP servers for Erlang is Cowboy. It is a small, fast and modern HTTP server for Erlang/OTP. If you were to write any web application in Elixir it will run on Cowboy, because the Elixir core team has built a Plug adapter for Cowboy, conveniently named plug_cowboy.

This means that if you include this package in your package, you will get the Elixir interface to talk to the Cowboy web server (and vice-versa). It means that you can send and receive requests and other stuff that web servers can do.

So why is this important?

Well, to understand Plug we need to understand how it works. Basically, using the adapter (plug_cowboy), Plug can accept the connection request that comes in Cowboy and turn it into a meaningful struct, also known as Plug.Conn.

This means that Plug uses plug_cowboy to understand Cowboy's nitty-gritty details. By doing this Plug allows us to easily build handler functions and modules that can receive, handle and respond to requests.

Of course, the idea behind Plug is not to work only with Cowboy. If you look at this SO answer from José Valim (Elixir's BDFL) he clearly states "Plug is meant to be a generic adapter for different web servers. Currently we support just Cowboy but there is work to support others."

Enter Plug

Okay, now that we've scratched the surface of Cowboy and it's Plug adapter, let's look at Plug itself.

If you look at Plug's README, you will notice that there are two flavours of plugs, a function or a module.

The most minimal plug can be a function, it just takes a Plug.Conn struct (that we will explore more later) and some options. The function will manipulate the struct and return it at the end. Here's the example from the README:

def hello_world_plug(conn, _opts) do
  conn
  |> put_resp_content_type("text/plain")
  |> send_resp(200, "Hello world")
end

Code blatantly copied from Plug's docs.

If you look at the function, it's quite simple. It receives the connection struct, puts its content type to text/plain and returns a response with an HTTP 200 status and "Hello world" as the body.

The second flavour is the module Plug. This means that instead of just having a function that will be invoked as part of the request lifecycle, you can define a module that takes a connection and initialized options and returns the connection:

defmodule MyPlug do
  def init([]), do: false
  def call(conn, _opts), do: conn
end

Code blatantly copied from Plug's docs.

Having this in mind, let's take a step further and see how we can use Plug in a tiny application.

Plugging a plug as an endpoint

So far, the most important things we covered was what's Plug and what is it used for on a high level. We also took a look at two different types of plugs.

Now, let's see how we can mount a Plug on a Cowboy server and essentially use it as an endpoint:

defmodule PlugTest do
  import Plug.Conn

  def init(options) do
    # initialize options

    options
  end

  def call(conn, _opts) do
    conn
    |> put_resp_content_type("text/plain")
    |> send_resp(200, "Hello world")
  end
end

What this module will do is, when mounted on a Cowboy server, will set the Content-Type header to text/plain and will return an HTTP 200 with a body of Hello world.

Let's fire up IEx and test this ourselves:

› iex -S mix
Erlang/OTP 21 [erts-10.2] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [hipe] [dtrace]

Interactive Elixir (1.7.4) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> {:ok, _ } = Plug.Cowboy.http PlugTest, [], port: 3000
{:ok, #PID<0.202.0>}

This starts the Cowboy server as a BEAM process, listening on port 3000. If we cURL it we'll see the response body and it's headers:

› curl -v 127.0.0.1:3000
> GET / HTTP/1.1
> Host: 127.0.0.1:3000
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 200 OK
< cache-control: max-age=0, private, must-revalidate
< content-length: 11
< content-type: text/plain; charset=utf-8
< date: Tue, 25 Dec 2018 22:54:54 GMT
< server: Cowboy
<
* Connection #0 to host 127.0.0.1 left intact
Hello world

You see, the Content-Type of the response is set to text/plain and the body is Hello world. In this example, the plug is essentially an endpoint by itself, serving plain text to our cURL command (or to a browser). As you might be able to imagine at this point, you can plug in much more elaborate Plugs to a Cowboy server and it will serve them just fine.

To shut down the endpoint all you need to do is:

iex(2)> Plug.Cowboy.shutdown PlugTest.HTTP
:ok

What we are witnessing here is probably the tiniest web application one can write in Elixir. It's an app that takes a request and returns a valid response over HTTP with a status and a body.

So, how does this actually work? How do we accept the request and build a response here?

Diving into the `Plug.Conn`

To understand this, we need to zoom in the call/2 function of our module PlugTest. I will also throw in an IO.inspect right at the end of the function so we can inspect what this struct is:

def call(conn, _opts) do
  conn
  |> put_resp_content_type("text/plain")
  |> send_resp(200, "Hello world")
  |> IO.inspect
end

If you start the Cowboy instance again via your IEx session and you hit 127.0.0.1:3000 via cURL (or a browser), you should see something like this in your IEx session:

%Plug.Conn{
  adapter: {Plug.Cowboy.Conn, :...},
  assigns: %{},
  before_send: [],
  body_params: %Plug.Conn.Unfetched{aspect: :body_params},
  cookies: %Plug.Conn.Unfetched{aspect: :cookies},
  halted: false,
  host: "127.0.0.1",
  method: "GET",
  owner: #PID<0.316.0>,
  params: %Plug.Conn.Unfetched{aspect: :params},
  path_info: [],
  path_params: %{},
  port: 3000,
  private: %{},
  query_params: %Plug.Conn.Unfetched{aspect: :query_params},
  query_string: "",
  remote_ip: {127, 0, 0, 1},
  req_cookies: %Plug.Conn.Unfetched{aspect: :cookies},
  req_headers: [
    {"accept",
     "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8"},
    {"accept-encoding", "gzip, deflate, br"},
    {"accept-language", "en-US,en;q=0.9"},
    {"connection", "keep-alive"},
    {"host", "127.0.0.1:3000"},
    {"upgrade-insecure-requests", "1"},
    {"user-agent",
     "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36"}
  ],
  request_path: "/",
  resp_body: nil,
  resp_cookies: %{},
  resp_headers: [
    {"cache-control", "max-age=0, private, must-revalidate"},
    {"content-type", "text/plain; charset=utf-8"}
  ],
  scheme: :http,
  script_name: [],
  secret_key_base: nil,
  state: :sent,
  status: 200
}

What are we actually looking at? Well, it's actually the Plug representation of a connection. This is a direct interface to the underlying web server and the request that the Cowboy server has received.

Some of the attributes of the struct are pretty self-explanatory, like scheme, method, host, request_path, etc. If you would like to go into detail what each of these fields is, I suggest taking a look at Plug.Conn's documentation.

But, to understand better the Plug.Conn struct, we need to understand the connection lifecycle of each connection struct.

Connection lifecycle

Just like any map in Elixir Plug.Conn allows us to pattern match on it. Let's modify the little endpoint we created before and try to add some extra IO.inspect function calls:

defmodule PlugTest do
  import Plug.Conn

  def init(options) do
    # initialize options

    options
  end

  def call(conn, _opts) do
    conn
    |> inspect_state
    |> put_resp_content_type("text/plain")
    |> inspect_state
    |> put_private(:foo, :bar)
    |> inspect_state
    |> resp(200, "Hello world")
    |> inspect_state
    |> send_resp()
    |> inspect_state
  end

  defp inspect_state(conn = %{state: state}) do
    IO.inspect state
    conn
  end
end

Because Plug.Conn allows pattern matching, we can get the state of the connection, print it out and return the connection itself so the pipeline in the call/2 function would continue working as expected.

Let's mount this plug on a Cowboy instance and hit it with a simple cURL request:

iex(6)> Plug.Cowboy.http PlugTest, [], port: 3000
{:ok, #PID<0.453.0>}

# curl 127.0.0.1:3000

iex(21)> :unset
:unset
:unset
:set
:sent

You see, when the connection enters the plug it's state changes from :unset to :set to finally :sent. This means that once the plug is invoked the state of the connection is :unset. Then we do multiple actions, or in other words, we invoke multiple functions on the Plug.Conn which add more information to the connection. Obviously, since all variables in Elixir are immutable, each of these function returns a new Plug.Conn instance, instead of mutating the existing one.

Once the body and the status of the connection are set, then the state changes to :set. Up until that moment, the state is fixed as :unset. Once we send the response back to the client the state is changed to :sent.

What we need to understand here is that whether we have one or more plugs in a pipeline, they will all receive a Plug.Conn, call functions on it, whether to extract or add data to it and then the connection will be passed on to the next plug. Eventually, in the pipeline, there will be a plug (in the form of an endpoint or a Phoenix controller) that will set the body and the response status and send the response back to the client.

There are a bit more details to this, but this is just enough to wrap our minds around Plug and Plug.Conn in general.

Next-level `Plug`ging using `Plug.Router`

Now that we understand how Plug.Conn works and how plugs can change the connection by invoking functions defined in the Plug.Conn module, let's look at a more advanced feature of plugs - turning a plug into a router.

In our first example, we saw the simplest of the Elixir web apps - a simple plug that takes the request and returns a simple response with a text body and an HTTP 200. But, what if we want to handle different routes or HTTP methods? What if we want to gracefully handle any request to an unknown route with an HTTP 404?

One nicety that Plug comes with is a module called Plug.Router, you can see its documentation here. The router module contains a DSL that allows us to define a routing algorithm for incoming requests and writing handlers (powered by Plug) for the routes. If you are coming from Ruby land, while Plug is basically Rack, this DSL is Sinatra.rb.

Let's create a tiny router using Plug.Router, add some plugs to its pipeline and some endpoints.

Quick aside: What is a pipeline?

Although it has the same name as the pipeline operator (|>), a pipeline in Plug's context is a list of plugs executed one after another. That's really it. The last plug in that pipeline is usually an endpoint that will set the body and the status of the response and return the response to the client.

Now, back to our router:

defmodule MyRouter do
  use Plug.Router

  plug :match
  plug :dispatch

  get "/hello" do
    send_resp(conn, 200, "world")
  end

  match _ do
    send_resp(conn, 404, "oops")
  end
end

Code blatantly copied from Plug.Router's docs.

The first thing that you will notice here is that all routers are modules as well. By useing the Plug.Router module, we include some functions that make our lives easier, like get or match.

If you notice at the top of the module we have two lines:

plug :match
plug :dispatch

This is the router's pipeline. All of the requests coming to the router will pass through these two plugs: match and dispatch. The first one does the matching of the route that we define (e.g. /hello), while the other one will invoke the function defined for a particular route. This means that if we would like to add other plugs, most of the time they will be invoked between the two mandatory ones (match and dispatch).

Let's mount our router on a Cowboy server and see it's behaviour:

iex(29)> Plug.Cowboy.http MyRouter, [], port: 3000
{:ok, #PID<0.1500.0>}

When we hit 127.0.0.1:3000/hello, we will get the following:

› curl -v 127.0.0.1:3000/hello
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 3000 (#0)
> GET /hello HTTP/1.1
> Host: 127.0.0.1:3000
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 200 OK
< cache-control: max-age=0, private, must-revalidate
< content-length: 5
< date: Thu, 27 Dec 2018 22:50:47 GMT
< server: Cowboy
<
* Connection #0 to host 127.0.0.1 left intact
world

As you can see, we received world as the response body and an HTTP 200. But if we hit any other URL, the router will match the other route:

› curl -v 127.0.0.1:3000/foo
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 3000 (#0)
> GET /foo HTTP/1.1
> Host: 127.0.0.1:3000
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 404 Not Found
< cache-control: max-age=0, private, must-revalidate
< content-length: 4
< date: Thu, 27 Dec 2018 22:51:56 GMT
< server: Cowboy
<
* Connection #0 to host 127.0.0.1 left intact
oops

As you can see, because the /hello route didn't match we defaulted to the other route, also known as "catch all" route, which returned oops as the response body and an HTTP 404 status.

If you would like to learn more about Plug.Router and its route matching macros you can read more in its documentation. We still need to cover some more distance with Plug.

Built-in Plugs

In the previous section, we mentioned the plugs match and dispatch, and plug pipelines. We also mentioned that we can plug in other plugs in the pipeline so we can inspect or change the Plug.Conn of each request.

What is very exciting here is that Plug also comes with already built-in plugs. That means that there's a list of plugs that you can plug-in in any Plug-based application:

Plug.CSRFProtection
Plug.Head
Plug.Logger
Plug.MethodOverride
Plug.Parsers
Plug.RequestId
Plug.SSL
Plug.Session
Plug.Static

Let's try to understand how a couple of them work and how we can plug them in our MyRouter router module.

`Plug.Head`

This is a rather simple plug. It's so simple, I will add all of its code here:

defmodule Plug.Head do
  @behaviour Plug

  alias Plug.Conn

  def init([]), do: []

  def call(%Conn{method: "HEAD"} = conn, []), do: %{conn | method: "GET"}
  def call(conn, []), do: conn
end

What this plug does is it turns any HTTP HEAD request into a GET request. That's all. Its call function receives a Plug.Conn, matches only the ones that have a method: "HEAD" and returns a new Plug.Conn with the method changed to "GET".

If you've been wondering what the HEAD method is for, this is from RFC 2616:

The HEAD method is identical to GET except that the server MUST NOT return a
message-body in the response. The metainformation contained in the HTTP headers
in response to a HEAD request SHOULD be identical to the information sent in
response to a GET request. This method can be used for obtaining
metainformation about the entity implied by the request without transferring
the entity-body itself. This method is often used for testing hypertext links
for validity, accessibility, and recent modification.

Let's plug this plug in our Plug.Router (pun totally intended):

defmodule MyRouter do
  use Plug.Router

  plug Plug.Head
  plug :match
  plug :dispatch

  get "/hello" do
    send_resp(conn, 200, "world")
  end

  match _ do
    send_resp(conn, 404, "oops")
  end
end

Once we cURL the routes we would get the following behaviour:

› curl -I 127.0.0.1:3000/hello
HTTP/1.1 200 OK
cache-control: max-age=0, private, must-revalidate
content-length: 5
date: Thu, 27 Dec 2018 23:25:13 GMT
server: Cowboy

› curl -I 127.0.0.1:3000/foo
HTTP/1.1 404 Not Found
cache-control: max-age=0, private, must-revalidate
content-length: 4
date: Thu, 27 Dec 2018 23:25:17 GMT
server: Cowboy

As you can see, although we didn't explicitly match the HEAD routes using the head macro, the Plug.Head plug remapped the HEAD requests to GET and our handlers still kept on working as expected (the first one returned an HTTP 200, and the second one an HTTP 404).

`Plug.Logger`

This one is a bit more complicated so we cannot inline all of its code in this article. Basically, if we would plug this plug in our router, it will log all of the incoming requests and response statuses, like so:

  GET /index.html
  Sent 200 in 572ms

This plug uses Elixir's Logger (docs under the hood, which supports four different logging levels:

:debug - for debug-related messages
:info - for information of any kind (default level)
:warn - for warnings
:error - for errors

If we would look at the source of its call/2 function, we would notice two logical units. The first one is:

def call(conn, level) do
  Logger.log(level, fn ->
    [conn.method, ?\s, conn.request_path]
  end)

  # Snipped...
end

This one will take Elixir's Logger and using the logging level will log the information to the backend (by default it's console). The information that is logged is the method of the request (e.g. GET, POST, etc) and the request path (e.g. /foo/bar). This results in the first line of the log:

GET /index.html

The second logical unit is a bit more elaborate:

def call(conn, level) do
  # Snipped...

  start = System.monotonic_time()

  Conn.register_before_send(conn, fn conn ->
    Logger.log(level, fn ->
      stop = System.monotonic_time()
      diff = System.convert_time_unit(stop - start, :native, :microsecond)
      status = Integer.to_string(conn.status)

      [connection_type(conn), ?\s, status, " in ", formatted_diff(diff)]
    end)

    conn
  end)
end

In short: this section records the time between the start and the stop (end) of the request and prints out the difference between the two (or in other words - the amount of time the response took). Also, it prints out the HTTP status of the response.

To do this it uses Plug.Conn.register_before_send/2 (docs) which is a utility function that registers callbacks to be invoked before the response is sent. This means that the function which will calculate the diff and log it to the Logger with the response status will be invoked by Plug.Conn right before the response is sent to the client.

Wrapping up with Plug

You actually made it this far - I applaud you. I hope that this was a nice journey for you in Plug and it's related modules/functions and that you learned something new.

We looked at quite a bit of details in and around Plug. For some of the modules that we spoke about we barely scratched the surface. For example, Plug.Conn has quite a bit of more useful functions. Or Plug.Router has more functions in its DSL where you can write more elaborate and thoughtful APIs or web apps. In line with this, Plug also offers more built-in plugs. It even has a plug which can serve static files with ease, and plugging it in your Plug-based apps is a breeze.

But, aside from all the things that we skipped in this article, I hope that you understood how powerful the Plug model is and how much power it provides us with such simplicity and unobtrusiveness.

In future posts, we will look at even more details about other plugs in Plug, but until then please shoot me a comment or a message if you've found this article helpful (or not).

Validate your passwords using Elixir and haveibeenpwned.com's API

Ilija Eftimov — Sun, 23 Dec 2018 19:20:25 +0000

This post was originally published on my blog, on December 23, 2018. You can see it here.

Unless you've been living under a rock for the last couple of years, you probably know what two-factor authentication (2FA) is. It's quite a neat trick actually - you have a password that you have to (obviously) enter correctly (first factor), but you also have to receive a second (random) code through a different medium, sometimes on a different device, that you have to enter to log in (second factor).

Now, obviously this adds quite a bit of overhead to logging in, but it adds a disproportionate value when it comes to security. If you work in any sort of organisation it was probably no surprise when you were asked to turn on 2FA for all your accounts. If you haven't been asked to (or haven't done it), it's time to act ;)

But, what about factor No. 1? The password. Did we give up on them?

Not really. But, for sure we had to become more vigilant and smarter when setting our passwords. Why? Allow me to explain.

Reader, meet haveibeenpwned.com

Let me introduce you to haveibeenpwned.com. It's a free resource for anyone to quickly assess if they may have been put at risk due to an online account of their's having been compromised or "pwned" in a data breach. As you can imagine, to fulfil its purpose, this service also contains quite a long list of pwned passwords (about 500 million of them to be more precise), which are open for querying through a REST API.

If you want to learn more about the project, or it's author, I suggest checking out the About page of the project.

Using the pwned passwords API

This API allows us to check if any password is present in haveibeenpwned database. This means that if you send an already pwned password it will tell you that this password has been pwned and that it's suggested to choose another one.

Imagine you have a website where people can set their passwords, and once the user finished typing their new password you can ping this service and check if the password they chose has been pwned before.

Now, if you are thinking along the lines of "are you telling me to send a plain-text password across the wire to some random API?" then you're a step ahead, well done!

Sorry to disappoint, but no, actually I am not saying that. Instead of sending the whole password in plain-text, this API only requires the 5 characters of the SHA-1 hash of the actual password.

In Elixir terms, that would look like:

:sha
|> :crypto.hash("password")
|> Base.encode16
|> String.slice(0..4)

Interestingly what it sends back is the remainder of the hashed passwords that match the 5 characters that you sent. Basically, this means if we take a SHA-1 of "password":

iex(1)> :crypto.hash(:sha, "password") |> Base.encode16
"5BAA61E4C9B93F3F0682250B6CF8331B7EE68FD8"

We will send only 5BAA6 to the API, while in the response body we will receive a big list of strings that will represent the rest of the SHA-1, or in our example that would be 1E4C9B93F3F0682250B6CF8331B7EE68FD8.

Troy Hunt, who's the author of haveibeenpwned has written quite an extensive explanation on how this works - you can read it here.

Pinging the API

For the purpose of this exercise, we will create a small Mix package that will encapsulate all of the behaviours. If you're not familiar with how to create new packages using Mix, I suggest reading my article Write and publish your first Elixir library.

We will call our package Pwnex because somehow my brain always thinks that I have to mix up the main word (pwned) with Elixir to come up with a name. Anyway, let's create it:

› mix new pwnex
* creating README.md
* creating .formatter.exs
* creating .gitignore
* creating mix.exs
* creating config
* creating config/config.exs
* creating lib
* creating lib/pwnex.ex
* creating test
* creating test/test_helper.exs
* creating test/pwnex_test.exs

Your Mix project was created successfully.
You can use "mix" to compile it, test it, and more:

    cd pwnex
    mix test

Run "mix help" for more commands.

Now that we have the package bootstrapped locally, let's open lib/pwnex.ex and
add some documentation:

defmodule Pwnex do
  @moduledoc """
  Consults haveibeenpwned.com's API for pwned passwords.
  """

  @doc """
  Checks if a given password is already pwned.

  ## Examples

      iex> Pwnex.pwned?("password")
      {:pwned, 3_000_000}

      iex> Pwnex.pwned?("m4Z2fJJ]r3fxQ*o27")
      {:ok, 0}

  """
  def pwned?(password) do
  end
end

The moduledoc briefly explains the purpose of the package, while the doc explains the purpose of the pwned?/1 function and has two examples that we
could use in the doctests.

Our little algorithm

Let's see what would be the steps to implement the Pwnex.pwned?/1 function:

def pwned?(password) do
  {hash_head, hash_tail} =
    password
    |> sanitize
    |> hash
    |> split_password

  hash_head
    |> fetch_pwns
    |> handle_response
    |> find_pwns(hash_tail)
    |> return_result
end

Once more - the pipeline operator in Elixir makes this function so clear and procedure-like that explaining feels a tad redundant. Still, here it is:

sanitize - we want to remove all leading and trailing whitespaces from the password
hash - we want to convert the password to a SHA1 hash and return it's first 5 characters
split_password - we want to split the head - first 5 characters that we will send to the API and the tail - the rest of the SHA-1 hash
fetch_pwns - we will send an API request to haveibeenpwned to get all (if any) pwns of the password
handle_response - depending on the response we will either get the body, or the reason for failure returned
find_pwns - we will take the response body, and because haveibeenpwned uses a k-Anonymity model we will need to find the actual match ourselves (if present)
return_result - will return the tuple which will contain a result atom and a pwns count

Let's take a step by step approach and implement these functions.

Manipulating the password

Let's start easy. In sanitize, we want to trim leading and trailing whitespaces, while in hash we want to turn the password to SHA1 and return it's first five characters.

def sanitize(password), do: String.trim(password)

There isn't much to explain here really. Instead of using sanitize we can use String.trim/1, but I prefer to have a separate function that we could extend and test for any edge cases.

defp hash(password) do
  :crypto.hash(:sha, password)
  |> Base.encode16
end

:crypto is an Erlang module that provides a set of cryptographic functions. Interestingly, it's not part of the standard library, but it comes included in the distribution. One of the functions, as you can see in the code above, is hash/2, which takes the hashing algorithm as the first argument and the actual string to be hashed as the second argument. It returns the binary hash, that we can convert to hex by using Base.encode16.

Sending request to an API

I bet you're thinking HTTPoison. Aren't you? While I was writing this article I was also wondering do we have to include to a whole package just to do a simple GET request. You guessed it - we do not.

Although Elixir does not ship an HTTP client, Erlang does. And just like with :crypto, you can use Erlang's HTTP client from Elixir using :httpc. This module provides the API to an HTTP/1.1 compatible client. I suggest giving it's documentation a quick scan before we move on.

Let's let's open up IEx and give :httpc a spin:

iex(1)> :httpc.request('https://api.pwnedpasswords.com/range/21FCB')
{:ok,
 \{\{'HTTP/1.1', 200, 'OK'},
  [
    {'cache-control', 'public, max-age=2678400'},
    {'connection', 'keep-alive'},
    {'date', 'Sat, 22 Dec 2018 11:09:46 GMT'},
    {'server', 'cloudflare'},
    {'vary', 'Accept-Encoding'},
    {'content-length', '19951'},
    {'content-type', 'text/plain'},
    {'expires', 'Tue, 22 Jan 2019 11:09:46 GMT'},
    {'last-modified', 'Thu, 12 Jul 2018 01:32:06 GMT'},
    {'set-cookie',
     '__cfduid=d51115381191fd7bd0a003d466916efc41545476986; expires=Sun, 22-Dec-19 11:09:46 GMT; path=/; domain=.pwnedpasswords.com; HttpOnly; Secure'},
    {'cf-cache-status', 'HIT'},
    {'access-control-allow-origin', '*'},
    {'arr-disable-session-affinity', 'True'},
    {'cf-ray', '48d2235cbe93bf5c-AMS'},
    {'expect-ct',
     'max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"'},
    {'strict-transport-security',
     'max-age=31536000; includeSubDomains; preload'},
    {'x-content-type-options', 'nosniff'},
    {'x-powered-by', 'ASP.NET'}
  ],
  'THEBODYISHERE\r\nOMGSOMUCHSTUFFHEREWHATISTHISEVEN' ++ ...}}

You see, although the output is pretty verbose if you have ever sent an HTTP request via cURL or you've opened your browser's debugging tools, there shouldn't be any surprises here. The request/1 function will send a request to the pwnedpasswords API and it will return a ton of nested tuples, most of them being the response headers and the raw body of the response.

For our purpose, we can keep it simple. We are only interested if the function will return :ok atom as the first item in the tuple, or :error. We can use pattern matching to do this:

iex(1)> {:ok, {_status, _headers, body }} =
  :httpc.request('https://api.pwnedpasswords.com/range/21FCB')

So, lets get back to our Pwnex.fetch_pwns/1 function. The function will receive the first 5 characters of the hashed password, it will send that to the API and will return the body of the response:

def fetch_pwns(head) do
  :httpc.request('https://api.pwnedpasswords.com/range/#{head}')
end

Handling the response

The handle_response will actually be one function with three bodies:

  def handle_response({:ok, {_status, _headers, body}}), do: body
  def handle_response({:error, {reason, _meta}}), do: reason
  def handle_response(_), do: nil

By using pattern matching we can have three types of function bodies. The first one will be invoked when the response status is HTTP 200 OK, the second one when there's an error and the third one for any other case.

As you can imagine, we could have used conditional logic here, but having the power of pattern matching allows us to have three tiny functions that are very easy to read, understand and test.

Parsing the response

Here's the body of the response:

003D68EB55068C33ACE09247EE4C639306B:3\r\n012C192B2F16F82EA0EB9EF18D9D539B0DD:1\r\n01330C689E5D64F660D6947A93AD634EF8F:1\r\n0198748F3315F40B1A102BF18EEA0194CD9:1\r\n01F9033B3C00C65DBFD6D1DC4D22918F5E9:2\r\n0424DB98C7A0846D2C6C75E697092A0CC3E:5\r\n047F229A81EE2747253F9897DA38946E241:1\r\n04A37A676E312CC7C4D236C93FBD992AA3C:5\r\n04AE045B134BDC43043B216AEF66100EE00:2\r\n0502EA98ED7A1000D932B10F7707D37FFB4:5\r\n0539F86F519AACC7030B728CD47803E5B22:5\r\n054A0BD53E2BC83A87EFDC236E2D0498C08:3\r\n05AA835DC9423327DAEC1CBD38FA99B8834:1\r\n05E0182DEAE22D02F6ED35280BCAC370179:4

If you look carefully you'll notice that it's actually a list of partial SHA-1 hashes separated by \r\n. With a closer inspection of the first one:

003D68EB55068C33ACE09247EE4C639306B:3

you notice that it's actually a part of the hash, a colon : and a number (3 in the example above). This is actually the hash without it's first 5 characters and the number of times that particular password has been pwned.

This means that the password who's SHA-1 hash is
5BAA6003D68EB55068C33ACE09247EE4C639306B has been pwned 3 times, according to haveibeenpwned.com.

What we need to with this response body is to split it, to convert it to a list
which we will iterate and find our matching hash in. Let's do that:

def find_pwns(response, hash_tail) do
  response
  |> to_string
  |> String.split
  |> String.split()
  |> Enum.find(&(String.starts_with?(&1, hash_tail)))
end

Although find_pwns/2 might look a bit loaded, let me assure you it's not. Let's see what each of the lines do here:

to_string will convert the character list we receive from fetch_pwns and will convert it to a string so we can parse it in the next steps
String.split will split the string on the \r\n characters and will create a list of strings, looking like: ["003D68EB55068C33ACE09247EE4C639306B:3", ...]
We will invoke Enum.find which takes the list and a function as arguments. The list is the parsed list of hash tails and their pwns count, while the function is String.starts_with?/2, which will return true when a line starts with the value of hash_tail.

That's all. At the end, the find_pwns/2 function will return either the line
that contains the matched hash tail or it will return nil.

Returning a meaningful result

Now that we have found the count of pwns for the hash (or just a nil), we need to handle that and return a meaningful tuple to the user of the module.

When the find_pwns function does find a count, we want to return a tuple
like {:pwned, count}. Otherwise, when find_pwns does not find a count it will return nil, which we handle in the second definition of the return_result function:

def return_result(line) when is_binary(line) do
  [_, count] = String.split(line, ":")
  {:pwned, count}
end
def return_result(_), do: {:ok, 0}

In the first function body we will take the line, which should be a binary
(string), split it at the : character and then return the tuple with the count.
In the second function body we take any argument (which in our case is nil) and
return a tuple with 0 as the count.

Using Pwnex

Now, let's load Pwnex in IEx and give it a spin. To load it in IEx, you need to open the root of the module and run iex -S mix. This will open a IEx session and execute mix in it, which will in fact, load and compile the module and make it available for invocation directly from IEx:

› iex -S mix
Erlang/OTP 21 [erts-10.2] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [hipe] [dtrace]

Interactive Elixir (1.7.4) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> h Pwnex

                          Pwnex

Consults haveibeenpwned.com's API for pwned passwords.
iex(2)> Pwnex.pwned?("password")
{:pwned, 3533661}
iex(3)> Pwnex.pwned?("123!@#asd*&(*123SAkjhda")
{:ok, 0}

As you can see, password has been pwned about 3.5 million times, while 123!@#asd*&(*123SAkjhda never.

That's basically it. We have Pwnex working - it takes our input as a function argument, talks to an API through an Erlang HTTP client, parses its response body, builds a map of hashes and finds any pwns for the given password.

As you can see, this whole package does quite a bit in 65 lines of code.

In this article, we saw how we can create a new package, use it to communicate with an API over HTTP, we learned why you don't always need HTTPoison, how you can parse a request body and how you can mingle with some data.

If you would like to see the actual code that we wrote in this article, head over to its repo on Github.

DEV Community: Ilija Eftimov

Deep dive in CORS: History, how it works, and best practices

The error in your browser's console

In the beginning was the first subresource

Origins & cross-origin

The many dangers of cross-origin requests

Same-origin policy

Enter CORS

Cross-origin writes

Preflight requests

Cross-origin reads

Fine-tuning CORS

Some best practices

Free for all

Keeping it in the family

NULL origins

Skip cookies, if you can

Additional reading

Testing in Go: Failing Tests

A short overview of the testing package

Logging with Log and Logf

Signaling test Failure

Delayed vs. immediate failure

Signal & report failures in one go

When to use Error(f) or Fatal(f)

In closing

Testing in Go: First Principles

What is testing?

What is a test?

Testing our Go code

Testing with Go’s testing package

What happens when we run go test

Dealing with test failures

In closing

Golang Datastructures: Trees

A touch of theory

Modelling an HTML document

Building MyDOM - a drop-in replacement for the DOM 😂

Implementing Node Lookup 🔎

Breadth-first search ⬅➡

Depth-first search ⬇

Finding by class name 🔎

Deleting nodes 🗑

Where to next?

OTP in Elixir: Learn GenServer by Building Your Own URL Shortener

Meet OTP 👋

Shortening a link ✂️

Enter GenServer 🚪

Server start & state

Sync & async messages 📨

Reimplementing URLShortener, using GenServer ♻️

Stopping the server ✋

Shortening a URL

Fetching a shortened URL 🔗

More configurations 🎛

Naming the server 📢

Outro 🖖

Understanding the basics of Elixir’s concurrency model

Creating processes

Receiving messages in processes

What if we never receive a message?

Long-running processes

WTF you mean by “an infinite loop”?!

Long-running processes, continued

Keeping state using processes

Putting a value

Getting values

Getting a dump of the store

A small disappointment

Some more reading

A deeper dive in Elixir's Plug

What's Plug?

Web servers, yeehaw!

Enter Plug

Plugging a plug as an endpoint

Diving into the Plug.Conn

Connection lifecycle

Next-level Plugging using Plug.Router

Built-in Plugs

Plug.Head

A short overview of the `testing` package

Logging with `Log` and `Logf`

When to use `Error(f)` or `Fatal(f)`

Testing with Go’s `testing` package

What happens when we run `go test`

Reimplementing `URLShortener`, using `GenServer` ♻️

Diving into the `Plug.Conn`

Next-level `Plug`ging using `Plug.Router`

`Plug.Head`

`Plug.Logger`