DEV Community: Ben Church

How to write a super fast link shortener with Mnesia

Ben Church — Thu, 21 May 2020 03:07:35 +0000

Let's start this month’s tutorial with two statements that are going to get me in trouble:

Elixir is the most productive language out there.
Bit.ly charges way much for their paid plan

Elixir, the Phoenix framework, and the Erlang VM allows us to make production ready systems fast, easily, and with very few moving parts. At this point, you can see where we’re going with this. Let’s use what comes out of the box with Elixir to create a link shortener and prevent a $500 monthly bill.

Let’s dive in!

Initial Setup

Before you start

Please make sure you have the following:

Create a new Phoenix Project

The first thing to do as always is to have Phoenix create the skeleton of the project. For this demo, we won't have a “frontend” so we can tell the initializer to leave those out.

In your terminal type:

mix phx.new shorten_api --no-html --no-brunch

Update your dependencies

Next, we're going to add a few dependencies to mix.exs. Go ahead and update the deps/0 function in that file to look like this:

# mix.exs

# ...
defp deps do
  [
    {:phoenix, "~> 1.3.2"},
    {:phoenix_pubsub, "~> 1.0"},
    {:phoenix_ecto, "~> 3.2"},
    {:postgrex, ">= 0.0.0"},
    {:gettext, "~> 0.11"},
    {:cowboy, "~> 1.0"},
    {:ecto_mnesia, "~> 0.9.1"}
  ]
end
# ...

Logic!

Ok basic set up out of the way. That means we’re into setting up the logic that will:

Allow us to save URLs
Reference these URLs by a unique HashId (ex. abc123)
Navigate to /abc123 and it redirects to the URL that it references

First up, creating a way to store these links.

Creating a way to store these links

Let's use Phoenix’s built-in generators to do this for us. In your terminal run:

mix phx.gen.json Links Link links hash:string:unique url:string:unique

This will create the

Links context
Link model
Link controller
Database migration

That’s honestly the end of that step. It will ask you to put a few lines in your router.ex file but for now, you can skip that if you want as we will touch on it later. Let's move onto how we can modify what was created above to automatically create the id’s we will use to reference these links.

Autogenerate the HashId on Link creation

By default in these systems, models are given an id column in the database that is a number, unique and auto increments: 1, 2, 3 and so on. In our system we want the id to be a:

Short
Unique
String
That auto-generates.

Ecto makes this really easy.

The first thing to do is make a custom Ecto Type which will handle all of this for us. Create a new file shorten_api/ecto/hash_id.ex and populate it as follows:

# shorten_api/ecto/hash_id.ex
defmodule ShortenApi.Ecto.HashId do
  @behaviour Ecto.Type
  @hash_id_length 8
  # ======================= #
  # Ecto Specific Callbacks #
  # ======================= #

  @doc "Called when creating an Ecto.Changeset"
  @spec cast(any) :: Map.t
  def cast(value), do: hash_id_format(value)

  @doc "Converts/accepts a value that has been directly placed into the ecto struct after a changeset"
  @spec dump(any) :: Map.t
  def dump(value), do: hash_id_format(value)

  @doc "Converts a value from the database into the HashId type"
  @spec load(any) :: Map.t
  def load(value), do: hash_id_format(value)

  @doc "Callback invoked by autogenerate fields."
  @spec autogenerate() :: String.t
  def autogenerate, do: generate()

  @doc "The Ecto type."
  def type, do: :string

  # ============ #
  # Custom Logic #
  # ============ #

  @spec hash_id_format(any) :: Map.t
  def hash_id_format(value) do
    case validate_hash_id(value) do
      true -> {:ok, value}
      _ -> {:error, "'#{value}' is not a string"}
    end
  end

  @doc "Validate the given value as a string"
  def validate_hash_id(string) when is_binary(string), do: true
  def validate_hash_id(other), do: false

  @doc "Generates a HashId"
  @spec generate() :: String.t
  def generate do
    @hash_id_length
    |> :crypto.strong_rand_bytes()
    |> Base.url_encode64
    |> binary_part(0, @hash_id_length)
  end
end

Don’t worry if you don’t understand all of this, it’s a little more advanced…

What we did above is essentially create a new type that can be used the same way we define a field as a String or an Integer. Now we can define a field as a HashId.

You can learn more about this in the Ecto documentation.

So let's do just that and update shorten_api/links/link.ex to use a HashId as it’s a primary key instead of an Integer:

# shorten_api/links/link.ex
defmodule ShortenApi.Links.Link do
  use Ecto.Schema
  import Ecto.Changeset
  alias ShortenApi.Ecto.HashId

  @primary_key {:hash, HashId, [autogenerate: true]}
  @derive {Phoenix.Param, key: :hash}
  schema "links" do
    field :url, :string

    timestamps()
  end

  @doc false
  def changeset(link, attrs) do
    link
    |> cast(attrs, [:url])
    |> validate_required([:url])
  end
end

Simple!

Update the migration

Now that the HashId is set up in our code, we want to update the migration to set up the database to reflect what's happening in our model file. You should have a file in your project that ends with _create_links.exs. Find it, open it, and modify it to resemble the below code:

# 20180806064709_create_links.exs
defmodule ShortenApi.Repo.Migrations.CreateLinks do
  use Ecto.Migration

  def change do
    create table(:links, primary_key: false) do
      add :hash, :string, primary_key: true
      add :url, :string

      timestamps()
    end

    create unique_index(:links, [:url])
  end
end

Note we are telling Ecto to use a different field as the primary key.

Alright, that's the majority of our plumbing steps, now we’re going to jump into the core logic of this whole project.

Redirect from an Id to a URL

First, we need a function in our controller that

Takes an Id of a Link
Looks up the Link
Redirects to the URL attached to that Link

To do this, let's add a new function to our link controller found here: shorten_api_web/controllers/link_controller.ex

# shorten_api_web/controllers/link_controller.ex
# ...

def get_and_redirect(conn, %{"id" => id}) do
  url = id
    |> Links.get_link!()
    |> Map.get(:url)
  redirect(conn, external: url)
end

# ...

Doesn’t Phoenix and Elixir make this so so pretty? 😍

Hook it all up to our router

Now that we have this new controller function, the only thing left is to hook it up. Update the router.ex file to reflect the following:

Note: we will also be adding the routes to mix phx.gen suggested earlier

# router.ex
defmodule ShortenApiWeb.Router do
  use ShortenApiWeb, :router

  pipeline :api do
    plug :accepts, ["json"]
  end

  scope "/api", ShortenApiWeb do
    pipe_through :api
    resources "/links", LinkController, except: [:edit]
  end

  scope "/", ShortenApiWeb do
    get "/:id", LinkController, :get_and_redirect
  end
end

TADA! 🎉

At this point, you should be able to run the project with mix phx.server and have everything function as expected! However, we’re not stopping here.

Secret sauce

Because link shorteners sit between a user and the actual content, it is crucial that these systems are fast. While Elixir is already fast, the main lag time in this process comes from our database. It takes time to look up the Link attached to an id.

To speed this up, link shorteners will often opt to use an in-memory data store like Redis as opposed to an on-disk database like Postgres (which Phoenix sets us up with by default). Thankfully because Elixir is built on top of the Erlang VM, we already have an in-memory datastore built-in: Mnesia!

In the next section, we’re going to alter our configuration to use Mnesia instead of Postgres.

Swapping from Postgres to Mnesia

This process is actually very simple. First update config.exs as shown:

# config.exs
# ...
# Configure your database
config :shorten_api, ShortenApi.Repo,
  adapter: EctoMnesia.Adapter

config :ecto_mnesia,
  host: {:system, :atom, "MNESIA_HOST", Kernel.node()},
  storage_type: {:system, :atom, "MNESIA_STORAGE_TYPE", :disc_copies}

config :mnesia,
  dir: 'priv/data/mnesia'
# ...

Create your Mnesia database

Then create the location where Mnesia will back data up to and initialize the database through Ecto:

mkdir priv/data
mkdir priv/data/mnesia
mix ecto.create
mix ecto.migrate

Boom done! You’re now using an In-Memory database to hold our Link information. Wasn’t that easy?

Kick it off

You now can

start the project: mix phx.server
Create a shortened link via curl:

curl --request POST \
  --url http://localhost:4000/api/links/ \
  --header 'content-type: application/json' \
  --data '{
 "link": {
  "url": "https://twitter.com/bnchrch"
 }
}'

Take the hash returned in the response

{“data”:{“url”:”[https://twitter.com/bnchrch](https://twitter.com/bnchrch)","hash":"7cJY_ckq"}}

And be redirected appropriately when you go to localhost:4000/{hash}:

Wrap up

Amazing how easy it was with all these tools to create a fast, easy to maintain and simple to extend link shortener. A lot of the credit here can go to the BEAM (Erlang VM), Jose Valim (creator of Elixir), and Chris McCord (creator of Phoenix). The rest of the praise goes to how simple the idea of a link shortener is, in no way justifying a $500 per month introductory price tag. Still looking at you Bit.ly.

🧞‍ This is open source! you can find it here on Github

❤️ I only write about programming and remote work. If you follow me on Twitter I won’t waste your time.

Writing a Reverse Proxy in just one line with Go

Ben Church — Fri, 31 Jan 2020 18:18:26 +0000

Leave your programming language hang ups at the door and come admire the best standard library I’ve ever come across.

Choosing a Programming Language for a project shouldn’t be like declaring who your favourite team is. it should be a matter of pragmatism and choosing the right tool for the right job.

In this post I want to show you when and why Go shines as a language. Specifically I want to show you how rock solid their standard lib is for basic internet programming. Even more specifically… were gonna write a Reverse Proxy!

“Go has a lot going for it but where it really struts it stuff is in these lower level network plumbing tasks, there’s no better language.”

What is a reverse proxy? A big fancy way of saying a traffic forwarder. I get a request send from a client, send that request to another server, receive a response from the server and forward it back to the client. The reverse part of this simply means the proxy itself determines where to send traffic and when

Why is it useful? Because the concept is so simple it can be applied to assist in many different cases: Load balancing, A/B Testing, Caching, Authentication etc…

By the end of this short post you will have learned how to:

Serve HTTP requests
Parse the body of a request
Serve traffic to another server using a Reverse Proxy

Our Reverse Proxy Project

Lets dive into the actual project. What we are going to do is have a web server that:

Takes requests
Reads the body of a request, specifically the proxy_condition field
If the proxy domain is equal to A send traffic to URL 1
If the proxy domain is equal to B send traffic to URL 2
If the proxy domain is neither then send traffic to the Default URL.

Prerequisites

Go for programming with.
http-server for creating simple servers with.

Setting up our environment

First thing we want to do is input all the required configuration variables into our environment so that we can both use them in our application while keeping them out of source code.

I find the best approach is to create a .env file that contains the desired environment variables.

Below is what I have for this specific project:

export PORT=1330
export A_CONDITION_URL="http://localhost:1331"
export B_CONDITION_URL="http://localhost:1332"
export DEFAULT_CONDITION_URL="http://localhost:1333"

This is a habit I picked up from the 12 Factor App

After you save your .env file you can run:

source .env

to configure load the config into your environment any time.

Laying the foundation of our project

Next lets create a file called main.go that does the following:

When started logs the PORT, A_CONDITION_URL, B_CONDITION_URL, and DEFAULT_CONDITION_URL environment variables to the console
Listen for requests on the path: /

package main

import (
  "bytes"
  "encoding/json"
  "io/ioutil"
  "log"
  "net/http"
  "net/http/httputil"
  "net/url"
  "os"
  "strings"
)

// Get env var or default
func getEnv(key, fallback string) string {
  if value, ok := os.LookupEnv(key); ok {
    return value
  }
  return fallback
}

// Get the port to listen on
func getListenAddress() string {
  port := getEnv("PORT", "1338")
  return ":" + port
}

// Log the env variables required for a reverse proxy
func logSetup() {
  a_condtion_url := os.Getenv("A_CONDITION_URL")
  b_condtion_url := os.Getenv("B_CONDITION_URL")
  default_condtion_url := os.Getenv("DEFAULT_CONDITION_URL")

  log.Printf("Server will run on: %s\n", getListenAddress())
  log.Printf("Redirecting to A url: %s\n", a_condtion_url)
  log.Printf("Redirecting to B url: %s\n", b_condtion_url)
  log.Printf("Redirecting to Default url: %s\n", default_condtion_url)
}

// Given a request send it to the appropriate url
func handleRequestAndRedirect(res http.ResponseWriter, req *http.Request) {
  // We will get to this...
}

func main() {
  // Log setup values
  logSetup()

  // start server
  http.HandleFunc("/", handleRequestAndRedirect)
  if err := http.ListenAndServe(getListenAddress(), nil); err != nil {
    panic(err)
  }
}

💀Let’s get the skeletons out of the closet so we can move onto the fun stuff.

Now you should be able to run

Parse the request body

Now that we have the skeleton of our project together we want to start creating the logic that will handle parsing the request body. Start by updating handleRequestAndRedirect to parse the proxy_condition value from the request body.

type requestPayloadStruct struct {
  ProxyCondition string `json:"proxy_condition"`
}

// Get a json decoder for a given requests body
func requestBodyDecoder(request *http.Request) *json.Decoder {
  // Read body to buffer
  body, err := ioutil.ReadAll(request.Body)
  if err != nil {
    log.Printf("Error reading body: %v", err)
    panic(err)
  }

  // Because go lang is a pain in the ass if you read the body then any susequent calls
  // are unable to read the body again....
  request.Body = ioutil.NopCloser(bytes.NewBuffer(body))

  return json.NewDecoder(ioutil.NopCloser(bytes.NewBuffer(body)))
}

// Parse the requests body
func parseRequestBody(request *http.Request) requestPayloadStruct {
  decoder := requestBodyDecoder(request)

  var requestPayload requestPayloadStruct
  err := decoder.Decode(&requestPayload)

  if err != nil {
    panic(err)
  }

  return requestPayload
}

// Given a request send it to the appropriate url
func handleRequestAndRedirect(res http.ResponseWriter, req *http.Request) {
  requestPayload := parseRequestBody(req)
    // ... more to come
}

Basic parsing of a JSON blob to a struct in Go.

Use `proxy_condition` to determine where we send traffic

Now that we have the value of the proxy_condition from the request we will use it to decide where we direct our reverse proxy to. Remember from earlier that we have three cases:

If proxy_condition is equal to A then we send traffic to A_CONDITION_URL
If proxy_condition is equal to B then we send traffic to B_CONDITION_URL
Else send traffic to DEFAULT_CONDITION_URL

// Log the typeform payload and redirect url
func logRequestPayload(requestionPayload requestPayloadStruct, proxyUrl string) {
  log.Printf("proxy_condition: %s, proxy_url: %s\n", requestionPayload.ProxyCondition, proxyUrl)
}

// Get the url for a given proxy condition
func getProxyUrl(proxyConditionRaw string) string {
  proxyCondition := strings.ToUpper(proxyConditionRaw)

  a_condtion_url := os.Getenv("A_CONDITION_URL")
  b_condtion_url := os.Getenv("B_CONDITION_URL")
  default_condtion_url := os.Getenv("DEFAULT_CONDITION_URL")

  if proxyCondition == "A" {
    return a_condtion_url
  }

  if proxyCondition == "B" {
    return b_condtion_url
  }

  return default_condtion_url
}

// Given a request send it to the appropriate url
func handleRequestAndRedirect(res http.ResponseWriter, req *http.Request) {
  requestPayload := parseRequestBody(req)
  url := getProxyUrl(requestPayload.ProxyCondition)
  logRequestPayload(requestPayload, url)
  // more still to come...
}

Reverse Proxy to that URL

Finally we are onto the actual reverse proxy! In so many languages a reverse proxy would require a lot of thought and a fair amount of code or at least having to import a sophisticated library.

However Golang’s standard library makes creating a reverse proxy so simple it’s almost unbelievable. Below is essentially the only line of code you need:

httputil.NewSingleHostReverseProxy(url).ServeHTTP(res, req)

Note that in the following code we add a little extra so it can fully support SSL redirection (though not necessary):

// Serve a reverse proxy for a given url
func serveReverseProxy(target string, res http.ResponseWriter, req *http.Request) {
  // parse the url
  url, _ := url.Parse(target)

  // create the reverse proxy
  proxy := httputil.NewSingleHostReverseProxy(url)

  // Update the headers to allow for SSL redirection
  req.URL.Host = url.Host
  req.URL.Scheme = url.Scheme
  req.Header.Set("X-Forwarded-Host", req.Header.Get("Host"))
  req.Host = url.Host

  // Note that ServeHttp is non blocking and uses a go routine under the hood
  proxy.ServeHTTP(res, req)
}

// Given a request send it to the appropriate url
func handleRequestAndRedirect(res http.ResponseWriter, req *http.Request) {
  requestPayload := parseRequestBody(req)
  url := getProxyUrl(requestPayload.ProxyCondition)

  logRequestPayload(requestPayload, url)

  serveReverseProxy(url, res, req)
}

The one time in the project it felt like Go was truly getting out of my way.

Start it all up

Ok now that we have this all wired up setup our application on port 1330 and our 3 simple servers on ports 1331–1333 (all in separate terminals):

source .env && go install && $GOPATH/bin/reverse-proxy-demo
http-server -p 1331
http-server -p 1332
http-server -p 1333

With all these up and ruuning we can start to send through a requests with a json body in another terminal like so:

curl --request GET \

  --url http://localhost:1330/ \

  --header 'content-type: application/json' \

  --data '{

    "proxy_condition": "a"

  }'

If your looking for a great HTTP request client I cannot recommend Insomnia enough.

and Viola we can start to see our reverse proxy directing traffic to one of our 3 servers based on what we set in the proxy_condition field!

Wrap Up

Go has a lot going for it but where it really struts it stuff is in these lower level network “plumbing” tasks, there’s no better language. What we’ve written here is simple, performant, reliable and very much ready for use in production.

For simple services I can see myself reaching for Go again in the future.

🧞‍ This is open source! you can find it here on Github

❤️ I only write about programming and remote work. If you follow me on Twitter I won’t waste your time.

Lexers and Parsers: Letting Business Write Business Logic

Ben Church — Wed, 13 Nov 2019 02:15:51 +0000

Today we’re talking about Lexers and Parsers, more importantly I want to show you how you:

Can give your business team a crazy amount of flexibility
With out compromising your systems.
Or spending a lot of effort.

After reading this you should know how to:

Tokenize a string with leex (an Erlang lexer library)
Turn those tokens into an Abstract Syntax Tree with yecc (an Erlang parser generating library)
Apply logic to that AST with Elixir.

Essentially how to build a boolean based templating language! (in very few lines of code)

The Problem.

Lets imagine a hypothetical project:

A application that is responsible for sending surveys to customers of veterinary clinics.

and the PM just created this story:

As an administrator I would like to be able to determine who gets a survey based on the location of the user, their pets age and the clinic they are a customer of.

Now right away there is a lot of quick sand here.

Many requirements and parameters; that could change often; using a lot of dev time.

For example in the near future we may want to send surveys based on a user or clinic feature flag, a pets species or breed and potentially some other model entirely. Leaving us a lot to edit, repeatedly.

The Solution.

Give your entire company commit access!

Just kidding!

This is a story of who will be responsible for the changes and how. Think of it as a spectrum ranging from 100% dev to 100% admin responsibility. Some solutions on that spectrum:

Teach administrators to code and let them create pull requests
Create an admin panel with an element per field and let admins use it
Devs hard code these rules at the requests of administrators

As with most spectrums, the ends are extreme, the same goes for options 1 and 3. 1 results in a severely unstable and vulnerable system, 3 would lead to a slow process and product cycle.

Option 2 is the obvious and sane path, but that doesn’t exempt it from drawbacks.

It will introduce a lifelong set of overhead into the system: The overhead of everytime we add/remove a new field we have to be conscious of adding/removing the views, routes and controllers that is related to the administration of that field in relation to these surveys. On a large team this may be trivial but for a small team this could take up a significant amount of total dev time.

The question is: if dev time is a constraint is there another solution between 1 and 2. Thankfully largely due to Elixirs interop with Erlang there is. With the help of yecc and leex: Erlangs built in lexing and parsing libraries we can easily create a boolean language to help business safely define the rules on the system.

Don't believe me? Here let me show you.

What are we going to create

For demo purposes we are going to stick to a simple template language that will include:

TRUE and FALSE values
AND operator
NOT operator
Variables

The priniciples from these will allow us to extend the solution to use:

Numerical values
Logical operators (< , >, ≤, ≥, =)

However with brevity in mind I will cover that in a subsequent post :)

Setup

Make sure you have Elixir installed then run:

mix new lexer_parser
cd lexer_parser
mix deps.get
mkdir src
touch src/business_lexer.xrl
touch src/business_parser.yrl

Next we want to create 2 additional files that will contain our code.

mkdir src
touch src/business_lexer.xrl
touch src/business_parser.yrl

These are the only files we are adding over the course of this whole demo! Simple right?

Start simple

Lets first focus on getting something working. What we want to build is a function that takes a string containing the keywords true, false and and then returns the resulting boolean value.

For example:

LexerParser.evaluate('true and false')
{:ok, false}

LexerParser.evaluate('true and true')
{:ok, true}

LexerParser.evaluate('false')
{:ok, false}

How this system works (Explained 4 different ways)

Lexer → Parser → Elixir interface

While this might seem new and intimidating I assure you its not.

Our system is easily broken into 3 steps

Tokenize: Identify substrings of significance
Parse: Identify the sequence of these tokens
Compute: Determine the result of the sequence

Another way to conceptualize it is the different shapes data will take through our system.

String → Array → Abstract Syntax Tree → Boolean

And finally in really pragmatic form:

Lexer (.xrl) → Parser (.yrl) → Apply Logic(.ex)

Lets kick this off with the Lexer.

Lexer (leex)

Definition

.xrl files are used by [leex](http://erlang.org/doc/man/leex.html) which is a lexer analyzer generator for Erlang. Again it sounds complicated, it’s not. Lexers apply rules to a string that will identify tokens and turn them into a format we can use.

It’s how we know that true, false and and are important while whitespace and other noise can be ignored.

Here lets dive into the code first, then we can pick it apart.

The Code

Add the following to src/business_lexer.xrl

Definitions.
WS    = ([\\000-\\s]|%.*)

Rules.
false   : {token, {false,   TokenLine}}.
true    : {token, {true,    TokenLine}}.

and     : {token, {and_op,  TokenLine, list_to_atom(TokenChars)}}.

{WS}+   : skip_token.

What does this do

Definitions: These are the patterns of characters we are looking for. This is how we go from list of characters to a token.

Rules: <pattern> : <result>. This defines what we do with a token after we've identified it. This is where we first apply meaning.

TokenLine: The line number where the token occurred. (provided by leex)

TokenChars: The list of the characters in the matched token. (provided by leex)

list_to_atom: Changes a list of characters to an atom, this is to be able to match on the :and atom later. (provided by erlang)

skip_token: Discard the token; Don’t apply any meaning to it.

Run the code

Let’s see what happens if we run some simple strings through the Lexer. You can do so anywhere in your elixir project via the :business_lexer.string/1 function. Lets do that! Start by opening your terminal and running iex -S mix . Next enter the following:

iex(1)> :business_lexer.string('true')
{:ok, [true: 1], 1}

iex(2)> :business_lexer.string('false')
{:ok, [false: 1], 1}

iex(3)> :business_lexer.string('true and false')
{:ok, [{true, 1}, {:and_op, 1, :and}, {false, 1}], 1}

iex(4)> :business_lexer.string('and')
{:ok, [{:and_op, 1, :and}], 1}

iex(5)> :business_lexer.string('doesnt exist')
{:error, {1, :business_lexer, {:illegal, 'd'}}, 1}

Notice that it should return :ok for tokens the lexer recognizes and :error for the tokens it does not.

This is what we will pipe into the Parser.

Parser (yecc)

Definition

.yrl files are used by [yecc](http://erlang.org/doc/man/yecc.html) which is a parser generator for Erlang. If Lexers were about determining which sequence of characters are important, Parsers are about determining what to do with them. Parsers give meaning to the order of these tokens.

1. Its how we determine that the tokens on the left and right of and are needed.

2. Its also how we determine true and false don't need any other information to be parsed.

The Code

Add the following to src/business_parser.yrl

Nonterminals expression bool.

Terminals and_op true false.

Rootsymbol expression.

expression -> bool : '$1'.
expression -> expression and_op expression  : {binary_expr, and_op, '$1', '$3'}.

bool -> true : true.
bool -> false : false.

What am I looking at

Terminals: These are what we built in the leex file. These are tokens we’ve found and thus are not worth further breaking up. Note here that expression and bool are keywords defined by us below. These Terminals can only appear on the RHS of the rules.

Nonterminals: These are essentially more complex structures that we define with multiple rules below. Essentially patterns of terminals. These can appear on either the right hand side or the left hand side.

Rootsymbol: The starting point for the whole AST. This tells our grammar where to start and should be define in at least one rule.

Grammar Rules: These are the lines in the format <nonterminal> -> <pattern of terminals and non terminals> : <result> . They define how we interpret patterns and the code we return. It's key to note that the result will be what we interpret in the elixir code. For example expression -> expression and_op expression : {binary_expr, and_op, '$1', '$3'}. when given true and true will return {:binary_expr, :and_op, true, false}

Binary expression: The binary_expr is an atom will will match on to denote that the operator (in this case :and_op) will have two operands (children).

Run the code

Alright lets run this, like the lexer this parser will be available anywhere in your project by calling :business_parser.parse/1 the thing to know is that the input for this function is the tokens from the output of :business_lexer.string/1 . Lets open our iex terminal again and you can see what exactly that looks like:

iex(1)> 'true'
  |> :business_lexer.string()
  |> fn {:ok, tokens, _} -> :business_parser.parse(tokens) end.()

{:ok, true}

iex(2)> 'false'
  |> :business_lexer.string()
  |> fn {:ok, tokens, _} -> :business_parser.parse(tokens) end.()

{:ok, false}

iex(3)> 'true and false'
  |> :business_lexer.string()
  |> fn {:ok, tokens, _} -> :business_parser.parse(tokens) end.()

{:ok, {:binary_expr, :and_op, true, false}}

iex(4)> 'true and true and true'
  |> :business_lexer.string()
  |> fn {:ok, tokens, _} -> :business_parser.parse(tokens) end.()

{:ok, {:binary_expr, :and_op, true, {:binary_expr, :and_op, true, true}}}

iex(5)> 'and'
  |> :business_lexer.string()
  |> fn {:ok, tokens, _} -> :business_parser.parse(tokens) end.()

{:error, {1, :business_parser, ['syntax error before: ', ['\\'and\\'']]}}

iex(6)> 'doesnt exist' |> :business_lexer.string() |> fn {:ok, tokens, _} -> :business_parser.parse(tokens) end.()
** (FunctionClauseError) no function clause matching in :erl_eval."-inside-an-interpreted-fun-"/1

    The following arguments were given to :erl_eval."-inside-an-interpreted-fun-"/1:

        # 1
        {:error, {1, :business_lexer, {:illegal, 'd'}}, 1}

And there we go! We are going from a string all the way to an AST. We are now lexing and parsing with no additional dependencies and very few lines of code. The final step from here is to interpret the tree. To do that we can use our favorite language…Elixir!

Applying the logic

We’re going to make use of 3 functions and some pattern matching to wrap this all up. Our goal here is to interpret the output of our parser into either true or false.

Lets start by modifying our lexer_parser.ex file to include a function that summarizes our system:

def evaluate(expression) do
    with {:ok, tokens, _} <- :business_lexer.string(expression),
         {:ok, tree} <- :business_parser.parse(tokens) do
      evaluate_tree(tree)
    end
  end

Next lets define the function that evaluates the tree, the rules to define are:

If it’s a binary expression, evaluate the tree at each operand and afterwards take the results and apply logic to them based on the operator.
If it’s true or false go straight to applying logic (hint: the logic will be fairly redundant at this stage)

# Tree functions
# =============

def evaluate_tree({:binary_expr, op, a, b}) do
  with {:ok, a} <- evaluate_tree(a),
        {:ok, b} <- evaluate_tree(b) do
    apply_logic({:binary_expr, op, a, b})
  end
end

def evaluate_tree(other) when other in [true, false] do
  apply_logic(other)
end

Finally lets apply the logic that will make use of true false and and:

# Logic functions
# ==============

def apply_logic(boolean) when boolean in [true, false], do: {:ok, boolean}

def apply_logic({:binary_expr, :and_op, a, b})
  when is_boolean(a) and is_boolean(b), do: {:ok, a and b}

Run the code

Again jump into your iex shell and test this all out:

iex(1)> LexerParser.evaluate('true')
{:ok, true}

iex(2)> LexerParser.evaluate('false')
{:ok, false}

iex(3)> LexerParser.evaluate('true and false')
{:ok, false}

iex(4)> LexerParser.evaluate('true and true and true')
{:ok, true}

iex(5)> LexerParser.evaluate('and')
{:error, {1, :business_parser, ['syntax error before: ', ['\\'and\\'']]}}

iex(6)> LexerParser.evaluate('doesnt exist')
{:error, {1, :business_lexer, {:illegal, 'd'}}, 1}

And Done! We’ve changed the whole game! Alright not really but from here are the foundations to iteratively add a-lot of functionality really easy. Like how about adding variables?

Variables

Adding the ability to define variables is trivial. The end result should look something like this:

iex(1)> LexerParser.evaluate('a and b', %{"a" => true, "b" => true})
{:ok, true}

Let’s dive in:

1. Add the appropriate definition and rule to the lexer

We want to add a definition that will encapsulate any variable name ([A-Za-z_][0-9a-zA-Z_]*) and a rule that will take that variable name and transform it into an atom and mark it as var.

This is what your business_lexer.xrl file should look like afterwards:

Definitions.

VAR   = ([A-Za-z_][0-9a-zA-Z_]*)
WS    = ([\\000-\\s]|%.*)

Rules.

false   : {token, {false,   TokenLine}}.
true    : {token, {true,    TokenLine}}.

and     : {token, {and_op,  TokenLine, list_to_atom(TokenChars)}}.

{VAR}   : {token, {var,     TokenLine, list_to_binary(TokenChars)}}.
{WS}+   : skip_token.

2. Update the parser to include a new Terminal, Rule and a small Erlang function

We essentially want to tell the parser that when given a variable to use it’s key as it’s value.

This is what your business_parser.yrl file should look like afterwards:

Nonterminals expression bool.

Terminals var and_op true false.

Rootsymbol expression.

expression -> bool : '$1'.
expression -> var : extract('$1').

expression -> expression and_op expression  : {binary_expr, and_op, '$1', '$3'}.

bool -> true : true.
bool -> false : false.

Erlang code.

extract({T,_,V}) -> {T, V}.

Note the additions of the Erlang code. section. Nothing fancy here just a function that will pull the value out for us.

3. Update our functions to allow for variables and to interpret the var atom.

Simply we want to include one additional rule to the apply_logic function that when it encounters :var to return the related value from a given map of variables.

This is what your lexer_parser.ex file should look like afterwards:

def evaluate(expression, variables \\ %{}) do
    with {:ok, tokens, _} <- :business_lexer.string(expression),
         {:ok, tree} <- :business_parser.parse(tokens) do
      evaluate_tree(tree, variables)
    end
  end

  # Tree functions
  # =============

  def evaluate_tree({:binary_expr, op, a, b}, variables) do
    with {:ok, a} <- evaluate_tree(a, variables),
         {:ok, b} <- evaluate_tree(b, variables) do
      apply_logic({:binary_expr, op, a, b}, variables)
    end
  end

  def evaluate_tree(other, variables) do
    apply_logic(other, variables)
  end

  # Logic functions
  # ==============

  def apply_logic(boolean, _variables) when boolean in [true, false], do: {:ok, boolean}

  def apply_logic({:binary_expr, :and_op, a, b}, _variables)
    when is_boolean(a) and is_boolean(b), do: {:ok, a and b}

  def apply_logic({:var, variable}, variables) do
    case Map.get(variables, variable, nil) do
      nil -> {:error, "variable \\"#{variable}\\" not provided in: #{inspect variables}"}
      value -> {:ok, value}
    end
  end

Run the code

iex(1)> LexerParser.evaluate('some_system_defined_bool and another_system_defined_bool', %{"some_system_defined_bool" => true, "another_system_defined_bool" => true})
{:ok, true}

iex(2)> LexerParser.evaluate('some_system_defined_bool and another_system_defined_bool', %{"some_system_defined_bool" => true, "another_system_defined_bool" => false})
{:ok, false}

NOT

Another type of rule we can add is an unary expression. Essentially an operator that takes one operand. The best demonstration of this is the not operator. This would be a rule that inverts a boolean value. For example:

iex(1)> LexerParser.evaluate('not false')
{:ok, true}

Let me show you how easy this would be to add.

1. Update the lexer

Your getting the hang of this now so I’ll make this more to the point. Add the following rule to business_lexer.xrl:

not     : {token, {not_op,  TokenLine, list_to_atom(TokenChars)}}.

2. Update the parser

First add the not_op to your list of terminals:

Terminals var and_op not_op true false.

Then add a rule, tagging the not_op as an unary expression and passing along the value to the right of it:

expression -> not_op expression : {unary_expr, not_op, '$2'}.

3. Update our logic

Add one extra definition to evaluate_tree2 that handles a unary expression:

def evaluate_tree({:unary_expr, op, a}, variables) do
    with {:ok, a} <- evaluate_tree(a, variables) do
      apply_logic({:unary_expr, op, a}, variables)
    end
  end

And add one extra definition to apply_logic/2 that applies the ! to any :not_op value:

def apply_logic({:unary_expr, :not_op, a}, _)
    when is_boolean(a), do: {:ok, !a}

Run the code

iex(1)> LexerParser.evaluate('true and not false and a', %{"a" => true})
{:ok, true}

The rest of the f**cking owl

Like I said at the start of the post we could keep going on to include numerical values and other operators like >, <, =, &c. But I won’t do that. Instead I’m going to point you towards a open source library that will do it for you and show you a glimpse of the code so that you know just how simple it still is.

Enter Expreso

Expreso is a great library maintained by ympons that does all of this heavy lifting for us. It includes:

+, -, /, *
in
>, <, ≥, ≤, =, ≠
and, or, not
floats and integers

Here’s an example of what it can accomplish:

iex(1)> Expreso.eval('not email_disabled and pet_age > 4.1 or clinic_id = 123', %{"email_disabled" => false, "pet_age" => 4.2, "clinic_id" => 124})
{:ok, true}

and the code isn’t much more complicated that what we’ve wrote above. Here’s what the parser from this library looks like:

Nonterminals expression predicate scalar_exp elements element bool.

Terminals atom var number string mult_op add_op and_op or_op in_op not_op eq_op comp_op '(' ')' ',' true false.

Rootsymbol expression.
Left 100 or_op.
Left 200 and_op.
Left 300 eq_op comp_op.
Left 400 in_op.
Left 500 add_op.
Left 600 mult_op.
Nonassoc 700 not_op.

expression -> bool : '$1'.
expression -> predicate : '$1'.
expression -> var : extract('$1').
expression -> expression or_op expression   : {binary_expr, or_op, '$1', '$3'}.
expression -> expression and_op expression  : {binary_expr, and_op, '$1', '$3'}.
expression -> not_op expression : {unary_expr, not_op, '$2'}.
expression -> '(' expression ')' : '$2'.

predicate -> bool eq_op bool : {binary_expr, extract('$2'), '$1', '$3'}.
predicate -> scalar_exp eq_op scalar_exp : {binary_expr, extract('$2'), '$1', '$3'}.
predicate -> scalar_exp comp_op scalar_exp : {binary_expr, extract('$2'), '$1', '$3'}.
predicate -> scalar_exp in_op '(' elements ')' : {binary_expr, in_op, '$1', '$4'}.
predicate -> scalar_exp not_op in_op '(' elements ')' : {binary_expr, not_in_op, '$1', '$5'}.

predicate -> scalar_exp in_op var : {binary_expr, in_op, '$1', extract('$3')}.
predicate -> scalar_exp not_op in_op var : {binary_expr, not_in_op, '$1', extract('$4')}.

scalar_exp -> scalar_exp add_op scalar_exp : {binary_expr, extract('$2'), '$1', '$3'}.
scalar_exp -> scalar_exp mult_op scalar_exp: {binary_expr, extract('$2'), '$1', '$3'}.
scalar_exp -> element : '$1'.

elements -> element : [extract_value('$1')].
elements -> element ',' elements : [extract_value('$1')|'$3'].

element -> atom : '$1'.
element -> var : extract('$1').
element -> string : extract('$1').
element -> number : extract('$1').

bool -> true : true.
bool -> false : false.

Erlang code.

extract_value({_,V}) -> V.
extract({T,_,V}) -> {T, V}.

You can see there's not alot of new concepts in here. Just more copy and pasting as new operators are added.

Wrap this all up

Give me a lever and a place to stand and I will move the earth.

- Archimedes

In the end the challenge to overcome with diving into Erlang and its tooling isn't complexity, it's syntax. If your willing to put in the time to learn the basics the leverage you can apply to building systems is immense.

And if you don't feel you have the time, theres always the community's shoulders to stand on because they've built some pretty great things.

DEV Community: Ben Church

How to write a super fast link shortener with Mnesia

Initial Setup

Before you start

Create a new Phoenix Project

Update your dependencies

Logic!

Creating a way to store these links

Autogenerate the HashId on Link creation

Update the migration

Redirect from an Id to a URL

Hook it all up to our router

TADA! 🎉

Secret sauce

Swapping from Postgres to Mnesia

Create your Mnesia database

Kick it off

Wrap up

🧞‍ This is open source! you can find it here on Github

❤️ I only write about programming and remote work. If you follow me on Twitter I won’t waste your time.

Writing a Reverse Proxy in just one line with Go

Our Reverse Proxy Project

Prerequisites

Setting up our environment

Laying the foundation of our project

Parse the request body

Use proxy_condition to determine where we send traffic

Reverse Proxy to that URL

Start it all up

Wrap Up

Lexers and Parsers: Letting Business Write Business Logic

The Problem.

The Solution.

What are we going to create

Setup

Start simple

How this system works (Explained 4 different ways)

Lexer (leex)

The Code

What does this do

Run the code

Parser (yecc)

What am I looking at

Run the code

Applying the logic

Run the code

Variables

1. Add the appropriate definition and rule to the lexer

2. Update the parser to include a new Terminal, Rule and a small Erlang function

3. Update our functions to allow for variables and to interpret the var atom.

Run the code

NOT

1. Update the lexer

2. Update the parser

3. Update our logic

Run the code

The rest of the f**cking owl

Enter Expreso

Wrap this all up

Use `proxy_condition` to determine where we send traffic