DEV Community: David Sulc

Validating Data in Elixir: Using Ecto and NimbleOptions

David Sulc — Tue, 14 Nov 2023 14:34:39 +0000

In the previous part of this series about validating data at the boundary of an Elixir application, we covered a few general programming tactics to try and reject invalid and unexpected data in our software.

Continuing with that subject, we'll now explore how two libraries, namely Ecto and NimbleOptions, can further assist us.

Let's get started!

Using Ecto

As we've seen previously, Elixir provides many native techniques to help us guarantee data quality in our systems.

But it's also possible to leverage Ecto to cast, validate, and prune data even if there's no database interaction.

Schemaless Changesets in Ecto

Schemaless changesets (basically Ecto changesets that aren't tied to a database table) are a convenient way to create data structures. They also prevent bad data from making its way into structs (this can happen when bad data makes it past constructors or is added directly to a struct).

This approach is typically helpful when dealing with untrusted input (e.g., from an API request or as part of a module's public API). We can expose a function that will accept a plain map and, after proper vetting, will yield a struct. Other functions in the same module can then accept such a struct instance (rather than a map) to indicate that the data has been vetted and is safe to consume.

Here's what that approach can look like:

# in the Account module

defstruct [:name, suspended: false]

def from_params(%{} = params) do
  data  = %{}
  types = %{name: :string, suspended: :boolean}

  changeset =
    {data, types}
    |> Ecto.Changeset.cast(params, Map.keys(types))
    |> Ecto.Changeset.validate_required(...)
    |> Ecto.Changeset.validate_length(...)

  case apply_action(changeset, :insert) do
    {:ok, data} -> {:ok, struct(__MODULE__, data)}
    {:error, _} = error -> error
  end
end

Note that we're applying an :insert action, so we could even use our function's error to display directly in a Phoenix form: those require an :insert or :update action to render possible errors with the form's data.

Applying This To Phoenix Forms

Indeed, Phoenix forms inspect the action to determine whether error hints should be displayed: if no action is set, no errors will be rendered in the form. This is useful if an empty changeset is being used to render a Phoenix form: the changeset is invalid, but we don't want to berate the user with errors when they haven't (yet) made any actual errors. But this also means that if you want the validation errors resulting from from_params to show up in a Phoenix form, you need to set the :action value yourself, which is what we're accomplishing with our call to apply_action.

Of course, if you don't plan on using this with a Phoenix-rendered form, it won't be needed and can be safely skipped.

Creating a new validated account is now a simple matter of Account.from_params(%{name: "ACME", suspended: false}).

And since the types are dynamic, they could also be passed in as arguments if your domain requires it.

Embedded Schemas in Ecto

Let's say you already have an Account struct that isn't persisted to a database but still want to use Ecto to validate the data. In that case, you can transition to using an embedded schema rather than a basic struct. The trade-off between a schemaless changeset and an embedded schema is that the latter provides a bit more convenience at the expense of flexibility.

Namely, embedded schemas require you to have a struct within which to define the schema. Schemaless changesets don't have that requirement since, as their name implies, they don't need a schema definition.

Also, the datatypes for attributes can be dynamic in the schemaless changeset case (and provided as an argument to a function call, for example). In contrast, embedded schema types cannot vary from their defined value. Here's the above example rewritten to use an embedded schema:

# in the Account module

@primary_key false
embedded_schema do
  field :name, :string
  field :suspended, :boolean
end

def from_params(%{} = params) do
  changeset =
    %__MODULE__{}
    |> Ecto.Changeset.cast(params, [:name, :suspended])
    |> Ecto.Changeset.validate_required(...)
    |> Ecto.Changeset.validate_length(...)

  case apply_action(changeset, :insert) do
    {:ok, data} -> {:ok, struct(__MODULE__, data)}
    {:error, _} = error -> error
  end
end

If you'd like to dig deeper into how Ecto can also help you with your non-persisted data, I highly recommend reading Ecto's Data mapping and validation guide.

Validating Options

There are many circumstances where data is passed in as keyword lists (e.g., options given to OTP modules such as GenServers). These values need to be validated for conformity, but we also want that validation to be easy to understand and communicate. Enter NimbleOptions!

The NimbleOptions Library for Elixir

NimbleOptions is a great tool for validating options, as it is a lightweight library that verifies keyword lists and returns errors on invalid data. Let's see how it can be used!

Say we want to email suspended Account records. The email template to use will be different if we email a corporate or personal account. Additionally, you want to specify which values to pass into the template.

We'll want to adapt the signature: corporate accounts, for example, should have an account manager's name attached, while private accounts can simply have a generic signature. We'll also specify which URL to include as the call to action:

def email(%Account{} = account, opts) when is_list(opts) do
  template = Keyword.fetch!(opts, :template)
  values = Keyword.get(opts, :values, [])
  account_url = Keyword.get(values, :landing_url)
  signature = Keyword.get(values, :signature, "Yours truly,\nACME.com")

  # email generation and sending goes here

  :ok
end

This is a great start: we're enforcing the :template to be provided and the program will crash if the template isn't given. This keeps out bad data, but isn't a great experience for callers: if they make a mistake (even as simple as a typo on the template name!) everything is going to crash instead of just generating an error they can handle.

We could, of course, do something like this:

def email(%Account{} = account, opts) when is_list(opts) do
  with template when template in ~w(personal corporate)a <- Keyword.get(opts, :template) do
    values = Keyword.get(opts, :values, [])
    landing_url = %URI{} = Keyword.get(values, :landing_url, URI.parse("https://www.example.com/sign_in"))
    signature = Keyword.get(values, :signature, "Yours truly,\nACME.com")

    # email generation and sending goes here

    :ok
  else
    nil -> {:error, :missing_tempate}
    other when is_atom(other) -> {:error, :invalid_template_value}
    _ ->  {:error, :invalid_template_type}
  end
end

But that doesn't scale very well once the number of options grows: it gets pretty difficult to see what options are permitted, what their expected type is, and so on. Not to mention, this is going to get repetitive really fast.

Bringing In NimbleOptions

Let's try out NimbleOptions. We need to specify a schema that the options are expected to conform to:

@email_opts_schema [
  template: [
    required: true,
    type: {:in, ~w(personal corporate)a}
  ],
  values: [
    type: :keyword_list,
    default: [],
    keys: [
      landing_url: [
        type: {:struct, URI},
        default: URI.parse("https://www.example.com/sign_in")
      ],
      signature: [
        type: :string,
        default: "Yours truly,\nACME.com"
      ]
    ]
  ]
]

def email(%Account{} = account, opts) when is_list(opts) do
  with {:ok, validated_options} <- NimbleOptions.validate(opts, @email_opts_schema) do
    template = Keyword.fetch!(validated_options, :template)
    landing_url = validated_options[:values][:landing_url]
    signature = validated_options[:values][:landing_url]

    # email generation and sending goes here

    {:ok, validated_options}
  end
end

Do note we've specified default: [] in the values configuration: that way, an empty list will be used as the default, enabling the cascading of default child values such as landing_url to be filled in. Without it, a default landing_url won't be filled in if a values keyword isn't present at all.

Here it is in action:

iex> email(an_account, template: :personal)

{:ok,
 [
   values: [
     signature: "Yours truly,\nACME.com",
     landing_url: %URI{
       authority: "www.example.com",
       fragment: nil,
       host: "www.example.com",
       path: "/sign_in",
       port: 443,
       query: nil,
       scheme: "https",
       userinfo: nil
     }
   ],
   template: :personal
 ]}

iex> email(an_account, template: :personal, values: [signature: "With love,\nBob"])

{:ok,
 [
   template: :personal,
   values: [
     landing_url: %URI{
       authority: "www.example.com",
       fragment: nil,
       host: "www.example.com",
       path: "/sign_in",
       port: 443,
       query: nil,
       scheme: "https",
       userinfo: nil
     },
     signature: "With love,\nBob"
   ]
 ]}

As we can see, default values are being filled in when required, and they won't overwrite any provided values.

Validation also works out of the box:

iex> email(an_account, template: :boom)

{:error,
 %NimbleOptions.ValidationError{
   key: :template,
   keys_path: [],
   message: "invalid value for :template option: expected one of [:personal, :corporate], got: :boom",
   value: :boom
 }}

It will also verify that we're not passing in unexpected option keys, such as typos:

iex> email(an_account, template: :personal, values: [singature: "With love,\nBob"])

{:error,
 %NimbleOptions.ValidationError{
   key: [:singature],
   keys_path: [:values],
   message: "unknown options [:singature], valid options are: [:landing_url, :signature]",
   value: nil
 }}

Allowing Strings for `landing_page`

While having the landing_page as a URI ensures it at least looks right, having to provide it as a string is a bit of a pain. It would be nice to provide the option as a string, but here's how that currently behaves:

iex> email(an_account, template: :personal, values: [landing_url: "https://my.app"])

{:error,
 %NimbleOptions.ValidationError{
   key: :landing_url,
   keys_path: [:values],
   message: "invalid value for :landing_url option: expected URI, got: \"https://my.app\"",
   value: "https://my.app"
 }}

Luckily, there's an easy fix — we just have to tweak the type definition:

@email_opts_schema [
  template: [
    required: true,
    type: {:in, ~w(personal corporate)a}
  ],
  values: [
    type: :keyword_list,
    default: [],
    keys: [
      landing_url: [
        # improved type to accept strings
        type: {:or, [{:struct, URI}, :string]},
        default: URI.parse("https://www.example.com/sign_in")
      ],
      signature: [
        type: :string,
        default: "Yours truly,\nACME.com"
      ]
    ]
  ]
]

Strings are now accepted for the landing_page value:

iex> email(an_account, template: :personal, values: [landing_url: "https://my.app"])

{:ok,
 [
   template: :personal,
   values: [signature: "Yours truly,\nACME.com", landing_url: "https://my.app"]
 ]}

While a step in the right direction, there are now two minor issues:

Our code has to handle both string and URI data types (or we need to manually convert one to another after validation).
Leaving the value as a string doesn't indicate anything. In particular, there's no indication that the value is a valid URI "safe" to process.

Elixir's NimbleOptions To the Rescue!

Once again, NimbleOptions comes to our rescue: we can use a :custom data type that specifies a parser function. In our case, that parser function can simply be URI.new/1, as it already conforms to the expectation set by :custom (namely, returning {:ok, value} or {:error, message}):

@email_opts_schema [
  template: [
    required: true,
    type: {:in, ~w(personal corporate)a}
  ],
  values: [
    type: :keyword_list,
    default: [],
    keys: [
      landing_url: [
        # also accept strings, but parse them in %URI{}
        type: {:or, [{:struct, URI}, {:custom, URI, :new, []}]},
        default: URI.parse("https://www.example.com/sign_in")
      ],
      signature: [
        type: :string,
        default: "Yours truly,\nACME.com"
      ]
    ]
  ]
]

Check it out:

iex> email(an_account, template: :personal, values: [landing_url: "https://my.app"])

{:ok,
 [
   template: :personal,
   values: [
     signature: "Yours truly,\nACME.com",
     landing_url: %URI{
       authority: nil,
       fragment: nil,
       host: "my.app",
       path: nil,
       port: 443,
       query: nil,
       scheme: "https",
       userinfo: nil
     }
   ]
 ]}

We've now got the best of both worlds: convenient argument types and a standardized representation post-validation!

Wrapping Up

I hope you've enjoyed these two blog posts covering how you can keep bad data out of your Elixir applications while also making your code more expressive.

As we've seen, there are both "plain vanilla" Elixir techniques as well as support you can get from libraries like NimbleOptions to help us reject invalid data from being processed in our code.

By introducing these approaches to our modules, we'll provide more immediate feedback to callers and also make our programs more resilient when faced with unexpected data.

Hopefully, you've now got a few more tools in your belt to try out. Enjoy!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Validate Data in a Phoenix Application for Elixir

David Sulc — Tue, 17 Oct 2023 14:13:36 +0000

In this first part of a two-part series, we'll explore how to avoid bad data and validate data at the boundary of a Phoenix application.

We'll use a few techniques to ensure that bad data doesn't degrade our application.

In part two, we'll specifically focus on leveraging Ecto under the hood to cast data.

Let's dive in!

Say No to Bad Data in Elixir

Bad data must be dealt with immediately, or it will spread throughout your system and degrade other data. Given how difficult and time-consuming it is to fix data issues, preventing bad data from entering your system is well worth the effort.

Here is what the official Elixir documentation has to say on the matter:

[...] when you don't validate the values at the boundary, the internals of your library are never quite sure which kind of values they are working with.

This advice does not only apply to libraries, but to any Elixir code. Every time you receive multiple options or work with external data, you should validate the data at the boundary and convert it to structured data. For example, if you provide a GenServer that can be started with multiple options, you want to validate those options when the server starts and rely only on structured data throughout the process life cycle. Similarly, if a database or a socket gives you a map of strings, after you receive the data, you should validate it and potentially convert it to a struct or a map of atoms.

But what is a boundary, and where does it live? The answer, dear reader, is mostly up to you. In a sense, boundaries exist wherever your functions accept unknown or unsafe data, to later be processed (once some assumptions about the data are made).

An Example of a Boundary in a Phoenix App

A typical example of a boundary in a Phoenix app is the boundary between the web layer and the business logic: parameters come in as JSON values and are parsed into maps with string keys. But a client is free to send any sort of data within the JSON payload: there could be incorrect keys, invalid values, extra key values, and so on.

Since you don't want to deal with this unwieldy data in every location within your app, the recommendation is to process this untrusted data in a single location and convert it into a well-known shape with validated content.

This is typically done in a Phoenix Context, where Ecto is leveraged under the hood to cast data (we'll explore this in our next post). Once this conversion has taken place, your domain logic can trust the content of the request payload without performing the same verification again.

More generally speaking, boundaries will typically crop up anywhere in your software where you accept some data of unknown quality and process it internally (typically over several iterations). Examples include:

GenServers - where data is sent and processed into state changes.
Web requests - which make their way to domain logic and database tables.
Console input - which needs to be processed into arguments provided to a CLI application.

Additionally, boundaries can also show up when data has different meaning in different contexts (often called bounded contexts): a Customer might, for example, have a billing address, and a User might have a username, but both would refer to the same person in the world. In effect, boundaries are everywhere, which makes these techniques very handy when they're skillfully deployed.

Let's now check out a few techniques to prevent bad data from degrading our application.

Pattern Matching and Guards in Elixir

At the very local level, we can leverage pattern matching and guards to ensure we're always working with the data we expect.

This type of defensive code is beneficial anywhere you add it, be it within domain code (such as a Phoenix Context), the interface layer (in a Phoenix Controller, for example), or even deep within a free-standing function in a script.

Case Clauses

Case clauses can be very helpful in ensuring we explicitly list the data we agree to handle, particularly if there's no catch-all clause:

case Account.setup(...) do
    {:ok, %Account{suspended: true}} -> ...
    {:ok, %Account{initialized: true}} -> ...
    {:error, ...} -> ...
end

The above code clearly communicates its intent: the only expected outcomes of the setup function are an account (that is either initialized or suspended) or an error. Further, the case where an account is suspended "supersedes" an initialized account since the pattern match comes first.

Any other return value isn't expected and is therefore considered a bug: we explicitly don't handle those other cases, as we wouldn't know how to (if we did, they'd have their own case clause as shown above). In the face of unexpected data, it's safest to crash so that the OTP system can restart the process from a clean "known good" state rather than propagate dirty data throughout a system.

Pattern Matching in Function Heads

Pattern matching in function heads is a great way to not only ensure that data conforms to your expectations, but also to communicate your intent.

Contrast:

defp suspend(%Account{} = account)

With:

defp suspend(%{suspended: _} = account)

These are very different functions: in the first, it's clear that an Account is expected, so the account variable can be safely used in functions expecting an Account instance, whereas those assurances can't be made for the second version. In the second example, we're relying on the presence of the suspended attribute and a variable name to infer the context. That's playing with fire: there's nothing preventing someone from calling the function with a %User{} struct (assuming it also has a suspended attribute).

That said, while matching in function heads is helpful and convenient, make sure you're not muddying the waters for the sake of convenience. Let's use the following example to explore the subject, even though it's not directly related to validation:

def handle_call(:checkout, %{workers: [h | t], monitors: monitors}) do
  # ...
end

def handle_call(:checkout, %{workers: [], idle_overflow: [h | t]}) do
  # ...
end

def handle_call(:checkout, %{workers: [], idle_overflow: [],
    overflow: overflow, overflow_max: max, worker_sup: sup,
    spec: spec, monitors: monitors})
  when overflow < max do
    # ...
end

def handle_call(:checkout, %{workers: [], idle_overflow: [],
  overflow: overflow, overflow_max: max, waiting: waiting}) do
# ...
end

A lot of matching is going on here, but how much is to differentiate the function heads, and how much is for convenience (i.e., binding for later use)?

By leaving only the matches that differentiate the heads and moving other bindings to the function bodies, we can improve readability:

def handle_call(:checkout, %{workers: [_|_]} = state) do
  %{monitors: monitors} = state
  # ...
end

def handle_call(:checkout, %{workers: [], idle_overflow: [_]}) do
  # ...
end

def handle_call(:checkout, %{workers: [], idle_overflow: [],
    overflow: overflow, overflow_max: max} = state) when overflow < max do
  %{worker_sup: sup, spec: spec, monitors: monitors} = state
  # ...
end

def handle_call(:checkout, state) do
  %{workers: [], idle_overflow: [], overflow: overflow,
  overflow_max: max, waiting: waiting} = state
  # ...
end

After refactoring the above (non-validation related) example code, it's now more obvious that:

The first function matches when there are workers in the list.
The second function matches when there are no workers, but there are values in the idle_overflow list.
The third function matches when there are no workers or idle_overflow, but the overflow hasn't reached its max level yet.
The fourth function is for the remaining case.

You'll also note that we've gone somewhat overboard on matching here, such as matching on empty worker lists, even though the first function would match in that case. This approach is a bit of defensive programming, as the code will stay functional if:

The function clauses are reordered (e.g., during a refactoring), and their ordering importance is overlooked.
Additional matching is added to the first clause, changing its match semantics. Keeping each function clause responsible for declaring the context it expects makes the code more resilient to change.

Avoiding excessive matching in function heads is also the approach recommended by José Valim:

FWIW, I tend to use this rule: if the key is necessary when matching the pattern, keep it in the pattern, otherwise, move it to the body. So I end up with code like this:

def some_fun(%{field1: :value} = struct) do
  %{field2: value2, field3: value3} = struct
  ...
end

Guard Clauses

Guard clauses should be used whenever the definition of "what data is valid" can be tightened to make your code safer by design and to prevent processing unexpected/invalid data:

def annotate(%Account{} = account, annotation) when is_binary(annotation)

case account do
  %Account{payment_method: payment_method} when not is_nil(payment_method) -> ...
end

This same approach can naturally also be used for with and cond statements.

Custom Guards

Declaring custom guards effectively communicates intent and prevents bad data, significantly improving your code.

First, you must declare the guard in a separate module located outside of the module(s) where you want to use the guard.

defmodule Account.Guards do
  defguard is_suspended(account) when is_struct(account, Account) and account.suspended
end

Now our code is even more expressive:

import Account.Guards

case fetch_account(...) do
    %Account{} = account when is_suspended(account) -> ...
    ...
end

And that's it!

Next Up

In this post, we looked at how to validate data at the boundary of an Elixir application. We also used a few pattern matching and guard clause techniques to reject bad data.

We've already covered a lot of ground: let's give it a rest for now. In the next article, we'll explore a few more options to ensure the data we work with remains squeaky clean, using Ecto.

See you then!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Parser Combinators in Elixir: A Deeper Dive

David Sulc — Tue, 22 Nov 2022 12:29:38 +0000

In our last post, we wrote a basic parser for phone numbers using Elixir. It was a bit simplistic since it didn't really respect the format phone numbers are expected to have, but it was a great start.

We'll now improve the parser to ensure we only accept phone numbers that fit the spec and make our return type an instance of structured data.

Let's dive straight in!

Formatting Parse Results in Elixir

At this point, we've got rudimentary parsers processing the main parts of a local phone number. Let's now make the parse results easier to put into a data structure.

Right now, our parser output looks like this:

iex(1)>  Demo.PhoneNumber.Parser.parse("020-42 84 105")
{:ok, ["0", "2", "0", "-", "4", "2", " ", "8", "4", " ", "1", "0", "5"], "", %{},
 {1, 0}, 13}

In passing, you might note that the "remaining unparsed" value (third element in the tuple) is the "" empty string, meaning our parser successfully consumes the entire input string.

For starters, let's remove noise from the parse result: we're not interested in the trunk value (it's always "0" and therefore provides no information) or the separator. Nor, for that matter, are we interested in the spaces in the subscriber number. Let's wrap each of those in an ignore/2 so they get dropped from the output:

# lib/demo/phone_number/parser.ex

defmodule Demo.PhoneNumber.Parser do
  @moduledoc false

  import NimbleParsec

  # ...

  area_code =
    ignore(trunk_prefix)
    |> times(digit, 2)

  separator = choice([string("-"), string(" ")])

  subscriber_number =
    choice([digit, ignore(string(" "))])
    |> times(min: 1)

  dutch_phone_number =
    area_code
    |> concat(ignore(separator))
    |> concat(subscriber_number)
    |> eos()

  defparsec(:parse, dutch_phone_number)
end

This indeed removes the trunk prefix and separator from the parse result (given they're now both ignored), but it isn't exactly straightforward to understand since all the parts are just dumped into an array:

iex(1)> Demo.PhoneNumber.Parser.parse("020-42 84 105")
{:ok, ["2", "0", "4", "2", "8", "4", "1", "0", "5"], "", %{}, {1, 0}, 13}

We should tag the results to identify which ones come from particular parsers. That sounds like a perfect fit for tag/2:

# lib/demo/phone_number/parser.ex

defmodule Demo.PhoneNumber.Parser do
  @moduledoc false

  import NimbleParsec

  # ...

  area_code =
    ignore(trunk_prefix)
    |> times(digit, 2)
    |> tag(:area_code)

  # ...

  subscriber_number =
    choice([digit, ignore(string(" "))])
    |> times(min: 1)
    |> tag(:subscriber_number)

  # ...

  defparsec(:parse, dutch_phone_number)
end

We're now closer to our goal:

iex(1)> Demo.PhoneNumber.Parser.parse("020-42 84 105")
{:ok,
 [area_code: ["2", "0"], subscriber_number: ["4", "2", "8", "4", "1", "0", "5"]],
 "", %{}, {1, 0}, 13}

These values are still a bit messy though: we want a single string value instead of a collection of digit strings. Essentially, we want to Enum.join(digits_strings, ""), which is where reduce/3 enters the picture:

# lib/demo/phone_number/parser.ex

defmodule Demo.PhoneNumber.Parser do
  @moduledoc false

  import NimbleParsec

  # ...

  area_code =
    ignore(trunk_prefix)
    |> times(digit, 2)
    |> reduce({Enum, :join, [""]})
    |> tag(:area_code)

  # ...

  subscriber_number =
    choice([digit, ignore(string(" "))])
    |> times(min: 1)
    |> reduce({Enum, :join, [""]})
    |> tag(:subscriber_number)

  # ...

  defparsec(:parse, dutch_phone_number)
end

We're steadily moving towards our goal of having parse results that are nicely formatted and easy to convert into a data structure:

iex(1)> Demo.PhoneNumber.Parser.parse("020-42 84 105")
{:ok, [area_code: ["20"], subscriber_number: ["4284105"]], "", %{}, {1, 0}, 13}

We still need a little tweak: instead of the tagged values containing an array of a single element, we want to just have the element directly. That's easy. We'll just replace our use of tag/3 with unwrap_and_tag/3:

# lib/demo/phone_number/parser.ex

defmodule Demo.PhoneNumber.Parser do
  @moduledoc false

  import NimbleParsec

  # ...

  area_code =
    ignore(trunk_prefix)
    |> times(digit, 2)
    |> reduce({Enum, :join, [""]})
    |> unwrap_and_tag(:area_code)

  # ...

  subscriber_number =
    choice([digit, ignore(string(" "))])
    |> times(min: 1)
    |> reduce({Enum, :join, [""]})
    |> unwrap_and_tag(:subscriber_number)

  # ...

  defparsec(:parse, dutch_phone_number)
end

Check it out:

iex(1)> Demo.PhoneNumber.Parser.parse("020-42 84 105")
{:ok, [area_code: "20", subscriber_number: "4284105"], "", %{}, {1, 0}, 13}

Creating a Struct

Let's create a PhoneNumber struct to hold our parsed data:

# lib/demo/phone_number.ex

defmodule Demo.PhoneNumber do
  alias Demo.PhoneNumber.Parser

  defstruct [:country_code, :area_code, :subscriber_number]

  def new(string) when is_binary(string) do
    case Parser.parse(string) do
      {:ok, results, "", _, _, _} -> {:ok, struct!(__MODULE__, results)}
      {:error, reason, _rest, _, _, _} -> {:error, reason}
    end
  end
end

What do we have here? After defining a struct with defstruct/1, we define a new/1 function that will accept a string input. Then it will run our fancy parser: if the parser is unable to process the string, it will return {:error, reason}.

If, on the other hand, our parser is able to process the string, we know that results will be a keyword list of the parsed components found in the phone number (such as [area_code: "20", subscriber_number: "4284105"] as shown above). And we know this will indeed be a keyword list, because we've done so much work getting our parser to format and tag the output in this manner.

All of this hard work is finally paying off: since our keyword list keys are the same as the struct's, we can create a new struct instance by simply calling struct!/2 with our parse results!

We now have the public API to parse phone numbers into a data structure. Time for a quick test in iex -S mix:

iex(1)> Demo.PhoneNumber.new("020-42 84 105")
{:ok,
 %Demo.PhoneNumber{
   area_code: "20",
   country_code: nil,
   subscriber_number: "4284105"
 }}

iex(2)> Demo.PhoneNumber.new("foobar")
{:error, "expected ..."}

Writing Custom Combinators Using Elixir

We've made great strides so far: our parser now returns a data structure! This will be extremely helpful down the line when the data it contains needs to be validated or otherwise processed.

There's still some work ahead of us, though: aside from the fact that it doesn't yet parse international numbers, it also doesn't check for number lengths. Area codes may have 2 or 3 digits, while subscriber numbers may have 7 or 6 digits respectively.

To assist us in achieving this goal, we'll be writing more advanced and powerful combinators. As you may recall from the previous article, combinators are parsers that can be combined into bigger and better parsers. We'll be using NimbleParsec to define a complex parser from smaller and simpler ones.

Since we want the parser behavior to vary according to provided arguments, we need to define some functions. But since NimbleParsec's defparsec/3 is executed during compilation, we can't invoke a function defined in the same module.

To work around this, we'll define a Helpers sub-module and our configurable parsers there:

# lib/demo/phone_number/parser/helpers.ex

defmodule Demo.PhoneNumber.Parser.Helpers do
  @moduledoc false

  import NimbleParsec

  @doc """
  Parses a single digit.
  """
  def digit(combinator \\ empty()) do
    combinator
    |> utf8_string([?0..?9], 1)
  end

  @doc """
  Parses `count` digits.

  For example, `digits(3)` would parse 3 digits.
  """
  def digits(combinator \\ empty(), count) do
    combinator
    |> duplicate(digit(), count)
    |> reduce({Enum, :join, [""]})
  end

  @doc """
  Parses a phone number area code.

  Since phone number area codes can only be 2 or 3 digits in length,
  only `2` and `3` are valid values for the `length` parameter.
  """
  def area_code(combinator \\ empty(), length) when length in [2, 3] do
    combinator
    |> digits(length)
  end
end

As you can see, we use combinator as the first function argument, defaulting to NimbleParsec's empty() combinator if none is provided. This enables us to compose combinators where appropriate, without having to use concat/2 as a crutch.

Updating the Parser with Functions

Let's update our parser to use these new functions:

# lib/demo/phone_number/parser.ex

defmodule Demo.PhoneNumber.Parser do
  @moduledoc false

  import NimbleParsec
  import Demo.PhoneNumber.Parser.Helpers

  # ...

  area_code_with_trunk_prefix =
    ignore(trunk_prefix)
    |> area_code(2)
    |> unwrap_and_tag(:area_code)

  # ...

  subscriber_number =
    choice([digit(), ignore(string(" "))])
    |> times(min: 1)
    |> reduce({Enum, :join, [""]})
    |> unwrap_and_tag(:subscriber_number)

  dutch_phone_number =
    area_code_with_trunk_prefix
    |> concat(ignore(separator))
    |> concat(subscriber_number)
    |> eos()

  defparsec(:parse, dutch_phone_number)
end

We've renamed area_code to area_code_with_trunk_prefix so it won't get confused with the area_code imported from the new Helpers module. Also take note that digit has become digit(), as it's now a function rather than a combinator.

Our code still works, but we've yet to fix the subscriber number parser: if the area code is 2 digits, then the subscriber number should be 7 digits. Conversely, if the area code is 3 digits long, the subscriber should contain only 6 digits. So let's now refactor to have a variable-length subscriber_number combinator along with a configurable local_number combinator:

# lib/demo/phone_number/parser/helpers.ex

defmodule Demo.PhoneNumber.Parser.Helpers do
  @moduledoc false

  import NimbleParsec

  # ... various combinator definitions ...

  def subscriber_number(combinator \\ empty(), length) when length in [6, 7] do
    combinator
    |> digit()
    |> times(ignore(optional(string(" "))) |> digit(), length - 1)
    |> ignore(optional(string(" ")))
    |> reduce({Enum, :join, [""]})
  end

  def local_number(combinator \\ empty(), area_code_length, subscriber_number_length) do
    separator = choice([string("-"), string(" ")])

    combinator
    |> area_code(area_code_length)
    |> unwrap_and_tag(:area_code)
    |> concat(ignore(separator))
    |> concat(subscriber_number(subscriber_number_length) |> unwrap_and_tag(:subscriber_number))
  end
end

There's quite a bit going on here, so let's take a look.

Side note: if you ever get stuck when writing sub-parsers, you can create entry points to only test that portion of your code. For example, by adding defparsec(:subscriber_number, subscriber_number(7)) at the bottom of Demo.PhoneNumber.Parser, you can run iex -S mix and test the parsing of subscriber numbers with Demo.PhoneNumber.Parser.subscriber_number("65 23 125").

The subscriber number is defined as (mandatorily) starting with a digit, followed by an optional space and then another digit, with the latter grouping repeating one time less than the desired length provided as an argument. That means that we'll end up with as many actual digits as requested via length (along with a variable number of spaces mixed in). We ignore the whitespace so it won't show up in the results, and then combine all parsed digits into a single string.

We then define a local_number combinator that will accept two lengths: one for the area code, and the other for the subscriber portion. We combine these with an ignored separator, and tag each segment. Since we know each area_code and subscriber_number combinator will yield a single result, we can use unwrap_and_tag/3 to extract the parsed result from the returned array and tag it with the provided atom.

As you may have seen, our subscriber_number combinator accepts a previous combinator as the first argument: we could have chained it directly to the previous combinator.

Why, then, are we using concat/2? Because it will, in effect, "separate" the combinator we're interested in, so that unwrap_and_tag/3 will apply only to the subscriber_number sub-parser rather than to the combinator result of subscriber_number and its prior combinator.

In addition, the Enum.join step we have in subscriber_number would run into problems without concat/2. It attempts to join the result of the previous combinator (which will include area_code: "20"), so it will crash (as Enum.join won't know how to process a tuple). Keep this in mind if you run into unexpected failures when writing your own combinators.

Rewriting the Main Parser in Elixir

With these new custom combinators available, let's go back and rewrite our main parser:

# lib/demo/phone_number/parser.ex

defmodule Demo.PhoneNumber.Parser do
  @moduledoc false

  import NimbleParsec
  import Demo.PhoneNumber.Parser.Helpers

  trunk_prefix = string("0")

  local_portion =
    choice([
      local_number(2, 7),
      local_number(3, 6)
    ])

  dutch_phone_number =
    ignore(trunk_prefix)
    |> concat(local_portion)
    |> eos()

  defparsec(:parse, dutch_phone_number)
end

Nice and readable: we can easily tell that the local portion of the number is always expected to be 9 digits long, with a longer area code resulting in a short subscriber number.

We're almost done! We just need to handle the international prefix:

# lib/demo/phone_number/parser.ex

defmodule Demo.PhoneNumber.Parser do
  @moduledoc false

  import NimbleParsec
  import Demo.PhoneNumber.Parser.Helpers

  # ...

  international_prefix =
    ignore(string("+"))
    |> digits(2)
    |> ignore(string(" "))
    |> map({String, :to_integer, []})
    |> unwrap_and_tag(:country_code)

  dutch_phone_number =
    choice([international_prefix, ignore(trunk_prefix)])
    |> concat(local_portion)
    |> eos()

  defparsec(:parse, dutch_phone_number)
end

We simply define the international_prefix combinator using functions that are now familiar to us, and then use map/3 to ensure the result is an integer: we may want to compare it to valid ISO country codes later on.

With that in place, it's simply a matter of telling our parser that we're either:

In the international case (where our parser does not expect the trunk prefix to be present)
In a national context, where the trunk prefix should indeed be part of the phone number (but removed from the output)

Voilà, we've got our phone number parser defined in a way that's understandable and easy to maintain. Take a look:

iex(1)> Demo.PhoneNumber.new("+31 20-42 84 105")
{:ok,
 %Demo.PhoneNumber{
   area_code: "20",
   country_code: 31,
   subscriber_number: "4284105"
 }}

Next Steps: Play Around with Parser Combinators

The best way to get a good grasp on how parser combinators work is to experiment with them.

Want to give it a spin but can't think of an example? Try extending our code to parse other cases.

Write a parser for ISO 8601 datetimes such as 2019-11-14T00:55:31.820Z. And if you want to get really crazy, you can extend it to handle durations and time intervals.

You can also try your hand at writing a recursive parser: there's an example in NimbleParsec's parsec/2 documentation to get you started.

As touched upon in the introduction to this article, NimbleParsec is more of a parser generator: take a look at Combine and try to convert the above code to make use of Combine, or come up with your own example.

The way Combine works is that you combine parser functions, and obtain a new parser function. Parser functions can then be applied via Combine.parse/3. Here's a toy example:

# the public parser
def username_and_id(string) do
  Combine.parse(string, username_and_id_parser())
end

# these are internal parsers for the combinator

# a "username and id" is simply a sequence of
# username, space, id
defp username_and_id_parser() do
  sequence([
    username(),
    ignore(string(" ")),
    id()
  ])
end

# a username consists of word characters
defp username(), do: word()

# an id is an integer
defp id(), do: integer()

Calling this function will return the (non-ignored) parse result:

iex> username_and_id("foobar 123")
[["foobar", 123]]

Ready for the big leagues? Write your own parser combinator from scratch to fully understand how everything works under the hood. Here's an example parsing a subset of SQL, written by Saša Jurić.

Wrapping Up

In the first part of this series, we briefly defined parser combinators before building a simple one in Elixir.

In this second and final part, we dived deeper into the parser combinator that we built last time and improved it. We also gave you some ideas for ways to further explore parser combinators.

Happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Parser Combinators in Elixir: Taming Semi-Structured Text

David Sulc — Tue, 25 Oct 2022 13:58:30 +0000

The need to manipulate strings comes up quite often, whether it's to validate user-provided values or transform text into structured data that can be used programmatically.

Most often, we'll reach for regular expressions to accomplish this task, but sometimes there's a better solution to the problem: parser combinators. In this two-part article, we'll explore how they work.

Before moving on, let's define what 'parsing' is:

A parser is a software component that takes input data (frequently text) and builds a data structure.

Source: Wikipedia.

In other words, when talking about transforming text into structured data, we essentially mean parsing.

Let's get into it!

Parser Combinators: The What, How, and Why

So what are parser combinators? Well, as their name implies, they are parsers that can be combined — to make bigger, better parsers.

In effect, parser combinators enable the writing of complex parsers from simpler ones. You're less likely to make mistakes, and the parsers are a lot easier to maintain due to better readability.

In contrast, regular expressions are often a poor choice for non-trivial parsing. Writing them is error-prone, they're challenging to maintain, and their format doesn't lend itself well to documentation/explanation.

To illustrate working with parser combinators, we'll work on transforming a string representation of a Dutch phone number into a canonical representation. This can prove useful, as a web form can capture a user's phone number to later be used programmatically, for example.

The "Parser" and "Combinator" Bits: NimbleParsec

We'll be using NimbleParsec as the resulting parsers are more efficient, but other libraries such as Combine also exist and work well.

As an aside, note that it's been pointed out that NimbleParsec is more akin to a parser generator than a parser combinator: that's true, but the difference won't matter when learning how to make use of parser generators.

Parser Examples in Elixir

As mentioned before, parsing is the act of taking unstructured data (typically strings) and transforming them into structured data.

We've got some examples right in Elixir.

We can turn a string containing a URI into a data structure via URI.new/1:

iex> URI.new("https://elixir-lang.org/")
{:ok, %URI{
  fragment: nil,
  host: "elixir-lang.org",
  path: "/",
  port: 443,
  query: nil,
  scheme: "https",
  userinfo: nil
}}

iex> URI.new("/invalid_greater_than_in_path/>")
{:error, ">"}

Similarly, we can convert a string representation of an integer into an actual integer by using Integer.parse/2:

iex> Integer.parse("34")
{34, ""}

iex> Integer.parse("34.5")
{34, ".5"}

iex> Integer.parse("three")
:error

Each of these examples will accept a string and either return a parsed result (possibly with "leftover" string content, as in the Integer.parse/2 example) or an error indication.

Combinator Examples in Elixir

What if we had lists indicating how many visitors came from URIs (formatted as "URI & count" pairs)? It would be nice to have a single function to do the heavy lifting for us:

iex> parse_visit_count("https://elixir-lang.org/ 123")
{:ok, %{
  referrer: %URI{
    fragment: nil,
    host: "elixir-lang.org",
    path: "/",
    port: 443,
    query: nil,
    scheme: "https",
    userinfo: nil
  },
  visits: 123
}}

Easy, I hear you say — just do something like:

def parse_visit_count(string) do
  with [uri_string, count_string | []] <- string |> String.trim() |> String.split(),
        {:ok, uri} <- URI.new(uri_string),
        {count, ""} <- Integer.parse(count_string) do
    {:ok, %{referrer: uri, visits: count}}
  else
    _ -> :error
  end
end

That's indeed pretty straightforward. But what if there are several URI/visits pairs per line? Then the logic is already much more involved. And what if these URI/visits pairs are in reverse order (visits first, then URI)?

Using a parser combinator approach, we look at the problem differently. We've got a parser for URIs, and another one for integers. Let's write a parser for white space, then combine those three parsers into a single one for our specific use case. In pseudo-code, it would look like this:

def parse_visit_count(...) do
   ...
   |> uri_parser()
   |> whitespace_parser()
   |> integer_parser()
end

And since we're now in combinator-land, we could then use parse_visit_count and combine it with other parsers to step up the complexity. Neat, right?

Code Walkthrough: Our Parser Combinator Example

Let's work through a concrete example of using a parser combinator: we'll parse a phone number into a data structure.

The phone number we'll use for our examples is one from the Netherlands: +31 20 42 84 105.

According to Wikipedia, this number can be formatted in various ways (note we'll only be covering a subset of the possibilities within this article):

The area code ('A') is commonly separated with a dash ('-') and sometimes a space from the subscriber's number ('B').
The length of the area code for landlines is either 2 or 3 digits, depending on the population density of the area. This leaves 7 or 6 digits for the subscriber's number, resulting in a format of either 0AA-BBBBBBB or 0AAA-BBBBBB. [...]
The trunk prefix '0' is dropped when prefixed by the country code: +31 AA BBBBBBBB, [...], etcetera. [...]

In other words, this phone number can also be formatted as (among others):

+31 20 4284105
+31 20-42 84 105
020-42 84 105
020-4284105
020 42 84 105

Deconstructing the Problem

From a quick look at the formats above, we can tell that the main "chunks" we need to extract are:

Country code - the "+31" prefix, which may be absent
Area code - the "020", "(0)20", or "20" value
Subscriber number - the remaining seven digits, which may be separated by spaces

Getting Started

We'll use NimbleParsec, so let's go ahead and create a new project with mix new demo and add the dependency in mix.exs:

# in mix.exs

defp deps do
  [
    {:nimble_parsec, "~> 1.2"}
  ]
end

Run mix deps.get to ensure everything's ready for us. With that in place, let's start writing our phone number parser, which we'll do within its own module:

# lib/demo/phone_number/parser.ex

defmodule Demo.PhoneNumber.Parser do
  @moduledoc false

  import NimbleParsec

  dutch_phone_number = string("+31 20 42 84 105")

  defparsec(:parse, dutch_phone_number)
end

Within this PhoneNumber.Parser module, we need to import NimbleParsec so we can call on its functions (such as string) more conveniently. Then, we define our parser. Right now, that consists of only parsing the exact phone number string via NimbleParsec's string/2 function (hence why the phone number value is hardcoded). And last but not least, we call defparsec/3 to "finalize" the parser and make it executable.

Let's give it a spin! Open an IEx session by typing iex -S mix in the project's root folder from within a terminal. We can then try our parser:

iex(1)> Demo.PhoneNumber.Parser.parse("+31 20 42 84 105")
{:ok, ["+31 20 42 84 105"], "", %{}, {1, 0}, 16}

iex(2)> Demo.PhoneNumber.Parser.parse("foobar")
{:error, "expected string \"+31 20 42 84 105\"", "foobar", %{}, {1, 0}, 0}

The result is a tuple containing six terms. The first is the :ok or :error tag indicating whether the provided string could be parsed.

The second value is (in the :ok case) a list of the parsed information, or (in the :error case) the error reason.

The remaining values can be ignored as we won't get into them in this article.

Parsing Local Numbers

Now that we've created a parser for an exact string, let's dig deeper into NimbleParsec's functionality by expanding the variety of phone numbers our parser will be able to process.

Local phone numbers look like this:

020-42 84 105
020-4284105
020 42 84 105

There's an area code, then a separator which can be a space or a dash. This is followed by the subscriber number, which can contain spaces to make it easier to read.

Note also that the "area code" portion consists of the trunk prefix (which is "0") followed by two or three numbers indicating the area itself.

So conceptually, we're looking at writing a parser that looks like area_code <> separator <> subscriber_number.

Let's start with the area code:

# lib/demo/phone_number/parser.ex

defmodule Demo.PhoneNumber.Parser do
  @moduledoc false

  import NimbleParsec

  trunk_prefix = string("0")

  area_code =
    trunk_prefix
    |> string("20")

  dutch_phone_number = area_code

  defparsec(:parse, dutch_phone_number)
end

Let's give it a whirl in iex -S mix:

iex(1)> Demo.PhoneNumber.Parser.parse("020-42 84 105")
{:ok, ["0", "20"], "-42 84 105", %{}, {1, 0}, 3}

We can see our parser is properly extracting out the area code: it's being returned as ["0", "20"] because the area_code combinator is composed of trunk_prefix (yielding "0"), followed by string("20") (logically yielding "20"). The remainder of the input string is left unparsed (as "-42 84 105"). Success!

So what's going on? We create a trunk_prefix parser that will match the specific "0" string by using string/2.

Then, we define an area_code parser that matches this trunk_prefix followed by the "20" string. And finally, we define the parse function as the parser specified by the area_code combinator by calling defparsec/3.

We're on a roll, so let's add to our current parser so that it also captures the separator. This separator can be either a dash or a space. It sounds like a perfect match for choice/3, which will apply the first parser it finds that can parse the input.

# lib/demo/phone_number/parser.ex

defmodule Demo.PhoneNumber.Parser do
  @moduledoc false

  import NimbleParsec

  # ...

  separator = choice([string("-"), string(" ")])

  dutch_phone_number =
    area_code
    |> concat(separator)

  defparsec(:parse, dutch_phone_number)
end

In choice([string("-"), string(" ")]), the choice([...]) combinator will attempt to parse a "-" string. If it's not able to, it will attempt parsing a " " string. Only if both of these options fail, does the entire choice combinator fail. Conversely, if the first string("-") option succeeds, none of the subsequent parsers provided to choice will be tested.

You'll notice that we weren't able to simply pipe the two area_code and separator combinators into each other: separator doesn't accept an argument, so we can't pass in area_code. NimbleParsec does, however, make it easy to concatenate two combinators with concat/2, so that's exactly how we plug these two functions into one another. Let's check things still work as expected:

# in `iex -S mix`

iex(1)> Demo.PhoneNumber.Parser.parse("020-42 84 105")
{:ok, ["0", "20", "-"], "42 84 105", %{}, {1, 0}, 4}

iex(2)> Demo.PhoneNumber.Parser.parse("020 42 84 105")
{:ok, ["0", "20", " "], "42 84 105", %{}, {1, 0}, 4}

iex(3)> Demo.PhoneNumber.Parser.parse("020/42 84 105")
{:error, "expected string \"-\" or string \" \"", "/42 84 105", %{}, {1, 0}, 3}

Great! Both the dash and space separators are getting picked up, while the invalid slash yields an error.

Let's now write an overly accepting parser that we can come back to and tighten down: we want to capture all digit and space characters until "end of string". To do so, we'll use utf8_string/3 to parse digits, times/3 to get us repetition, and finally, eos/1 to represent "end of string". It looks like this:

# lib/demo/phone_number/parser.ex

defmodule Demo.PhoneNumber.Parser do
  @moduledoc false

  import NimbleParsec

  # ...

  digit = utf8_string([?0..?9], 1)

  area_code =
    trunk_prefix
    |> times(digit, 2)

  # ...

  subscriber_number =
    choice([digit, string(" ")])
    |> times(min: 1)

  dutch_phone_number =
    area_code
    |> concat(separator)
    |> concat(subscriber_number)
    |> eos()

  defparsec(:parse, dutch_phone_number)
end

We define our digit combinator as a UTF8 string that contains values in the 0-9 range and is of length 1: in other words, it's a string of a single digit.

Note we've also improved the area_code combinator: instead of matching only the specific "20" area code, it will match any two digits. The side effect is that the parse results will be slightly different: ["0", "2", "0", "-", ...] instead of ["0", "20", "-", ...].

Finally, since eos() has an optional combinator argument, there's no need to wrap it with a concat() call within the pipeline.

Time for a Short Intermission

We've covered a decent amount of ground so far. Let's take a break and let all of that sink in.

Our parser is starting to take shape, but it's still not very good. We'll improve it in the next installment where we'll investigate writing custom parsers, among other topics.

Until next time, happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

DEV Community: David Sulc

Validating Data in Elixir: Using Ecto and NimbleOptions

Using Ecto

Schemaless Changesets in Ecto

Applying This To Phoenix Forms

Embedded Schemas in Ecto

Validating Options

The NimbleOptions Library for Elixir

Bringing In NimbleOptions

Allowing Strings for landing_page

Elixir's NimbleOptions To the Rescue!

Wrapping Up

Validate Data in a Phoenix Application for Elixir

Say No to Bad Data in Elixir

An Example of a Boundary in a Phoenix App

Pattern Matching and Guards in Elixir

Case Clauses

Pattern Matching in Function Heads

Guard Clauses

Custom Guards

Next Up

Parser Combinators in Elixir: A Deeper Dive

Formatting Parse Results in Elixir

Creating a Struct

Writing Custom Combinators Using Elixir

Updating the Parser with Functions

Rewriting the Main Parser in Elixir

Next Steps: Play Around with Parser Combinators

Wrapping Up

Parser Combinators in Elixir: Taming Semi-Structured Text

Parser Combinators: The What, How, and Why

The "Parser" and "Combinator" Bits: NimbleParsec

Parser Examples in Elixir

Combinator Examples in Elixir

Code Walkthrough: Our Parser Combinator Example

Deconstructing the Problem

Getting Started

Parsing Local Numbers

Time for a Short Intermission

Allowing Strings for `landing_page`