DEV Community: Mathias Polligkeit

Sharing secrets with the 1Password CLI

Mathias Polligkeit — Fri, 05 Jun 2020 19:54:24 +0000

Sometimes you need secrets in your local dev environment, be it a token for an api or a client secret for an OAuth2 provider. But of course you cannot commit those secrets to the Git repository. So whenever a new secret is added or a new member joins the team, you will need to have a way to share those secrets.

Often times it is badly documented which secrets you need and how to obtain them, so that new team members need to ask around, and the old team members need to remember where to find them, and they will likely share them in a direct Slack message. There must be a better way.

If your team already uses 1Password, you can make use of the 1Password CLI. Let’s see how you can use it in a shell script that you can safely commit to our repo.

Set up

Install the 1password cli as described in the Getting Started guide. As described there, you need to sign in once after the installation with:

op signin example.1password.com someone@example.com

After that, you can just sign in with:

# Fish
eval (op signin example)

# Bash
eval $(op signin example)

You will need to do this before running the example script below. You will stay signed in for 30 minutes.

If you have multiple 1Password accounts, you can specify the account to use with the --account flag.

Basics

You can get the item with the name SomeItem from the vault TeamVault with:

op get item SomeItem --vault TeamVault

The output will be raw json, but you can use jq to bring it into a more usable form. To pretty print the output, just run:

op get item SomeItem --vault TeamVault | jq

In general, items can be referenced by their display name (overview.title) or by their uuid (uuid).

The fields (username, password etc.) can be found under the details key. Unfortunately, the output of the 1Password CLI is not very consistent. For login items, the details are an object like this:

{
  "details": {
    "fields": [
      {
        "name": "password",
        "type": "P",
        "value": "hk0ikb7ki0ae7k07bk"
      },
      {
        "designation": "username",
        "name": "username",
        "type": "T",
        "value": "mail@example.com"
      }
    ]
  }
}

A password item has details like this:

{
  "details": {
    "password": "somepassword",
    "sections": [
      {
        "name": "linked items",
        "title": "Related Items"
      },
      {
        "fields": [
          {
            "k": "string",
            "n": "HWGKWG86PGWPHKGWP67PKW",
            "t": "iaeui",
            "v": "iaeuiaoo"
          }
        ],
        "name": "Section_FKFGZKFG87ZFGZKZFGHKZ87",
        "title": "section"
      }
    ]
  }
}

And a server item has multiple sections under details, while the keys in the field objects are just single characters:

{
  "details": {
    "sections": [
      {
        "fields": [
          {
            "inputTraits": {
              "autocapitalization": "none",
              "autocorrection": "no"
            },
            "k": "string",
            "n": "username",
            "t": "client_id",
            "v": "ifugae7b0u98i7ape078b"
          },
          {
            "k": "concealed",
            "n": "password",
            "t": "client_secret",
            "v": "up908mußpffugh7pb-89bu7iae"
          }
        ],
        "name": "",
        "title": ""
      }
    ]
  }
}

This can make parsing the data a bit tricky. For our use case, I think it is a good idea to stick to the item type server. In the last example, you can extract the client_secret like this:

eval op get item MyItem | jq ".details.sections[0].fields[] | select(.t==\"client_secret\").v" -r

The script

Now that we know how to retrieve items and secrets via the command line, we can put the logic into a script.

This is the fish version (you could put the 1p_server_field function in a separate file in ~/.config/fish/functions to make it available on your system):

#!/usr/bin/env fish

set vault TeamVault

function 1p_server_field -d "retrieves a field from a 1password server item" -a item_name section_index field_name
  eval op get item $item_name --vault $vault | jq ".details.sections[$section_index].fields[] | select(.t==\"$field_name\").v" -r
end

set -x SOME_CLIENT_ID (1p_server_field some_item_name 0 client_id)
set -x SOME_CLIENT_SECRET (1p_server_field some_item_name 0 client_secret)

And here the same as a bash script:

#!/bin/bash -eux
vault=TeamVault
1p_server_field () {
  echo "$(eval op get item $1 --vault $vault | jq ".details.sections[$2].fields[] | select(.t==\"$3\").v" -r)"
}
export SOME_CLIENT_ID=$(1p_server_field some_item_name 0 client_id)
export SOME_CLIENT_SECRET=$(1p_server_field some_item_name 0 client_secret)

Save this as env.sh and run . env.sh.

You can safely commit a script like this to your repository. Nobody has to ask anymore where to find the credentials, and after rolling a secret, all you have to do is to run this script again.

Don’t stop believing.

originally published on Medium

Guideline for (remote) meetings

Mathias Polligkeit — Fri, 05 Jun 2020 19:47:45 +0000

While remote meetings allow you to bridge the distance between distributed workers, attendees may still feel isolated, disengaged and separated from the team if the meetings aren't set up properly. All of the points listed here are common sense, but still, all of them are disregarded at times. This guideline is meant to remind you of a few things to keep in mind.

In the conference room

1. Set up conference microphones suitable for the number of attendees.

You want to make sure that the audio is clear and everyone in the room can be understood by the remote attendees.

2. Ensure that the attendees are distributed evenly around the microphones.

The best microphone is of no use if only attendees on one side of the table can be heard.

3. Set up a camera that captures all attendees.

Using a single laptop camera directed at the meeting facilitator is not enough. A camera that captures all attendees helps everyone who joins remotely to see who is attending the meeting, who is talking and whatever is going on in the meeting room. Think about someone looking for something in their notes. If you can see it, you know. If you don’t, you wonder why nothing is happening anymore.

Make sure that nobody sits outside of the picture frame. If remote attendees don’t see you, they may assume you are not there and be surprised or even startled when they suddenly hear your voice from the off.

4. Set up a big screen to display the video of the remote attendees and the screen shares.

This is the other side of the previous point. It is easy to forget that there is someone joining remotely if you cannot see them. If they are visible on the screen, you are more likely to directly address them and ask for their opinion.

5. Ensure that all attendees can see the screen.

Again, it doesn’t help if only the meeting facilitator can see the video.

6. Ensure that the internet connection is sufficient.

This is obvious but still worth mentioning. It is difficult to have a meeting if you have to deal with audio artefacts, grainy stop-and-go video and audio dropouts.

7. Set up the equipment before the start of the meeting.

Don’t waste anybody’s time by searching for adapters, cables and video conference IDs just when the meeting is supposed to start.

As lonely remote attendee

8. Use a decent microphone, consider headphones or a headset.

If you use a low-quality laptop microphone, you will be hard to understand. Using headphones or a headset can alleviate problems with echos and feedback loops.

9. Turn on the camera.

This helps other attendees to remember that you are there, gauge your reactions and create a sense of nearness.

10. Mute the audio when there is background noise or when you cause noise.

Background noise transmitted through the microphone makes everything harder to understand. Also, there is no need to transmit coughs, sneezes or noises from drinking or arranging items on your desk. Don’t breath into the microphone.

11. Ensure that the internet connection is sufficient.

Same as in the previous section.

12. Join the meeting from a room.

Don’t join a meeting while walking through the city. Be aware of your background. Nobody needs to see the laundry on your bed.

Conducting the meeting

Most of these points are valid for on-site meetings as well, but it’s still worth mentioning them.

13. Prepare the meeting.

Don’t try to figure out what to do during the meeting. Have your notes, documents and tools ready.

14. Add a link to the video conference to the calendar event.

This allows everyone to join on time without asking for the meeting ID.

15. Supply a meeting agenda ahead of time.

Give everyone the chance to think about the topic in advance, gather their notes or jot down questions. Use the description of the calendar event to state the goal, list the agenda points and add links to relevant documents and further information.

16. Invite the right people.

Make sure to only invite the people you need. If you need someone only for some of the agenda points, move those topics to the beginning of the meeting and let them leave when you move on to the remaining topics.

17. RSVP to all calendar invitations.

This lets the organiser know who attends the meeting ahead of time and reschedule if necessary. It also helps to avoid waisting time at the beginning of the meeting for figuring out whether someone will still join or not.

18. Ensure that there is a facilitator who leads the meeting.

There should be someone responsible for leading through the agenda, moderating the discussions and keeping the meeting on topic. This person should also make sure that there is a conclusion to each topic and that the next steps are clearly named. Of course, this responsibility doesn’t need to lie with a single person. You can always have dedicated facilitators for different agenda points.

19. Ensure that there is someone who writes meeting notes.

Put the salient points as well as the decisions and tasks emerging from the meeting in writing.

20. Be on time.

Be on time!

21. Start the meeting on time.

Don't wait another ten minutes until the last attendee has joined.

22. Clearly start the meeting.

Don’t move seamlessly from chit-chat right into the topic. Say welcome, state the goal of the meeting, say who is attending, then start with the first topic.

23. Acknowledge the attendees

Say hello to anybody who joins the meeting. If you have a conference room full of people and some lonely remote attendees, it is easy for remote attendees to feel like outsiders and to fall into a passive role. Make them feel part of the team. Ask for their opinion. Try to engage them.

24. Don’t silently join the meeting.

Say hello when you join the meeting. Even when you’re late. If you join at a moment where you cannot interrupt, use the first opportunity to say hello.

25. Share your screen.

Whenever you talk about something that you see on your screen, make sure to share it so that everybody can follow.

26. Don’t use a whiteboard.

Whiteboard, flip chart or sticky note drawings are hard to recognise via video stream. There are online-collaboration tools for everything. If on the other hand you conduct a pure on-site meeting, by all means, use all the low-tech tools you have.

27. End the meeting on time.

Learn to use the meeting time efficiently and don't use more time for the meeting than was scheduled. If there is still more to discuss at the end of the meeting, add a remark in the meeting notes and schedule another, perhaps more focused meeting on another day.

If you are in a conference room, end a few minutes before the next scheduled meeting in the room so that you have time to pack up and so that the next meeting party has time to set up.

28. Clearly end the meeting.

Don’t mumble into your beard that the meeting is over and let the meeting devolve into chit-chat, letting the remote attendees wonder whether there is something else to come. Wrap up, thank everyone, say bye to the remote attendees and end the meeting.

29. Send a link to the meeting notes after the meeting.

Don’t let the meeting notes rot on your computer. Make them available to all attendees and everyone interested. Notify the attendees when they are available. Ask them to read them through for correctness and to add missing points, corrections or clarifications if necessary.

originally published on Medium

Adventures in Ecto and PostgreSQL: Partial unique indexes and upserts with associations

Mathias Polligkeit — Fri, 05 Jun 2020 19:42:43 +0000

Let’s say you were asked by your government to implement a national dog registry in order to help them to enforce their new one-dog-policy. Everyone who wants to buy a dog will need to register it at the municipal government.

Since your government cares a lot about the well-being of the nation’s dogs and generally invests a lot of time and resources into research endeavours, they want to collect some additional data on all the dogs and want all dog owners to check in once a year to keep the information up to date. Because a year is a long time between updates to conduct proper research and also to keep the feedback loop short regarding new dog-happiness policies, the government also plans to equip dogs and dog-related products with sensors in the future.

Setup

Versions: Elixir 1.10.0, ecto 3.3.3, ecto_sql 3.3.4, PostgreSQL 11.5

Let’s start off with this Ecto migration:

defmodule DogRegistry.Repo.Migrations.CreateTables do
  use Ecto.Migration

  def change do
    create table(:owners, primary_key: false) do
      add :id, :binary_id, primary_key: true
      add :name, :string, null: false
      add :city, :string, null: false
      timestamps()
    end

    create table(:dogs, primary_key: false) do
      add :id, :binary_id, primary_key: true
      add :name, :string, null: false
      add :status, :string, null: false
      add :owner_id, references(:owners, on_delete: :nilify_all, type: :uuid)
      timestamps()
    end

    create table(:dog_toys, primary_key: false) do
      add :id, :binary_id, primary_key: true
      add :product, :string
      add :rating, :integer
      add :dog_id, references(:dogs, on_delete: :nilify_all, type: :uuid)
      timestamps()
    end

    create index(:dog_toys, :dog_id)
  end
end

With this we create an owners table, a dogs table and a dog_toys table. A dog belongs to an owner and a dog toy belongs to a dog. The government will need a lot more information on the dogs, of course, including various psychological measures and health statistics, but let’s keep it simple for now.

We also create Ecto schemas for each of these tables, but since they are quite unremarkable, I’ll only give you the Dog schema:

defmodule DogRegistry.Dog do
  use Ecto.Schema

  import Ecto.Changeset

  alias DogRegistry.DogToy
  alias DogRegistry.Owner
  alias DogRegistry.Repo

  @primary_key {:id, :binary_id, autogenerate: true}
  @foreign_key_type :binary_id
  schema "dogs" do
    field :name, :string
    field :status, :string
    belongs_to :owner, Owner
    has_many :toys, DogToy, on_replace: :delete
    timestamps()
  end

  def changeset(dog, attrs) do
    dog
    |> cast(attrs, [:name, :owner_id, :status])
    |> cast_assoc(:toys)
    |> validate_required([:name, :owner_id, :status])
    |> validate_inclusion(:status, ["alive", "dead"])
    |> foreign_key_constraint(:owner_id)
  end
end

Partial unique indexes

You decided to handle both dog registrations and updates to existing dogs with a single function. Don’t ask me why, you didn’t tell me.

This means we need to implement this as an upsert. The first thing we need is a unique index on the dogs table to prevent the clerk from inserting a second dog with the same owner. However, one-dog-policy does not mean that you may only have one single dog throughout your entire lifetime. Once your dog died, you can get a new one. This means the unique index may only apply if the dog is alive.

PostgreSQL allows you to define partial indexes to do just that.
In SQL that index would be defined like this:

CREATE UNIQUE INDEX dogs_owner_id_alive_ix ON dogs (owner_id, status)
WHERE status = 'alive';

Ecto.Migration.unique_index/3 allows you to add a where option, so the statement above looks like this in your application:

defmodule DogRegistry.Repo.Migrations.DogsOwnerIdAliveIx do
  use Ecto.Migration
  def change do
    create unique_index(:dogs, [:owner_id, :status],
             name: "dogs_owner_id_alive_ix",
             where: "status = 'alive'"
           )
  end
end

The first tests

We create a test module that uses the DataCase module Phoenix creates and sets up some fixtures. The complete code example is linked below. The first describe block only has a simple test to make sure you can insert a new dog into the database:

describe "create_or_update/1" do
  test "creates a dog" do
    owner = insert_owner!(@owner)
    assert {:ok, %Dog{}} = Dog.create_or_update(params(@living_dog_1, owner))
    assert [%Dog{}] = Repo.all(Dog)
  end

  test "allows to add multiple dead dogs" do
    owner = insert_owner!(@owner)
    assert {:ok, %Dog{}} = Dog.create_or_update(params(@dead_dog_1, owner))
    assert {:ok, %Dog{}} = Dog.create_or_update(params(@dead_dog_2, owner))
    assert {:ok, %Dog{}} = Dog.create_or_update(params(@living_dog_1, owner))
    assert {:ok, %Dog{}} = Dog.create_or_update(params(@dead_dog_3, owner))
    assert [_, _, _, _] = Repo.all(Dog)
  end
end

The params/2 function does nothing but add the ID of the second argument to the map that is passed as the first argument. I know that you know that there is a pipe operator.

This is the function that passes the tests:

def create_or_update(attrs) do
  %__MODULE__{}
  |> changeset(attrs)
  |> Repo.insert()
end

The upsert test

Now on to the more interesting test case.

test "replaces previous dog if alive" do
  owner = insert_owner!(@owner)
  assert {:ok, %Dog{}} = Dog.create_or_update(params(@living_dog_1, owner))
  assert {:ok, %Dog{}} = Dog.create_or_update(params(@living_dog_2, owner))
  assert [%Dog{name: name}] = Repo.all(Dog)
  assert name == @living_dog_2.name
end

This looks similar as the previous test, but now you ensure that there is still only one dog in the database after calling the function a second time. This means if there is already a living dog for the owner in the database, the entry will be replaced silently. You slowly come to the conclusion that this behaviour is probably not what you would want in this scenario, but since you are already invested in this example, you decide to continue.

Unsurprisingly, this test fails with the current implementation. To perform an upsert, you can set the :on_conflict option when calling Repo.insert/2. When using PostgreSQL, you also need to set the :conflict_target option to tell PostgreSQL which constraints you expect. This can be a list of columns or the constraint name itself. Since we named the unique index, we can try this:

def create_or_update(attrs) do
  %__MODULE__{}
  |> changeset(attrs)
  |> Repo.insert(
    on_conflict: :replace_all,
    conflict_target: {:constraint, :dogs_owner_id_alive_ix}
  )
end

However, now we're getting this error in all the tests:

** (Postgrex.Error) ERROR 42704 (undefined_object) constraint "dogs_owner_id_alive_ix" for table "dogs" does not exist

What’s going on?

Ecto turns the options above into this SQL fragment:

ON CONFLICT ON CONSTRAINT "dogs_owner_id_alive_ix"`

This is syntactically correct, but PostgreSQL requires you to specify the constraint including the whole WHERE clause. So we need to find a way to tell Ecto to do that. Unfortunately, there is no shortcut for this at the moment, but instead of passing the constraint name, Ecto also allows us to pass an unsafe SQL fragment.

def create_or_update(attrs) do
  %__MODULE__{}
  |> changeset(attrs)
  |> Repo.insert(
    on_conflict: :replace_all,
    conflict_target:
      {:unsafe_fragment, "(owner_id, status) WHERE status = 'alive'"}
  )
end

This works!

Associations

Your plan was to update all the associations of the dog (except for the owner) with that same function call. So far, the test fixtures look like this:

@dead_dog_1 %{name: "Clarence", status: "dead"}
@dead_dog_2 %{name: "Penelope", status: "dead"}
@dead_dog_3 %{name: "Christopher", status: "dead"}
@living_dog_1 %{name: "Dwayne", status: "alive"}
@living_dog_2 %{name: "Claire", status: "alive"}

Very simple, no associations. Let’s give Dwayne some toys and see what happens.

@living_dog_1 %{
    name: "Dwayne",
    status: "alive",
    toys: [
      %{product: "round chewie thing", rating: 5},
      %{product: "long catchie thing", rating: 4}
    ]
  }

Running the tests again, we get a foreign key constraint error for the test create_or_update/1 replaces previous dog if alive:

1) test create_or_update/1 replaces previous dog if alive (DogRegistry.DogTest)
     test/dog_registry/dog_test.exs:50
     ** (Ecto.ConstraintError) constraint error when attempting to insert struct:
* dog_toys_dog_id_fkey (foreign_key_constraint)
If you would like to stop this constraint violation from raising an
     exception and instead add it as an error to your changeset, please
     call `foreign_key_constraint/3` on your changeset with the constraint
     `:name` as an option.
The changeset defined the following constraints:
* dogs_owner_id_fkey (foreign_key_constraint)

The other tests pass, though. So inserting a dog including toys is no problem, but updating an existing dog with toys is.
Let’s remove the toys from the fixtures for a moment and update the test like this:

test "replaces previous dog if alive" do
  owner = insert_owner!(@owner)
  assert {:ok, %Dog{id: id_1}} =
               Dog.create_or_update(params(@living_dog_1, owner))
  assert {:ok, %Dog{id: id_2}} =
               Dog.create_or_update(params(@living_dog_2, owner))
  IO.inspect(id_1, label: "ID1")
  IO.inspect(id_2, label: "ID2")
  assert [_] = Repo.all(Dog)
end

You will get an output similar to this:

ID1: "6a274d9b-f0c1-4f06-b8db-1ef3c7103da4"
ID2: "7e9f7b9c-7a64-4a96-a0a3-2aaa08360b33"

As it happens, passing on_conflict: :replace_all to Repo.insert/2 will not only replace the values in the changeset, but also generate new values for all autogenerated fields including timestamps and the ID. If we want to keep some of the existing values, we can pass {:replace_all_except, fields} or {:replace, fields} as the option value. Let's change the on_conflict option of create_or_update/1 to:

on_conflict: {:replace_all_except, [:id, :inserted_at]}

We want to keep the original ID, but we also want to keep the inserted_at value, while we are happy to get an updated updated_at value.

If we run the test again, we will still see two different IDs. Why is that?

We configured our application with UUIDs as primary keys. Unlike auto-incrementing integer IDs, UUIDs are auto-generated by Ecto, not in the database (unless you set autogenerate to false in your schema and configure your database to generate them). And since Ecto already knows all values it generated or set itself, it doesn’t ask them from the database when running the query. So in our case, Ecto generates a UUID, sends an insert statement to the database, the database sees a conflict and makes a replacement instead, but keeps the old ID, and this existing ID is not returned back to Ecto.

The fix is easy, though. Repo.insert/2 accepts another option called :returning, which specifies which fields should be returned from the database. If you set returning to true, all fields will be returned.

So the final version of create_or_update/1 looks like this:

def create_or_update(attrs) do
  %__MODULE__{}
  |> changeset(attrs)
  |> Repo.insert(
    returning: [:id],
    on_conflict: {:replace_all_except, [:id, :inserted_at]},
    conflict_target:
      {:unsafe_fragment, "(owner_id, status) WHERE status = 'alive'"}
  )
end

Now the IDs in the test output will match and the tests will also pass with fixtures including associations.

However, our tests still don't have any assertions about the associations. Let's change that by adding this test:

test "replaces all associations" do
  owner = insert_owner!(@owner)
  assert {:ok, %Dog{}} = Dog.create_or_update(params(@living_dog_1, owner))
  assert {:ok, %Dog{}} = Dog.create_or_update(params(@living_dog_2, owner))
  assert [%DogToy{product: "small squeaky thing"}] = Repo.all(DogToy)
end

Similar to the previous test, except that we actually have a look at the dog toys after the upsert. @living_dog_1 had two toys, @living_dog_2 had only one, so we would expect to be only the single toy of dog 2 in the database after the update. But this test fails, the database includes both toys of dog 1 and the toy of dog 2.

The problem is that PostgreSQL can handle the replacement of the dog entry itself, but it doesn't know anything about the associations, and since we only passed an empty struct as the first argument to Ecto.Changeset.changeset/2 in our create_or_update/2 function, Ecto has no way of tracking the associations either.

Now, we could change that by 1) changing the function signature of create_or_update/2 and 2) passing a fully preloaded struct to the function, but that would defeat the purpose of the upsert function, because we could then as easily call a regular update function with that preloaded resource. Which means that if you want to do a complete replacement of the existing associations, you shouldn't use an upsert function after all.

Complete source code: github.com/woylie/article-dog-registry

Don't look directly into the sun.

originally published on Medium

Property-Based Test-Driven Development in Elixir

Mathias Polligkeit — Fri, 05 Jun 2020 19:31:21 +0000

Recently I came across a small function in our code base that was meant to transform a list of arbitrarily sorted and possibly overlapping time ranges into a sorted list of non-overlapping time ranges, or in other words, to merge the time ranges in a list. The function was covered by example-based unit tests, but unfortunately, once in a while the function would throw an error. This was a good opportunity to use property-based testing in order to find inputs that would cause the errors.

In this blog post, I would like to turn it around and build a function like that from scratch by using a property-based test-driven approach. The principles of property-based testing are easy to understand, but the challenge is to find good properties that fully describe a function.

We will use the StreamData library, but it should be easy to apply the steps to other libraries like PropEr.

The problem

We need to write a function that takes a list of date ranges and returns a list of date ranges sorted by start date that does not contain any overlapping date ranges.

Date ranges are represented by 2-tuples (we could also have used the Date.Range type instead). The end date in the tuple always has to be greater than or equal to the start date.

Example input:

[
  {~D[1999-01-01], ~D[1999-06-01]},
  {~D[1998-11-01], ~D[1999-02-01]},
  {~D[1999-06-02], ~D[1999-08-01]},
  {~D[1999-10-01], ~D[1999-12-01]}
]

The function should turn this into:

[
  {~D[1998-11-01], ~D[1999-08-01]},
  {~D[1999-10-01], ~D[1999-12-01]}
]

Setup

Let’s start by creating a new mix application.

mix new date_range_merger
cd date_range_merger

Add {:stream_data, "~> 0.4.3"} to mix.exs and run mix deps.get.

Change the content of .formatter.exs to the following in order to prevent mix format from adding brackets to StreamData macro calls:

# Used by "mix format"
[
  inputs: ["{mix,.formatter}.exs", "{config,lib,test}/**/*.{ex,exs}"],
  line_length: 80,
  import_deps: [:stream_data]
]

Generators

We need to create a generator that generates a valid date range tuple. Let’s start by creating a generator for a date and build on that.

Date generator

Replace the code in test/date_range_merger_test.exs with this:

defmodule DateRangeMergerTest do
  use ExUnit.Case
  use ExUnitProperties
  import DateRangeMerger
  import StreamData

  defp date do
    gen all year <- integer(1970..2050),
            month <- integer(1..12),
            day <- integer(1..31),
            match?({:ok, _}, Date.from_erl({year, month, day})) do
      Date.from_erl!({year, month, day})
    end
  end

  property “generator test” do
    check all generated_date <- list_of(date()) do
      IO.inspect(generated_date)
    end
  end
end

What’s happening here? We define a private function called date which uses the ExUnit.Properties.gen/1 macro to define a generator. In the first three lines of the function, we call StreamData.integer/1 to generate values for the date parts.

The fourth line is a bit different: It doesn’t generate new data, but acts as a filter. We try to build a date from the parts. When it succeeds, match?/2 will return true and the generator returns whatever is returned in the do block. When it fails (e. g. when we generated a 31st of February), match?/2 will return false, the generated year, month and day are discarded and the generator will try again with different values.

Why do we use match?/1? Why don’t we just write {:ok, date} <- Date.from_erl({year, month, day})? While it is possible to filter generated values on the left side of the arrow through pattern matching, gen/1 expects a generator on the right side of the arrow.

After the date generator, we define the first property test, although it is only a dummy test to verify our generator works as expected. Property tests are defined with the ExUnitProperties.property/2 macro. The test uses the ExUnitProperties.check/1 macro to generate test data and run assertions within its do block.

When you run mix test, you should see a list of 100 random dates between 1st January 1970 and 31st December 2050.

Date range generator

Now that we have our date generator, we can build a date range generator on top of it. It is easy to compose generators. Our requirement is that the end date has to be greater than or equal to the start date.

Add this right below the date/0 function:

defp date_range do
  gen all start_date <- date(),
          days_between <- integer(0..18_250) do
    {start_date, Date.add(start_date, days_between)}
  end
end

We use our custom date generator to create the start date and generate a number of days to add between 0 and 365 * 50. Of course this will cause our generator to return date ranges between 1970 and 2100. If we still want to keep our date range within 1970 and 2050, we can add a filter clause like this:

defp date_range do
  gen all start_date <- date(),
          days_between <- integer(0..18_250),
          end_date = Date.add(start_date, days_between),
          Date.compare(end_date, ~D[2050–12–31]) != :gt do
    {start_date, end_date}
  end
end

Let’s change the property test to:

property "generator test" do
  check all date_range <- date_range() do
    IO.inspect(date_range)
  end
end

Now run mix test, and you should… whoa, what’s this?

1) property generator test (DateRangeMergerTest)
     test/date_range_merger_test.exs:25
     ** (StreamData.FilterTooNarrowError) too many consecutive elements were filtered out.
      To avoid this:
       * make sure the generation space contains enough values that the chance of a generated
         value being filtered out is small. For example, don’t generate all integers and filter
         out odd ones in order to have a generator of even integers (since you’d be taking out
         half the generation space).
       * keep an eye on how the generation size affects the generator being filtered. For
         example, you might be filtering out only a handful of values from the generation space,
         but small generation sizes might make the generation space much smaller hence increasing
         the probability of values that you’d filter out being generated.
       * try to restructure your generator so that instead of generating many values and taking
         out the ones you don’t want, you instead generate values and turn all of them into
         values that are suitable. For example, multiply integers by two to have a generator of
         even values instead of filtering out all odd integers.

The filter is too narrow! Turns out that StreamData doesn’t like it if you generate too many invalid values. Let’s try to fix this.

One way to do this would be to reduce the maximum difference between the dates, thereby reducing the risk that the end date is after 2050:

days_between <- integer(0..1000),

That seems to work, but it isn’t particularly nice. We want a broad range of inputs in a property-based test, and now we reduced the maximum date range to a mere 1000 days. Let’s try a different approach.

defp date_range do
  gen all start_date <- date(),
          max_difference = Date.diff(~D[2050-12-31], start_date),
          days_between <- integer(0..max_difference) do
    {start_date, Date.add(start_date, days_between)}
  end
end

This is much better. Instead of generating a lot of values and filtering out the ones we don’t want, we calculate the maximum allowed difference from the generated start date, so that we only generate valid end dates in the first place. No need for a filter.

Run mix test again and you should see a glorious list of date ranges between 1970 and 2050.

First property: Always returns a list

Time for our first property. We’re going to start simple: We know that the function should always return a list for valid input.
Replace the dummy property test with this:

property "always returns a list" do
  check all date_ranges <- list_of(date_range()) do
    assert is_list(merge(date_ranges))
  end
end

This might not seem very exciting, but it ensures that whatever implementation we come up with will never throw an exception. This property was already enough to uncover the problem with the faulty implementation mentioned in the introduction.

Unsurprisingly, the compiler will complain about an undefined function. Since we want to follow a pure TDD approach, let’s write the bare minimum to make this test pass. Replace the code in lib/date_range_merger.ex with:

defmodule DateRangeMerger do
  @moduledoc """
  Documentation for DateRangeMerger.
  """

  @doc """
  Takes a list of date ranges and returns a list of date ranges sorted by first
  date that does not contain any overlapping date ranges.
  """
  @spec merge([{Date.t(), Date.t()}]) :: [{Date.t(), Date.t()}]
  def merge(date_ranges) when is_list(date_ranges) do
    []
  end
end

Brilliant. There’s only one tiny problem: The input date ranges are not contained within the output. This seems like a good property to tackle next.

Second property: Each input date range is covered

So let’s implement that as a property test. We still want to generate a list of input date ranges. Then we want to make sure that each date range in the list is covered by one of the date ranges in the result list. Since we’re supposed to merge overlapping date ranges, we don’t look for an exact match. Our property could look something like this:

property "input date ranges are contained within output date ranges" do
  check all date_ranges <- list_of(date_range()) do
    result = merge(date_ranges)

    for {start_date, end_date} <- date_ranges do
      assert {_, _} =
               Enum.find(result, fn {result_start_date, result_end_date} ->
                 Date.to_erl(start_date) >= Date.to_erl(result_start_date) &&
                   Date.to_erl(end_date) <= Date.to_erl(result_end_date)
               end)
    end
  end
end

Note that we have to convert the dates to tuples with Date.to_erl/1 before we can compare them, because otherwise Dates would be ordered structurally according to Erlang’s term ordering. Alternatively, we could use Date.compare/2, but I think the version above is more readable than this:

Date.compare(start_date, result_start_date) != :lt &&
  Date.compare(end_date, result_end_date) != :gt

When we run mix test, we get this:

1) property input date ranges are contained within output date ranges (DateRangeMergerTest)
     test/date_range_merger_test.exs:31
     Failed with generated values (after 0 successful runs):

         * Clause:    date_ranges <- list_of(date_range())
           Generated: [{~D[1973-07-10], ~D[1973-07-10]}]

     match (=) failed
     code:  assert {_, _} = Enum.find(result, fn {result_start_date, result_end_date} -> Date.compare(start_date, result_start_date) != :lt && Date.compare(end_date, result_end_date) != :gt end)

StreamData has shrunk the input list to a single date range, which obviously cannot be found in the empty list returned by the function. We can make this test pass with a simple change:

def merge(date_ranges) when is_list(date_ranges) do
  date_ranges
end

Instead of an empty list, we just return the untouched input list.

Third property: Output is sorted

It’s time that the function actually does something. Let’s add another easy property: The output list is supposed to be sorted by start date. We can test this by chunking the result list in pairs of date ranges and asserting that in each pair, the start date of the first date range is lower than or equal to the start date of the second date range.

property "output date ranges are sorted by start date" do
  check all date_ranges <- list_of(date_range()) do
    pairs =
      date_ranges
      |> merge()
      |> Enum.chunk_every(2, 1, :discard)

    for [{start_date_a, _}, {start_date_b, _}] <- pairs do
      assert Date.to_erl(start_date_a) <= Date.to_erl(start_date_b)
    end
  end
end

To make this pass, all we have to do is to sort the input list:

def merge(date_ranges) when is_list(date_ranges) do
  Enum.sort_by(date_ranges, fn {start_date, _} -> Date.to_erl(start_date) end)
end

So far, so good. Now on to the actual merging part.

Fourth property: No overlap

Let’s recapitulate what we have done so far. We have made sure that all input date ranges are somehow contained within the output date ranges and that the output date ranges are sorted by start date. The other requirement is that the result does not contain any overlapping date ranges. What does this mean?

In a date range list sorted by start date, the start dates of two subsequent date ranges may not be the same.

In a date range list sorted by start date, for each pair of subsequent date ranges, the end date of the first range must be smaller than the start date of the second range.

We can implement this similarly to the third property (sorted list).

property "output date ranges do not overlap" do
  check all date_ranges <- list_of(date_range()) do
    pairs =
      date_ranges
      |> merge()
      |> Enum.chunk_every(2, 1, :discard)

    for [{start_date_a, end_date_a}, {start_date_b, _}] <- pairs do
      assert Date.to_erl(start_date_a) != Date.to_erl(start_date_b)
      assert Date.to_erl(end_date_a) < Date.to_erl(start_date_b)
    end
  end
end

Of course we could merge the first point, or even both, into the previous property, but I prefer to keep these properties separate for reasons of clarity.

To make this test pass, we can build up the output list from the sorted listed in a reduce function.

def merge(date_ranges) when is_list(date_ranges) do
  date_ranges
  |> Enum.sort_by(fn {start_date, _} -> Date.to_erl(start_date) end)
  |> Enum.reduce([], &merge_overlapping/2)
  |> Enum.reverse()
end

defp merge_overlapping(
       {next_start_date, next_end_date},
       [{previous_start_date, previous_end_date} | rest] = acc
     ) do
  if Date.to_erl(next_start_date) <= Date.to_erl(previous_end_date) do
    # date ranges overlap, merge them into one
    new_end_date =
      Enum.max_by([previous_end_date, next_end_date], &Date.to_erl/1)

    [{previous_start_date, new_end_date} | rest]
  else
    # date ranges don't overlap, prepend to list unchanged
    [{next_start_date, next_end_date} | acc]
  end
end

defp merge_overlapping({next_start_date, next_end_date}, []),
  do: [{next_start_date, next_end_date}]

So basically we walk through the list and figure out whether the current range overlaps with the previous range. If yes, we merge them and prepend the new date range tuple to the accumulated result list. If no, we prepend the current range to the result list as it is.

Fifth property: Adjacent date ranges are merged

This makes our tests pass, but there is one small problem: We didn’t consider that the date ranges are meant to be inclusive. That means that when the start date of a date range lies exactly one day after the end date of the previous date range, these ranges still have to be merged (e. g. [{~D[1982-06-01], ~D[1982-07-01]}, {~D[1982-07-02], ~D[1982-07-04]}] should be merged to [{~D[1982-06-01], ~D[1982-07-04]}]). Let’s update our "output date ranges do not overlap" property test to reflect this.

Change this line:

assert Date.to_erl(end_date_a) < Date.to_erl(start_date_b)

To this:

assert Date.diff(start_date_b, end_date_a) > 1

If you run the tests again, they will still pass most of the time. It seems like this particular case does not occur very frequently in the generated data. Let’s add another more specific generator:

defp adjacent_date_ranges do
  gen all {start_date_a, end_date_a} <- date_range(),
          start_date_b = Date.add(end_date_a, 1),
          max_difference = Date.diff(~D[2050-12-31], start_date_b),
          days_between <- integer(0..max_difference),
          end_date_b = Date.add(start_date_a, days_between) do
    Enum.shuffle([{start_date_a, end_date_a}, {start_date_b, end_date_b}])
  end
end

First we generate a single date range tuple with our existing generator, then we build a second range that starts exactly one day after the first one ends. As a last step, we build a list from both date ranges and shuffle it.

We could now use this new generator in the existing "output date ranges do not overlap" property, but I prefer to add a separate property, as it makes the intent clearer. Adjacent date ranges and overlapping date ranges are two different things, after all.
Add this at the end of the test module:

property "adjacent date ranges are merged" do
  check all date_range_pairs <- list_of(adjacent_date_ranges()) do
    date_ranges = date_range_pairs |> Enum.concat() |> Enum.shuffle()

    pairs =
      date_ranges
      |> merge()
      |> Enum.chunk_every(2, 1, :discard)

    for [{start_date_a, end_date_a}, {start_date_b, _}] <- pairs do
      assert Date.to_erl(start_date_a) != Date.to_erl(start_date_b)
      assert Date.diff(start_date_b, end_date_a) > 1
    end
  end
end

The assertions are exactly the same as in "output date ranges do not overlap". The only difference is the use of another generator.

If you run the tests now, they should fail reliably. The fix is easy. In lib/date_range_merger.ex, replace this line:

if Date.to_erl(next_start_date) <= Date.to_erl(previous_end_date) do

With this:

if Date.diff(next_start_date, previous_end_date) < 2 do

Are we there yet?

So far, we made sure that:

the function always returns a list,
each input date range is contained within the output date ranges,
the output is sorted by start date,
the output date ranges do not overlap, and adjacent date ranges are merged.

Is this enough to fully describe the function? Let’s try to trick the tests with a faulty implementation. Replace DateRangeMerger.merge/1 with this:

def merge([]), do: []

def merge(date_ranges) when is_list(date_ranges) do
  start_date =
    date_ranges
    |> Enum.min_by(fn {a, _} -> Date.to_erl(a) end)
    |> elem(0)

  end_date =
    date_ranges
    |> Enum.max_by(fn {_, b} -> Date.to_erl(b) end)
    |> elem(1)

  [{start_date, end_date}]
end

Now run the tests again, and you should see:

5 properties, 0 failures

That was easy! Just by returning a list with a single date range that spans all the input date ranges, we made all tests pass.
We can make it even easier:

def merge(date_ranges) when is_list(date_ranges) do
  [{~D[0000-01-01], ~D[9999-12-31]}]
end

This will also pass our tests.

Let’s tackle the easier issue first.

Sixth property: Min and max dates match in input and output

We were able to make our tests pass by returning a single date range that extends way beyond the test values we generate. We should make sure that:

the min start dates in the input and output are the same, and
that the max end dates in the input and output are the same.

That doesn’t seem so difficult. Let’s add this property test:

property "min and max dates in input and output match" do
  check all date_ranges <- list_of(date_range()) do
    result = merge(date_ranges)

    assert min_start_date(date_ranges) == min_start_date(result)
    assert max_end_date(date_ranges) == max_end_date(result)
  end
end

defp min_start_date([]), do: nil

defp min_start_date(date_ranges) do
  date_ranges
  |> Enum.map(&elem(&1, 0))
  |> Enum.min_by(&Date.to_erl/1)
end

defp max_end_date([]), do: nil

defp max_end_date(date_ranges) do
  date_ranges
  |> Enum.map(&elem(&1, 1))
  |> Enum.max_by(&Date.to_erl/1)
end

This will make our tests fail with the second naive implementation, but our test suite is still being fooled by the first one.

Seventh property: Gaps in input list are preserved

So what’s the problem exactly? While we do test whether all input date ranges are covered in the output list, we do not ensure that the gaps between the input date ranges are preserved. Which strategies could we employ to test this property?

We could write a function that takes a list of date ranges and returns a list of date ranges that are not covered (the gaps). We could then assert that none of those gap date ranges are covered in the output list. However, the implementation of that function would probably be very similar to the function we want to test. Also, this gap-finder function would warrant its own unit tests.
We could pass the list of input date ranges to a second generator that also generates date ranges, but filters them so that it only returns date ranges not covered by the input date ranges. The generator would thus generate a list of random gap ranges that should not be covered. Unfortunately, this would cause a FilterTooNarrowError as we have seen earlier (I’ve tried).
We could build a generator similar to the adjacent_date_ranges generator, but have it generate a longer list of adjacent date ranges that will be marked as either cover or gap. The date ranges marked as cover would be used as input, while the date ranges marked as gap would be used in the assertion.

Since I cannot come up with a better solution, let’s try to implement option 3.

The generator:

defp date_ranges_with_gaps() do
  gen all initial_date <- date(),
          count <- integer(2..1000),
          date_ranges <- do_date_ranges_with_gaps(initial_date, count) do
    {cover, gap} =
      Enum.split_with(date_ranges, fn
        {:cover, _, _} -> true
        {:gap, _, _} -> false
      end)

    cover_ranges =
      cover
      |> Enum.map(fn {_, start_date, end_date} -> {start_date, end_date} end)
      |> Enum.shuffle()

    gap_ranges =
      gap
      |> Enum.map(fn {_, start_date, end_date} -> {start_date, end_date} end)
      |> Enum.shuffle()

    %{
      cover: cover_ranges,
      gap: gap_ranges
    }
  end
end

defp do_date_ranges_with_gaps(previous_end_date, count, acc \\ []) do
  if count <= 0 || Date.to_erl(previous_end_date) >= {2050, 12, 31} do
    constant(acc)
  else
    gen all type <- member_of([:cover, :gap]),
            start_date = Date.add(previous_end_date, 1),
            max_difference = Date.diff(~D[2050-12-31], start_date),
            days_between <- integer(0..max_difference),
            end_date = Date.add(start_date, days_between),
            result <-
              do_date_ranges_with_gaps(
                end_date,
                count - 1,
                [{type, start_date, end_date} | acc]
              ) do
      result
    end
  end
end

This generator is a bit trickier than the previous one, because it generates a list where each item depends on the previous item.

First we generate an initial date to start at and a count of list items to be generated. Then we call the sub-generator do_date_ranges_with_gaps/3. This is a recursive generator. Each call results in the generation of a single date range including the mark for :cover or :gap, which is prepended to the accumulator. As a last step, the generator uses itself to generate the subsequent date range, based on the generated end date of the current one. To eventually break out of the recursion, we decrement the count value on each iteration. Also, we still want to constrain the dates between 1970 and 2050, which is why we have the second break out condition.

Because the function needs to return a generator in any case, we wrap the accumulated list in the constant/1 generator. We’ll end up with a list of 3-tuples. Back in date_ranges_with_gaps/0, we turn this list into a map.

This is an example output of this generator:

%{
  cover: [
    {~D[2050-12-15], ~D[2050-12-17]},
    {~D[2050-03-11], ~D[2050-04-20]},
    {~D[2050-12-20], ~D[2050-12-21]},
    {~D[2046-05-22], ~D[2048-01-13]},
    {~D[2048-01-14], ~D[2048-04-11]},
    {~D[2050-12-22], ~D[2050-12-22]},
    {~D[2048-08-18], ~D[2050-03-10]},
    {~D[2050-12-14], ~D[2050-12-14]}
  ],
  gap: [
    {~D[2025-05-15], ~D[2046-05-21]},
    {~D[2050-12-18], ~D[2050-12-19]},
    {~D[2050-12-29], ~D[2050-12-29]},
    {~D[2048-04-12], ~D[2048-08-17]},
    {~D[2050-04-21], ~D[2050-12-13]},
    {~D[2050-12-23], ~D[2050-12-28]},
    {~D[2050-12-30], ~D[2050-12-31]},
    {~D[2005-10-12], ~D[2025-05-14]}
  ]
}

One shortcoming of this generator is that it does not generate any overlapping date ranges in :cover. We could actually modify it to do that by calling do_date_ranges_with_gaps/3 with both the previous start and end date and allowing the start date to be anywhere between the previous start date and one day after the previous end date, but only if the type is :cover. I’ll leave this as an exercise to the reader.

With the generator in place, we can now write the property test.

property "gaps are preserved" do
  check all date_ranges <- date_ranges_with_gaps() do
    result = merge(date_ranges[:cover])

    for {start_date, end_date} <- date_ranges[:gap] do
      assert is_nil(
               Enum.find(result, fn {result_start_date, result_end_date} ->
                 Date.to_erl(start_date) >= Date.to_erl(result_start_date) &&
                   Date.to_erl(end_date) <= Date.to_erl(result_end_date)
               end)
             )
    end
  end
end

This works exactly as the output date ranges do not overlap test, except that we assert that the gap date ranges are actually not covered by the output ranges.

The faulty implementation of merge/1 we added earlier will finally fail this test. We can now revert the function to the correct implementation to make the test suite pass again.

You can find the final version on github.com/woylie/property-based-tdd.

Conclusion

We started with a simple task and a basic property test and gradually built on that to create a test suite that should hopefully thoroughly cover the requirements.

We came up with these properties:

always returns a list
input date ranges are contained within output date ranges
output date ranges are sorted by start date
output date ranges do not overlap
adjacent date ranges are merged
min and max dates in input and output match
gaps are preserved

We saw how to write and combine custom generators, how to write recursive generators and how to write generators that are more specific to certain edge cases. We also saw how we can come up with better properties by thinking about how to trick a test suite with faulty implementations.

If you find better, cleaner, more readable or more concise solutions or if you find another faulty implementation that does not fail the tests, let me know.

Don’t run with scissors.

originally published on Medium