DEV Community: Peter Solnica

Introducing Elixir Drops

Peter Solnica — Wed, 25 Oct 2023 08:38:45 +0000

A few years ago my Ruby friends asked me if it would be possible to port some of the dry-rb libraries to Elixir. I remember some early community attempts at porting dry-validation specifically, I did my best to support the effort but back then my Elixir knowledge was too basic and I was too busy with other work.

Fast forward to today and I'm happy to announce the very first release of Elixir Drops! 🎉 In this article I'll introduce you to the concepts of the library and show you how you could use it.

Contracts with schemas

One of the core features of Drops is its Contract extension with the Schema DSL. It allows you to define the exact shape of the data that your system relies on and data domain validation rules that are applied to type-safe data that was processed by the schema.

There are multiple benefits of this approach:

Casting and validating data using schemas is very precise, producing detailed error messages when things go wrong
Schemas give you type safety at the boundaries of your system - you can apply a schema to external input and be 100% sure it's safe to work with
It's very easy to see the shape of data, making it easier to reason about your system as a whole
You can restrict and limit larger data structures, reducing them to simpler representations that your system needs
It's easy to convert schemas to other representations ie for documentation purposes or export them to JSON Schema format
Schemas capture both the structure and the types of data, making it easy to reuse them across your codebase
Your domain validation rules become simple and focused on their essential logic as they apply to data that meets type requirements enforced by schemas

This of course sounds very abstract, so here's a simple example of a data contract that defines a schema for a new user:

defmodule Users.Signup do
  use Drops.Contract

  schema do
    %{
      required(:name) => string(:filled?),
      optional(:age) => integer()
    }
  end
end

Users.Signup.conform(%{
  name: "Jane Doe",
  age: 42
})
# {:ok,
#  %{
#    name: "Jane Doe",
#    age: 42
#  }}

{:error, errors} = Users.Signup.conform(%{})

Enum.map(errors, &to_string/1)
# ["name key must be present"]

{:error, errors} = Users.Signup.conform(%{
  name: "",
  age: "42"
})

Enum.map(errors, &to_string/1)
# ["age must be an integer", "name must be filled"]

Let's take a closer look at what we did here:

The contract defines a schema with two keys:
- required(:name) means that the input map is expected to have the key :name
- string(:filled?) means that the :name must be a non-empty string
- optional(:age) means that the input could have the key :age
- integer() means that the :age must be an integer

Even though this is a very basic example, notice that the library does quite a lot for you - it processes the input map into a new map that includes only the keys that you specified in the schema, it checks both the keys and the values according to your specifications. When things go wrong - it gives you nice error messages making it easy to see what went wrong.

Type-safe Casting

One of the unique features of Drops Schemas is type-safe casting. Schemas breaks down the process of casting and validating data into 3 steps:

Validate the original input values
Apply optional casting functions
Validate the output values

It's better to explain this in code though:

defmodule Users.Signup do
  use Drops.Contract

  schema do
    %{
      required(:name) => string(:filled?),
      optional(:age) => cast(string(match?: ~r/\d+/)) |> integer()
    }
  end
end

Users.Signup.conform(%{
  name: "Jane Doe",
  age: "42"
})

{:error, errors} = Users.Signup.conform(%{
  name: "Jane Doe",
  age: ["oops"]
})

Enum.map(errors, &to_string/1)
# ["cast error: age must be a string"]

{:error, errors} = Users.Signup.conform(%{
  name: "Jane Doe",
  age: "oops"
})

Enum.map(errors, &to_string/1)
# ["cast error: age must have a valid format"]

Notice that when the age input value cannot be casted according to our specification, we get a nice "cast error" message and we immediately know what went wrong.

Domain validation rules

Contracts allow you to split data validation into schema validation and arbitrary domain validation rules that you can implement. Thanks to this approach we can focus on the essential logic in your rules as we don't have to worry about the types of values that the rules depend on.

Let me explain this using a simple example:

defmodule Users.Signup do
  use Drops.Contract

  schema do
    %{
      required(:name) => string(:filled?),
      required(:password) => string(:filled?),
      required(:password_confirmation) => string(:filled?)
    }
  end

  rule(:password, %{password: password, password_confirmation: password_confirmation}) do
    if password != password_confirmation do
      {:error, {[:password_confirmation], "must match password"}}
    else
      :ok
    end
  end
end

Users.Signup.conform(%{
  name: "John",
  password: "secret",
  password_confirmation: "secret"
})
# {:ok, %{name: "John", password: "secret", password_confirmation: "secret"}}

{:error, errors} = Users.Signup.conform(%{
  name: "John",
  password: "",
  password_confirmation: "secret"
})

Enum.map(errors, &to_string/1)
# ["password must be filled"]

{:error, errors} = Users.Signup.conform(%{
  name: "John",
  password: "foo",
  password_confirmation: "bar"
})

Enum.map(errors, &to_string/1)
# ["password_confirmation must match password"]

Here we check whether password and password_confirmation match but first we define in our schema that they must be both non-empty strings. Notice that our domain validation rule is not applied at all if the schema validation does not pass.

As you can probably imagine, this type of logic could be easily implemented as a higher-level macro, something like validate_confirmation_of :password. This is something that I'll most likely add to Drops in the near future.

Safe map atomization

Another very useful feature is support for atomizing input maps based on your schema definition. This means that the output map will include only the keys that you defined turned into atoms, in the case of string-based maps.

Here's an example:

defmodule Users.Signup do
  use Drops.Contract

  schema(atomize: true) do
    %{
      required(:name) => string(:filled?),
      optional(:age) => integer()
    }
  end
end

Users.Signup.conform(%{
  "name" => "Jane Doe",
  "age" => 42
})
# {:ok, %{name: "Jane Doe", age: 42}}

About the first release

Today I'm releasing v0.1.0 of Drops and even though it's the first release, it already comes with a lot of features! It's already used in our backend system at valued.app, processing and validating millions of JSON payloads regularly.

It is an early stage of development though and I encourage you to test it out and provide feedback! Here are some useful links where you can learn more:

Check it out and let me know what you think!

10 Years of Open Source

Peter Solnica — Fri, 04 Jun 2021 17:40:40 +0000

On June 4th, 2011 I released the first version of Virtus, a ruby gem that I extracted from the DataMapper project.

I remember how I felt about Open Source back then, and I have to tell you that my perspective has changed a lot.

10 years, a freaking decade, is a lot of time...I've gone through a lot of ups-and-downs during that time, as my open-source contributions sky-rocketed, and that changing perspective is something I continuously think about. I don't know how many projects I contributed to, how many libraries I've written, I honestly do not know.

This 10-year anniversary made me super emotional and I have a lot of thoughts about Open Source that I’d like to share.

This will be bitter-sweet and probably too long. Please keep in mind that this is based on my own experience and I'll do my best not to generalize too much.

How my perspective changed

I remember that in my mid 20s, when I was already working as a software developer, I had this concept in my head that Open Source is this amazing thing that everybody is doing. Maybe it sounds ridiculous but it is exactly how I used to see it. I'm a heavy-thinker and a natural problem-solver and on top of that I'm super impatient.

This was a perfect mix to become an Open Source contributor because whenever something didn't work for me I was either working on a patch 10 seconds later, or I was already half-way through prototyping my own solution.

This is exactly how I ended up contributing to DataMapper, then joining the core team, then releasing Virtus, then working on DataMapper 2.0, then turning it into rom-rb, then joining dry-rb and building 1.25 library / month on average for about 2 years or so to eventually join Hanami team...and, yeah, it's been kinda nuts now when I look back.

For the most part I kept thinking "this is what everybody does" until I realized it's absolutely not the case.

The first thing that hit me was the amount of work that's needed "just" to maintain a library. Even when you have a library that is used by, let's say, 27 people (which is a relatively low number) this can easily turn into 10+ hours of work per month, every month. If you have more than one library, and the total number of people using your stuff is now counted in hundreds or even thousands, this can easily get out of hand. There will obviously be downtime in various projects, especially the more mature ones -- but on average once you maintain something, it becomes super hard to find time to do the work that you actually want to do.

This is when things become difficult.

What's the deal here?

You're obviously responsible for managing your time and what you do and we could easily just say "well, nobody forces you to work on OSS" and end the discussion...but is it that simple?

People think that Open Source is just this one big, global family. Thousands of people working on all these amazing libraries, frameworks and tools.There's this idealistic view of Open Source that was absolutely, clearly visible to me. I feel like it's changing a bit these days but it's still there for sure.

I keep meeting people who are almost apologetically telling me that they wish they had contributed more.

My question is: what's the deal here? What makes us think that we're obliged to do this?

The way I see it is that we've created this impression ourselves as an industry. We kept telling ourselves that we need to build Open Source Software because it's the future of software (and it is!). We also kept convincing ourselves that this is how you can become a better developer (and you can!) and so some folks just double down on working harder.

Then you have companies and their expectations. It used to be - and probably still is - very common to see job offers where "contributing to OSS" was mentioned as either a "nice to have" or even a requirement. I think it's a bad practice now, except for cases where the job really does involve contributing to OSS.

I believe companies giving you extra props for OSS work contributed heavily to this notion of OSS as something that we're supposed to do.

The question, however, remains unanswered. What’s the deal here? I feel like we can all try to come up with our own answers that will work for us at an individual level. It's very hard to come up with a universal answer though.

When you have a product and a client using your product who's paying you to do your work, then the deal is simple - you work on what they need, and they pay you for it. But When it comes to Open Source this is extremely complicated and vague. What's the product? Who's the client? Can we even treat an open-source project like that? Or maybe the deal is that there is no deal and that's the beauty of it?

Let's be real

If you're a maintainer then this will probably ring some bells. If you're not and you're using OSS then this could be useful.

Let's be real - we're not one big, global, family. Most of the work is done by a tiny little fraction of the global developer population. We like to think that we're all in this together but we're not.

You can come up with great looking stats, showing that thousands of people are contributing to OSS projects -- but in reality the actual bulk of work, the countless hours spent on building core functionality, testing, fixing bugs, making performance improvements, supporting users, writing docs, responding to issues, dealing with pull requests and so on are done by maintainers.

Let me also make it very clear: all contributions count, they are important and highly appreciated.

But my point is to underscore the fact that when 1000 people did a little bit of work each, then you get significant results in total, but please realize that the time that each person spent on their contributions cannot compare to the time that maintainers spend doing their work.

To illustrate the type of situations maintainers have to deal with on a regular basis, here are some real world scenarios:

You get a bug report from somebody. You spend 3 hours debugging and coming up with a patch. You ask the user to test it out. The user never gets back to you.
You get a feature request. You spend hours discussing it. You're going through source code figuring out how it could be implemented. You end up with a nice PoC. You ask the user if it would work well. The user never gets back to you.
You get a Pull Request. You review it and it's looking good but you leave some comments because a couple of small things must be addressed. The user never gets back to you.
You get a Pull Request out of the blue with an angry user trying to point out your mistakes. You do your best to be polite but the user keeps attacking you. You spend hours in total just to discuss it. You reach some level of consensus and then guess what, the user never gets back to you.
There's this issue that somebody reported over a year ago and you want to address it eventually but you can't find time/energy/motivation/etc. to do it. It is marked as "help wanted" but people keep pinging you with comments like "any progress on this?". The uncomfortable sense of guilt starts to creep in even though it's not even your job.

...and man I could go on.

Here's something to think about: nobody has the time to work on Open Source. Nobody. There is no time, yet we somehow do it. Many of these situations are caused by the fact that a user hasn’t enough time. In other cases it's the maintainer who hasn’t enough time. We struggle to push things forward and it's often extremely unproductive.

Thank You ❤️

On a very personal level - I want to thank everybody from the Ruby community and all the people who helped me grow, who inspired me and who supported me through difficult times. Thank you for your trust and for putting up with me for so long 😆

I know this post is bitter-sweet, but overall, what I actually want to say is that I'm very grateful. I've learned so much and I'm sure I will learn even more.

The 5 Rules of Simple RSpec Tests

Peter Solnica — Wed, 12 May 2021 12:51:48 +0000

The 5 "rules" I try to follow in order to write simple RSpec tests.

Let's GO.

1. Max 2 levels of describe/context nesting

Everything above 2 is a code-smell and causes alarm bells in my head to ring. The more levels of nesting you have, the harder it is to understand what a given example is doing. If you add before/after hooks to the mix, it'll become even worse.

I often reduce nesting by simply using example descriptions like that:

RSpec.describe CreateUser do
  let(:create_user) do
    CreateUser.new
  end

  context "with valid params" do
    it "returns success" do
      expect(create_user.(name: "Jane", email: "jane@doe.org")).to be_success
    end
  end

  context "with invalid params" do
    it "returns failure when name is missing" do
      expect(create_user.(name: "", email: "jane@doe.org")).to be_failure
    end

    it "returns failure when email is missing" do
      expect(create_user.(name: "Jane", email: "")).to be_failure
    end
  end
end

Even though this could use more contexts and have attributes provided as per-context lets, I still prefer the simpler form. There are cases where let is helpful but I'm optimizing for having as little nesting levels as possible.

2. Top-down `let` definitions

When defining let statements, I do my best to organize a spec in a way that you don't have to jump from an inner describe/context block to outer blocks in order to understand the spec setup. This means that if you're reading a spec from the top, every subsequent let statement should ultimately lead you to a full understanding of the whole setup for a given spec example. A setup where an inner let overrides an outter let is a code-smell.

Here's an example of what I try to avoid:

RSpec.describe CreateUser do
  subject(:create_user) do
    CreateUser.new
  end

  context "with valid params" do
    it "returns success" do
      expect(create_user.(name: "Jane", email: "jane@doe.org")).to be_success
    end
  end

  context "with invalid params" do
    # THIS RIGHT HERE - BAD, VERY BAD `LET`
    let(:create_user) do
      CreateUser.new(some_custom: "stuff for this example")
    end

    it "returns failure when name is missing" do
      expect(create_user.(name: "", email: "jane@doe.org")).to be_failure
    end

    it "returns failure when email is missing" do
      expect(create_user.(name: "Jane", email: "")).to be_failure
    end
  end
end

3. Meaningful example descriptions

I've stopped using short DSL syntax years ago. I used to do this:

RSpec.describe CreateUser do
  subject do
    CreateUser.new
  end

  context "with valid params" do
    specify {
      expect(create_user.(name: "Jane", email: "jane@doe.org")).to be_success
    }
  end
end

You can streamline a lot with RSpec's powerful DSL but I prefer to optimize for readability and having nice descriptions is part of this. In general, I try to describe actual scenarios that can happen in the actual client code (as in, the code that will be using the thing I'm writing a spec for).

4. No mocking by default

I've stopped obsessing about "pure unit testing". This topic is big enough to have a separate post or even a series of posts about it but I'll try to summarize it here.

Using mocks in RSpec, even if you use verified doubles, is still prone to ending up with false-positivies. I feel uncomfortable every time I use a double, especially if it's not a verified double, and so I try to avoid mocking in general.

There are situations where mocking is absolutely helpful and justified - a great example is some kind of a 3rd-party authorization system. You definitely want to have it mocked in various testing scenarios.

If I were to come up with some guideline here, it would probably be this: use a mock only if using the real thing causes at least one of the following problems:

Makes tests setup significantly more complex
Makes tests significantly slower
Causes unwanted and/or problematic side-effects in external systems (ie a file system)
Depends on the internet connection

There's a lot that can be done to improve our "mocking situation" in the Ruby ecosystem in general. I've got some ideas how to achieve a truly safe and easy-to-work-with setup with mocks and stubs using our Hanami 2.0 setup, so stay tuned.

5. Higher tolerance to code duplication

There's this pretty good advise that you shouldn't "dry up" your code until you have the same concept repeated in more than 3 places. This works well but not in tests. That's why I have a much higher tolerance to duplication in tests. Reducing duplication in tests makes your setup more complex and we don't want that. Simply be more wary when reducing duplication and make sure it's worth the extra complexity.

This is especially relevant when you're using RSpec as it provides very powerful tools to reduce duplication. It's easy to overcomplicate your test suite when you go too far when trying to reduce duplication.

Wrapping up

Overall this really boils down to the famous "with great power comes great responsibility" quote, very aplicable here. RSpec has a lot of features but you don't have to use all of them all of the time. What I really appreciate about RSpec though is that it works so well out of the box and its core built-in features are very helpful. It's a powerful tool so it takes time to learn how to use it effectively!

Last but not least, this is based on my personal experience with RSpec and your experience may be different. I'm more than happy to learn about your approach to writing simple and maintainable RSpec tests.