Review of the Book: Design Elixir Systems with OTP

#elixir #phoenix #review #book

This book by Bruce Tate is, for sure, one of the best findings that I did this year.
For those of you who don't know me, my name is Thiago Ramos, I am a senior software engineer for a California based startup and I work mainly with ruby and rails.
Some years ago I discovered Elixir seeing one of Jose Valim's presentations and he got my attention. As a Brazilian, I had heard of Elixir before in 2013, but it was only in 2016 that I looked into it.

Well, life gets in the way and I never really studied Elixir. But at the end of 2019, I decided that I would learn Elixir and all the PETAL stack because I loved the language and the fact that it is functional and runs inside the Beam Virtual Machine which is a 40+ virtual machine that already solved the concurrent problems that we are mostly facing now, all of this got me thinking, hmm, I think this is a great language and ecosystem for the future. I must learn it now.

From there to now I read a lot of Elixir books including Programming Elixir, Phoenix Programming, Programming Ecto, and Functional Web Development With Elixir, OTP and Phoenix. All excellent books, especially the last one that I will do a review later.
This year I found Design Elixir Systems with OTP and I am glad I did.

This book is somewhat comparable to the Functional Web Development with Elixir in the sense that both authors code layer by layer which means that the book is structured in a way that each chapter or part of the book is responsible for one layer.
In the case of this book, the author is splitting the code into 5 layers plus two chapters related to testing.
The following is an attempt to explain a little bit of the book for those of you who don't know it or already know but are not sure to buy it.

Layers Explained

First of all, I got say that this book is not for a programmer that is learning elixir. For that I think you should go first to the Programming Elixir book, to understand the data structures, the functional language idiom using elixir and to understand how Elixir uses recursion, how do you create functions, how to use them, and other aspects of the language.

That being said I think if you understand a little bit about all those topics above you will find that this book will help you greatly. First, because it's almost a hands on book, where you can see the author implementing the code and discussing why he chooses to use Map instead of a Struct or vice-versa, you can see when he uses Tuples and when he prefers to use Keyword Lists.

What I am trying to say here is that seeing the choices being made regarding the problem at hand is great to understand better why and how to use particular data structures over others. And that's, for a person that is learning the language, it is awesome to see.

I am kind of biased because I always learned better by doing and I was never a big fan of Bible type of books anyway.
Through all the book the author is creating a Quiz system where a particular user can build quizzes through templates and can apply those quizzes to other users. There are two main core parts in the system:
The Quiz Manager is where a quiz is created, all the templates and fields populated. You can have multiple templates for a single quiz, like addition, multiple additions, and so on.

The other is the Quiz Session where a user can take a quiz. In this process, the user needs to correctly answer several questions so he can pass the quiz. All of that is configurable through the Quiz Manager. The Quiz Session will also timeout and that we will see when I talk about the Scheduler structure.

So, the author starts by creating the core of the application which consists solely of modules that contain some structs and functions. He does not talk about Ecto or changesets or schemas or anything like that, he is not worried about any persistence layer. Just the intricate relations between modules and their functions. He goes on and creates all the core structure with logic for the Quiz problem. In this part, you will have a fully functional core that you can test and even work within the console.

This approach has a name in the architecture world which is Hexagonal architecture.

In the hexagonal architecture your core business logic is in the center and all the infrastructure code is in the boundaries, which in this case means:

Processes
Supervisors
Database (Ecto)
Web Layer (Phoenix)

So, in the core the author does not create any GenServer or worries about processes or anything like that, it is just plain old Elixir modules with functions. I have some thoughts about this approach that are conflicted.

I did a lot of projects where we did the separation of concerns using the hexagonal architecture. I will do another post about my experiences on this but one thing I can say now is that you should analyze if your application is too data-centric, which means that your web layers almost always send data back and forth from databases to the web or other delivery channel. Or if your application has a high "density" of business logic.

In my experience, I think you don't need to adopt one structure and use it everywhere for all the cases in your system. I think you can understand the parts of your system that would be good to be extracted from concerns about persistence layer and delivery mechanisms and extract them so you can work on them without worry too much about database schema structures or JSON/XML structures or whatever it is your delivery mechanism.

What I see a lot is that people tend to be rigid about their early choices and create code in the system using the choices they made early on even when it is clear that it doesn't need to be that way. And we tend to see a lot of layers with no logic and layers of conversion data being used where it is most unnecessary, just because the choices were made and who the hell is brave enough to go against the system in place, right?

I am rambling here. Going back to the book.

The next step is to create the boundaries where all the GenServer processes exist.

So, with all the core done it is easy enough to create the modules that will start the processes that will use the core modules to create quizzes and make it possible for users to take the quizzes.

What I found good in this part of the book is that you can see how an experienced elixir developer thinks about it and how he approaches the creation of the servers and the API for that servers.

In this part, he goes on and creates the GenServers and the API for the GenServers which consists of functions on the module that hide the calls and the logic that is executed in the GenServer callbacks like "handle_call" and "handle_cast"
The author does a great explanation of why it is better to use GenServer.call instead of GenServer.cast and uses the Logger code in elixir to explain how they use those methods and callbacks to apply back pressure and limit the flow of data so the system continues healthy.

All GenServer.call are synchronous and the GenServer.cast are asynchronous which means that if you use only the "cast" function you can end up receiving more messages than you can process and once your mailbox is full you will start to reject messages and one away to deal with this is to use the "call" function instead. Depending on your business requirements you can create logic to use "cast" and change it to "call" if needed.
I will not spoil more than I did so far so you read the book too.

The next step he goes to create the supervisors. In this book, he creates both the configuration for static and dynamic supervisors. He does a great job explaining why we need dynamic supervisors for the Quiz Session part and why we don't need for the Quiz Manager part.
Quiz Sessions are created by the user. So each user taking a quiz is a different process in the tree.
Quiz Managers don't need to have more than one process because they are not dynamic. Once we have a manager with all the templates we can start taking quizzes using the templates already created.

Of course that he goes on to explain that this is the case for the business logic he thought about. We, of course, could have multiple Quiz Manager processes if we want to.

The last part of the book is related to the Workers. What are they exactly?

At first, I thought about them as background jobs because I am a full-time ruby/rails developer, and every time someone talks about workers in my mind they are talking about some work that is done in the background. But in Elixir every process you create has its own world there and does not share anything with any other process.

In this case, the author says that we should, in general, avoid naked processes and use Tasks instead or always work with supervisors and he goes on to write examples using the erlang library poolboy. I may be wrong here since I don't remember the words he wrote.

But in this part of the book, the author creates a module called Proctor which the basic function of the module is to schedule quizzes, start them, and terminate them when they timeout.

This layer uses all the other layers built in the book and
the code specifically creates processes and terminates them when they should be terminated.
So, the worker word is used in a meaningful way here, they are processes that do the work of schedule quizzes and create and kill quizzes processes when it should.
In a sense, this layer is adding one more process in the tree that is responsible to create processes and kill them.

What did I think of the book?

My general feeling about the book is that it is a great book to learn a lot about OTP. I learned when to create processes, the type of code that should go within, I learned how to use Supervisors to leverage a system that is reliable and scalable. I lost the fear of creating trees of processes and I got to see how an experienced developer usually thinks when coding in elixir.
Overall I am much better at elixir than I was before reading the book.

I have to be careful though because following someone's path (code) does not mean that you really get the knowledge and by now I have baggage enough to know that I need to write and write a lot of code and solve a lot of problems so I can be good at something.
Keep in mind that I am not learning how to code and I do study elixir for some years now. So I do think that if you are starting now I am not sure if Elixir is the best language for you and neither this book.

I do think that if you know the basics this book will greatly improve your Elixir idiom and you got to see pretty good tricks.

I wish you all be safe and well and let's learn Elixir.