DEV Community: Christian Blavier

On Elixir Metaprogramming

Christian Blavier — Mon, 23 Nov 2020 17:00:00 +0000

Metaprogramming is a scary word: it sounds like voodoo for programmers, and in some manners it is. But it is used by most popular Elixir libraries (Ecto, Phoenix and Elixir itself) and as I leveled up as an Elixir programmer, I needed to understand how it worked under the hood.

I personally felt intimidated by the topic, and I owned the Metaprogramming Elixir book for more than a year before actually reading it 😱

My Elixir personal library

But for any seasoned Elixir programmer, it is an excellent handbook and very pleasant to read. It is also pragmatic and I concur 100% with Chris McCord’s introductory advice:

Rule 1 : Don't Write Macros - Chris McCord

Be cautious with metaprogramming as it is hardly maintainable and a pain to debug; don't even think about writing your application core features with it. It is more relevant for technical libraries, custom DSL and any other specific subjects.

It's a kind of magic

As many of you, I came from a Ruby on Rails development background and I remember that while learning Rails, I was so impressed by the language expressivity and its magic features. Coming then from a Java background, it was a blessing to me!

irb> 2.years.from_now + 5.days.from_now 
#=> Mon, 28 Nov 2022 13:38:36 UTC

irb> Person.find_by_user_name_and_password(user_name, password)
#=> #<Person id: 10, user_name: "Ryan">

Ruby magic in action

Rails made indeed an intensive usage of Ruby dynamic features to extend the language at runtime: the find_by_user_name_and_password call just above is an example of dynamically generated class method based on the Person model attributes.

How I felt when learning Ruby on Rails

But there are caveats to Ruby magic:

too much magic may hurt as you lack the understanding of what really happens under the hood. Same logic also applies to Elixir metaprogramming (remember Chris McCord's motto? "Don't Write Macros" 😅)
Ruby magic is dynamic by nature which means it is performed at runtime, at the expense of performance. Whereas Elixir macros are evaluated at compile time and inserted in your bytecode as if it were manually written code.

What is metaprogramming?

Before going deeper in the subject, it’s worth reminding that metaprogramming is the art of writing code that will generate code. Metaprogramming features are first class members of the Elixir language which provides macro API to rapidly extend the core language.

Did you know that if-then-else or unless block syntax are written in Elixir through its macro mechanism? It provides syntaxic sugar over a binary case pattern matching structure.

defmacro macro_unless(clause, do: expression) do
  quote do
    if(!unquote(clause), do: unquote(expression))
  end
end

unless block written with an Elixir macro

You could also be using meta-programmed features everyday without knowing it:

ExUnit test definitions leverages on macros
Ecto querying DSL can perfectly mimic SQL language thanks to metaprogramming
Of course Chris McCord’s Phoenix framework uses macro a lot, like the Routing DSL

By the way, did you know that more than 90% of the Elixir sourcecode is written in Elixir? Meta enough for you? 🤯

Elixir Macros 101

Your first Elixir macros will be a bit tedious to write. I won’t give a full course about Elixir metaprogramming (buy Chris McCord’s book!) but I will instead give you a few keys to understand it.

Can't blame Matthieu, happened to me as well!

The main thing you need to understand is that all code you write in Elixir is represented in memory as an Abstract Syntax Tree (AST). The compiler will parse your code and turn it into an AST, then the BEAM will evaluate the AST in order to run it.

Abstract Syntax Tree (AST)

Metaprogramming, which we previously defined as « code to generate code » could be achieved by generating raw Elixir code (as strings) but it would be unefficient and hardly maintainable.

That’s why Elixir provides API’s to generate and evaluate AST. Here comes quote: an Elixir primitive which can convert any Elixir code into its AST.

iex(1)> quote do 5 + 8 * 3 end
#=> {:+, [context: Elixir, import: Kernel],
#=> [5, {:*, 
#=> [context: Elixir, import: Kernel], 
#=> [8, 3]
#=> ]}]
#=> }

This code can be evaluated the same way you would have evaluated code in a string (think eval in Javascript)

iex(1)> Code.eval_string("5 + 8 * 3")
#=> {29, []}
iex(2)> Code.eval_quoted(quote do 5 + 8 * 3 end)
#=> {29, []}

Now what happens if you try to evaluate code with dynamic parts?

iex(1)> a = 5
iex(2)> Code.eval_string("3 * a")
#=> ** (CompileError) nofile:1: undefined function a/0

Of course a is unknown to your evaluated code and a simple way to fix this would be to use text interpolation.

iex(1)> a = 5
iex(2)> Code.eval_string("3 * #{a}")
#=> {15, []}

Well the good news is you can interpolate within your quoted code as well: use the unquote primitive.

iex(1)> a = 5
iex(2)> Code.eval_quoted(quote do 3 * unquote(a) end)
#=> {15, []}

Simple as that! So whenever you think you are lost between your quote / unquote blocks, just look at it as if it were text interpolation.

Then you need to know about defmacro which you probably already encountered in your Phoenix project. It's a special construct able to produce code, from quoted expressions and parameters.

Here is an example that generates a few functions :

defmodule MyMacro do
  defmacro generate_functions(name_and_values) do
    for {name, value} <- name_and_values do
      quote do
        def unquote(name)(), do: unquote(value)
      end
    end
  end
end

defmodule MyModule do
  require MyMacro

  MyMacro.generate_functions([{:one, 1}, {:two, 2}, {:three, 3}])
end

iex(1)> MyModule.two + MyModule.three 
#=> 5

There are quite a few other things to know when it comes to Elixir metaprogramming, but quote, unquote and defmacro are the basics and can already help you to build powerful stuff.

Ok, but what for?

I will end this post with some examples of the way we use metaprogramming in our Elixir projects.

Extends Ecto query API

Ecto provides a fully featured querying API that mimics SQL. It relies on Elixir Macros and will be able to trigger compilation errors whenever your query is malformed.

iex(1)> query = from u in User, where: u.age > 0, select: u.name
iex(2)> Repo.all(query)

Typical Ecto query

But every now and then you'll want to use some specific database functions not available in Ecto API. You can then use fragment Ecto function which let you insert raw SQL in your Ecto Query:

query = 
  from u in User,
  where: is_nil(fragment("?->>?", u.metadata, "phone"))

Using fragment to access a field within a Postgres JSONB column

But the proper way is to leverage on macros to extend the Ecto query API:

defmodule MyQueryMacros do
  defmacro jsonb_get(column, key) do
    quote do
      fragment("?->>?", unquote(column), unquote(key))
    end
  end
 end

 query = 
   from u in User,
   where: is_nil(jsonb_get(u.metadata, "phone"))

Same Ecto query using our homemade jsonb_get macro

Aliases & Imports

As your Elixir codebase grows, you can see that you repeat over and over the same alias and import blocks at the head of your files. Not really DRY...

Phoenix already offers a solution you can use through the use / __using__ macro. To declare a controller or a view, phoenix suggests a bunch of default aliases and imports declared in my_app_web.ex file. Here is an example of changelog.com *_web.ex file : changelog.com/changelog_web.ex file

You can then use the macro with the following snippet, which will alias and import a few Phoenix modules and even generate some authorization related functions:

defmodule MyAppWeb.MyController do
  use MyAppWeb, :controller

  # ...
end

In our own project, we also brewed our own aliasing system that we can use like this to prevent us from declaring the same aliases over and over.

defmodule MyApp.Contracts.SomeContractService do
  use MyApp,
    aliases: [:campaigns, :contracts, :users],
    private_aliases: [:contracts]

  # ...
end

I may write further about these specific macros in another post 😉

changix

A final example of how we use macros in our main Elixir application is our application changelog. We present new features, bug fixes and improvement to our users with a changelog sidebar (and a blinking badge to draw their attention).

A changelog sidebar embedded in our webapp

Instead of feeding this changelog from database, we use markdown files with a YAML front matter header.

Our markdown changelog files, commited right along source code

At first, we used to parse these files at runtime everytime a user triggered the sidebar, then we cached the HTML output in memory in order to improve performance.

But after reading Chris McCord's book, we realized that these markdown files could be considered as source code and be compiled as well (or at least processed at compile-time). Here comes changix: a library relying on macros to provide compile-time changelog features.

cblavier/changix on Github

Final words

I hope that this post will help you feel more confortable with Elixir metaprogramming and give you enough courage to hack yours; but remember Chris's advice and always be cautious 😅

Boring is the new Black

Christian Blavier — Mon, 02 Mar 2020 13:35:56 +0000

The more I get experienced as a software engineer and a developer, the more I like to keep things simple. In this article I will explain why I believe that boring - yet pragmatic - choices are often the most powerful.

Did you say "boring"?

I borrowed this term from a talk given by two Doctolib software engineers (Doctolib is a very successful french startup in the health space). In this presentation, Nicolas & Michel explain how Doctolib is dealing with 30k+ simultaneous users with a very simple and straightforward Ruby on Rails architecture: mostly a monolithic web application and a SQL database.
Doctolib software architecture
Before going deeper into software architecture, you may want to know that the boring trend has also been mentioned in quite a few other fields:

LA Times journalist Christopher Hawthorne is praising boring building architecture in his "Boring architecture? Yes please" paper.
Nick Rollins helps us to remember that a Good User Experience involves a Boring User Interface and urges not be creative when dealing with UI: don't mess with your users and just stick to the - boring - established standards.
And even Elon Musk named his company The Boring Company ;-) (terrible example, they are actually boring, ie. drilling, tunnels)

Boring architecture

Let's focus again on software architecture: a few years ago I wrote an article about the choice of not using any JS Framework when libraries such as Angular or EmberJS were on the rise, but instead leveraging on a very classic MVC architecture with a thin JS layer. I since had excellent feedbacks on this approach even if I occasionally rely on ReactJS to handle very dynamic pieces of UI.

Unfortunately it seems that my ideas are against the current flow, since average web applications kept getting more and more complex with:

a lot of different storage solutions (NoSQL)
GraphQL replacing REST APIs
The rise of micro-services architectures (M/SOA)
Containerized (Docker) and orchestrated deployments (Kubernetes)

These architectural solutions are impressive at helping large scale companies to deal with their storage or operation issues. But what drives me crazy is to see how many, seed or low-scale companies / softwares start using these technologies from the ground-up.

Micro Services architectures

Let's take an example : Micro Services. It looks so neat and clever on your PowerPoint slides, making you think you will divide and conquer on your software by splitting the big business problems you are trying to solve in much smaller pieces of software - easier to manage.

But M/SOA also induces a very important development overhead:

If you want your micro-services to talk together, you'll have to tie them up with APIs (whereas a simple function call would have been much simpler and faster).
You can't simply share code between your services. You'll have to build and release libraries shared between your micro-services, even for sharing a simple one-liner function across your services.
Micro-services also mean multiple units of deployment, which is way more complicated to release and run than a single web application. People usually end up with solutions like Kubernetes.
Versioning and release-management is really tough.

This overhead will eventually drive you away from delivering actual business value, which is not only a terrible idea for early stage companies, but a handicap for any size of business.

Document databases

Another example is the use of document oriented databases (such as MongoDB or CouchDB). Such databases are designed with very attractive benefits compared to relational databases:

you can store document (think JSON object) instead of simples rows.
database is schema-less meaning a single collection (ie. table) can hold documents of different nature (new attributes ...) without any schema migration involved.

Unfortunately, what these databases are not marketed for, is the feature they lack regarding SQL databases:

you cannot join different collections together. Meaning that if you want to reach best performance, your must really think your schema upfront considering all your use cases and potential queries to store and structure your documents in an optimal way. But your business will change and so will your queries ... leading you eventually to either sub-optimal N+1 queries or de-normalized data structures which are a nightmare to maintain, making you build over-engineered CQRS architectures (overhead!)
no ACID transaction. You can live without it 90% of the time, but the remaining 10% are a real pain to work-around with software transaction and compensation mechanisms

I used MongoDB for years and learned these lessons the hard way at the expense of days at tweaking for improved performance and using CQRS design pattern to maintain integrity between denormalized data collections.

A few years later, when came the time to build a new product I adopted PostgreSQL and it was such a relief! Postgres even became stronger over years by including new features previously only available in some noSQL system (such as JSON column datatype, partitioning and sharding, or geographic search).

The Majestic Monolith

Monolith, by Rene Aigner
Monolithic software architectures sound awkward, clumsy - of course boring - and unable to face agile and fast changing requirements. But they are quite the opposite.

Doctolib previous example has already proven me right, but so did Basecamp. In The Majestic Monolith post DHH explains how Basecamp is dealing with million of users with their large and monolithic application.

Run a small team, not a tech behemoth? Embrace the monolith and make it majestic. You Deserve It! - DHH, 2019

Building your software as a monolith doesn't mean that you don't care about architecture and are willing to build a big plate of spaghetti-ware. I sincerely believe that removing the clutter and the boilerplate of non essential technology will help you at focusing on things that really matter such as a well designed business layer and tons of unit tests.

You can totally craft well-thought and battle-tested code in a - majestic - monolithic fashion

Boring organizations

Some might argue that such a simplistic architecture does not add up very well with the size of their team or organization. Please look at it the other way around: maybe that if you started building less complicated software, you might figure out that you need smaller teams?

*"Organizations which design systems are constrained to produce designs which are copies of the communication structures of these organizations" - *Conway, 1967.

50 years ago, John Conway stated about the deep link existing between the structure of software and the structure of organizations which build them. So keep with small teams, hire jack-of-all-trades full-stack developers and build clean, simple and pragmatic software. I'm quite sure this advice should resonate with most companies building software ; not everyone is building Google or Facebook.

The patterns that make sense for organizations orders of magnitude larger than yours, are often the exact opposite ones that’ll make sense for you. It’s the essence of cargo culting. If I dance like these behemoths, surely I too will grow into one. I’m sorry, but that’s just not how the tango goes. - DHH, 2019

Wrapping up

I don't know if building boring and simple softwares with smaller teams will become a new trend in upcoming years, but it is my motto since a few years and it gives strong results until now.

You might wonder "Will I get bored at building boring stuff? . I don't think you will:

Building simple software does not mean at all being bored: you will deliver more value and joy to your users and it is among the most exciting things and what should fuel your days as a software developer.
I'm still tech savvy and really interested in new technologies and frameworks, but I'm also getting older and wiser ;-) When considering a new technology I'm always thinking about the tradeoff between the provided value and the induced overhead.

How we improved our Elixir build speed by 5x

Christian Blavier — Tue, 08 Oct 2019 20:53:00 +0000

I’m really happy and proud to say that since this story was first posted, I received great feedback and relay throughout the Elixir community! The last advice given in this article has indeed been taken as an inspiration by José Valim to add a test partitioning feature to Elixir 🤗

Some context

I have been using Elixir + Phoenix on a daily basis for 3 years now, and almost since day one, my team and I have set up continuous integration (CI) and continuous delivery (CD) processes.

Since we’re deploying our app on Heroku, we’ve been using:

SemaphoreCI as our CI tool (because SemaphoreCI is available as a Heroku add-on, simple as that!)
Heroku native pipelines to perform CD whenever CI is successful on our Git master branch

Although our CD process has always been very reliable, our CI process gradually started to under-perform, becoming both slower and unreliable. Like boiling frogs, we did not really pay attention until very recently when we had to wait almost for 1 hour before being able to deploy our code to staging : each CI run would take 15 minutes, and random test failures forced us to re-run each build a few times before getting the green light. This drove us really crazy, and led us to take action!

In this article, we’ll talk about the actions we took in order to improve an unreliable — randomly failing — 15 minutes long CI process to a solid and steady 3 minutes build.

#1 — Improve worst tests

First action is to find out the culprits: which tests are performing really slowly and which are failing randomly?

Running mix test --trace is very useful as it gives you detailed execution time, per test. But my favorite is mix test --slowest 10 which prints the list of the 10 worst performing tests. I have no general purpose advice to give about fixing these tests, but the slowest are usually quite easy to improve.

Regarding randomly failing tests, two things can really help:

a simple bash loop to run a single test over and over. If it returns anything else than 0, the test is unreliable.

(for i in {1..10}; do mix test test/lib/my_app/features/my_browser_test.exs:10 > /dev/null; echo $?; done) | paste -s -d+ - | bc

a wait_until function that helps to fix unreliable browser tests (we're using Hound with chromedriver). Here is the gist.

A last ugly trick 🙈 I am ashamed of and I urge you not to use is to give another chance to your failing tests! Running your tests with mix test || mix test --failed will automatically rerun any failed test and still give you a 0 exit code if it passes on the second time.

Pro tip: I strongly encourage you to read mix help test output: hidden gems in there!

#2 — Cache all things

At this stage, the build became way more reliable (99% success rate) but still really slow (~ 12 min).

My first attempt to improve this was to see if the grass was greener at GitHub’s with their brand new GitHub Actions, and if it would result in an improved build duration. Although promising, I did not manage to run any of our chromedriver tests with their product, and execution time was anyway quite similar.

I then decided to go back to Semaphore and migrate from the classic plan to their new SemaphoreCI 2.0. The new Semaphore (like GitHub Actions btw) requires you to describe your CI pipeline in a YAML file committed along your source code. In this YAML file you can run docker containers, arbitrary shell commands or specific semaphore commands, including the very useful cache commands.

By using cache store and cache restore at strategic locations you can make most of your builds way faster:

by not re-downloading Elixir dependencies
not re-compiling all Elixir code, only updated code
not re-downloading NPM/Yarn dependencies

Pro tip: have a look at fail_fast and auto_cancel Semaphore settings. Real time savers!

#3 —Leverage on parallelism

Caching really helped, and at this step we could have stopped improving the process as our build time already went down to 6 minutes.

But reading SemaphoreCI documentation is insightful, and their concepts of blocks and jobs that can run either sequentially or in parallel suggest there is still room for improvement!

Our build was really straight-forward then:

The first improvement is to run in parallel what can be done in parallel:

mix credo (the Elixir code analysis tool) and mix test can be run at the same time. With a fail-fast strategy that stops running tests whenever credo fails, it can be a real time saver!
JS & CSS assets can be compiled at the same time than Elixir code: it can save up to one minute.

The final step is to split our tests in different batches to also run them in parallel. I wrote this custom test runner (which simply wraps mix test shell command) that splits all test files in N batches (by hashing their filename) and only runs a single batch out of N:

# Run all tests in 4 random batches
$> mix n_test 1 4
$> mix n_test 2 4
$> mix n_test 3 4
$> mix n_test 4 4

You can then leverage on SemaphoreCI parallelism to run a single task N times with an incremental index passed as an environment variable.

Our CI finally looks like this, running at a solid and steady 3mn pace!

Pro tip: save time at learning Semaphore by reading our build pipeline setup

Wrapping up

It has been known for years how important Continuous Integration is, for any size of software team. But it should be a source of trust, happiness and increased productivity, never a source of frustration!

So start paying attention, and maybe some of the guidelines in this post will help you at improving your CI build!

NB - I was not paid by SemaphoreCI to write this article, so feel free to experiment with any other CI software, as long as it offers you caching & parallelism.

Rails with no JS framework

Christian Blavier — Fri, 28 Feb 2014 21:57:00 +0000

Because all apps don’t need Ember or Angular

I’m writing code for over a decade now, consider myself as a geek, and as a geek I’m always very curious and excited about any new fancy technology released every now and then.

These days client-side MVC frameworks are very trendy, and to be honest what I read and learned regarding Ember or Angular (among many others) looked really shiny and well-thought. These are typically frameworks that I would love to love.

But right now, I don’t need them. Do you?

The playground

In addition to my freelancing activity, I’m running Folyo (which by the way, can help you to find great designer talents). It’s as Ruby on Rails website with a UI that could be defined as slick, reactive and responsive (especially thanks to Sacha, my co-founder), but by no means complex.

As a job board, Folyo has quite a common page driven architecture where any action performed on a page will often lead you to another page or refresh current page content. Folyo’s UI does not involve rich interactions like live data charts or dynamic data binding. No, just web pages and it’s perfectly fine!

So what about client-side frameworks?

Why not, since it seems like the way to go these days?

Because such frameworks are complex tools designed to solve complex interaction issues. You must be aware that they also have serious drawbacks regarding productivity, testing, SEO, etc. (read this) and also have a steep learning curve.

Building a website with client-side MVC frameworks will require you to build your Rails server-side application as an API application only, which means depriving you of some of Rails beauty and increasing your development effort (API + UI) significantly.

Still, using a framework also brings some virtues. One of the first being to help you with organizing your code in a more manageable way. While Rails is really directive regarding the way you need to organize server-side code, it dictates only one rule for Javascript:

Put whatever you want in application.js.

Eh. Really?

Time to clean it up

I will now explain the way we decided to organize Javascript code on Folyo, in a much more manageable way than everything in application.js. I believe it’s quite pragmatic and I’m sure it will fit with a lot of other applications.

Required libraries:

jQuery, obviously.
Coffeescript, not mandatory but its syntax for class definition is very convenient.
Turbolinks. It will really improve user experience by making navigation from page to page AJAX only.
HeadJS, used through headjs-rails gem, will speed up your app by loading your JS asynchronously.

Code hierarchy

To organize the code, I will simply obey to following rules:

Folyo JS code hierarchy

Any page requiring JS code will have it’s own class (a JS view).
If you need to share piece of code between views, put it in a widget class.
application.js is just doing the glue between your classes, jQuery, HeadJS and Turbolinks.
That’s all!

Views

ApplicationView is the default view, inherited by any other view, and instantiated by default when no specific view is provided.

window.Views ||= {}
class Views.ApplicationView

 render: ->
   Widgets.FancyBox.enable()
   Widgets.MarkdownEditor.enable()

 cleanup: ->
   Widgets.FancyBox.cleanup()
   Widgets.MarkdownEditor.cleanup()

Since we want FancyBox and our Markdown editor to work on a lot of pages, we put it in the ApplicationView as a default behavior.

Then a typical view would look like that:

window.Views.Newsletters ||= {}
class Views.Newsletters.EditView extends Views.ApplicationView

 render: ->
   super()
   $('a.preview').click (e) ->
     e.preventDefault()
     url = $(e.target).attr('href')
     window.open(url, '_blank', 'width=800,height=800')

 cleanup: ->
   super()

Simple as that! But why the cleanup stuff?

Because with Turbolinks, the Javascript environment is not reset between each page. For instance, if you define a timer on a page, it will keep ticking on next pages. So, just remember to stop your timer in the cleanup method (or remove any document-wide event listener)

Widgets

Sorry, no rocket science here.

window.Widgets ||= {}

class Widgets.FancyBox
  @enable: -> $(".fancybox").fancybox()
  @cleanup: -> $(".fancybox").off()

The glue

application.js is now only the entry point which will listen to Turbolinks events to render proper views and clean them up.#= require everything you need

#= require everything you need

pageLoad = ->
  className = $('body').attr('data-class-name')
  window.applicationView = try
    eval("new #{className}()")
  catch error
    new Views.ApplicationView()
  window.applicationView.render()

head ->
  $ ->
    pageLoad()
    $(document).on 'page:load', pageLoad

    $(document).on 'page:before-change', ->
      window.applicationView.cleanup()
      true

    $(document).on 'page:restore', ->
      window.applicationView.cleanup()
      pageLoad()
      true

You also need to define a specific data attribute on each page, to indicate which JS view needs to be rendered.

In your application_controller.rb, define js_class_name method:

def js_class_name
   action = case action_name
     when 'create' then 'New'
     when 'update' then 'Edit'
    else action_name
   end.camelize
   "Views.#{self.class.name.gsub('::', '.').gsub(/Controller$/, '')}.#{action}View"
 end

And then use it in your layout (as well as HeadJS initialization)

%html
  %head
    = javascript_include_tag 'head.min'
    = headjs_include_tag 'vendor', 'application'
  %body{'data-class-name' => js_class_name}

Final words

I’m seing most and most Rails projects bootstrapped using either Angular, Ember or Backbone as their de facto choice.

Whilst these frameworks are very powerful, I hope that after reading this post you will consider that not using a JS framework is also a valid choice.