DEV Community: Elvio Vicosa

How my side projects failed

Elvio Vicosa — Fri, 07 Jan 2022 10:22:17 +0000

This content was first published in https://elviovicosa.com/writings/reflection-failed-side-projects/

In the last couple of years, I launched some side projects. All digital. Three of them are permanently retired, and their domains are already expired. I built 2 iOS apps, one e-book, and a micro-SaaS¹.

Like many indie developers, I like the idea of building products, solving a problem, and charging money for it. When listening to some startup founders' interviews and reading TechCrunch-styled websites, it seems that coming up with an idea and turning it into a business is something simple. I found the reality different. Launching something that people are willing to pay is hard. Making it consistently is even harder. Especially if you want to bootstrap it while keeping a full-time job.

This post is a self-reflection, where I want to think about what went well, but most importantly, what went wrong with my past projects. The mistakes I should avoid and the things I must focus on when building new projects.

1. iLearn (iOS app)

iLearn was my first project. It was an iOS application i built to help people to learn and practice English.

It was fun to build it. I learned lots of things, like iOS development (Objective-C and Swift), design, and a bit about marketing. There was no planning, just building something I thought would be useful.

In a few weeks, a number of education blogs featured iLearn. It was free and people started using it. I honestly have no idea how blogs discovered the application. The only marketing I did was to put a website live. A couple of months after the launch, I was shocked to see the number of downloads. Thousands of downloads in a single day. To my surprise and amusement, i discovered the app was featured in the App Store, inside the education category.

I made no money with it. Zero. To be honest, I never had planned on making money out of it. I was happy to see people using it.

I kept the application for two years. In that time, lots of similar apps joined the App Store and iLearn got outdated. Since there weren’t many people using it, I decided to shut it down.

2. Mamae (iOS app)

Mamae (mummy in Portuguese) was my second project. It was an iOS application that I built to help new parents to track their baby's daily activities. Parents could track things like breastfeeding, pumping, sleep pattern, etc.

Since the beginning, I planned to make money with Mamae. The app had one feature free (Nursing) and all the other features were behind In-App Purchases ($1,99 each). It was the first time I sold a product online and I felt really good about it.

Motivated by sales, I invested in improving the app. I worked on stability, a feature to sync data among multiple different devices, and translation to Portuguese, Spanish and German languages.

I reached out to a few parenting blogs. Most of them wanted to charge me to share about Mamae. A couple of them shared for free. The application got some momentum and was featured by Apple. That brought lots of users, and some ended up converting to paid. The application was featured by a well-known blog about design: Abduzeedo.

The growth numbers were good. I was invited to join a Berlin Startup Incubator to expand the application. I thought about the idea for a couple of weeks but decided to not join. At the time I had no kids and I didn't feel like working full-time on it.

I put Mamae on the auto-pilot for a year. There is lots of competition in App Store and the sales drained after a few months. I made the app totally free for another year, and then shut it down.

3. Phoenix for Rails Developers (e-book)

Phoenix for Rails Developers is an e-book I published. It was the most successful project I had so far, and it's the only project that is still alive.

Back in 2012, I discovered the Erlang programming language, and I have been a big fan ever since. Around 2016 I started playing with Elixir and it was love at first sight.

After reading Nathan Barry's Authority book, I decided to write and publish a book about Elixir. More precisely, a book covering the Phoenix framework, using my Ruby on Rails background.

I invested time to build an audience around the book. I wrote blog posts, free PDF guides, etc. On the launch day, there were around 1k people on the book waiting list. I emailed them with a special 30% discount and the first sales happened less than 5 minutes after.

While writing the book I had serious doubt that anyone would buy it. Seeing people paying for it, sending emails sharing how they used the book to adopt Elixir in their companies just amazed me. It still amazes me today.

So far, the book has made a total of $16,180.40. It's broken down into two license categories: 1) Personal ($14,422.20) and 2) Company ($1,758.20).

I wrote about the writing, publishing, and marketing approach in an Indie Hacker interview.

4. Statictastic (micro-SaaS)

Statictastic was my first micro-SaaS project. The idea was simple: help companies to easily publish their Jekyll or Hugo websites. My goal was to build a tool to be used by anyone, not only developers.

The idea was simple, but the execution... I ended up building too many features. I built a CMS, a form submission tool, managed redirects, password protection, etc. All of it without never talking to potential customers. The quality of the features was not good enough. It was not a surprise to me.

Several people tried the product. A few became paid customers. When talking to some of the churned customers, the feedback was similar: they liked the features, but they were not solving their problems.

I believe I could have improved Statictastic to meet customer expectations. It could at some point become a sustainable business. But after almost one year of working on it, I was not willing to keep investing in it.

I thanked the few customers for their trust and retired the project.

Things I can learn from my past projects

Decide whether I want to build a business or learn new tools

While building Statictastic I tried to include features that were not core to the problem, just because it was fun to learn. I spent a considerable amount of time learning and researching how to build them. As a software engineer, it's easy to get distracted by tech and neglect things like marketing and customer research.

Building a business with the tools I already know is hard. Trying to build a business while learning new tools, is just too much. For my upcoming projects, it's important, to be honest from the beginning: do I want to learn new tech, or do I want to build a business? In the case of the latter, using boring technology² is the way to go.

Launching is hard, but it is the easiest part

Lost of people have ideas and plans to build something. Few people launch their ideas. Launching is hard, but launching is the easiest part. In my projects, the "building" phase is exciting. Launching is exhilarating. Keeping working on it until it grows is tough. Phoenix for Rails Developers was the only project I kept investing time in. Not surprisingly, it was also the only project that had meaningful financial returns.

I think it has a lot to do with mindset. I started most of the projects with "launching" as the final goal, and it's not. Launching is just the first step and I should plan for that.

Audience

The only project I invested time in building an audience was the one that was "successful". Was it a coincidence? I don't think so.

For upcoming projects, I want to either use the existing audience I have access to (Elixir, software development) or be ready to spend time start building a new one on the way.

There's no path to follow

Some people on the internet pretend to have the success formula³. The recipe to launch a side project following a number of steps, and voilà, success. I always questioned myself: if they knew how to do it, why are they not doing it consistently? I have no idea.

This time I won't spend time reading more books about past experiences. Instead, I want to use this time to build, test and improve my projects.

Start small

Building things is nice to please a software engineer's mind. It's fun and exciting. But it's also distracting and delays learnings.

I want to focus on the minimum necessary to start testing things. It's not about launching landing pages with a form to collect emails. It's about launching a real product, that solves a real problem well. Then, with customers, decide on how to build additional stuff.

Foot notes

Micro-SaaS is a term coined by Tyler Tringas.
Boring tech-stack for me is: Elixir, Phoenix, PostgreSQL, plain HTML+CSS, plain JS and deploying to a VPS with Ubuntu, Nginx and Let's Encrypt.
I'm more aligned with @levelsio, on launching things and seeing what gets traction.

Caching with Elixir and ETS

Elvio Vicosa — Tue, 12 Nov 2019 13:32:16 +0000

In this post, you'll learn how to use ETS as a caching mechanism in your Elixir applications, get familiar with different available options, and be made aware of some things to keep in mind.

The Concept of Caching

My two-year-old son loves eating cookies. He loves them so much that even while playing soccer, he keeps going back to the jar to grab one more.

To stop playing so as to get another cookie is not something he enjoys, so he started grabbing multiple cookies and keeping them in his hands.

That concept of keeping things that are important, but are costly to get, close to the subject that needs to use them, is commonly referred to as caching.

In computing, caching is a ubiquitous concept that is used in both hardware and software. It has been around for a long time and is a smart concept, ported from real-life situations.

How Popular Languages Deal with Caching

Popular programming languages usually rely on external dependencies such as Memcache or Redis for caching. Using these tools is almost a standard practice now.

Although that's a valid approach, it introduces yet another dependency to a project. As simple as it initially seems, the operational costs of keeping that dependency alive (running, monitoring, patching with the latest security updates, etc.), might be overwhelming when the requirements are rather simple.

Cache in Elixir (and Erlang)

In Elixir, the need for caching information is common as a way of avoiding the unnecessary hurdle of accessing information that has a high-cost tag attached to it.

Elixir comes with a huge advantage that might help simplify the lives of developers in need of caching—ETS.

ETS stands for Erlang Term Storage. It enables developers to store and access data using a key.

Here's a small example of the usage of an ETS table:

iex(4)> :ets.new(:security_level, [:named_table])
:security_level

iex(5)> :ets.insert(:security_level, {1, :high})
true

iex(6)> :ets.insert(:security_level, {2, :low})
true

iex(7)> :ets.insert(:security_level, {3, :none})
true

iex(8)> :ets.lookup(:security_level, 1)
[{1, :high}]

In that example, an ETS table named :security_level is created, and a couple of values are inserted into it.

ETS tables have four different types:

Set: That's the default type—the one used in the above example. Each key can occur only once.
Ordered set: An ordered set has the same property as the set, but ordered by Erlang/Elixir term.
Bag: An ETS using the "bag" type supports multiple items per key.
Duplicate bag: A "Duplicate bag" allows both duplicated keys and items.

The best ETS type to use depends on the specific needs of an application.

The data structures used to implement the ETS tables in the Erlang VM are optimized to provide the best possible access time. Depending on the type, the Erlang VM uses either Hash Tables or Binary Trees to represent the ETS table. When compared to linear access times for a list, both cases have better performance.

Here are two examples where I recently used ETS for caching purposes:

Feature Flags

We use continuous delivery at the company I work in, and feature flags are crucial if you're doing trunk-based-development and want to integrate new, albeit not finished code, without enabling it for customers.

We store our feature flags in AWS Parameter Store, fetching them once every 5 minutes and caching them locally in an ETS table.

Soft Real-Time Stats

There are several cases where we need to display things like "X people have purchased that plan" or "That product was already used by Y customers".

A naive approach would be to perform database queries every time such values are required, but that would be extremely heavy (and unnecessary) on our database.

We cache those values and update them every couple of minutes. Although they might not reflect the real number, they indicate a "good enough" number, which is meaningful to customers.

Things to Keep in Mind When Using ETS as a Cache Solution

Integration of Unnecessary Dependencies

ETS is part of the Erlang VM, so there are no additional dependencies necessary when using it.

Optimized Lookup

The lookup of an item (or group of items) is optimized depending on the type of the ETS table.

No Serialization

ETS tables can store Elixir data structures. This means you can store a group of %Accounts.User{...} or %PlayerScore{} and you'll be able to fetch those structures back without having to serialize them.

Not Garbage Collected

When an ETS table is no longer necessary, you must delete it manually using :ets.delete, otherwise, it might stay around forever.

A common solution is to create a table inside a process (e.g. GenServer). Since the ETS table is linked to the process that created it, if the process dies (or is terminated), the table also gets "garbage collected".

Use of ETS in Distributed Systems

ETS plays nice when using distributed Elixir, but in both the examples I shared, the projects didn't connect multiple nodes in a distributed Elixir fashion. Nevertheless, those features are part of a project deployed to several web servers, each one with its copy of ETS locally.

In both examples—"Feature Flags" and "Soft real-time stats"—ETS works well because Eventual Consistency is an acceptable side-effect.

How to Create and Use ETS Within a `GenServer`

Previously, I mentioned a case where I used ETS for "Feature Flags". In this section, I want to give you a better understanding of its internals.

The non-cached code is quite simple:

defmodule FeatureFlags do
  def enabled?(name) do
    name in get_list_of_enabled_flags()
  end

  defp get_list_of_enabled_flags() do
    # ... Access AWS, fetch values for specific project/env
  end
end

FeatureFlags.enabled?/1 returns either true if the feature flag is enabled, or false when it is not enabled.

The problem with this code lies in the fact that for every call to FeatureFlags.enabled?/1, the code performs requests to AWS and parses the response.

Since the changes to the features happen once or twice a day, we can safely cache successful returns from AWS and use the cached version.

What should we use for caching? If you guessed ETS, you're right!

Here's the cached version:

defmodule FeatureFlags do
  use GenServer

  @table :features

  def start_link(_args) do
    GenServer.start_link(__MODULE__, nil, name: __MODULE__)
  end

  def is_enabled?(name) do
    :ets.lookup_element(@table, name, 2)
  rescue
    _ -> false
  end

  def init(nil) do
    for feature_name <- get_list_of_enabled_flags() do
      :ets.insert(@table, {feature_name, true})
    end

    {:ok, nil}
  end

  defp get_list_of_enabled_flags() do
    # ... Access AWS, fetch values for specific project/env
  end
end

The first difference is the use of a GenServer. As previously mentioned, ETS tables are not automatically garbage collected. When started by a GenServer, the ETS will be garbage-collected together with the GenServer process, in case it exists.

In init/1, we fetch all the features from AWS and insert them in the ETS table. The init/1 function is called only once. After that, all the requests to the FeaturesFlags modules are going to use the cached version.

(In that example, for simplicity reasons, we are not dealing with error handling or the update of the features).

With our modified version, new calls to the FeatureFlags.enabled?/1 uses ETS to look up the values.

You might be wondering: "But since you're using a GenServer, why not use the GenServer state itself to hold the feature list instead?".

And that's a totally valid point. The main reason behind that decision is that every call to access a GenServer state (e.g. using handle_call or handle_cast), goes to the process mailbox and is processed serially. That's the reason why FeatureFlags.enabled?/1 uses the ETS directly, not calling the GenServer state.

That approach achieves two essential points:

It makes the ETS garbage collected by design and
It achieves concurrency, by not serializing requests inside the GenServer.

Summary

Developers building applications using Elixir are fortunate to have ETS available as part of the Erlang VM toolbelt.

ETS provides a right mix of simplicity (removing the need for integrating yet another tool to a project) with performance (it's a fast and battle-tested piece of software).

Keep in mind that, like anything, ETS is not suitable for every caching problem. There are cases where ETS is not a good idea, and you need to rely on other tools to solve the problem, so take this content with a grain of salt and make sure ETS meets your needs before sticking to it.

Financial Portfolio rebalancing with Elixir

Elvio Vicosa — Tue, 03 Sep 2019 21:34:06 +0000

What's a portfolio?

A portfolio is a grouping of financial assets such as stocks, bonds, commodities, currencies and cash equivalents, as well as their fund counterparts, including mutual, exchange-traded and closed funds. A portfolio can also consist of non-publicly tradable securities, like real estate, art, and private investments.

Source: investopedia

The “Permanent Portfolio”

The “Permanent Portfolio” was created by Harry Browne in the 1980s. It assumes the economy has four possible states:

Prosperity: A period of economic growth, thriving business, low unemployment rates and low-interest rates.
Inflation: A period when consumer prices generally are rising.
Deflation: The opposite of inflation.
Recession: A significant decline in economic activity spread across the economy.

Distribution

Based on those economic conditions, Browne identified the most volatile asset classes, proposing the following distribution:

Prosperity: 25% Stocks
Inflation: 25% Gold
Deflation: 25% Long term government bonds
Recession: 25% Cash

The self-regulation factor of the "Permanent Portfolio" is one of the fundamental concepts of Browne's idea. It covers the most different scenarios of the economy, looking for ways to compensate losses and increase returns. This concept makes the "Permanent Portfolio" appealing to people who don't want to be actively managing a portfolio on a daily (or even monthly) basis.

Let’s take a look at the historical prices of Gold and S&P500 (an index that measures the stock performance of the 500 large companies listed on stock exchanges in the United States).

Source: Yahoo Finance

In 2008, the year of the global financial crisis, stock investors definitively got worried about the market, and as a result, the value of the S&P index plunges. For 4 years, the S&P value stayed low, only getting back to its pre-crisis value after 2013. In case your investments were purely stocks,
it wouldn't be a fun ride.

At the same time, around 2008, the price of gold has started rising. When comparing it to the S&P value at the same time, its behaviour presents almost an inverted shape. The gold value has not only covered the stock losses but also has contributed to a profitable year.

While many investors were incredibly affected by the 2008 crisis, "Permanent Portfolio" holders had the following returns:

2008: +1.97%
2009: +7.8%
2010: +14.5%
2011: +11.36%
2012: +6.8%
2013: -2.4%
2014: +9.1%

Despite a small glitch in 2013, the returns from the "Permanent Portfolio" were always positive. Considering the whole period between 2008 and 2014, the portfolio had an average return of 7% per year.

That's one of the properties of the "Permanent Portfolio", which makes its usage so appealing to more risk-averse humans like myself.

Fast forward to 2019, many people are concerned about a new financial crisis. Germany (where I live) is about to declare "technical recession" (two consecutive quarterly contractions in the economy). If you look at the chart, you can again see the unbalanced form between S&P and gold.

Of course that by looking at this chart, I (an amateur investor) can't draw any conclusions about whether there's a recession coming next or not. What's essential to highlight is that the behaviour of the "Permanent Portfolio" is again supporting a potential crisis scenario.

Rebalancing a portfolio

The portfolio distribution (Stocks: 25%, Gold: 25%, Long term government bonds: 25%, Cash: 25%), will change over time. Some individual classes will increase, while others will decrease. That change shifts the asset classes desired allocation.

The “Permanent Portfolio” embraces the fact that the market is cyclical. There will be prosperity times where stocks value will rise
and gold will plunge, but there will also be times where the stock market is not the best option.

Rebalancing a portfolio means getting the asset classes back to their desired allocations (25% each),
by selling the ones that are overweight and buying the underweight ones.

The financial crisis in 2008 has created an unbalanced portfolio situation. It "forced" the portfolio holder to sell parts of the gold and use the profits to buy stocks (at an extremely low value) to balance it back to the desired weights.

In the long run (remember the market is cyclical), by continuously rebalancing a portfolio, you are using
the built-in volatility of the portfolio to "buy cheap and sell high".

Using Elixir to rebalance a “Permanent Portfolio”

Let's use Elixir to rebalance a permanent portfolio. For the purpose of this tutorial, we will use the market data from the free Alpha Vantage stock APIs. There are also other finance APIs you might want to consider as your stock market data source:

Sell overweight items (e.g. current allocation is higher than 25%)
Buy underweight items (e.g. current allocation is lower than 25%), using the profits from the overweight sell.
Reduce the number of operations (e.g. buy and sell), because every transaction also has a cost.
Take decisions based on real-time information.

Let's imagine the following interface:

allocations = %{"IAU" => 10, "TLT" => 10, "VGSH" => 10, "VTI" => 10}
orders = Rebalancex.rebalance_portfolio(allocations)

Based on a number of allocations (the amount of units that a specific item has), you can call the Rebalancex.rebalance_portfolio,
which will return a number of orders to be performed.

Those orders contain information if a given symbol must be bought, sold or kept as it is:

[
  [:sell, "IAU", 4],
  [:sell, "TLT", 1],
  [:sell, "VGSH", 1],
  [:buy, "VTI", 8]
]

Considering the points above, that's the code responsible for rebalancing the portfolio:

defmodule Rebalancex do
  alias Rebalancex.Quote

  @underweight 0.25
  @overweight 0.27

  def rebalance_portfolio(allocations, quote_service \\ Quote) do
    portfolio = %{cash: 0, allocations: allocations}
    prices = get_prices(allocations, quote_service)

    new_portfolio = do_rebalance_portfolio(portfolio, prices)
    create_order(portfolio, new_portfolio)
  end

  defp do_rebalance_portfolio(portfolio, prices) do
    positions = get_positions(portfolio.allocations, prices)
    weights = get_weights(positions)

    underweight_symbol = get_underweight(weights)
    overweight_symbol = get_overweight(weights)

    portfolio
    |> maybe_buy_underweight(underweight_symbol, prices)
    |> maybe_sell_overweight(overweight_symbol, prices)
    |> maybe_rebalance_again(underweight_symbol, overweight_symbol, prices)
  end

  defp get_prices(allocations, quote_service) do
    allocations
    |> Enum.reduce(%{}, fn {symbol, _}, acc ->
      Map.put(acc, symbol, quote_service.price_for(symbol))
    end)
  end

  defp get_price(prices, symbol) do
    Map.fetch!(prices, symbol)
  end

  defp get_positions(portfolio, prices) do
    portfolio
    |> Enum.reduce(%{}, fn {symbol_name, units}, acc ->
      Map.put(acc, symbol_name, get_price(prices, symbol_name) * units)
    end)
  end

  defp get_weights(positions) do
    total_value = Enum.reduce(positions, 0, fn {_, position}, acc -> position + acc end)

    positions
    |> Enum.reduce(%{}, fn {symbol_name, position}, acc ->
      Map.put(acc, symbol_name, position / total_value)
    end)
  end

  defp get_underweight(weights) do
    {symbol, _weight} =
      weights
      |> Enum.filter(fn {_, value} -> value < @underweight end)
      |> Enum.min_by(fn {_, value} -> value end, fn -> {nil, nil} end)

    symbol
  end

  defp get_overweight(weights) do
    {symbol, _weight} =
      weights
      |> Enum.filter(fn {_, value} -> value > @overweight end)
      |> Enum.max_by(fn {_, value} -> value end, fn -> {nil, nil} end)

    symbol
  end

  defp maybe_buy_underweight(portfolio, nil, _) do
    portfolio
  end

  defp maybe_buy_underweight(portfolio, symbol, prices) do
    price = get_price(prices, symbol)
    maybe_buy_underweight(portfolio, symbol, price, portfolio.cash)
  end

  defp maybe_buy_underweight(portfolio, symbol, price, cash) when cash > price do
    portfolio
    |> incr(symbol)
    |> withdraw(price)
  end

  defp maybe_buy_underweight(portfolio, _symbol, _price, _cash) do
    portfolio
  end

  defp maybe_sell_overweight(portfolio, nil, _prices) do
    portfolio
  end

  defp maybe_sell_overweight(portfolio, symbol, prices) do
    price = get_price(prices, symbol)

    portfolio
    |> decr(symbol)
    |> deposit(price)
  end

  defp maybe_rebalance_again(portfolio, nil, nil, _prices) do
    portfolio
  end

  defp maybe_rebalance_again(portfolio, _, _, prices) do
    do_rebalance_portfolio(portfolio, prices)
  end

  defp incr(%{allocations: allocations} = portfolio, symbol) do
    new_allocations = Map.put(allocations, symbol, allocations[symbol] + 1)
    %{portfolio | allocations: new_allocations}
  end

  defp decr(%{allocations: allocations} = portfolio, symbol) do
    new_allocations = Map.put(allocations, symbol, allocations[symbol] - 1)
    %{portfolio | allocations: new_allocations}
  end

  defp deposit(%{cash: cash} = portfolio, amount), do: %{portfolio | cash: cash + amount}

  defp withdraw(%{cash: cash} = portfolio, amount), do: %{portfolio | cash: cash - amount}

  defp create_order(%{allocations: old_allocations}, %{allocations: new_allocations}) do
    Enum.map(old_allocations, fn {symbol, old_units} ->
      cond do
        new_allocations[symbol] > old_units ->
          [:buy, symbol, new_allocations[symbol] - old_units]

        new_allocations[symbol] < old_units ->
          [:sell, symbol, old_units - new_allocations[symbol]]

        true ->
          [:keep, symbol]
      end
    end)
  end
end

Some things to point out:

The Rebalancex.Quote module fetches real-time prices using the AlphaVantage API.
It buys or sells single units at a time and re-evaluates how the weights were affected.
It keeps rebalancing the portfolio until there's no underweight or overweight item in it.
The order is created by comparing the original portfolio to the rebalanced one.

Demo

I created a project to visualize the evolution of a “Permanent Portfolio” value over time.
Starting with the amount of $10k, you can change different variables like monthly deposits,
rebalancing strategy, region and see the impact on the final value.

The demo project is:

An API built using Elixir and Plug.
A frontend application built using React and Redux.
Data visualization using D3.

Visit https://elviovicosa.com/permanent-portfolio
to test it live.

ActiveRecord vs. Ecto Part Two

Elvio Vicosa — Tue, 09 Oct 2018 12:41:16 +0000

This is the second part of the "ActiveRecord vs. Ecto" series, in which Batman and Batgirl fight over querying databases and we compare apples and oranges.

After looking into database schemas and migrations in ActiveRecord vs. Ecto part one, this post covers how both ActiveRecord and Ecto enable developers to query the database, and how both ActiveRecord and Ecto compare when dealing with the same requirements. Along the way, we'll also find out Batgirl's 1989-2011 identity.

Seed data

Let's get started! Based on the database structure defined in the first post of this series, assume the users and the invoices tables have the following data stored in them:

users

id	full_name	email	created_at*	updated_at
1	Bette Kane	bette@kane.test	2018-01-01 10:01:00	2018-01-01 10:01:00
2	Barbara Gordon	barbara@gordon.test	2018-01-02 10:02:00	2018-01-02 10:02:00
3	Cassandra Cain	cassandra@cain.test	2018-01-03 10:03:00	2018-01-03 10:03:00
4	Stephanie Brown	stephanie@brown.test	2018-01-04 10:04:00	2018-01-04 10:04:00

* ActiveRecord's created_at field is named inserted_at in Ecto by default.

invoices

id	user_id	payment_method	paid_at	created_at*	updated_at
1	1	Credit Card	2018-02-01 08:00:00	2018-01-02 08:00:00	2018-01-02 08:00:00
2	2	Paypal	2018-02-01 08:00:00	2018-01-03 08:00:00	2018-01-03 08:00:00
3	3			2018-01-04 08:00:00	2018-01-04 08:00:00
4	4			2018-01-05 08:00:00	2018-01-05 08:00:00

* ActiveRecord's created_at field is named inserted_at in Ecto by default.

Queries performed through this post assume that the data above is stored in the database, so keep this information in mind while reading it.

Find item using its primary key

Let's start with getting a record from the database using its primary key.

ActiveRecord

irb(main):001:0> User.find(1)
User Load (0.4ms)  SELECT  "users".* FROM "users" WHERE "users"."id" = $1 LIMIT $2  [["id", 1], ["LIMIT", 1]]
=> #<User id: 1, full_name: "Bette Kane", email: "bette@kane.test", created_at: "2018-01-01 10:01:00", updated_at: "2018-01-01 10:01:00">

Ecto

iex(3)> Repo.get(User, 1)
[debug] QUERY OK source="users" db=5.2ms decode=2.5ms queue=0.1ms
SELECT u0."id", u0."full_name", u0."email", u0."inserted_at", u0."updated_at" FROM "users" AS u0 WHERE (u0."id" = $1) [1]
%Financex.Accounts.User{
  __meta__: #Ecto.Schema.Metadata<:loaded, "users">,
  email: "bette@kane.test",
  full_name: "Bette Kane",
  id: 1,
  inserted_at: ~N[2018-01-01 10:01:00.000000],
  invoices: #Ecto.Association.NotLoaded<association :invoices is not loaded>,
  updated_at: ~N[2018-01-01 10:01:00.000000]
}

Comparison

Both cases are quite similar. ActiveRecord relies on the find class method of the User model class. It means that every ActiveRecord child class has its own find method in it.

Ecto uses a different approach, relying on the Repository concept as a mediator between the mapping layer and the domain. When using Ecto, the User module has no knowledge about how to find itself. Such responsibility is present in the Repo module, which is able to map it to the underneath datastore, which in our case is Postgres.

When comparing the SQL query itself, we can spot a few differences:

ActiveRecord loads all the fields (users.*), while Ecto loads only the fields listed in the schema definition.
ActiveRecord includes a LIMIT 1 to the query, while Ecto doesn't.

Fetching all items

Let's go a step further and load all users from the database.

ActiveRecord

irb(main):001:0> User.all
User Load (0.5ms)  SELECT  "users".* FROM "users" LIMIT $1  [["LIMIT", 11]]
=> #<ActiveRecord::Relation [#<User id: 1, full_name: "Bette Kane", email: "bette@kane.test", created_at: "2018-01-01 10:01:00", updated_at: "2018-01-01 10:01:00">, #<User id: 2, full_name: "Barbara Gordon", email: "barbara@gordon.test", created_at: "2018-01-02 10:02:00", updated_at: "2018-01-02 10:02:00">, #<User id: 3, full_name: "Cassandra Cain", email: "cassandra@cain.test", created_at: "2018-01-03 10:03:00", updated_at: "2018-01-03 10:03:00">, #<User id: 4, full_name: "Stephanie Brown", email: "stephanie@brown.test", created_at: "2018-01-04 10:04:00", updated_at: "2018-01-04 10:04:00">]>

Ecto

iex(4)> Repo.all(User)
[debug] QUERY OK source="users" db=2.8ms decode=0.2ms queue=0.2ms
SELECT u0."id", u0."full_name", u0."email", u0."inserted_at", u0."updated_at" FROM "users" AS u0 []
[
  %Financex.Accounts.User{
    __meta__: #Ecto.Schema.Metadata<:loaded, "users">,
    email: "bette@kane.test",
    full_name: "Bette Kane",
    id: 1,
    inserted_at: ~N[2018-01-01 10:01:00.000000],
    invoices: #Ecto.Association.NotLoaded<association :invoices is not loaded>,
    updated_at: ~N[2018-01-01 10:01:00.000000]
  },
  %Financex.Accounts.User{
    __meta__: #Ecto.Schema.Metadata<:loaded, "users">,
    email: "barbara@gordon.test",
    full_name: "Barbara Gordon",
    id: 2,
    inserted_at: ~N[2018-01-02 10:02:00.000000],
    invoices: #Ecto.Association.NotLoaded<association :invoices is not loaded>,
    updated_at: ~N[2018-01-02 10:02:00.000000]
  },
  %Financex.Accounts.User{
    __meta__: #Ecto.Schema.Metadata<:loaded, "users">,
    email: "cassandra@cain.test",
    full_name: "Cassandra Cain",
    id: 3,
    inserted_at: ~N[2018-01-03 10:03:00.000000],
    invoices: #Ecto.Association.NotLoaded<association :invoices is not loaded>,
    updated_at: ~N[2018-01-03 10:03:00.000000]
  },
  %Financex.Accounts.User{
    __meta__: #Ecto.Schema.Metadata<:loaded, "users">,
    email: "stephanie@brown.test",
    full_name: "Stephanie Brown",
    id: 4,
    inserted_at: ~N[2018-01-04 10:04:00.000000],
    invoices: #Ecto.Association.NotLoaded<association :invoices is not loaded>,
    updated_at: ~N[2018-01-04 10:04:00.000000]
  }
]

Comparison

It follows the exact same pattern as the previous section. ActiveRecord uses the all class method and Ecto relies on the repository pattern to load the records.

There are again some differences in the SQL queries:

The same as the previous section, ActiveRecord loads all the fields (users.*), while Ecto loads only the fields listed in the schema definition.
ActiveRecord also defines a LIMIT 11, while Ecto simply loads everything. This limit comes from the inspect method used on the console (https://github.com/rails/rails/blob/master/activerecord/lib/active_record/relation.rb#L599).

Querying with conditions

It's very unlikely that we need to fetch all the records from a table. A common need is the use of conditions, to filter out the data returned.

Let's use that example to list all the invoices which are still to be paid (WHERE paid_at IS NULL).

ActiveRecord

irb(main):024:0> Invoice.where(paid_at: nil)
Invoice Load (18.2ms)  SELECT  "invoices".* FROM "invoices" WHERE "invoices"."paid_at" IS NULL LIMIT $1  [["LIMIT", 11]]
=> #<ActiveRecord::Relation [#<Invoice id: 3, user_id: 3, payment_method: nil, paid_at: nil, created_at: "2018-01-04 08:00:00", updated_at: "2018-01-04 08:00:00">, #<Invoice id: 4, user_id: 4, payment_method: nil, paid_at: nil, created_at: "2018-01-05 08:00:00", updated_at: "2018-01-05 08:00:00">]>

Ecto

iex(19)> where(Invoice, [i], is_nil(i.paid_at)) |> Repo.all()
[debug] QUERY OK source="invoices" db=20.2ms
SELECT i0."id", i0."payment_method", i0."paid_at", i0."user_id", i0."inserted_at", i0."updated_at" FROM "invoices" AS i0 WHERE (i0."paid_at" IS NULL) []
[
  %Financex.Accounts.Invoice{
    __meta__: #Ecto.Schema.Metadata<:loaded, "invoices">,
    id: 3,
    inserted_at: ~N[2018-01-04 08:00:00.000000],
    paid_at: nil,
    payment_method: nil,
    updated_at: ~N[2018-01-04 08:00:00.000000],
    user: #Ecto.Association.NotLoaded<association :user is not loaded>,
    user_id: 3
  },
  %Financex.Accounts.Invoice{
    __meta__: #Ecto.Schema.Metadata<:loaded, "invoices">,
    id: 4,
    inserted_at: ~N[2018-01-04 08:00:00.000000],
    paid_at: nil,
    payment_method: nil,
    updated_at: ~N[2018-01-04 08:00:00.000000],
    user: #Ecto.Association.NotLoaded<association :user is not loaded>,
    user_id: 4
  }
]

Comparison

In both examples, the where keyword is used, which is a connection to the SQL WHERE clause. Although the generated SQL queries are quite similar, the way how both tools get there have some important differences.

ActiveRecord transforms the paid_at: nil argument to the paid_at IS NULL SQL statement automatically. In order to get to the same output using Ecto, developers need to be more explicit about their intent, by calling the is_nil().

Another difference to be highlighted is the "pure" behaviour of the function where in Ecto. When calling the where function alone, it doesn't not interact with the database. The return of the where function is a Ecto.Query struct:

iex(20)> where(Invoice, [i], is_nil(i.paid_at))
#Ecto.Query<from i in Financex.Accounts.Invoice, where: is_nil(i.paid_at)>

The database is only touched when the Repo.all() function is called, passing the Ecto.Query struct as argument. This approach allows query composition in Ecto, which is the subject of the next section.

Query composition

One of the most powerful aspects of database queries is composition. It describing a query in a way that contains more than a single condition.

If you are building raw SQL queries, it means you'll probably use some kind of concatenation. Imagine you have two conditions:

not_paid = 'paid_at IS NOT NULL'
paid_with_paypal = 'payment_method = "Paypal"'

In order to combine those two conditions using raw SQL, means you'll have to concatenate them using something similar to:

SELECT * FROM invoices WHERE #{not_paid} AND #{paid_with_paypal}

Luckily both ActiveRecord and Ecto have a solution for that.

ActiveRecord

irb(main):003:0> Invoice.where.not(paid_at: nil).where(payment_method: "Paypal")
Invoice Load (8.0ms)  SELECT  "invoices".* FROM "invoices" WHERE "invoices"."paid_at" IS NOT NULL AND "invoices"."payment_method" = $1 LIMIT $2  [["payment_method", "Paypal"], ["LIMIT", 11]]
=> #<ActiveRecord::Relation [#<Invoice id: 2, user_id: 2, payment_method: "Paypal", paid_at: "2018-02-01 08:00:00", created_at: "2018-01-03 08:00:00", updated_at: "2018-01-03 08:00:00">]>

Ecto

iex(6)> Invoice |> where([i], not is_nil(i.paid_at)) |> where([i], i.payment_method == "Paypal") |> Repo.all()
[debug] QUERY OK source="invoices" db=30.0ms decode=0.6ms queue=0.2ms
SELECT i0."id", i0."payment_method", i0."paid_at", i0."user_id", i0."inserted_at", i0."updated_at" FROM "invoices" AS i0 WHERE (NOT (i0."paid_at" IS NULL)) AND (i0."payment_method" = 'Paypal') []
[
  %Financex.Accounts.Invoice{
    __meta__: #Ecto.Schema.Metadata<:loaded, "invoices">,
    id: 2,
    inserted_at: ~N[2018-01-03 08:00:00.000000],
    paid_at: #DateTime<2018-02-01 08:00:00.000000Z>,
    payment_method: "Paypal",
    updated_at: ~N[2018-01-03 08:00:00.000000],
    user: #Ecto.Association.NotLoaded<association :user is not loaded>,
    user_id: 2
  }
]

Comparison

Both queries are answering the same question: "Which invoices were paid and used Paypal?".

As already expected, ActiveRecord offers a more succinct way of composing the query (for that example), while Ecto requires developers to spend a bit more on writing the query. As usual, Batgirl (the Orphan, mute one with the Cassandra Cain identity) or Activerecord is not as verbose.

Don't be fooled by the verbosity and apparent complexity of the Ecto query shown above. In a real world environment, that query would be rewritten to look more like:

Invoice
|> where([i], not is_nil(i.paid_at))
|> where([i], i.payment_method == "Paypal")
|> Repo.all()

Seeing from that angle, the combination of the "pure" aspects of the function where, which does not perform database operations by itself, with the pipe operator, makes query composition in Ecto really clean.

Ordering

Ordering is an important aspect of a query. It enables developers to ensure that a given query result follows a specified order.

ActiveRecord

irb(main):002:0> Invoice.order(created_at: :desc)
Invoice Load (1.5ms)  SELECT  "invoices".* FROM "invoices" ORDER BY "invoices"."created_at" DESC LIMIT $1  [["LIMIT", 11]]
=> #<ActiveRecord::Relation [#<Invoice id: 4, user_id: 4, payment_method: nil, paid_at: nil, created_at: "2018-01-05 08:00:00", updated_at: "2018-01-05 08:00:00">, #<Invoice id: 3, user_id: 3, payment_method: nil, paid_at: nil, created_at: "2018-01-04 08:00:00", updated_at: "2018-01-04 08:00:00">, #<Invoice id: 2, user_id: 2, payment_method: "Paypal", paid_at: "2018-02-01 08:00:00", created_at: "2018-01-03 08:00:00", updated_at: "2018-01-03 08:00:00">, #<Invoice id: 1, user_id: 1, payment_method: "Credit Card", paid_at: "2018-02-01 08:00:00", created_at: "2018-01-02 08:00:00", updated_at: "2018-01-02 08:00:00">]>

Ecto

iex(6)> order_by(Invoice, desc: :inserted_at) |> Repo.all()
[debug] QUERY OK source="invoices" db=19.8ms
SELECT i0."id", i0."payment_method", i0."paid_at", i0."user_id", i0."inserted_at", i0."updated_at" FROM "invoices" AS i0 ORDER BY i0."inserted_at" DESC []
[
  %Financex.Accounts.Invoice{
    __meta__: #Ecto.Schema.Metadata<:loaded, "invoices">,
    id: 3,
    inserted_at: ~N[2018-01-04 08:00:00.000000],
    paid_at: nil,
    payment_method: nil,
    updated_at: ~N[2018-01-04 08:00:00.000000],
    user: #Ecto.Association.NotLoaded<association :user is not loaded>,
    user_id: 3
  },
  %Financex.Accounts.Invoice{
    __meta__: #Ecto.Schema.Metadata<:loaded, "invoices">,
    id: 4,
    inserted_at: ~N[2018-01-04 08:00:00.000000],
    paid_at: nil,
    payment_method: nil,
    updated_at: ~N[2018-01-04 08:00:00.000000],
    user: #Ecto.Association.NotLoaded<association :user is not loaded>,
    user_id: 4
  },
  %Financex.Accounts.Invoice{
    __meta__: #Ecto.Schema.Metadata<:loaded, "invoices">,
    id: 2,
    inserted_at: ~N[2018-01-03 08:00:00.000000],
    paid_at: #DateTime<2018-02-01 08:00:00.000000Z>,
    payment_method: "Paypal",
    updated_at: ~N[2018-01-03 08:00:00.000000],
    user: #Ecto.Association.NotLoaded<association :user is not loaded>,
    user_id: 2
  },
  %Financex.Accounts.Invoice{
    __meta__: #Ecto.Schema.Metadata<:loaded, "invoices">,
    id: 1,
    inserted_at: ~N[2018-01-02 08:00:00.000000],
    paid_at: #DateTime<2018-02-01 08:00:00.000000Z>,
    payment_method: "Credit Card",
    updated_at: ~N[2018-01-02 08:00:00.000000],
    user: #Ecto.Association.NotLoaded<association :user is not loaded>,
    user_id: 1
  }
]

Comparison

Adding order to a query is straight-forward in both tools.

Although the Ecto example uses a Invoice as first parameter, the order_by function also accepts Ecto.Query structs, which enables the order_by function to be used in compositions, like:

Invoice
|> where([i], not is_nil(i.paid_at))
|> where([i], i.payment_method == "Paypal")
|> order_by(desc: :inserted_at)
|> Repo.all()

Limiting

What would be a database without limit? A disaster. Luckily, both ActiveRecord and Ecto help to limit the number of returned records.

ActiveRecord

irb(main):004:0> Invoice.limit(2)
Invoice Load (0.2ms)  SELECT  "invoices".* FROM "invoices" LIMIT $1  [["LIMIT", 2]]
=> #<ActiveRecord::Relation [#<Invoice id: 1, user_id: 1, payment_method: "Credit Card", paid_at: "2018-02-01 08:00:00", created_at: "2018-01-02 08:00:00", updated_at: "2018-01-02 08:00:00">, #<Invoice id: 2, user_id: 2, payment_method: "Paypal", paid_at: "2018-02-01 08:00:00", created_at: "2018-01-03 08:00:00", updated_at: "2018-01-03 08:00:00">]>

Ecto

iex(22)> limit(Invoice, 2) |> Repo.all()
[debug] QUERY OK source="invoices" db=3.6ms
SELECT i0."id", i0."payment_method", i0."paid_at", i0."user_id", i0."inserted_at", i0."updated_at" FROM "invoices" AS i0 LIMIT 2 []
[
  %Financex.Accounts.Invoice{
    __meta__: #Ecto.Schema.Metadata<:loaded, "invoices">,
    id: 1,
    inserted_at: ~N[2018-01-02 08:00:00.000000],
    paid_at: #DateTime<2018-02-01 08:00:00.000000Z>,
    payment_method: "Credit Card",
    updated_at: ~N[2018-01-02 08:00:00.000000],
    user: #Ecto.Association.NotLoaded<association :user is not loaded>,
    user_id: 1
  },
  %Financex.Accounts.Invoice{
    __meta__: #Ecto.Schema.Metadata<:loaded, "invoices">,
    id: 2,
    inserted_at: ~N[2018-01-03 08:00:00.000000],
    paid_at: #DateTime<2018-02-01 08:00:00.000000Z>,
    payment_method: "Paypal",
    updated_at: ~N[2018-01-03 08:00:00.000000],
    user: #Ecto.Association.NotLoaded<association :user is not loaded>,
    user_id: 2
  }
]

Comparison

Both ActiveRecord and Ecto have a way of limiting the number of records returned by a query.

Ecto's limit works similarly to order_by, being suitable for query compositions.

Associations

ActiveRecord and Ecto have different approaches when it comes to how associations are handled.

ActiveRecord

In ActiveRecord, you can use any association defined in a model, without having to do anything special about that, for example:

irb(main):012:0> user = User.find(2)
User Load (0.3ms)  SELECT  "users".* FROM "users" WHERE "users"."id" = $1 LIMIT $2  [["id", 2], ["LIMIT", 1]]
=> #<User id: 2, full_name: "Barbara Gordon", email: "barbara@gordon.test", created_at: "2018-01-02 10:02:00", updated_at: "2018-01-02 10:02:00">
irb(main):013:0> user.invoices
Invoice Load (0.4ms)  SELECT  "invoices".* FROM "invoices" WHERE "invoices"."user_id" = $1 LIMIT $2  [["user_id", 2], ["LIMIT", 11]]
=> #<ActiveRecord::Associations::CollectionProxy [#<Invoice id: 2, user_id: 2, payment_method: "Paypal", paid_at: "2018-02-01 08:00:00", created_at: "2018-01-03 08:00:00", updated_at: "2018-01-03 08:00:00">]>

The example above shows that we can get a list of the user invoices when calling user.invoices. When doing so, ActiveRecord automatically queried the database and loaded the invoices that are associated with the user. While this approach makes things easier, in the sense of writing less code or having to worry about extra steps, it might be a problem if you are iterating over a number of users and fetching the invoices for each user. This issue is known as the "N + 1 problem".

In ActiveRecord, the proposed fix to the "N + 1 problem" is to use the includes method:

irb(main):022:0> user = User.includes(:invoices).find(2)
User Load (0.3ms)  SELECT  "users".* FROM "users" WHERE "users"."id" = $1 LIMIT $2  [["id", 2], ["LIMIT", 1]]
Invoice Load (0.6ms)  SELECT "invoices".* FROM "invoices" WHERE "invoices"."user_id" = $1  [["user_id", 2]]
=> #<User id: 2, full_name: "Barbara Gordon", email: "barbara@gordon.test", created_at: "2018-01-02 10:02:00", updated_at: "2018-01-02 10:02:00">
irb(main):023:0> user.invoices
=> #<ActiveRecord::Associations::CollectionProxy [#<Invoice id: 2, user_id: 2, payment_method: "Paypal", paid_at: "2018-02-01 08:00:00", created_at: "2018-01-03 08:00:00", updated_at: "2018-01-03 08:00:00">]>

In this case, ActiveRecord eager-loads the invoices association when fetching the user (as seen in the two SQL queries shown).

Ecto

As you might already have noticed, Ecto really doesn't like magic or implicitness. It requires developers to be explicit about their intents.

Let's try the same approach of using user.invoices with Ecto:

iex(7)> user = Repo.get(User, 2)
[debug] QUERY OK source="users" db=18.3ms decode=0.6ms
SELECT u0."id", u0."full_name", u0."email", u0."inserted_at", u0."updated_at" FROM "users" AS u0 WHERE (u0."id" = $1) [2]
%Financex.Accounts.User{
  __meta__: #Ecto.Schema.Metadata<:loaded, "users">,
  email: "barbara@gordon.test",
  full_name: "Barbara Gordon",
  id: 2,
  inserted_at: ~N[2018-01-02 10:02:00.000000],
  invoices: #Ecto.Association.NotLoaded<association :invoices is not loaded>,
  updated_at: ~N[2018-01-02 10:02:00.000000]
}
iex(8)> user.invoices
#Ecto.Association.NotLoaded<association :invoices is not loaded>

The result is a Ecto.Association.NotLoaded. Not so useful.

To have access to the invoices, a developer needs to let Ecto know about that, using the preload function:

iex(12)> user = preload(User, :invoices) |> Repo.get(2)
[debug] QUERY OK source="users" db=11.8ms
SELECT u0."id", u0."full_name", u0."email", u0."inserted_at", u0."updated_at" FROM "users" AS u0 WHERE (u0."id" = $1) [2]
[debug] QUERY OK source="invoices" db=4.2ms
SELECT i0."id", i0."payment_method", i0."paid_at", i0."user_id", i0."inserted_at", i0."updated_at", i0."user_id" FROM "invoices" AS i0 WHERE (i0."user_id" = $1) ORDER BY i0."user_id" [2]
%Financex.Accounts.User{
  __meta__: #Ecto.Schema.Metadata<:loaded, "users">,
  email: "barbara@gordon.test",
  full_name: "Barbara Gordon",
  id: 2,
  inserted_at: ~N[2018-01-02 10:02:00.000000],
  invoices: [
    %Financex.Accounts.Invoice{
      __meta__: #Ecto.Schema.Metadata<:loaded, "invoices">,
      id: 2,
      inserted_at: ~N[2018-01-03 08:00:00.000000],
      paid_at: #DateTime<2018-02-01 08:00:00.000000Z>,
      payment_method: "Paypal",
      updated_at: ~N[2018-01-03 08:00:00.000000],
      user: #Ecto.Association.NotLoaded<association :user is not loaded>,
      user_id: 2
    }
  ],
  updated_at: ~N[2018-01-02 10:02:00.000000]
}

iex(15)> user.invoices
[
  %Financex.Accounts.Invoice{
    __meta__: #Ecto.Schema.Metadata<:loaded, "invoices">,
    id: 2,
    inserted_at: ~N[2018-01-03 08:00:00.000000],
    paid_at: #DateTime<2018-02-01 08:00:00.000000Z>,
    payment_method: "Paypal",
    updated_at: ~N[2018-01-03 08:00:00.000000],
    user: #Ecto.Association.NotLoaded<association :user is not loaded>,
    user_id: 2
  }
]

Similarly to ActiveRecord includes, the preload with fetch the associated invoices, which will make them available when calling user.invoices.

Comparison

Once again, the battle between ActiveRecord and Ecto ends up with a known-point: explicitness. Both tools enable developers to easily access associations, but while ActiveRecord makes it less verbose, the result of it might have unexpected behaviours. Ecto follows the WYSIWYG kind of approach, which only does what is seen in the query defined by the developer.

Rails is well-known for using and promoting caching strategies to all the different layers of the application. One example is about using the "Russian doll" caching approach, which relies entirely on the "N + 1 problem" for its caching mechanism to perform its magic.

Validations

Most validations present in ActiveRecord are also available in Ecto. Here's a list of common validations and how both ActiveRecord and Ecto define them:

ActiveRecord	Ecto
`validates :title, presence: true`	`validate_required(changeset, [:title])`
`validates :email, confirmation: true`	`validate_confirmation(changeset, :email)`
`validates :email, format: {with: /@/`}	`validate_format(changeset, :email, ~r/@/)`
`validates :start, exclusion: {in: %w(a b)}`	`validate_exclusion(changeset, :start, ~w(a b))`
`validates :start, inclusion: {in: %w(a b)}`	`validate_inclusion(changeset, :start, ~w(a b))`
`validates :terms_of_service, acceptance: true`	`validate_acceptance(changeset, :terms_of_service)`
`validates :password, length: {is: 6}`	`validate_length(changeset, :password, is: 6)`
`validates :age, numericality: {equal_to: 1}`	`validate_number(changeset, :age, equal_to: 1)`

Wrap up

There you have it: the essential apples versus oranges comparison.

ActiveRecord focuses on the ease of performing database queries. The great majority of its features are concentrated on the model classes themselves, not requiring developers to have a deep understanding of the database, nor the impact of such operations. ActiveRecord does lots of things implicitly by default. Although that makes it easier to get started, it makes it harder to understand what is happening behind the scenes and it only works if you follow the "ActiveRecord way".

Ecto, on the other hand, requires explicitness which results in more verbose code. As a benefit, everything is in the spotlight, nothing behind the scenes, and you can specify your own way.

Both have their upside depending on your perspective and preference. So having compared apples and oranges, we come to the end of this BAT-tle. Almost forgot to tell you BatGirl's codename (1989 - 2001) was .... Oracle. But let's not go into that. 😉

This post is written by guest author Elvio Vicosa. Elvio is the author of the book Phoenix for Rails Developers.

ActiveRecord vs. Ecto - Part One

Elvio Vicosa — Fri, 28 Sep 2018 11:22:43 +0000

Data is a core part of most software applications. Mapping and querying data from a database is a recurring task in the life of a developer. Because of this, it is important to understand the process and be able to use abstractions that simplify the task.

In this post, the first of a series of two, you'll find a comparison between ActiveRecord (Ruby) and Ecto (Elixir). We'll see how both tools enable developers to migrate and map database schemas.

So we’ll be comparing Apples and Oranges. (Original) Batgirl, who never needed to say a word, versus Batman, explicitly stating 'I'm Batman'. Implicit, convention over configuration, versus Explicit intention. Round one. Fight!

ActiveRecord

With more than 10 years since its release, chances are, you've already heard about ActiveRecord - the famous ORM that is shipped by default with Ruby on Rails projects.

ActiveRecord is the M in MVC - the model - which is the layer of the system responsible for representing business data and logic. ActiveRecord facilitates the creation and use of business objects whose data requires persistent storage in a database. It is an implementation of the ActiveRecord pattern which is itself, a description of an Object Relational Mapping system.

Although it is mostly known to be used with Rails, ActiveRecord can also be used as a standalone tool, getting embedded in other projects.

Ecto

When compared to ActiveRecord, Ecto is a quite new (and at the moment not as famous) tool. It is written in Elixir and is included by default in Phoenix projects.

Unlike ActiveRecord, Ecto is not an ORM, but a library that enables the use of Elixir to write queries and interact with the database.

Ecto is a domain specific language for writing queries and interacting with databases in Elixir.

By design, Ecto is a standalone tool, being used in different Elixir projects and not connected to any framework.

Aren't you Comparing Apples and Oranges?

Yes we are! Although ActiveRecord and Ecto are semantically different, but common features like database migrations, database mappings, queries and validations are supported by both ActiveRecord and Ecto. And we can achieve the same results are achieved using both tools. For those interested in Elixir coming from a Ruby background we thought this would be an interesting comparison.

The Invoice System

Throughout the rest of the post, a hypothetical invoice system will be used for demonstration. Let's imagine we have a store selling suits to super heroes. To keep things simple, we'll only have two tables for the invoice system: users and invoices.

Below is the structure of those tables, with their fields and types:

users

Field	Type
full_name	string
email	string
created_at (ActiveRecord) / inserted_at (Ecto)	datetime
updated_at	datetime

invoices

Field	Type
user_id	integer
payment_method	string
paid_at	datetime
created_at (ActiveRecord) / inserted_at (Ecto)	datetime
updated_at	datetime

The users table has four fields: full_name, email, updated_at and a fourth field that is dependent on the tool used. ActiveRecord creates a created_at field while Ecto creates an inserted_at field to represent the timestamp of the moment the record was first inserted in the database.

The second table is named invoices. It has five fields: user_id, payment_method, paid_at, updated_at and, similar to the users table, either created_at or inserted_at, depending on the tool used.

The users and invoices tables have the following associations:

A user has many invoices
An invoice belongs to a user

Migrations

Migrations allow developers to easily evolve their database schema over time, using an iterative process. Both ActiveRecord and Ecto enable developers to migrate database schema using a high-level language (Ruby and Elixir respectively), instead of directly dealing with SQL.

Let's take a look at how migrations work in ActiveRecord and Ecto by using them to create the users and invoices tables.

ActiveRecord: Creating the Users Table

Migration

class CreateUsers < ActiveRecord::Migration[5.2]
  def change
    create_table :users do |t|
      t.string :full_name, null: false
      t.string :email, index: {unique: true}, null: false
      t.timestamps
    end
  end
end

ActiveRecord migrations enable the creation of tables using the create_table method. Although the created_at and updated_at fields are not defined in the migration file, the use of t.timestamps triggers ActiveRecord to create both.

Created Table Structure

After running the CreateUsers migration, the created table will have the following structure:

   Column   |            Type             | Nullable |              Default
------------+-----------------------------+----------+-----------------------------------
 id         | bigint                      | not null | nextval('users_id_seq'::regclass)
 full_name  | character varying           | not null |
 email      | character varying           | not null |
 created_at | timestamp without time zone | not null |
 updated_at | timestamp without time zone | not null |
Indexes:
    "users_pkey" PRIMARY KEY, btree (id)
    "index_users_on_email" UNIQUE, btree (email)

The migration is also responsible for the creation of a unique index for the email field. The option index: {unique: true} is passed to the email field definition. This is why the table has listed the "index_users_on_email" UNIQUE, btree (email) index as part of its structure.

Ecto: Creating the Users Table

Migration

defmodule Financex.Repo.Migrations.CreateUsers do
  use Ecto.Migration

  def change do
    create table(:users) do
      add :full_name, :string, null: false
      add :email, :string, null: false
      timestamps()
    end

    create index(:users, [:email], unique: true)
  end
end

The Ecto migration combines the functions create() and table() to create the users table. The Ecto migration file is quite similar to its ActiveRecord equivalent. In ActiveRecord the timestamps fields (created_at and updated_at) are created by t.timestamps while in Ecto the timestamps fields (inserted_at and updated_at) are created by the timestamps() function.

There's a small difference between both tools on how indexes are created. In ActiveRecord, the index is defined as an option to the field being created. Ecto uses the combination of the functions create() and index() to achieve that, consistent with how the combination is used to create the table itself.

Created Table Structure

   Column    |            Type             | Nullable |              Default
-------------+-----------------------------+----------+-----------------------------------
 id          | bigint                      | not null | nextval('users_id_seq'::regclass)
 full_name   | character varying(255)      | not null |
 email       | character varying(255)      | not null |
 inserted_at | timestamp without time zone | not null |
 updated_at  | timestamp without time zone | not null |
Indexes:
    "users_pkey" PRIMARY KEY, btree (id)
    "users_email_index" UNIQUE, btree (email)

The table created on running the Financex.Repo.Migrations.CreateUsers migration has an identical structure to the table created using ActiveRecord.

ActiveRecord: Creating the `invoices` Table

Migration

class CreateInvoices < ActiveRecord::Migration[5.2]
  def change
    create_table :invoices do |t|
      t.references :user
      t.string :payment_method
      t.datetime :paid_at
      t.timestamps
    end
  end
end

This migration includes the t.references method, that wasn't present in the previous one. It is used to create a reference to the users table. As described earlier, a user has many invoices and an invoice belongs to a user. The t.references method creates a user_id column in the invoices table to hold that reference.

Created Table Structure

     Column     |            Type             | Nullable |               Default
----------------+-----------------------------+----------+--------------------------------------
 id             | bigint                      | not null | nextval('invoices_id_seq'::regclass)
 user_id        | bigint                      |          |
 payment_method | character varying           |          |
 paid_at        | timestamp without time zone |          |
 created_at     | timestamp without time zone | not null |
 updated_at     | timestamp without time zone | not null |
Indexes:
    "invoices_pkey" PRIMARY KEY, btree (id)
    "index_invoices_on_user_id" btree (user_id)

The created table follows the same patterns as the previously created table. The only difference is an extra index (index_invoices_on_user_id), which ActiveRecord automatically adds when the t.references method is used.

Ecto: Creating the `invoices` Table

Migration

defmodule Financex.Repo.Migrations.CreateInvoices do
  use Ecto.Migration

  def change do
    create table(:invoices) do
      add :user_id, references(:users)
      add :payment_method, :string
      add :paid_at, :utc_datetime
      timestamps()
    end

    create index(:invoices, [:user_id])
  end
end

Ecto also supports the creation of database references, by using the references() function. Unlike ActiveRecord, which infers the column name, Ecto requires the developer to explicitly define the user_id column name. The references() function also requires the developer to explicitly define the table the reference is pointing to, which in this example, is the users table.

Created Table Structure

     Column     |            Type             | Nullable |               Default
----------------+-----------------------------+----------+--------------------------------------
 id             | bigint                      | not null | nextval('invoices_id_seq'::regclass)
 user_id        | bigint                      |          |
 payment_method | character varying(255)      |          |
 paid_at        | timestamp without time zone |          |
 inserted_at    | timestamp without time zone | not null |
 updated_at     | timestamp without time zone | not null |

Indexes:
    "invoices_pkey" PRIMARY KEY, btree (id)
    "invoices_user_id_index" btree (user_id)
Foreign-key constraints:
    "invoices_user_id_fkey" FOREIGN KEY (user_id) REFERENCES users(id)

Both migrations are also quite similar. When it comes to the way the references feature is handled, there are a few differences:

Ecto creates a foreign-key constraint to the user_id field ("invoices_user_id_fkey" FOREIGN KEY (user_id) REFERENCES users(id)), which maintains the referential integrity between the users and invoices tables.
ActiveRecord automatically creates an index for the user_id column. Ecto requires the developer to be explicit about that. This is why the migration has the create index(:invoices, [:user_id]) statement.

ActiveRecord: Data Mapping & Associations

ActiveRecord is known for its "conventions over configurations" motto. It infers the database table names using the model class name, by default. A class named User, by default, uses the users table as its source. ActiveRecord also maps all the columns of the table as an instance attribute. Developers are only required to define the associations among the tables. These are also used by ActiveRecord to infer the involved classes and tables.

Take a look at how the users and invoices tables are mapped using ActiveRecord:

users

class User < ApplicationRecord
  has_many :invoices
end

invoices

class Invoice < ApplicationRecord
  belongs_to :user
end

Ecto: Data Mapping & Associations

On the other hand, Ecto requires the developer to be explicit about the data source and its fields. Although Ecto has similar has_many and belongs_to features, it also requires developers to be explicit about the associated table and the schema module that is used to handle that table schema.

This is how Ecto maps the users and invoices tables:

users

defmodule Financex.Accounts.User do
  use Ecto.Schema

  schema "users" do
    field :full_name, :string
    field :email, :string
    has_many :invoices, Financex.Accounts.Invoice
    timestamps()
  end
end

invoices

defmodule Financex.Accounts.Invoice do
  use Ecto.Schema

  schema "invoices" do
    field :payment_method, :string
    field :paid_at, :utc_datetime
    belongs_to :user, Financex.Accounts.User
    timestamps()
  end
end

Wrap Up

In this post, we compared apples and oranges without a blink. We compared how ActiveRecord and Ecto handle database migrations and mapping. A battle of the implicit slient original Batgirl versus the explicit 'I'm Batman' Batman.

Thanks to "convention over configuration", using ActiveRecord usually involves less writing. Ecto goes in the opposite direction, requiring developers to be more explicit about their intents. Other than "less code" being better in general, ActiveRecord has some optimal defaults in place that save the developer from having to make decisions on everything and also having to understand all the underlying configurations. For beginners, ActiveRecord is a more suitable solution, because it makes "good enough" decisions by default as long as you strictly follow its standard.

The explicit aspect of Ecto makes it easier to read and understand the behavior of a piece of code, but it also requires the developer to understand more about the database properties and the features available. What might make Ecto look cumbersome at first glance, is one of its virtues. Based on my personal experience in both ActiveRecord and Ecto world, Ecto's explicitness removes the "behind the scene" effects and uncertainty that is common in projects with ActiveRecord. What a developer reads in code, is what happens in the application and there is no implicit behavior.

In a second blog in a few weeks, in the two part "ActiveRecord vs Ecto" series, we'll cover how queries and validations work in both ActiveRecord and Ecto.

We'd love to know what you thought of this article. We're always on the lookout for new topics to cover, so if you have a subject you'd like to learn more about, please don't hesitate to leave a comment!

This post is written by guest author Elvio Vicosa. Elvio is the author of the book Phoenix for Rails Developers.

DEV Community: Elvio Vicosa

How my side projects failed

1. iLearn (iOS app)

2. Mamae (iOS app)

3. Phoenix for Rails Developers (e-book)

4. Statictastic (micro-SaaS)

Things I can learn from my past projects

Decide whether I want to build a business or learn new tools

Launching is hard, but it is the easiest part

Audience

There's no path to follow

Start small

Foot notes

Caching with Elixir and ETS

The Concept of Caching

How Popular Languages Deal with Caching

Cache in Elixir (and Erlang)

Feature Flags

Soft Real-Time Stats

Things to Keep in Mind When Using ETS as a Cache Solution

Integration of Unnecessary Dependencies

Optimized Lookup

No Serialization

Not Garbage Collected

Use of ETS in Distributed Systems

How to Create and Use ETS Within a GenServer

Summary

Financial Portfolio rebalancing with Elixir

What's a portfolio?

The “Permanent Portfolio”

Distribution

Rebalancing a portfolio

Using Elixir to rebalance a “Permanent Portfolio”

Demo

ActiveRecord vs. Ecto Part Two

Seed data

Find item using its primary key

ActiveRecord

Ecto

Comparison

Fetching all items

ActiveRecord

Ecto

Comparison

Querying with conditions

ActiveRecord

Ecto

Comparison

Query composition

ActiveRecord

Ecto

Comparison

Ordering

ActiveRecord

Ecto

Comparison

Limiting

ActiveRecord

Ecto

Comparison

Associations

ActiveRecord

Ecto

Comparison

Validations

Wrap up

ActiveRecord vs. Ecto - Part One

ActiveRecord

Ecto

Aren't you Comparing Apples and Oranges?

The Invoice System

Migrations

ActiveRecord: Creating the Users Table

Ecto: Creating the Users Table

ActiveRecord: Creating the invoices Table

Ecto: Creating the invoices Table

ActiveRecord: Data Mapping & Associations

Ecto: Data Mapping & Associations

Wrap Up

How to Create and Use ETS Within a `GenServer`

ActiveRecord: Creating the `invoices` Table

Ecto: Creating the `invoices` Table