DEV Community: Ana Nunes da Silva

Authentication with Bcrypt

Ana Nunes da Silva — Fri, 08 Jul 2022 09:10:36 +0000

Authentication

Authentication is a process that confirms a user's identity - making sure they are who they say they are. We usually do this before granting access or allowing privileged actions.

If we take this concept from the online to the real world, a key (🔑) is one of the simplest forms of authentication there is. Anyone with a key should be granted access to a building/house/room/cabinet. If you have the key, you are an authorized user.

Going back to the web world, a key usually takes the form of credentials. The most common credentials are a combination of a username and a password.

How can we make sure that we are storing these credentials safely?

Storing passwords

It is a very bad practice to store passwords in plaintext. If the data gets compromised, the intruder will have access to all the users' passwords. And since people tend to use the same password in different places, it will compromise the users not only on your application but also on other applications as well. This poses a great security threat.

Instead, you should store user passwords as secure undeciphered strings. This way, if a hacker manages to get access to your database, only hashed passwords will be leaked and, in theory, the hacker will not be able to decode them (or at least it will have a really hard time).

Ok, but how can we create and store these secured passwords?

Hashing algorithms

A hashing algorithm is a complex mathematical function that transforms a string of data into a seemingly random output string of fixed length.

Here's the trick of hashing algorithms:

Usually, encryption means transforming the string temporarily until a key is used to transform it back to the original string. But a hashing algorithm is a one-way encryption. One way means that it is non-reversible. Even if you have access to the database, you cannot get the original password back. But if we cannot decrypt the password, how can we match it?

A hashing algorithm is deterministic, meaning that the same input will always return the same output. When a user writes their credentials on the login form and submits it, the password is hashed and compared to the stored hashed password. If they are the same, then the login is successful.

All the above works great in theory, though in practice things are a bit more complex. With time, hackers have built tools to decode passwords. I will not go into much detail here but some of the common strategies are:

Rainbow tables: databases that have been precomputed with the most commonly used passwords and their hashed values (remember that a password will always have the same hash). Hackers can use a hashed password to get the original string. These databases have grown to store billions of records! Hackers
Dictionary attack: An attempt to guess passwords by using well-known words or phrases. These words/phrases will be hashed and compared with the hash they are trying to decode.
Brute force attack: a trial and error attack. It is the most expensive/time-consuming strategy. It implies trying out all the possible variations of characters up to a certain maximum length until you eventually get one right.

But as these hackings develop, so do the hashing algorithms. There are different hashing algorithms out there, some not recommended anymore for having become vulnerable like MD5, SHA-1, or SHA-256. Which one should we use then?

Bcrypt

Bcrypt is a cryptographic hashing algorithm created in 1999, and designed with passwords in mind. There are two main characteristics that make Bcrypt safer than other algorithms:

Bcrypt is a slow algorithm - this is good, as it reduces the number of passwords by second that can be hashed by a hacker. Prevents dictionary attacks.
Bcrypt uses salt - a string that is appended to the hash and stored together. So decoding the password requires knowing the salt string. Prevents rainbow attacks.

Rails comes with Bcrypt support. The ActiveModel has a has_secure_password method that we can use to set and authenticate against a Bcrypt password. It will provide a set of methods that will help set up authentication.

Without further ado, let's see how it works:

Install bcrypt

Look into your Gemfile and you will see a commented line with the bcrypt gem. It is included by default when a new rails app is created.

Uncomment the line and install it.

#Gemfile

# Use Active Model has_secure_password [https://guides.rubyonrails.org/active_model_basics.html#securepassword]
gem 'bcrypt', '~> 3.1.7'

Generate a user model and corresponding db table

The password_digest attribute is what will store the hashed password.

rails g model User username:string password_digest:string
rails db:migrate

This is what the schema should look like:

# db/schema.rb
create_table "users", force: :cascade do |t|
    t.string "username"
    t.string "password_digest"
    t.datetime "created_at", null: false
    t.datetime "updated_at", null: false
end

Add the `has_secure_password` method to the User model

# models/user.rb
class User < ApplicationRecord
  has_secure_password
end

The following validations are added automatically:

Password must be present on creation
Password length should be less than or equal to 72 bytes
Confirmation of password (using a XXX_confirmation attribute)

Create user/session routes

The following routes will allow us to sign up, log in, and log out:

# config/routest.rb
  resources :users, only: [:create]

  get '/signup', to: 'users#new'
  delete '/logout', to: 'sessions#destroy'
  get '/login', to: 'sessions#new'
  post '/sessions', to: 'sessions#create'

Build the signup feature

# controllers/users.rb
class UsersController < ApplicationController
  def new; end

  def create
    @user = User.new(user_params)

    if @user.valid?
      @user.save
      redirect_to login_path
    else
      redirect_to signup_path
    end
  end

  private

  def user_params
    params.require(:user).permit(:username, :password, :password_confirmation)
  end
end

<%# views/users/new.html.erb %>
<%= form_for @user do |f| %>
  <%= f.label :username %>
  <%= f.text_field :username, placeholder: "Username" %>
  <%= f.label :company %>
  <%= f.text_field :company, placeholder: "Company Name" %>
  <%= f.label :password %>
  <%= f.password_field :password, placeholder: "Password" %>
  <%= f.label :password_confirmation %>
  <%= f.password_field :password_confirmation, placeholder: "Confirm Password" %>
  <%= f.submit "Create Account" %>
<% end %>

Note that when we create a new user, we do not ask for a password_digest but rather a password (and password_confirmation) - two attributes that were made available by the has_secure_password included before. Bcrypt will then convert the password string into a hash and store it in the password_digest.

Build the login/logout features

This is where another new method will be handy - the authenticate method. It will take the password string provided by the user, hash it and compare it with the stored hash.

Let's say I have a user with username 'ana' and password '1234' (not a safe one, I know, just to simplify the example):

When providing the wrong password:

User.find_by(username: 'ana').authenticate('dasdas')
# => false

When providing the right password:

User.find_by(username: 'ana').authenticate('1234')
# => 
# #<User:0x00007fd0f635df8
#id: 3,
#username: "ana",
#password_digest: "[FILTERED]", 
#created_at: Thu, 07 Jul 2022 14:20:56.012150000 UTC +00:00,
#updated_at: Thu, 07 Jul 2022 14:20:56.012150000 UTC +00:00>

If both are the same, then a user instance is returned and we can proceed with setting the session's user with the authenticated user session[:user_id] = @user.id). When logging out, we will just need to clear this user from the session (session[:user_id] = nil)

# controllers/sessions.rb
class SessionsController < ApplicationController
  def new; end

  def create
    @user = User.find_by(username: params[:username])
    if @user && @user.authenticate(params[:password])
      session[:user_id] = @user.id
      redirect_to schedule_path(@user.company.name)
    else
      redirect_to login_path
    end
  end

  def destroy
    session[:user_id] = nil

    redirect_to login_path
  end
end

<%# views/sessions.new.html.erb %>
<%= form_tag sessions_path, remote: true do %>
  <%= label_tag "Username" %>
  <%= text_field_tag :username, nil, placeholder: "Username" %>
  <%= label_tag "Password" %>
  <%= password_field_tag :password, nil, placeholder: "Password" %>
  <%= submit_tag "Log In"%>
<% end %>

Require authentication

To be able to protect pages from unwanted visits, we can create an authenticate_user! method that will check if the user is logged in. If not, it will require the user to log in to be able to access that page.

For example:

class PostsController < ApplicationController
  before_action :authenticate_user!, only: [:show]

  def show
    @posts = Post.all
  end
end

# controllers/application_controller.rb
class ApplicationController < ActionController::Base
  helper_method :current_user

  def authenticate_user!
    redirect_to login_path unless current_user
  end

  def current_user
    @current_user ||=
      begin
        return nil unless session[:user_id]

        User.find(session[:user_id])
      end
  end
end

Add login/logout buttons

Finally, we can add the login/logout buttons to our app:

<% if current_user %>
  <%= button_to "Logout", logout_path, method: :delete %>
<% else %>
  <%= button_to "Login Page", login_path, method: :get %>
<% end %>

Happy authentication!

Exploring ViewComponent

Ana Nunes da Silva — Fri, 01 Jul 2022 09:02:42 +0000

What is ViewComponent

ViewComponent is an open-source framework built by Github to solve some of the Rails views pattern issues that emerged from building a large-scale application.

Views pattern issues?

Rails views are not encapsulated, they exist in a single execution context - meaning they can share state. This lack of isolation makes them hard to organize, reuse, scale, and also test.

We have all been there. Between views, partials, presenters, and helpers it might become hard to make sense of all the view logic, and duplication quickly starts to emerge. Maintaining and making changes in the frontend can be a cumbersome task.

Testing these views is also expensive. You will have to run slow integration or system tests that go through the router and controller layers, in addition to the view layer.

This is what ViewComponent decided to tackle. Github understood that there was a missing abstraction in the Rails view layer.

A good metaphor is written on the official documention:

"ViewComponent is to UI what ActiveRecord is to SQL"

Enter encapsulation

Inspired by React, the main idea is to make ruby objects render views.

In practice, a ViewComponent will be a combination of a ruby file and a template file.

This will force the single responsibility principle to be applied to the views. No more scattering view-related logic across models, controllers, and helpers. The view logic will now be the responsibility of a single class, encapsulating it in a proper object-oriented fashion. And that logic will be accessible only to a matching template.

At the end of the day, this encapsulation is what will make the view components reusable and testable.

Let's see it in practice by refactoring views built the rails way.

Refactoring rails views into components

Installation

In Gemfile, add:

gem "view_component"

Once that is installed you will have access to a new component generator.

This generator accepts a name for the ViewComponent and the arguments.

Unless you pass other options, the generator will create a test file (defaults to minitest), a ruby file, and a template file (defaults to html):

bin/rails generate component <name of the component> <params to be used by the component>
      invoke  test_unit
      create  test/components/example_component_test.rb
      create  app/components/example_component.rb
      create  app/components/example_component.html.erb

You can see the full generator documentation here.

Reusable components

As a rule of thumb, a template that has the potential to be reused is a good candidate for a ViewComponent.

Let's start with a simple example - page titles.

I have a heading that will be the same on all pages, only the content is dynamic, like this:

<h1 class="mb-4 text-2xl font-bold">Posts</h1>

Let's create a component:

rails g component Heading --test-framework rspec

Note that I am not passing any arguments, since only the content will be dynamic. I'm also using the --test-framework flag to create a rspec test file instead of minitest.

So my component class will look like this:

# components/heading_component.rb
class HeadingComponent < ViewComponent::Base
end

And here's the component's html:

<%#components/heading_component.html.erb%>
<h1 class="mb-4 text-2xl font-bold"><%= content %></h1>

Now I can call this component in all the application's pages. For example:

<%#views/posts.html.erb%>
<%= render(HeadingComponent.new.with_content("Posts")) %>

Passing arguments

Ok, so what if I also have buttons on different pages but they might be of different types (e.g. primary, danger,...). If I want to make a reusable component, I will have to pass the type as an argument:

rails g component Button type --test-framework rspec

Now the component class will be initialized with the type.

# components/button_component.rb
class ButtonComponent < ViewComponent::Base
  def initialize(type: type)
    @type = type
  end
end

And this instance variable can now be used in the component's view file.

<%#components/button_component.html.erb%>
<button type="button" data-view-component="true" class="btn-<%= @type %> btn">
  Click me
</button>

Now we just need to render the component in the rails views:

<%#views/books/show.html.erb%>
<%= render(ButtonComponent.new(type: "primary").with_content("Click me")) %>

Collections

When we are listing something, for example in an index page:

<%#views/books/index.html.erb%>
<% @books.each do |book| %>
  <h2>book.title</h2>
  <p>book.description</p>
<% end %>

We could start the refactoring by creating a component for a book:

rails g component Books book --test-framework rspec

#components/book_component.rb
class BookComponent < ViewComponent::Base
  def initialize(book:)
    @book = book
  end
end

<%#components/book_component_html.erb%>
<h2>@book.title</h2>
<p>@book.description</p>

<%#views/books/index.html.erb%>
<% @books.each do |book| %>
  <%= render(BookComponent.new(book: book)) %>
<% end %>

But what if we could also abstract the iteration by somehow telling the component to render itself for each element of the collection?

Gladly, ViewComponent has two handy methods for this: .with_collection and .with_collection_parameter.

So let's adjust the component class and reference the parameter that should be used to create a collection:

#components/book_component.erb
class BookComponent < ViewComponent::Base
  with_collection_parameter :book

  def initialize(book:)
    @book = book
  end
end

No need to adjust the component html. The only thing we need is to go to the rails view, remove the iteration and render theBookComponent with the .with_collection method, passing the collection that we want to render.

<%# views/books/index.html.erb%>
<%= render(BookComponent.with_collection(@books)) %>

The `content` block

In some of the previous examples, I have been passing text as content to the component. That's because, as mentioned in the documentation:

"Content passed to a ViewComponent as a block is captured and assigned to the content accessor."

So instead of passing text, we can actually pass a component to another.

Let's start from the end result now:

<%#views/products/index.html.erb%>
<%= render(GridListComponent.new) do %>
  <%= render(ProductComponent.with_collection(@products)) %>
<% end %>

Just by reading the code, we can already infer that we will have some sort of list that will render each item of the product collection.

Let's look at the individual components now, starting from the outside - the grid list component first:

<%#components/grid_list_component.html.erb%>
<div class="grid grid-cols-2 gap-4">
  <ul class="col-span-1 grid grid-cols-2 gap-x-4 gap-y-8 sm:grid-cols-3 sm:gap-x-6 lg:grid-cols-4 xl:gap-x-8">
    <li>
      <%= content %>
    </li>
  </ul>
</div>

Note the content accessor on the <li> element. This accessor will render whatever it is passed to the block. So if we pass the product component, which is:

<%#components/product_component.html.erb%>
<a href="<%= product_path(@product) %>">
  <img src="..." alt="..." class="...">
</a>

In the end, this is what the rendered html will look like:

<div class="grid grid-cols-2 gap-4">
  <ul class="col-span-1 grid grid-cols-2 gap-x-4 gap-y-8 sm:grid-cols-3 sm:gap-x-6 lg:grid-cols-4 xl:gap-x-8">
    <li>
      <a href="/products/1">
        <img src="..." alt="..." class="...">
      </a>
    </li>
  </ul>
</div>

Slots

Another way to compose components is through slots. As stated in the documentation,

"In addition to the content accessor, ViewComponents can accept content through slots. Think of slots as a way to render multiple blocks of content, including other components."

You will have to mention which slots the component accepts, using either renders_one or renders_many.

If you want to use an existing component to render the slot, you will have to pass it as a second argument.

It is also possible to define a new component as a subclass of the StoreComponent class. For that, you will need to pass the name of the component as a string on the second argument/

Let's look at an example with a Store component that will render a header and books:

# components/store_component.rb
class StoreComponent < ViewComponent::Base
  renders_one :header, "HeaderComponent"
  renders_many :books, BookComponent

  class HeaderComponent < ViewComponent::Base
    attr_reader :classes

    def initialize(classes:)
      @classes = classes
    end

    def call
      content_tag :h1, content, { class: classes }
    end
   end
end

<%# store_component.html.erb %>
<%= header %>

<% books.each do |book| %>
  <%= book %>
<% end %>

<%# index.html.erb %>
<%= render StoreComponent.new do |c| %>
  <% c.with_header(classes: "") do %>
    <%= link_to "My Store", root_path %>
  <% end %>

  <% c.with_book(title: "A book") do %>
     Nice Book
  <% end %>

  <% c.with_book(title: "Another book") do %>
    Read books
  <% end %>
<% end %>

Testing

Since components are isolated, we can skip the heavy system tests for now and just focus on simple and fast unit tests.

I'm going to use Rspec, but you can read the documentation for some minitest examples.

To use Rspec, you will need to add some ViewComponent's helpers and capybara matchers to spec/rails_helper.rb.

Make sure you have both rspec and capybara installed:

group :development, :test do
  gem 'capybara'
  gem "rspec-rails", "~> 5.0.0"
end

Add ViewComponent and Capybara configs:

# spec/rails_helper.rb
require "view_component/test_helpers"
require "capybara/rspec"

RSpec.configure do |config|
  config.include ViewComponent::TestHelpers, type: :component
  config.include Capybara::RSpecMatchers, type: :component
end

And using the BookComponent from earlier we can test that it prints the book titles, both individually and for a collection:

require "rails_helper"

RSpec.describe BookComponent, type: :component do
  book_1 = Book.create(title: 'Well Grounded Rubyist')
  book_2 = Book.create(title: '99 Bottles')

  it "renders a book" do
    render_inline(described_class.new(listing: book_1))

    expect(page).to have_text('Well Grounded Rubyist')
  end

  it "renders a listing collection" do
    render_inline(described_class.with_collection(Listing.all))

    expect(page).to have_text('Well Grounded Rubyist')
    expect(page).to have_text('99 Bottles')
  end
end

Going further

There is still a lot I want to explore using ViewComponent. For now, I feel that the beginning is fairly straightforward but I can see that it is not always easy to figure out what the right abstraction should be and how to best compose related components. This is not something inherent to ViewComponent per se though. It's the principles behind encapsulation and reusability that forces you to think a little beforehand so we can pick up the fruits later!

As always, the more the community explores and shares its findings, the more we can learn from each other, see the good/bad patterns and even contribute to the project.

I found the official documentation and videos really helpful and there are topics that I would like to cover in future explorations:

The content accessor Vs. slots
How to use ViewComponents with stimulus
Previewing components in isolation
Building a components library with ViewComponents and Storybook

There are also great ViewComponent based libraries that we can use or take ideas from:

References:

viewcomponent.org
ViewComponents in the Real World - Joel Hawksley
Encapsulating ruby on rails view

Finding all tables with a specific column name (postgres)

Ana Nunes da Silva — Mon, 14 Feb 2022 10:51:00 +0000

The information schema

The information schema consists of a set of views that contain information about the objects defined in the current database.

This is how Postgres defines the information schema. If you look at the official documentation, you will find two relevant views in this schema:

the tables view that contains all tables and views defined in the database
and the columns view that contains all columns (or view columns) defined in the database

These two views will give you access to information on the tables and the columns in our database, so let's see how we can use them:

The query

Let's say we want to get all tables with a student_id column.

If you are working with a single public schema (the default schema that Postgres will use to hold your tables), this query will do the trick:

SELECT 
  tables.tables_name
FROM information_schema.tables AS tables
JOIN information_schema.columns AS columns
  ON columns.table_name = tables.table_name
  AND columns.table_schema = tables.table_schema
WHERE columns.column_name = 'student_id'
  AND tables.table_schema = 'public'
  AND tables.table_type = 'BASE TABLE'
ORDER BY tables.table_name

A few notes on this query:

To access any of the views, you need to add information_schema as a prefix to the view name;
For the query to retrieve all tables with a student_id column it will have to join the two views that hold this information - information_schema.tables and information_schema.columns;
Postgres has its own internal schemas that you will want to exclude (e.g. the information_schema itself and the pg_catalog schema). In this case, we want the public schema only;
You can filter tables by table type: BASE TABLE for a persistent base table (the normal table type), VIEW for a view, FOREIGN TABLE for a foreign table, or LOCAL TEMPORARY for a temporary table. In this case, I only want the BASE TABLE.

Example of the query output:

	table_name
1	course_statistics
2	student_classes

Multiple schemas

If you are working with multiple schemas, the query needs some adjustments.

You can filter the results to a specific schema or it might be useful to select the table names together with the corresponding schema name. In the latter case, you will need to add tables.table_schema to the SELECT statement and, instead of choosing a specific schema, exclude Postgres internal schemas:

SELECT
   tables.table_schema,
   tables.table_name
FROM information_schema.tables as tables
JOIN information_schema.columns AS columns
  ON columns.table_name = tables.table_name
  AND columns.table_schema = tables.table_schema
WHERE columns.column_name = 'student_id'
  AND tables.table_schema not in ('information_schema', 'pg_catalog')
  AND tables.table_type = 'BASE TABLE'
ORDER by tables.table_schema

Example of the query output:

	table_schema	table_name
1	public	student_classes
2	public	course_statistics
3	tenant_1	student_classes
4	tenant_1	course_statistics

Conclusion

The information_schema contains metadata about the data objects stored in a database. It provides plenty of information not only on tables and columns but also permissions, triggers, constraints, and others.

The example I've used in this article is a real case example. I have recently had to run a migration where all data of a student had to be migrated to another student. So instead of manually checking and writing down all tables with a student reference, I queried information_schema and was then able to iterate each one to perform an update.

The information schema views do not, however, contain information about PostgreSQL-specific features; to inquire about those you need to query the system catalogs or other PostgreSQL-specific views.

Whether you need this kind of data for information purposes, migrations, or other actions, know that you will most probably find the information you need by querying Postgres internal schemas.

Polymorphic joins in Active Record

Ana Nunes da Silva — Fri, 11 Feb 2022 17:24:00 +0000

The polymorphic associations

In Active Record, you can use polymorphic associations to allow a model to belong to more than one other model, on a single association.

Here's an example, where a booking can belong to an accommodation or an office:

class Booking < ApplicationRecord
  belongs_to :bookable, polymorphic: true
end

class Accommodation < ApplicationRecord
  has_many :bookings, as: :bookable
end

class Office < ApplicationRecord
  has_many :bookings, as: :bookable
end

Rember that for this bookable association to work, the bookings table will have to hold the bookable_id and bookable_type columns. The Rails official documentation explains how to implement polymorphic associations, here.

The issue with joining polymorphic associations

In polymorphic associations if you try joining bookable directly, you will get an error:

Booking.joins(:bookable)

ActiveRecord::EagerLoadPolymorphicError (Cannot eagerly load the polymorphic association :bookable)

This is because Active Record does not know what table or tables to join, since there can be many tables under the generic bookable association.

One of the ways to solve this problem is to pass an SQL string, explicitly stating which bookable table we want to join, using the foreign key and type column:

Booking.joins("INNER JOIN accommodations ON accommodations.id = bookings.bookable_id AND bookings.bookable_type = 'Accommodation'")

But note that we are excluding the offices table from the query. If we would like to join it also, we would have to add a similar join statement but with a left join:

Booking
  .joins("LEFT OUTER JOIN accommodations ON accommodations.id = bookings.bookable_id AND bookings.bookable_type = 'Accommodation'")
  .joins("LEFT OUTER JOIN offices ON offices.id = offices.bookable_id AND bookings.bookable_type = 'Office'")

It is easy to imagine the mess it can potentially become if you need to add more bookable associations and perform additional queries on top of them.

It would help if instead of passing these verbose SQL strings, we could do something like:

Booking.left_joins(:accommodation, :office)

If you wanted to join all bookables, you would still have to pass all the bookable associations, though that could be done by passing the associations as a symbol instead.

If you try running that now, you will still get an error:

ActiveRecord::ConfigurationError: Can't join 'Booking' to association named 'accommodation'; perhaps you misspelled it?

The Booking model only knows about a bookable entity, so Active Record does not recognize Accommodation and Office individually as associations.

What if we could add these associations individually to the Booking model?

Using scoped associations

We can define specific associations by scoping them by bookable_type and foreign_key:

class Booking < ApplicationRecord
  belongs_to :bookable, polymorphic: true
  belongs_to :accommodation, -> { where(bookings: { bookable_type: 'Accommodation' }) }, foreign_key: 'bookable_id'
  belongs_to :office, -> { where(bookings: { bookable_type: 'Office' }) }, foreign_key: 'bookable_id'
end

Now, if you run the previous query again, you will get all bookings of the type 'Accommodation' and 'Office'. The SQL under the hood will be exactly the same as the one we have written before in the custom joins. You can confirm that by calling the .to_sql method on the query:

Booking.left_joins(:accommodation, :office).to_sql

=> "SELECT \"bookings\".* FROM \"bookings\" 
LEFT OUTER JOIN \"accommodations\" ON \"accomodations\".\"id\" = \"bookings\".\"bookable_id\" AND \"bookings\".\"bookable_type\" = 'Accommodation' 
LEFT OUTER JOIN \"offices\" ON \"offices\".\"id\" = \"bookings\".\"bookable_id\" AND \"bookings\".\"bookable_type\" = 'Office'

Happy querying!

Pagination in Rails from scratch (no gems)

Ana Nunes da Silva — Mon, 22 Nov 2021 13:11:38 +0000

Pagination usually requires having to answer two main questions:

What will the selected page contain?
- How many items?
- Which items?
How is the page related to the collection?
- What is the total number of pages for this collection?
- Is there a page after? And before?

Whilst the first will return the right subset of items back to the user, the latter will provide the frontend with the information needed to build the pagination controls. There are different types of controls you can build depending on your goals and UX preferences. In this example I will stick with a classic, something like this:

If you're having doubts about which controls to go with, I recommend this article.

So without further ado, let us write some code:

1. Creating a pagination mixin

Why a mixin?

When you're building an application you will probably need pagination in more than one endpoint. To avoid repeating the same implementation in different controllers we can make use of mixins. In practice, this means creating a Pagination module that can be included in different classes across your application.

The `Pagination` module

Following the pattern of most of the pagination gems, this Pagination module will consist of one public method: a paginate method that receives a collection and params. In our case, that collection will be an ActiveRecord collection. The params is a hash that will hold some of the current page attributes needed to paginate the collection. For instance, params will need to include page (the current page) and per_page (items that each page will retrieve from the collection).

module Pagination
  def paginate(collection:, params: {})
    pagination = Services::Pagination.new(collection, params)

    [
      pagination.metadata,
      pagination.results
    ]
  end
end

This module will be responsible for returning two sets of information: the page items (pagination.results) and the metadata that will support the pagination controls (pagination.metadata).

How do we get this data? We do not know yet, let's hold it for a while. What we know for now is that we will have to return this information to the controller. These are the answers to the questions we have raised before, in the intro of this article.

Now we can include this module in whatever class that needs pagination. In our case, that will be the controller:

class PostsController < ApplicationController
  include Pagination

  POSTS_PER_PAGE = 6

  def index
    @pagination, @posts = paginate(collection: Post.all, params: page_params)
  end

  private

  def page_params
    params.permit(:page).merge(per_page: POSTS_PER_PAGE)
  end
end

The collection will be all the posts, and the params will hold the page selected by the user and what we have defined to be the limit of each page (the POSTS_PER_PAGE constant).

Note that since the pagination module returns an array, you can 'unpack' that array and assign its values to multiple variables in one line. The first variable will hold the first value of the array and the second variable will hold the second value. You can learn more about this technique here.

2. Writing the pagination logic

Ok, so let us understand what is behind the magic of our module. The module calls a pagination service that will be responsible for getting the page items based on two values: items per page and the number that sets the start of each page. For an ActiveRecord collection that can be achieved with two methods only: limit and offset. As long as you know the values that these two methods receive - that is all you need to paginate.

module Services
  class Pagination
    attr_reader :collection, :params

    def initialize(collection, params = {})
      @collection = collection
      @params = params.merge(count: collection.size)
    end

    def metadata
      @metadata ||= ViewModel::Pagination.new(params)
    end

    def results
      collection
        .limit(metadata.per_page)
        .offset(metadata.offset)
    end
  end
end

But how can we calculate these values?

limit limits the returned results to a maximum number. In our case, all pages should have 6 items each (the last page can have less, depending on the total size of the collection). So 6 is our limit. Remember this was defined as constant in the controller - POSTS_PER_PAGE - and passed as a param to our pagination service.

But what about offset? The offset will help you define the boundaries of your page. It skips over a number of posts when returning results. If your offset is 12 it will skip the first 12 posts and return all resulting records after that. When combined with limit it allows us to see pages beyond the first page. So if the offset is 12 and limit is 6, we will return 6 results after skipping the first 12. That will be page 3.

But these values are calculated in a separate class that will be solely responsible for building all the metadata needed for the pagination and for the frontend. This leads us to the final part of our backend:

Preparing the pagination metadata

module ViewModel
  class Pagination
    DEFAULT = { page: 1, per_page: 6 }.freeze

    attr_reader :page, :count, :per_page

    def initialize(params = {})
      @page     = (params[:page] || DEFAULT[:page]).to_i
      @count    = params[:count]
      @per_page = params[:per_page] || DEFAULT[:per_page]
    end

    def offset
      return 0 if page == 1

      per_page * (page.to_i - 1)
    end

    def next_page
      page + 1 unless last_page?
    end

    def next_page?
      page < total_pages
    end

    def previous_page
      page - 1 unless first_page?
    end

    def previous_page?
      page > 1
    end

    def last_page?
      page == total_pages
    end

    def first_page?
      page == 1
    end

    def total_pages
      (count / per_page.to_f).ceil
    end
  end
end

Besides providing our pagination service with the limit and offset values that we saw before, this class will work as a frontend helper as most of the methods in this class will hold the information needed to build the pagination controls.

Going back to our module, remember that the first element of the returned array will be an instance of this class. It will later be assigned to an instance variable @pagination in our PostsController and made available for the frontend views to consult.

3. Building the frontend component

At the bottom of your collection view, in this case posts/index.html.erb you can now add the pagination controls. It might be a good idea to isolate them in a component:

<%# posts/index.html.erb %>

<%= render partial: 'shared/pagination', locals: { pagination: @pagination } %>

<%# _pagination.html.erb %>

<div>
  <% if  pagination.previous_page? %>
    <%= link_to 'First', posts_path(page: 1) %>
    <%= link_to '< Previous', posts_path(page: pagination.previous_page) %>
  <% else %>
    <p>First</p>
    <p>&#60; Previous</p>
  <% end %>
  <p><%= "Page #{pagination.page} of #{pagination.total_pages}" %></p>
  <% if pagination.next_page? %>
    <%= link_to 'Next >', posts_path(page: pagination.next_page) %>
    <%= link_to 'Last', posts_path(page: pagination.total_pages) %>
  <% else %>
    <p>Next &#62;</p>
    <p>Last </p>
  <% end %>
</div>

So there you go, you can now access the ViewModel::Pagination methods like previous_page, next_page, or total_pages and direct the links to the right pages. Now go ahead and add the CSS styling as you please.

Have fun paginating! 📖

ruby methods - #tap and #then

Ana Nunes da Silva — Mon, 01 Nov 2021 18:44:16 +0000

In this post, I'll share the main difference between #tap and #then with specific use cases.

The tap and then methods look very similar, at first. But If you look at their source code, there's a slight nuance that makes their purpose quite different.

Looking in

If we'd implement these methods ourselves in ruby, we could write:

def tap_into
  yield(self)
  self
end

def and_then
  yield(self)
end

I've purposefully named these methods differently, so we don't override ruby's methods, in case we'd like to play with them in irb.

Looking at this implementation in ruby, the difference is in one line only. While tap yields self to the block and returns self, then only yields self and will return whatever the block returns.

The difference is in the return

Let's call these two methods, using the same code sample, so we can grasp the difference:

'Ana'.tap { |name| "Hi, #{name.upcase}" }
# => "Ana"

'Ana'.then { |name| "Hi, #{name.upcase}" }
# => "Hi, ANA"

As you can see, tap passed 'Ana' to the block and returned 'Ana'.

It did not transform the string name that was passed to it. But though it might look like nothing happened inside that block, it did. We can confirm that by adding a puts before the string:

'Ana'.tap { |name| puts "Hi, #{name.upcase}" }
# Hi, ANA
# => "Ana"

On the other hand, then is more straightforward: it took 'Ana', passed it to the block, and returned the result of that block. It transformed the self.

Actions vs Transformations

The previous example provides a hint of the different use cases for each method:

With tap you can take an object and run sequential actions with it, whilst then lets you take an object and apply sequential transformations on it.

Use cases

Using `tap` to call dependent actions

This one I don't use as often as the others, but sometimes it's useful to run related actions while still returning the original object.

def confirm_booking(booking)
  booking
    .update!(status: confirmed)
    .tap(&method(:notify_hosts))
    .tap(&method(:notify_guests))
end

This would be the same as:

def confirm_booking(booking)
  booking.update!(status: confirmed)
  notify_hosts(booking)
  notify_guests(booking)
  booking
end

Using `tap` to create data with associations

I use this one frequently when creating data for my tests with factory bot:

let(:author_with_book) do 
  create(:author, name: 'Ana').tap do |author|
    create(:book, author: author, title: 'taptap')
  end
end

Using `tap` to add multiple test expectations for the same object:

Avoids Booking.last repetitions and adds the idea of a booking scope that you'd otherwise miss if you used Booking.last in a variable.

Booking.last.tap |booking|
  expect(booking.check_in).to be_present
  expect(booking.check_out).to be_present
  expect(booking.guest_id).to eq(guest.id)
end

Using `then` to transform objects like strings or hashes:

def build_sql_string
  Array(select_clause)
    .then { |sql_string| sql_string << where_clause }
    .then { |sql_string| sql_string << order_by_command }
    .then { |sql_string| sql_string << limit_clause }
    .then { |sql_string| sql_string.join(' ') }
end

private

def select_clause
  "SELECT * FROM students"
end

def where_clause
  "WHERE school_id = #{@school_id}"
end

def order_by_command
  "ORDER BY name ASC"
end

def limit_clause
  "LIMIT 20"
end

Using `then` to chain queries

This is probably the most common use case for me. Using then is really useful to build complex queries and/or conditional queries (e.g. apply a filter if the filter is sent).

def result
  Post
    .then(&method(:filter_by_tag))
    .then(&method(:filter_by_status))
    .then(&method(:order))
end

private

def filter_by_tag(posts)
  return posts unless @tag

  posts
    .joins('left outer join tag_posts on tag_posts.post_id = posts.id')
    .joins('left outer join tags on tags.id = tag_posts.tag_id')
    .where('tags.name' => @tag)
end

def filter_by_status(posts)
  return posts unless @status

  posts.where(status: Post.statuses[@status])
end

def order(posts)
  posts.order('published_at DESC')
end

Final notes

I've come across really good content out there on how these two methods work, but not that much on real examples on how people use them, hence the motivation to write this post.

The use cases exposed here, are a collection of examples that I've seen to have benefited from the use of tap and then. I find then especially useful in refactoring complex code to simple, explicit, chainable methods. There are surely many other use cases, and it would be great to hear about them - if you have other ideas please share!

Rails templates with docker

Ana Nunes da Silva — Sat, 16 Oct 2021 17:58:28 +0000

Starting new rails apps has never been easier for me. Here's why and how.

Before docker

You know when you want to start a new app, but you end up losing too much time setting it up, and solving issues with the dependencies on your machine? And after a while, you want to start another new project, only to find yourself going through the same steps and pains, to a point where you almost lose interest?

I've always wanted to find a way to start a new project on my machine, seamlessly. No hassle, just plug and play. But I never found a solution for that until I started using docker. Being able to start a rails project in under 5 minutes has given me back the joy of working on side projects and exploring new things.

After docker

So, recently I've decided to put together simple docker based templates with the tools I generally work with on a rails project:

Using github templates I can easily create new repositories based on a chosen template, clone it to my machine, run some docker-compose commands documented in the project's README, and voilà!

These templates are mostly based on the official docker quickstart tutorial for rails. They are currently set for ruby 3 and rails 6 but the beauty of docker is that you can go ahead and easily change them to whatever version you like, without the need to have anything installed on your computer. The same goes for postgres, mysql or whatever other tools you might add to these templates.

I'll be adding more as I start new rails apps with different toolsets that might be helpful in the future. For instance, I've currently started a new rails app with sidekiq and since I use sidekiq often, I know it's a good candidate for a template. I'll not create templates for all possible combinations of tools but mostly for those that involve adding containers, networking, or installing other dependencies. The idea is to save time with cumbersome configurations.

I'm happy to get feedback on these, feel free to use them, adapt them to your needs or just take some ideas to start your own. If you're fairly new to docker like me, I recommend you build your own from scratch. You'll learn a lot about docker and docker tools.

Understanding docker - playing with ruby containers

Ana Nunes da Silva — Thu, 16 Sep 2021 19:49:35 +0000

Understanding the basics of docker and docker commands is the first step into app containerization.

Docker, Dockerfile, docker-compose... When you first get into the docker world it can be a bit daunting to understand how all of these tools work and relate to each other.

In this post, I'll be playing around with a simple ruby container just using docker commands. No Dockerfile nor docker-compose for now. Just the basic docker commands.

It sure helped me really grasp the basic concepts behind images and containers in isolation before moving into more complex concepts.

Images vs Containers

In docker, it all begins with an image. The image sets the minimum base for a container to run. An image is an immutable file that will serve as a snapshot for the container, and the container will be the actual process on which you can work.

It might help to look at it with an OOP mindset - think of an image as a class and the container as an instance of that class. A container is an instance of an image.

For the purpose of this exercise, I want to run a simple ruby sandbox with irb. This sandbox will be available in a container but for this container to run I'll need its snapshot - the ruby image.

Luckily, we don't have to build the ruby image ourselves. There are plenty of ready-made images available on Docker hub that you can use. If you search for ruby you'll find the official ruby image in different versions.

The `docker run` command

The command that starts a container process from an image is docker run.

Let's understand how the command works by starting a ruby container.

If you run docker run ruby for the first time, it will first fetch the ruby image in its latest version from the docker hub and download it into your local docker.

You'll get an output similar to this one:

Unable to find image 'ruby:latest' locally
latest: Pulling from library/ruby
4c25b3090c26: Pull complete 
1acf565088aa: Pull complete 
b95c0dd0dc0d: Pull complete 
5cf06daf6561: Pull complete 
942374d5c114: Pull complete 
420c5b579440: Pull complete 
cdfe7730cd5c: Pull complete 
bfeceb400e58: Pull complete 
Digest: sha256:8849efdb1f006c5d7b26980f3aeb15f00fa2b5428cfbef8761aef5fc87491b89
Status: Downloaded newer image for ruby:3.0.2
Switch to inspect mode.

Ok, it looks like nothing happened yet but it does say that it downloaded the ruby image successfully. How can I confirm that?

The `docker images` command

If you run docker images you will see that you'll have the ruby with the latest tag available for you. You should see something like this:

REPOSITORY     TAG       IMAGE ID       CREATED        SIZE
ruby           latest    b28f54b6ce55   3 days ago    881MB

Note that the created timestamp you see in the docker images list is the date that the image was added to the docker library and not the date of the download to your local docker.

Going back to our command, it should have also started a container based on this image but where is it?

The `docker ps` command

Run docker ps to list all the running containers.

CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

The list is empty. Why? The run command should have started a new container process.

If you run docker ps -a where -a is the flag to show all containers (running and exited), your container will be there.

CONTAINER ID   IMAGE                    COMMAND                  CREATED          STATUS                        PORTS     NAMES                
948b3179783a   ruby:latest              "irb"                    10 seconds ago   Exited (0) 5 seconds ago                eager_snyder

Look at the Status column. It reads Exited (0) 5 seconds ago. But why was this container exited?

You may not have noticed, amongst the printed download messages, but there was an error message at the end. Let's see that message again by re-running the docker run command:

-> docker run ruby
Switch to inspect mode.

This is because the ruby image by default sends irb as a command to the container. Once the container process is opened the irb command should run. But the container cannot run irb unless we tell it to run in the interactive terminal mode.

In summary: the process was opened, the container ran irb, got the error message, the process was closed right after it, exiting the container with it.

We can run any command with the interactive terminal mode using the -it flag.

-> docker run -it ruby
irb(main):001:0>

Yeah, our sandbox is running! 🎉

Go ahead and play with ruby a bit.

irb(main):001:0> ['hello', 'world'].join(', ').capitalize
=> "Hello, world"

Before you exit irb, open another terminal window and run docker ps

CONTAINER ID   IMAGE         COMMAND   CREATED         STATUS         PORTS     NAMES
2ef882ad2b5b   ruby:latest   "irb"     4 minutes ago   Up 4 minutes             elated_satoshi

There it is! We have a running container!

Now exit irb and run docker ps again. It's empty. If you run docker ps -a you should see your container there. It has an exited status again.

CONTAINER ID   IMAGE                    COMMAND                  CREATED          STATUS                      PORTS     NAMES
2ef882ad2b5b   ruby:latest              "irb"                    6 minutes ago    Exited (0) 47 seconds ago             elated_satoshi
948b3179783a   ruby:latest              "irb"                    15 minutes ago   Exited (0) 15 minutes ago             eager_snyder

We've just learned another essential aspect of containers. The container has one main process. If that process stops, the container stops. When we left irb, the process stopped and the container was exited.

Note also that we have two stopped containers. The first container with the random name elated_satoshi is the one we've just opened and closed after playing with irb, and the last one eager_snyder is the container that was created when we ran the command without the interactive terminal flag.

So, every time you run the ruby image, you're creating a different container.

Passing other commands to the container

We've seen that running docker run -it ruby is the same as running docker -it ruby irb. We omit it because that's the default for ruby, but we can pass other commands for the container to run once it's opened.

For instance, if we run docker run -it ruby bash we will create an interactive bash shell in the container.

docker run -it ruby bash
root@b1c73a63eef9:/# ls
bin  boot  dev  etc  home  lib  lib64  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var
root@b1c73a63eef9:/# ruby -v
ruby 3.0.2p107 (2021-07-07 revision 0db68f0233) [x86_64-linux]
root@b1c73a63eef9:/#

Stopping and removing containers

As we saw earlier, every time you run the docker run command a new container will be created. This means that with time you'll get a big list of stopped containers:

If you'd like to remove stopped containers, you can just run docker rm <container id>.

If the container is still running you might have to use the stop or kill command before removing it.

docker stop <container id>
docker rm <container id>

If you'd like to avoid exited containers to be kept after the process is closed there's a useful flag you can use with docker run, the -rm.

docker run -rm -ti ruby

Keeping container processes open

Remember that every time we left irb or bash in the previous exercises, we closed the process and the container.

But there's a way you can exit yourself from the process without closing it. If you use ctrl+p + ctrl+q, you will no longer see the process but it will still be running in the background. If you run docker ps -l, where the -l flag will get the latest container, you'll see your container there.

If you want to start a new container in the detached mode you can pass the detach flag, -d:

-> docker run -d -ti ruby
e68f13762ee83ea315b3d0a679432dd1963f276deb387aa3bdce9b4801f21979

It prints out the container id for you, but again, you can also confirm that there's a new process running with docker ps.

CONTAINER ID   IMAGE        COMMAND   CREATED         STATUS         PORTS     NAMES
e68f13762ee8   ruby:latest  "irb"     4 seconds ago   Up 3 seconds             crazy_varahamihira

If you want to jump in the process again, you can run the docker attach <container id> command.

-> docker attach e68f13762ee8
irb(main):001:0>

You can also add a new process to the same container, with the exec command. Imagine you have irb running in the background but you want to jump in the bash:

docker exec -it e68f13762ee8 bash
root@e68f13762ee8:/#

Naming containers

As we noticed earlier, a container has a name but if you don't give it one, a random one will be assigned to it.

If you want to give it a name, you can pass it in a --name flag.

docker run --name ruby-sandbox -ti ruby`

And now run docker ps to make sure there's a container with that name:

CONTAINER ID   IMAGE                    COMMAND                  CREATED          STATUS                   PORTS     NAMES                
aee3cd98a190   ruby:latest              "irb"                    10 seconds ago   Up 9 seconds                       ruby-sandbox

First master the fundamentals!

A lot of commands were covered in this article but I believe that there is only a handful of fundamental concepts in docker. The goal of this article was to share a bit of the process and exercises I've gone through to understand what docker really is. It's a compilation of stuff I was first taught by my colleague Leandro (he has great content on this topic, go check him out) and that I also got from basic docker courses. Have fun! 🐳

Testing CSV and Excel uploads with tempfiles

Ana Nunes da Silva — Mon, 05 Jul 2021 20:05:04 +0000

Lately, I've been working on a couple of features that allow bulk data to be uploaded through .csv or .xlsx files.

When working with files, passing file fixtures to tests is a common practice. File fixtures are sample files that you can store under spec/fixtures/files (or in your own customized path). This way, you can read an actual file in your tests or just pass the file path to other classes that might need to access the file data.

File fixtures are simple to use but I've found them useful only on cases where either the file data is not that relevant or it is not going to vary.

The problem is that in most cases the user can upload files with different structures or data, depending on the use case. This means that, for test purposes, if we want to cover different scenarios, we will eventually end up stuffing our fixtures folder with enough file variations that would cover all the scenarios. It's just not flexible enough and it looks hard to maintain. Plus, it has the downside of hiding data from your tests since you have to open the files to know what's inside.

Ideally, we should be able to create the sample data explicitly in each test scenario, run our expectations, and clean up the data afterward. Sounds like a good case for Tempfiles - a file that will be available to test but that will be deleted after the test runs.

# students_uploader_spec.rb
describe 'Services::StudentsUploader' do 
  it 'creates students' do 
    Tempfile.open(['students.csv', '.csv']) do |temp|
       CSV.open(temp, 'wb', col_sep: ';', headers: true) do |csv|
         csv << ['email', 'name']
         csv << ['student@example.com', 'Student Name']
       end

       result = described_class.call(temp)

       expect(result[:count]).to eq(1)
       expect(Student.count).to eq(1)
     end
  end
end

Looking at the example above I can cleary define three necessary steps:

Open a Tempfile and pass it a filename (extension is optional).
Inside the Tempfile block, open a CSV, pass it the tempfile instance and build the rows with the data you need
Run the expectations inside the tempfile block

A fourth automatic step will delete the tempfile so you don't end up with a bunch of tempfiles lying around in your project.

It's important to reinforce that everything that depends on the tempfile will have to be run inside the tempfile block. That's because the tempfile is deleted immediately after its block is closed. If you try to run your expectation after the tempfile block is closed, you'll get an error similar to this one:

Failures:
  Failure/Error: result = described_class.call(temp)

  NameError:
    undefined local variable or method `temp' for #<RSpec::ExampleGroups::Services::SudentsUploader:0x00007fb74000c0b8>
  # ./spec/services/students_uploader_spec.rb:1:in `block (3 levels) in <top (required)>'

But going back to our happy path, I can also write a similar context for xlsx files (I'm using the axlsx gem to write excel files):

describe 'Services::StudentsUploader' do
  context 'when uploading a xlsx' do  
    it 'creates students' do 
      Tempfile.open(['students.xlsx', '.xlsx']) do |temp|
        Axlsx::Package.new do |p|
          p.workbook.add_worksheet(name: "test") do |sheet|
            sheet.add_row ['name', 'email', 'code']
            sheet.add_row ['Jane Doe', nil, 123]
            sheet.add_row ['John Doe', 'john@test.com', nil]
          end
          p.serialize(temp.path)
        end

        result = described_class.call(temp.path)

        expect(result[:count]).to eq(1)
        expect(Student.count).to eq(1)
      end
    end
  end
end

Now I can play around with different file structures, knowing that I will have my data explicitly available in each scenario and that I will be able to tweak it easily without the hassle of having to create and maintain external files.

Happy testing! 🧪

Ordering fuzzy search results by relevancy using trigrams

Ana Nunes da Silva — Sat, 08 May 2021 10:07:41 +0000

Say that I want to search users by name or email. If I type 'ana' I want to get all the users whose name or email contains the character combination 'ana'. I'm saying contains since there might be emails or names that are not an exact match to 'ana ' but can contain it - e.g. 'joana', 'diana', 'user@banana.com' - I don't want to exclude those from my results. Other names or emails similar to 'ana' can be valuable to whoever is searching.

This type of search is called fuzzy string matching and can be implemented with SQL using wildcards. I'll be using PostgreSQL but similar SQLs can be written in other databases:

SELECT id, name, email
FROM users
AND users.name ILIKE '%ana%'
OR users.email ILIKE '%ana%'
LIMIT 20

This query will return all the users whose name or email contains 'ana' but I won't get them in any specific order. I could order them alphabetically by 'name' and 'email' but that does not guarantee relevance. Since I'm limiting results to the first 20 I could even be excluding relevant results by ordering alphabetically.

Luckily I've learned that Postgres has this very interesting extension called pg_trgm. Quoting from the postgresql documentation:

[pg_trgm] provides functions and operators for determining the similarity of alphanumeric text based on trigram matching, as well as index operator classes that support fast searching for similar strings.

In linguistics, a trigram is a group of three consecutive written units such as letters, syllables, or words. What Postgres will do is to split words into trigrams. For instance, the 'ruby' trigram would be {" r"," ru","by ",rub,uby}. Note that some trigrams have empty spaces. That's because each word is considered to have two spaces prefixed and one space suffixed when determining the set of trigrams contained in the string. What we really need to take away from this is that postgres will split words into smaller chunks of 3 alphanumeric characters and then check the similarities with the same size chunks of other words.

So when comparing two strings using trigrams, Postgres can score the similarity between them from 0 to 1 (where 1 is a perfect match). Let's play with the pg_trgm extension to understand how it works.

First, I'll add the extension:

CREATE EXTENSION pg_trgm;

Now, I can compare two words:

SELECT SIMILARITY('Ana', 'Addriana'); 
-- returns 0.3

SELECT SIMILARITY('Ana', 'Alana'); 
-- returns 0.42857143

This illustrates the consequences of ordering alphabetically. If I search 'Ana', 'Addriana' will show up in the results before 'Alana' despite having a lower similarity score. It might be ok for you, depending on the requirements of your product, to order alphabetically. Using similarity adds a bit of complexity to your code but in this case, it does add value to the search feature I'm working on.

So now that we know how trigrams and the similarity function work, let's add that to our initial query. To order by similarity I'll first have to SELECT that score, give it an alias, and pass that variable to the ORDER BY clause:

SELECT
id,
name,              
email,
SIMILARITY(name, 'ana') AS name_score,
SIMILARITY(email, 'ana') AS email_score
FROM users
WHERE users.name ILIKE '%ana%'
OR users.email ILIKE '%ana%'
ORDER BY name_score DESC NULLS LAST, email_score DESC NULLS LAST, name
LIMIT 20;

Looking closely at the ORDER BY clause, note that since I'm searching on two columns, I've considered that the most relevant is name_score. Then I'll look at the email_score and I'll finally order alphabetically if necessary.

Fast searches with trigram indexes

Now that we've got our query, I've analyzed its performance and it seems fairly fast (~100-130ms). Still, since I want to return search results as my users are typing, I don't want this search to be just fast - I want it blazing-fast. I need immediate feedback. I'm going to test adding a couple of indexes to see if they can improve the response time.

GIN (Generalized Inverted Index) and GiST (Generalized Search Tree) indexes are very useful in text search features since they can map multiple values (e.g.: list of trigrams) to a single row. On the other hand, Postgres default B-Tree indexes are optimized for when a row has a single key-value.

When choosing between GIN vs GiST, I've decided to go with the GIN index. Although it might be a bit slower to write it is faster to read than the GiST index. To make sure that it will not raise any performance issues, I've tested creating my indexes locally on data amounts similar to the production data. Creating an index locks out the table for writes and in problematic cases, it can cause downtime. To avoid this, I'll tell Postgres to create these indexes concurrently. These can take longer to build but they will not require a lock that blocks writes.

So with the help of pg_trgm, which supports indexing based on trigrams, I'll add a GIN index to each of the columns that I'm searching - name and email - and pass the operator class gin_trgm_ops to instruct Postgres to build these indexes with trigrams.

CREATE INDEX CONCURRENTLY index_users_on_name_trigram
ON users 
USING gin (name gin_trgm_ops);

CREATE INDEX CONCURRENTLY index_users_on_email_trigram
ON users 
USING gin (email gin_trgm_ops);

With these two indexes, I've managed to reduce the query runtime to ~60-80ms 🚀.

Other performance remarks: `ILIKE` vs `LIKE`

There are two reasons why I've decided to use the Postgresql ILIKE function instead of the LIKE. One has to do with the functionality per se and the other with performance.

In terms of functionality, the main difference between the two is that ILIKE is case-insensitive. So if I want to know if 'Ana' and 'ana' are the same, ILIKE will tell you that statement is true, and LIKE will tell you otherwise:

SELECT 'Ana' ILIKE 'ana'; 
-- returns true

SELECT 'Ana' LIKE 'ana'; 
-- returns false

You can make LIKE case-insensitive as well with the help of the LOWER function:

SELECT 'Ana' ILIKE 'ana'; 
--returns true

SELECT LOWER('Ana') LIKE LOWER('ana'); 
-- returns true

But this is a more verbose and prone to errors version that does not add any value to this feature. Besides, I've noted a slight improvement in performance using ILIKE.

If you're interested in understanding the performance differences between these two approaches, I recommend this stackoverflow thread. Here are my 2 cents after reading about it and doing some benchmarks: The latter option (LIKE with LOWER) can be significantly faster than the first (ILIKE) if you're not using indexes. When you use good indexes, those differences are reduced or even inverted. In my case, creating different GIN indexes for name, email ended up showing that the ILIKE option is the most performant (though with indexes this difference is not that relevant).

Conclusion

Using the PostgresSQL pg_trgm extension not only improved the relevancy of the search results but also brought this feature with great performance improvements (vs. a previously sluggish user experience). To come up with the final query and index solutions, I've used pgAdmin to test different hypotheses. Postgres EXPLAIN ANALYZE is an awesome command that I've also used heavily throughout this development. Remember that each problem is different so I'd advise anyone to use the tools that Postgres provides to analyze and measure different potential solutions.

Testing external APIs with Rspec and WebMock

Ana Nunes da Silva — Fri, 16 Apr 2021 16:05:40 +0000

Testing code you don't own can be a bit trickier than writing tests for code you own. If you're interacting with 3rd party APIs, you don't actually want to make real calls every time you run a test.

Reasons why you should not make real requests to external APIs from your tests:

it can make your tests slow
you're potentially creating, editing, or deleting real data with your tests
you might exhaust the API limit
if you're paying for requests, you're just wasting money
...

Luckily we can use mocks to simulate the HTTP requests and their responses, instead of using real data.

I've recently written a simple script that makes a POST request to my external newsletter service provider every time a visitor subscribes to my newsletter. Here's the code:

module Services
  class NewSubscriber
    BASE_URI = "https://api.buttondown.email/v1/subscribers".freeze

    def initialize(email:, referrer_url:, notes: '', tags: [], metadata: {})
      @email = email
      @referrer_url = referrer_url
      @notes = notes
      @tags = tags
      @metadata = metadata
    end

    def register!
      uri = URI.parse(BASE_URI)
      request = Net::HTTP::Post.new(uri.request_uri, headers)

      http = Net::HTTP.new(uri.host, uri.port)
      http.use_ssl = true
      http.request(request, payload.to_json)
    end

    private

    def payload
      {
        "email" => @email,
        "metadata" => @metadata,
        "notes" => @notes,
        "referrer_url" => @referrer_url,
        "tags" => @tags
      }
    end

    def headers
      {
        'Content-Type' => 'application/json',
        'Authorization' => "Token #{Rails.application.credentials.dig(:buttondown, :api_key)}"
      }
    end
  end
end

The first approach could be to mock the ::Net::HTTP ruby library that I'm using to make the request:

describe Services::NewSubscriber do
  let(:email) { 'user@example.com' }
  let(:referrer_url) { 'www.blog.com' }
  let(:tags) { ['blog'] }
  let(:options) { { tags: tags, notes: '', metadata: {} } }

  let(:payload) do
    {
      email: email,
      metadata: {},
      notes: "",
      referrer_url: referrer_url,
      tags: tags
    }
  end

  describe '#register' do
    it 'sends a post request to the buttondown API' do
      response = Net::HTTPSuccess.new(1.0, '201', 'OK')

      expect_any_instance_of(Net::HTTP)
        .to receive(:request)
        .with(an_instance_of(Net::HTTP::Post), payload.to_json)
        .and_return(response)

      described_class.new(email: email, referrer_url: referrer_url, **options).register!

      expect(response.code).to eq("201")
    end
  end

This test passes but there are some caveats to this approach:

I'm too tied to the implementation. If one day I decide to use Faraday or HTTParty as my HTTP clients instead of Net::HTTP, this test will fail.
It's easy to break the code without making this test fail. For instance, this test is indifferent to the arguments that I'm sending to the Net::HTTP::Post and Net::HTTP instances.

Testing behavior with `WebMock`

WebMock is a library that helps you stub and set expectations on HTTP requests.

You can find the setup instructions and examples on how to use WebMock, in their github documentation. WebMock will prevent any external HTTP requests from your application so make sure you add this gem under the test group of your Gemfile.

With Webmock I can stub requests based on method, uri, body, and headers. I can also customize the returned response to help me set some expectations base on it.

require 'webmock/rspec'

describe Services::NewSubscriber do
  let(:email) { 'user@example.com' }
  let(:referrer_url) { 'www.blog.com' }
  let(:tags) { ['blog'] }
  let(:options) { { tags: tags, notes: '', metadata: {} } }

  let(:payload) do
    {
      email: email,
      metadata: {},
      notes: '',
      referrer_url: referrer_url,
      tags: tags
    }
  end

  let(:base_uri) { "https://api.buttondown.email/v1/subscribers" }

  let(:headers) do
    {
      'Content-Type' => 'application/json',
      'Accept'=>'*/*',
      'Accept-Encoding'=>'gzip;q=1.0,deflate;q=0.6,identity;q=0.3',
      'User-Agent'=>'Ruby'
    }
  end

  let(:response_body) { File.open('./spec/fixtures/buttondown_response_body.json') }

  describe '#register!' do
     it 'sends a post request to the buttondown API' do
        stub_request(:post, base_uri)
          .with(body: payload.to_json, headers: headers)
          .to_return( body: response_body, status: 201, headers: headers)

       response = described_class.new(email: email, referrer_url: referrer_url, **options).register!

       expect(response.code).to eq('201')
    end
  end
end

Now I can pick another library to implement this script and this test should still pass. But if the script makes a different call from the one registered by the stub, it will fail. This is what I want to test - behavior, not implementation. So if, for instance, I run the script passing a different subscriber email from the one passed to the stub I'll get this failure message:

1) Services::NewSubscriber#register! sends a post request to the buttondown API
     Failure/Error: http.request(request, payload.to_json)                                                                                                                        

     WebMock::NetConnectNotAllowedError:
       Real HTTP connections are disabled. Unregistered request: POST https://api.buttondown.email/v1/subscribers with body '{"email":"ana@test.com","metadata":{},"notes":"","ref
errer_url":"www.blog.com","tags":["blog"]}' with headers {'Accept'=>'*/*', 'Accept-Encoding'=>'gzip;q=1.0,deflate;q=0.6,identity;q=0.3', 'Authorization'=>'Token 26ac0f19-24f3-4ac
c-b993-8b3d0286e6a0', 'Content-Type'=>'application/json', 'User-Agent'=>'Ruby'}

       You can stub this request with the following snippet:

       stub_request(:post, "https://api.buttondown.email/v1/subscribers").
         with(                              
           body: "{\"email\":\"ana@test.com\",\"metadata\":{},\"notes\":\"\",\"referrer_url\":\"www.blog.com\",\"tags\":[\"blog\"]}",
           headers: {                                                                    
          'Accept'=>'*/*',                                                                                                                                                        
          'Accept-Encoding'=>'gzip;q=1.0,deflate;q=0.6,identity;q=0.3',
          'Authorization'=>'Token 26ac0f19-24f3-4acc-b993-8b3d0286e6a0',            
          'Content-Type'=>'application/json',                                                                                                                                     
          'User-Agent'=>'Ruby'
           }).                                                                           
         to_return(status: 200, body: "", headers: {})

       registered request stubs:            

       stub_request(:post, "https://api.buttondown.email/v1/subscribers").                                                                                                        
         with(                              
           body: "{\"email\":\"user@example.com\",\"metadata\":{},\"notes\":\"\",\"referrer_url\":\"www.blog.com\",\"tags\":[\"blog\"]}",                                                    headers: {                       
          'Accept'=>'*/*',
          'Accept-Encoding'=>'gzip;q=1.0,deflate;q=0.6,identity;q=0.3',
          'Content-Type'=>'application/json',
          'User-Agent'=>'Ruby'
           })                               

       Body diff:                           
        [["~", "email", "ana@test.com", "user@example.com"]]

There's not much more to it than this. VCR is another tool that also stubs API calls but works a little differently by actually making a first real request that will be saved in a file for future use. For simple API calls, WebMock does the trick for me!

How to access joined data with ActiveRecord

Ana Nunes da Silva — Sun, 11 Apr 2021 10:55:32 +0000

So far we've been using joins to support filtering based on related associations. But what if we not only want to filter but also access and use data from those associations?

Let's bring back the domain models we've been using during this series. This time Booking and Rating will be enough to illustrate some examples:

class Booking < ApplicationRecord
  has_many :ratings
end

class Rating < ApplicationRecord
  belongs_to :booking
end

Booking.all

Id	Check In	Check Out
62	20 Jun 2020	23 Jun 2020
63	07 Aug 2021	08 Aug 2021
64	22 May 2021	28 May 2021
65	01 Apr 2021	02 Apr 2021

Rating.all

Id	Rate	Comments	Booking Id
25	4	'Amazing'	62
26	3	'Coming back soon'	64
27	5	'friendly staff'	63

`pluck` vs `select`

Get a list of bookings that have been rated. The list should contain booking `ids` with their corresponding rating `comments`.

A simple way to return this list could be to join ratings to bookings and pluck the id values from bookings and the corresponding comments values from ratings. Remember that we can pass an SQL string to explicitly say which <table>.<column> we'd like to pluck:

Booking.joins(:ratings).pluck('bookings.id', 'ratings.comments')

In SQL terms, this means that we're inner joining ratings and selecting only the two columns that we need, id from bookings table and the comments from the ratings table.

SELECT "bookings"."id", "ratings"."comments" 
FROM "bookings" 
INNER JOIN "ratings" 
ON "ratings"."booking_id" = "bookings"."id"

Since we're using pluck, the output is an array of arrays with the id, comments combinations:

[[62, 'amazing'], [63, 'friendly staff'], [64,  'Coming back soon']]

If we'd like to transform this into an index/dictionary structure that is easier to read and consult we can use the .to_h method:

Booking.joins(:ratings).pluck('bookings.id', 'ratings.comments').to_h

There, much better:

{ 62 => 'Amazing',
  63 => 'friendly staff',
  64 => 'Coming back soon' }

So pluck returns the data prepared for us in a ruby array object. If we'd rather return an ActiveRecord collection, we can solve this exercise in a similar way using select instead:

Booking.joins(:ratings).select('bookings.id', 'ratings.comments')

This will translate in exactly the same SQL as before:

SELECT "bookings"."id", "ratings"."comments" 
FROM "bookings" 
INNER JOIN "ratings" 
ON "ratings"."booking_id" = "bookings"."id"

But now, instead of an array we're returning ActiveRecord objects:

 [#<Booking:0x00007f86084a4530 id: 62>,
 #<Booking:0x00007f86084a4418 id: 63>,
 #<Booking:0x00007f86084a4288 id: 64>]

Looking at this output, it looks like we've lost the comments data. But notice what happens when we ask one of these objects what attributes they have:

bookings = Booking.joins(:ratings).select('bookings.id', 'ratings.comments')

bookings.first.attributes

=> {"id"=>62, "comments"=>"Amazing"}

What's going on here? Activerecord's select is basically making all attributes that we've selected into ActiveRecord instance methods.

So if we do:

bookings = Booking.joins(:ratings).select('bookings.id', 'ratings.comments')

bookings.first.comments

=> "Amazing"

All this without triggering any extra queries.

So to get the list that we want, we can now iterate on each booking and print the available attributes:

Booking.joins(:ratings).select('bookings.id', 'ratings.comments').each do |booking|
   puts booking.id
   puts booking.comments
end

Which will print:

62
"Amazing"
63
"friendly staff"
64
"Coming back soon"

Be aware of n + 1 queries

It's important to mention that joins per se does not avoid n + 1 queries. We've been managing to avoid them so far because we're asking SQL to join and select the data we need in a single query.

Watch what happens if you don't use the select method:

Booking.joins(:ratings).each do |booking|
   puts booking.id
   puts booking.ratings.map(&:comments)
end

Note that now comments is no longer a booking instance method. That means that for each booking we'll have to get its ratings and from there add another iteration to get each rating's comments.

  Booking Load (3.0ms)  SELECT "bookings".* FROM "bookings" INNER JOIN "ratings" ON "ratings"."booking_id" = "bookings"."id"                                                                                      
  🤯 Rating Load (29.8ms)  SELECT "ratings".* FROM "ratings" WHERE "ratings"."booking_id" = $1  [["booking_id", 63]]                                                                                       
  🤯 Rating Load (0.3ms)  SELECT "ratings".* FROM "ratings" WHERE "ratings"."booking_id" = $1  [["booking_id", 64]]                  
  🤯 Rating Load (2.0ms)  SELECT "ratings".* FROM "ratings" WHERE "ratings"."booking_id" = $1  [["booking_id", 65]]

`eager_load`

If you don't want to select attributes prior to the iteration, you can eager_load the ratings data. What this will not do, however, is to make comments available as a booking instance method. So we'll have to get comments through the ratings:

Booking.joins(:ratings).eager_load(:ratings).each do |booking|
   puts booking.id
   puts booking.ratings.map(&:comments)
end

In SQL this will be:

 SELECT "bookings"."id" AS t0_r0, "bookings"."guest_id" AS t0_r1, "bookings"."accommodation_id" AS t0_r2, "bookings"."check_in" AS t0_r3, "bookings"."check_out" AS t0_r4, "bookings"."status" AS t0_r5, "bookings"."total_guests" AS t0_r6, "bookings"."created_at" AS t0_r7, "bookings"."updated_at" AS t0_r8, "ratings"."id" AS t1_r0, "ratings"."rate" AS t1_r1, "ratings"."comments" AS t1_r2, "ratings"."booking_id" AS t1_r3, "ratings"."created_at" AS t1_r4, "ratings"."updated_at" AS t1_r5 FROM "bookings" INNER JOIN "ratings" ON "ratings"."booking_id" = "bookings"."id"

Note that now you'll have all booking and all rating attributes available which can make the query heavier in some cases.

You can also use eager_load without the joins which will default into a left outer join:

Booking.eager_load(:ratings).each do |booking|
   puts booking.id
   puts booking.ratings.map(&:comments) if booking.ratings.present?
end

Here's the SQL:

SELECT "bookings"."id" AS t0_r0, "bookings"."guest_id" AS t0_r1, "bookings"."accommodation_id" AS t0_r2, "bookings"."check_in" AS t0_r3, "bookings"."check_out" AS t0_r4, "bookings"."status" AS t0_r5, "bookings"."total_guests" AS t0_r6, "bookings"."created_at" AS t0_r7, "bookings"."updated_at" AS t0_r8, "ratings"."id" AS t1_r0, "ratings"."rate" AS t1_r1, "ratings"."comments" AS t1_r2, "ratings"."booking_id" AS t1_r3, "ratings"."created_at" AS t1_r4, "ratings"."updated_at" AS t1_r5 FROM "bookings" LEFT OUTER JOIN "ratings" ON "ratings"."booking_id" = "bookings"."id"

Conclusion

There are several ways you can access data from joined associations:

You can use pluck if returning a simple ruby object is enough for you
If you'd like to get the benefits of returning an ActiveRecord collection use select instead
If you want to work with all the model and the joined association(s) attributes, you can use eager_load
Don't forget to keep a close eye on performance. While pluck and select can reduce the load time of your queries, combining multiple joins in a single query and making all the attributes accessible with eager_load can easily bloat them. In that case, consider breaking up your queries either using a preload strategy or writing separate queries that support each other.

That's it! Hope this 4 part series on understanding ActiveRecord joins was useful! Would love to hear about how you've been using joins too!

DEV Community: Ana Nunes da Silva

Authentication with Bcrypt

Authentication

Storing passwords

Hashing algorithms

Bcrypt

Install bcrypt

Generate a user model and corresponding db table

Add the has_secure_password method to the User model

Create user/session routes

Build the signup feature

Build the login/logout features

Require authentication

Add login/logout buttons

Happy authentication!

Exploring ViewComponent

What is ViewComponent

Views pattern issues?

Enter encapsulation

Refactoring rails views into components

Installation

Reusable components

Passing arguments

Collections

The content block

Slots

Testing

Going further

References:

Finding all tables with a specific column name (postgres)

The information schema

The query

Example of the query output:

Multiple schemas

Example of the query output:

Conclusion

Polymorphic joins in Active Record

The polymorphic associations

The issue with joining polymorphic associations

Using scoped associations

Pagination in Rails from scratch (no gems)

1. Creating a pagination mixin

Why a mixin?

The Pagination module

2. Writing the pagination logic

Preparing the pagination metadata

3. Building the frontend component

ruby methods - #tap and #then

In this post, I'll share the main difference between #tap and #then with specific use cases.

Looking in

The difference is in the return

Actions vs Transformations

Use cases

Using tap to call dependent actions

Using tap to create data with associations

Using tap to add multiple test expectations for the same object:

Using then to transform objects like strings or hashes:

Using then to chain queries

Final notes

Rails templates with docker

Before docker

After docker

Understanding docker - playing with ruby containers

Understanding the basics of docker and docker commands is the first step into app containerization.

Images vs Containers

The docker run command

The docker images command

The docker ps command

Passing other commands to the container

Stopping and removing containers

Keeping container processes open

Naming containers

First master the fundamentals!

Testing CSV and Excel uploads with tempfiles

Ordering fuzzy search results by relevancy using trigrams

Fast searches with trigram indexes

Other performance remarks: ILIKE vs LIKE

Conclusion

Testing external APIs with Rspec and WebMock

Testing code you don't own can be a bit trickier than writing tests for code you own. If you're interacting with 3rd party APIs, you don't actually want to make real calls every time you run a test.

Add the `has_secure_password` method to the User model

The `content` block

The `Pagination` module

Using `tap` to call dependent actions

Using `tap` to create data with associations

Using `tap` to add multiple test expectations for the same object:

Using `then` to transform objects like strings or hashes:

Using `then` to chain queries

The `docker run` command

The `docker images` command

The `docker ps` command

Other performance remarks: `ILIKE` vs `LIKE`

Testing behavior with `WebMock`

`pluck` vs `select`

Get a list of bookings that have been rated. The list should contain booking `ids` with their corresponding rating `comments`.

`eager_load`