DEV Community: K Putra

Rails: Fixing ‘Invalid query parameters: invalid %-encoding’

K Putra — Tue, 23 Mar 2021 02:45:22 +0000

Table of Contents:
Problem
A. Drop the query string
B. Fix the query string
Final word

Problem

ActionController::BadRequest
Invalid query parameters: invalid %-encoding (miconazole%202%%20cr)

Rack::QueryParser::InvalidParameterError
invalid %-encoding (miconazole%202%%20cr)

ActionController::BadRequest
ActionView::Template::Error: Invalid query parameters: invalid %-encoding (%2Fsearch%2Fall%Forder%3Ddescending%26page%3D5%26sort%3Dcreated_at)

In a full stack apps, sometimes users manually edit query strings in the address bar, and make requests that have invalid encodings.

In a api apps, sometimes front end or mobile engineer forget to encoding the query strings, so they just give raw query strings to your apps, then they left it for years until it became very hard to fixed on the client side, and the management decide the server side should fix it.

These corrupted query strings are not that important, but users are receiving 500 server error pages.

What can we do?

I can offer two choice: Drop the query string OR Fix the query string.

A. Drop the query string

I got this from this post right here.

First, make a middleware:

# You can put this everywhere
# In my case, I put this in app/middlewares/hande_bad_encoding_middleware.rb

class HandleBadEncodingMiddleware
  def initialize(app)
    @app = app
  end

  def call(env)
    begin
      Rack::Utils.parse_nested_query(env['QUERY_STRING'].to_s)
    rescue Rack::Utils::InvalidParameterError
      env['QUERY_STRING'] = ''
    end

    @app.call(env)
  end
end

Second, use this middleware before Rack::Runtime:

# config/application.rb

require_relative 'boot'

require 'rails/all'

# If you get NameError: uninitialized constant,
# you have to use require like this one:
require_relative '../app/middlewares/handle_bad_encoding_middleware.rb'

Bundler.require(*Rails.groups)

module YourAppsName
  class Application < Rails::Application

    ...
    # add this:
    config.middleware.insert_before Rack::Runtime, HandleBadEncodingMiddleware
    ...

  end
end

The result is, you drop the query string:

# from this
"https://your-url.com?query=alcohol%2070%"

# to this
"https://your-url.com"

B. Fix the query string

I did some research and modify the middleware. This is the step:

First (optional), I use gem rack-utf8_sanitizer to remove invalid UTF-8 characters from the query strings. This prevents errors like "invalid byte sequence in UTF-8".

I personally prefer rack-utf8_sanitizer than utf8-cleaner because it is more robust, more option, and less bugs.

Second, make a middleware:

# You can put this everywhere
# In my case, I put this in app/middlewares/hande_bad_encoding_middleware.rb

class HandleBadEncodingMiddleware
  def initialize(app)
    @app = app
  end

  def call(env)
    begin
      Rack::Utils.parse_nested_query(env['QUERY_STRING'].to_s)
    rescue Rack::Utils::InvalidParameterError
      params = URI.decode_www_form(env['QUERY_STRING'])
      query_string = URI.encode_www_form(params)
      env['QUERY_STRING'] = query_string
    end

    @app.call(env)
  end
end

There are many ways to encode and decode url:

URI.escape
URI.unescape
URI.encode_www_form
URI.decode_www_form
URI.encode_www_form_component
URI.decode_www_form_component
CGI.escape
CGI.unescape

I've tried them all, and I picked the best for me, which are URI.encode_www_form and URI.decode_www_form. Probably you have to pick which is best for you.

Third, use this middleware before Rack::Runtime:

# config/application.rb


require_relative 'boot'

require 'rails/all'

# If you get NameError: uninitialized constant,
# you have to use require like this one:
require_relative '../app/middlewares/handle_bad_encoding_middleware.rb'

Bundler.require(*Rails.groups)

module YourAppsName
  class Application < Rails::Application

    ...
    # you add this for the rack-utf8_sanitizer,
    # as described in the gem's readme:
    config.middleware.insert 0, Rack::UTF8Sanitizer
    # add this:
    config.middleware.insert_before Rack::Runtime, HandleBadEncodingMiddleware
    ...

  end
end

The result is, you fix the query string:

# from this
"https://your-url.com?query=alcohol%2070%"

# to this
"https://your-url.com?query=alcohol%2070%25"

Final Word

If you have any problems or questions, let me know in the comment.

If you are helped, I am glad I can help you.

Rails + PostgreSQL Array

K Putra — Mon, 29 Jun 2020 09:34:57 +0000

If you continue to read this article, I assume that you know Ruby, OOP in Ruby, RoR, and Active Record.

Yes, Postgresql support Array types to store. Based on their documentation:

PostgreSQL allows columns of a table to be defined as variable-length multidimensional arrays. Arrays of any built-in or user-defined base type, enum type, composite type, range type, or domain can be created.

Let's start our journey! (I use Rails API-only as example, but this article can be implemented in normal Rails as well)

Table of Contents:
1. Migration
2. Create
3. Show
4. Update
5. Query
6. Final Word

1. Migration

It is simple:

# db/migrate/*_create_books.rb
class CreateBooks < ActiveRecord::Migration[6.0]
  def change
    create_table :books do |t|
      t.string :title
      t.string :tags, array: true, default: []
      t.integer :ratings, array: true, default: []

      t.timestamps
    end
    add_index :books, :tags, using: 'gin'
    add_index :books, :ratings, using: 'gin'
  end
end

If you want to add new column:

# db/migrate/*_add_subjects_to_books.rb
class AddSubjectsToBooks < ActiveRecord::Migration
  def change
    add_column :books, :subjects, :string, array:true, default: []
  end
end

Notice:
I define the column as t.string :tags, array: true not t.array :tags. Compare to jsonb, which t.jsonb :payload. This is because there is no "array" type in PostgreSQL, only "array of column type". PostgreSQL arrays aren't generic containers like Ruby arrays, they are more like arrays in C, C++, etc.

2. Create

Create a record is very simple too:

irb(main):001:0> Book.create(title: "Hacking Growth", tags: ["business", "startup"], ratings: [4, 5])
   (0.1ms)  BEGIN
  Book Create (0.6ms)  INSERT INTO "books" ("title", "tags", "ratings", "created_at", "updated_at") VALUES ($1, $2, $3, $4, $5) RETURNING "id"  [["title", "Hacking Growth"], ["tags", "{business,startup}"], ["ratings", "{4,5}"], ["created_at", "2020-06-29 08:48:42.440895"], ["updated_at", "2020-06-29 08:48:42.440895"]]
   (0.4ms)  COMMIT
=> #<Book id: 1, title: "Hacking Growth", tags: ["business", "startup"], ratings: [4, 5], created_at: "2020-06-29 08:48:42", updated_at: "2020-06-29 08:48:42">

3. Show

Both tags and ratings now an array object:

irb(main):002:0> book = Book.first
  Book Load (0.3ms)  SELECT "books".* FROM "books" ORDER BY "books"."id" ASC LIMIT $1  [["LIMIT", 1]]
irb(main):003:0> book.tags
=> ["business", "startup"]
irb(main):004:0> book.tags[0]
=> "business"

4. Update

To update, the most easiest way is:

irb(main):005:0> book.tags << 'management'
=> ["business", "startup", "management"]
irb(main):0006:0> book.save!
   (0.1ms)  BEGIN
  Book Update (1.2ms)  UPDATE "books" SET "tags" = $1, "updated_at" = $2 WHERE "books"."id" = $3  [["tags", "{business,startup,management}"], ["updated_at", "2020-06-29 08:54:36.731328"], ["id", 1]]
   (0.4ms)  COMMIT
=> true
irb(main):007:0> book.tags
=> ["business", "startup", "management"]

And any other way to add a value to an array object:

# This works
book.tags << 'management'

#This will work too
book.tags.push 'management'

# This is also will work
book.tags += ['management']

But do not do this: Book.first.tags << 'finance', it won't be saved to the database. Prove:

irb(main):008:0> Book.first.tags << "finance"
  Book Load (0.3ms)  SELECT "books".* FROM "books" ORDER BY "books"."id" ASC LIMIT $1  [["LIMIT", 1]]
=> ["business", "startup", "management", "finance"]
irb(main):009:0> Book.first.save!
  Book Load (0.3ms)  SELECT "books".* FROM "books" ORDER BY "books"."id" ASC LIMIT $1  [["LIMIT", 1]]
=> true
irb(main):010:0> Book.first.tags
  Book Load (0.3ms)  SELECT "books".* FROM "books" ORDER BY "books"."id" ASC LIMIT $1  [["LIMIT", 1]]
=> ["business", "startup", "management"]

If you want to use raw SQL, you can check to the official documentation.

5. Query

Let say we want to search every single Book that have tags management:

# This is valid
irb(main):011:0> Book.where("'management' = ANY (tags)")

# This is more secure
irb(main):012:0> Book.where(":tags = ANY (tags)", tags: 'management')

# This is also valid
irb(main):013:0> Book.where("tags @> ?", "{management}")

What if we want to search every single book that DO NOT HAVE tags management:

irb(main):013:0> Book.where.not("tags @> ?", "{management}")

You can see the operators and their description in the official documentation.

Now, what if we want to search book that contain multiple tags, like management and startup:

# This is valid
irb(main):014:0> Book.where("tags @> ARRAY[?]::varchar[]", ["management", "startup"])

# This is valid
irb(main):015:0> Book.where("tags &&  ?", "{management,startup}")

# If you use where.not, you basically search for all that do not contain the parameter given.

Now what if we want to search all book that have rating more than 3:

irb(main):016:0> Book.where("array_length(ratings, 1) >= 3")

How about making our search a little bit more robust and supporting pattern matching:

# %gem% is manaGEMent 
irb(main):017:0> Book.where("array_to_string(tags, '||') LIKE :tags", tags: "%gem%")

You can see all the operators and functions and their description in the official documentation.

6. Final Word

That's all from me. I'll update if I find something interesting.

source: myself and extract from many articles

Rails + PostgreSQL JSONB (Part 3)

K Putra — Sun, 28 Jun 2020 14:44:32 +0000

Rails + PostgreSQL JSONB Series
Part 1: Migration and CRUD
Part 2: store_accessor, Query, and Index
Part 3: Paginate JSONB Data

If you continue to read this article, I assume that you know Ruby, OOP in Ruby, RoR, Active Record, and basic postgresql query

This Part 3 is covering about how to Paginate JSONB Data. But first, I'll deep dive into "playing" with the JSONB data. If you are not interested into deep dive, you can directly jump to Chapter 3: Select + flatten + pagination

The database schema in this article will be very different from Part 1 and Part 2. No connection between the previous Parts and this part.

The schema we used in this article:

class CreateUsers < ActiveRecord::Migration[5.2]
  def change
    create_table :users do |t|
      t.jsonb :payload, null: false, default: '{}'
    end
  add_index :users, :payload, using: :gin
  end
end

Then we seed our database with these data:

irb(main):001:1* payload_1 = [
irb(main):002:1*   {"name": "A", "age": 20},
irb(main):003:1*   {"name": "B", "age": 21},
irb(main):004:1*   {"name": "C", "age": 22},
irb(main):005:1*   {"name": "D", "age": 23},
irb(main):006:1*   {"name": "E", "age": 24},
irb(main):007:1*   {"name": "F", "age": 25},
irb(main):008:1*   {"name": "G", "age": 26}
irb(main):009:0> ]
irb(main):010:0>
irb(main):011:1* payload_2 = [
irb(main):012:1*   {"name": "H", "age": 27},
irb(main):013:1*   {"name": "I", "age": 28},
irb(main):014:1*   {"name": "J", "age": 29},
irb(main):015:1*   {"name": "K", "age": 30},
irb(main):016:1*   {"name": "L", "age": 31}
irb(main):017:0> ]
irb(main):018:0>
irb(main):019:1* payload_3 = [
irb(main):020:1*   {"name": "H", "age": 27},
irb(main):021:1*   {"name": "I", "age": 28},
irb(main):022:1*   {"name": "J", "age": 29},
irb(main):023:1*   {"name": "K", "age": 30},
irb(main):024:1*   {"name": "L", "age": 31}
irb(main):025:0> ]
irb(main):026:0> User.create(payload: payload_1)
   (0.1ms)  BEGIN
  User Create (1.0ms)  INSERT INTO "users" ("payload", "created_at", "updated_at") VALUES ($1, $2, $3) RETURNING "id"  [["payload", "[{\"name\":\"A\",\"age\":20},{\"name\":\"B\",\"age\":21},{\"name\":\"C\",\"age\":22},{\"name\":\"D\",\"age\":23},{\"name\":\"E\",\"age\":24},{\"name\":\"F\",\"age\":25},{\"name\":\"G\",\"age\":26}]"], ["created_at", "2020-06-29 00:05:49.725982"], ["updated_at", "2020-06-29 00:05:49.725982"]]
   (0.3ms)  COMMIT
=> #<User id: 1, payload: [{"name"=>"A", "age"=>20}, {"name"=>"B", "age"=>21}, {"name"=>"C", "age"=>22}, {"name"=>"D", "age"=>23}, {"name"=>"E", "age"=>24}, {"name"=>"F", "age"=>25}, {"name"=>"G", "age"=>26}], created_at: "2020-06-29 00:05:49", updated_at: "2020-06-29 00:05:49">
irb(main):027:0> User.create(payload: payload_2)
   (0.1ms)  BEGIN
  User Create (0.3ms)  INSERT INTO "users" ("payload", "created_at", "updated_at") VALUES ($1, $2, $3) RETURNING "id"  [["payload", "[{\"name\":\"H\",\"age\":27},{\"name\":\"I\",\"age\":28},{\"name\":\"J\",\"age\":29},{\"name\":\"K\",\"age\":30},{\"name\":\"L\",\"age\":31}]"], ["created_at", "2020-06-29 00:05:49.804221"], ["updated_at", "2020-06-29 00:05:49.804221"]]
   (0.5ms)  COMMIT
=> #<User id: 2, payload: [{"name"=>"H", "age"=>27}, {"name"=>"I", "age"=>28}, {"name"=>"J", "age"=>29}, {"name"=>"K", "age"=>30}, {"name"=>"L", "age"=>31}], created_at: "2020-06-29 00:05:49", updated_at: "2020-06-29 00:05:49">
irb(main):028:0> User.create(payload: payload_3)
   (0.1ms)  BEGIN
  User Create (0.3ms)  INSERT INTO "users" ("payload", "created_at", "updated_at") VALUES ($1, $2, $3) RETURNING "id"  [["payload", "[{\"name\":\"H\",\"age\":27},{\"name\":\"I\",\"age\":28},{\"name\":\"J\",\"age\":29},{\"name\":\"K\",\"age\":30},{\"name\":\"L\",\"age\":31}]"], ["created_at", "2020-06-29 00:05:51.025203"], ["updated_at", "2020-06-29 00:05:51.025203"]]
   (0.8ms)  COMMIT
=> #<User id: 3, payload: [{"name"=>"H", "age"=>27}, {"name"=>"I", "age"=>28}, {"name"=>"J", "age"=>29}, {"name"=>"K", "age"=>30}, {"name"=>"L", "age"=>31}], created_at: "2020-06-29 00:05:51", updated_at: "2020-06-29 00:05:51">

So, we have 3 row of User.

Let's start our journey!

Table of Contents:
1. Simple select
2. Select + flatten
3. Select + flatten + pagination
4. Select + flatten + pagination + nested
5. Select + flatten + pagination + nested + custom object
6. Select + flatten + pagination + nested + custom object + nested

1. Simple select

If we simply select all from User we get 3 rows with a JSONB array in the payload column.

SELECT * 
FROM users;  

----+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 id | payload
----+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  1 | [{"age": 20, "name": "A"}, {"age": 21, "name": "B"}, {"age": 22, "name": "C"}, {"age": 23, "name": "D"}, {"age": 24, "name": "E"}, {"age": 25, "name": "F"}, {"age": 26, "name": "G"}]
  2 | [{"age": 27, "name": "H"}, {"age": 28, "name": "I"}, {"age": 29, "name": "J"}, {"age": 30, "name": "K"}, {"age": 31, "name": "L"}]
  3 | [{"age": 32, "name": "M"}, {"age": 33, "name": "N"}, {"age": 34, "name": "O"}, {"age": 35, "name": "P"}, {"age": 36, "name": "Q"}]
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

In Rails (using Active Record), this method equal to:

irb(main):029:0> User.all
  User Load (0.3ms)  SELECT "users".* FROM "users" LIMIT $1  [["LIMIT", 11]]
=> #<ActiveRecord::Relation [#<User id: 1, payload: [{"age"=>20, "name"=>"A"}, {"age"=>21, "name"=>"B"}, {"age"=>22, "name"=>"C"}, {"age"=>23, "name"=>"D"}, {"age"=>24, "name"=>"E"}, {"age"=>25, "name"=>"F"}, {"age"=>26, "name"=>"G"}], created_at: "2020-06-29 00:05:49", updated_at: "2020-06-29 00:05:49">, #<User id: 2, payload: [{"age"=>27, "name"=>"H"}, {"age"=>28, "name"=>"I"}, {"age"=>29, "name"=>"J"}, {"age"=>30, "name"=>"K"}, {"age"=>31, "name"=>"L"}], created_at: "2020-06-29 00:05:49", updated_at: "2020-06-29 00:05:49">, #<User id: 3, payload: [{"age"=>27, "name"=>"H"}, {"age"=>28, "name"=>"I"}, {"age"=>29, "name"=>"J"}, {"age"=>30, "name"=>"K"}, {"age"=>31, "name"=>"L"}], created_at: "2020-06-29 00:05:51", updated_at: "2020-06-29 00:05:51">]>
irb(main):050:0>

2. Select + flatten

We can then flatten the JSONB array in the data column by using the jsonb_array_elements function. This will return 17 rows of id and data JSONB objects.

SELECT id, 
       jsonb_array_elements(payload) 
FROM users;

---------+--------------------------
 id | payload
---------+--------------------------
  1 | {"age": 20, "name": "A"}
  1 | {"age": 21, "name": "B"}
  1 | {"age": 22, "name": "C"}
  1 | {"age": 23, "name": "D"}
  1 | {"age": 24, "name": "E"}
  1 | {"age": 25, "name": "F"}
  1 | {"age": 26, "name": "G"}
  2 | {"age": 27, "name": "H"}
  2 | {"age": 28, "name": "I"}
  2 | {"age": 29, "name": "J"}
  2 | {"age": 30, "name": "K"}
  2 | {"age": 31, "name": "L"}
  3 | {"age": 32, "name": "M"}
  3 | {"age": 33, "name": "N"}
  3 | {"age": 34, "name": "O"}
  3 | {"age": 35, "name": "P"}
  3 | {"age": 36, "name": "Q"}
------------------------------------

In Rails (using Active Record), this method equal to:

irb(main):030:0> User.select("id", "jsonb_array_elements(payload) AS payload").all
  User Load (0.4ms)  SELECT "users"."id", jsonb_array_elements(payload) AS payload FROM "users" LIMIT $1  [["LIMIT", 11]]
=> #<ActiveRecord::Relation [#<User id: 1, payload: {"age"=>20, "name"=>"A"}>, #<User id: 1, payload: {"age"=>21, "name"=>"B"}>, #<User id: 1, payload: {"age"=>22, "name"=>"C"}>, #<User id: 1, payload: {"age"=>23, "name"=>"D"}>, #<User id: 1, payload: {"age"=>24, "name"=>"E"}>, #<User id: 1, payload: {"age"=>25, "name"=>"F"}>, #<User id: 1, payload: {"age"=>26, "name"=>"G"}>, #<User id: 2, payload: {"age"=>27, "name"=>"H"}>, #<User id: 2, payload: {"age"=>28, "name"=>"I"}>, #<User id: 2, payload: {"age"=>29, "name"=>"J"}>, ...]>

3. Select + flatten + pagination

We can then paginate the previous flatten query above.

SELECT id, 
       jsonb_array_elements(payload) 
FROM users 
LIMIT 5;

---------+---------------------------
 id | payload
---------+---------------------------
  1 | {"age": 20, "name": "A"}
  1 | {"age": 21, "name": "B"}
  1 | {"age": 22, "name": "C"}
  1 | {"age": 23, "name": "D"}
  1 | {"age": 24, "name": "E"}
-------------------------------------

SELECT id, 
       jsonb_array_elements(payload) 
FROM users 
LIMIT 5 OFFSET 5;

---------+---------------------------
 id | payload
---------+---------------------------
  1 | {"age": 25, "name": "F"}
  1 | {"age": 26, "name": "G"}
  2 | {"age": 27, "name": "H"}
  2 | {"age": 28, "name": "I"}
  2 | {"age": 29, "name": "J"}
-------------------------------------

In Rails (using Active Record), this method equal to:

irb(main):031:0> User.select("id", "jsonb_array_elements(payload) AS payload").all.limit(5).offset(5)
  User Load (0.3ms)  SELECT "users"."id", jsonb_array_elements(payload) AS payload FROM "users" LIMIT $1 OFFSET $2  [["LIMIT", 5], ["OFFSET", 5]]
=> #<ActiveRecord::Relation [#<User id: 1, payload: {"age"=>25, "name"=>"F"}>, #<User id: 1, payload: {"age"=>26, "name"=>"G"}>, #<User id: 2, payload: {"age"=>27, "name"=>"H"}>, #<User id: 2, payload: {"age"=>28, "name"=>"I"}>, #<User id: 2, payload: {"age"=>29, "name"=>"J"}>]>

The controller and routes might look like this:

# app/controllers/users_controller.rb
class UsersController < ApplicationController
  def index
    page = params[:page].to_i
    per_page = params[:per_page].to_i
    offset = (page - 1) * per_page
    payload = User.select("id", "jsonb_array_elements(payload) AS payload").all.limit(per_page).offset(offset)
    render json: { object: payload }, status: 200
  end
end

# config/routes.rb
Rails.application.routes.draw do
  resources :users
end

Let's fire up the server with rails s, then go to localhost:3000/users?page=1&per_page=5 in the browser/postman (GET method):

{
    "object": [
        {
            "id": 1,
            "payload": {
                "age": 25,
                "name": "F"
            }
        },
        {
            "id": 1,
            "payload": {
                "age": 26,
                "name": "G"
            }
        },
        {
            "id": 2,
            "payload": {
                "age": 27,
                "name": "H"
            }
        },
        {
            "id": 2,
            "payload": {
                "age": 28,
                "name": "I"
            }
        },
        {
            "id": 2,
            "payload": {
                "age": 29,
                "name": "J"
            }
        }
    ]
}

If you want to only record from one User, say User with id 1, the controller might look like this:

# app/controllers/users_controller.rb
class UsersController < ApplicationController
  def index
    payload = JSON.parse "[#{ActiveRecord::Base.connection.exec_query(sql).rows.join(',')}]"
    render json: { object: payload }, status: 200
  end

  private

  def sql
    "
    SELECT jsonb_array_elements(payload)
    FROM users
    WHERE id = #{params[:id]}
    LIMIT #{params[:per_page]} OFFSET #{(params[:page].to_i - 1) * params[:per_page].to_i}
    "
  end
end

Let's go to localhost:3000/users?page=1&per_page=5&id=1 again, and the result will be:

{
    "object": [
        {
            "age": 20,
            "name": "A"
        },
        {
            "age": 21,
            "name": "B"
        },
        {
            "age": 22,
            "name": "C"
        },
        {
            "age": 23,
            "name": "D"
        },
        {
            "age": 24,
            "name": "E"
        }
    ]
}

4. Select + flatten + pagination + nested

We can take this 1 step further and can group by id again and put the JSON back into an array using the jsonb_agg function:

WITH users_unnested as (
    SELECT id, 
           jsonb_array_elements(payload) as payload
    FROM users 
    LIMIT 5 OFFSET 5
)
SELECT id, jsonb_agg (payload) 
FROM users_unnested
GROUP BY id;

---------+--------------------------------------------------------------------------------
 id | payload
---------+--------------------------------------------------------------------------------
  1 | [{"age": 25, "name": "F"}, {"age": 26, "name": "G"}]
  2 | [{"age": 27, "name": "H"}, {"age": 28, "name": "I"}, {"age": 29, "name": "J"}]
---------+--------------------------------------------------------------------------------

5. Select + flatten + pagination + nested + custom object

We can take the previous query and re-construct a new object with new fields e.g. user_id and user_info. This will return a single column with a new custom JSONB object (again a row per id).

WITH users_unnested as (
    SELECT id, 
           jsonb_array_elements(payload) as payload
    FROM users
    LIMIT 5 OFFSET 5
),
users_info as (
    SELECT jsonb_build_object (
               'user_id', id,
               'user_info', jsonb_agg (payload)
           ) as user
     FROM users_unnested
     GROUP BY id
)
SELECT user from users_info;

----------------------------------------------------------------------------------------------------------------------
 user
----------------------------------------------------------------------------------------------------------------------
 {"user_id": 1, "user_info": [{"age": 25, "name": "F"}, {"age": 26, "name": "G"}]}
 {"user_id": 2, "user_info": [{"age": 27, "name": "H"}, {"age": 28, "name": "I"}, {"age": 29, "name": "J"}]}
----------------------------------------------------------------------------------------------------------------------

6. Select + flatten + pagination + nested + custom object + nested

The previous query returned 2 rows, we can create a single row by once again using the jsonb_agg function i.e.

WITH users_unnested as (
    SELECT id, 
           jsonb_array_elements(payload) as payload
    FROM users
    LIMIT 5 OFFSET 5
),
users_info as (
    SELECT jsonb_build_object (
               'user_id', id,
               'user_info', jsonb_agg (payload)
           ) as user
     FROM users_unnested
     GROUP BY id
)
SELECT jsonb_agg(user) from users_info;
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 user
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 [{"person_id": 1, "person_info": [{"age": 25, "name": "F"}, {"age": 26, "name": "G"}]}, {"person_id": 2, "person_info": [{"age": 27, "name": "H"}, {"age": 28, "name": "I"}, {"age": 29, "name": "J"}]}]
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

source: here

Rails: Policy Objects Implementation

K Putra — Thu, 02 Jan 2020 07:17:42 +0000

If you continue to read this article, I assume that you know Ruby, OOP in Ruby, and RoR.

What is Policy Object Design Pattern?

Policy Objects encapsulate and express a single business rule.

In our applications we might have different business rules coded mostly as if-else or switch statements. These rules represent concepts in your domain like whether “a customer is eligible for a discount” or whether “an email is supposed to be sent or not” or even whether “a player should be awarded a point”.

source: this article

Let's start our journey! (I use Rails API-only as example, but this article can be implemented in normal Rails as well)

Table of Contents:
1. Problem
2. Moving Policy Logic to Model
3. Create a separate class
4. Conclusion

1. Problem

Let say we have this kind of controller:

# app/controllers/discounts_controller.rb
class DiscountsController < ApplicationController
  def create
    if can_user_get_discount?
      code = GenerateDiscountVoucherCode.new(@current_user.id).call
      render json: { status: "OK", message: "Your Discount Voucher Code: #{code}" }, status: 201
    else
      render json: { status: "Failed", message: "You are not allowed to get Discount Voucher" }, status: 422
    end
  end

  private

  def can_user_get_discount?
    is_premium? &&
    last_discount_more_than_10_days_ago? &&
    high_buyer?
  end

  def is_premium?
    @current_user.premium?
  end

  def last_discount_more_than_10_days_ago?
    @current_user.last_discount_sent_at < ten_days_ago
  end

  def ten_days_ago
    Time.now - 10.days
  end

  def high_buyer?
    @current_user.total_purchase_this_month > 5_000
  end
end

You put all the policy logic (business rule) in your controller. This is not good. Especially if you have many/complex policy logic.

Btw, you can ignore GenerateDiscountVoucherCode class. We just need to know that this class is the one that responsible for generating the discount voucher code.

2. Moving Policy Logic to Model

Well, we can move the logic to our model. So, it become like this:

# app/controllers/discounts_controller.rb
class DiscountsController < ApplicationController
  def create
    if @current_user.can_get_discount?
      code = GenerateDiscountVoucherCode.new(@current_user.id).call
      render json: { status: "OK", message: "Your Discount Voucher Code: #{code}" }, status: 201
    else
      render json: { status: "Failed", message: "You are not allowed to get Discount Voucher" }, status: 422
    end
  end
end

# app/models/user.rb
class User < ApplicationRecord
  enum membership: ['regular', 'premium']
  MINIMUM_PURCHASE = 5_000

  def can_get_discount?
    self.premium? &&
    self.last_discount_more_than_10_days_ago? &&
    self.high_buyer?
  end

  def last_discount_more_than_10_days_ago?
    self.last_discount_sent_at < ten_days_ago
  end

  def ten_days_ago
    Time.now - 10.days
  end

  def high_buyer?
    self.total_purchase_this_month > MINIMUM_PURCHASE
  end
end

Still not good. Now we just bloat our model. Let's implement Policy Object Design Pattern!

3. Create a separate class

Now, we create a class:

# app/lib/discount_voucher_policy.rb
class DiscountVoucherPolicy
  MINIMUM_PURCHASE = 5_000

  def initialize(user)
    @user = user
  end

  def allowed?
    is_premium? &&
    last_discount_more_than_10_days_ago? &&
    high_buyer?
  end

  private

  def is_premium?
    @user.premium?
  end

  def last_discount_more_than_10_days_ago?
    @user.last_discount_sent_at < ten_days_ago
  end

  def ten_days_ago
    Time.now - 10.days
  end

  def high_buyer?
    @user.total_purchase_this_month > MINIMUM_PURCHASE
  end
end

Now, what are our controller and model looks like?

# app/controllers/discounts_controller.rb
class DiscountsController < ApplicationController
  def create
    if policy.allowed?
      code = GenerateDiscountVoucherCode.new(@current_user.id).call
      render json: { status: "OK", message: "Your Discount Voucher Code: #{code}" }, status: 201
    else
      render json: { status: "Failed", message: "You are not allowed to get Discount Voucher" }, status: 422
    end
  end

  private

  def policy
    DiscountVoucherPolicy.new(@current_user)
  end
end

# app/models/user.rb
class User < ApplicationRecord
  enum membership: ['regular', 'premium']
end

4. Conclusion

Compared to our first controller, this is "just" putting our policy logic to a separate class. It is true, as our main intention is moving our policy logic out of controller and model.

Policy Objects encapsulate and express a single business rule.

You'll find this design pattern useful if you have complex policy logic. Example I gave just a simple policy logic.

source: myself

Rails + PostgreSQL JSONB (Part 2)

K Putra — Mon, 30 Dec 2019 04:19:52 +0000

Rails + PostgreSQL JSONB Series
Part 1: Migration and CRUD
Part 2: store_accessor, Query, and Index
Part 3: Paginate JSONB Data

If you continue to read this article, I assume that you know Ruby, OOP in Ruby, RoR, Active Record, and PostgreSQL Query Language.

Before we start, if you haven't read Part 1, here is our schema:

# db/migrate/*_create_books.rb
class CreateBooks < ActiveRecord::Migration[5.2]
  def change
    create_table :books do |t|
      t.integer :user_id
      t.jsonb :payload, null: false, default: '{}'
    end
  add_index :books, :payload, using: :gin
  end
end

book_payload = {
  title: 'Hacking Growth',
  publisher: 'Currency',
  published_date: '2017-04-07',
  authors: [
    {
      id: 1,
      name: 'Sean Ellis'
    },
    {
      id: 2,
      name: 'Morgan Brown'
    }
  ]
}

Book.create(user_id: 1, payload: book_payload)

Let's start our journey!

Table of Contents:
1. store_accessor
2. Query
3. Index
4. Final Word

1. store_accessor

If you frequently access some attributes, you can use store_accessor. When you use store_accessor, you can call the column like normal column type.

# app/models/book.rb
class Book < ApplicationRecord
  belongs_to :user
  serialize :payload, JsonbSerializers
  store_accessor :payload, :title, :publisher, :authors
end

Book.last.payload['title']
# => "Hacking Growth"

Book.last.title
# => "Hacking Growth"

Book.last.publisher
# => "Currency"

Book.last.authors
# => [{"id"=> 1, "name"=>"Sean Ellis"}, {"id"=>2, "name"=>"Morgan Brown"}]

This is not the only usability of store_accessor. More about it:

# Create
Book.create(user_id: 1, title: 'The godvader', publisher: 'Mario', published_date: '2002-03-01', authors: [{ id: 19, name: 'Mario Puzo' }])

# Update
book = Book.last
book.title = 'The Godfather'
book.publisher = 'NAL'
book.save

You can add validation too if you want! By this, you can combine JSON Schema and Active Record Validations for more powerful validation.

# app/models/book.rb
class Book < ApplicationRecord
  belongs_to :user
  serialize :payload, JsonbSerializers
  store_accessor :payload, :title, :publisher, :authors

  validates :title, length: { in: 3..50 }
end

In short, store_accessor is just a shortcut which defines getter and setter methods.

BUT, store_accessor does not allow you to access nested keys. You can only access the first layer. Say, you have this schema inside column payload in a model named City:

{
  geolocation: {
    latitude: 48.856613,
    longitude: 2.352222
  },
  detail: {
    name: "Paris",
    url: "https://www.latlong.net/c/?lat=48.856613&long=2.352222"
  }
}

You can only do this:

# app/models/city.rb
class City < ApplicationRecord
  store_accessor :payload, :geolocation, :detail
end

You can't access latitude, longitude, name, and url using store_accessor.

If you still want to do something like City.last.geolocation_latitude, then we can set our getter and setter methods manually:

# app/models/city.rb
class City < ApplicationRecord
  store_accessor :payload, :geolocation, :detail

  def geolocation_latitude
    self.geolocation['latitude']
  end

  def geolocation_latitude=(value)
    self.geolocation['latitude'] = value
  end

  # add these getter and setter each for 3 more times:
  # geolocation_longitude, detail_name, detail_url
end

This way, you can access them as it is using store_accessor.

city = City.last
city.geolocation_latitude
# => 48.856613

city.detail_name
# => "Paris"

city.detail_name = 'Rome'
city.save

city.detail
# => {"name"=>"Rome", "url"=>"https://www.latlong.net/c/?lat=48.856613&long=2.352222"}

Tips: You can implement Ruby Metaprogramming, so you don't have to create getter and setter methods one by one:

# app/models/city.rb
class City < ApplicationRecord
  store_accessor :payload, :geolocation, :detail
  SCHEMA = {
    'geolocation' => ['latitude', 'longitude'],
    'detail' => ['name', 'url']
  }

  SCHEMA.each do |key, val|
    val.each do |method|
      define_method "#{key}_#{method}" do
        self.send("#{key}")[method]
      end

      define_method "#{key}_#{method}=" do |arg|
        self.send("#{key}")[method] = arg
      end
    end
  end
end

I've never done this before, because, as I've mentioned, we use store_accessor for some attributes that we use frequently.

2. Query

I'll just explain queries that I frequently used. Please refer to official documentation of PostgreSQL for full information about query.

Using JSONB column type, we can't rely 100% using ORM. We have to understand basic query, especially JSONB type query.

# Matches where 'Book' contains 'title': 'Hacking Growth'
Book.where("payload ->> 'title' = :title", title: 'Hacking Growth')

# This is just the same as above, but not save from SQL injection
Book.where("payload ->> 'title' = ?", "Hacking Growth")

# Same as the first example
Book.where("payload @> ?", { title: 'Hacking Growth' }.to_json)

# Matches where 'Book' first author's name is Sean Ellis
Book.where("payload -> 'authors' -> '0' ->> 'name' = :name", name: 'Sean Ellis')

# Same as above
Book.where("payload #>> '{authors,0,name}' = :name", name: 'Sean Ellis'

# Same as above
Book.where("payload -> 'authors' -> '0' @> :val", val: { name: 'Sean Ellis' }.to_json)

Now, go back to our City model from #1.

# Matches where 'City' contains 'detail':{'name': 'Paris'}
City.where("payload -> 'detail' ->> 'name' = :name", name: 'Paris')

# Same as above
City.where("payload #>> '{detail,name}' = :name", name: 'Paris')

# Same as above
City.where("payload @> :val", val: { detail: { name: 'Paris'}}.to_json)

# Same as above
City.where("payload -> 'detail' @> :val", val: { name: 'Paris'}.to_json)

# Search City that has name AND url in payload['detail']
City.where("payload ->> 'detail' ?& array[:keys]", keys: ['name', 'url'])

# Search City that has name OR url in payload['detail']
City.where("payload ->> 'detail' ?& array[:keys]", keys: ['name', 'url'])

Notice:

payload do not use quotation mark ('') because it is the name of the column. Whereas the attributes of the json always use quotation mark (''). Remember, you can name your column as you want, as long as ruby, rails, and postgresql allowed it.

When to use -> and ->> ? We use the -> operator to keep returning objects until we reach the final attribute where it’s ok to use ->> to return as text.

So if you have:

{
  this: {
    very: {
      deep: {
        nested: 'yeah'
      }
    }
  }
}

You do:

Model.where("payload -> 'this' -> 'very' -> 'deep' ->> 'nested' = :val", val: "yeah")

# Btw, this is shorter:
Model.where("payload #>> '{this,very,deep,nested}' = :val", val: "yeah")

If you want to search for integer/float, you need to add explicit type casts:

Book.where("(payload #>> '{authors,0,id}')::int = :val", val: 1)

City.where("(payload -> 'geolocation' ->> 'latitude')::float = :val", val: 48.856613)

3. Index

Add index for JSONB column types is supported for rails >=5.0.0. I only cover for rails >=5.0.0.

Mostly, we are using GIN index. (Read this official documentation about GIN/GiST index types for more information.)

Let say we want to add partial index for title in our Book model, and we want to add partial index for name of the first author. Create it is as simple as this:

# db/migrate/*_create_books.rb
class CreateBooks < ActiveRecord::Migration[5.2]
  def change
    create_table :books do |t|
      t.integer :user_id
      t.jsonb :payload, null: false, default: '{}'
    end
  add_index :books, :payload, "payload ->> 'title'", using: :gin, name: "index_pictures_on_title"

  add_index :books, :payload, "payload #>> '{authors,0,name}'", using: :gin, name: "index_pictures_on_first_author_name"
  end
end

4. Final Word

This is the end of Part 2, and also the end of Series Rails + PostgresQL JSONB. I'll update this Series if I found something useful.

source: myself and extract from many source, I don't save them, I just write what I remember

Rails + PostgreSQL JSONB (Part 1)

K Putra — Sat, 28 Dec 2019 16:31:55 +0000

Rails + PostgreSQL JSONB Series
Part 1: Migration and CRUD
Part 2: store_accessor, Query, and Index
Part 3: Paginate JSONB Data

If you continue to read this article, I assume that you know Ruby, OOP in Ruby, RoR, and Active Record.

Not so many article cover this topic. Even the existing articles did not cover in details. I have to read many articles before I can perform well.

This is the first time I'll deep dive into a topic.

You need Rails >= 4.2, PostgreSQL >= 9.4

PostgreSQL has JSON and JSONB column type. They are look the same, but there are many differences, especially in functionality. I recommend to use JSONB instead of JSON, as JSONB column type is the upgraded version of JSON.

Why we should use JSONB column type?

The advantage of using jsonb is that you can easily integrate relational and non-relation data, with performance that can be better than most non-relational databases like MongoDB.

source: this article

Let's start our journey! (I use Rails API-only as example, but this article can be implemented in normal Rails as well)

Table of Contents:
1. Migration
2. Create
3. Show
4. Validation
5. Update
6. Final Word

1. Migration

It is as simple as any other column types.

# db/migrate/*_create_books.rb
class CreateBooks < ActiveRecord::Migration[5.2]
  def change
    create_table :books do |t|
      t.integer :user_id
      t.jsonb :payload, null: false, default: '{}'
    end
  add_index :books, :payload, using: :gin
  end
end

Notice:

First, you don't have to use null: false, default: '{}', but I recommend to use it. Simply because you don't have to check if it is nil or not.

# if you allow nil OR dont't state default value to a hash
Book.create(user_id: 1)
Book.last.payload['title']
# => NoMethodError (undefined method `[]' for nil:NilClass)

# if you don't allow nil AND state default value to a hash
Book.create(user_id: 1)
Book.last.payload['title']
# => nil

Second, we’re also defining a GIN index. I'll cover this in Part 2.

Third, I named the column name as payload, but you can named it as you want as long as ruby, rails, and postgresql allowed it.

2. Create

Create a record is very simple too:

book_payload = {
  title: 'Hacking Growth',
  publisher: 'Currency',
  published_date: '2017-04-07',
  authors: [
    {
      id: 1,
      name: 'Sean Ellis'
    },
    {
      id: 2,
      name: 'Morgan Brown'
    }
  ]
}

Book.create(user_id: 1, payload: book_payload)

There you go!

3. Show

It is very simple to show the data. Note that any column defined as JSONB will be represented as a hash, with all keys are returned as string.

Book.last.user_id
# => 1

Book.last.payload['title']
# => "Hacking Growth"

Book.last.payload['authors'][0]['name']
# => "Sean Ellis"

Now, if you want a custom serializer so you can access your JSON object using symbols as well, you can do like this:

# app/models/book.rb
class Book < ApplicationRecord
  belongs_to :user
  serialize :payload, JsonbSerializers
end

# app/serializers/jsonb_serializers.rb
class JsonbSerializers
  def self.dump(hash)
    hash.to_json
  end

  def self.load(hash)
    (hash || {}).with_indifferent_access
  end
end

Book.last.payload[:title]
# => "Hacking Growth"

4. Validation

The problems with JSOB column type is if you don't give validation then it will be schema free. You can input any schema to this column. Take a look at the code below:

book_payload = {
  title: 'Getting to Plan B',
  pubs: 'Harvard Business Review Press',
  authors: 'John W Mullins'
  rating: 3.5
}

Book.create(user_id: 1, payload: book_payload)

The record above will be committed in DB. In most scenario, this is a big problem.

How to validate the params given by API consumer is right or wrong? We have to do two steps:

Using JSON Schema
Whitelist params

First step, we are using JSON Schema. I won't cover what JSON Schema is. The official page here is perfect enough to be understood.

How to use JSON Schema in Rails? We are using gems. There are few gems, but the only one I've ever use is activerecord_json_validator gems. (Take note that this gem only provide using draft-04 of JSON Schema)

Let's make our JSON Schema for our Book model. We can put the schemas wherever we want, but I like to put them in app/models/schemas.

To learn about JSON Schema, you can use this and this. Those two are enough.

// app/models/schemas/book_payload.json
{
  "$schema": "http://json-schema.org/draft-04/schema",
  "type": "object",
  "required": [ "title", "publisher", "published_date", "authors" ],
  "properties": {
    "title": { "type": "string" },
    "publisher": { "type": "string" },
    "published_date": { "type": "string" },
    "authors": {
      "type": "array",
      "items": {
        "$ref": "authors.json#/definitions/data"
      },
      "minItems": 1,
      "uniqueItems": true
    }
  }
}

// app/models/schemas/authors.json
{
  "definitions": {
    "data": {
      "type": "object",
      "required": [ "id", "name" ],
      "properties": {
        "id": { "type": "integer" },
        "name": { "type": "string" }
      }
    }
  }
}

Then, we put to our Gemfile, and bundle install:

gem 'activerecord_json_validator'

Then we update our model. (To learn about this gem, their github page is more than enough.)

# app/models/book.rb
class Book < ApplicationRecord
  ...

  PAYLOAD_SCHEMA = "#{Rails.root}/app/models/schemas/book_payload.json"
  validates :payload, presence: true, json: { message -> (err) { err }, schema: PAYLOAD_SCHEMA }
end

First step is finish. Now, look at the code below:

book_payload = {
  title: 'Getting to Plan B',
  pubs: 'Harvard Business Review Press',
  authors: 'John W Mullins'
  rating: 3.5
}

Book.create!(user_id: 1, payload: book_payload)

The code above will throw validation error, as the json is not valid. There are no publisher param, no publisher_date param, and authors param is not an array.

Then take a look at this code:

book_payload = {
  title: 'Getting to Plan B',
  publisher: 'Harvard Business Review Press',
  published_date: '2009-09-08',
  pubs: 'Harvard Business Review Press',
  authors: [
    {
      id: 3,
      name: 'John W Mullins',
      web: 'http://www.johnwmullins.com'
    }
  ],
  rating: 4.2
}

Book.create!(user_id: 1, payload: book_payload)

The code above will NOT throw validation error! Because it is a valid json. And you can call pubs and rating too!

Book.last.payload['pubs']
# => "Harvard Business Review Press"

Book.last.payload['rating']
# => 4.2

Book.last.payload['authors'][0]['web']
# => "http://www.johnwmullins.com"

Imagine if API Consumer give 100 parameters for every request, then your data storage will be used for garbages that you'll never need!

That's why we need the whitelisting params.

Second step, we are whitelisting params in controllers. This is necessary not only for JSONB column type, but for any other column types.

Let's update our controller:

# app/controllers/books_controller.rb
class BooksController < ApplicationController
  def create
    book = Book.create!(book_params)
    render json: { status: "OK", message: "Book created!", object: book }, status: 201
  end

  private

  def book_params
    params.permit(
      payload: [
        :title,
        :publisher,
        :published_date,
        authors: [
          :id,
          :name
        ]
      ]
    )
  end
end

You know how whitelisting params works. I think I don't have to explain the code above.

That's it. Now you have validation for your jsonb column.

5. Update

Note: All of the examples in this chapter are assuming you don't have validation for payload column in Book.

Updating JSONB column is a little tricky. Let say you only want to change the title of the last record of Book. If you do this:

Book.last.payload
# => {"title"=>"Hacking Growth", "publisher"=>"Currency", "published_date"=>"2017-04-07", "authors"=>[{"id"=>1, "name"=>"Sean Ellis"}, {"id"=>2, "name"=>"Morgan Brown"}]}

book_payload = {
  title: 'Blue Ocean'
}

Book.last.update(payload: book_payload)

Now your payload only consist of title!

Book.last.payload
# => {"title"=>"Blue Ocean"}

Book.last.payload['publisher']
# => nil

Book.last.payload['authors']
# => nil

This is the right way:

book = Book.last
book.payload['title'] = 'Blue Ocean'
book.save!

Book.last.payload
# => {"title"=>"Blue Ocean", "publisher"=>"Currency", "published_date"=>"2017-04-07", "authors"=>[{"id"=>1, "name"=>"Sean Ellis"}, {"id"=>2, "name"=>"Morgan Brown"}]}

Tips for updating JSONB column:

# app/controllers/books_controller.rb
class BooksController < ApplicationController
  def update
    UpdateBookPayload.new(update_params).call
    render json: { status: "OK", message: "Book updated!" }, status: 200
  end

  private

  def update_params
    params.permit(
      :book_id,
      :title,
      :publisher,
      :published_date,
      authors: [
        :id,
        :name
      ]
    )
  end
end

# app/lib/update_book_payload.rb
class UpdateBookPayload
  def initialize(params)
    @params = params
    @book   = book
  end

  def call
    iterate_params
    @book.save!
  end

  private

  def book
    Book.find(@params[:book_id])
  end

  def iterate_params
    params = @params.delete('book_id')
    params.each do |key1, value1|
      if key1 == 'authors'
        iterate_authors(key1, value1)
      else
        @book.payload[key1] = (value1 || @book.payload[key1])
      end
    end
  end

  def iterate_authors(key1, value1)
    value1.each_with_index do |value2, key2|
      value2.each do |key3, value3|
        @book.payload[key1][key2][key3] = (value3 || @book.payload[key1][key2][key3])
      end
    end
  end
end

Therefor, the API Consumer can pass any json to BE. We can check the class in perform using rails console:

Book.last.id
# => 10

Book.last.payload
# => {"title"=>"Hacking Growth", "publisher"=>"Currency", "published_date"=>"2017-04-07", "authors"=>[{"id"=>1, "name"=>"Sean Ellis"}, {"id"=>2, "name"=>"Morgan Brown"}]}

params = {
  book_id: 10,
  title: 'Blue Ocean',
  authors: [
    {},
    { id: 5 }
  ]
}

UpdateBookPayload.new(params).call

Book.last.payload
# => {"title"=>"Blue Ocean", "publisher"=>"Currency", "published_date"=>"2017-04-07", "authors"=>[{"id"=>1, "name"=>"Sean Ellis"}, {"id"=>5, "name"=>"Morgan Brown"}]}

6. Final Word

This is the end of Part 1. In Part 2, I'll cover about store_accessor, Query and Indexing.

source: myself and extract from many source, I don't save them, I just write what I remember

Rails: Background Job Done Right (Case 1)

K Putra — Fri, 27 Dec 2019 17:27:44 +0000

If you continue to read this article, I assume that you know Ruby, OOP in Ruby, RoR, Active Record, Sidekiq, and ruby metaprogramming.

This article series is the implementation of sidekiq's best practice using real case from my work.

Perhaps you should read these articles before we start:
Coding Sidekiq Workers the Right Way
Improving Rails Performance with Better Background Jobs

Let's start our journey!

Table of Contents:
1. Problem
2. First Solution (bad)
3. Second Solution (better)
4. Third Solution (much better)
5. Flow Comparison
6. The Code
7. Final Word

1. Problem

Every 4 hours, I have to check if there is any Invoice that need to be checked to 3rd party API. If there is any, I have to do http request to 3rd API per record found. There are 6 types of Invoices. They are unique, so there are 6 models: InvoiceA..InvoiceF.

To do this, I have to use background job. And Sidekiq is my choice.

For the cron job, I use sidekiq-cron gems

2. First Solution (bad)

The example of my initial solution for the worker was:

# app/workers/scheduled/check_status_worker.rb
class Scheduled::CheckStatusWorker
  include Sidekiq::Worker

  def perform
    invoice_as = InvoiceA.where(status: "unfinished")
    check_status_of_(invoice_as)

    # looping b, c, d ,e

    invoice_fs = InvoiceF.where(status: "unfinished")
    check_status_of_(invoice_fs)
  end

  def check_status_of_(invoices)
    invoices.each do |invoice|
      invoice.check_status
    end
  end
end

Notice:

I don't use sidekiq_options retry: false. Well, if I use it, then if an exception happen in the middle of the job, the rest of the records won't be checked to 3rd party. Let say I have 10,000 records, and exception occured in 500th record, then 9,500 records won't be checked.

I use each method for each models. If there are 10,000 records per model, so this worker will be running for a loong time.

I call check_status from the instance model. This is not good. I should make a class to checked to 3rd party API. Remember this

I put the logic in the worker. This is also not good.

In this solution, I have 6 models and 1 worker.

What can we improve?

3. Second solution (better)

This solution consist of 3 phases.

First, I'll move the logic from worker to class. In this phase, I'll have 6 models, 1 worker, and 1 class (new).

The code of the worker and the class:

# app/workers/scheduled/check_status_worker.rb
class Scheduled::CheckStatusWorker
  include Sidekiq::Worker

  def perform
    StartScheduledCheckStatus.new.call
  end
end

# app/lib/start_scheduled_check_status.rb
class StartScheduledCheckStatus
  INVOICE = %w(invoice_a invoice_b invoice_c invoice_d invoice_e invoice_f)

  def call
    INVOICE.each { |inv| send("check_#{inv}" }
  end

  private

  INVOICE.each do |method|
    define_method "check_#{method}" do
      clazz = method.camelize
      invoices = clazz.constantize.where(status: "unfinished")
      check_status_of_(invoices)
    end
  end

  def check_status_of_(invoices)
    invoices.each do |invoice|
      invoice.check_status
    end
  end
end

In this phase, no improvement in perfomance. Only clean code.

Second, I'll move check_status from every models to stand alone classes. The check_status method for every models is simulated like this:

# app/models/invoice_a.rb
class InvoiceA < ApplicationRecord
  def check_status
    # action-1. build message
    # action-2. do http request
    # action-3. action based on the response
  end
end

The url of the API is the same (https://url/check_status), so action-2 is the same across models. Action-3 is the same across models. But the messages (action-1) is unique for each models.

Knowing this, I'll make 1 class to do action-2 and action-3, and 6 classes to do action-1.

In this phase, I'll have 6 models, 1 worker, and 1 class + 7 classes (new).

The code will be look like these:

# app/lib/start_scheduled_check_status.rb
class StartScheduledCheckStatus
  ...
  private

  INVOICE.each do |method|
    define_method "check_#{method}" do
      clazz = method.camelize
      invoices = clazz.constantize.where(status: "unfinished")
      check_status_of_(invoices, clazz: clazz)
    end
  end

  def check_status_of_(invoices, clazz: clazz)
    invoices.each do |invoice|
      CheckStatus.new(invoice, clazz).call
    end
  end
end

# app/lib/check_status.rb
class CheckStatus
  def initialize(invoice, clazz)
    @invoice = invoice
    @clazz   = clazz
  end

  def call
    message = "#{@clazz}MessageBuilder".constantize.new(@invoice).call
    # action-2: do http request using message
    # action-3: action based on the response
  end
end

# app/lib/invoice_a_message_builder.rb
class InvoiceAMessageBuilder
  def initialize(invoice)
    @invoice = invoice
  end

  def call
    # action-1: build message using @invoice
  end
end

Phew, that was long (remember, there are 6 message builder class). But in this phase, no improvement in perfomance. Only clean code. Let's go to next phase.

Third, I'll make a worker for CheckStatus. This is the improvement in performance. We push CheckStatus to a worker per record, so Scheduled::CheckStatusWorker does not have to run http request at all! If there are 10,000 records found, so there will be 10,000 jobs. This is background job done right!!

In this phase, I'll have 6 models, 1 worker + 1 worker (new), and 8 classes.

So, the code will be like these:

# app/lib/start_scheduled_check_status.rb
class StartScheduledCheckStatus
  ...

  def check_status_of_(invoices, clazz: clazz)
    invoices.each do |invoice|
      CheckStatusWorker.perform_async(invoice.id, clazz)
    end
  end
end

# app/workers/check_status_worker.rb
class CheckStatusWorker
  include Sidekiq::Worker
  sidekiq_options retry: false

  def perform(invoice_id, clazz)
    CheckStatus.new(invoice_id, clazz).call
  end
end

# app/lib/check_status.rb
class CheckStatus
  def initialize(invoice_id, clazz)
    @invoice_id = invoice_id
    @clazz      = clazz
    @invoice    = invoice
  end

  def call
    message = "#{@clazz}MessageBuilder".constantize.new(@invoice).call
    # action-2: do http request using message
    # action-3: action based on the response
  end

  private

  def invoice
    @clazz.constantize.find(@invoice_id)
  end
end

Notice:

Now I use sidekiq_options retry: false, because if one job is failed, it won't affect another jobs. Let say there are 10,000 jobs, 1 job failed the 9,999 other jobs will still running.

Can I improve again? Yes, we can!

4. Third Solution (much better)

This solution consist of 2 phases.

First, don't use each method. But use push_bulk method instead! Why should we use this?

The push_bulk method allows us to push a large number of jobs to Redis. This method cuts out the redis network round trip latency. I wouldn't recommend pushing more than 1000 per call but YMMV based on network quality, size of job args, etc. A large number of jobs can cause a bit of Redis command processing latency.

source: github

In this phase, I'll have 6 models, 2 workers, and 8 classes.

Let's upgrade our code a bit:

# app/lib/start_scheduled_check_status.rb
class StartScheduledCheckStatus
  INVOICE = %w(invoice_a invoice_b invoice_c invoice_d invoice_e invoice_f)
  SIZE = 1000

  def call
    INVOICE.each { |inv| send("check_#{inv}" }
  end

  private

  INVOICE.each do |method|
    define_method "check_#{method}" do
      clazz = method.camelize
      invoices_ids = get_invoices_ids_from_(clazz)
      check_status_of_(invoices_ids, clazz: clazz)
    end
  end

  def get_invoices_ids_from_(clazz)
    clazz.constantize.where(status: "unfinished").pluck(:id)
  end

  def check_status_of_(invoices_ids, clazz: clazz)
    args = get_array_of_args_from_(invoices_ids, clazz)
    args.each do |arg|
      Sidekiq::Client.push_bulk('class' => CheckStatusWorker, 'args' => arg)
    end
  end

  def get_array_of_args_from_(ids, clazz)
    arrs = ids.in_groups_of(SIZE, false)
    arrs.map! { |arr| arr.in_groups_of(1) }
    arrs.map! do |arr|
      arr.map! do |a|
        a << clazz
      end
    end
    arrs
  end
end

Notice:

I still use each. Before, every each means I put 1 invoice into a single job per invoice. But this time, every each means I put 1,000 invoices into a single job per invoice (bulk push).

The size of invoices is divided into 1000, as it is the best practice of sidekiq based on the official guide (see quoted text aboout push_bulk)

Second, we can improve our first worker. Have a little bit logic in worker is not a crime.

In this phase, I'll have 6 models, 2 workers + 1 worker (new), and 8 classes.

# app/workers/scheduled/check_status_worker.rb
class Scheduled::CheckStatusWorker
  include Sidekiq::Worker

  INVOICE = %w(invoice_a invoice_b invoice_c invoice_d invoice_e invoice_f)

  def perform
    INVOICE.each do |inv|
      SeparatedScheduledCheckStatusWorker.perform_async(inv)
    end
  end
end

# app/workers/separated_scheduled_check_status_worker.rb
class SeparatedScheduledCheckStatusWorker
  include Sidekiq::Worker

  def perform(inv)
    StartScheduledCheckStatus.new(inv).call
  end
end

# app/lib/start_scheduled_check_status.rb
class StartScheduledCheckStatus
  INVOICE = %w(invoice_a invoice_b invoice_c invoice_d invoice_e invoice_f)
  SIZE = 1000

  # inv is one of INVOICE
  def initialize(inv)
    @inv = inv
  end

  def call
    send("check_#{@inv}")
  end
  ...

Notice:

Before, we have 1 job for 6 models. Now we separate 1 job to become 6 jobs, which means 1 job per model.

This is another memory-usage improvement.

And that's all !

5. Flow Comparison

Flow in First Solution:

Every 4 hour, cron will start 1 job: Scheduled::CheckStatusWorker
This job will search unfinished invoice from Invoice A
Every invoice found will be checked one-by-one to 3rd party API using http request
The job will continue to search unfinished invoice from Invoice B
Once again, every invoice found will be checked one-by-one to 3rd party API
And over and over until Invoice F.

This is like The One Ring:

One Job to rule them all, One Job to find them, One Job to bring them all, and in the http request bind them.

Remember, The One Ring is the source of problems!

Flow in Third Solution:

Every 4 hour, cron will start 1 job: Scheduled::CheckStatusWorker
This job will start 6 other jobs: SeparatedScheduledCheckStatusWorker, one job for one model
All the 6 jobs will search unfinished invoice from their respective model
Every invoice found will be pushed bulk to job CheckStatusWorker. One job for one invoice

You can see the huge differences from First Solution and Third Solution. There will be huge improvement to our apps !

6. The Code

From 6 models and 1 worker, we end up have 6 models, 3 workers, and 8 classes.

This is the entire code:

# app/workers/scheduled/check_status_worker.rb
class Scheduled::CheckStatusWorker
  include Sidekiq::Worker

  INVOICE = %w(invoice_a invoice_b invoice_c invoice_d invoice_e invoice_f)

  def perform
    INVOICE.each do |inv|
      SeparatedScheduledCheckStatusWorker.perform_async(inv)
    end
  end
end

# app/workers/separated_scheduled_check_status_worker.rb
class SeparatedScheduledCheckStatusWorker
  include Sidekiq::Worker

  def perform(inv)
    StartScheduledCheckStatus.new(inv).call
  end
end

# app/lib/start_scheduled_check_status.rb
class StartScheduledCheckStatus
  INVOICE = %w(invoice_a invoice_b invoice_c invoice_d invoice_e invoice_f)
  SIZE = 1000

  def initialize(inv)
    @inv = inv
  end

  def call
    send("check_#{@inv}")
  end

  private

  INVOICE.each do |method|
    define_method "check_#{method}" do
      clazz = method.camelize
      invoices_ids = get_invoices_ids_from_(clazz)
      check_status_of_(invoices_ids, clazz: clazz)
    end
  end

  def get_invoices_ids_from_(clazz)
    clazz.constantize.where(status: "unfinished").pluck(:id)
  end

  def check_status_of_(invoices_ids, clazz: clazz)
    args = get_array_of_args_from_(invoices_ids, clazz)
    args.each do |arg|
      Sidekiq::Client.push_bulk('class' => CheckStatusWorker, 'args' => arg)
    end
  end

  def get_array_of_args_from_(ids, clazz)
    arrs = ids.in_groups_of(SIZE, false)
    arrs.map! { |arr| arr.in_groups_of(1) }
    arrs.map! do |arr|
      arr.map! do |a|
        a << clazz
      end
    end
    arrs
  end
end

# app/workers/check_status_worker.rb
class CheckStatusWorker
  include Sidekiq::Worker
  sidekiq_options retry: false

  def perform(invoice_id, clazz)
    CheckStatus.new(invoice_id, clazz).call
  end
end

# app/lib/check_status.rb
class CheckStatus
  def initialize(invoice_id, clazz)
    @invoice_id = invoice_id
    @clazz      = clazz
    @invoice    = invoice
  end

  def call
    message = "#{@clazz}MessageBuilder".constantize.new(@invoice).call
    # action-2: do http request using message
    # action-3: action based on the response
  end

  private

  def invoice
    @clazz.constantize.find(@invoice_id)
  end
end

# app/lib/invoice_a_message_builder.rb
class InvoiceAMessageBuilder
  def initialize(invoice)
    @invoice = invoice
  end

  def call
    # action-1: build message using @invoice
  end
end

Note: There are 6 models, so there are 6 message builder classes.

7. Final Word

Sorry for bad english. And sorry for bad naming classes, methods, and workers. I'll improve this article later.

source: myself

Rails: Bulk Insert to DB

K Putra — Fri, 27 Dec 2019 13:58:40 +0000

If you continue to read this article, I assume that you know Ruby, OOP in Ruby, RoR, and Active Record.

The purposes of this article is only to let you know that you can do bulk insert, and tell you some available options to do that. Therefor, I won't deep dive into each options.

Table of Contents:
Problem
A. Using gems
B. Using the SQL INSERT statement
C. Rails 6
Final Word

Problem

Do you have import function for csv/excel files into your rails apps and then put the records into database? If you have, I believe some of you still do something like this:

def import_record
  CSV.read(file_path).each do |record|
     Article.create!(
       title:  record[0],
       author: record[1],
       body:   record[2]
     )
  end
end

The code above will invoke SQL calls as much as the total records in the csv file. If you have 20 records in the file, then it will make 20 SQL calls.

If you have only 10 records per files, perhaps it is not a big problem. But, what if you were facing 10,000 records per files? 1,000,000 records per file?

Well, 1,000,000 SQL calls are.....

Can we 'refactor' it? Yes we can! Just use bulk insert, then we will only call 1 SQL calls !

Let's start our journey!

A. Using gems

I believe most of the reader is not using Rails 6. But if you do, you can go straigth to C. Rails 6.

I will only cover 1 gem: activerecord-import.

Put this in your Gemfile, then run bundle install:

gem 'activerecord-import'

As the name suggests, one of the dependency of this gem is active record. Make sure your apps use active record as ORM.

Then, let's update our code above:

def import_record
  articles = array_of_records
  Article.import articles
end

def array_of_records
  records = []
  CSV.read(file_path).each do |record|
    records << extract_(record)
  end
  records
end

def extract_(record)
  {
    title: record[0],
    author: record[1],
    body: record[2]
  }
end

So, instead of make 20 SQL calls, your code will make only 1 SQL calls!

Not so much improvement for 20 records. But so much improvement for 10,000 records, 1,000,000 records, 20,000,000 records, and so on.

Read the github documentation for full information.

B. Using the SQL INSERT statement

No need to install any gems if you are using rails. But you have to understand query language.

def import_record
  articles = array_of_records
  sql = build_sql_from_(articles)
  ActiveRecord::Base.connection.insert_sql(sql)
end

def build_sql_from_(articles)
  sql = "INSERT INTO articles VALUES"
  sql_values = []
  articles.each do |article|
    sql_values << "(#{article.values.join(", ")})"
  end
  sql += sql_values.join(", ")
end

def array_of_records
  ...
end

def extract_(record)
  ...
end

I never use this options, as this require deep knowledge of the database itself.

C. Rails 6

Rails 6 support bulk insert without any gems.

So, let's update our code:

def import_record
  articles = array_of_records
  Article.insert_all articles
end

def array_of_records
  ...
end

def extract_(record)
  ...
end

Not so much different from using activerecord-import. We just change import to insert_all.

Of course there are many bulk insert method besides insert_all in Rails 6. There are many options too, just like activerecord-import.

If you want to know in detail, there are many great articles with great authors that write in detail about bulk insert in Rails 6. You can easily search for it.

Final Word

Bulk Insert is not only for import file like the scenario given. There are many situation where bulk insert can save the day.

That's all from me.

Rails: Facade Design Pattern for Index

K Putra — Thu, 26 Dec 2019 16:14:10 +0000

If you continue to read this article, I assume that you know Ruby, OOP in Ruby, RoR, Active Record, and a little bit metaprogramming.

What is Facade Design Pattern?

Facade Design Patter provide a unified interface to a set of interfaces in a subsystem. Facade defines a higher-level interface that makes the subsystem easier to use.

source: GoF in Design Patterns - Elements of Reusable Object-Oriented Software

Perhaps you should read these articles before we start:
Rails: Query Object Pattern Implementation
Rails: skinny controller, skinny model

Let's start our journey! (I use Rails API-only as example, but this article can be implemented in normal Rails as well)

Table of Contents:
1. Problems
2. First Solution (bad)
3. Second Solution (a bit better)
4. Third Solution (much better)
5. The Cost
6. Final Word

1. Problems

In Fintax, the FE team asked me (the only one BE developer) to give this kind of response for index api:

{
    "object": [
        {
            "column": "object1"
        },
        {
            "column": "object2"
        }
    ],
    "page": {
        "total_record": 20,
        "per_page": 2
    }
}

Okay, it's not hard. Let's move to first solution.

2. First Solution (bad)

Let assume we use will_paginate gems (I actually use that). For beginner, these will be their controllers:

# app/controllers/users_controller.rb
class UsersController < ApplicationController
  def index
    users = User.where(nil)
    users = users.where(role: params[:role]) if params[:role]
    users = users.where(status: params[:status]) if params[:status]
    users = users.where(public_id: params[:public]) if params[:public]
    users = users.paginate(page: params[:page], per_page: 2)
    users_count = users.count
    render json: { object: users, page: { total_record: users_count, per_page: 2 }}, status: 200
  end
end

# app/controllers/companies_controller.rb
class CompaniesController < ApplicationController
  def index
    companies = Company.where(nil)
    companies = companies.where("name like ?", "#{params[:name]}%") if params[:name]
    companies = companies.where(tax_id: params[:tax]) if params[:tax]
    companies = companies.paginate(page: params[:page], per_page: 2)
    companies_count = companies.count
    render json: { object: companies, page: { total_record: companies_count, per_page: 2 }}, status: 200
  end
end

This kind of controllers is very bad, because of various reasons:

I have a dozen of models that need to be indexed (and this number will grow in the future), so this kind of code not only very repetitive, but also very hard to maintained;
The total objects per page are not dynamic, so if the FE team asked to change it, we have to change all controllers;
We send every columns exist in the model to FE;
etc.

Let's move to our next solution.

3. Second Solution (a bit better)

We will use filterable module which I explain in my previous article.

We will use pluck. To be precise, I use pluck_all gems. Why we use pluck? So we don't send every columns to FE. We just give them what they need. Why we don't use as_json or map? Because pluck is better in performance.

We will make our pagination to become dynamic. Tell FE team that they can throw per_page parameter if someday they want to change total objects per page.

So, our controllers and models will be like this:

# app/controllers/users_controller.rb
class UsersController < ApplicationController
  def index
    users = User.call_index(params)
    users_count = users.count
    users = users.show_index
    render json: { object: users, page: { total_record: users_count, per_page: (params[:per_page] || 2) }}, status: 200
  end
end

# app/controllers/companies_controller.rb
class CompaniesController < ApplicationController
  def index
    companies = Company.call_index(params)
    companies_count = companies.count
    companies = companies.show_index
    render json: { object: companies, page: { total_record: companies_count, per_page: (params[:per_page] || 2) }}, status: 200
  end
end

# app/models/user.rb
class User < ApplicationRecord
  include Filterable

  scope :role,   -> (role) { where(role: role) }
  scope :status, -> (status) { where(status: status) }
  scope :public, -> (public_id) { where(public_id: public_id) }

  def self.call_index(params)
    filter(params.slice(:role, :status, :public))
    .paginate(page: params[:page], per_page: (params[:per_page] || 2)
  end

  def self.show_index
    pluck_all(:id, :role, :status, :public)
  end
end

# app/models/company.rb
class Company < ApplicationRecord
  include Filterable

  scope :name, -> (name) { where("name like ?", "#{name}%") }
  scope :tax,  -> (tax_id) { where(tax_id: tax_id) }

  def self.call_index(params)
    filter(params.slice(:name, :tax))
    .paginate(page: params[:page], per_page: (params[:per_page] || 2)
  end

  def self.show_index
    pluck_all(:id, :name, :tax)
  end
end

I hope you realize 2 things:

First, I use pluck after counting total object.

If I use count after pluck, I call method count from Array class, not count from Active Record. So, I'll get length of array instead of total object from active record.

Second, if you have already read Rails: Refactor Your Where Method, notice that I delete filtering_(params) method from controllers, and filter the params in models.

If I filter the params in controller as before and add page and per_page parameters, then the filter method will search scope named page and per_page. Because these scopes do not exist, then rails will throw error.

Tips: There are many ways to move params.slice() to another method. Why we need to move it? Imagine you have 10 keys in the params hash! One of the ways are: we create class inside model, like the code below. That way, the code is more cleaner (method should do one thing), and params.slice() is reusable inside the model.

# app/models/user.rb
class User < ApplicationRecord
  include Filterable

  scope :role,   -> (role) { where(role: role) }
  scope :status, -> (status) { where(status: status) }
  scope :public, -> (public_id) { where(public_id: public_id) }

  def self.call_index(params)
    filter(Whitelisting.call(params))
    .paginate(page: params[:page], per_page: (params[:per_page] || 2)
  end

  def self.show_index
    pluck_all(:id, :role, :status, :public)
  end

  class Whitelisting
    def self.call(params)
       params.slice(:role, :status, :public)
    end
  end
end

Okay, this is better than initial solution. Now, let's implement Facade Pattern !

4. Third Solution (much better)

Let's implement Facade Pattern. We'll make app/lib directory, and create index_facade.rb.

In short, index_facade.rb is the unified interface to a set of interfaces in a subsystem.

# app/lib/index_facade.rb
class IndexFacade
  def initialize(model, params)
    @model   = model.constantize
    @params  = params
    @objects = objects
  end

  def call
    {
      object: @objects.show_index
      page: {
        total_record: @objects.counts
        per_page: per_page
      }
    }
  end

  private

  def objects
    @model.call_index(@params)
  end

  def per_page
    @params[:per_page] || 2
  end
end

Now, we'll update our controllers:

# app/controllers/users_controller.rb
class UsersController < ApplicationController
  def index
    users = IndexFacade.new('User', params).call
    render json: users, status: 200
  end
end

# app/controllers/companies_controller.rb
class CompaniesController < ApplicationController
  def index
    companies = IndexFacade.new('Company', params).call
    render json: companies, status: 200
  end
end

Voila! Look at our new controllers!

5. The Cost

The only costs are:

In every model, we have to include Filterable.

In every model, we have to add call_index and show_index method, as in User and Company.

What if I want index of Article is giving every column and have no filter?

# app/models/article.rb
class Article < ApplicationController
  include Filterable

  def self.call_index(params)
    filter(params.slice())
    .paginate(page: params[:page], per_page: (params[:per_page] || 2)
  end

  def self.show_index
    pluck_all
  end
end

# app/controllers/articles_controller.rb
class ArticlesController < ApplicationController
  def index
    articles = IndexFacade.new('Article', params).call
    render json: articles, status: 200
  end
end

6. Final Word

If you have opinion or better implementation, or may be I was wrong about Facade Pattern, let discuss in comment.

source: myself

Rails: skinny controller, skinny model

K Putra — Wed, 25 Dec 2019 16:34:08 +0000

If you continue to read this article, I assume that you know Ruby, OOP in Ruby, and RoR. Perhaps this article is for beginner in rails.

Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live.

This is important. Well-structured code has many benefits, one of it is easy to maintained. In rails, the first sign of well-structured code is skinny controller, skinny model.

Let's start our journey! (I use Rails API-only as example, but this article can be implemented in normal Rails as well)

Table of Contents:
Phase 1: Fat Controller, Skinny Model
Phase 2: Skinny Controller, Fat Model
Phase 3: Skinny Controller, Skinny Model
Supplement

Phase 1: Fat Controller, Skinny Model

So, if you are new in rails, probably you just put your business logics in controllers, causing your controllers have hundreds lines of code. On the other hand your models are only have a few lines of code. This is called fat controller, skinny model.

It is not ilegal. But always remember what I quoted in the beginning of this article!

I'll use example. Let say we have these 2 controllers and 2 models:

# app/controllers/users_controller.rb
# total: 37 lines
class UsersController < ApplicationController
  def create
    user = User.create!(user_params)
    # 10 lines of BusinessLogic-A
    # 5 lines of BusinessLogic-B
    render json: { status: "OK", message: "User created!" }, status: 201
  end

  def update
    user = User.update!(user_params)
    # 5 lines of BusinessLogic-B
    render json: { status: "OK", message: "User updated!" }, status: 200
  end

  private

  def user_params
    params.permit(:role, :public_id, :email, :password)
  end
end

# app/controllers/companies_controller.rb
# total: 80 lines
class CompaniesController < ApplicationController
  def create
    company = Company.create!(company_params)
    # 20 lines of BusinessLogic-C
    # 14 lines of BusinessLogic-D
    render json: { status: "OK", message: "Company created!" }, status: 201
  end

  def update
    company = Company.create!(company_params)
    # 20 lines of BusinessLogic-C
    # 9 lines of BusinessLogic-E
    render json: { status: "OK", message: "Company updated!" }, status: 200
  end

  private

  def company_params
      params.permit(:name, :tax_id)
  end
end

# app/models/user.rb
# total: 3 lines
class User < ApplicationRecord
  has_secure_password
end

# app/models/company.rb
# total: 2 lines
class Company < ApplicationRecord
end

What is the first thing you notice? I hope it is that the controllers both have redundant code: UsersController has two BusinessLogic-B written, CompaniesController has two BusinessLogic-C written.

Well, you might get killed!

You can put, let say, BusinessLogic-C as method in CompaniesController, so it does look like this:

# app/controllers/companies_controller.rb
# total: 46 lines (from 80 lines)
class CompaniesController < ApplicationController
  def create
    @company = Company.create!(company_params)
    business_logic_c
    # 14 lines of BusinessLogic-D
    render json: { status: "OK", message: "Company created!" }, status: 201
  end

  def update
    @company = Company.create!(company_params)
    business_logic_c
    # 9 lines of BusinessLogic-E
    render json: { status: "OK", message: "Company updated!" }, status: 200
  end

  private

  def company_params
      params.permit(:name, :tax_id)
  end

  def business_logic_c
    # 20 lines of BusinessLogic-C
  end
end

So you reduce CompaniesController from 80 lines to 46 lines. Not so bad, you have implemented DRY. But it is not good enough.

Why? Let's move to next phase.

Phase 2: Skinny Controller, Fat Model

To answer your last question, I'll quote from other:

“Fat Model, Skinny Controller” refers to how the M and C parts of MVC ideally work together. Namely, any non-response-related logic should go in the model, ideally in a nice, testable method. Meanwhile, the “skinny” controller is simply a nice interface between the view and model.

In practice, this can require a range of different types of refactoring, but it all comes down to one idea: by moving any logic that isn’t about the response to the model (instead of the controller), not only have you promoted reuse where possible but you’ve also made it possible to test your code outside of the context of a request.

source

So, let's move our BusinessLogic from controllers to models!

# app/controllers/users_controller.rb
# total: 20 lines (from 37 lines)
class UsersController < ApplicationController
  def create
    user = User.create!(user_params)
    user.business_logic_a
    user.business_logic_b
    render json: { status: "OK", message: "User created!" }, status: 201
  end

  def update
    user = User.update!(user_params)
    user.business_logic_b
    render json: { status: "OK", message: "User updated!" }, status: 200
  end
  ...
end

# app/controllers/companies_controller.rb
# total: 21 lines (from 80 lines)
class CompaniesController < ApplicationController
  def create
    company = Company.create!(company_params)
    company.business_logic_c
    company.business_logic_d
    render json: { status: "OK", message: "Company created!" }, status: 201
  end

  def update
    company = Company.create!(company_params)
    company.business_logic_c
    company.business_logic_e
    render json: { status: "OK", message: "Company updated!" }, status: 200
  end
  ...
end

# app/models/user.rb
# total: 24 lines (from 3 lines)
class User < ApplicationRecord
  has_secure_password

  def business_logic_a
    # 10 lines of BusinessLogic-A
  end

  def business_logic_b
    # 5 lines of BusinessLogic-A
  end
end

# app/models/company.rb
# total: 53 lines (from 2 lines)
class Company < ApplicationRecord
  def business_logic_c
    # 20 lines of BusinessLogic-C
  end

  def business_logic_d
    # 14 lines of BusinessLogic-D
  end

  def business_logic_e
    # 9 lines of BusinessLogic-E
  end
end

As you can see, the more complex your code is, the skinnier your controller, and the fatter your model.

Note: You can use callbacks if it is suitable for your need. Callbacks are powerful tools in rails. I won't cover callbacks in this article.

And finally, we will be going to the last phase!

Phase 3: Skinny Controller, Skinny Model

"Fat Model, Skinny Controller" is a very good first step, but it doesn't scale well once your codebase starts to grow.

Let's think on the Single Responsibility of models. What is the single responsibility of models? Is it to hold business
logic? Is it to hold non-response-related logic?

No. Its responsibility is to handle the persistence layer and its abstraction.

Business logic, as well as any non-response-related logic and non-persistence-related logic, should go in domain
objects.

Domain objects are classes designed to have only one responsibility in the domain of the problem. Let your classes
"Scream Their Architecture" for the problems they solve.

In practice, you should strive towards skinny models, skinny views and skinny controllers. The architecture of your
solution shouldn't be influenced by the framework you're choosing

source

In practice, you can put Domain Objects in lib or you make app/lib. If you put in app/lib, so you have to know about autoloading in rails. If you put in lib, this is worth reading

In this article, I'll just put our Domain Objects in app/lib. You have to make this directory first. In your root folder, run:

$ mkdir app/lib

Now the directory is ready. Let's start!

# app/controllers/users_controller.rb
# total: 20 lines (from 20 lines)
class UsersController < ApplicationController
  def create
    user = User.create!(user_params)
    BusinessLogicA.new
    BusinessLogicB.new
    render json: { status: "OK", message: "User created!" }, status: 201
  end

  def update
    user = User.update!(user_params)
    BusinessLogicB.new
    render json: { status: "OK", message: "User updated!" }, status: 200
  end
  ...
end

# app/controllers/companies_controller.rb
# total: 21 lines (from 21 lines)
class CompaniesController < ApplicationController
  def create
    company = Company.create!(company_params)
    BusinessLogicC.new
    BusinessLogicD.new
    render json: { status: "OK", message: "Company created!" }, status: 201
  end

  def update
    company = Company.create!(company_params)
    BusinessLogicC.new
    BusinessLogicE.new
    render json: { status: "OK", message: "Company updated!" }, status: 200
  end
  ...
end

# app/models/user.rb
# total: 3 lines (from 24 lines)
class User < ApplicationRecord
  has_secure_password
end

# app/models/company.rb
# total: 2 lines (from 53 lines)
class Company < ApplicationRecord
end

# app/lib/business_logic_a.rb
class BusinessLogicA
  # 10 lines of BusinessLogic-A
end

# app/lib/business_logic_b.rb
class BusinessLogicB
  # 5 lines of BusinessLogic-B
end

# app/lib/business_logic_c.rb
class BusinessLogicC
  # 20 lines of BusinessLogic-C
end

# app/lib/business_logic_d.rb
class BusinessLogicD
  # 14 lines of BusinessLogic-D
end

# app/lib/business_logic_e.rb
class BusinessLogicE
  # 9 lines of BusinessLogic-E
end

So you have skinny controller, skinny model, but you add 5 more files with total of 78 lines. Adding these files is necessary. Why? Read again the quoted text in this phase (read quoted text).

Supplement

This is not the end. You still has many things to learn so your code is more beautiful and easier to maintained. Some of them are:

Ruby Design pattern: you can start from here
Rails Design pattern: you can start from here
Sandi Matz Rule: you can start from here
Clean code: you can start from here
Rails callbacks: you can start from here
Rails concerns: you can start from here
Ruby Metaprogramming: you can start from here
And many more!

Thanks for reading.

Rails: Query Object Pattern Implementation

K Putra — Wed, 25 Dec 2019 03:50:38 +0000

If you continue to read this article, I assume that you know Ruby, OOP in Ruby, RoR, Active Record, and a little bit Ruby Metaprogramming.

What is Query Object Pattern?

Query Objects are classes specifically responsible for handling complex SQL queries, usually with data aggregation and filtering methods, that can be applied to your database.

source: this article

In this article, we won't create separate class, but we create a module that can be reusable for all models.

Let's start our journey! (I use Rails API-only as example, but this article can be implemented in normal Rails as well)

Table of Contents:
1. Beginning
2. Using scope in model
3. Add metaprogramming
4. Endgame - skinny controller
5. The hidden cost
6. Final thoughts

1. Beginning

Let say we have these kind of controllers:

# app/controllers/users_controller.rb
class UsersController < ApplicationController
  def index
    users = User.where(nil)
    users = users.where(role: params[:role]) if params[:role]
    users = users.where(status: params[:status]) if params[:status]
    users = users.where(public_id: params[:public]) if params[:public]
    render json: users, status: 200
  end
end

# app/controllers/companies_controller.rb
class CompaniesController < ApplicationController
  def index
    companies = Company.where(nil)
    companies = companies.where("name like ?", "#{params[:name]}%") if params[:name]
    companies = companies.where(tax_id: params[:tax]) if params[:tax]
    render json: companies, status: 200
  end
end

Not so good, eh. Let's use scope in our models.

2. Using scope in model

Now we implement scope in User model and Company model. What is scope? And how we use it? You can start with this official guide of rails.

Let's add scope in our models, and we will update our controllers too!

# app/models/user.rb
class User < ApplicationRecord
  scope :role,   -> (role) { where(role: role) }
  scope :status, -> (status) { where(status: status) }
  scope :public, -> (public_id) { where(public_id: public_id) }
end

# app/models/company.rb
class Company < ApplicationRecord
  scope :name, -> (name) { where("name like ?", "#{name}%") }
  scope :tax,  -> (tax_id) { where(tax_id: tax_id) }
end

# app/controllers/users_controller.rb
class UsersController < ApplicationController
  def index
    users = User.where(nil)
    users = users.role(params[:role]) if params[:role]
    users = users.status(params[:status]) if params[:status]
    users = users.public(params[:public]) if params[:public]
    render json: users, status: 200
  end
end

# app/controllers/companies_controller.rb
class CompaniesController < ApplicationController
  def index
    companies = Company.where(nil)
    companies = companies.name(params[:name]) if params[:name]
    companies = companies.tax(params[:tax]) if params[:tax]
    render json: companies, status: 200
  end
end

Well, our controllers is a bit more beautiful. But, they aren't satisfying enough.

3. Add metaprogramming

If you don't know about Ruby Metaprogramming, you can start with my article about Ruby Metaprogramming. If you don't want to read the full article, just read chapter #3.

Now, let's upgrade our controllers using send()

# app/controllers/users_controller.rb
class UsersController < ApplicationController
  def index
    users = User.where(nil)
    params.slice(:role, :status, :public).each do |key, value|
      users = users.send(key, value) if value
    end
    render json: users, status: 200
  end
end

# app/controllers/companies_controller.rb
class CompaniesController < ApplicationController
  def index
    companies = Company.where(nil)
    params.slice(:name, :tax).each do |key, value|
      companies = companies.send(key, value) if value
    end
    render json: companies, status: 200
  end
end

Okay, but we should not satisfied enough. We can refactor our controllers!

4. Endgame - skinny controller

Let's make a module to store our code.

# app/models/concerns/filterable.rb
module Filterable
  extend ActiveSupport::Concern

  module ClassMethods
    def filter(filtering_params)
      results = self.where(nil)
      filtering_params.each do |key, value|
        results = results.send(key, value) if value
      end
      results
    end
  end
end

Now, we have to update our models and upgrade our controllers.

# app/models/user.rb
class User < ApplicationRecord
  include Filterable
  ...
end

# app/models/company.rb
class Company < ApplicationRecord
  include Filterable
  ...
end

# app/controllers/users_controller.rb
class UsersController < ApplicationController
  def index
    users = User.filter(filtering_(params))
    render json: users, status: 200
  end

  private

  def filtering_(params)
    params.slice(:role, :status, :public)
  end
end

# app/controllers/companies_controller.rb
class CompaniesController < ApplicationController
  def index
    companies = Company.filter(filtering_(params))
    render json: companies, status: 200
  end

  private

  def filtering_(params)
    params.slice(:name, :tax)
  end
end

5. The hidden cost

a. Remember to whitelist your params! In my example, I whitelist my params using filtering_(params) method in each controller. If you don't whitelist your params, imagine this:

params = { destroy: 1 }

User.filter(params)

b. In Ruby 2.6.0, method filter is added on enumerables that takes a block. You can check the code below. But I strongly suggest to change the method name, from filter to filter_by.

# This will throw error, because ruby use filter for enumerables.
def index
    @companies = Company.includes(:agency).order(Company.sortable(params[:sort]))
    @companies = @companies.filter(params.slice(:ferret, :geo, :status))
end

# This won't throw error, because we call filter for class Company
def index
    @companies = Company.filter(params.slice(:ferret, :geo, :status))
    @companies = @companies.includes(:agency).order(Company.sortable(params[:sort]))
end

6. Final thoughts

Is this really necessary? May be it is not if the parameters given is not much, or you have only 1 or 2 models. But, what if the parameters is more than 4 per models, and you have 10 models?

Btw, this code is not my work. I only rewrite this article + it's gist for my future use (in case the link is broken).

Ruby Metaprogramming: part 1

K Putra — Tue, 24 Dec 2019 09:41:33 +0000

Metaprogramming is, write code that writes code. There are many articles out there which explain the fundamental of ruby metaprogramming, but I just want to cover how to use metaprogramming.

Do I need to learn metaprogramming? Yes, you do! Ruby Metaprogramming is a powerful tool to refactor your code (besides design pattern).

Let's start our first journey!

Table of Content
1. Open Classes
2. Dynamic method: write code that define method
3. Calling method using string
4. Calling class using string
5. Method missing

1. Open Classes

Let say you have this method:

def remove_leading_zeros(str)
  str.sub!(/^0*/, '')
end

str = '0090'
puts remove_leading_zeros(str)
# => "90"

You can just put the method inside String class

class String
  def remove_leading_zeros
    sub!(/^0*/, '')
  end
end

str = '0090'
puts str.remove_leading_zeros
# => "90"

arr = ('0090'..'0100').to_a
arr.map! { |a| a.remove_leading_zeros }
puts arr.join(' ')
# => "90 91 92 93 94 95 96 97 98 99 100"

Be careful of Monkeypatching: you redefine existing method in a class.

str = 'Ruby Language'

# Normal
puts str.upcase
# => "RUBY LANGUAGE"

# Monkeypatching
class String
  def upcase
    downcase
  end
end
puts str.upcase
# => "ruby language"

This is not ilegal, just be careful, because may be you will get un-notice bugs in the future. Always check for existing method before adding new method to a class.

2. Dynamic method: write code that define method

We know how to define method in ruby. But let's write code that define a method.
Let's say we have this kind of class:

class WhereAmI
  def in_indonesia
    puts 'I am in Indonesia'
  end

  def in_america
    puts 'I am in America'
  end
end

i_am = WhereAmI.new
i_am.in_indonesia
i_am.in_america
# => "I am in Indonesia"
# => "I am in America"

Pretty repetitive if you have 5 countries, right. Let's add metaprogramming:

class WhereAmI
  ['indonesia', 'america', 'england', 'germany', 'japan'].each do |method|
    define_method "in_#{method}" do
      puts "I am in #{method.capitalize}"
    end
  end
end
i_am = WhereAmI.new
i_am.in_indonesia
i_am.in_america
i_am.in_england
# => "I am in Indonesia"
# => "I am in America"
# => "I am in England"

You can pass arguments, options, and blocks too!

Well, these codes will explain better:

class Bar
  define_method(:foo) do |arg=nil|
    puts arg
  end
end

Bar.new.foo
# => nil

Bar.new.foo("baz")
# => "baz"

class Bar
  define_method(:foo) do |arg1, arg2, *args|
    puts arg1 + ' ' + arg2 + ' ' + args
  end
end

Bar.new.foo
# => wrong number of arguments

Bar.new.foo("one", "two", "three", "four", "five")
# => "one two ["three", "four", "five"]

class Bar
  define_method(:foo) do |arg, *args, **options, &block|
    puts arg
    puts args
    puts options
    puts 'Hey you use block!' if block_given?
  end
end

Bar.new.foo("one", "two", 3, 4, this_is: 'awesome', yes: 'it is') do
  'foobar'
end
# => "one"
# => ["two", 3, 4]
# => {:this_is => "awesome", :yes => "it is"}
# => "Hey you use block!"

3. Calling method using string

Okay, but what if I want to call all the method in previous class (#2) in one line? We call method using string (because it is String, you can use string interpolation)

i_am = WhereAmI.new
countries = ['indonesia', 'america', 'england', 'germany', 'japan']
countries.each { |country| i_am.send("in_#{country}") }
# => "I am in Indonesia"
# => "I am in America"
# => "I am in England"
# => "I am in Germany"
# => "I am in Japan"

The point is, use send(). To make it clear:

str = 'Ruby Languange'

# Normal calling method
str.upcase
# => "RUBY LANGUAGE"

# Call method using string
str.send('upcase')
# => "RUBY LANGUAGE"

There are send() and public_send(). The code below will explain what is the difference. It'll also explain how to use argument.

class MyClass
  private def priv_method(var)
    puts "Private #{var}"
  end
end

# This will throw error, because we call for private method
MyClass.new.priv_method('Ruby')
# => NoMethodError

# This won't throw error
MyClass.new.send('priv_method', 'Ruby')
# => "Private Ruby"

# This will throw error, because public_send calls public method only
MyClass.new.public_send('priv_method', 'Ruby')
# => NoMethodError

4. Calling class using string

What if I want to create a class instance using string? Yes you can!(Note: because it is String, you can use string interpolation)

There is Rails's way, and there is Ruby's way. Let's use WhereAmI class (#2) again, and I add new class inside a module.

module Foo
  class Bar
  end
end
# Normal
i_am = WhereAmI.new
foo  = Foo::Bar.new

# Rails's way, you can only use this in Ruby on Rails
i_am = 'WhereAmI'.constantize.new
foo  = 'Foo::Bar'.constantize.new


# Ruby's way
i_am = Object.const_get('WhereAmI').new
foo  = ('Foo::Bar'.split('::').inject(Object) {|o,c| o.const_get c}).new

5. Method missing

Let's back to our WhereAmI class (#2). What if we want to call undefined method?

i_am = WhereAmI.new
i_am.in_mars
# => NoMethodError

Yes, we get NoMethodError. WhereAmI does not know how to handle a method that it does not have! What if I don't want to get NoMethodError? So let's add method_missing to our class:

class WhereAmI
  # Code in #2 is copied to here

  def method_missing(method, *args, &block)
    puts "You called method #{method} using argument #{args.join(', ')}"
    puts "--You also using block" if block_given?
  end
end

i_am = WhereAmI.new
i_am.in_mars
i_am.in_mars('with', 'elon musk')
i_am.in_mars('with', 'ariel tatum') { "foobar" }

# => "You called in_mars using argument "
# => "You called in_mars using argument with, elon musk"
# => "You called in_mars using argument with, ariel tatum"
# => "--You also using block"

Now, probably you don't really know what to do with this function. I'll give you one. Let's upgrade our WhereAmI class in #2.

Before, we can only call 5 countries as given in the arrays. What if I want to able to call all countries and all planet and also all known-stars in the universe? Let's combine define_method and method_missing!

class WhereAmI
  def method_missing(method, *args, &block)
    return super method, *args, &block unless method.to_s =~ /^in_\w+/
    self.class.send(:define_method, method) do
      puts "I am in " + method.to_s.gsub(/^in_/, '').capitalize
    end
    self.send(method, *args, &block)
  end
end

i_am = WhereAmI.new
i_am.in_indonesia
i_am.in_mars
i_am.in_betelgeuse
# => "I am in Indonesia"
# => "I am in Mars"
# => "I am in Betelgeuse"

Still don't know when to use method_missing? Easy, you will figure it out soon.

That's all for Part 1. I still don't know what to write in other Parts. But, see you in other Parts!