DEV Community: Parulian Sinaga

Ruby cursor pagination (2)

Parulian Sinaga — Sat, 11 Jul 2020 06:52:30 +0000

Following this post, I would like to make it open source. So, people around the world can share with me for improvement and feature suggestion. Then, I will explain here my thought.

"If this project should become open source, then I need to make it become general for all case, configurable, and people can extend functionality in dynamic way", this is what I thinking at first. So, I list down for possibility code that make it not general.

Hardcoded primary_key

We hardcoded id as primary_key as default cursor when order not specified. This become a problem if this project will be used by project that have uuid or other column name as primary key. For this first problem, can be fixed by using active record attribute method primary_key to get primary_key column name

# from
...where('id > ?', cursor)

# to
...where(arel_table[primary_key].gt(cursor))

Limitation of order column

Since in our case we only have one order value for every page, we hardcoded order method also. Other people may have case when data need to order by name, quanty, etc for example.

# from
...where('(name, id) > (?, ?)', *cursor).order(:name, :id)

# to
# sorted_columns = { :name => :asc, :id => :asc }
# cursor = { :name => 'Shampoo', :id => 23 }
...where(
  Arel::Nodes::GreaterThan.new(
    Arel::Nodes::Grouping.new(cursor.keys.map{ |col| arel_table[col] }),
    Arel::Nodes::Grouping.new(
      cursor.values.map do |col|
        Arel::Nodes.build_quoted(col)
      end
    )
  )
).order(sorted_columns)

Above code will generate this sql in our case.

-- row values support PostgreSQL (since 8.4)
select * 
from products 
where (title, id) > ('Shampoo', 23)
order by name asc, id asc

-- equal to
select *
from products
where (title > 'Shampoo'
  or (title = 'Shampoo' and id > 23))
order by title asc, id asc

Encode and Decode Json

Usually cursor values encode and decode into Base64. In our case, we store cursor values as a Hash ex: next_cursor = {created_at:'2020-07-11T00:49:06Z'}, parse it to json and then encode to Base64. But, lately we found the issue when we collect data after '2020-07-11T00:49:06Z'.

[
  [
    # should not include in collection
    #<Product:0x00007fd53c33fc70 id: 5, name: "Shampo0", created_at: 2020-07-11 00:49:06 UTC, updated_at: 2020-07-11 00:49:06 UTC>,

    #<Product:0x00007fd53c33f950 id: 6, name: "Tooth brush", created_at: 2020-07-11 00:49:07 UTC, updated_at: 2020-07-11 00:49:06 UTC>,
    #<Product:0x00007fd53c33f590 id: 7, name: "Lifeboy Shoap", created_at: 2020-07-11 00:49:08 UTC, updated_at: 2020-07-11 00:49:06 UTC>,
    #<Product:0x00007fd53c33f3b0 id: 8, name: "Handsanitizer efe", created_at: 2020-07-11 00:49:09 UTC, updated_at: 2020-07-11 00:49:06 UTC>,
    #<Product:0x00007fd53c33eaa0 id: 9, name: "Pantene", created_at: 2020-07-11 00:49:10 UTC, updated_at: 2020-07-11 00:49:06 UTC>
  ],
  {next_cursor: ''}
]

We forgot that created_at store store micro/nano seconds ex: 2020-07-11T00:49:06.234632Z. So, we need to add micro/nano seconds into cursor values next_cursor = {created_at:'2020-07-11T00:49:06.234632Z'}.
After thingking further, I prefer store cursor values in string so that after encoded, it will be shorter than json ex: next_cursor = 'created_at:2020-07-11T00:49:06.234632Z'.

All of this code available here

Ruby Cursor Pagination (1)

Parulian Sinaga — Sat, 04 Jul 2020 13:44:47 +0000

Intro

As consumer expectations ramp up, API performance has never been more important than it is today. It’s a well-known statistic that 53% of web users will abandon a web page if it takes more than 3 seconds to load.
These expectations don’t necessarily line up with the technical requirements of an API. In the era of big data and analytics, APIs are dealing with larger amounts of data in their backend than ever before. To truly stand their ground in today’s digital economy, APIs must be optimized for peak efficiency. API pagination is a key strategy for making sure your APIs run smoothly and effectively.

Implementation

In this article, I am not going to talk about what is cursor pagination? cursor pagination vs offset pagination? why do I need them, what problem is solved by cursor pagination?

But I'm gonna sharing, how we can achieve cursor pagination that meet our product expectation. Our project currently only have mobile android as client and every index per page implement infinite scroll to access all of the items collection which is fit for cursor.

We use postgres as our main database in this project. So below the problem will be solve using active-record. Also, we just cover after cursor here but actually they have same solution for both after and before cursor.

def pager(after:, before: nil, limit: nil, sort: nil)
  # ... initialize parameter
  collection = if after.nil?
                 order(sorter).extending(ActiveRecordRelationMethods).limit(page_limit)
               else
                 custom_expression = create_custom_expression(after, sorted_columns)
                 where(custom_expression).order(sorter).extending(ActiveRecordRelationMethods).limit(page_limit)
               end

  # ...create_paginate_meta
end

In if condition you can tell me that it simple just return first page if cursor not set. Then if cursor defined, return the page after cursor value. For example, cursor = {id: 125} will return page after id 125. Take a notice, value of cursor usually encoded so client and server need to have contract to choose how this cursor should encode/decode. We encode/decode cursor value in base64, eyJpZCI6MTI1fQ== instead of {id: 125}.

Sorting by something other than id is possible by remembering the last value of the field being sorted on. For example, if you’re sorting by title, then the last value is the title of the last record in the page. If the sort value is not unique, then if we used it alone we would potentially be skipping records. For example, assume you have the following products

Id	Title
3	Hat
2	Pants
4	Pants
1	Shoes

Requesting a page size of two sorted by title would return product with ids 3 and 2. To request the next page, just querying by title > “Pants” would skip product 4 and start at product 1.

Id	Title
3	Hat
2	Pants
`4`	`Pants`
1	Shoes

Whatever the use case of the client that requests these records, it’s likely to have problems if records are sometimes skipped. The solution is to set a secondary sort column on a unique value, like id, and then remembering both the last value and last id. In that case the query for the second page would look like this:

SELECT *
FROM `products`
WHERE (`products`.`title` > "Pants" 
  OR (`products`.`title` = "Pants" AND `products`.`id` > 2))
ORDER BY `products`.`title` ASC, `products`.`id` ASC
LIMIT 2

Here is our implementation in ruby

def create_custom_expression(cursor_params, sorted_columns)
  decode_cursor_params = JSON.parse(Base64.strict_decode64(cursor_params))
  return arel_table[primary_key].gt(decode_cursor_params[primary_key]) if sorted_columns.blank?

  filter_ordered_columns = filter_with_ordered_columns(decode_cursor_params, sorted_columns)
  filter_primary_key = filter_with_primary_key(decode_cursor_params)

  filter_ordered_columns.or(filter_primary_key)
end

def filter_with_ordered_columns(decode_cursor_params, sorted_columns)
  result = self
  sorted_columns.each_with_index do |(column, type), index|
    result = if index.zero?
               result.arel_table[column].send(AREL_ORDER[type])
             else
               result.or(arel_table[column].send(AREL_ORDER[type]))
             end
  end

  result
end

def filter_with_primary_key(decode_cursor_params)
  result = self

  decode_cursor_params.each_with_index do |(column, value), index|
    result = if index.zero?
               result.arel_table[column].gt(value)
             else
               result.and(arel_table[column].eq(value))
             end
  end

  result
end

Summary

To ensure the query is performant as the number of records increases you need a database index set up on title and id. If an appropriate index is not set up then it could be even slower than using offset pagination. If you wanna try on your own, I already push the code on github.

Still, there is so much improvement we need on the system which is:

implement before cursor
meta url for first, last, next and previous page
collection size
error-cases
any other? :D ,please commented below if you have any concern or for any system requirement

Improve Rspec Pipeline Performance

Parulian Sinaga — Thu, 22 Aug 2019 09:57:52 +0000

At Mekari, we highly encourage that developers should write tests on every projects. Testing prevents bugs in production, forces better code, and encourages developer confidence when making changes to existing code bases. We use rspec, selenium with capybara as our standard unit tests and integration tests tools.

Rspec best practice

If you're new to ruby on rails and never used rspec before, here are some links which will give little guidance to start.

Working with Large Code Bases

In a large code base, testing becomes even more important. While some argue that testing slows development down, we argue that it increases development velocity over time, especially when refactoring existing code and eliminating technical debt.

Test Driven Development(TDD) encourages running your tests frequently and often. What you might find in a large code base when running the full test suite are it may takes 25+ minutes to run 2500+ tests. That will make you unproductive when you push changes and wait test suite to finish.

In this post, I will share how we accelerate our pipeline test. For my project, we use dedicated CI (Continuous Integration) server bitbucket for running the full test suite. Every minutes, developers make changes on a branch, commit, and push changes to bitbucket. Then, create PR (Pull Request) regarding ticket and bitbucket will pull down these changes and run the entire test suite.

Increase Hardware capability

When we want to achieve performance without doing much, our first option come to improve hardware specifications. You can easily see double the non parallel performance on your test suite by upgrading your CI server from an old or low spec machine into faster one.

Parallel Tests

Running specs across multiple CPU cores can further decrease the test time for your suite. We use parallel_tests gem for this. It uses a database per thread and can run on CI servers like bitbucket. Running test suite in parallel also make hardware upgrades even more effective, scaling out to machines with more cores. Single core performance is no longer your bottleneck.

When parallelizing your suite, beware of shared state between tests that lives outside of the database such as Elasticsearch. These interdependencies need to be addressed too in parallel.

This solution solve your issue about slow running your test suite, but they come at a price.

Profiling The Specs

When you have implemented the above solutions, but you still feel slow with your test suite. Then, what you should try to consider is your database strategy setup and test suite.

Database Cleaner configuration

If you write ruby or rails code that interacts with a database and write unit test, chances are you have heard of or used the database_cleaner gem. It's a terrific gem that abstracts away the various ORM APIs for getting the DB into a "blank slate" state. Some setup needed to use it in optimal mode. Here are some article that give you idea to setup your own:

Factory cascade

Let's say you have User model and each user has one Contact and one Location record. Your factory should be looking similar like this:

factory :user do
  contact
  location
end

Now, try to guess how many records are created in the database once you call create(:user)?

Each time you call create(:user), then 2 additional records are created. Yes, even with build. Now imagine that you are using this base factory across all tests. There is 2 ways to eliminate this issue.

Explicit associations The first thing that comes to mind is to remove all (or almost all) associations from our factories:

factory :user do
  # do not declare associations
  # contact
  # location
end

With this approach, you have to explicitly specify all required associations when using a factory

create(:user, contact: contact, location: location)

# But!!
create(:user) # => raises ActiveRecord::InvalidRecord

Traits Your base factory should always have only these attributes that are needed to pass validation. If you sometimes need associations then consider using traits

factory :user do
  name { "Zuned #{Faker::Name.unique.name}" }
  username { "#{ name.parameterize.underscore}_#{ SecureRandom.hex(3) }" }
  email { "#{ username + Random.rand(1000).to_s }@example.com"}

  trait :with_location do
    location
  end
end

and you can call your factory using

create :user, :with_location

let_it_be and Anyfixture

Our specs always have relations or required company and 2 or more employees (with association to user, job title, location, and job level). Imagine if we call create :company and so on for every specs. 1 spec need 1 company + 2 employee + 2 locations + 2 job_title + 2 job_level, it means 9 records only for initial data. Can we generate those 9 records only once for every specs?

Yes, here comes anyfixture. Built in fixture are different with anyfixture. Both represent different approach. Fixture declare a static state of the data that is loaded into a test database right away and usually persists between test runs. But with anyfixture, we able to hard reload record to get fresh copy of record via refind method.
Here are our implementation with (Anyfixture)[https://test-prof.evilmartians.io/#/any_fixture].

require 'test_prof/any_fixture/dsl'
require 'test_prof/ext/active_record_refind'

using TestProf::AnyFixture::DSL
using TestProf::Ext::ActiveRecordRefind

# Fixtures are kept to be outside db transaction such that it can be leveraged
# for specs. https://github.com/palkan/test-prof/blob/master/docs/any_fixture.md
RSpec.shared_context 'company with staffs' do
  before(:all) do
    # register to fixture
    fixture(:company) { FactoryBot.create(:company) }
    fixture(:location) { FactoryBot.create(:location, company: fixture(:company)) }
    fixture(:job_title_ceo) {
      FactoryBot.create(:job_title, :ceo, company: fixture(:company))
    }

    fixture(:job_title_hr) {
      FactoryBot.create(
        :job_title,
        title: 'HR Manager', parent: fixture(:job_title_ceo), company: fixture(:company)
      )
    }

    fixture(:job_title_staff) {
      FactoryBot.create(
        :job_title,
        title: 'Staff', parent: fixture(:job_title_ceo), company: fixture(:company)
      )
    }

    fixture(:org_general) {
      FactoryBot.create(
        :organisation_node,
        name: 'General', company_id: fixture(:company).id
      )
    }

    fixture(:org_hr) {
      FactoryBot.create(
        :organisation_node,
        name: 'HR', company_id: fixture(:company).id
      )
    }

    fixture(:job_level_ceo) {
      FactoryBot.create(:job_level, :CEO, company: fixture(:company))
    }

    fixture(:job_level_hr) {
      FactoryBot.create(:job_level, name: 'HR', company: fixture(:company))
    }

    fixture(:job_level_staff) {
      FactoryBot.create(:job_level, name: 'Staff', company: fixture(:company))
    }

    fixture(:owner) {
      FactoryBot.create(
        :employee, :owner,
        company: fixture(:company),
        location: fixture(:location),
        job_level: fixture(:job_level_ceo),
        organisation_node: fixture(:org_general),
        job_title: fixture(:job_title_ceo)
      )
    }

    fixture(:hr_admin) {
      FactoryBot.create(
        :employee, :hr_admin,
        company: fixture(:company),
        location: fixture(:location),
        job_level: fixture(:job_level_hr),
        organisation_node: fixture(:org_hr),
        job_title: fixture(:job_title_hr)
      )
    }

    fixture(:staff) {
      FactoryBot.create(
        :employee, :staff,
        company: fixture(:company),
        location: fixture(:location),
        job_level: fixture(:job_level_staff),
        organisation_node: fixture(:org_general),
        job_title: fixture(:job_title_staff)
      )
    }
  end

  # Ensure that every example has a fresh copy of a record
  %i[
    company location job_title_ceo job_title_hr org_general org_hr
    job_level_ceo job_level_hr owner hr_admin staff
  ].each do |object|
    let(object) { fixture(object).refind }
  end
end

Imagine this example too.

describe User do
  let!(:user) { FactoryGirl.create :user }

  it 'does something' do
    # test with user
  end

  it 'does something' do
    # test with user
  end

  it 'does something' do
    # test with user
  end
end

How many User records are created in the database? One record for each example so the answer is 3 records. To avoid it you can use (let_it_be)[https://test-prof.evilmartians.io/#/let_it_be]. This helper uses Rails transactional tests feature which means that the record is created only once at the beginning of the test and removed when the test is finished.

describe User do
  let_it_be(:user) { FactoryGirl.create :user }

  it 'does something' do
    # test with user
  end

  it 'does something' do
    # test with user
  end

  it 'does something' do
    # test with user
  end
end

Additional suggestion and resource

There are some condition where we doesn't need data to persists first into db like checking custom validation on model and unit test on view or mailer specs. Then if this happer, we can utilize build and build_stubbed create strategy.

let(:company) { FactoryBot.build_stubbed(:company) }

some usefull links:

(Test Profiling)[https://test-prof.evilmartians.io]
(Factory therapy for your ruby tests)[https://evilmartians.com/chronicles/testprof-2-factory-therapy-for-your-ruby-tests-rspec-minitest#bonus-anyfixture]
(build and build_stubbed)[https://thoughtbot.com/blog/use-factory-girls-build-stubbed-for-a-faster-test]

RSpec best practice

Parulian Sinaga — Tue, 20 Aug 2019 08:31:28 +0000

I use RSpec in my daily work. It’s really hard to overemphasize how helpful it is and how much easier becomes your life if you have good specs coverage. But its outstanding flexibility enables many ways to make your specs awful: horribly slow, over-bloated, even non-readable sometimes. I do not want to teach you BDD and RSpec here, but instead I will give you some ideas how to improve your specs quality and increase efficiency of your BDD workflow.

Here are some tips how we are at Mekari structure test code. These should help you to keep your tests well structured, easy to read, comprehend, and DRY.

1. describe

Be clear about what method you're describing. For instance, use the ruby documentation convention of . when referring to a class method's name and # when referring to an instance method's name.

### Bad example
describe 'the authenticate method for User' do
describe 'if the user is an admin' do

### Good example
describe '.authenticate' do
describe '#admin?' do

2. context

context starts either with "with" or "when", such "when status is pending"

describe '#status_badge' do
  context 'returns css class based on status' do
    context 'when status is pending' do
      let(:request_status) { 'pending' }

      it 'returns css for grey badge' do
        expect(subject.status_badge).to eql 'c-badge--grey'
      end
    end

    context 'when status is approved' do
      let(:request_status) { 'approved' }

      it 'returns css for green badge' do
        expect(subject.status_badge).to eql 'c-badge--green'
      end
    end
  end
end

3. it

it describes a test case (one expectation) and specify only one behavior. Multiple expectations in the same example are signal that you may be specifying multiple behaviors. By specify only have one expectation, helps you on finding possible errors, going directly to the failing test, and to make your code readable.

Anyway, in tests that are not isolated (e.g. ones that integrate with a DB, an external webservice, or end-to-end-tests), you take a massive performance hit to do the same setup over and over again, just to set a different expectation in each test. In these sorts of slower tests, I think it's fine to specify more than one isolated behavior.

it { is_expected.to belong_to(:job_title).class_name('JobTitle').optional }
it { is_expected.to belong_to(:template).class_name('Approval::Template') }

# or

it 'has relations' do
  is_expected.to belong_to(:job_title).class_name('JobTitle').optional
  is_expected.to belong_to(:template).class_name('Approval::Template')
end

4. subject

subject makes way clearer about what you're actually testing and helps you stay DRY in your tests

describe Book do
  describe '#valid_isbn?' do
    subject { Book.new(isbn: isbn).valid_isbn? }

    context 'with a valid ISBN number' do
      let(:isbn) { 'valid' }

      # ...
    end

    context 'with an invalid ISBN number' do
      let(:isbn) { 'invalid' }

      # ...
    end
  end  
end

There is more what you can do with RSpec and you can read them on betterspecs

5. Mocking

mocking is interesting and usually we're doing mocking when the scenario which we want to test require another service.

You may mock just everything so your spec will never hit the database or another service. But, this is something wrong. When your model code changed or the initiliaze method of service you are call changed, your code will break without get failing specs before merge to production.

By mocking objects in advance, you can allow yourself to focus on the thing that you’re working on at the moment. Let's say that you are working on a new part of the system, and you realize that the code you're currently describing and implementing will require two new collaborating objects. Using mocks, you can define their interfaces as you write a spec for the code you're currently working on.

That way, you maintain a clean environment by having all your tests pass, before moving on to implement the collaborating objects. Without mocks, you'd be required to immediately jump to writing the implementation for the collaborating objects, before having your tests pass. This can be distracting and may lead to poor code design decisions. Mocking helps us by reducing the number of things we need to keep in our head at a given moment.

Since you're able to mock, remember so as not to mock everything to green your specs or to make your spec never hit the database. This is something wrong. When your object code changed or the initialize method of object you're call changed, your code will break without get failing specs before merge to production.

class OvertimeRequest do
  #...
  def allow?
    current_company.active_subscribe?
  end
end

RSpec.describe OvertimeRequest do
  before do
    allow_any_instance_of(Company).to receive(:active_subscribe?).and_return true
  end

  it ... do
  #..
end

6. Custom matchers

Rspec has many useful matchers, we already used be true, be false, and so on. Sometimes, when expecting given values, we repeat the same code over and over again. Let's consider the following example

RSpec.feature 'some feature', type: :feature do
  it 'saves data' do
    #..

    expect(page).to have_css('.c-alert--success', text: 'Saved successfully', visible: :all)
  end

  it 'returns errors' do
    #..

    expect(page).to have_css('.c-alert--failed', text: 'Failed to save', visible: :all)
  end
end

# -- another 100 example to check flash message

wouldn’t be easier if you could write just this

expect(page).to have_flash_message("Saved successfully", type: :success)

If you've consistent logic across the whole app, it's better to create such custom matcher. To create such matcher you've to create matchers.rb file (name it as you want) in spec/support directory and put there your matcher definition:

RSpec::Matchers.define :have_flash_message do |message, opts = {}|
  match do |container|
    css_class = opts[:type].present? ? ".c-alert--#{opts[:type]}" : ".c-alert"

    expect(page).to have_css(
      css_class,
      text: message,
      visible: :all
    )
  end
end

The last step is to require your matcher by including require 'support/matchers' in the spec_helper.rb file.

resources

Some links that can help