DEV Community: Tiago Padrela Amaro

Rails url_for and strong parameters aren't fast

Tiago Padrela Amaro — Sun, 15 Jul 2018 00:09:34 +0000

This post was also published on my website.

In this article I'll describe a Rails performance caveat that I've recently faced on a project: Rails strong parameters and its routing are not performant and should be avoided if you are generating many URLs.

Scenario

The project needed a faceted filter which would generated thousands of URLs. Within a single faceted filter object, I had to generate an URL with the current URL merged with a given object's set of options.

A common approach to this issue would be using Rails' url_for and strong parameters. I ended up writing the following method for my view object:

def current_path(options = {})
  permitted_params = params.permit(DEFAULT_PERMITTED_PARAMS).merge(options)
  url_for(permitted_params)
end

Cool! That's concise, secure and it'll give me the current URL with all extra options I want to merge with it.

As you can imagine, there's was a caveat: this was slow.

Benchmarks

How slow? Pages were taking 500 milliseconds to 800 milliseconds to load, with a single user request and a heavy CPU load. I knew there was something wrong on that view.

After profiling that the current_path method was the one to blame, I ran a couple of benchmarks to test how the Rails way would compare to a pure Ruby stdlib approach. Using Ruby's stdlib URI, the method could be written as:

def current_path(options = {})
  permitted_params = request.parameters.slice(*DEFAULT_PERMITTED_PARAMS) # Avoid strong parameters 
  uri = URI(request.original_url) # Get the current URL through the request
  uri.query = permitted_params.merge(options).to_param
  uri.to_s
end

Here's the benchmark code:

# Inside the `app/controllers/benchmark_controller.rb` file

def show
  uri = URI(request.original_url)

  Benchmark.ips do |x|
    x.report("ActionDispatch::Routing::UrlFor#url_for with query string") do
      params.permit(:id)
      url_for(params.permit(:id)) # => http://localhost:3000/?id=42
    end

    x.report("Ruby stdlib URI") do
      uri.query = request.params.slice(:id).to_param
      uri.to_s # => http://localhost:3000/?id=42
    end

    x.compare!
  end
end

The results were quite interesting, given two different scenarios I often observed:

Users requests fit strong parameters, it didn't need to cleanup user input
Users requests didn't fit strong parameters, it needed to cleanup user input

Results with parameters that were permitted

Example request to URL http://lvh.me:3000/?id=42

Warming up --------------------------------------
ActionDispatch::Routing::UrlFor#url_for with query string
                       735.000  i/100ms
     Ruby stdlib URI     9.807k i/100ms
Calculating -------------------------------------
ActionDispatch::Routing::UrlFor#url_for with query string
                          7.449k (± 4.8%) i/s -     37.485k in   5.044535s
     Ruby stdlib URI    103.079k (± 4.9%) i/s -    519.771k in   5.054106s

Comparison:
     Ruby stdlib URI:   103079.2 i/s
ActionDispatch::Routing::UrlFor#url_for with query string:     7448.5 i/s - 13.84x  slower

Results with parameters that were not permitted

Example request to URL http://lvh.me:3000/?id=42&count=123

Warming up --------------------------------------
ActionDispatch::Routing::UrlFor#url_for with query string
                       309.000  i/100ms
     Ruby stdlib URI     9.325k i/100ms
Calculating -------------------------------------
ActionDispatch::Routing::UrlFor#url_for with query string
                          3.051k (± 8.4%) i/s -     15.450k in   5.101804s
     Ruby stdlib URI     81.424k (± 3.8%) i/s -    410.300k in   5.046415s

Comparison:
     Ruby stdlib URI:    81423.9 i/s
ActionDispatch::Routing::UrlFor#url_for with query string:     3050.7 i/s - 26.69x  slower

Analyzing results and conclusion

Given that url_for + strong parameters had a 3k iteration per second (on its worst case scenario) and that some rendered pages had more than 1500 anchor tags, that easily could consume 500 milliseconds of processing time, only to render faceted filtering!

Rails ease and magic comes with a performance cost. Its router methods are easy to use, but keep in mind you will need to move away from the Rails way if you want faster responses.

Postgres Locks and SKIP LOCKED - Updating Records in Parallel

Tiago Padrela Amaro — Thu, 11 Jan 2018 16:56:37 +0000

This article was originally posted on my personal website.

In this post, I'll show how I solved the following challenge: avoid database deadlocks while updating Postgres records
in parallel using Postgres' locking clause.

In an application I was developing, I had the following data structure: a product with many offers, where offers needed
to be ranked by their price grouped per product. With that logic, I thought on possible approaches to calculate ranking:

Calculate the ranking whenever the offer price was updated
Create background workers at the application level to handle the offer ranking update
Delegate ranking calculation to the database (using triggers)

With the KISS principle in mind, I tried the first implementation.

First Approach: Updating Ranking Through a Callback

This first approach was the naive one: creating an operation to update an offer's ranking whenever its price was updated.
This works fine if your operations are executed in serial, but if you update several records in parallel, you'll face some
issues with your database locking system.

In my case, I had the Puma server executing multiple calls to fetch prices in real time.

With multiple processes updating prices simultaneously, updating offers prices got me into a deadlock scenario,
where the same database row was being updated multiple times on different transactions. In this case, wrapping the ranking
update operation into a transaction would also not work, since multiple transactions would still fall in deadlocks.

Since this scenario looked like a "queue", a background queueing logic could solve the deadlock issue.

Second Approach: Background Workers

The second approach wasn't even implemented, since background workers should always be executed in parallel, as its goal is
to speed up or defer costly jobs.

Updating a single attribute isn't expensive, and it's serial execution nature is the real problem.

Third Approach: Database Triggers

Since deadlocks were the problem, deferring the issue to the database seemed to be the most logical approach.

Using triggers at the offers' table seemed the right approach, but the deadlocks would also occur on triggers if records
were not selected with the right locking mechanism. So, my research began on which mechanism would be the right one to
approach this issue.

Going through Postgres' documentation, the SKIP LOCKED locking clause
seemed to solve the issue with the following caveat:

With SKIP LOCKED, any selected rows that cannot be immediately locked are skipped. Skipping locked rows provides an
inconsistent view of the data, so this is not suitable for general purpose work, but can be used to avoid lock contention
with multiple consumers accessing a queue-like table.

This is an acceptable trade-off, since offer ranking is updated as a "queue-like table".

With this, I started porting my update operation to a Postgres trigger.

Postgres Implementation

With the "one to many" relationship between offers and products, and the product_id being the foreign key for the products
table, I wrote the following trigger:

CREATE FUNCTION calculate_offer_product_ranking() RETURNS trigger
    LANGUAGE plpgsql
    AS $$
BEGIN
  WITH offer_scope AS (
      SELECT *
      FROM offers
      WHERE product_id = NEW.product_id
      FOR UPDATE SKIP LOCKED
  ), ranked_offers AS (
      SELECT
        id,
        row_number()
        OVER (
          PARTITION BY product_id
          ORDER BY CAST(price AS FLOAT) ) AS ranking
      FROM offer_scope
      WHERE offer_scope.product_id = NEW.product_id
  )

  UPDATE offers
  SET product_price_ranking = ranked_offers.ranking
  FROM ranked_offers
  WHERE offers.id = ranked_offers.id;

  RETURN NEW;
END;
$$;

CREATE TRIGGER calculate_offer_product_ranking_trigger AFTER INSERT OR UPDATE OF price ON offers FOR EACH ROW EXECUTE PROCEDURE calculate_offer_product_ranking();

If you're not familiar with the WITH clause, nor Postgres window functions (PARTITION BY), I would highly recommend
you reading their docs, as these functions are quite useful when grouping data:

WITH queries docs

Window functions docs

Conclusion

This challenge reminded me that not all business rules will go to your application layer, as your database is a powerful
tool. Databases are incredible, reliable and optimized pieces of software, being able of more than "just storing data".

With this database trigger calculating offers' ranking, I could guarantee ranking without creating any workarounds on my
application layer to execute a task native to the database: atomicity and consistency.

What made you switch your main programming language?

Tiago Padrela Amaro — Fri, 22 Sep 2017 14:40:32 +0000

This post's goal is to share experience, telling a bit of your story and helping people who are in doubt.

What made you switch your main programming language?

Hi, I'm Tiago Amaro

Tiago Padrela Amaro — Sun, 12 Mar 2017 23:56:57 +0000

I have been coding for 10 years.

You can find me on Twitter as @tiagopadrela

I live in Rio de Janeiro.

I work for my own

I mostly program in these languages: Ruby and Javascript.

I am currently learning more about Marketing and self employment.

Nice to meet you.