It Will Do vs It Will Scale

A lot of people wonder how to design for scale. Honestly, learning how to code for scale takes a lot of practice. In my opinion, it is easier to do once you have experienced some of those scaling pains. After you have seen the downfalls of slow code it can be easier to pick it out in the future.

In this post I want to share one example of the difference between "It will do" code vs "It will scale" code.

Finding the Offending Code

First and foremost, how did I find this unoptimized code? We had some jobs that were taking a long time to run and I decided to try and figure out why. I took some sample data and ran the job inline in a console with Rails.logger.level = 0. Here's a snippet of what I saw.

Role Load (0.6ms)  SELECT  `roles`.* FROM `roles` WHERE `roles`.`description` = 'read-only'  ORDER BY description ASC LIMIT 1
Role Load (0.6ms)  SELECT  `roles`.* FROM `roles` WHERE `roles`.`description` = 'basic'  ORDER BY description ASC LIMIT 1
Role Load (0.6ms)  SELECT  `roles`.* FROM `roles` WHERE `roles`.`description` = 'admin'  ORDER BY description ASC LIMIT 1
...
Role Load (0.6ms)  SELECT  `roles`.* FROM `roles` WHERE `roles`.`description` = 'read-only'  ORDER BY description ASC LIMIT 1
Role Load (0.6ms)  SELECT  `roles`.* FROM `roles` WHERE `roles`.`description` = 'basic'  ORDER BY description ASC LIMIT 1
Role Load (0.6ms)  SELECT  `roles`.* FROM `roles` WHERE `roles`.`description` = 'admin'  ORDER BY description ASC LIMIT 1
...
Role Load (0.6ms)  SELECT  `roles`.* FROM `roles` WHERE `roles`.`description` = 'read-only'  ORDER BY description ASC LIMIT 1
Role Load (0.6ms)  SELECT  `roles`.* FROM `roles` WHERE `roles`.`description` = 'basic'  ORDER BY description ASC LIMIT 1
Role Load (0.6ms)  SELECT  `roles`.* FROM `roles` WHERE `roles`.`description` = 'admin'  ORDER BY description ASC LIMIT 1

Over and over again we were looking up these same three roles. I knew to process the data we would check the user's role to figure out the permissions, but I had no clue why we were making 3 database requests to do it. I eventually traced the MySQL requests to this line of code: Role.system_roles.include?(user.role). From there I went to the Role model to see what that method was doing.

def self.system_roles
  [readonly_role, default_role, admin_role]
end

Each of those three roles were defined in their own methods.

def self.admin_role
  find_by(:description => ADMIN_ROLE)
end

def self.default_role
  find_by(:description => DEFAULT_ROLE)
end

def self.readonly_role
  find_by(:description => READONLY_ROLE)
end

Now it all made complete sense. Immediately, I scoffed at the code I had found. But then I stopped myself. Rather than going on a rant about how inefficient this code was, I took some time to understand why it was written this way in the first place.

Writing Code That Will Do

Turns out, the first set of individual role methods were written first. The individual find methods were used throughout the code to do simple one off permission checks for a single role.

A little later on, when we did our first pass at role based access control (RBAC) we implemented the system_roles method. We used this method to check access permissions for users in controller actions. Since we were only using it to do a single lookup in a controller action, we were not worried about 3 0.6ms MySQL queries. We wrote code that "would do".

Eventually, we expanded RBAC to include controlling access to data from our Elasticsearch cluster. Every time we made a request to Elasticsearch, we would check the role permissions of the user that made the request. When implementing this, I(yep, this one is on me!) found the system_roles method on the Role model and used it. At the time, I did not think about what would happen if we started calling that method millions of times. It was a simple method that we already had, so I figured I would use it. It worked great for over a year.

Writing Code For Scale

Today, we still use this code when we are getting data from Elasticsearch. Except now, we run this code hundreds of thousands of times in parallel background workers. Those 3 individual calls to MySQL were adding up and adding a not so insignificant amount of time to our jobs. To fix the problem, I updated the system_roles method to use a single where clause.

def self.system_roles
  where(:description => [ADMIN_ROLE, DEFAULT_ROLE, READONLY_ROLE])
end

Instead of making 3, 0.6ms MySQL requests, we now only have to make a single 0.3ms MySQL request. Not only does this save us time, but this also saves us additional load on our database.

Lesson 1 Learned - Try to write code that will scale from the start.

When you are writing code, it can be easy to only have the present use case in mind and write code that "will do". Try to look beyond the current use case when you are crafting code. Could this code be used differently in the future? If so, would it be performant? Right off the bat try to write the most performant code possible and your future self or coworkers will thank you for it.

Lesson 2 Learned - Don't blindly reuse code.

It is easy to grab a method or class that is already in place and use it when you are adding a new feature. Be weary of this! That class or method probably was written for an entirely different purpose than what you are planning to use it for. The author of the original code might not have intended it to be used the way you are going to use it. Make sure it fits all of your needs in terms of logic and performance.

The same concept applies to gems. There are a lot of great gems out there that were not designed to scale. Be aware of this. If you intend to use a gem frequently, take a little tour around the source code. Make sure it is optimized for your use case. More than once I have run into slow code when trying to implement plugins for background workers.

When you go to write your next line of code, ask yourself, is this code that "will do" or is this code that "will scale"?