DEV Community: Loran Kloeze

Revived extension to autoformat with Rubocop in VSCode

Loran Kloeze — Fri, 29 Jul 2022 14:24:00 +0000

There is a nice VSCode extension called vscode-ruby-rubocop that you can use to automatically format a Ruby file using Format document (Shift+Alt+F). Unfortunately, that extension seems to be unmaintained for over 6 months now... 😢

But there is some good news. 🎉🎉 I have forked the repository and fixed a few things:

Changed the Rubocop argument --auto-correct to --autocorrect because --auto-correct is deprecated.
Added configuration option ruby.rubocop.useServer. When you enable this option, Rubocop is started with --server which dramatically increases formatting speed. Keep in mind, this option only works with Rubocop >= 1.31.

You can find the revived extension at GitHub and in the VSCode marketplace.

How to create middleware in your Rails application

Loran Kloeze — Wed, 27 Jul 2022 15:37:10 +0000

Creating your own middleware in a Rails application is not something a lot of developers do or have to do. But when you want to add custom headers, authentication or some kind of rate limiting, you should definitely consider using middleware. I want to show you how you can implement a small rate limiter using your own middleware.

Middleware in Rails

Middleware in Rails is actually middleware in Rack. Rack connects a web framework (e.g. Rails) with a web server (e.g. Puma). It is responsible for wrapping http data to present it in a single method call where you can work with requests and responses. That single method call is where you put the middleware logic. You can consider middleware to be the entry and exit of you Rails application.

Each piece of middleware receives the current environment from the previous middleware (if any) and reads and/or changes this environment to pass it on to the next middleware (again, if any).

There is actually already a lot of default middleware in a vanilla Rails application created with rails new. You can list that middleware by running ./bin/rails middleware. For a new Rails 7 application it looks like this:

use ActionDispatch::HostAuthorization
use Rack::Sendfile
use ActionDispatch::Static
use ActionDispatch::Executor
use ActionDispatch::ServerTiming
use ActiveSupport::Cache::Strategy::LocalCache::Middleware
use Rack::Runtime
use Rack::MethodOverride
use ActionDispatch::RequestId
use ActionDispatch::RemoteIp
use Sprockets::Rails::QuietAssets
use Rails::Rack::Logger
use ActionDispatch::ShowExceptions
use Sentry::Rails::CaptureExceptions
use WebConsole::Middleware
use ActionDispatch::DebugExceptions
use Sentry::Rails::RescuedExceptionInterceptor
use ActionDispatch::ActionableExceptions
use ActionDispatch::Reloader
use ActionDispatch::Callbacks
use ActiveRecord::Migration::CheckPending
use ActionDispatch::Cookies
use ActionDispatch::Session::CookieStore
use ActionDispatch::Flash
use ActionDispatch::ContentSecurityPolicy::Middleware
use ActionDispatch::PermissionsPolicy::Middleware
use Rack::Head
use Rack::ConditionalGet
use Rack::ETag
use Rack::TempfileReaper
use ActionDispatch::Static
run MyApp::Application.routes

Let's look at small piece of default middleware called ActionDispatch::RequestId (source). It already contains a good explaination of the inner workings in the comments of that file.

# https://github.com/rails/rails/blob/81c5c9971abe7a42a53ddbfede2683081a67e9d1/actionpack/lib/action_dispatch/middleware/request_id.rb

require "securerandom"
require "active_support/core_ext/string/access"

module ActionDispatch
  # Makes a unique request id available to the +action_dispatch.request_id+ env variable (which is then accessible
  # through ActionDispatch::Request#request_id or the alias ActionDispatch::Request#uuid) and sends
  # the same id to the client via the X-Request-Id header.
  #
  # The unique request id is either based on the X-Request-Id header in the request, which would typically be generated
  # by a firewall, load balancer, or the web server, or, if this header is not available, a random uuid. If the
  # header is accepted from the outside world, we sanitize it to a max of 255 chars and alphanumeric and dashes only.
  #
  # The unique request id can be used to trace a request end-to-end and would typically end up being part of log files
  # from multiple pieces of the stack.
  class RequestId
    def initialize(app, header:)
      @app = app
      @header = header
    end

    def call(env)
      req = ActionDispatch::Request.new env
      req.request_id = make_request_id(req.headers[@header])
      @app.call(env).tap { |_status, headers, _body| headers[@header] = req.request_id }
    end

    private
      def make_request_id(request_id)
        if request_id.presence
          request_id.gsub(/[^\w\-@]/, "").first(255)
        else
          internal_request_id
        end
      end

      def internal_request_id
        SecureRandom.uuid
      end
  end
end

The methods initialize and call are the key methods. The initialize method is called only once at booting the app and can be used to configure the middleware and set up data that is shared with all requests. The most important part is the call method definition. It is called at every request a client makes to a Rails app. The call method receives the current Rack environment as Ruby hash and should return a three element array with the status code, response headers and optionally a body. That array is generated by running @app.call(env). Of the two methods, you'll probably work most of the time with call.

In the RequestId middleware above, the environment (env) is changed by adding a header with a request id. That id is determined in the private methods of the middleware class. The header name is taken from the header keyword argument in initalize. This makes sense: the name of the header doesn't change during the app lifetime but the contents of the header do change each request. That's why the header name is set in initalize (runs once at boot time) and the value is set in call (runs each new request).

Our own middleware

Now that we understand a little more about middleware, let's create our own. We will create middleware that implements a simple rate limiter using Kredis. Kredis is a nice wrapper around Redis. Other than that, I assume you have some basic knowledge of Rails.

Basic setup

Make sure you have a working Redis server available. Run redis-cli in your shell to check if Redis is running. Something like 127.0.0.1:6379> should pop up.

Start by creating a new Rails app and add the Kredis gem.

$ rails new myratelimiter
$ cd myratelimiter
$ ./bin/bundle add kredis
$ ./bin/rails kredis:install

Check config/redis/shared.yml to make sure the configuration matches your Redis server. When you have a default Redis setup you probably don't have to change anything.

We need something to request from our Rails application so create a controller and model with ./bin/rails g scaffold User name email and run .bin/rails db:migrate.

Create five users by running 5.times {|i| User.create(name: "Name #{i}", email: "user#{i}@example.com")} in the Rails console.

Start the Rails server and run curl http://localhost:3000/users.json. You should see some json containing the five users.

Rate limiter

We have a basic api Rails app in place that shows users through /users.json. The problem is that consumers of our api can keep on sending requests so it's time to implement rate limiting.

The requirements of our rate limiter are simple for the sake of simplicity:

the consumer of the api cannot send more than 50 requests per 5 minutes
the consumer receives an http status 429 - Too many requests when the limit is reached
the server always responds with 3 headers:
- Rate-Limit-Reached: a boolean which tells the consumer if the rate limit is reached
- Rate-Limit-Requests-Left: a number which tells how many requests are left in the current time window
- Rate-Limit-Requests-Reset: a datetime which tells when the rate limiter will reset

Let's start with a few tests for the first two requirements:

# test/controllers/users_controller_test.rb
require 'test_helper'

class UsersControllerTest < ActionDispatch::IntegrationTest
  setup do
    Kredis.redis.flushall
    @user = users(:one)
  end

  test 'should not rate limit normal use' do
    49.times do
      get users_url
      assert_response :success
    end
  end

  test 'should rate limit abnormal use' do
    50.times do
      get users_url
      assert_response :success
    end

    get users_url
    assert_response :too_many_requests
  end
end

If you run the test suite you'll see that the second tests fails. Of course that's correct since we haven't implemented the rate limiter yet so let's go ahead with that.

Create lib/middleware/rate_limiter.rb with this class:

# lib/middleware/rate_limiter.rb - first draft
class RateLimiter
  def initialize(app)
    @app = app
  end

  def call(env)
    @app.call(env)
  end
end

Add this middleware to your app in config/application.rb:

...
require_relative '../lib/middleware/rate_limiter'
...
module Myratelimiter
  class Application < Rails::Application
    ...
    config.middleware.insert_before 0, RateLimiter
  end
end

That's it. You now have your own middleware in place. It doesn't do a lot at this point and running the test suite still yields the same failing tests.
Time to implement the first two requirements. I'm going to show you the complete middleware class for those requirements and go through it step by step.

# lib/middleware/rate_limiter.rb - second draft
class RateLimiter
  MAX_PER_WINDOW = 50
  WINDOW_SIZE = 1.minute

  def initialize(app)
    @app = app
  end

  def call(env)
    @req = ActionDispatch::Request.new(env)
    rate_limited? ? response_limit_reached : response_normal
  end

  private

  def rate_limited?
    request_counter.value >= MAX_PER_WINDOW
  end

  def kredis_key
    "rate_limiter:#{remote_ip}"
  end

  def request_counter
    # Only set the expires_in when the key is created
    # for the first time. Otherwise expires_in is
    # reset each time the key is accessed.

    if key_exists?
      Kredis.counter(kredis_key)
    else
      Kredis.counter(kredis_key, expires_in: WINDOW_SIZE)
    end
  end

  def key_exists?
    Kredis.redis.exists(kredis_key).positive?
  end

  def remote_ip
    # No need to re-invent logic to calculate the remote IP. It's already
    # available to use in ActionDispatch.

    ActionDispatch::RemoteIp::GetIp.new(@req, false, []).calculate_ip
  end

  def response_normal
    # Give back a normal response after incrementing the counter

    request_counter.increment
    @app.call(@req.env)
  end

  def response_limit_reached
    # We can also just put 429 here but this is more explicit.
    status_code = Rack::Utils::SYMBOL_TO_STATUS_CODE[:too_many_requests]

    [status_code, {}, []]
  end
end

If you run the test suite now, it will pass. If you want to test it yourself, set MAX_PER_WINDOW to 3 and run curl -v http://localhost:3000/users.json four times. The first three times the server responds as usual. The fourth time it returns an empty body with status code 429 - Too many requests.

What's happening in our newly created middleware? On each request the method call is called where an ActionDispatch::Request class is instantiated. With that object we can work easier with the data in env. Then we check if the rate limit is reached and return an appropriate response.

In rate_limited? the method request_counter is called which brings us to the part where Kredis is used. We use Kredis to initialize a counter in Redis. Kredis 'instantiates' the value from Redis. In other words, when you call Kredis.counter("mykey") we have an object that points to a Redis value under mykey. On that object we can call #increment which increments the current value in Redis. As you can see we check if the Redis key exists so that we can decide to use the call with expires_in. Each time you call #counter with expires_in, the expire timer resets. We don't want that because then the key will never expire. Checkout the Kredis docs for more information about Kredis.

We need some way to identify the consumer (or visitor if you will). A session is not an option since we don't have access to it yet and it's easy to circumvent by simple deleting cookies client side. Maybe an IP address? It's not really easy to spoof and you have to go through greater lengths to change the IP each n requests. On the other hand, a public IP is often shared amongst many users behind some router. For now, we go with an IP address but in the real world you'll probably identify a user with a token. In the method kredis_key the key is composed from remote_ip. In remote_ip we use logic that Rails already provides from ActionDispatch.

If the rate limit is not reached, response_normal is called where the request counter in Redis is incremented and the control is given back to Rails with @app.call(@req.env). But if the rate limit is reached, response_limit_reached is called. In that method we return a three element array containing the http status code, extra headers and a body. The last two are empty since the status code 'too many requests' is enough to inform the consumer the rate limit is reached.

You now have a working rate limiter as middleware. It might be too basic for a real world application but it does the job.

Extra headers

We still need to fulfill the third requirement: three headers containing information about the rate limiter.

Let's start with the first header called Rate-Limit-Reached. First we write a new test:

class UsersControllerTest < ActionDispatch::IntegrationTest
  ...
  test 'should return correct Rate-Limit-Reached header' do
    get users_url

    # You may be tempted to use `refute` but we want to make sure it really is false
    # instead of falsey like nil.
    assert_equal false, response.headers['Rate-Limit-Reached']

    50.times { get users_url } # Trigger rate limiter
    assert_equal true, response.headers['Rate-Limit-Reached']
  end
  ...
end

Run the test suite and it should fail. Add/replace the following methods:

class RateLimiter
  ...
  def rate_limit_headers
    {
      'Rate-Limit-Reached' => rate_limited?
    }
  end

  def response_normal
    # Give back a normal response after incrementing the counter

    request_counter.increment
    @app.call(@req.env).tap do |_status, headers, _body|
      rate_limit_headers.each { |key, value| headers[key] = value }
    end
  end

  def response_limit_reached
    # We can also just put 429 here but this is more explicit.
    status_code = Rack::Utils::SYMBOL_TO_STATUS_CODE[:too_many_requests]

    [status_code, rate_limit_headers, []]
  end
  ...
end

Now the test suite passes and we have implemented the first header requirement.

On to the next header called Rate-Limit-Requests-Left. Append the following to the test suite:

class UsersControllerTest < ActionDispatch::IntegrationTest
  ...
  test 'should return correct Rate-Limit-Left header' do
    get users_url

    assert_equal 49, response.headers['Rate-Limit-Left']

    50.times { get users_url } # Trigger rate limiter
    assert_equal 0, response.headers['Rate-Limit-Left']
  end
  ...
end

Run the test suite again to make sure it fails and add/replace the following methods:

class RateLimiter
  ...
  def rate_limit_headers
    {
      'Rate-Limit-Reached' => rate_limited?,
      'Rate-Limit-Left' => requests_left
    }
  end

  def requests_left
    MAX_PER_WINDOW - request_counter.value
  end
  ...
end

This will yield a passing test suite. We now have implemented the first two required headers.

Let's finish up with the third header called Rate-Limit-Requests-Reset. Again we start with a new test:

class UsersControllerTest < ActionDispatch::IntegrationTest
  ...
  test 'should return correct Rate-Limit-Reset header' do
    get users_url

    seconds_left = Time.parse(response.headers['Rate-Limit-Reset']) - Time.now.utc
    assert seconds_left.between?(1.minute - 5.seconds, 1.minute)

    50.times { get users_url } # Trigger rate limiter
    seconds_left = Time.parse(response.headers['Rate-Limit-Reset']) - Time.now.utc
    assert seconds_left.between?(1.minute - 5.seconds, 1.minute)
  end
  ...
end

And the logic to add/replace in the middleware:

class RateLimiter
  ...
  def rate_limit_headers
    {
      'Rate-Limit-Reached' => rate_limited?,
      'Rate-Limit-Left' => requests_left,
      'Rate-Limit-Reset' => reset_time
    }
  end

  def reset_time
    ttl = Kredis.redis.ttl(kredis_key) # Ask Redis how long the key has left to live
    (ttl >= 0 ? ttl.seconds.from_now : Time.zone.now).iso8601 # Create a datetime from TTL
  end
  ...
end

Run the test suite again, all tests should pass.

Wrapping it up

And now you have created some middleware to implement a rate limiter with a test suite. You can find the source of the above Rails application here.

The main purpose of this article was to show you how to develop middleware and not how to develop the world's most sophisticated rate limiter. But I hope it gave you some inspiration!

Notes

The test suite is kind of slow because we run the n.times { get users_url } blocks a lot... I'm going to let you figure out a refactor of the test suite to increase the performance. A small pointer: think about running two response cycles (normal and abnormal) and caching the results so each test can use the cached responses.
There is one important case not tested in the suite and that is the moment the request limit has been reset. Normally you can use travel but the TTL is set in Redis and not in the Rails app. You can refactor the middleware to accept a configuration value for the window size and set it to a few seconds so that it can be tested.

Alternatives for Ruby's OpenStruct

Loran Kloeze — Sun, 10 Jul 2022 16:47:45 +0000

This week I was updating Rubocop in an older Rails application. And that means Rubocop gets to throw new rules in my face. One of those new rules was about me using OpenStruct which is apparently now considered an anti-pattern. Time for an alternative for OpenStruct.

Using OpenStruct is a quick way to create some kind of hash where the keys are accessible through methods. But OpenStruct has a huge drawback: there are concerns regarding performance according to this and this source. Rubocop says:

Instantiation of an OpenStruct invalidates Ruby global method cache as it causes dynamic method definition during program runtime. This could have an effect on performance, especially in case of single-threaded applications with multiple OpenStruct instantiations.

So if using OpenStruct is most of the time an anti-pattern, what are some good alternatives for OpenStruct and how do they benchmark against it in Ruby 3.1.2?

First of all: there are cases where using OpenStruct is valid. Creating some kind of Ruby script that runs once? Just use OpenStruct at your own discretion, run it and throw the script away. Using OpenStruct in a Rails application where it is used very often? Don't do that and check out the alternatives below. My point is: don't just let some random guy like me on the internet tell you what you should do in each and every case.

OpenStruct

Let's start with a quick refresher on OpenStruct.

require 'ostruct'

car = OpenStruct.new
car.wheels = 4
car.mileage = 13_337

puts "My car has {car.wheels} wheels and" 
puts "a mileage of #{car.mileage} miles"
# => My car has 4 wheels and 
# => a mileage of 13337 miles

Wonderful. The syntax is quite clean, dynamic and easy to work with. Now let's do it 250,000 times.

require 'benchmark'
require 'ostruct'

cars = []
time = Benchmark.measure do
  250_000.times do 
    car = OpenStruct.new
    car.wheel = 4
    car.mileage = rand((1..100_000))
    cars << car
  end
end

puts time
# => 6.662331   0.400225   7.062556 (  7.062667)

That took 7.06 seconds. Let's continue with some alternatives.

Class

The good old class. Always there in Ruby when we need it. When we use a customer class as an alternative for the above OpenStruct it looks like this:

class Car
  attr_accessor :wheels, :mileage
end

cars = []
time = Benchmark.measure do
  250_000.times do
    car = Car.new
    car.wheels = 4
    car.mileage = rand((1..100_000))
    cars << car
  end
end

puts time
# => 0.091187   0.000000   0.091187 (  0.101195)

This time it took 0.10 seconds. That's 70 times faster than using OpenStruct.

Using a class is a good alternative. It requires some more code but that's fine. And in this case, using a class conveys nice what this piece of your business domain looks like.

Struct

Of course we should look at Struct too:

car_struct = Struct.new(:wheels, :mileage)
cars = []

time = Benchmark.measure do
  250_000.times do
    cars << car_struct.new(4, rand((1..100_000)))
  end
end

puts time
# => 0.085577   0.000649   0.086226 (  0.086235)

With Struct it took 0.09 seconds, similar to using a class.

Hash

The last alternative is a plain hash:

cars = []
time = Benchmark.measure do
  250_000.times do
    cars << { wheels: 4, mileage: rand((1..100_000)) }
  end
end

puts time
# => 0.085295   0.000000   0.085295 (  0.085301)

And hash is also similar to using a class or Struct with a run time of 0.09 seconds

Benchmark

To summarize:

OpenStruct   - 6.66 0.40  7.06 (7.06) - baseline
Custom class - 0.09 0.00  0.09 (0.10) - ~70 times faster
Struct       - 0.08 0.00  0.08 (0.08) - ~80 times faster
Hash         - 0.08 0.00  0.08 (0.08) - ~80 times faster

What should I choose?

Any of the above alternatives of OpenStruct is fine. If you want to squeeze out the last few percentages of performance, go with a Struct or hash. If you only use the data at a certain point in your code, a hash is also appropriate. But as soon as that data flows through multiple paths in your code, don't be afraid to upgrade the hash to a custom class or Struct . A class will convey better what your data model looks like and can eventually enforce it as well so it is more resilient. You cannot really make a wrong choice here and you can always upgrade the data structure at a later moment, Ruby is nice that way.

Creating lots of records in Rails with insert_all

Loran Kloeze — Fri, 01 Jul 2022 09:19:21 +0000

Imagine importing a lot of data in you Rails app. I'm talking about > 100,000 records. Is it a good idea to just run 100,000 x Address.create(data)? No, probably not. Let's find out what a better way is to bulk create data in your Rails database.

The app

We're creating an app that imports data from a csv file with customer data. The file contains 1,000 rows consisting of 6 fields. It looks something like this:

id,first_name,last_name,email,gender,ip_address
1,Lianne,Hosten,lhosten0@cnbc.com,Female,112.33.73.252
2,Aubrie,Dorin,adorin1@unesco.org,Female,254.88.120.47
...

The requirement is simple: we want each row in the csv to have a saved ActiveRecord counterpart called Customer.

Model

The model and associated migration:

# The model
class Customer < ApplicationRecord
end

# And its migration
class CreateCustomer < ActiveRecord::Migration[7.0]
  def change
    create_table :uploads do |t|
      t.string :first_name
      t.string :last_name
      t.string :email
      t.string :gender
      t.string :ip_address

      t.timestamps      
    end
  end
end

Inserting data

There are many ways to introduce the csv file in your app. You might download it from an url, read it from the filesystem or from a submitted form with a file input. Either way, you'll mostly end up with a csv file in a controller action.

Making it work

When you're developing software, most of the time you should first focus on getting the stuff to work before moving on to focus on performance. So let's create something that works.

require 'csv'
require 'benchmark'

class CustomersController < ApplicationController

  def create_bulk
    file = params[:csv_file] # Uploaded through a form

    time = Benchmark.measure do
      CSV.open(file.path, headers: true).each do |row|
        Customer.create(row)
      end
    end

    puts time
  end
end

Benchmark results: 3.884280 0.871490 4.755770 ( 13.748814)
Yeah, that took 13 seconds on my machine for creating only 1,000 customers. Imagine importing 250,000 ones...

Making it work fast

So, it's too slow. What are we going to do? Well, there's a method from ActiveRecord called insert_all.

The method insert_all receives an array of attributes per record and creates an SQL INSERT statement to insert all the records in one trip. For example:

customers = [
  {first_name: 'Lianne', last_name: 'Hosten', email: 'lhosten0@cnbc.com', gender: 'Female', ip_address: '112.33.73.252'},
  {first_name: 'Aubrie', last_name: 'Dorin', email: 'adorin1@unesco.org', gender: 'Female', ip_address: '254.88.120.47'},
  ...
]
Customer.insert_all(customers)

The resulting SQL statement:

Customer Bulk Insert (7.9ms)  INSERT INTO "customers"
("first_name","last_name","email","gender","ip_address","created_at","updated_at") 
VALUES 
('Lianne', 'Hosten', 'lhosten0@cnbc.com', 'Female', '112.33.73.252', STRFTIME('%Y-%m-%d %H:%M:%f', 'NOW'), STRFTIME('%Y-%m-%d %H:%M:%f', 'NOW')), 
('Aubrie', 'Dorin', 'adorin1@unesco.org', 'Female', '254.88.120.47', STRFTIME('%Y-%m-%d %H:%M:%f', 'NOW'), STRFTIME('%Y-%m-%d %H:%M:%f', 'NOW')),
...

Let's give this a try in our app.

...
class CustomersController < ApplicationController

  def create_bulk
    file = params[:csv_file] # Uploaded through a form

    time = Benchmark.measure do
      CSV.open(file.path, headers: true).each_slice(250) do |rows|
        Customer.insert_all(rows.map(&:to_h))
      end
    end

    puts time
  end
end

Benchmark results: 0.047267 0.035247 0.082514 ( 0.130676)

Hey, what is each_slice? Normally each enumerates an array and gives you each element to work on. But each_slice gives you back n elements to work on. Yep, it gives you back an array.

The csv contains 1,000 rows so the code above generates 1,000 / 250 = 4 SQL INSERT queries instead of 1,000 INSERT queries. Another thing that speeds up the import is that the Customer model is not instantiated which shaves off quite a few more cpu cycles.

It took 0.13 seconds instead of 13 seconds... 100 times faster.
Alright, why should I ever use #create again? Well, read on.

Should I use `insert_all` whenever possible?

Wow, 100 times faster, there must be a catch. Yes, there is and it's a big one. The huge drawback here is that callbacks and validations are not triggered.

I repeat: callbacks and validations are not triggered when using insert_all.

So make sure the data you use with insert_all is valid(ish). Check your callbacks to find out if there is anything that should run before or after your insert_all.

Important notes

Use insert_all only when callbacks and validations are okay to be ignored.
The benchmarks results that I compared are the so called elapsed real times. Those are nice to give you a human sense of the time elapsed but they are less ideal when you really want to compare the time the CPU was busy chewing on your code.