Ruby class pattern to work with API requests with built-in async approach

#ruby #concurrency #oop #async

Intro

I have been working with dozens of different Web APIs (mostly JSON) as a Ruby developer. Some of them can provide an official Ruby gem that you can install and use in order to interact with the API in your Ruby code. Others don't have anything like It at all and you need to implement the interaction with the API yourself. In this case, there are a lot of helpful API ruby gems that are easier to use than the default Ruby URI module https://github.com/ruby/uri. Let's list a few of them:

Faraday - https://github.com/lostisland/faraday. Personally, I use It, just because historically it was linked to every project I worked with. In addition, you can install the Faraday Curl gem https://github.com/mauricio/faraday_curl to get nice Curl requests in logs.
HTTPParty - https://github.com/jnunemaker/httparty. Heard a lot about It, but never used it.
RESTClient - https://github.com/rest-client/rest-client for folks who prefer Sinatra's actions syntax.
etc...

Those tools are just tools that you can use in order to implement the connection and interaction with any Web API. But the problem is that every API has Its interface, authorization rules, rate limiter, the way It handles errors, etc. So if in your project you need to connect 2 totally different APIs - you need to implement the interaction to each one yourself (assuming that those APIs have no gems available). If that API interaction code will not be properly encapsulated in Ruby classes, every developer in your team will need to carefully read API documentation in order to use the code that triggers those API calls. With every added API to your project, this issue will be a bigger and bigger problem with time. In this article, I will provide my own idea of how to create a simple, but powerful Ruby class pattern to hide HTTP interaction for any API in a Ruby class. This idea is based on the work that I did with my colleague years ago. Every such class will have the same external interface. As a bonus - my solution includes a way to do async HTTP requests.

Pattern

We should start with the list of Ruby gems that I am gonna use In my example:

gem 'concurrent-ruby'
gem 'faraday'
gem 'faraday_curl'

concurrent-ruby - to add async requests ability. https://github.com/ruby-concurrency/concurrent-ruby
faraday - to do HTTP requests (you can use any approach to do HTTP requests as I mentioned in the Intro section)
faraday_curl - to get nice Curl requests in logs (totally optional)

The next step is to create a module that I am gonna call FaradayConnector - this module will have all the needed logic in order to make any kind of HTTP requests to Web API (JSON by default). This module will be used to mix in other modules (API-specific ones). I will add more clarity on that later in the article:

require 'faraday'
require 'faraday_curl'
require 'concurrent'

module FaradayConnector
  class ServerError < RuntimeError
    attr_reader :response

    def initialize(response)
      @response = response
      super
    end
  end
  class TimeoutError < RuntimeError; end
  class ClientError < RuntimeError; end

  def request
    return @_request if defined?(@_request)

    # creating a Promise for async approach
    @_request = Concurrent::Promises.future { do_request }
  end

  def process
    return @_process if defined?(@_process)

    request
    @_process = do_process
  end

  def as_json(_options = {})
    process
  end

  protected

  def do_request
    # implement the real request in Child
  end

  def do_process
    # implement additional response decorations in Child
    request.value!
  end

  def url
    # must be added in Child
    raise 'Undefined url'
  end

  def auth
    # must be added in Child or use nil, if API has no Authorization
    raise 'Undefined auth'
  end

  def additional_headers
    {}
  end

  def content_type
    'application/json'
  end

  def request_type
    :url_encoded
  end

  def get(path, params = {})
    handle_request { connection.get(path, params) }
  end

  def post(path, body = {})
    formatted_body = json_content? ? body.to_json : body
    handle_request { connection.post(path, formatted_body) }
  end

  def delete(path, params = {})
    handle_request { connection.delete(path, params) }
  end

  def put(path, body = {})
    formatted_body = json_content? ? body.to_json : body
    handle_request { connection.put(path, formatted_body) }
  end

  def timeout
    45
  end

  def connection
    @connection ||= Faraday.new(url: url) do |faraday|
      faraday.request request_type
      faraday.headers['Authorization'] = auth if auth
      faraday.headers['Content-Type'] = content_type
      faraday.headers = faraday.headers.merge(additional_headers) if additional_headers
      faraday.options.timeout = timeout
      faraday.response(:logger)
      faraday.request :curl, Logger.new($stdout), :info
      faraday.adapter Faraday.default_adapter
    end
  end

  def handle_request
    response = handle_errors { yield }
    parse_response(response)
  end

  # just an easier way to handle HTTP errors
  def handle_errors
    response = yield
    e = if [502, 504].include?(response.status)
          TimeoutError.new(response)
        elsif [500, 503].include?(response.status)
          ServerError.new(response)
        elsif [400, 401, 404, 422].include?(response.status)
          ClientError.new(response)
        end
    return response unless e

    raise e
  end

  def parse_response(response)
    return {} unless response.body

    json_content? ? JSON.parse(response.body) : response
  rescue JSON::ParserError
    {}
  end

  def json_content?
    content_type == 'application/json'
  end
end

Now, assume we need to connect 2 different APIs (I just googled 2 public ones to show as an example):

OpenMeteo - https://open-meteo.com/ to get weather forecast
SportMonks - https://docs.sportmonks.com/football/api/ to get stats for Football teams

Every API has Its own Authorization rules, OpenMeteo has none, and SportMonks uses an Authorization token. We need to create 2 folders with modules called ApiConnector:

open_meteo/api_connector.rb

require './faraday_connector'

module OpenMeteo
  module ApiConnector
    include FaradayConnector

    def url
      'https://api.open-meteo.com'
    end

    def auth
      # no auth for this example
    end
  end
end

sport_monks/api_connector.rb

require './faraday_connector'

module SportMonks
  module ApiConnector
    include FaradayConnector

    def url
      'https://api.sportmonks.com'
    end

    def auth
      # a token goes here based on the API documentation
      # https://docs.sportmonks.com/football/welcome/authentication
      'some-token'
    end
  end
end

With these 2 easy steps, we created a codebase that can successfully authorize those 2 APIs we are using.
Time to create some HTTP requests. Let's start with OpenMeteo. On their home page, there are 2 examples of 2 different endpoints - forecast and historical. The next step is to create a separate class for every endpoint using the already created ApiConnector module.

forecast

module OpenMeteo
  class Forecast
    include ApiConnector

    attr_reader :latitude, :longitude

    def initialize(latitude, longitude)
      @latitude = latitude
      @longitude = longitude
    end

    def do_request
      get('v1/forecast',
        latitude: latitude,
        longitude: longitude
      )
    end

    def do_process
      request.value!
      # additional data manipulations goes here
    end
  end
end

historical

module OpenMeteo
  class Historical
    include ApiConnector

    attr_reader :latitude, :longitude, :start_date, :end_date

    def initialize(latitude, longitude, start_date, end_date)
      @latitude = latitude
      @longitude = longitude
      @start_date = start_date
      @end_date = end_date
    end

    def do_request
      get('v1/era5',
        latitude: latitude,
        longitude: longitude,
        start_date: start_date,
        end_date: end_date
      )
    end

    def do_process
      request.value!
      # additional data manipulations goes here
    end

    private

    def url
      # you can change url when needed easily
      'https://archive-api.open-meteo.com/'
    end
  end
end

There are 3 essential methods here:

initialize - a class constructor is only needed when an API request has input parameters
do_request - a place where you are triggering an HTTP request (available methods are get, post, put, delete). Remember this method returns Concurrent::Promise class instances.
do_process - calling value! method on Concurrent::Promise a class instance is essential to get the request-response.

Note: You probably noticed the url method override in the second file, we need this because a second API call has a different URL, and with this pattern, it's very easy to do)

This is the way to use those classes to do requests:

forecast = OpenMeteo::Forecast.new(42.361145, -71.057083)
historical = OpenMeteo::Historical.new(42.361145, -71.057083, '2024-01-01', '2024-01-30')
# Async part
forecast.request
historical.request

puts 'historical', historical.process
puts 'forecast', forecast.process

With the async approach, it means that we are doing both requests in 2 different threads, so they will work faster. When you call process method on any of It, promise will wait until all requests will be finished before returning any results, this is the way to synchronize code back.

The output will look like this:

I, [2024-05-09T14:34:30.756703 #74264]  INFO -- request: GET https://api.open-meteo.com/v1/forecast?latitude=42.361145&longitude=-71.057083
I, [2024-05-09T14:34:30.756857 #74264]  INFO -- request: Content-Type: "application/json"
User-Agent: "Faraday v2.9.0"
I, [2024-05-09T14:34:30.756822 #74264]  INFO -- request: GET https://archive-api.open-meteo.com/v1/era5?end_date=2024-01-30&latitude=42.361145&longitude=-71.057083&start_date=2024-01-01
I, [2024-05-09T14:34:30.761810 #74264]  INFO -- request: Content-Type: "application/json"
User-Agent: "Faraday v2.9.0"
I, [2024-05-09T14:34:31.084956 #74264]  INFO -- response: Status 200
I, [2024-05-09T14:34:31.085185 #74264]  INFO -- response: date: "Thu, 09 May 2024 18:34:31 GMT"
content-type: "application/json; charset=utf-8"
transfer-encoding: "chunked"
connection: "keep-alive"
content-encoding: "gzip"
I, [2024-05-09T14:34:31.090995 #74264]  INFO -- response: Status 200
I, [2024-05-09T14:34:31.091058 #74264]  INFO -- response: date: "Thu, 09 May 2024 18:34:31 GMT"
content-type: "application/json; charset=utf-8"
transfer-encoding: "chunked"
connection: "keep-alive"
content-encoding: "gzip"
historical
{"latitude"=>42.355007, "longitude"=>-71.12906, "generationtime_ms"=>0.0050067901611328125, "utc_offset_seconds"=>0, "timezone"=>"GMT", "timezone_abbreviation"=>"GMT", "elevation"=>9.0}
forecast
{"latitude"=>42.36515, "longitude"=>-71.0618, "generationtime_ms"=>0.00095367431640625, "utc_offset_seconds"=>0, "timezone"=>"GMT", "timezone_abbreviation"=>"GMT", "elevation"=>9.0}

As we can see, both requests are running at the same time, so we are sure the async approach is working. If we just run process methods for both instances one by one, we will have a synchronized approach, that will look like this:

forecast = OpenMeteo::Forecast.new(42.361145, -71.057083)
historical = OpenMeteo::Historical.new(42.361145, -71.057083, '2024-01-01', '2024-01-30')
puts 'historical', historical.process
puts 'forecast', forecast.process

I, [2024-05-09T14:37:15.830733 #74386]  INFO -- request: GET https://archive-api.open-meteo.com/v1/era5?end_date=2024-01-30&latitude=42.361145&longitude=-71.057083&start_date=2024-01-01
I, [2024-05-09T14:37:15.830765 #74386]  INFO -- request: Content-Type: "application/json"
User-Agent: "Faraday v2.9.0"
I, [2024-05-09T14:37:16.154027 #74386]  INFO -- response: Status 200
I, [2024-05-09T14:37:16.154220 #74386]  INFO -- response: date: "Thu, 09 May 2024 18:37:16 GMT"
content-type: "application/json; charset=utf-8"
transfer-encoding: "chunked"
connection: "keep-alive"
content-encoding: "gzip"
historical
{"latitude"=>42.355007, "longitude"=>-71.12906, "generationtime_ms"=>0.0050067901611328125, "utc_offset_seconds"=>0, "timezone"=>"GMT", "timezone_abbreviation"=>"GMT", "elevation"=>9.0}
I, [2024-05-09T14:37:16.157443 #74386]  INFO -- request: GET https://api.open-meteo.com/v1/forecast?latitude=42.361145&longitude=-71.057083
I, [2024-05-09T14:37:16.157511 #74386]  INFO -- request: Content-Type: "application/json"
User-Agent: "Faraday v2.9.0"
I, [2024-05-09T14:37:16.495799 #74386]  INFO -- response: Status 200
I, [2024-05-09T14:37:16.496003 #74386]  INFO -- response: date: "Thu, 09 May 2024 18:37:16 GMT"
content-type: "application/json; charset=utf-8"
transfer-encoding: "chunked"
connection: "keep-alive"
content-encoding: "gzip"
forecast
{"latitude"=>42.36515, "longitude"=>-71.0618, "generationtime_ms"=>0.00095367431640625, "utc_offset_seconds"=>0, "timezone"=>"GMT", "timezone_abbreviation"=>"GMT", "elevation"=>9.0}

Now let's get back to the second API that we planned to connect. It has a lot more examples that you can check out here https://docs.sportmonks.com/football/api/demo-response-files. To be very simple, let's say we want to get data about some team with an id 8.

module SportMonks
  class Team
    include ApiConnector

    attr_reader :team_id

    def initialize(team_id)
      @team_id = team_id
    end

    def do_request
      get("v3/football/teams/#{team_id}")
    end

    def do_process
      request.value!
      # additional data manipulations goes here
    end
  end
end

team = SportMonks::Team.new(8)
team.request
team.process

Same interface, but a different API.

Summary

Feel free to use this pattern for your API needs. I have been using this for years for a wide variety of services. I prefer using this, then connecting API-specific gems. From my perspective, this approach has a few benefits:

integrated async way of doing requests
same external interface for every API you will use in your codebase, which makes It very easy to use by any developer in your team
once ApiConnector is created with configured authorization, you will need to think about It ever again. (always use a secure way to store tokens or other secrets). If you don't know what to use, I can suggest dotenv gem - https://github.com/bkeepers/dotenv
flexible request configuration that is encapsulated in one class with a method override approach

PS: It's a copy of my original post on Medium - https://medium.com/@zozulyak.nick/ruby-class-pattern-to-work-with-api-requests-with-built-in-async-approach-bf0713a7dc96