DEV Community: sylvance

Implementing the Outbox Pattern with Sinatra, Sidekiq, and Postgres

sylvance — Mon, 07 Aug 2023 19:33:11 +0000

Prerequisites:

Ruby installed on your system
Postgres installed and running
Redis server installed and running (default port: 6379)
Bundler gem installed (gem install bundler)

Step 1: Setup the Project

1.1 Create a new project directory and navigate to it:

Copy code

mkdir outbox_sinatra_example
cd outbox_sinatra_example

1.2 Create a Gemfile to manage dependencies:

source 'https://rubygems.org'

gem 'sinatra'
gem 'pg'
gem 'sidekiq'

1.3 Install the required gems using Bundler:

bundle install

Step 2: Create the Outbox Table

2.1 Access your Postgres database

Assuming you have a database named 'your_database', using psql or any other PostgreSQL client.

2.2 Create the Outbox table:

In the psql shell do below, this will create a table for oubox messages;

CREATE TABLE outbox (
  id SERIAL PRIMARY KEY,
  message_type VARCHAR(255),
  message_body TEXT,
  created_at TIMESTAMPTZ DEFAULT NOW(),
  processed_at TIMESTAMPTZ
);

Step 3: Implement the Sinatra Endpoint and OutboxWorker

3.1 Create a new file named app.rb in the project directory:

# app.rb
require 'sinatra'
require 'pg'
require 'sidekiq'
require 'json'

# Configure Sidekiq with Redis URL
Sidekiq.configure_server do |config|
  config.redis = { url: 'redis://localhost:6379/0' }
end

class OutboxWorker
  include Sidekiq::Worker
  sidekiq_options retry: 5 # Set the number of retries on failure

  def perform(message_id)
    conn = PG.connect(dbname: 'your_database')
    message = conn.exec_params('SELECT * FROM outbox WHERE id = $1', [message_id]).first
    return unless message

    # Ensure idempotency - check if the message has already been processed
    return if message['processed_at']

    begin
      # Process the message here (e.g., send the message to the appropriate destination)
      # ...

      # Mark the message as processed and record the processing time
      conn.exec_params(
        'UPDATE outbox SET processed_at = $1 WHERE id = $2',
        [Time.now, message_id]
      )
    rescue StandardError => e
      # If an error occurs, log it and let Sidekiq retry the job
      puts "Error processing message (id=#{message['id']}): #{e.message}"
      raise e
    end
  end
end

post '/create-message' do
  content_type :json
  data = JSON.parse(request.body.read)

  message_type = data['message_type']
  message_body = data['message_body']

  conn = PG.connect(dbname: 'your_database')
  result = conn.exec_params(
    'INSERT INTO outbox (message_type, message_body) VALUES ($1, $2) RETURNING id',
    [message_type, message_body]
  ).first

  # Enqueue the message for background processing
  OutboxWorker.perform_async(result['id']) if result

  status 201
  { message: 'Message created and enqueued for processing' }.to_json
end

Step 4: Set Up the config.ru File

4.1 Create a config.ru file to define the application for use with Rack:

# config.ru
require './app'

run Sinatra::Application

Step 5: Start the Application

5.1 Start Sidekiq worker in a separate terminal window:

bundle exec sidekiq -r ./app.rb

5.2 Start the Sinatra application using the Rack server:

bundle exec rackup

Step 6: Testing the Outbox Endpoint

Use a tool like curl or Postman to send a POST request to create a new message:

curl -X POST -H "Content-Type: application/json" -d '{"message_type":"notification","message_body":"Hello, World!"}' http://localhost:9292/create-message

Step 7: Verify Message Processing

Check the logs in the terminal running Sidekiq to see the message processing:

[INFO] 2023-08-07T00:00:00.000Z: [OutboxWorker] start
Processing message: 1, Type: notification, Body: Hello, World!
[INFO] 2023-08-07T00:00:00.000Z: [OutboxWorker] done: 3.123 sec

With these changes, the OutboxWorker now handles retries on failure using Sidekiq's retry mechanism. If a processing error occurs, the job will be retried up to 5 times (or any number you specify in sidekiq_options) before being moved to the Sidekiq dead queue.

Additionally, we've introduced idempotency in the worker by checking if the message has already been processed before attempting to process it again. If the processed_at field in the Outbox table is already set, the worker will skip processing the message again, ensuring that the same message isn't processed multiple times.

Keep in mind that idempotency also depends on the processing logic of the actual message handler, so ensure that your message processing code is idempotent as well.

Monitoring using Honeybadger and Rails Activesupport instrumentation

sylvance — Tue, 04 Feb 2020 20:02:15 +0000

Introduction

This tutorial explains how to report data of rails application usage based critical events. These events are subjective but we will look at a more general type of event namely an endpoint that takes too long to load.

We will report the events using Honeybadger. Honeybadger, as we will see, is easy to setup and has many features that allow.

Setting up Honeybadger

First, we will need a Rails application to plug Honeybadger into. Conveniently, I have created a template that setups a quick rails application with simple authentication and user data seeded to a database. To use it, do the following

Download this rails template gist https://gist.github.com/Sylvance/5c7d5db8ce8609fa16403e74255dee39
Change into the directory this gist is in.
Then run rails new your_app_name --database=postgresql -m ./template.rb to create the rails app with a postgres database. When asked to override stuff say Yes.

There's that. We now have a full Rails app which is simple but complex enough to show where Honeybadger shines.

Sign up for a trial account at Honeybadger. Then choose rails as the framework you will use. Follow the instructions shown for Rails then click "Complete Setup". I will just repeat those instructions just in case needed. Now in your Gemfile add gem 'honeybadger', '~> 4.0' then run bundle install. Next use your apikey to run this command; bundle exec honeybadger install [apikey]. This command will generate a honeybadger.yml file under the config directory then sends a test notification to your Honeybadger errors dashboard which will look something like below.

Congrats. You've setup Honeybadger and plugged it into your Rails app.

Reporting using Honeybadger.notify

Reporting using Honeybadger is just a call away. But first we shall add this to our configuration file, config/honeybadger.yml;

env: 'production'

By default, Honeybadger suppresses reports sent on a development environment. By adding the line above we are able to send reports to the Honeybadger account so that they are displayed on the dashboard.

Now we will use Honeybadger.notify to report everytime any endpoint is hit in our app. Add the following to application_controller.rb;

before_action do
    Honeybadger.notify("Reporting live from an arbitrary endpoint!")        
end

If we run the server using rails s then go to http://localhost:3000 within the application we will see this in the terminal;

If we look at the Honeybadger errors dashboard we will see this;

That's great! We have made our first report using Honeybadger. However, it is not that helpful since it has no insights as to what problem might have occurred. Next let's build on this by providing more insights.

Using Honeybadger.context to give more context to error reports

To give context as to how an error occurred Honeybadger gives you the context method to help with this. For instance, we can use the context method to include data about the user associated with a report.
To do this in our application, within the application_controller.rb replace the previous code with;

before_action :build_context

# place below authenticate_request method
def build_context
    Honeybadger.context({
        user_id: current_user.id,
        user_email: current_user.email
    })
end

Now whenever a report is sent using Honeybadger it will contain the details of the user that sent it. We will still need to call Honeybadger.notify to do this we can add this line to the app/controllers/users_controller.rb within the index action:

def index
    @users = User.all
    ActiveSupport::Notifications.instrument "index.event"
end

Everytime this action is executed an index.event event is published. We will need a subscriber in order to use this to send Honeybadger reports. Create a new file in config/initializers and call it events.rb. Insert this:

ActiveSupport::Notifications.subscribe "index.event" do |*args|
    event = ActiveSupport::Notifications::Event.new(*args)
    Honeybadger.notify("This report is from the index action")
end

This subscriber listens to our custom "index.event" event and triggers Honeybadger to send a report. We will need to add this in the app/controller/sessions_controller.rb;

skip_before_action :build_context, only: [:new, :create]

This should skip building the context when logging in

Now to test how context works, restart the server with rails s and go to http://localhost:3000/signup and create a new user then login at http://localhost:3000/login. Then go to http://localhost:3000/users. A report will be sent from the index action in app/controllers/users_controller.rb. This is how context will look like in the Honeybadger dashboard with the user details in it.

Using Honeybadger with the ActiveSupport instrumentation API

Let's say we want to send a report every time an endpoint request takes longer than 2 seconds to execute. To do this, we can leverage the ActiveSupport instrumentation API which acts in publisher-subscriber type of way.

Inside events.rb insert this code:

ActiveSupport::Notifications.subscribe 'process_action.action_controller' do |*args|
    event = ActiveSupport::Notifications::Event.new(*args)
    Rails.logger.info "Event received: #{event}"
    if event.duration > 2000
        Honeybadger.notify("This action has lasted more than 2s")
    end
end

Above we have a subscriber that 'listens' to the process_action.action_controller event in Rails. These events occur when a controller action is executed. When the event occurs, we then check for how long it took using event.duration. If the duration is longer than 2s we send a report using Honeybadger.notify.

Now to test this out let us go to app/controllers/users_controller.rb and add code to cause a 3-second delay within the show action as shown:

def show
    sleep 3
end

This action is guaranteed to execute longer than 2 seconds every time and thereby sending a report to Honeybadger. Now restart the server with rails s and go to http://localhost:3000/users. Pick any user and click on their respective Show link. This will execute for 3 seconds or longer and a report will be seen in the Honeybadger errors dashboard as shown below.

Using Honeybadger with breadcrumbs

Breadcrumbs give us the ability to see statements that have been executed leading up to an error. This is powerful when debugging an issue that has been reported by Honeybadger. We will update the Honeybadger yaml config to enable breadcrumbs:

breadcrumbs:
  enabled: true

And that is all you need to get breadcrumbs running for Honeybadger. To test it out, restart the server then go to http://localhost:3000/users. Pick any user and click on their respective Show link. This will trigger a report to be sent. In the dashboard the breadcrumb will look something like this;

Conclusion

Honeybadger is an excellent tool to use for monitoring an application. It is relatively easy and quick to setup. Using it with Rails ActiveSupport gives you full control of monitoring any parts of your application. All the resulting code for this article can be found here https://github.com/Sylvance/badge-honey.