How to Monitor Ruby on Rails Background Jobs (Sidekiq) with Vigilmon

Background jobs are the backbone of most Rails applications — sending emails, processing uploads, syncing data. When Sidekiq goes down, jobs silently queue up and users wonder why their emails never arrived.

This guide covers monitoring both your Rails web app and Sidekiq worker health with Vigilmon.

What Can Go Wrong with Sidekiq

• Redis connection drops (Sidekiq can't process)
• Sidekiq process crashes (workers stop silently)
• Queue backup (jobs processing too slowly)
• Scheduled jobs (cron) stop running
• Job retry storms consuming all resources

None of these show up in web request logs. You need dedicated monitoring.

Step 1: Add a Sidekiq Health Endpoint to Rails

Add a health controller that checks both Rails and Sidekiq:

# app/controllers/health_controller.rb
class HealthController < ApplicationController
  skip_before_action :authenticate_user!, raise: false
  skip_before_action :verify_authenticity_token

  def show
    checks = {
      database: database_ok?,
      redis: redis_ok?,
      sidekiq: sidekiq_ok?,
    }

    status = checks.values.all? ? :ok : :service_unavailable
    render json: { status: status, checks: checks }, status: status
  end

  private

  def database_ok?
    ActiveRecord::Base.connection.execute("SELECT 1")
    true
  rescue
    false
  end

  def redis_ok?
    Sidekiq.redis { |conn| conn.ping == "PONG" }
  rescue
    false
  end

  def sidekiq_ok?
    stats = Sidekiq::Stats.new
    # Alert if dead job count is growing or queue is backed up
    stats.dead_size < 100 && stats.enqueued < 1000
  rescue
    false
  end
end

Add the route:

# config/routes.rb
Rails.application.routes.draw do
  get '/health', to: 'health#show'
  # ...
end

Step 2: Sidekiq Built-in Web UI (optional)

Sidekiq ships with a web dashboard. Mount it for internal visibility:

# config/routes.rb
require 'sidekiq/web'

Rails.application.routes.draw do
  # Protect with authentication
  authenticate :user, lambda { |u| u.admin? } do
    mount Sidekiq::Web => '/sidekiq'
  end
end

But this only works when you're logged in and watching. Vigilmon watches 24/7.

Step 3: Heartbeat Monitoring for Scheduled Jobs

For cron-style jobs (using sidekiq-scheduler or sidekiq-cron), use Vigilmon's heartbeat monitoring to verify jobs actually run:

# app/jobs/daily_report_job.rb
class DailyReportJob < ApplicationJob
  queue_as :default

  def perform
    # ... your job logic
    GenerateReport.call

    # Ping Vigilmon to confirm job completed
    heartbeat_url = ENV['VIGILMON_HEARTBEAT_URL']
    Net::HTTP.get(URI(heartbeat_url)) if heartbeat_url
  rescue => e
    Rails.logger.error "DailyReportJob failed: #{e.message}"
    raise
  end
end

In Vigilmon, create a Heartbeat Monitor:

• Expected ping interval: every 24 hours
• Alert if no ping received within: 25 hours (buffer for job delays)

Vigilmon alerts you if your scheduled job stops running — even if Sidekiq itself is "up".

Step 4: Queue Depth Monitoring

Create a separate endpoint for queue metrics:

# app/controllers/health_controller.rb (extended)
def queues
  stats = Sidekiq::Stats.new
  queue_data = Sidekiq::Queue.all.map do |q|
    { name: q.name, size: q.size, latency: q.latency.round(2) }
  end

  render json: {
    processed: stats.processed,
    failed: stats.failed,
    enqueued: stats.enqueued,
    dead: stats.dead_size,
    workers: Sidekiq::Workers.new.size,
    queues: queue_data,
  }
end

Route it:

get '/health/queues', to: 'health#queues'

Set up a Vigilmon monitor on /health that returns 503 when the queue is backed up beyond your threshold.

Step 5: Connect Vigilmon

1. Sign up at vigilmon.online
2. Monitor 1: HTTP Monitor → https://yourapp.com/health (checks Rails + Redis + Sidekiq basics)
3. Monitor 2: Heartbeat Monitor → ping URL for your critical scheduled jobs
4. Monitor 3: SSL Certificate Monitor → your domain
5. Set alert channels: email, Slack, PagerDuty

Failure Coverage Map

Failure Mode	Application Logs	Vigilmon
Sidekiq process crash	Silent	✅ Health endpoint 503
Redis connection lost	Errors in Sidekiq logs	✅ Health endpoint 503
Scheduled job stopped running	Silent	✅ Heartbeat alert
Rails web process crash	Silent	✅ HTTP monitor
Queue backed up > 1000 jobs	Only in Sidekiq Web UI	✅ Custom threshold
SSL expiry	Manual check	✅ SSL monitor

Dockerfile Integration

If running Sidekiq in Docker:

# Sidekiq-specific health check
HEALTHCHECK --interval=30s --timeout=10s --retries=3 \n  CMD bundle exec sidekiq-healthcheck || exit 1

Or use a simple Redis ping:

HEALTHCHECK --interval=30s \n  CMD redis-cli -h $REDIS_HOST ping || exit 1

Alert Thresholds to Configure

• Dead job count > 50: Something is failing repeatedly
• Queue latency > 60 seconds: Workers can't keep up
• No heartbeat from scheduled job: Job stopped running
• Health endpoint returns 503: Critical dependency down

Free Tier

Vigilmon's free plan covers 5 monitors with 1-minute intervals — enough for:

• 1 web health check
• 1 Sidekiq health check
• 1 heartbeat for your most critical cron job
• 1 SSL monitor
• 1 spare

Summary

1. Add a /health endpoint that probes Rails DB, Redis, and Sidekiq queue state
2. Use heartbeat monitors to verify scheduled jobs actually execute
3. Connect Vigilmon for 24/7 external monitoring
4. Get alerted when Sidekiq silently dies before your users notice

---

Monitor your Rails app free at vigilmon.online