Run Sidekiq 6 as daemon in Production environment on Ubuntu 20.04

I also have Traditional Chinese version

利用 systemd 在 Ubuntu 執行 Sidekiq 6

Kevin Luo ・ Aug 20 '21

#rails #sidekiq #systemd

Sidekiq is a very convenient queuing tool for developers to handle the asynchronous tasks and response clients sooner.

Deploying Sidekiq on production server is not hard but there are many details which should be reminded.

I'd like to share my experience of executing Sidekiq 6 by systemd as daemon service on Ubuntu 20.04

What is our goal?

• Execute Sidekiq
• Sidekiq should be start automatically after reboot or crash
• Sidekiq should be restarted after we deploy Rails application via Capistrano

How do we execute Sidekiq?

To know how to execute Sidekiq in the production server. We need to know how do we Execute Sidekiq in the development environment.

bundle exec sidekiq

It is in fact very simple. You can append more arguments such as -C xxxx.yml for specific config file, but we can keep it simple now.

So, actually, you can do the same thing on the production server.
If you want it to be more like a daemon. You can add a & to make the command run in the background:

RAILS_ENV=production bundle exec sidekiq &

If you just want to execute Sidekiq in the production environment, that’s it.

But it is not very practical and clearly doesn’t fit our needs:

1. It won’t be restarted after the server reboot
2. It won’t restart automatically after we deploy the application or it crashes due to exceptions
3. It does not feel “Pro”. You want a more “Pro” way to do this even you’re not using Sidekiq Pro….(just kidding😂)

To make it more ideal and practical, we need to use systemd

What is systemd?

systemd is a service manager program to manage the daemon program and it is used by Linux system.
With systemd, we can…

• use the command systemctl to start/stop any service
• enable services, and the enabled services will be started automatically after the system boot.
• You can specify what to do if the service fails, for example, restarting it.

Those are what we need to know about systemd for now, feel free to google more information about it.
It sounds like it can fit our needs, so let’s use systemd to control Sidekiq!

Add Sidekiq as a service unit

Every service is treat as an unit and it has a unit file to describe each service.
We can make a file under /lib/systemd/system/sidekiq.service

By the way the user to execute Sidekiq is named deployer

# /lib/systemd/system/sidekiq.service
[Unit]
Description=sidekiq
After=syslog.target network.target

[Service]
Type=simple
WorkingDirectory=/path/to/your/app

# If you use rbenv:
# ExecStart=/bin/bash -lc 'exec /home/deploy/.rbenv/shims/bundle exec sidekiq -e production'
# If you use the system's ruby:
# ExecStart=/usr/local/bin/bundle exec sidekiq -e production
# If you use rvm in production without gemset and your ruby version is 2.6.5
# ExecStart=/home/deploy/.rvm/gems/ruby-2.6.5/wrappers/bundle exec sidekiq -e production
# If you use rvm in production with gemset and your ruby version is 2.6.5
# ExecStart=/home/deploy/.rvm/gems/ruby-2.6.5@gemset-name/wrappers/bundle exec sidekiq -e production
# If you use rvm in production with gemset and ruby version/gemset is specified in .ruby-version,
# .ruby-gemsetor or .rvmrc file in the working directory
ExecStart=/home/deployer/.rvm/bin/rvm in /path/to/your/app/current do bundle exec sidekiq -e production

User=deployer
Group=deployer
UMask=0002

# Greatly reduce Ruby memory fragmentation and heap usage
# https://www.mikeperham.com/2018/04/25/taming-rails-memory-bloat/
Environment=MALLOC_ARENA_MAX=2

# if we crash, restart
RestartSec=1
Restart=on-failure

# output goes to /var/log/syslog
StandardOutput=syslog
StandardError=syslog

# This will default to "bundler" if we don't specify it
SyslogIdentifier=sidekiq

[Install]
WantedBy=multi-user.target

Because systemd scans all services under /lib/systemd/system/, so your sidekiq.service can be found by systemd now.

After creating sidekiq.service, you can execute the following commands

# reload serivces
sudo systemctl daemon-reload
# enable the sidekiq.service so it will start automatically after rebooting
sudo systemctl enable sidekiq.service
# start the sidekiq
sudo service sidekiq start
# we can check the log in /var/log/syslog
sudo cat /var/log/syslog
# we can check if Sidekiq is started
sudo ps aux | grep sidekiq
# or
sudo systemctl status

Restart Sidekiq by Capistrano

I use Capistrano to deploy rails because it saves my time and prevent human mistakes.
After I deploy the latest code to production server, I also want Sidekiq to be restarted so it can load the latest code.
To add this behavior into Capistrano workflow, we can use the gem capistrano-sidekiq
Adding this gem into Gemfile then do bundle install

# Gemfile
gem 'capistrano-sidekiq', group: :development

In the Capfile, add:

# Capfile
require 'capistrano/sidekiq'
# Default sidekiq tasks
install_plugin Capistrano::Sidekiq
# We specify that we want to use systemd to control sidekiq
install_plugin Capistrano::Sidekiq::Systemd

Then during the cap production deploy, it will do the following tasks in order:

• stop get new task from redis
• stop Sidekiq service
• start Sidekiq service

Make an user-wide sidekiq.service

There is one thing I didn’t mention on purpose.
The service unit we just created is a system-wide and always need sudo to use systemctl.
If we want any unprivileged user to utilize systemd, we need a user-wide service unit.
Moreover, capistrano/sidekiq default setting expects the user-wide service unit to control the Sidekiq.

And Yes, we can just modify the default setting of capistrano/sidekiq to control to use the system-wide systemd, but I just want to explain there are many options we can choose from. (And I was also confused why my sidekiq.service not work…when I was learning how to execute Sidekiq on production server)

If you have already enabled the system-wise sidekiq.service, you need to disable it and delete the service-unit:

sudo systemctl stop sidekiq
sudo systemctl disable sidekiq.service
sudo rm /lib/systemd/system/sidekiq.service

The user-wide services should be put under ~/.config/systemd/user/, where is the location systemd scans for the user-wide services. Add a file named sidekiq.service again:

[Unit]
Description=sidekiq
After=syslog.target network.target

[Service]
# notify can be used only after Sidekiq 6.0.6
# if the version is under 6.0.6, use Type=simple
Type=notify
WatchdogSec=10

WorkingDirectory=/path/to/your/app/current

# If you use rbenv:
# ExecStart=/bin/bash -lc 'exec /home/deploy/.rbenv/shims/bundle exec sidekiq -e production'
# If you use the system's ruby:
# ExecStart=/usr/local/bin/bundle exec sidekiq -e production
# If you use rvm in production without gemset and your ruby version is 2.6.5
# ExecStart=/home/deploy/.rvm/gems/ruby-2.6.5/wrappers/bundle exec sidekiq -e production
# If you use rvm in production with gemset and your ruby version is 2.6.5
# ExecStart=/home/deploy/.rvm/gems/ruby-2.6.5@gemset-name/wrappers/bundle exec sidekiq -e production
# If you use rvm in production with gemset and ruby version/gemset is specified in .ruby-version,
# .ruby-gemsetor or .rvmrc file in the working directory
ExecStart=/home/deployer/.rvm/bin/rvm in /path/to/your/app/current do bundle exec sidekiq -e production
ExecReload=/usr/bin/kill -TSTP $MAINPID

# Greatly reduce Ruby memory fragmentation and heap usage
# https://www.mikeperham.com/2018/04/25/taming-rails-memory-bloat/
Environment=MALLOC_ARENA_MAX=2

# if we crash, restart
RestartSec=1
Restart=on-failure

# output goes to /var/log/syslog
StandardOutput=syslog
StandardError=syslog

# This will default to "bundler" if we don't specify it
SyslogIdentifier=sidekiq

[Install]
WantedBy=default.target

Then we can use following commands to control sidekiq.service:

systemctl --user daemon-reload 
systemctl --user enable sidekiq.service

# you can use systemctl to control sidekiq
systemctl --user {start,stop,restart} sidekiq

With this sidekiq.service file and enabled it by systemd, the capistrano/sidekiq should work properly : )

Common problems

Sidekiq always turns down while start it via user-wise systemd

User-wise systemd actually only allows logged-in user to execute the services.
Therefore, all the services will be shut down while the last user session is ended.
To keep the users session to execute the user-wise systemd service, we can use the command below:

loginctl enable-linger [user_name]

Then the user's session will be created once the system starts, and will be kept. And the user-wise systemd services can be preserved to run.

When running Sidekiq:quiet task during deploying, it has error below

sidekiq:quiet
    01 systemctl --user reload sidekiq
    01 Failed to reload sidekiq.service: Job type reload is not applicable for unit sidekiq.service.
  ✘ 01 deployer@xxxxxxxx 0.067s

Solution

• add ExecReload=/bin/kill -TSTP $MAINPID into
• Although sidekiq readme has said that it is recommended to use this command to stop receiving tasks from redis. Capistrano/sidekiq does not have the corresponding explanation.

The target is not found

• The target is a group of service unit
• You can use systemctl list-units —type=target to find the available targets

Redis is not started yet

• Actually this is the situation I imagine…I haven’t had this situation yet.
• I think add redis.service in After may solve this… After=syslog.target network.target redis.service

The -L log/sidekiq.log is not work

• Log redirection is not supported in Sidekiq 6 anymore, please read Logging · mperham/sidekiq Wiki · GitHub
• we can Add Sidekiq tag to the logs in the syslog by bundle exec sidekiq 2>&1 | logger -t sidekiq

If you think this article is helpful, you can buy me a coffee to encourage me 😉

Reference

• An example of sidekiq.service provided by sidekiq repo: sidekiq/sidekiq.service at master · mperham/sidekiq · GitHub
• GitHub - seuros/capistrano-sidekiq: Sidekiq integration for Capistrano