Google Cloud Tasks with Active Job

TL;DR; It is now possible to use Google Cloud Tasks with Active Job using the cloudtasker gem with the release of v0.11.0. If you intend to deploy your Rails app on Cloud Run, Cloudtasker is the easiest solution to get started with background jobs on Rails.

Serverless has become an integral part of deployment infrastructures nowadays. While not designed for it originally, Rails can easily be deployed in serverless mode using platforms like GCP Cloud Run.

Cloud Run takes care of scaling your applications automatically based on volume on concurrent requests and bills you per millisecond of CPU usage. It also allows you to scale down to zero instances, making it a perfect choice for test environments.

Your app is a bit slow to boot? Cloud Run also takes care of keeping a warm pool of instances to ensure your service doesn't suffer from slow starts.

For all the reasons above, Cloud Run is certainly one of the most (cost) efficient infrastructure to deploy your Rails app.

The problem? Well with serverless you can't rely on background processes anymore - everything must be managed as an HTTP request. If your application requires background jobs then traditional solutions such as Sidekiq or Resque will not work because they require an active daemon.

Introducing Cloudtasker

Comes cloudtasker, the easiest way to have background jobs for Rails on Cloud Run. We wrote previously about cloudtasker, but this time it's even easier thanks to a native Active Job integration for Rails.

Cloudtasker is an interface to Google Cloud Tasks, a Google-managed job scheduler. It allows you to enqueue and process jobs in exactly the same way as other background job solutions (Active Job, Sidekiq, Resque).

When you enqueue jobs using Cloudtasker, they get pushed to Google Cloud Tasks. When the job is due for processing, Cloud Tasks sends a webhook to your application to process the job. Because this whole enqueue/process logic is HTTP based, it's completely compatible with serverless deployments.

What about working locally then? We've got you covered. Cloudtasker is shipped with a local server that mimics Cloud Tasks so you can work locally without relying on ngrok to receive webhooks.

Getting started with Cloudtasker and Active Job

In order to run jobs locally with Cloudtasker you'll need to install and run Redis. Redis is only required when working locally with the local processing server, not in production.

# E.g. using brew
brew install redis

Then add the following to your Gemfile:

# Active Job is available since v0.11.0
gem 'cloudtasker', '~> 0.11.0'

Add an initializer to configure Cloudtasker:

# config/initializers/cloudtasker.rb

Cloudtasker.configure do |config|
  # Configure the processor host
  #
  # This App URL is required to that Cloud Tasks knows where to send the job webhooks
  # Adapt the server port to be the one used by your Rails web process
  #
  # You may as well use the cloud run provided URL for your production environment
  #
  config.processor_host = Rails.env.production? 'https://app.myawesomesite.com' : 'http://localhost:3000'
end

Edit your config/application.rb and configure Active Job to use cloudtasker:

# config/application.rb

require_relative 'boot'
require 'rails/all'

Bundler.require(*Rails.groups)

module MyRailsApp
  class Application < Rails::Application
    # Initialize configuration defaults for originally generated Rails version.
    config.load_defaults 6.0

    # Settings in config/environments/* take precedence over those specified here.
    # Application configuration can go into files in config/initializers
    # -- all .rb files in that directory are automatically loaded after loading
    # the framework and any gems in your application.

    # Use cloudtasker as the ActiveJob backend:
    config.active_job.queue_adapter = :cloudtasker
  end
end

You're done with the setup part. Now let's define a job.

# app/jobs/example_job.rb

class ExampleJob < ApplicationJob
  queue_as :default

  def perform(some_arg)
    logger.info("Job run with #{some_arg}. This is working!")
  end
end

Launch Rails and the cloudtasker local processing server. This local processing server is only required in development to mimic Google Cloud Tasks locally.

# In one terminal
> rails s -p 3000

# In another terminal
> cloudtasker

# Note you might prefer to define a Procfile with a web and worker process 
# then use foreman to launch both in one go.

Now launch a Rails console and try enqueuing your first job.

# Process job as soon as possible
ExampleJob.perform_later('foo')

# Process job in 60 seconds
ExampleJob.set(wait: 60).perform_later('foo')

That's it! You should now see your jobs running in your logs.

Deploying to production

In order to deploy to production we'll need to add a few Cloud Task configuration details in our Cloudtasker initializer.

# config/initializers/cloudtasker.rb

Cloudtasker.configure do |config|
  # GCP project configuration
  config.gcp_location_id = 'us-central1' # defaults to 'us-east1'
  config.gcp_project_id = 'my-gcp-project'

  # GCP queue prefix. 
  # With the prefix below, jobs will be pushed to Cloud Tasks on the 'my-app-default' queue.
  #
  # If you configure an Active Job to `queue_as: 'low'` then jobs will be pushed
  # to Cloud Tasks on the 'my-app-low' queue.
  #
  config.gcp_queue_prefix = 'my-app'

  # Configure the processor host
  #
  # This App URL is required to that Cloud Tasks knows where to send the job webhooks
  # Adapt the server port to be the one used by your Rails web process
  #
  # You may as well use the cloud run provided URL for your production environment
  #
  config.processor_host = Rails.env.production? 'https://app.myawesomesite.com' : 'http://localhost:3000'
end

You should now create the Cloud Task queues in your project. Active Jobs will be pushed to the gcp_prefix-queue_as queue (e.g. myapp-critical). By default, this is gcp_prefix-default (e.g. myapp-default).

You can create your job queues via the Cloud Task UI, using the gcloud CLI or using the cloudtasker-provided rake task.

# Create your default queue and a low queue using the gcloud CLI
gcloud tasks queues create <gcp_queue_prefix>-default --max-concurrent-dispatches=20
gcloud tasks queues create <gcp_queue_prefix>-low --max-concurrent-dispatches=5

# Create your default queue and a low queue using the Cloudtasker setup task
# This still requires the gcloud CLI to be installed and authenticated
rake cloudtasker:setup_queue name=default concurrency=20
rake cloudtasker:setup_queue name=low concurrency=20

The bare minimum is to create the gcp_prefix-default queue.

Once you are done creating your queues, just deploy your app and enjoy seamless background jobs!

Kudos to the author

I'd like to thank Roberto Quintanilla for developing the Active Job integration. It's a really great addition to the gem.