DEV Community: Hopsoft

Optimizing Rails connections

Hopsoft — Fri, 18 Mar 2022 00:17:09 +0000

This is the 3rd post in a series that starts here..

Have you ever experienced a Rails app that suddenly and inexplicably delivers dismal response times or becomes completely unresponsive? It can be incredibly confusing, especially when nothing in the environment has changed recently.

Let's look at one of the most common reasons this might happen.

Datastore Connections

Correctly configuring Rails connections can be complicated. Here's a short video that explains how easy it is to get wrong.

Note that this problem isn't isolated to your primary database either. It's possible to misconfigure any datastore that has limited connections. Postgres, Redis, Memcached, etc...

There are a few aspects of this problem to consider.

Connection Limits

Datastores have an upper limit to how many active connections they can reasonably support. System resources typically govern these limits. For example, PostgreSQL defaults to 100 connections while Redis defaults to 10,000. It's possible to override default connection limits. Just be sure you know what you're doing.

Hosting providers impose further limits. Consider that Heroku's Standard 0 Postgres plan caps at 120 connections and their Premium 0 Redis plan caps at 40.

Connection Pools

Establishing a connection is an expensive and slow operation. Connection pooling reduces overall latency by limiting how frequently connections are set up by reusing pre-established connections.

The improved efficiency gained from connection pooling enables applications to serve more concurrent requests than otherwise possible based on datastore connection limits alone. Pooling is a force multiplier.

Deployment Topology

The deployment configuration plays a significant role. Consider a database with a limit of 40 connections. We'd be at capacity with 2 servers configured to use 20 connections each.

This problem compounds as the deployment becomes more complex. And the challenge isn't limited to the number of servers. We also need to account for the number of processes running per server and the number of threads active in each process.

The Variables

Typical Rails applications use the following environment variables.

WEB_CONCURRENCY - the number of processes
RAILS_MAX_THREADS - the number of threads

These are descriptive names, but I think we can do better. Given that Rails applications do more than serve web requests, let's consider some more suitable names before looking at formulas.

SERVERS - the number of servers/dynos
PROCESSES - the number of processes
THREADS - the number of threads

We also need to factor in external systems that have access to the database. This adds another variable to consider.

EXTERNAL_CONNECTIONS

That's all the variables, now onto the formula.

The Formula

The formula is quite simple.



(SERVERS * PROCESSES * THREADS) + EXTERNAL_CONNECTIONS

Let's look at a standard Rails deployment.

2 web servers, 2 processes, 8 threads
3 worker servers, 4 processes, 16 threads
10 external connections

Here's the equation for the number of connections.



(2 * 2 * 8) + (3 * 4 * 16) + 10 = 234

It's vital to ensure that you don't configure more connections than are available. You may need to increase the provisioned hardware or reduce the number of connections your application uses.

Configuration

Here's an example of configuring Rails for the deployment described above.



# config/database.yml
default: &default
  adapter: postgresql
  encoding: unicode
  pool: <%= ENV["THREADS"] %>
  # it's generally a good idea to configure timeouts
  connect_timeout: 1
  variables:
    statement_timeout: 5s
    lock_timeout: 2s



# config/initializers/sidekiq.rb
settings = {
  url: ENV["REDIS_QUEUE_URL"], 
  size: ENV["THREADS"].to_i + 2, # pool size
  # it's generally a good idea to configure timeouts
  connect_timeout: 0.2,
  read_timeout: 0.5,
  write_timeout: 0.5
}

Sidekiq.configure_server do |config|
  config.redis = settings
end

Sidekiq.configure_client do |config|
  config.redis = settings
end



# config/environments/production.rb
Rails.application.configure do
  config.cache_store = :redis_cache_store, {
    url: ENV["REDIS_CACHE_URL"],
    size: ENV["THREADS"] + 2, # pool size
    # it's generally a good idea to configure timeouts
    connect_timeout: 0.2,
    read_timeout: 0.5,
    write_timeout: 0.5
  }
end



# config/puma.rb
workers ENV["PROCESSES"]
max_threads_count = ENV["THREADS"]
min_threads_count = ENV["THREADS"]
threads min_threads_count, max_threads_count



# Procfile
web: PROCESSES=2 THREADS=8 bin/start-pgbouncer bundle exec puma -C config/puma.rb
worker: PROCESSES=4 THREADS=16 bin/start-pgbouncer bundle exec sidekiq -C config/sidekiq.yml

The formula and configuration may appear simple, but they are deceptively complex. The complexity becomes more apparent as the deployment grows more sophisticated. For example, consider a deployment with multiple worker pools of varying sizes backed by Redis instances with different connection limits. Can you think of other deployment topologies? How does the math work out?

Advanced Options

Stacked Pooling

It's possible to stack connection pooling with tools like pg_bouncer, which can help further ensure your Rails application doesn't exhaust database connections. Heroku supports this with a buildpack.

Sidekiq Swarm

The number of processes and threads can be hard to determine when using tools like Sidekiq swarm. Just pay attention and ensure that you know what the number is.

Sophisticated, Simple, and Affordable Background Workers

Hopsoft — Fri, 11 Mar 2022 02:05:49 +0000

This is the 2nd post in a series that starts here.

CodeFund

At CodeFund, we processed 5+ million background jobs per day at our peak. Some of those jobs performed computationally expensive operations. Others performed the heavy database work required to operate the business. We hosted on Heroku with 3 Standard 1x dynos responsible for background work. It cost $75/month and had plenty of capacity to scale without additional costs.

How did we accomplish this?

The first thing to consider is how best to separate the work. Think about the different types of jobs your application performs. Are there any jobs that should be isolated from the primary work queue? At CodeFund we pulled financial settlement and ETL processes out and set up a dedicated dyno to run these jobs on a schedule which allowed the primary worker dynos to focus on real-time operations, things like recording ad impressions and clicks.

Here's our Heroku worker dyno configuration.

And here's how we configured Sidekiq queues.

# config/sidekiq.yml
:concurrency: 5
:queues:
  - critical
  - mailers
  - schedule
  - impression
  - click
  - default
  - rollbar
  - action_mailbox_routing
  - active_storage_analysis
  - action_mailbox_incineration
  - low
  - traffic

# config/sidekiq_data.yml
:concurrency: 5
:queues:
  - create_debits_for_campaign_and_date
  - create_debit_for_campaign_and_date
  - recalculate_organizataion_balances
  - ensure_daily_summaries
  - daily_summaries
  - daily_summary
  - ensure_scoped_daily_summaries

Note that we didn't use weights. Instead, we leveraged Sidekiq's default behavior of draining queues in linear order.

Considering the worker domain or category will help you organize queues into groups that can be configured and managed independently. For example, this relatively simple approach ensured that ETL processes never blocked or slowed down real time operations.

Importing Third Party Data

Most companies I've worked with need to import and standardize third-party data. How that data is obtained, how often it changes, and what's required to standardize can vary dramatically.

Third party data providers often impose restrictions like rate-limiting to prevent problems within their own systems. They typically require a means of authentication to enforce such controls. Sometimes upstream data is volatile and requires continuous importing.

I've worked on applications where several of our customers shared credentials for accessing third party data with other customers. Sharing credentials can prove challenging, given data access limits, and is further compounded if customers have different requirements for data freshness and accuracy.

Standardizing and augmenting this data can be expensive. It can also trigger large quantities of adjacent work. I've seen small upstream changes that cascaded into millions of jobs for a single customer, which flooded the queue and blocked other critical work.

How can we solve these problems?

I tend to use similar strategies whenever I encounter requirements like this. I lean on Sidekiq Pro and Enterprise features, use great libraries like AcidicJob, and apply my own Rails-fu when needed.

Let's tackle each problem in turn.

Honor Third Party Constraints

Begin with the expectation that some customers will share credentials for accessing third-party data and that rate limits exist for accessing that data. We'll need to throttle requests for all customers sharing the same third party credentials.

Let's create an ActiveRecord model to manage credentials and include the ability to throttle activity. We'll use Sidekiq Enterprise's bucket-based rate limiting to accomplish this.

# app/models/credential.rb
class Credential < ApplicationRecord
  has_many :customers

  # Throttles the passed block for this credential
  def throttle(count = 60, interval = :minute, options={}, &block)
    options = options.reverse_merge(
      wait_timeout: 5, 
      lock_timeout: 60
    )
    Sidekiq::Limiter
      .bucket(to_gid_param, count, interval, options)
      .within_limit(&block)
  end
end

# app/jobs/fetch_data_job.rb
class FetchDataJob < ActiveJob

  def perform(customer)
    customer.credential.throttle do
      # 1. fetch third-party data
    end
    # 2. queue adjacent work
  end
end

customer = Customer.find(1)
FetchDataJob.perform_later customer

This ensures that we don't DOS third parties, even when our customers share credentials and request data at the same time.

Prevent Customers from Consuming all Resources

Remember that importing data triggers a cascade of adjacent work. We need to ensure that a single customer doesn't flood the queue(s), consume all system resources, and block other important work.

We can manage this with clever use of Sidekiq queue weights. First, let's update our Sidekiq configuration and set up partitions for the default and low queues.

# config/sidekiq.yml
:concurrency: 8
:queues:
  - [critical, 6]
  - [high, 4]
  - [default, 2]
  - [default_0, 2]
  - [default_1, 2]
  - [default_2, 2]
  - [default_3, 2]
  - [low, 1]
  - [low_0, 1]
  - [low_1, 1]
  - [low_2, 1]
  - [low_3, 1]

Note that the partitions for default and low queues share the same weights. This means that jobs in these queues will have an equal chance of being dequeued when multiple customers are queueing millions of jobs in parallel. A single customer won't block others.

So how do we get jobs into a partitioned queue?

Let's create a helper method to obtain a random partitioned queue name.

# config/initializers/sidekiq.rb
def Sidekiq.partitioned_queue_name(name = "default", size: 4)
  "#{name}_#{rand size}"
end

Now let's look at how to best use these partitioned queues. We'll revisit our FetchDataJob and create a few others to handle the adjacent work.

# app/jobs/fetch_data_job.rb
class FetchDataJob < ActiveJob
  def perform(customer)
    source_documents = []

    customer.credential.throttle do
      # fetch third-party data and assign source_documents
    end

    source_documents.each do |doc|
      StandardizeDocumentJob
        .set(
          queue: Sidekiq.partitioned_queue_name("default"),
          wait: (0..60).to_a.sample.seconds
        )
        .perform_later(document, url)
    end
  end
end

Note that we're queueing up adjacent work into StandardizeDocumentJob.

# app/jobs/standardize_document_job.rb
class StandardizeDocumentJob < ActiveJob
  def perform(source_document)
    document = Document.new
    # standardize the source document's data
    document.save

    images = []
    # scan document for linked images

    images.each do |url|
      ImportImageJob
        .set(
          queue: Sidekiq.partitioned_queue_name("low"),
          wait: (0..60).to_a.sample.seconds
        )
        .perform_later(document, url)
    end
  end
end

The work then cascades and fans out to ImportImageJob.

# app/jobs/import_image_job.rb
class ImportImageJob < ActiveJob
  def perform(document, url)
    # save url to cloud provider
    # update document
  end
end

A few things to note:

We only throttle the request to fetch third party data
After the initial import, the work cascades exponentially
Partitioned jobs have equal priority if enqueued at around the same time

That's a lot to digest, but the primary takeaway is to be thoughtful when architecting background workers.

Your solution can be as simple or as elaborate as needed. And we've barely scratched the surface of what's possible here. What if we configured our worker infrastructure to run the low queue on a large cluster of servers. How might that affect throughput? I've partitioned worker pools similar to PostgreSQL's Hash based table partitioning to help prevent individual customers or processes from blocking others.
What about other infrastructure changes?

Next up, we'll learn about configuring Rails connections.

Distributed Systems with Rails

Hopsoft — Thu, 10 Mar 2022 20:57:15 +0000

When most people think of distributed systems, domains like microservices and blockchains or some variant of orchestrated complexity come to mind.

But let's take a moment to consider Wikipedia's definition.

A distributed system is a system whose components are located on different networked computers, which communicate and coordinate their actions by passing messages to one another from any system. The components interact with one another in order to achieve a common goal.

Three significant characteristics of distributed systems are:
  1. concurrency of components
  2. lack of a global clock
  3. independent failure of components

If we stretch this definition a bit, we'll discover that Ruby on Rails supports creating distributed systems. In fact, most full-stack web frameworks do.

Consider that you're already running multiple services, often on separate hardware, things like databases and caches. Now examine purpose-driven deployments. A server dedicated to standard web requests and another for API requests. Or think of the third-party services that web applications rely on for things like logging, email, search, analytics, image processing, etc... Such services are endless. Just look at what's offered in Heroku's marketplace.

While it may not be self-evident, thinking of Rails as a distributed system platform will help frame the mental model for the ensuing discussion.

I want to focus on a particular feature in Rails that helps accomplish some distributed system characteristics, ActiveJob. ActiveJob provides background job processing for Rails.

The principal component in the Rails toolbelt for managing background jobs is the backplane or data-bus which allows us to store metadata about expensive operations that should move off the critical path and be processed in the background. These days, Redis is generally used as this backplane for serious Rails applications. It's a multipurpose in-memory data store that serves several functions: a cache, message broker, and database.

ActiveJob uses Redis as a message broker by saving or queueing metadata about operations to run in the background. It also dequeues that metadata and runs those operations. Sidekiq is another popular library that serves the same purpose. Sidekiq implements the ActiveJob interface and includes several advanced features not available in ActiveJob.

It's common for Rails applications to run ActiveJob background workers on separate hardware, but sophisticated deployments might even run isolated worker pools on different machines for different types of background work. It's also viable to run multiple instances of Redis to further separate different types of background work.

The possibilities are endless, and Rails supports all of them. I hope it's becoming clear that purpose-driven deployments can comprise a distributed system even when all the code is contained inside a monolith. For our purposes, it might help to think of distributed systems in terms of units of deployment instead of isolated units of execution.

Now, let's consider some scenarios where segregating different types of background work makes sense.

Limiting resource contention on a centralized database
Constraining requests to third party API endpoints
Partitioning customers to ensure a single user can't exhaust system resources and cause problems for everyone
Isolating ETL processes to prevent disrupting production

All of these scenarios can be managed by a single monolithic Rails codebase. Why might this be a good idea? Team productivity is exceptionally high in a well-designed Rails monolith. Even if the code is more tightly coupled, it's a tradeoff worth considering as you'll get several benefits of
"distributed systems" without all the downsides.

Stay tuned for the next post covering how to structure a queueing system to manage all that background work.

Rails, Hotwire, CableReady, and StimulusReflex are BFFs

Hopsoft — Mon, 08 Nov 2021 16:27:46 +0000

Enforcing strict RESTful routes and controllers is perhaps the most impactful technique that influenced my usage of Ruby on Rails for the better. I cannot overstate how much I love traditional REST semantics and encourage their usage on every team that I have influence over.

Having said that, I also think rigidly applying this pattern to smaller and smaller use cases has diminishing returns. One example of a smaller use case is TurboFrames. TurboFrames are great and I use them along with their attendant REST semantics, but I try to be very thoughtful about how far I take this approach.

For example, libs like CableReady and Futurism can lazy load partials so unobtrusively that the notion of adhering to the formality of REST, with the required new routes, controllers, etc..., would be far too much ceremony for such small use cases.

One of the original goals of CableReady and StimulusReflex was to work seamlessly with traditional HTTP server rendered Rails apps (pre Hotwire) without requiring significant architectural changes or forcing a proliferation of new routes, controllers, or views/partials etc... We basically wanted a way to gradually introduce robust real-time and reactive behavior into traditional Rails apps with as little friction as possible. The idea being to allow people to leverage the work that had already been done rather than forcing a rethinking of the app. I view CableReady/StimulusReflex as: REST + RPC sprinkles + async server triggered DOM behavior.

Hotwire, while very cool, introduces new concepts that impose a higher cognitive cost and forces you to rethink how to best structure a Rails app. I view Hotwire as: REST semantics for everything + async server triggered CRUD updates.

There are pros and cons to each approach. Hotwire has more obvious and strict conventions, while CableReady and StimulusReflex adhere more to Ruby's philosophy of flexibility and expressiveness.

For me, using both Hotwire and CableReady + StimulusReflex techniques together is like "having my cake and eating it too." Admitedly, this is a power move and requires some experience to know when to apply each approach.

FYI - There are some great conversations on the StimulusReflex Discord server about this stuff. We'd love it if you joined us.

Also, I should note how much I dislike the umbrella marketing term "Hotwire" as it forces a false dichotomy in this conversation. Both CableReady and StimulusReflex are designed to work well with Hotwire libs and even have hard dependencies on some of them.

Build a real-time Twitter clone in 10 mins with Rails and StimulusReflex

Hopsoft — Wed, 29 Apr 2020 14:07:20 +0000

Demo

Learn how to build interactive real-time applications with Ruby on Rails, CableReady, and StimulusReflex. We'll create a Twitter style timeline that broadcasts updates to all users in real-time ...and we'll build the entire application in less than 10 minutes.

Conceptual Compression

These tools and techniques take conceptual compression to a new level by helping developers of all experience levels build real-time reactive applications faster than ever. We've been able to accomplish this with a sustained focus on developer happiness and productivity that has reduced much of the complexity associated with "modern" web development.

We use these tools at CodeFund to build some of the most sophisticated parts of our application.

The best part is that we're open source, so you can see exactly how we do it. For example, here's the backend reflex code used to build the form shown in the video. https://github.com/gitcoinco/code_fund_ads/blob/master/app/reflexes/campaign_bundles_reflex.rb

Keep an eye out for an in depth write up on the CodeFund blog where we'll dig into more detail. Coming soon...

The Ideal Bootstrapped Business

Hopsoft — Mon, 03 Feb 2020 14:33:11 +0000

I recently watched Jason Cohen's excellent presentation from MicroConf in 2013. Jason is a successful entrepreneur and has founded several companies including WP Engine and Smart Bear. His presentation was chock full of great information.

https://vimeo.com/74338272

Here are my notes. Hopefully you'll find them useful.

Most companies don't work. They either fail to build a product that people want, or they build something that doesn't embrace the constraints of a small company.

The Self-Funded Startup

A self-funded startup is a cash machine with predictable acquisition of recurring revenue that offers annual prepay in a good market.

Revenue Models

🚫 One Offs - Every month starts at zero and requires new sales efforts... the opposite of a cash machine.
🚫 Picking Up Pennies - Skimming a percentage from other financial activity i.e. donations, transactions, etc... an inefficient cash machine.
✅ Recurring Revenue - The only way to build a cash machine.

Getting 1,000 customers at any price point is really hard. 150 customers is more realistic to start.

150 Customers

50 from scratching and clawing
25 from social media marketing
75 from standard marketing

Average Revenue per User (ARPU) is the most important metric that you can track while you're small. Set a goal to hit $10,000 in monthly recurring revenue with 150 customers. That means your pricing should average $66/mo per customer.

Founders are hesitant to charge enough because they know their product is shitty in the beginning. Find ways to justify a higher price. Also, embrace the fact that you're running a boutique business. People are willing to pay more to help small companies they believe in.

Pricing Tactics

Tiers - monthly/quarterly/yearly
Specials - higher monthly price with various coupons and discounts
Annual Prepay - will solve cash flow problems

If just 25% of your customers elect annual prepay, it will yield over 3x the cash flow that monthly plans provide. Here are some tactics to convince more customers to opt for annual prepay.
- Annual Only Coupons - 3 months off
- High Monthly + 1/2 off Annual - raise monthly prices & offer steep annual discounts
- Business Tier - name the most expensive plan "Business"... some will opt for this simply because they want to spend the most

Revenue Tactics

Money Back Guarantee - will produce more revenue than free trials.
Premium Support - paid offer to move a customer's support tickets to a priority position in the queue. Doesn't require additional labor but is very much a value add for the customer.

Market Models

🚫 Consumer (B2C) - Extremely low price and very high maintenance.
🚫 Point-in-Time - Pain points are temporary and not recurring or sustainable. Weddings, events, etc...
🚫 Viral - Very low probability of actual virality or revenue.
🚫 Marketplace - Difficult to build. Requires creating two different businesses and balancing their growth.
✅ Naturally Recurring - Anything tied to regular cycles or underlying volatility. Hosting costs, marketing tools, invoicing, reporting, SEO, HR/compliance, etc...
✅ Naturally Latent Anything without real-time requirements. Decision support (analytics & reports), content, project management, etc...
✅ Completable - Smaller product scope mitigates feature creep and allow you to focus on customers without racing to add every competitor's features.
✅ After Market - Add ons to an existing product or service.

Choose a big market. Product validation already exists and customers are easier to find. There's plenty of money and room for competition. Servicing a niche in a big market provides the flexibility to pivot or expand as needed.

Acquisition Models

🚫 Social Media - Very expensive (time and money) and doesn't produce predictable recurring revenue.
✅ Paid Advertising - Very predictable and repeatable.

Rules of thumb for acquisition costs.

Low bar is a $300 spend for $50/mo in revenue.
Basic CPC (Cost per Click) Formula: CPC = Monthly ARPU / 25

Your formulas will become more sophisticated as you experiment and gather data.

End Game

Successful companies continue to grow. They don't stop. You need a plan.

Sell and take your exit
Deliberately limit growth (raise prices, waiting list, etc... will change your clientele)
Raise money to fuel accelerated growth

Ruby is designed for humans

Hopsoft — Sun, 19 Jan 2020 15:53:56 +0000

I just finished reading this great interview with Matz. It helps build an understanding of his thought process behind the design of the Ruby programming language. Here's a short excerpt.

People don’t understand the nature of designing a language. Language is kind of a scaffolding of the mind, a way of structuring your thinking. It’s same to human languages, like Russian, Japanese and English.

Programming languages such as Ruby, Python, JavaScript, and so on, help to develop the mind, to allow you to turn ideas into something tangible and useful. That’s the primary purpose of a programming language.

I think a programming language should have a philosophy of helping our thinking, and so Ruby’s focus is on productivity and the joy of programming. Other programming languages, for example, focus instead on simplicity, performance, or something like that. Each programming language has a different philosophy and design. If you feel comfortable with Ruby’s philosophy, that means Ruby is your language.

-Matz

This is the philosophical foundation that I attempted to communicate in my post, Why I still love Ruby. Matz simplifies the message even further. ❤️

Why I still love Ruby

Hopsoft — Mon, 13 Jan 2020 14:58:08 +0000

Language and Cognition

Philosophers and linguists have long asked the question, Does the language we speak shape the way we think?

We now have academic research to help answer this fascinating question. The data seems to indicate that language does indeed affect the way we think. It can even impact our ability to perform.

Lera Boroditsky, Ph.D has a wonderful TED talk that introduces linguistic and cognitive studies to the masses.

She demonstrates that language influences a tremendous amount of the way we perceive and interact with the world. Everything from our sense of direction to our understanding of time and distance. It touches our ability to count or quantify, our awareness of colors, our perception of intent, credit and blame, even our concepts of gender.

Our language changes what we pay attention to and notice. What we place emphasis on. What our biases are. It impacts our capacity and speed at which we can accomplish various tasks. It can even change the physical characteristics of our brain.

Language acts as a stepping stone into entirely new cognitive realms and can lead to big differences in creative and intellectual ability.

The Ruby Language

Yukihiro Matsumoto (Matz) has indicated that Ruby was inspired and influenced by several other programming languages. Lisp, Smalltalk, Perl, Eiffel, Ada, Basic, and others. More recently, Rust, Go, and Elixir.

I knew many languages before I created Ruby, but I was never fully satisfied with them. They were uglier, tougher, more complex, or more simple than I expected.

Throughout the development of the Ruby language, I’ve focused my energies on making programming faster and easier.

Ruby is designed to make programmers happy.

-Matz

Why I still love Ruby

Ruby has a long history of assimilating the best ideas, patterns, and practices from other programming languages. In many ways, it is the convergence of the most exceptional features from other languages.

Admittedly this can feel messy or chaotic at times, but it delivers the widest range of options to the programmer. Ruby's culture of embracing and including the best from other languages dramatically increases the solution space and directly impacts our capacity to think and perform as developers.

Does this mean we should ignore other languages and trust the MRI team will always get it right? Of course not. Learning another language will improve your ability to work with Ruby... or any language for that matter.

To have a second language is to have a second soul.

-Charlemagne

Having said that, Ruby is the most malleable and flexible language I've ever worked with. It frees my mind to explore solutions that I may not have considered or even been aware of in another language.

A short note on Rails

I should point out that Rails wouldn't exist without Ruby. DHH appropriated Ruby's culture of inclusivity and created a framework that is a delightful melting pot of tools, patterns, and techniques. All made possible by Ruby. Language choice may also help explain why no other web framework has materialized that shares the same eminence and stamina.

This is why I still love Ruby, even after 10 years of working with it.

Build a server updated async progress bar with Rails in 5 steps

Hopsoft — Tue, 07 Jan 2020 15:18:03 +0000

This tutorial demonstrates how simple it is to perform DOM updates from Rails background jobs with CableReady.

Intro

Ruby on Rails supports websockets out of the box via a built in library known as ActionCable. I created a library named CableReady that works with ActionCable to perform common DOM operations from background jobs without requiring you to write any custom JavaScript. And, it's very performant.

1. Create the Rails project

rails new progress_bar_demo
cd progress_bar_demo

2. Create the restful resource

First create the controller and HTML page.

bundle exec rails generate controller progress_bars
touch app/views/progress_bars/show.html.erb

<!-- app/views/progress_bars/show.html.erb -->
<h1>Progress Bar Demo</h1>
<div id="progress-bar">
  <div></div>
</div>

Then update the routes file.

# config/routes.rb
Rails.application.routes.draw do
  resource :progress_bar, only: [:show]
  root "progress_bars#show"
end

3. Setup the styling

First create the stylesheet.

mkdir app/javascript/stylesheets
touch app/javascript/stylesheets/application.scss

// app/javascript/stylesheets/application.scss
#progress-bar {
  background-color: #ccc;
  border-radius: 13px;
  padding: 3px;
}

#progress-bar>div {
  background-color: green;
  width: 0;
  height: 20px;
  border-radius: 10px;
}

Then update the JavaScript pack to include the stylesheet.

// app/javascript/packs/application.js
require("@rails/ujs").start()
require("turbolinks").start()
require("@rails/activestorage").start()
require("channels")

import "../stylesheets/application.scss" // <-- add this line

Finally, update the application layout to use the stylesheet pack.

<!-- app/views/layouts/application.html.erb -->
<!DOCTYPE html>
<html>
  <head>
    <title>ProgressBarDemo</title>
    <%= csrf_meta_tags %>
    <%= csp_meta_tag %>

    <!-- line below was updated to use stylesheet_pack_tag -->
    <%= stylesheet_pack_tag 'application', media: 'all', 'data-turbolinks-track': 'reload' %>
    <%= javascript_pack_tag 'application', 'data-turbolinks-track': 'reload' %>
  </head>

  <body>
    <%= yield %>
  </body>
</html>

4. Setup the ActionCable channel

yarn add cable_ready
bundle exec rails generate channel progress_bar

// app/javascript/channels/progress_bar_channel.js
import consumer from "./consumer"
import CableReady from 'cable_ready'

consumer.subscriptions.create("ProgressBarChannel", {
  received: data => {
    if (data.cableReady) CableReady.perform(data.operations)
  }
})

# app/channels/progress_bar_channel.rb
class ProgressBarChannel < ApplicationCable::Channel
  def subscribed
    stream_from "ProgressBarChannel"
  end
end

5. Setup the backend

bundle add cable_ready
bundle exec rails generate job progress_bar

When this job fires, it runs a loop that fills in the progress bar a little bit on each iteration. This is possible because CableReady allows us to send commands to the browser that update the DOM without the need to write custom Javascript.

# app/jobs/progress_bar_job.rb
class ProgressBarJob < ApplicationJob
  include CableReady::Broadcaster
  queue_as :default

  def perform
    status = 0
    while status < 100
      status += 10
      cable_ready["ProgressBarChannel"].set_attribute(
        selector: "#progress-bar>div",
        name: "style",
        value: "width:#{status}%"
      )
      cable_ready.broadcast
      sleep 1 # fake some latency
    end
  end
end

# app/controllers/progress_bars_controller.rb
class ProgressBarsController < ApplicationController
  def show
    ProgressBarJob.set(wait: 1.second).perform_later
  end
end

6. Run and watch the magic

bundle exec rails s

Then visit http://localhost:3000 in a browser.

Disclaimer

⚠️ This demo is tailored for the development environment. In a production setup you'd need to configure both ActionCable and ActiveJob to use Redis. You'd also want to secure the ActionCable channel.