DEV Community: NejŘemeslníci

How to detect classes contained in ruby gems in Tailwind 4

Matouš Borák — Wed, 05 Mar 2025 21:40:06 +0000

Our main web app uses Tailwind CSS and we are happy to have recently migrated it to Tailwind version 4. Among the nice features that Tailwind 4 brings to the table is an automatic detection of CSS classes in the application source files. This allowed us to completely remove the content section from the former Tailwind v3 configuration and just forget about it.

For a moment only, that is.

As it quickly turned out, we forgot about the fact that not all CSS classes are used within our application itself. Our app includes an internal gem that implements several Flowbite components and some of the Tailwind classes it contains are unique to that gem. As the gem sources reside outside the app root directory, the Tailwind program cannot see them and doesn’t scan them for potential classes.

How we did it in Tailwind 3

In Tailwind v3 we actually had the following code in the tailwind.config.js configuration file, handling this issue:

// tailwind.config.js

const execSync = require('child_process').execSync;

function getGemPath(gem) {
  return execSync(`bundle show ${gem}`).toString().trim()
}

module.exports = {
  content: [  
    ...,
    `${getGemPath('flowbite')}/app/components/**/*.{slim,rb}`
  ]
}

The config defined a function which called the bundle show command to get the absolute path to the gem source code folder. It then combined it with glob patterns to the actual source code files. This solution was probably inspired from here and just worked for us for several years.

But how to do it in Tailwind 4?

In Tailwind 4, things have changed dramatically. Using a JS configuration is still possible but not really recommended and we did not want to keep that tech debt in our project.

The official way now is to configure Tailwind via a CSS file. In a CSS file though, there is no way to run code dynamically, everything has to be pre-defined and static. So how can we reference gem source files when each environment that our app runs on stores them under a different path?

We came up with a solution that we are happy with using two small tricks: 1) we made the gem path static and 2) we made it accessible relative to the main app.

As it turns out, Bundler, has the concept of plugins and one of them does precisely what we need: the bundler-symlink plugin provides a post-install hook that adds symlinks to all gems of the main application under its local directory. Specifically, under the .bundle/gems/ directory of your project. This is great because all the different absolute gem paths are now accessible from a known place within the main application.

To ensure the gem is available in your setup, you can add it to the Gemfile like this:

# Gemfile

plugin "bundler-symlink"

Then, whenever bundle install is run, the plugin adds / updates a set of symlinks to the gems:

$ bundle install

Fetching gem metadata from https://rubygems.org/..
Resolving dependencies...
Fetching bundler-symlink 0.4.0
Installing bundler-symlink 0.4.0
Installed plugin bundler-symlink
Symlinking bundled gems into /home/matous/projekty/nejremeslnici/web/.bundle/gems
Bundle complete! ...


$ ls -l .bundle/gems

lrwxrwxrwx 1 matous users 51 Mar  5 21:32 actioncable-8.0.1 -> /home/matous/.gem/ruby/3.4.1/gems/actioncable-8.0.1/
lrwxrwxrwx 1 matous users 53 Mar  5 21:32 actionmailbox-8.0.1 -> /home/matous/.gem/ruby/3.4.1/gems/actionmailbox-8.0.1/
lrwxrwxrwx 1 matous users 52 Mar  5 21:32 actionmailer-8.0.1 -> /home/matous/.gem/ruby/3.4.1/gems/actionmailer-8.0.1/
lrwxrwxrwx 1 matous users 50 Mar  5 21:32 actionpack-8.0.1 -> /home/matous/.gem/ruby/3.4.1/gems/actionpack-8.0.1/
lrwxrwxrwx 1 matous users 50 Mar  5 21:32 actiontext-8.0.1 -> /home/matous/.gem/ruby/3.4.1/gems/actiontext-8.0.1/
lrwxrwxrwx 1 matous users 50 Mar  5 21:32 actionview-8.0.1 -> /home/matous/.gem/ruby/3.4.1/gems/actionview-8.0.1/
...

Now, you can easily add a relative gem path to the Tailwind CSS configuration file using the @source directive (the path is relative to the location of the CSS file):

/* app/assets/tailwind/application.css */

@import "tailwindcss";

@source "../../../.bundle/gems/flowbite-components-517d2087e439/app/components/**/*.{slim,rb}";

And voila, all styles work again!

There is still one issue though: the bundler plugin creates symlinks that have the gem versions in their names so this setup would break again whenever we upgraded our Flowbite components gem. This does not seem like a viable solution, long-term.

We briefly tried to use a glob in the gem path in the Tailwind configuration (as in @source ".../.bundle/gems/flowbite-*/...") but this did not work, probably due to a limitation of the Tailwind scanner regarding symlinks.

Using the ”bare symlinks“ plugin

We ended up cloning the bundler plugin and amending its source to create symlinks that were not versioned instead. We released the new plugin under the bundler-bare_symlink name. It is exactly the same as the original ¹ except for one thing: it uses the bare gem name instead of the (versioned) gem directory name in the symlink name.

So, to use it, we first uninstalled the original plugin:

$ bundle plugin uninstall bundler-symlink

Then put our new one in the Gemfile:

# Gemfile

plugin "bundler-bare_symlink"

And after running bundle install again, we got the following symlink structure:

$ ls -l .bundle/gems

lrwxrwxrwx 1 matous users 51 Mar  5 22:09 actioncable -> /home/matous/.gem/ruby/3.4.1/gems/actioncable-8.0.1/
lrwxrwxrwx 1 matous users 53 Mar  5 22:09 actionmailbox -> /home/matous/.gem/ruby/3.4.1/gems/actionmailbox-8.0.1/
lrwxrwxrwx 1 matous users 52 Mar  5 22:09 actionmailer -> /home/matous/.gem/ruby/3.4.1/gems/actionmailer-8.0.1/
lrwxrwxrwx 1 matous users 50 Mar  5 22:09 actionpack -> /home/matous/.gem/ruby/3.4.1/gems/actionpack-8.0.1/
lrwxrwxrwx 1 matous users 50 Mar  5 22:09 actiontext -> /home/matous/.gem/ruby/3.4.1/gems/actiontext-8.0.1/
lrwxrwxrwx 1 matous users 50 Mar  5 22:09 actionview -> /home/matous/.gem/ruby/3.4.1/gems/actionview-8.0.1/
...

And finally, we were able to use a fully relative and static path in the Tailwind configuration file:

/* app/assets/tailwind/application.css */

@import "tailwindcss";

@source "../../../.bundle/gems/flowbite/app/components/**/*.{slim,rb}";

Conclusion

This setup works for us on all developer machines as well as on the servers. The fact that the plugin uses a bundle install hook makes it very convenient and we did not have to explicitly install or configure it anywhere. And, most importantly, all of our web styles work again!

Would you like to read more stuff like this? Follow us on Bluesky.

It even keeps the original license, the WTFPL, of course, haha. ↩

Speed up Kamal deploys in GitHub Actions

Matouš Borák — Wed, 13 Nov 2024 12:09:29 +0000

Have you got your application successfully deployed using Kamal and GitHub Actions but the deploy time still seems a bit too much? Depending on what changes you are trying to push to the server, ”a few minute“ deploys should be perfectly possible with GitHub Actions, even shorter ones:

Let’s take a look at some ways you can tweak your workflow to speed things up. This post was inspired by the nice conversations with Adrien on BlueSky and I also wanted to share our experience with optimizing our own GitHub Actions deploy flow that we went through recently. Part of this post will be Ruby on Rails–specific, the rest should be rather universal.

Deploying with Kamal involves building a Docker image, a process with its own set of intricate rules. There are many best practices around optimizing the Dockerfile itself and/or the building process but as they are not Kamal nor GitHub Actions–specific, we won’t cover them here. Besides, the default Ruby on Rails Dockerfile is already pretty optimized. So what other options do we have?

Cash everything you can

While I never recommend employing caching as the first step of optimizing things, Docker is an exception. The Docker image format is well suited for caching, the cached layers are immutable and their invalidation strategy seems simple and robust. Therefore, we really want to cache the builds right after we get our deployment workflow working.

Caching Docker build images

To cache build image layers, we need to tweak two things, one of them is the Kamal config:

# config/deploy.yml
builder:
  cache:
    type: gha
    options: mode=max
    image: kamal-app-build-cache

Since GitHub offers a cache storage back-end supported by Docker, we use the gha cache type so that the cache storage is as close to our runners as possible. The mode=max option instructs Docker to cache even the intermediate build layers, not only those exported to the final image. And we also give our build image some (arbitrary) name.

But in the context of GitHub Actions, this is not yet sufficient for caching to work. If you looked at the Actions ⟶ Caches tab in your GitHub repository, it would still be empty. How come?

By default, Kamal uses the docker-container driver to build images which, in turn, uses the BuildKit toolkit internally. While Kamal sets up registry caching correctly, caching still fails in the end because the BuildKit process is isolated from our GitHub Action runtime process. To connect the two, we need to expose the GitHub runtime to the workflow. Luckily, there is a GitHub Action ready just for this so all that is needed is adding the action to the workflow file. We put it right after setting up Docker Buildx:

# .github/workflow/deploy.yml
jobs:
  deploy:
    steps:
      ...
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Expose GitHub Runtime for cache
        uses: crazy-max/ghaction-github-runtime@v3

If you run the build action again, you should start seeing "buildkit" entries in the Actions tab ⟶ Caches in your GitHub repository:

More importantly, your builds should now be running about twice as fast!

Caching application dependencies

There is one more thing that we can cache – application dependencies, i.e. the libraries it uses, etc. GitHub Actions support caching the assets of various package managers. For ruby, caching the bundled gems is configured in the workflow file with a single bundler-cache: true option like this:

# .github/workflow/deploy.yml
jobs:
  deploy:
    steps:
      ...
      - name: Set up Ruby
        uses: ruby/setup-ruby@v1
        with:
          bundler-cache: true

Respect the build architecture

While Docker allows cross–platform builds quite easily, they tend to be much slower than when building an image on the same platform as is the target server. Especially those builds that involve a lot of computational work, such as when building ruby gems with native extensions.

Most of the standard GitHub Action runners are based on the x64 architecture so if your target server is ARM-based, your options are currently quite limited. You can:

switch to one of the larger Linux arm64–based runners but these are billed separately and are not free even for public repositories,
self-host a GitHub Action runner on a server of your choice,
migrate your application to an x64–architecture server,
switch to a completely different CI service which provides ARM-based builds (out of scope for this post),
or, of course, you can bite the bullet and reconcile to waiting for cross–platform builds.

Since 2024, there actually are some standard ARM–based runners available in GitHub Actions, namely macos-14, macos-15, and macos-latest that run on the Apple silicon (M1) chip. Unfortunately, this setup has its own limitations and quirks, most importantly, the M1 chip does not support nested virtualization. Since the GitHub Action runner itself is virtualized and Docker needs that too, there is effectively no way to run Docker on M1–chip runners. Until GitHub releases hosted runners based on the newest M3 Apple silicon chips (which allegedly do support nested virtualization), there won’t be a feasible way to deploy to ARM-based servers natively using MacOS runners. In other words, until then, standard GitHub Action runners are best suited for deploying to x64 architecture servers only.

Don’t build multi-platform images unless you really need them

Unless you have multiple target servers that are mixed in their architectures, do not build multi-platform images. Building images for multiple architectures is inherently slow and there isn’t much you can do about it.

So, be sure you have a single arch specified in the builder section of Kamal config. For reasons mentioned above, preferably amd64:

# config/deploy.yml
builder:
  arch: amd64

Parallelize building native gems (misc tweak)

We have also tested whether Bundler is able to determine the number of processors in a GitHub Action run so that it can parallelize gem downloads and builds (see its --jobs option) and yes, it works great without having to configure anything.

This is not the case though when Bundler builds gems with native extensions. Usually, it calls make to compile the C extensions but these calls are not parallelized out of the box. So if you build a lot of gems with native extensions, you might want to define an environment variable that instructs make to run multiple compile jobs in parallel:

MAKE="make -j4"

where 4 is the number of processors available in your runner or even slightly more, you can experiment with that.

Watch your runners performance

There is a new feature that has been released by GitHub just a few days ago: a place to monitor the performance statistics of your GitHub Actions. To view them, go to Insights ⟶ Actions Performance Metrics in your repository and you will see something like the following:

Just keep on mind that the numbers here are average numbers. In the case of deployment workflows the run time usually differs quite a lot based on whether gems need to be re-bundled, assets recompiled or not.

Conclusion

GitHub Actions are a specific CI/CD environment and it is quite easy to unknowingly create a Kamal deployment workflow that will under-perform in it. We shared a few tips that could help leveling up the runners speed so that we don’t have to wait for our deploys for longer than necessary.

Want more stuff like this? Follow us here or at BlueSky 🦋.

Strict locals in Slim / Haml partials in Rails

Matouš Borák — Thu, 22 Feb 2024 17:13:09 +0000

When I saw the announcement about partial template strict locals in Rails 7.1, I was excited but it took me a long time to realize that this is not an ERB-only feature but that it should actually work in any template language, including my favorite – Slim! I even remember trying it back then but failing, probably due to an incorrect syntax. While there are many nice tutorials explaining template strict locals out there, they are all ERB-centric so let’s take a look how this feature can be used in Slim or Haml templates, respectively.

Preparations

Suppose we want to render a partial template from a main one one, to show an important message:

/ main.html.slim

= render "notice"

/ _notice.html.slim

h1
  ' Note:
  = message

If we run this code, it will fail with an ugly and misleading error message:

undefined local variable or method message for an instance of #<...>
Hint: message is probably misspelled.

saying that we probably misspelled message which we did not - we forgot to pass the local variable in the first place.

Adding a magic comment

Let’s make the partial template recognize our local variable using a strict locals magic comment:

/ _notice.html.slim

/# locals: (message:)

h1
  ' Note:
  = message

Note that the exact syntax is important here: it must be a comment (denoted by the slash / character in Slim), then a hash (#), a space, then locals:, another space and finally something that resembles a method arguments definition with keyword arguments specifying the needed local variables. The reason this syntax is so strict is that it gets matched by a regular expression deep in the Action View templates processing code.

In case of a HAML template, the syntax differs only by the magic comment leading character, it must be -# instead of /:

/ _notice.html.haml

-# locals: (message:)

%h1 Note!
  = message

Note that both magic comments are code comments, i.e. they are not output to the final HTML.

Once we have this magic comment in place, we get a much better error:

missing local: :message

and the stack trace leads us to the proper place (to the line with render in the main template) right away.

And of course, all other strict locals goodies work, too, such as the default values or prohibiting any locals.

Closing thoughts

Up till now, we commonly used the local_assigns hash to notify the reader that our variable is to be passed from the outside and to set its default value. We think that strict locals make this hash a little obsolete as they serve very similar purposes with a nicer syntax.

Although we still prefer View Components for our most sophisticated view template blocks, we will definitely use Rails partials with more pleasure since we can be sure now to write them in a safer way. 👍🏻

How to use View Transitions in Hotwire Turbo

Matouš Borák — Thu, 16 Feb 2023 09:55:37 +0000

Have you heard the news? Google Chrome 111 will be released on March 1st 2023 and – among other things – will ship a feature I was eager to try since I first heard about it: View Transitions. This is a new API proposed by Google at W3C, currently as a 1st Working Draft. An informal summary for it can be found in this Explainer which seems to be a more up-to-date version of the original introductory article published at the Chrome Developers site last year. I highly recommend reading either of the documents. Update: there is now some MDN documentation available, too!

So what are View Transitions good for? In short, they allow adding animated page transitions. Although we already have several standard options to animate stuff on web pages (CSS Transitions, CSS Animations or the Web Animations API) and countless more options in particular JavaScript frameworks and libraries (Framer Motion for React, Vue Transitions, Svelte Transitions, Swup, Barba.js or Animate.css to name just a few), the web still lacks a generic, standards-based and easy-to-use solution to animate transitions between pages or during DOM updates. At least that’s what Google engineers say and I tend to agree with them.

In the Hotwire Turbo world specifically, several discussions about integrating transition animations also took place and a few promising approaches emerged, namely the Turn project or the transitions in Bridgetown. There is also a chapter in the Noel Rappin’s Modern Front-End book and an interesting article but overall, frankly, this topic still fells somewhat early-stage and exploratory.

And here comes the View Transitions proposal. Could it change the situation for Hotwire and others? It of course depends on many things, most importantly on the adoption rate by other browser vendors and the community in general and whether the W3C Draft Recommendation track succeeds. But I have a gut feeling that yes, it might change things!

A ”Hello world“ example

Let’s take a look at a simple example. Coincidentally, a bunch of examples for an animated counter have recently been created by Jake Archibald in React, Svelte and Lit, so let’s try building the same in Turbo! I will demonstrate it in a Ruby on Rails app as this is what I’m most accustomed to but it should work the same in any other Turbo-enabled framework.

So, we will create a simple page with a big bold number and a link that will increment this number using a Turbo Frame. First, we’ll build a bare-bones page and then we’ll add a View Transition animation.

Turbo Frame needs to route it’s requests somewhere so let’s add a route first:

# config/routes.rb
Rails.application.routes.draw do
  resources :counter, only: :index
end

The corresponding controller (by the way, a ”singular“ one) can stay basically empty:

# app/controllers/counter_controller.rb
class CounterController < ApplicationController
  def index; end
end

The template renders the <turbo-frame> tag and inside it the link and the counter itself (the Slim template language and Tailwind styling are used here, hopefully the notation is sufficiently self-explaining):

/ app/views/counter/index.html.slim
= turbo_frame_tag(:incrementing_counter)
  - counter = params[:counter].to_i
  = link_to "Increment", counter_index_path(counter: counter + 1),
            class: "block p-2 bg-gray-100 active:bg-gray-200"

  #counter.absolute.w-full.text-8xl.font-bold.text-center
    = counter

And this is how it looks so far: Turbo makes requests and replaces the Frame upon clicking the link and no animations are triggered (as can be seen in the lower-right panel):

Now, let’s add View Transitions to the Turbo Frame. To do this, we need to override the default rendering function for Turbo Frames in the turbo:before-frame-render event handler with a custom one that utilizes View Transitions:

// app/javascript/application.js
addEventListener("turbo:before-frame-render", (event) => {
  if (document.startViewTransition) {
    const originalRender = event.detail.render
    event.detail.render = (currentElement, newElement) => {
      document.startViewTransition(()=> originalRender(currentElement, newElement))
    }
  }
})

The handler code first checks whether View Transitions are supported by the browser and if so, it wraps the original rendering function with the document.startViewTransition function. And that’s it! These couple of lines trigger a default cross-fade animation of the whole page whenever any Turbo Frame renders. Yes, the whole page is indeed animated by default but since most elements on the page are either identical (those outside the Turbo Frame) or replaced by identically looking elements (e.g. the link inside the Frame), in practice only the counter itself feels animated. It looks like this:

Finally, we can customize the transition a bit. To match the examples mentioned above, let’s rotate the counter during transitions. For this we will need to utilize ”named“ View Transitions, explained here in the documentation. Whenever we select a part of the screen with a CSS selector and name it using the view-transition-name attribute, the browser starts to animate the corresponding screen region independently of all other parts. This way, we can animate one or more elements on the page without affecting the rest of the page.

So let’s add the following CSS to the index template (Slim recognizes a css: block that just renders a normal <style> tag):

/ app/views/counter/index.html.slim
/ (anywhere outside the Turbo Frame tag)
css:
  /* (1) */
  #counter {
    view-transition-name: counter;
    contain: layout;
  }

  /* (2) */
  @keyframes rotate-out {
    to {
      transform: rotate(90deg);
    }
  }

  @keyframes rotate-in {
    from {
      transform: rotate(-90deg);
    }
  }

  /* (3) */
  ::view-transition-old(counter) {
    animation-duration: 200ms;
    animation-name: -ua-view-transition-fade-out, rotate-out;
  }
  ::view-transition-new(counter) {
    animation-duration: 200ms;
    animation-name: -ua-view-transition-fade-in, rotate-in;
  }

Let’s break this code down a bit:

The CSS selector #counter matches the counter div and the view-transition-name property names this area of the screen, for the purpose of View Transitions, as counter. This name will be used in the animation declarations below.

The clone property currently must be added here for some reasons internal to the current View Transitions implementation in Chrome and must be set to paint or layout. This restriction is planned to be removed from the specification, though, and in fact I’ve heard that it is not needed in Chrome Canary any more.
The rotation animation keyframes are defined here. Note that while the transition also uses fade-in and fade-out animations, they don’t have to be defined here because the spec requires browsers to implement them natively under the name -ua-view-transition-fade-in/out.
The CSS animations for the counter (the View Transition area named counter) are configured here. The CSS selectors here are some of the pseudo-elements automatically created during the transition. The -old pseudo-element represents a screenshot of the old DOM state that should somehow disappear or ”go away“ from the viewport and the -new pseudo-element represents a live version of the final DOM state that should be brought into sight.

So, overall, this code selects a portion of the page and animates it independently from the rest of the page during Turbo Frames DOM updates. Behind the scenes, the default cross-fade for the rest of the page still also takes place, it just is not visible because all its elements are visually identical. The result looks like this:

A few initial tips & tricks

Does this work for Turbo Drive visits, too?

Sure it does and it’s actually pretty easy! All we have to do is define the same event handler as we did above but attach it to the turbo:before-render event instead. By default we’ll get a cross-fade animation of the whole page during Turbo Drive page visits.

Do not try to ”name“ the Turbo Frame itself

When playing with Turbo Frame View Transitions I first tried to use a custom animation for the whole Turbo Frame element by naming it via the view-transition-name property. For some reason, this does not work and you end up with a very cryptic and misleading error message in the console (yes I did have the contain property in the CSS declaration):

Aborting transition. Element must contain paint or layout for view-transition-name : counter

So, when using custom animations, an element from inside the Frame must be selected and named.

Debugging View Transitions

Since View Transitions are technically just normal CSS animations, they can be inspected with the Animations panel in the Dev Tools. Also, the automatically created pseudo-elements are visible in the Elements tab during the transitions:

Conclusions

I confess I am quite excited about the new View Transitions API. Among the things I particularly like about it are the following:

It is surprisingly easy to plug this inside Hotwire Turbo and you get the default cross-fade transition animation immediately for free (in latest Chrome-like browsers, that is).
Since this is implemented natively in the browser, the animations are highly optimized and performant.
View Transitions should allow (today or in the future) building highly interactive transitions similar to those in Material Design.
There is some initial support for Multi-Page Applications, too, which is great news because we can bring transition animations declared in CSS to our old but gold apps.
It should be possible to use a different animation based on the ”direction“ of the visit (Back/Forward) using the Navigation API (also still experimental and not very well supported, though).

Things I am still concerned about:

Browser support: the Firefox team evaluates it, the Safari team is silent. This will be a log run and making a polyfill is probably too difficult. For web sites where transition animations are critical, this is still a no go.
If you’re not careful enough, the transition feels more fluid but also a little bit slower. The reason for it is that View Transitions start the animations at the moment when both the old and new DOM states are already rendered. This means that the exit animation is delayed until new content is available and until that time, nothing happens. Also, the entry animations for the new state usually delay its appearance a little bit more.

This is not a problem of View Transitions themselves but rather a more generic one. If the exit animation (e.g. a fade out) started immediately after user interaction (e.g. a link click), sometimes the user would have to stare at a blank page until the new page content is grabbed, rendered and run through an entry animation. Still, some kind of support for this scenario (possibly with custom loaders or skeletons) would be nice.
Tailwind support: I think the current Tailwind syntax does not allow targeting the HTML document-connected pseudo-elements so we have to resort to custom CSS (which is not a big problem, actually).
All transitions target the whole page, there is currently no option to make, say, two components (Frames) animate totally independently. An initial proposal for ”scoped transitions“ can be found here.

Overall, I like this feature and wish it matures enough and gets wider support soon!

Styling Simple Form forms with Tailwind

Matouš Borák — Mon, 20 Jun 2022 17:24:30 +0000

During our recent redesign of the admin section of our web, we wanted to set up a system that would allow us to quickly build good looking forms. We like to use the Simple Form gem for our forms, we have been using it for nearly 10 years, and we still appreciate its very succinct but flexible syntax. Also, we style our web with Tailwind and we definitely wanted to leverage it for the new admin pages, too. So, at one point of the redesign effort we had to deal with the ”How do we style our new admin forms with Tailwind?“ task and we wanted to do it in the most reusable way possible and without losing Simple Form's flexibility.

In our previous post we ran through the basics of Rails forms rendering and how custom Rails form builders work. In this post, we will build on that remembering that Simple Form is basically just a custom Rails form builder. We will try to briefly describe what components it uses to render the forms and how to amend them to enable flexible Tailwind styling.

Does Simple Form support Tailwind?

Well, yes and no. Simple Form has no Tailwind-specific configuration baked in currently – the details are discussed in this GitHub issue. However, Simple Form is styling system-agnostic so nothing should stop us if we sprinkled the Simple Form configuration with our set of Tailwind classes for each of the form components. (A ”component“, in Simple Form jargon, is an input, its label, a hint and error message by default and we can define a custom component for anything else if we like.).

And indeed, for simple scenarios, this should work well. Problems arise when we need to divert from the default look for a specific field. To understand that, we need to talk a bit about how Simple Form combines the default classes from the configuration with the custom classes that are meant to override them. For each component (input, label, etc.), Simple Form recognizes a <component>_html hash of options. CSS classes are handled via the :class key of this hash. So, for example, label_html: { class: "custom-class" } represents a way to tell Simple Form that we want a specific input’s label to have the "custom-class" class.

Now, how does this "custom-class" get merged with the default classes that we may have defined for the labels? The answer is that Simple Form simply appends the custom class(es) to the default classes. This may work well for traditional CSS styling where we can easily target arbitrary combinations of elements, ids and classes in the CSS and thus ensure that the specificity of the "custom-class" will be high enough to override the defaults. In Tailwind, most classes have the same specificity and in that situation, according to the CSS cascading rules, the class that appears last in the Tailwind-generated CSS file is the one that wins which is most probably not what we want.

To demonstrate this problem with an example, let’s say we have the following default styles defined for a label in the Simple Form configuration:

# config/initializers/simple_form.rb
config.wrappers :default do |b|
  ...
  b.use :label, class: "font-medium"
  ...
end

This config sets a ”medium“ font weight for our form labels by default. Now, suppose we want a specific input’s label to be bold instead, we might want to try the following naive approach (we’re using the Slim template notation here):

/ some_template.html.slim
= simple_form_for(@user) do |f|
  = f.input :e_mail, label: "Important e-mail", \
                     label_html: { class: "font-bold" }

When we inspect the output HTML, everything seems to be in order – the "font-bold" is properly added to the set of classes generated for the label and to the default class "font-medium":

But the label is still not rendered bold! The explanation is that the "font-medium" class happens to be listed below the "font-bold" class in the CSS file generated by Tailwind so, according to the CSS cascade rules, it wins:

When trying to override default Tailwind classes in Simple Form, we need to be able to not only add new classes but also remove some of the default ones. This is not something that Simple Form supports out of the box so we must help it a bit.

One option is the simple_form_tailwind_css gem by Abe Voelker, mentioned also in the GitHub issue. It uses a combination of default Tailwind styles in the Simple Form configuration with a custom form builder and a few custom inputs. The gem supports replacing the default classes related to error messages with custom ones.

This gem was a great source of inspiration for us but we wanted to add a bit more flexibility to the process of overriding default Tailwind classes so we chose our own custom solution that we’ll further describe below.

Our solution to the problem

Let’s first recap the goals that we wanted to achieve with this solution:

we wanted great looking, Tailwind-styled Simple Form forms on our new admin pages by default,
but with the option to amend the style of a particular field or its part if needed,
from a technical point of view, we wanted the default form style and structure to be defined in a single place in the code base,
and we preferred to be able to replace default classes selectively rather than all of them in bulk.

For the visual design and layout of the forms, we chose Tailwind UI, a beautifully crafted set of Tailwind-styled components (more on that in a previous post if you like). Also, inspired by Tailwind UI, we decided to layout our new admin forms on a 12-column grid. That was the relatively easy part.

Coming to a default class overriding API

To handle flexible overriding of the class methods, we explored several options that Simple Form would allow. First, we looked into the conventional way – defining default classes for all form parts in the Simple Form initialization file but in the end overriding them didn’t seem possible without monkey-patching Simple Form or re-defining all input types as custom inputs.

What we wanted to achieve here, was to be able to override the defaults selectively, i.e. we wanted to list the particular classes to remove from the defaults and add the ones that should override them. See the following sample usage for an overridden label style:

= simple_form_for(@user) do |f|
  = f.input :e_mail, label: "Important e-mail", \
                     label_html: { remove_default_class: "font-medium", \
                                   class: "font-bold" }

So, by specifying the :remove_default_class key in the options hash we wanted to selectively remove the named default classes and then the ”standard“ :class key would add the overriding classes. Thus, this API would allow flexible and selective replacement of the form defaults.

In the end, we decided to divert a bit from Simple Form conventions and build a combination of a style-less wrapper configuration and a custom form builder. Let’s show you how.

The style-less wrapper configuration

We decided to put all class defaults not in the Simple Form configuration but into a custom form builder. The configuration then defined a ”wrapper“, i.e. a form layout structure, called :plain, that was intentionally class-less. The main part of it looks like this:

  # config/initializers/simple_form.rb
  config.wrappers :plain do |b|
    ...

    b.use :label
    b.wrapper :input_wrapper, tag: :div do |component|
      component.use :input
    end
    b.use :hint,  wrap_with: { tag: :p }
    b.use :error, wrap_with: { tag: :p }
  end

This wrapper can render the following raw form structure:

<form>
 <!-- for each field: -->
 <div> <!-- wrapper -->
   <label>...</label>  <!-- optional label -->
   <div> <!-- input wrapper -->
     <input .../>  <!-- the input itself -->
     <p>...</p>  <!-- optional hint -->
     <p>...</p>  <!-- optional error message -->
   </div>
 <div>
 ...
</form>

A few notes about this structure:

the input is wrapped in a div: this is not essential but it makes it easier to add margins or set a flexbox layout for certain types of inputs (namely booleans),
all other form components are as basic as it gets, no styling, no wrappers,
this configuration allows the raw form layout to be potentially reused for form variants that should look different – we would just have to create a custom form builder for each form type.

The custom form builder

Next, we built a custom form builder that inherits from the default Simple Form builder and that takes care of both defining and overriding the default classes. (Have a look at our previous post if you’re not sure what a Rails form builder is.)

In the custom builder, we re-defined the most important methods to build various types of form components, such as the input or label methods. The methods basically do just a couple of things:

they define the default Tailwind classes for the given component (with a bit of programmatic logic applied, if needed),
they allow to amend the defaults,
and finally they pass the handling to the parent method via super.

Let’s show an example for a label method in the custom builder:

class Builders::AdminFormBuilder < SimpleForm::FormBuilder
  def label(attribute_name, *args, &block)
    options = args.extract_options!.dup

    default_class = "block text-sm font-medium text-gray-700"
    options = arguments_with_updated_default_class(default_class, **options)

    super(attribute_name, *[args.first, options], &block)
  end
end

It works as follows:

the label method signature is the same as the one in the default Simple Form builder (we’re only using a named block variable instead of an implicit block),
the method extracts the options that are passed to a field via the label_html options hash,
it defines the default Tailwind classes for our form labels, including the "font-medium" class,
it calls the arguments_with_updated_default_class method to update the hash with overridden classes (more on that below),
and it reconstructs the original arguments and passes them to the parent method.

The arguments_with_updated_default_class method is a private method in the builder. It takes the default classes string and a hash of options, such as the label_html hash. From it it removes the classes named in the :remove_default_class key of the options hash and adds the new classes listed in the :class key:

def arguments_with_updated_default_class(default_class, **kwargs)
  kwargs = kwargs.dup
  classes = default_class.to_s

  remove_key = :remove_default_class
  class_key = :class

  if kwargs[remove_key].present?
    classes = (classes.split - kwargs[remove_key].split).join(" ")
    kwargs.delete(remove_key)
  end

  # simple_form sometimes uses array of classes instead of strings
  # so we have to account for that
  if kwargs[class_key].is_a?(Array)
    kwargs[class_key] = kwargs[class_key].map(&:to_s).join(" ")
  end

  kwargs[class_key] = (classes.split + kwargs[class_key].to_s.split)
  kwargs[class_key] = kwargs[class_key].join(" ")

  kwargs
end

The only non-obvious line is hopefully the one with the comment – it is here because we found out that Simple Form uses arrays of strings instead of a space-separated string in certain cases so we had to unify the two.

With these methods in the custom builder, repeating the ”bold label“ example above:

= simple_form_for(@user) do |f|
  = f.input :e_mail, label: "Important e-mail", \
                     label_html: { remove_default_class: "font-medium", \
                                   class: "font-bold" }

…finally produces the correct HTML with the default classes applied except for the default "font-medium" class which is replaced by "font-bold". And the label finally gets rendered bold, phew!

Miscellaneous goodies

Using a custom builder allows to play with the form building API in unlimited ways. For example, as we’ve said above, we lay out our admin forms in a 12-column grid. To support spanning columns in a convenient way:

= f.input :e_mail, col_span: 4

…we defined a private helper method in the builder that translates this custom col_span option to the Tailwind col-span-X class on the wrapper div.

We added more syntactic sugar to e.g. autofocus the first field of the form or to add some data attributes to connect a Stimulus controller for certain special input types. The options are endless 🙂.

(An almost) complete form builder example

We saw a few requests to publish some more of the actual code of our custom form builder. OK, please have a look at the TailwindFormBuilder at this gist! The builder works as a thin wrapper around the default Simple Form builder. It cooperates closely with the style-less Simple Form wrapper configuration mentioned above.

The builder defines a few methods that override methods in the Simple Form default builder. Their main purpose is that they add a set of predefined Tailwind classes to the various elements of the form field wrapper and to the field itself. Still, the methods allow to override every default class involved, by the remove_default_class and class parameters discussed above. Just note that in the gist there is a slightly expanded version of the helper method arguments_with_updated_default_class than in the sample listing above.

Please also note that the default classes defined in the builder are by no means complete and you will have to fill them in to make the builder actually usable. The reason is that we use Tailwind UI for styling the form components and its license (as far as we understand) does not allow us to publish the whole styling as the builder is a ”derivative library“ rather than an end-product.

To use this custom builder, you can just name it together with the custom wrapper when rendering a Simple Form form:

/ app/views/users/edit.html.slim
= simple_form_for(@user, builder: Builders::TailwindFormBuilder, wrapper: :plain, ...) do |f|
  = f.input(:name)

Or, to use it even more conveniently, you can define a Rails helper to set the custom form builder and wrapper for us as well as style the form tag itself, for example in a 12-columns grid:

# app/helpers/form_helper.rb
module FormHelper

  def tailwind_form_for(object, **options, &block)
    default_form_class = "grid grid-cols-1 gap-x-4 gap-y-6 items-start sm:grid-cols-12"
    options[:html] ||= {}
    options[:html][:class] = default_form_class # you can even use arguments_with_updated_default_class here to make the default classes more flexible
    options[:wrapper] = :plain
    options[:builder] = Builders::TailwindFormBuilder

    simple_form_for(object, **options, &block)
  end

end

Then you can just call this helper instead in a form template:

/ app/views/users/edit.html.slim
= tailwind_form_for(@user, ...) do |f|
  = f.input(:name)

Please get in touch with us if something isn’t clear enough, we can pair a bit or something…

Summary

Using a style-less configuration and a custom form builder, we were able to create a system for styling our Simple Form forms with Tailwind, without giving up any flexibility and without resorting to monkey-patches. This combo lives happily on our production web now. If you try a similar approach, please drop us a note!

If you don’t want to miss future posts like this, follow me here or on Twitter. Cheers!

From HTML to Simple Form: anatomy of Rails forms

Matouš Borák — Fri, 17 Jun 2022 17:08:09 +0000

In our main project, we like to use the Simple Form gem for our forms, we have been using it for nearly 10 years, and we still appreciate its very succinct but flexible syntax.

We also like that using Simple Form, we can build consistently looking and structured forms with minimum effort. If you had a chance to read our previous post about our initial experience with View Components, you may remember that we decided to not use View Components at all for our forms. Why? Mainly because Simple Form has its own component system already built in and in this post we will briefly describe how it works under the hood. Also, as we will see, the same principles are valid for all Rails form builders so findings in this post can be helpful for understanding the default style of building forms under Rails in general.

That said, diving into the details of Simple Form forms may be a bit confusing at first because it’s syntax looks so different from the default Rails form helpers. For example, in Simple Form, a single line of code may define all of: the form input field, its label, a hint or even the corresponding validation error message. So how does Simple Form actually generate the final HTML markup?

Anatomy of a Simple Form (and Rails) form

To show this, let’s try a bottom-up approach. We’ll start with pure HTML and will build gradually, layer by layer, the same form using more and more abstracted syntax until we reach the Simple Form style. This will hopefully help us clarify the underlying concepts.

HTML

A basic form may be rendered like the following in the final HTML:

<form action="/users" method="post">
  <div>
    <label for="name">Enter your name: </label>
    <input type="text" name="name" id="name">
  </div>
  <div>
    <label for="email">Enter your email: </label>
    <input type="email" name="email" id="email">
  </div>
  <div>
    <input type="submit" value="Subscribe!">
  </div>
</form>

This HTML may be equally coded in a template (we use a Slim template syntax here) like this:

form action="/users" method="post"
  div
    label> for="name" Enter your name:
    input#name type="text" name="name"
  div
    label> for="email" Enter your email:
    input#email type="text" name="email"
  div
    input type="submit" value="Subscribe!"

Rails form tag helpers

We can get the same HTML output (except negligible details) using the Rails form tag helpers. The word ”tag“ here illustrates that these are helper functions that can each do only one thing – render a particular HTML tag. In fact we are just using a slightly different syntax to render our form than before while the code structure stays the same:

= form_tag("/users", method: :post)
  div
    = label_tag :name, "Enter your name: "
    = text_field_tag :name
  div
    = label_tag :email, "Enter your email: "
    = text_field_tag :email
  div
    = submit_tag "Subscribe!"

Rails form helpers

OK, now we’re getting to more interesting stuff: Rails form helpers, form_with or its older cousin form_for, make a bigger change, especially when the form is bound to an model object, such as @user:

= form_with(model: @user) do |f|
  div
    = f.label :name, "Enter your name: "
    = f.text_field :name
  div
    = f.label :email, "Enter your email: "
    = f.text_field :email
  div
    = f.submit "Subscribe!"

The form_with form helper used here does at least the following:

it checks whether @user is a new or an existing record and generates the proper route (action and method) for the form,
it recognizes that the @user has e.g. its name attribute set and prefills the name field with this value,
and it follows the Rails conventions to name the fields (e.g. with the name="user[name]" attribute) so that the submitted POST data can be nicely processed in the target controller.

Note that we still have to layout the form by ourselves: we define the form structure and Rails form helpers take care of the field / form tags. This pattern is the default in Rails and gives us great flexibility – we can structure each form any way we like – but perhaps makes it a bit harder to keep the forms consistent, especially when we have many complex forms (as is typical in a web admin section) because we have to keep all the form templates structure the same, too. To achieve greater consistency, we could get some help from Rails form builders.

Rails form builders

A Rails form builder is an object used by Rails to build forms. It is instantiated in the form helper form_with / form_for and is yielded in the form block. In the code sample above, the f variable is a Rails form builder object. The builder methods (text_field, label, etc.) know about the form model object (@user) and use the form tag helpers to render the corresponding fields, labels or other elements, potentially wrapped with arbitrary HTML tags. The default Rails form builder though serves just as a simple proxy to the form tag helpers and that is why the overall form HTML structure is, by default, fully on the developer.

Custom form builder

Now, we can create a custom form builder to, for example, tighten the rendered form HTML. The simplest option is to derive it from the default Rails form builder. There are a few internal variables that we need to be aware of when working with form builders: @template represents the view context that we use to call the underlying tag helper methods, @object is the model object (@user) and @object_name is its name ("user").

So, for example, to support a possibility to wrap a text field and a label together with a wrapper div, we can define the following custom builder:

# app/form_builders/labelling_form_builder.rb
class LabellingFormBuilder < ActionView::Helpers::FormBuilder
  def text_field(attribute_name, options = {})
    @template.content_tag(:div, class: "wrapper") do
      label(attribute_name, options[:label]) + 
         super(attribute_name, options.except(:label))
    end
  end
end

…and use it in a form template like this:

/ app/views/users/new.html.slim
= form_with(model: @user, builder: LabellingFormBuilder) do |f|
  = f.text_field :email, label: "Enter your email"

…which generates the following HTML output:

<form action="/users" method="post">
  <div class="wrapper">
    <label for="user_email">Enter your email</label>
    <input type="text" name="user[email]" id="user_email" />
  </div>
</form>

Let’s see what is going on here:

we inherit our custom builder class LabellingFormBuilder from the default Rails form builder
we override its method called text_field so that it renders both label and the input field itself
we can pass a custom label text via our custom option key :label
both label and input are wrapped with a wrapper div, note that we have to use the @template variable the a view context for calling Rails view helpers
in the template, we specify our custom builder with the builder option and pass the custom option for the label

OK, we successfully amended the generated fields HTML structure using a custom form builder and now we can build a text field including its label using a single line of code and have both elements wrapped with a div that we can style.

Also note that we’ve actually created an initial version of a ”component“ to build forms – the code for the layout and structure of the form HTML is defined in a single place in the code base. We can change this one place – the custom builder – and it will affect all forms using that builder.

Overall, custom form builders make it possible to:

alter the way input fields and their accompanying HTML elements are rendered and structured in the form,
add methods for special types of fields,
define a specific API for building forms with a concrete structure and layout, thus helping forms consistency,
use all of ruby syntax to define a logic around our forms, define presets / default values, styles and so on.

Simple Form (and others)

By now, we should have enough information to understand how the Simple Form gem works. Technically, Simple Form just provides a custom Rails form builder. The same holds for other Rails-based form builder gems, such as Formtastic or bootstrap_form (as opposed to Rails-independent form building gems such as Forme).

So, once again, Simple Form is nothing but a custom Rails form builder that comes with a rich set of features around it:

Using the builder’s helper methods, especially the one called input we can generate everything related to a form field: the field itself, its label, hint or error message.
Or anything else, actually! Simple Form makes no assumptions about the form HTML markup and the form structure is fully configurable in a single place – the Simple Form initializer file.
It also provides a set of custom fields with their own logic for processing and rendering model data. Notably, Simple Form supports various automatic processing of records associated to the main model object or generic collections.

With Simple Form, our sample form could be encoded in a template like this:

= simple_form_for(@user) do |f|
  = f.input :name, label: "Enter your name"
  = f.input :email, label: "Enter your email"
  = f.button :submit, "Subscribe!"

Note how everything about a form field is defined in a single line (and we only touched the surface here). Simple Form is great if we prefer consistency when building our forms. By configuring Simple Form ”wrappers“, we can define the default structure and layout of all our forms (plus, of course, alternative form versions if we need them). Moreover, all layout options can be overridden in a particular form or field so we don’t lose any flexibility if we need that.

Mixing different types of form helpers together

As a final note, we can mix different layers of form building helpers, even in a single form. For example, nothing would stop us if we tried to render a form using Simple Form but used a Rails form tag helper for some particular field that must be handled specially.

Doing so, we only have to think about Rails conventions, especially those for naming form fields, and obey them manually when rendering the field. The Rails documentation has an example showing how to mix form layers.

Summary

We went through all layers from the bottom of the generated HTML to the custom form builders and Simple Form. In a future post, we will build on this information to describe how we created a custom Rails form builder to conveniently allow styling our new admin forms with Tailwind.

If you don’t want to miss future posts like this, follow me here or on Twitter. Cheers!

From partials to ViewComponents: writing reusable front-end code in Rails

Matouš Borák — Fri, 03 Jun 2022 16:21:46 +0000

Recently, we began using ViewComponents in our project to help us build the redesigned admin section of our web. We want to share our decision process that made us try this framework in the first place and how well it went. TL;DR? We love it! ❤

The situation

Being a long-term product-oriented project, every now and then we find ourselves rewriting some, even fully working, pages from scratch: to refresh the design, to speed up the page load or to apply new coding standards and get rid of the old ones. Now, the time has come to our old and rusty admin section.

We have lots of – fairly standard – admin pages: with index tables, details and edit forms. We estimated that about 80-90% of our admin pages could look and behave in a more or less unified way, the rest being special pages that must be carefully optimized for the most essential needs our administrators have when doing their job. Although a large part of our admin was already written in a reusable way, we wanted to revamp the look and feel, bringing in new standards including styling via Tailwind CSS or higher interactivity with the Hotwire stack.

We knew from the start that rebuilding admin section was a perfect case for developing and applying components in the view layer. By ”a component“ we mean a piece of reusable code, isolated and encapsulated from others that in the end gets rendered as a visually and functionally distinct part of a webpage. A component should easily provide a default look but still allow flexibility if needed (some support for more visual variants and / or behaviors).

The design

To build a user interface made of reusable components, we first had to choose a template or a design library that would promise a consistent look for all the various page sections that we needed. We assessed many such Tailwind-ready templates focusing on admin interfaces but eventually decided to invest in Tailwind UI. And we never regretted since!

Tailwind UI is a set of wonderfully crafted visual components made with an eye for detail and with responsiveness in mind, perfectly suitable for the latest Tailwind CSS versions. It’s a showcase of what the Tailwind design gurus think is nice and functional and even though we didn’t use the advanced features such as Vue/React templates, we learned a lot from the code samples. What we particularly liked about Tailwind UI is that there was often more than one alternative laid out for a visual feature giving us more options to choose and be inspired from.

What about a full-fledged Rails admin gem?

We briefly considered migrating to a full-grown Rails admin interface, such as ActiveAdmin, RailsAdmin, Administrate or Avo. We especially liked Avo which is built on a very modern stack similar to ours (Tailwind + Hotwire + ViewComponents). In the end, we didn’t go this route as we found some of the options a bit too restrictive (even though Avo is very flexible) and we did not feel like trying to amend it to our needs. For example, Avo renders forms in a 1-field-per-row layout while we wanted something more similar to the Tailwind UI Stacked form layout. Nevertheless, we found a great deal of inspiration in the Avo code and its design principles.

Options for reusable front-end code in Rails

After we settled down on the design, we pondered about a suitable way to add the front-end components to our code base. What options do Rails actually give us in the first place? We considered partial templates, Rails helpers and then we moved on to other possible solutions. Below is a brief summary of our thought process at that time.

Partial templates

Partial template (or ”partial“) is a standard Rails way to extract a piece of template code to its own file. The partial then can be called (rendered) from other templates, helpers or controllers.

Partials, like all templates, are HTML-centric – they are the strongest for embedding various HTML tags in a structure which is then rendered on a web page. However, they are not that great if you need to add some non-trivial logic – a template with more than a few control statements can quickly become messy.

In our opinion, the biggest issue, from a ”component“ point of view, is their lack of isolation. A partial template freely recognizes all instance variables (@variable) and this makes the template tightly coupled with the controller layer where these variables are typically defined. Suppose we’d use a partial template on five different pages: we would have to define the same instance variable(s) in all five actions of the corresponding controllers. Even worse, instance variables default to nil so forgetting to properly set a variable does not raise an exception, instead it can just lead to an unexpected blank output.

Sure, we could pass the data as local variables instead (and we heartily encourage such a more explicit style of passing data to partials) but this convention would have to be guarded and enforced all over the team. We actually run a custom Overcommit hook to encourage this explicit style in our project. Still, the locals hash is just… a Hash and makes the partials API only moderately flexible for us, especially since we settled down on using keyword argument APIs wherever possible.

Update as of January 2024: Actually, since Rails 7.1, we have the option to define strict locals in Rails partials, see our post about them. We consider strict locals a very nice feature as they allow to enforce an "API" for calling / rendering Rails partials. This is definitely a bonus point for Rails partials.

The nice thing about partial templates is that templates are unit-testable with View specs (or similarly in Minitest) and the rendered output can even be verified using Capybara matchers.

Rails helpers

Helpers are another ”Railsy“ option to componentize things in the view layer. They are simple ruby functions, living inside a module.

As opposed to templates, Rails helpers are ruby-centric. Being just normal ruby functions, they support any API style for passing parameters that is supported by your ruby version, including keyword arguments.

But the easier it is to call and embed ruby code in a helper, the harder it tends to be to combine more HTML tags there into a renderable structure. Simple tags are fine and there are a multitude of pre-defined ActionView helpers available, such as content_tag or link_to that reduce repetition. But building a more complex HTML structure inside a helper can become a painful experience. Sooner than later one finds that he or she needs to concat things, perhaps even capture things and all the time they must be aware of the possible security implications of building such HTML structure and learn about html_safe, safe_concat and similar stuff while templates partly mitigate this problem by regarding raw HTML tags as html_safe by default.

While helpers can deal with ruby code logic more elegantly than templates, they are nowhere near ruby classes or objects in terms of flexibility. Helpers are – similarly to templates – not isolated from instance variables but it is perhaps a bit more straightforward to obey a convention of using only function parameters to pass data to them. But helpers are also global: each helper is available in the whole view layer, which means that their names must be unique and that categorizing them into multiple files (modules) makes less sense as it brings no real encapsulation. Add to it the fact that Rails itself defines dozens of helpers so the helpers namespace can become quite cluttered and name collisions with hard-to-debug surprises may occur.

It is easy to unit-test Rails helpers but only as plain ruby functions, for example it is not possible to use Capybara on the generated output by default.

Combination of partials and helpers

We’ve seen that partial templates and helpers have each their own strengths and weaknesses in terms of building components. So why not let each of them focus on what they can do best? Indeed, partials and helpers are meant to cooperate: one can easily call helpers from templates as well as render partials from helper functions.

While we were sure that we could get pretty far using a combination of helpers and templates, by the time we were assessing this option, we already knew we wanted to look elsewhere. To us, the two worlds are too distinct for building components, the two types of code are located too far from each other without an obvious interconnection between them. Partials and helpers may play together well but they still don’t make a clear unit.

So what about the world outside Rails defaults? There are quite a few independent projects trying to help build components in the Rails view layer, among the more famous being Draper (utilizing the decorators pattern) or Cells (full-featured components in views). In the end, we decided to take a deeper look into a relatively new one – the ViewComponent framework.

View Components

The ViewComponent framework has originally been developed and used extensively at GitHub. It provides a set of conventions to build components in the view layer that should make them well encapsulated, reusable, flexible and testable. Below are our comments to features that we particularly liked about View Components:

View Components have an explicit home in the code base. By ”having a home“ we not only mean that view components reside under the app/components folder but also the fact that the code for the component behavior as well as its template live next to each other, in the same place in the code base. The components code can be categorized into folders by their meaning or function rather than technology.
The component ruby file supports logic of any complexity. A component is just a ruby class so we can leverage all features of object-oriented programming in them such as private methods, composition, inheritance and just about anything else, if needed. The template file, on the other hand, can stay virtually logic-less.
View components are truly encapsulated. All configuration and data for the component must be explicitly passed in via the initialize method arguments or blocks. Even Rails helpers are not automatically recognized and must be explicitly included or accessed through a helpers proxy object.
The rendering of components is flexible, too. The developer can choose whether to render the output in a template file or inline in the ruby code (in the call method). The former style suits well for components with a more complex HTML structure, the latter for smaller and simpler ones.
View Components support and encourage testing via unit tests. The tests are then very fast and validate the rendered HTML output (with support for Capybara matchers included).
Good tests coverage of the view layer is – frankly – not that common in Rails projects because it is notoriously unpleasant. System tests tend to give good coverage but are slow and hard to maintain while view unit tests are possible, as we saw above, but hard to isolate and prepare test data for. View Components profit from their natural encapsulation and explicit data flow, so writing unit tests for them should be much easier.
There is a preview mode for View Components. This is especially handy because it encourages components reuse and provides an obvious place for sample code and documentation.

This all looks very well but did we see any disadvantages before starting with View Components? Not much, really. We expected that building components including meaningful previews and tests definitely requires a bit more work than creating a partial template, for example, but the benefits of doing so, in our eyes, outweighed the pain.

The biggest unclear area that we saw related to view components were forms. There were glimpses of compatibility issues with Rails form helpers in the documentation and we saw a recent effort of the team to mitigate them. Moreover, we were used to building forms with Simple Form which added another variable to the equation. And, in general, we considered the Rails form builders (as well as the Simple Form builder) a system of form-related components in the first place so we were unsure how this would fit into the View Components ecosystem or whether we should even try to do that.

Nevertheless, we decided to try building a few View Components for our new admin interface, and see how it goes.

View Components after a few weeks of intensive usage

And yes, we have some findings to share after a few weeks of using and building View Components.

It was easy to start with View Components

The conventions of View Components seemed so clear and obvious that after a few tries we were able to build new components without hesitation. We routinely added ruby code, templates, unit tests and previews and used the emerging components on the admin pages that we were rebuilding. The feeling that more and more of a page is compiled from a few well-defined and well-tested components is very addictive!

The previews are great, Lookbook is awesome

One of the features that we liked the most were previews, especially after we found about the Lookbook project. It is a user interface for viewing, documenting and interacting with View Component previews. The best thing is that Lookbook does not deviate from View Component preview conventions so a developer just has to write a VC preview with perhaps a few optional annotations in the comments and Lookbook automatically converts it into something like this:

We really love Lookbook and it immediately became the official developer version of our ”Design manual“ accessible for everyone in our company.

Building view components can be a hard core API coding job

After a few weeks we realized something unexpected. Building components, especially the more complex and universal ones (that you might expect in an admin interface), felt more like back-end rather than front-end work. Of course, we had to encode the component into a HTML template and style it but this seemed like an icing on the cake. Instead, thinking how to meaningfully pass data into the components and how to interconnect them while still allowing reasonable flexibility became the main focus of our work. Which brings us to the next point…

We had the best results with an outside-in approach

We put a lot of energy into trying to make the components as easily reusable for the developers as possible. For this we always started by writing code samples that would use the (at that time non-existent) component. We tried a few variants and attempted to cover all the use/edge cases known at that time. Only after we were happy with the external API for the component, we moved on to solving its internals. The result is a set of components that we find lovable to use.

Helpers can help components rendering substantially

For the components that were meant to be reused frequently, we always added a helper for the sole purpose of simplifying calling the component. For example the HeadingComponent from the image above, is actually meant to be called via an admin_heading helper which is just a simple wrapper around the component rendering:

module AdminComponentsHelper
  def admin_heading(**options, &block)
    render Containers::Admin::HeadingComponent.new(**options), &block
  end
end

Luckily, View Component previews as well as Lookbook work with helpers without issues so there was nothing stopping us from documenting the actual encouraged style of using the components.

We did not use View Components for forms at all

In important conclusion regarding forms came from this effort as well: we decided to not use View Components for forms at all. While we were not particularly happy about having to maintain two different component systems in our code base, we took this pragmatic decision because we like the Simple Form style and Simple Form itself is a very flexible component system, just focused on forms building.

Theoretically, we could be able to mimic the Simple Form API with a set of form-related View Components but we didn’t think the effort was worth it. Instead, we dove deep in Simple Form builders and managed to create a Tailwind-styled one that suits our needs perfectly (this might deserve a separate post; update: there it is). And the best part of all: both unit tests and Lookbook work very well even for Simple Form tests and previews so we didn’t have to compromise anything important. Have a look at this gist for a basic example of such preview.

Summary

Overall, we are very happy with adding View Components to our project. Throughout the first few weeks, we built around ten universal components covering most of the needs for our admin pages and are quickly adding new pages in the new style using them. View Components seem like the missing piece that fit perfectly to our current view layer evolution needs.

Since then, the new convention for choosing a pattern in the view layer has become as simple as:

Is it a form? Use Simple Form with our new helpers.
Is it supposed to be reusable? Build a View Component and think well about the API and helpers.
Is it critical? Build a View Component and ensure a good test coverage.
Is there a non-trivial logic involved in the rendering? Build a View Component.
None of the above? Choose freely among View Components, templates and helpers, whatever seems like a good fit.

Thank you for your attention, if you feel tempted to try View Components, good!

If you don’t want to miss future posts like this, follow me here or on Twitter. Cheers!

How to run a really long task from a Rails web request

Matouš Borák — Tue, 19 Apr 2022 06:22:35 +0000

Recently, our management needed a way to export invoices in bulk. After the manager selects the first and last invoice for the batch in a web form, an asynchronous process should start that generates PDF files for the invoices, packs them into a zip file and sends the manager an email with a link to download the export. Now, generating the PDFs is slow, very slow. For larger batches involving hundreds or thousands of invoices, this process can easily take 10 or 15 minutes or even more.

So how do we trigger such a long-running process from a Rails request? The first option that comes to mind is a background job run by some of the queuing back-ends such as Sidekiq, Resque or DelayedJob, possibly governed by ActiveJob. While this would surely work, the problem with all these solutions is that they usually have a limited number of workers available on the server and we didn’t want to potentially block other important background tasks for so long.

What we wanted instead was to run a new, separate process from the Rails request. Something like running a Rake task but triggered by a web request. In fact, we even had the bulk export already implemented as a Rake task, so what we actually wanted was to make this task accessible from our admin web interface.

”Forking“ the process

The standard way on Unix-like systems to spawn a new process is to fork it. In a Rails controller, forking a rake task could look like this:

class BulkInvoiceExportsController < ApplicationController
  def create
    child = fork do
      exec("bin/rails export_invoices FROM=20220001 TO=20220100 \\
            >> /tmp/bulk_invoices_export.log 2>&1")
    end
    Process.detach(child)
  end
end

Let’s note a few things about the code inspired by this StackOverflow answer:

The Process#fork method splits the current process (its current thread) into two copies and the new child process runs the code in the block.
The child process is then replaced with a newly loaded process using Process#exec.
The final child process inherits all important settings from the parent process, such as environment variables, open file descriptors or current working directory. This is why we can simply run bin/rails without having to set up the correct ruby first (even when using a ruby version manager such as rvm, rbenv or chruby) and without specifying an absolute path to the Rails binary.
Because the code in the block uses shell redirection, the child Rails process is not executed directly but using a standard shell (usually /bin/sh). Redirection allows us to debug and monitor what is going on in the rake task.
By default, the operating system expects that the parent process is interested in the child process termination status. We are not – we want to run the rake task and forget about it, the task handles everything else such as sending the final email by itself. That’s why we call Process#detach to let the OS know we don’t care about the child process and to prevent accumulating zombie processes.

”Spawning“ the process

If we wanted to make our code more portable (usable on Windows, for example), we would have to use Process#spawn instead of fork, as suggested in the ruby documentation. The spawn method also allows to fine-tune the child process environment, file descriptors, limits or working directory.

An almost equivalent way of scheduling the rake task using spawn could be written this way:

class BulkInvoiceExportsController < ApplicationController
  def create
    child = spawn("bin/rails export_invoices FROM=20220001 TO=20220100",
                  %i[out err] => %w[/tmp/bulk_invoices_export.log a])
    Process.detach(child)
  end
end

Security caveats

Please keep in mind that triggering such a long-running process from the controller is not safe. In the previous examples, each request to the create action of the controller leads to spawning one external Rails process, consuming perhaps a substantial portion of the CPU and memory resources and opening more connections to your database servers. This is a setup very vulnerable to DoS attacks.

The technique is probably OK only in very controlled environments such as in an internal admin area accessible to a limited number of people who know what they are doing and when the function is used only sparingly. If we wanted to make this rake task publicly accessible (as in a ”data take out“ function, for example), we would definitely resort to a real queuing system such as those mentioned above or perhaps a queuing daemon on the system level (e.g. atd which can hold the tasks based on the server load).

Anyway, for our use case, directly forking the rake task from the controller was the most pragmatic way to go and we are happy about the result.

If you don’t want to miss future posts like this, follow me here or on Twitter. Cheers!

Track and fix excessive Active Record instantiation

Matouš Borák — Fri, 21 Jan 2022 12:45:34 +0000

When working with a Rails back-end code, grabbing too many records from the database is discouraged, and for very good reasons:

your database server has to work hard to find the records,
a lot of data needs to be transferred from the database to your app,
and — last but not least — Active Record instantiates too many objects, leading to a memory bloat in your application.

All of this slows down the response time of your web app for the given request. But even worse, it also affects all future requests because the memory, newly allocated by the Rails process, usually becomes fragmented and hard to release back, unless you apply special tweaks.

OK but is this a real issue?

Even though limiting the db queries is rather a basic rule, mentioned early in the Rails Guides, we noticed that a huge SELECT still occasionally slips into our production code. How come? Turns out, it is surprisingly easy for several reasons:

developers usually work with a small dev database and forget about the scale of production data,
even though our dev team actually works with a large subset of production data, it’s too easy to forget to test a worse case scenario,
the Active Record syntax is very succinct and lets an unsuspecting developer build huge JOINs very easily; for example, this innocent-looking query: User.recent_customers.eager_load(orders: :order_logs) can suddenly cause a gigantic data load when a power user with many orders falls into the recent_customers scope,
and, sometimes devs simply forget to have large data processed in batches.

We first became aware of the issue when we profiled some exceptionally slow requests in DataDog and noticed large Active Record instantiation spans, such as this one:

The trace shows that in this particular request, Active Record instantiated over 2,100 ZipCode model objects which delayed the response by ~160 ms (and this even excludes the time needed to run the query and transfer the results to the Rails app). That’s insane! We don’t think we need information about two thousand zip codes anywhere on our site.

Tracking the problem in production

After finding and fixing a few places, we decided that we needed a continuous monitoring of this problem in production. We could have probably done this in DataDog itself via a custom metric generated from the Indexed APM spans. Other APM systems may have different options, such as ScoutAPM’s memory bloat detection.

But in the end, we chose to build a custom solution that we could more easily send to our reporting system instead. Because, it turns out, tracing the instantiations is very well supported using Rails instrumentation. Each time Active Record instantiates objects after retrieving data from the database, it generates the instantiation.active_record event which we can hook into.

Below is the complete code for a custom log subscriber (i.e. a class to consume the given instrumentation events and log them) that processes "instantiation" events.

# app/subscribers/active_record_instantiation_subscriber.rb
require "active_support/log_subscriber"

# Send Rollbar error when ActiveRecord instantiates too many objects.
# Log all AR instantiation in dev log.
class ActiveRecordInstantiationSubscriber < ::ActiveSupport::LogSubscriber
  MAX_TOLERATED_RECORDS = 2000
  MAX_TOLERATED_DURATION = 200 # in milliseconds

  def instantiation(event)
    return if Rails.env.test?

    payload = event.payload
    excessive_load = payload[:record_count].to_i > MAX_TOLERATED_RECORDS || event.duration > MAX_TOLERATED_DURATION
    if Rails.env.development?
      message = "  Instantiated #{payload[:record_count]} records 
                 of class #{payload[:class_name]} 
                 in #{event.duration} ms".squish
      excessive_load ? error(message) : debug(message)

    elsif excessive_load && Rails.env.production?
      Rollbar.error("Too many ActiveRecord objects instantiated",
                    record_count: payload[:record_count],
                    class_name: payload[:class_name],
                    duration: event.duration,
                    source_code: Rails.backtrace_cleaner.clean(caller))
    end
  end
end

The location of the log subscriber file is arbitrary given that it is set up from a Rails initializer:

require "./app/subscribers/active_record_instantiation_subscriber"

ActiveSupport::Notifications.subscribe("instantiation.active_record", 
                                       ActiveRecordInstantiationSubscriber.new)

In essence, the subscriber serves two purposes:

It logs a message to the Rails log about each instantiation in development environment, so that the developer can see potential problems with creating too many model objects as early as possible.
In production, the code reports a custom error to our tracking system if the number of instantiated records is especially high (above 2000, as configured in the MAX_TOLERATED_RECORDS constant) or the instantiation too slow (above 200 ms, see the MAX_TOLERATED_DURATION constant). This error message includes:
- the number of instantiated objects,
- the time the instantiation took (in ms),
- the instantiated class,
- and a stack trace pointing to the proper place in code.

This way, we can continuously monitor excessive instantiation in our tracking system from the real traffic.

How to fix the issues found

OK, excessive Active Record instantiation warnings successfully fill your reporting system so… what next? The general effort here is to try to decrease the amount of data loaded from the database. The specific way to do that will depend on why the data is loaded in the first place. Let’s have a look on some of the options:

Add a limit clause to your queries wherever it makes sense. Use pagination for data listings and index pages.
Use select or even better, pluck: the select method still leads to model objects instantiation but limits instantiating attributes only to those explicitly listed. The pluck method skips model objects creation altogether and returns a simple array of the results data, saving a lot of memory and CPU cycles.
In case you only look for an aggregate value, use calculation methods instead of grabbing all records and aggregating them in ruby.
If you really need to process a lot of records (e.g. in a rake task), use methods for loading the data in batches. The in_batches method can even be combined with pluck, for example. Or better yet, use mass update if appropriate.
Try rewriting complex (especially nested) eager load queries into multiple simpler queries that you can control more precisely.

By the way, most of these tips are more thoroughly explained in the Complete Guide to Rails Performance by Nate Berkopec, a book we recommend with love. So, may your memory be free!

If you don’t want to miss future posts like this, follow me here or on Twitter. Cheers!

How to get rid of Internet Explorer faster but carefully

Matouš Borák — Mon, 17 Jan 2022 10:41:45 +0000

Many web developers eagerly look forward to Internet Explorer End of Life, scheduled on June 15th, 2022. We definitely do as well! Of course, this EOL date doesn’t mean that all IE users will be gone by then. But it will be OK to take IE into consideration even less when updating the site.

Nevertheless, we’ve already set out that journey about a year ago:

we show an overlay popup to all IE users, asking them to switch:
throughout last year, we upgraded to Tailwind 2 and Stimulus 3 even though both frameworks drop official support for IE,
we still use a few IE-related polyfills that solve the biggest issues, but only for the most critical use cases,
and we generally don’t test newly built features in IE any more.

(For more details about our previous IE-related measures, see our last post.)

Since then, we saw a clear trend: relative IE usage dropped from 1.7% in January 2021 down to ~0.5% in early November. Still, we were thinking: could we help this trend even more? Our site is a marketplace connecting customers with various craftspeople. We observed that almost all IE visits were from our customers, especially first-time visitors. We knew our customers were often not very tech-savvy and we couldn’t expect them to migrate to a newer browser unless the experience was really seamless. So, could we still help them make the switch somehow? Turns out we could!

The ”Need Microsoft Edge List“

A nice migration process is actually offered by Microsoft itself although the feature can be harder to find. Microsoft keeps an official list of websites that want their users to switch from IE to Edge. Windows systems periodically download this file and take care of the migration process on the sites listed there.

The list is called IE Compatibility List and the process to get there is both surprising and lovely — because very old-school and manual 😍:

as a prerequisite, you need to show a message to your IE visitors on your web, asking them to switch,
then you can officially request Microsoft to add your site to the list; you do that by sending a free-form email including the required information,
a human replies (in our case it was Kelly, hi! 👋) to confirm your request or clarify details,
the same person takes care of you during the whole time and informs you about the progress,
about a week later, your site is added to the list itself and starts being recognized by Windows immediately.

From now on, when a user visits your site with IE, he or she is redirected to the Edge browser and a localized explanation pops up. Most importantly, all bookmarks, settings, cookies and passwords are automatically transferred so everything works just the same as in IE (but better).

The effect

Of course we were very curious how effective this IE compatibility list was. Our analytics data shows that the number of IE visits dropped to half of the previous numbers within a few days after the addition to the list.

That’s great! Using the official list, we were able to migrate even many first-time visitors and the relative proportion of IE visits dropped to ~0.2%. Still, as can be seen in the chart, there are a few IE visitors left, making around one hundred visits per week. We guess these are users of very old and long-time-not-updated Windows systems who must experience increasingly serious problems accessing the internet overall… We are sorry about that but we believe we did all we could. So, good luck to all and see you in better browser times!

Upgrade to Stimulus 3, say bye to IE11, and celebrate 🎉

Matouš Borák — Tue, 19 Oct 2021 20:53:23 +0000

Most of our application JavaScript code is already written as Stimulus controllers, the rest being slowly assimilated or removed. Recently, we wanted to upgrade the Stimulus framework to version 3 to gain access to the new cool features, such as:

debug mode that greatly helps understanding what exactly your controllers are doing and why,
dispatching events among controllers - previously, communication between controllers required various ”hacks“, not any more as it is now official and straightforward,
action parameters for even more flexibility when calling controller actions,
default values no more need to be specified in HTML , they can reside in the controller itself,
and more.

So we started by fixing all deprecation warnings, then updated the Stimulus package and all imports to the new package name. Since we are still using Webpacker (not for long, you bet…), we added the – now separate – stimulus-webpack-helpers package and updated the controllers initialization. All easy and clear, right?

Well, not so fast. We did not read the whole release notes properly enough and did not notice at first that Stimulus 3 drops support for IE11. This made us stop for a while and do some browser usage analyses.

IE11 measures

Luckily, we’ve had most of the work done from almost a year ago, when we adopted Tailwind in our project. Tailwind 2.0 also dropped official support for IE11 and we made an important decision at that time: while the IE11 usage numbers were small, we could not afford making our web totally unusable for these users. So we employed a few polyfills, added a few styling fixes specific to IE11 so that our web was still – somehow – accessible via this old browser. Also, we put up an alert that tried to persuade people to switch. And we waited… until today.

So now we looked at the numbers again and found that all seemed very good! The usage numbers, both absolute and relative, decreased steadily, our providers didn’t use IE almost at all, our customers a bit more but still negligibly. Who knows whether our pop up, Microsoft or a general innovation pressure contributed to the effect, the important thing was that we were ready to make the next step.

So, we decided to continue freely with the Stimulus upgrade and we also added our site to the Need Microsoft Edge list. Being listed here will automatically redirect IE11 users to Edge when they visit our site.

”Not IE 11“

To our surprise, we hit a weird and at first confusing error during the Stimulus upgrade process: Uncaught (in promise) TypeError: class constructors must be invoked with 'new'". No controllers worked at all. We double-, triple-checked the configs and all seemed OK. The solution clicked after we read this response on Stack Overflow. Our JS code was transpiled to ES5 but Stimulus itself now uses ES6 as the compile target. So our ES5 controllers could not extend ES6 Stimulus classes.

We found the cause in the browserslist section of our package.json file. This setting is used by Babel to transpile various modern JS features to their safer alternatives according to browsers usage, and we needed to explicitly exclude IE11 support to compile our JS code to ES6 and the error disappeared.

  "browserslist": [
-    "defaults"
+    "defaults",
+    "not IE 11"
  ],

By the way, targeting our JavaScript code to ES6 alone decreased our production bundle size by about 15% (unzipped). Nice!

We also quickly checked with Can I Use that we are OK with ES6 considering our browser usage pattern, and yes, sure:

Finally, as we recently added the Stimulus-Use library to our project, we made sure to upgrade it to current beta which supports Stimulus 3.

Conclusion

Our tests show that everything works nicely under Stimulus 3. We enjoy the lovely debug mode and other new features. Stimulus has grown to a mature framework, perfectly usable in HTML-first application stacks.

While for the few remaining IE11 users it will be increasingly difficult to use our site, we are quite OK with it as we have tried to reduce the harm before and continue to do so to some (lesser and lesser) extent. You can’t stop progress. Bye IE! 👋

If you like reading stuff like this, you might want to follow us on Twitter.

Migrating Selenium system tests to Cuprite

Matouš Borák — Mon, 04 Oct 2021 20:58:51 +0000

In our project, we’ve been running system tests (then called rather "Feature tests") since around 2016. System tests use a real browser in the background and test all layers of a Rails application at once: from the database all the way up to the nuances of JavaScript loaded together with the web pages. Back then, we wrote our system tests using Capybara with Poltergeist, a driver that ran a headless Phantom JS browser. Since this browser stopped being actively developed, we migrated our test suite to the Selenium / Webdriver wrapper around Chrome browser around ~2018. Chrome was itself fine for tests automation but the Selenium API was quite limited and we had to rewrite several Poltergeist features using 3rd party gems and tools.

That is why we were happy to find out that a new ruby testing driver approach is being developed. It is called Cuprite, it runs the Ferrum library under the hood which, in turn, is an API that directly instruments the Chrome browser using the Chrome DevTools Protocol (CDP). About a week ago, we finally made a serious attempt to make our system test suite run on Cuprite, with especially two questions in our minds:

would the tests run faster?
would the Cuprite API be easier to use?

As a little spoiler we are glad to say that both points turned true for us and we kind of fell in love with these wonderful pieces of software, Cuprite and Ferrum. If you’d like to hear more details, read on.

The migration

All important parts of the basic installation process are shown in the Cuprite README and in the customization section of the Ferrum README. Great resources and tips can also be found in this article by Evil Martians.

The very lovely thing about Cuprite is that it very much resembles the old but good Poltergeist API. The CDP protocol is much more versatile than Selenium driver and thus Cuprite allows e.g. the following things which were hard or even impossible with Selenium:

blocking / allowing requests to external domains and URLs,
setting cookies, even before visiting the given page,
setting request headers,
opening the Chrome DevTools with a single line of code,
inspecting and/or logging all communication between the test and the Chrome browser (all CDP messages).

For a lot of these features, we previously had to adopt various 3rd party gems, such as the Puffing Billy proxy (for blocking domains), the webdrivers gem (for auto-updating the Chrome drivers), etc. and although they certainly did a good job for us, now we were able to finally rip them off the project completely:

The Cuprite speed-up is real and can be helped even more

OK, let’s talk numbers. We have ~140 system tests in our project, covering the most important use cases in our web application. Several of the test cases go through some very complex scenarios, slowing down the whole test suite run time considerably. Overall, our system tests used to run approximately 12 minutes on Selenium, while the same suite finishes in ~7 minutes under Cuprite. That is approximately a 40% speed-up 😲!

Not all of this can be attributed to Cuprite speed alone though as, in the end, we configured the new driver slightly differently. For example we used whitelisting of specific domains instead of blocking the others as we did on Selenium. It is now a much stronger and stricter setup that probably blocks more domains than before, speeding up the page loads. Still, the speed up was clear and apparent since the first run of Cuprite.

Faster sign-in in tests

And we added a few more tricks. We rewrote our sign-in helper method in a more efficient way. This was possible because Cuprite allows setting a cookie (i.e. the session cookie) even prior to visiting a page, unlike Selenium. Thus, we could manually generate a session token and store it both to our back-end session store as well as the session cookie. We just needed to make sure the session cookie had the same options as the real session cookie.

def login_via_cookie_as(user)
  public_session_id = SecureRandom.hex(16)
  page.driver.set_cookie("session_test", 
                         public_session_id, 
                         domain: ".example.com", 
                         sameSite: :Lax, 
                         secure: true, 
                         httpOnly: true)

  private_session_id = Rack::Session::SessionId.new(public_session_id)
                                               .private_id
  Session.create!(session_id: private_session_id, 
                  data: { user_id: user.id })
end

This lead to another noticeable speed-up of the tests suite run.

Test fixes needed

Initially, about 30 tests (~20%) that were OK under Selenium, failed under Cuprite. Some of the failures were easy to fix, others were more puzzling. Overall, we came to a feeling that the Cuprite driver was less forgiving than Selenium, forcing us to be a bit more precise in our tests.

For example, we filled a value of "10 000" into a number input field in a test (note the whitespace). This works without issues inside Selenium but fails under Cuprite. Now, let’s show a few more types of fixes that we had to deal with.

Scrolling and clicking issues

A lot of tests failed because Cuprite tried to click an element that was covered by another element on the page. Cuprite seems to scroll and center the element a bit less (compared to Selenium) prior to clicking it.

Here is a typical example – the test was trying to click on the button covered by the sticky header, as we could easily see by saving the page screenshot upon test failure:

The failure log message would show a Capybara::Cuprite::MouseEventFailed error with details about which element was at the same position as the clicked-on element.

We had to manually scroll to an element in a few tests. To further mitigate this issue in a more generic way, we also overloaded the click_button method from Capybara to scroll and center the button on the page before clicking it:

def click_button(locator, *options)
  button = find_button(locator, *options)
  page.scroll_to(button, align: :center)
  button.click
end

File uploads needed absolute file paths

We use Dropzone JS to support uploading files. Under Cuprite, uploading stopped working and an ERR_ACCESS_DENIED error was shown in the JavaScript console each time a test attempted to upload a file.

It took a while to debug this but in the end the issue was quite prosaic – Chrome needed absolute paths when simulating the file upload in the test. So, the fix was just along the following lines:

- attach_file("file-input", 
-             "./app/assets/images/backgrounds/brown-wood-bg_512.png")
+ attach_file("file-input", 
+             Rails.root.join("app/assets/images/backgrounds/brown-wood-bg_512.png").to_s)

We are not sure if this issue is only when using Dropzone or rather related to generic file uploads in system tests.

AJAX / Fetch issues due to Cuprite being ”too fast“

Surprisingly, some more tests started failing randomly. Soon it turned out that all of them deal somehow with JavaScript sending requests to the back-end via AJAX or Fetch. Again, these tests were rather stable under Selenium and – as we investigated – the issue was that under some circumstances the Cuprite driver generated multiple Fetch requests and sent them too fast.

For example, we have a few ”live search“ fields, backed by back-end Fetch requests, on some pages. The live search function was usually triggered by the keyup event and Cuprite was such a fast typewriter that it frequently sent multiple requests almost at once. If some of the responses got a bit late or out of sync, the front-end JavaScript code began hitting issues. We solved this by adopting a technique called debouncing and, frankly, we should have done this since the beginning. By the way, we used the useDebounce module from the marvelous Stimulus-use library to achieve this.

Custom Cuprite logger

A lot of our migration effort went to developing a logger for some of the events that Cuprite / Ferrum handles when talking to the browser. In general, Cuprite offers a stream of all CDP messages exchanged between the driver and the browser. To use it, one has to filter out the events that he or she is interested in.

We used this feature to track two kinds of data in the log:

JavaScript errors printed in the JS console in the Chrome browser,
details about the requests and responses sent to/from the server as this information sometimes greatly helps debugging tests.

Usually, we let the test fail when a JavaScript error occurs. Ferrum has a js_errors option in the driver configuration to do just that. It works nice but we used a custom solution instead because we wanted some of the JavaScript errors to actually be ignored and we didn’t want a test failure then. In the end, we made a helper class (similar to this one) that collected all JS errors during a test run and checked this array of errors in the after block, allowing for ignoring preconfigured types of errors. Note that care must be taken about cleaning-up the state in RSpec as triggering an expectation error in the after block otherwise skips all later code in the block.

def catch_javascript_errors(log_records, ignored_js_errors)
  return if log_records.blank?

  aggregate_failures "javascript errors" do
    log_records.each do |error|
      next if ignored_js_error?(error, ignored_js_errors)
      expect(error).to be_nil, "Error caught in JS console:\n#{error}"
    end
  end
end

RSpec.configure do |config|
  # this is run after each test
  config.after do |example|
    catch_javascript_errors(page.driver.browser.logger.error_logs, 
                            example.metadata[:ignored_js_errors])
  ensure
    # truncate the collected JS errors
    page.driver.browser.logger.truncate
    # clean up networking
    page.driver.wait_for_network_idle
  end
end

Other CDP protocol events (namely Network.requestWillBeSent, Network.responseReceived and Network.requestServedFromCache) served as the basis for logging all requests and their responses. We chose a custom log format that enables us to better understand what’s going on – network wise – in each test and if you’re curious, it looks like this:

Summary

We are indeed very happy about the migration to Cuprite. Our tests are much faster, the API to handle them is simpler and the migration forced us to take a closer care while handling some special situations, benefiting not only the tests but the users visiting our site, too. Overall this feels like a great move and we heartily recommend anyone to do the same! 🤞

If you like reading stuff like this, you might want to follow us on Twitter.