DEV Community: Tom de Bruijn

Expand Your Monitoring Capabilities with AppSignal's Standalone Agent Docker Image

Tom de Bruijn — Tue, 04 Jul 2023 11:44:14 +0000

Want to monitor all of your application's services? Our Standalone Agent allows you to monitor processes our standard integrations don't monitor by default, helping you effortlessly expand your monitoring capabilities.

To help simplify the process of configuring our standalone agent, we're excited to announce the launch of our Standalone Agent's Docker image, available on Docker Hub under the name appsignal/agent.

What Is the AppSignal Standalone Agent?

Based on the same software that powers our Ruby, Elixir, and JavaScript integrations, the standalone agent can be used to monitor:

Infrastructure: machines that are part of our system but do not run application code.
Background jobs: Think intensive cron jobs or long-running data-processing scripts. You can use the standard integration if these background jobs are written in a supported language (Ruby, Elixir, or Node.js).
More languages: Programs written in languages other than those supported by an AppSignal integration.

For instance, with the standalone agent, you can track a machine learning model written in Java, instrument backup scripts, or monitor Kafka brokers.

Monitor Traces, Custom Metrics, and More

The AppSignal standalone agent is a powerful addition to monitoring your applications. With the standalone agent Docker image, you can report OpenTelemetry traces, StatsD metrics, and NGINX metrics to a single endpoint within your internal network.

You can view all this information on AppSignal alongside your application's core metrics, giving you a bird's eye view of the performance of your application and its services.

Getting Started with AppSignal's Docker Image

Previously, running the AppSignal standalone agent required running it as another process on a Docker container for your application or infrastructure like NGINX. However, with our new docker image, you can run the AppSignal standalone agent in an isolated container, making it easier to manage and more efficient to use.

Ready to get started? You can find more information on how to install and configure the AppSignal standalone docker image in our installation and configuration documentation.

AppSignal, The Developer Driven APM

AppSignal's standalone agent is just one of our many developer-driven tools designed to help you to monitor your application. Developers also enjoy using our monitoring because we have:

An intuitive interface that is easy to navigate.
Simple and predictable pricing.
Developer-to-developer support.

If you're new to AppSignal, remember to ask us for some free stroopwafels! They taste almost as good as it feels to have all of your application's metrics at your fingertips 😉🍪

Write your own Domain Specific Language in Ruby

Tom de Bruijn — Mon, 06 Feb 2023 12:36:00 +0000

Let's do some metaprogramming in Ruby! This article assumes you have some familiarity with writing Ruby and are interested in learning how to use metaprogramming.

This article is also available as a video recording of a talk.

What's a DSL?

Let's start with some definitions on what we're going to be looking at. The abbreviation DSL stands for Domain Specific Language.

A DSL is a sub-language within a Ruby app (or any other programming language) that is specific to a certain domain: like a Ruby gem or part of an app. Without that gem or app, the DSL won't work. It's specific to that domain.

In Ruby there's a gem named Rake. This gem provides a DSL to create tasks to be run from the command line. A small example looks like this:

# Rakefile
task :hello do
  puts "Hello there"
end

You can call this task from the terminal with the following command. When called, it will perform the block we've given to the task method.

$ rake hello
Hello there

This is the power of a DSL. We don't need to know how to listen to arguments given to Ruby from the command line. Rake does this for us. We only need to define the commands.

Why use a DSL?

Configuration

One of the most common reasons to create your own DSL is for configuration, for an app or part of a gem. In Rails, the Application class contains configuration for the different gems that make up Rails. It's even possible to define your own custom config options specific to your app.

# config/application.rb
module MyApp
  class Application < Rails::Application
    config.load_defaults 7.0

    config.time_zone = "UTC"

    config.app.custom_option = "some value"
  end
end

Within the Application class, the config object can be used to set config options or load defaults, like shown above.

Automate writing code

A DSL can be used to abstract away a bunch of repetitive code. In a Rails app there can be many different HTTP endpoints the app listens to. To write a Ruby app that listens to these endpoints can require a lot of code. In Rails, we can declare several different routes with one method: resources

Rails.application.routes.draw do
  resources :posts do
    resources :comments
  end
  root "homepage#show"
end

The resources method will do several things:

create several endpoints like GET /posts, GET /posts/new, POST /posts, DELETE /posts, etc.;
nest endpoints, if they are nested within another resources method call's block, like in the example above for comments;
route the request data on the endpoint to the relevant controller.

This is all defined with one method call. We don't need to go through several files and methods to make sure these steps are all done for every route. This makes, what would be high churn code, more maintainable*.

*: At times, we may be swapping out a lot of code with less code, but that code is more complex because of the metaprogramming involved.

Writing your own Ruby DSL

The start of a DSL

The first thing that usually gets a DSL is configuration. A gem ships with a configuration class or a part of an app does. A new Config class gets created with some config options. (In this case a CLI tool that has options to print more information with verbose and a particular output format called documentation.)

class Config
  def initialize(options = {})
    @options = options
  end
end

config = Config.new(
  :verbose => true,
  :format => :documentation
)
# => #<Config:...
#        @options={:verbose=>true, :format=>:documentation}>

The above example is usually enough for a small component.

Let's look how we can make this DSL feel more like some of the DSLs we've seen. In the example below we don't configure the Config class with a Hash of options, but use method calls like the verbose= attribute writer.

Ruby has helpers for defining reader and writer methods on a class for instance variables. Calling attr_accessor :verbose in a class definition will define the verbose or verbose= methods.

class Config
  attr_accessor :verbose, :format
end

config = Config.new
config.verbose = true
config.verbose # => true
config.format = :documentation

pp config
# => #<Config:...
         @format=:documentation,
         @verbose=true>

While this is quick to set up, it has some downsides. Like before, we want to store all config options on a @options Hash instead of on their own instance variables. To export the config options as a Hash, we would need to export them all manually. Now there are more places that need to be updated the more config options are added. Easily forgotten, easily broken.

class Config
  # ...

  def options
    {
      :verbose => @verbose,
      :format => @format
    }
  end
end

pp config.options
# => {:verbose=>true, :format=>:documentation}

To make the Config class work with a Hash of options, we can define our own methods for each config option. In these methods we set the values on the @options Hash, using the key of the config option we want to set. Great! We now can export our configuration with ease.

class Config
  attr_reader :options

  def initialize
    @options = {}
  end

  def verbose=(value)
    @options[:verbose] = value
  end

  def format=(value)
    @options[:format] = value
  end
end

config = Config.new
config.verbose = true
config.format = :documentation

pp config.options
# => {:verbose=>true, :format=>:documentation}

With this setup, we run into a familiar problem. Every time a new config option is added we need to add another method for that config option. As more config options are added, the Config class grows and grows, becoming more difficult to maintain.

Dynamically define methods in Ruby

Instead of defining a method per config option, we can dynamically define these methods. Dynamically defining methods allow us to quickly define several methods that share almost the same implementation.

In the example below I've created a new class method, as indicated with the def self.def_options syntax (1). This method, "define options", receives an array of option names to define methods for (2). In the def_options method it will loop through these option names (3). In each iteration of the loop it will call the define_method method (4). With this method we can define a new method, like with the def <method name> syntax, but with a dynamic method name.

class Config
  def self.def_options(*option_names) # 1
    option_names.each do |option_name| # 3
      define_method "#{option_name}=" do |value| # 4
        @options[option_name] = value # 5
      end
    end
  end

  def_options :verbose, :format # 2

  def initialize
    @options = {}
  end

  def options
    @options
  end
end

config = Config.new
config.verbose = true
config.format = :documentation

pp config.options
# => {:verbose=>true, :format=>:documentation}

The define_method method receives the name of the method, and a block (4). This block will be the implementation of the method we are defining. In this case, it will call the @options instance variable to receive the Hash of options configured, and add a new value for the option name we've declared (5). The DSL syntax for the end-user remains the same.

Using Ruby blocks

In my opinion, no Ruby DSL is complete without blocks*. They look good for one, but they also provide a new scope of context to users of our DSL, in which our new DSL rules apply.

*: It's perfectly fine to have a DSL without blocks.

Config.configure do |config|
  config.verbose = true
  config.format = :documentation
end

In the next example, I've created a new class method called configure. To interact with blocks given to a method, we can use the yield keyword. If a method has been given a block, yield will call that block. If we give a value to yield, it will make it so that the block receives that value as an argument in the block, as is shown with |config| in the example above.

class Config
  def self.configure
    instance = new # Create a new instance of the Config class
    yield instance # Call the block and give the Config instance as an argument
    instance
  end

  # ...
end

Using instance_eval

We can make this shorter, and in my opinion, feel more like a DSL with its own block context. We can drop the |config| block argument and avoid having to call config. in front of every config option.

Config.configure do
  verbose true
  format :documentation
end

I've omitted the equals sign (=) from the example here. More on that later.

To make this work we go back to our configure class method. Instead of yield, we'll use instance_eval. (Eval may sound scary, and it has some things to look out for, but in this small example it's quite safe.)

To call instance_eval we need to pass in the block given to the method. We can't use yield for this. We need to explicitly specify the block argument of the configure method with &block. The key part being the ampersand (&) of the argument name &block, telling Ruby it's the block argument.

class Config
  def self.configure(&block)
    instance = new
    instance.instance_eval(&block)
    instance
  end

  # ...
end

(In Ruby 3.1 and newer the &block argument can also be written as only the ampersand &, the name is optional.)

When called on another object, like our Config class instance, instance_eval will execute that block within the context of the Config instance. The context of that block is changed from where it was defined to the Config instance. When a method is called within that block, it will call it directly on the Config instance.

Local variable assignment gotcha

Remember how in the example above I omitted the equals sign from the method names? When calling a method name ending with an equals sign in instance_eval, Ruby will interpret it as a local variable assignment instead. It will not call the method on the Config instance. That's how Ruby works, and we can't change that. I omitted the equals sign to make the DSL still work.
Config.configure do
  # This won't work
  verbose = true
  format = :documentation
end
It's possible to work around this by explicitly calling the methods on self., but this syntax becomes basically the same as using yield. Instead, I removed the equals sign from the method name.
Config.configure do
  # This works, but requires `self.` in front of every method call
  self.verbose = true
  self.format = :documentation
end

When to use yield or instance_eval?

I've shown two ways of calling blocks in Ruby. Which one should be used is up to the authors of the DSL. I have my personal preference for using instance_eval, but there are definitely scenarios where using yield works better.

When something needs to be configured, using a lot of methods with an equals sign (writer methods), using yield with a block argument is quite common.

Config.configure do |config|
  config.verbose = true # Set a value
end

When defining something or calling actions, I see the instance_eval approach used a lot.

Config.configure do
  load_defaults! # Perform an action
end

That doesn't mean one excludes the other. Gems like Puma use the instance_eval method for their configuration file.

# config/puma.rb
workers 3
preload_app!

Use the approach you feel works best for your DSL.

Share DSL behavior with modules

With the above methods you can start building your DSL. This gives us a good toolbox. As the DSL grows, or more DSLs get added to the domain, it's good to share behavior between the DSL classes.

For example, this domain has two separate configuration classes: a Config and an Output class. One configures how the gem works, the other how it outputs results to the terminal.

# Multiple configuration classes
Config.configure do
  enabled true
end

Output.configure do
  verbose true
  format :documentation
end

We don't want to repeat ourselves, so we move the DSL definition behavior to a module called Configurable. When that module is included the class method def_options is made available with the class definition.

class Config
  include Configurable

  def_options :enabled
end

class Output
  include Configurable

  def_options :verbose, :format
end

When the Configurable module is included, it adds the options method to the class. The module also extends the class it's included in, using the included callback in Ruby modules. In that callback, the class is extended with the ClassMethods module, defined inside the Configurable module. This module will add the class methods (def_options and configure) to the class. This is also how things like ActiveSupport::Concern work.

module Configurable
  def self.included(base)
    base.extend ClassMethods
  end

  module ClassMethods
    def def_options(*option_names)
      # ...
    end

    def configure(&block)
      # ...
    end
  end

  def options
    @options ||= {}
  end
end

With all the logic moved to the Configurable module, we now have a reusable module to create as many config classes as the domain needs.

Nesting DSLs

The Output class is really a sub domain of the Config class though. It's part of the gem config: the configurations should be nested within one another.

Config.configure do
  enabled true

  output do
    verbose true
    format :documentation
  end
end

Rather than Output being a top-level config class, we store it on the Config class. In the output method in the example below, the method creates a new Output instance. When a block is given, it will instance_eval the block on the @output object. The method will always return the created @output object so the sub-config can be accessed.

class Config
  include Configurable

  def_options :enabled

  class Output
    include Configurable

    def_options :verbose, :format
  end

  def output(&block)
    @output ||= Output.new
    @output.instance_eval(&block) if block_given?
    @output
  end
end

Module builder pattern

The Module builder pattern is a really neat design pattern in Ruby that allows us to do away with the two step process of including a module and then calling methods included by it. This pattern is described in more detail in The Ruby Module Builder Pattern by Chris Salzberg.

In the example below, a new ConfigOption module is used to include a dynamically defined module. For the end-user, the resulting DSL remains the same.

class Config
  include ConfigOption.new(:verbose, :format)
end

When ConfigOption.new is called, the desired config option names are given to the initialize method. Like before, we iterate over this list. Using define_method the necessary methods are defined on the module. It's possible to do so in this initialize method, because it's creating a new implementation of the ConfigOption module.

class ConfigOption < Module
  def initialize(*option_names)
    define_method :options do
      @options ||= {}
    end

    option_names.each do |option_name|
      define_method option_name do |value|
        options[option_name] = value
      end
    end
  end
end

For a much more in-depth look, I can highly recommend reading The Ruby Module Builder Pattern by Chris Salzberg.

Create DSL objects

Another step I recommend is to separate the behavior of the DSL itself and the actual app code that uses it, by creating DSL classes. These classes are only used when the end-user interacts with the DSL, like configuring a gem. When the configuration is done, the options are read from the DSL class and set on whatever config object the gem uses.

Like before, we have a Config class. When configured using Config.configure, it creates a ConfigDSL class (1). This ConfigDSL class will become the context of the block given to the configure class method.

class Config
  def self.configure(&block)
   dsl = ConfigDSL.dsl(&block) # 1
   new(dsl.options) # 2
  end

  attr_reader :options

  def initialize(options)
    @options = options
  end
end

class ConfigDSL
  include ConfigOption.new(:verbose, :format)

  def self.dsl(&block)
    instance = new
    instance.instance_eval(&block)
    instance
  end
end

After the configure block has been called, the Config class reads the options from the ConfigDSL and initializes a new instance of itself with these options (2).

With this approach the DSL that configures the gem is only used when the app starts. The app doesn't carry around all that extra weight of how the DSL works all the time. The separation of this behavior makes it easier to not accidentally call the configure DSL when it should not be available.

This approach also solves a downside of using instance_eval. When calling blocks this way, private methods are accessible inside the block. That's something we usually don't want to allow.

class Config
  def self.configure(&block)
    instance = new
    instance.instance_eval(&block)
    instance
  end

  private

  def secret_method
    "super secret method"
  end
end

Config.configure do
  # This should raise an error about calling
  # a private method, but it doesn't
  secret_method
  # => "super secret method"
end

Private methods can be accessed in blocks called using instance_eval, because the block is evaluated in the context of the instance. It's as if it's being run from a method within that object. With separate DSL classes you have to worry less about private methods that you don't want anyone to call.

Conclusion

Now that our toolbox is complete we can create our own Domain Specific Language using Ruby. We have the following tools at our disposal:

Use define_method to dynamically define new methods on classes.
Use Ruby blocks to give your DSL that real DSL feeling and create a context wherein your DSL shines.
- Use yield to return a DSL object as a block argument, or;
- Use instance_eval to change how to block works, and allow end-users to directly call methods on the new context of new block.
Use Modules to:
- share behavior between many classes, and;
- use the Module builder pattern to remove any logic of how the DSL is constructed from the class that uses the DSL.
DSL objects separate the logic of how the DSL works and how the rest of the gem works. Creating a better separation of responsibilities in your code.

Now go forth, and build your own DSL!

Calculating String length and width – Fun with Unicode

Tom de Bruijn — Thu, 16 Jun 2022 11:01:14 +0000

Let's calculate how string length in Rust¹! How many characters there really are and how much space these strings take up when displayed.

¹: This article may also apply to other languages. This article will only focus on strings with the Rust default UTF-8 encoding. There's an appendix for how it works in Ruby at the end of the article. I'm simplifying this content to keep the article short.

This post was first published on my own blog.

String.len()

The first function you'll probably come across is String.len()/str/len(), or length of string. Given the string "abc" it will returns the length of three. All looks good so far.

"abc".len() // => 3

That is, until we take a closer look at the docs for this function. It says the following:

Returns the length of this string, in bytes, not chars or graphemes. In other words, it might not be what a human considers the length of the string.

Source: String.len()

The Rust docs are giving us a warning here that it may not always return the number we'd expect. It will return the string length in bytes, and it sounds like not all characters are counted as one byte.

Let's try something that's not just plain "a" through "z", but something like a character with an accent.

"é".len() // => 2 bytes

We can see here that the result is a larger number than what we consider the string length to be. A lot of characters are comprised of multiple bytes. There are only so many characters we can make from an eight number byte, this is what ASCII is. To support all characters of all languages in the world in 256 possible different bytes wouldn't fit. Let's try another approach.

Chars.count()

Rust has a built-in "Chars" module we can use to split up the string into a list of characters, this gives a more accurate result.

"abc".chars().count() // => 3 characters
"é".chars().count()   // => 1 characters

When we ask the str.chars() method to give us a breakdown of what it considers characters we get a pretty good result. The character with the accent is seen as one character.

"é".chars() // => Chars(['é'])

We've learned that characters can be composed of multiple bytes. Relying on the string byte length for the actual length is not accurate enough.

Will Chars always return an accurate string length? This depends on your use-case. If you're looking for the number of characters in a string, probably yes, but also no. If you're looking for the actual string display width as rendered, the size it takes up on screen, then very much no.

If we look at the documentation again it will give us another warning:

It's important to remember that char represents a Unicode Scalar Value, and might not match your idea of what a 'character' is. Iteration over grapheme clusters may be what you actually want.

Source: str.chars()

Graphemes

What if the string being checked doesn't only contain numbers and letters, accents or not? Emojis are very popular nowadays and present in every kind of string. Emojis can have a byte size of much more than two bytes, but also consist of multiple characters even though it looks like one object.

// Person emoji
"🧑".len()           // => 4 bytes
"🧑".chars().count() // => 1 characters

// Woman scientist emoji
"👩‍🔬".len()           // => 11 bytes
"👩‍🔬".chars().count() // => 3 characters

// Family: Man, Man, Girl, Boy emoji
"👨‍👨‍👧‍👦".len()           // => 25 bytes
"👨‍👨‍👧‍👦".chars().count() // => 7 characters

The "woman scientist" emoji shown above takes up eleven bytes and three characters. The family takes up 27 bytes and seven characters, even though we only see one item in the string.

If we print the list of characters we'll get a better idea of what is happening.

"👩‍🔬".chars()
// => Chars(['👩', '\u{200d}', '🔬'])

"👨‍👨‍👧‍👦".chars()
// => Chars([
//   '👨',
//   '\u{200d}',
//   '👨',
//   '\u{200d}',
//   '👧',
//   '\u{200d}',
//   '👦'
// ])

Emoji can consists of multiple characters, that together construct another emoji and tell computers what to render. In the example above, we first see the "woman" emoji, then what is called a "Zero Width Joiner" character, and finally the microscope emoji. For the family we see the whole list of different genders and ages joined together in the same way.

The Zero Width Joiner character used by these combined emoji is, as the name describes, a Unicode character with zero width (which makes it invisible). This character is used to join together multiple emoji to make a new one.

These additional emoji characters do mess up our string length calculation. Luckily we can use the unicode-segmentation Rust crate to do more accurate character counting. We can ask the crate to split the string based on something called Graphemes and return the list of characters. A grapheme cluster in this context is a group of Unicode codepoints that consist of "user-perceived character", what we consider a character when we type it.

use unicode_segmentation::UnicodeSegmentation;

"abc".graphemes(true).count // => 3
"é".graphemes(true).count   // => 1
"🧑".graphemes(true).count() // => 1
"👩‍🔬".graphemes(true).count() // => 1
"👨‍👨‍👧‍👦".graphemes(true).count() // => 1

(The true argument given to graphemes function means we want to count it using the Unicode extended grapheme clusters and not the legacy clusters. Let's use the non-legacy cluster.)

Finally! We have an accurate string length. Right?

Yes. For the scenarios in which you want to know the number of characters in a string, I'd say yes.

But...

Graphemes widths

As you can see in the code example below, emoji aren't shown one character column wide. Most emoji are rendered with a display width of two columns, two other Latin characters, like A through Z. For illustration purposes I'll be using a monospaced font so that the latin characters have a predicable width.

// These two lines should render with the same width
"ab"
"🧑"

While the graphemes-count-method (shown in the previous section) will give you a correct string length in terms of how many characters you see, it's not always same as the horizontal space in columns a string takes up.

In my Lintje project (a Git linter) I ran into this string length vs display width problem for the program's rules and output formatting. The terminal output didn't align properly when a string in a Git commit message included a emoji. In the example output below, the ^ marker should be aligned with the last character of the string, but with an emoji in the string it wouldn't be properly aligned.

# Badly aligned output
String with a ❌ emoji
                    ^ End of sentence
# Well aligned output
String with a ✅ emoji
                     ^ End of sentence

For rules that checked string length, I decided to count the display width of how every character is rendered as the string length. If a string has a max length of 50 characters, because of readability and width constraints, it should not be allowed to jam that string full with 50 emoji. It would take up double the horizontal space of a string with the same length without emoji.

Using the unicode-width Rust crate we can calculate a more accurate string display width.

UnicodeWidthStr::width("🧑") // => 2
UnicodeWidthStr::width("👩‍🔬") // => 4
// Consists of 👩, a Zero Width Joiner and 🔬
UnicodeWidthStr::width("👨‍👨‍👧‍👦") // => 8
// Consists of multiple face emoji and Zero Width Joiner characters

That is, the display width that every character is defined as being part of each grapheme cluster in Unicode. As you can see, the emoji with a Zero Width Joiner characters are as wide as the number of emoji they join times two.

There are also "emoji" that only return a display width of one, not two, columns. These one column display width characters are considered to have a "narrow" display width. The two column wide characters are "wide". This difference is something you can see if your terminal and other applications, as it sometimes overlaps with the next character. This will not be visible in the browser you're reading this in.

"❤️".len()           // => 6 bytes
"❤️".chars().count() // => 2 characters
"❤️".chars()         // => Chars(['❤', '\u{fe0f}'])
UnicodeWidthStr::width("❤️") // => 1 display width

The heart emoji ❤️ is a "Heavy black heart" character ❤ from the dingbat collection combined with what's called a "Variation Selector-16" character. This is another invisible character in graphemes clusters to indicate certain characters should be displayed as their emoji counterparts.

And the actual width is?

Why is this happening? From what I understand the unicode-width library is following the Unicode reference to the letter, counting only the base character's display width and not the emoji variation. That results in output that I would not have expected. It's not something that can be fixed, unless you change something in Unicode first.

Unfortunately I don't have another function or library to call on to fix this and return what we see as the actual display width.

What I did in my Lintje project was first split up the string into graphemes clusters, then scan every sub string for Zero Width Joiner characters, and then only return "two" as the width. This is somewhat more accurate to what we see displayed, but it's not 100% accurate. It's good enough for my purpose right now. There probably are better ways out there to do this, but I fear that it will mean maintaining a list of the display width of all emojis. At least the exceptions with a display width of one. Let me know if you know of a better solution!

// Very simplified example
let string = "👨‍👨‍👧‍👦";
let mut width = 0;
let unicode_chars = string.graphemes(true);
for character in unicode_chars.into_iter() {
    if character.contains("\u{200d}") {
        width += 2;
    } else {
        width += UnicodeWidthStr::width(character);
    }
}
width // 2

The full implementation with additional checks for other modifier characters can be found in the Lintje GitHub project's utils module.

In conclusion

We've learned that what we humans consider string length is not the byte length. Characters can be multiple characters joined together, like emoji. Not all characters have the same display width. Some characters take up more horizontal space than others and some don't even take up any horizontal space.

So what should you use to calculate string length or display width?

Do you want the length of a string in bytes? Use String.len().
Do you want the number of characters in a string? The Chars.count() method is an option, but I suggest the next solution.
Do you want the visible number of characters in a string? Use str.graphemes(true).count().
Do you want the somewhat accurate width of the string? Use UnicodeWidthStr.width(str).
If you're like me an what the string display width calculated more accurately? You'll have to write something yourself I'm afraid.

Appendix: Ruby

I write a lot of Ruby code, so here's how the above applies to Ruby, as much as I looked into it.

When calling Ruby's String#length, it returns the length of characters like Rust's Chars.count. If you want the length in bytes you need to call String#bytesize.

"abc".length   # => 3 characters
"abc".bytesize # => 3 bytes
"é".length     # => 1 characters
"é".bytesize   # => 2 bytes

Calling the length on emoji will return the individual characters as the length. The 👩‍🔬 emoji is three characters and eleven bytes in Ruby as well.

"👩‍🔬".length   # => 3 characters
"👩‍🔬".bytesize # => 11 bytes

Do you want grapheme clusters instead? You're in luck: it's built-in to Ruby with String#grapheme_clusters.

"👩‍🔬".grapheme_clusters.length # => 1 cluster

To calculate the display with, we can use the unicode-display_width gem. The same multiple counting of emoji in the grapheme cluster still applies here.

require "unicode/display_width"

Unicode::DisplayWidth.of("👩‍🔬") # => 4
Unicode::DisplayWidth.of("❤️") # => 1

When not to use instance variables in RSpec

Tom de Bruijn — Tue, 02 Feb 2021 13:06:58 +0000

Using RSpec there is some confusion about the differences between let, let!, and instance variables in specs. I'd like to focus on how instance variables work in RSpec in combination with before :context blocks, and in what kind of scenarios you should and should not use them.

Why use instance variables?

The advantage of declaring instance variables in before :context (or before :all) blocks is that whatever value is assigned is only queried or calculated once for many specs. The before :context block is only executed once for all specs in that context. Those specs can use the instance variable without repeating the same setup for every spec, which should speed up the test suite.

Reading the above it might be tempting to put a lot of spec setup in before :context blocks. A fast test suite creates happy developers, right? But there's a downside to using instance variables in RSpec, which could make for very unhappy developers.

From the RSpec docs:

It is very tempting to use before(:context) to speed things up, but we recommend that you avoid this as there are a number of gotchas, as well as things that simply don't work.

[...]

Instance variables declared in before(:context) are shared across all the examples in the group. This means that each example can change the state of a shared object, resulting in an ordering dependency that can make it difficult to reason about failures.

The RSpec docs give us a warning about changing values of the instance variable. State can leak between specs using instance variables defined in a before :context block this way.

Let's look at some examples of specs using instance variables and in what scenarios in will break.

Specs sharing instance variables

In the example below specs only assert if the value of the instance variable matches the expected value. Since the instance variable is not changed, all the specs will pass. If the instance variable value took a long time to query or calculate we have saved that time for two specs in this file.

# spec/lib/example_1_spec.rb
describe "Example 1" do
  before :context do
    # Imagine this being a complex value to prepare
    # This block is only run once in the `describe "Example 1"` block
    @my_instance_variable = :my_value
  end

  it "spec 1" do
    expect(@my_instance_variable).to eql(:my_value)
  end

  it "spec 2" do
    expect(@my_instance_variable).to eql(:my_value)
  end
end

$ bundle exec rspec spec/lib/example_1_spec.rb --order defined
Finished in 0.00223 seconds
2 examples, 0 failures

(I'm using --order defined in the examples in this post so that the spec execution order is predictable and reproducible.)

But specs can be more complicated than this. They may pass the instance variable to some other part of the app, which modifies the given value. This is where things go wrong, what the RSpec docs warn us about.

Reassigning the instance variable

If changing the instance variable is the problem, let's reassign it and see what happens in other specs.

In the example below the "spec 1" spec changes the instance variable to test a slightly different scenario.

# spec/lib/example_2_spec.rb
describe "Example 2" do
  before :context do
    # Imagine this being a complex value to prepare
    @my_instance_variable = :my_value
  end

  it "spec 1" do
    @my_instance_variable = :new_value
    expect(@my_instance_variable).to eql(:new_value)
  end

  it "spec 2" do
    expect(@my_instance_variable).to eql(:my_value)
  end
end

$ bundle exec rspec spec/lib/example_2_spec.rb --order defined
Finished in 0.00223 seconds
2 examples, 0 failures

In this example "spec 2" does not fail, even though "spec 1"—which runs before "spec 2"—changes the instance variable. There was no need for us to reset the original value of the instance variable at the end of the spec even though we changed it.

The way that RSpec runs the specs ensures that every spec uses the original instance variables. Every spec in RSpec is its own Ruby class, in which the spec is performed. Before the spec class run, RSpec sets the instance variables from the before :context block on the spec class. When an instance variable is reassigned in a spec, it only reassigns it on that spec class instance. It doesn't not reassign the instance variable on the same scope as the before :context instance variables are stored, and so does not interfere with any other specs. An example of how this looks can be found later on in this post.

This behavior doesn't always quite work though. Let's see what happens if we use a bit more complex value. That way we know the limitations of using instance variables in RSpec.

Modifying instance variable values

In the next example the @my_instance_variable is assigned a more complex value: an array with multiple values. We will intentionally break the spec in this scenario.

In "spec 1" we're testing a slightly different scenario again, modifying the value before running the assertion. Instead of reassigning the variable we're adding a value to the array on @my_instance_variable.

# spec/lib/example_3_spec.rb
describe "Example 3" do
  before :context do
    @my_instance_variable = [:one, :two]
  end

  it "spec 1" do
    @my_instance_variable << :three
    expect(@my_instance_variable).to eql([:one, :two, :three])
  end

  it "spec 2" do
    expect(@my_instance_variable).to eql([:one, :two])
  end
end

$ bundle exec rspec spec/lib/example_3_spec.rb
Failures:

  1) Example 3 spec 2
     Failure/Error: expect(@my_instance_variable).to eql([:one, :two])

       expected: [:one, :two]
            got: [:one, :two, :three]

       (compared using eql?)

       Diff:
       @@ -1 +1 @@
       -[:one, :two]
       +[:one, :two, :three]

     # ./spec/lib/example_3_spec.rb:14:in `block (2 levels) in <top (required)>'

Finished in 0.01596 seconds (files took 0.10576 seconds to load)
2 examples, 1 failure

Failed examples:

rspec ./spec/lib/example_3_spec.rb:12 # Example 3 spec 2

Unlike before, the "spec 2" spec has now failed. It fails because the instance variable still has the value from the first spec. State has leaked from "spec 1" into "spec 2". Let's look at how the values from these instance variables have moved between specs.

How instance variables work in RSpec

To recap what we learned from the examples earlier:

Assigning instance variables in before :context means they'll only be assigned once, as the before :context block is only once run before all specs in the spec context.
Every spec in RSpec is performed as its own class, with its own scope. Instance variable from the before :context block are set on the spec class before it is performed.
Instance variable values do not leak between specs when the values are basic Ruby objects such as Symbols, numbers, etc.
Instance variable values do leak between specs when the values are more complex objects such as Arrays, Strings, Class instances, etc.

Let's take a closer look at how RSpec handles instance variables for spec classes to see how this could break in our test suite.

How RSpec handles instance variables

When RSpec runs specs in a context, it first runs the before :context blocks. After a before :context block is executed RSpec then stores the list of the instance variables on the class it creates for the context.

When RSpec then starts a spec in that context it creates a new class for that spec and sets instance variables of that context on the spec class.

Let's look at how this works using the same scenario from the second example, where we reassigned the instance variables.

class Context # RSpec context class
  def self.before_context_ivars
    # Where the `before :context` instance variables are stored
    @ivars ||= {}
  end

  def self.set_before_context_ivars_on(spec_instance)
    # Set instance variables from `before :context` on spec instance
    before_context_ivars.each do |name, value|
      spec_instance.instance_variable_set(name, value)
    end
  end
end

class Spec # RSpec spec class
  def run # During spec
    @var = 2 # Reassign instance variable
  end
end

# Mock a `before :context` block and
# store an instance variable on the Context class
Context.before_context_ivars[:@var] = 1 # Basic Ruby object value
# Initialize the spec class
spec_instance = Spec.new
# Set the `before :context` instance variables on the spec
Context.set_before_context_ivars_on(spec_instance)

puts "Context @var before spec:",
  Context.before_context_ivars[:@var].inspect

# Run the spec
spec_instance.run

puts "Spec @var:", spec_instance.instance_variable_get(:@var).inspect
puts "Context @var after spec:",
  Context.before_context_ivars[:@var].inspect

(Simplified example for demonstration purposes.)

Context @var before spec:
1
Spec1 @var:
2
Context @var after spec:
1

RSpec sets the original value of the instance variable on every spec class instance. Reassigning the instance variable in the spec class does not modify the value of the instance variable on the context. It only changes it on the spec class instance.

When the instance variable value was modified however, the value stored on the Context class is also modified, as it is still the same value.

# Using the same context class as from the previous example

# Mock a `before :context` block and
# store an instance variable on the Context class
Context.before_context_ivars[:@var] = [1, 2] # More complex Ruby object
# Initialize the spec class
spec_instance = Spec.new
# Set the `before :context` instance variables on the spec
Context.set_before_context_ivars_on(spec_instance)

puts "Context @var before spec:",
  Context.before_context_ivars[:@var].inspect

# Run the spec
spec_instance.run

puts "Spec @var:", spec_instance.instance_variable_get(:@var).inspect
puts "Context @var after spec:",
  Context.before_context_ivars[:@var].inspect

Context @var before spec:
[1, 2]
Spec @var:
[1, 2, 3]
Context @var after spec:
[1, 2, 3]

In this example we can see the instance variable's value has changed, because the value was modified, rather than the instance variable being reassigned. This value was modified in memory, also changing the value in the context instance variable storage, which causes the modified state to leak into other specs.

Pointers

What happens is, RSpec (or rather Ruby) is assigning pointers to the values in memory when it sets the instance variable values on the spec class instance. Variables in Ruby are pointers to a place in the application's memory. Assigning a value to another variable does not make a copy of it, but points to the same location in memory.

When RSpec sets the instance variables, it doesn't set a copy of the original Array value. Instead it sets the pointer to the Array value in memory. If the Array in memory has changed during the spec run, it will set not the original value for the next spec, but the modified Array instead. This is part of how Ruby works, this is not something RSpec can "fix". And which is why the RSpec docs warn us about using instance variables in before :context blocks.

Alternatives

To prevent state from leaking into other specs by modified values, it's possible to "freeze" objects in Ruby. If we freeze an Array, String or other object instance, Ruby will not allow any modifications.

var = [1, 2].freeze
var << 3
# Raises an error to prevent modification
# => FrozenError (can't modify frozen Array: [1, 2])

But this will be more difficult to do for larger objects with nested objects, as it only freezes the top object and not all nested objects.

var = [[1, 2], [4, 5]].freeze
var[0] << 3
puts var.inspect
# => [[1, 2, 3], [4, 5]] # The nested value was modified

Alternatively it's possible to deep clone or dup the object. The problem with this is that it will take up a lot more memory, as every object will be kept in memory multiple times, so I can't recommend it.

Conclusion

RSpec scoping instance variables between specs in classes is a great help from RSpec for basic Ruby objects, but this behavior shouldn't be relied upon for more complex Ruby objects such as Arrays, Strings, and other object instances.

If a spec is modifying an instance variable value, you can't be sure what the value of that instance variable will be in the next spec. State may leak to other specs, breaking them in unexpected ways. This will be especially difficult to track down when the specs are run in a random order each time.

Make sure that if you use instance variables, you absolutely do not modify any value set on the instance variable if you want a predictable and reproducible test suite. And that's something we should all want. I'm all for fast test suites, but what I like more is a stable test suite.

A big thanks to Benoit Tigeot for fact checking this article!

Git: Review changes before committing

Tom de Bruijn — Mon, 26 Oct 2020 12:52:32 +0000

Making a git commit can be as quick as git add --all && git commit -m "WIP". The problem with this approach is that we didn't check what changes we committed. Who knows what got committed that shouldn't have. Usually we find out when it's too late.

While working on an issue, I personally make a lot of changes that don't belong in the same commit. For that reason the approach of "now commit everything I have changed" rarely works for me.

Let's look at the things we don't want to commit first and then improve the way we make commits by reviewing what we commit first and how.

Things we don't want to commit

Some things that we don't want stored in git commits are:

App secrets
- Tokens, passwords, personal information, etc. These are things we don't want to commit to the git history, especially public repositories. A bot will scrape our hosting provider credentials and spin up a lot of machines on our dime before we know what happened.
Debugging code
- Some code may still be present in the project that helped debug an issue, such as a print/puts, a console.log, or a binding.pry/debugger-statement. While not necessarily harmful it creates a lot of output or hangs the process while running the app and test suites. It creates unwanted noise.

Removing these changes from commits isn't always as easy. Force pushing branches can mess up the team's flow and commits in public repositories don't disappear. Someone with a direct link can still access the old commits.

To go through the process of creating new app secrets, rolling those out across the app is very time consuming and cumbersome. Removing debugging code isn't as much trouble but it does require another commit, which adds noise to the git history. This doesn't help communicating in git.

Review what to commit

To avoid including things that shouldn't be committed and create better commits, let's look at ways to create commits.

This is the process I recommend going through when creating a commit:

Start with reviewing changes:
- Are we about to stage files we should not be staging, such as credential files?
- Are there debug statements in the changes we should remove first?
- Are there changes that are not required to fix the bug or add the feature? In a commit where a bug is fixed, don't include code style changes in an unrelated part of the code.
- Finally, this is also a review for you, the committer, to double check the changes that were made are actually all necessary and correct. Are there tests for every logic change? Did we add or update the docs?
Stage the changes that you approve and are necessary.
Only after this review, commit the changes.

The main idea of this is that we look at what files we stage, but more importantly, we review the changes in each file before we stage them.

Commit tools

Let's look at some tools that help with creating git commits. These are alternatives to git add --all command from the beginning of this post, which stages all changes: modifications, additions and deletions.

There are some basic tools that git provides us and there are third party tools we can use to make staging specific changes easier.

Git add file

Using the git add command it's possible to add one file at a time with git add path/to/file.md. This makes sure we don't add files we shouldn't commit, such as secret files, which is good.

$ git status
On branch main
Changes not staged for commit:
  modified:   path/to/file.md

$ git add path/to/file.md

$ git status
On branch master
Changes to be committed:
  modified:   path/to/file.md

But what about the changes in those files? We still don't know exactly what changes are being staged and will end up getting committed. We need a more detailed staging process.

Interactively stage changes

To review and select changes within a file we want to stage we can use git add --patch. This opens a basic selection interface in the terminal with which we can select "hunks" to be staged. Hunks are groups of changed lines in a file that git can easily stage and unstage.

$ git add --patch
diff --git a/app/models/blog_post.rb b/app/models/blog_post.rb
index 8b13789..257cc56 100644
--- a/app/models/blog_post.rb
+++ b/app/models/blog_post.rb
@@ -154,6 +154,12 @@ class BlogPost < ActiveRecord::Base
   end
+
+  def published?
+    binding.pry
+    Time.now >= published_at
+  end

   def active?
(1/10) Stage this hunk [y,n,q,a,d,e,?]?

In the example above we run git add --patch and it prompts us with the question if we want to "Stage this hunk". What's useful here is that we can see the "hunk" it's referring to in the preview above it, and review if it's any code we actually want to commit.

Choosing "y" for "yes", or "n" for "no", we can step by step stage changes or skip them. Git will take us through all the changes one "hunk" at a time until we've reviewed them all.

But maybe we spot an issue with the hunk. We can "q" for "quit" and open the file in our editor to fix the issue and then restart the process by running git add --patch again.

This interface is quite limiting. If we don't agree with git's hunk boundaries we can't easily change them. It's not very easy to only stage one line or omit a line. If we stage the hunk in the example above we also commit some debugging code that will break things for us later.

Git GUI

A git GUI can help give us more control over what we commit. Apps that give us humans a more easy to digest interface for git and may even help with more advanced tools such as merge conflict resolution and rebasing.

Git GUIs are great! Don't let anyone tell you otherwise.

When making commits it's important to have a view of the current changes, stage them interactively, and commit knowing we only commit those changes that were selected. A GUI can display very detailed changes in a concise way, and the selection process for staging changes is more precise.

Personally I use tig most of the time in the terminal, and Fork any time I run into tig's limitations. To get started I recommend Fork, or a similar GUI app, as it's a very complete Git tool. Both tools give great overviews of git history and staged and unstaged files. It's possible to stage entire files in one go, or deep dive and only add the one specific line we want to stage and nothing else.

There are many other git GUIs, and no one tool is the best or only one you're allowed to use. Experiment, try them out, and use the tool that works best for you.

An example commit flow

When I am making a commit I often use tig in the terminal, but for convenience I'll use Fork in this example as it will look the same for everyone.

First I open the app, then I'm presented with the git history. After opening the "Local Changes" view I get an overview of all uncommitted changes.

In this example I want to stage the env_helpers.rb file, but not all changes. There are changes in the same file for another feature I'm working on. If I stage the entire hunk I also stage the code I don't want to include. I could open the file in my editor again and remove the line temporarily, but then I should remember to restore it again later. (I'm going to forget that, aren't I?)

This is the process I go through in the video:

In the left panel's "Unstaged Changes" box I look for the file I've just made changes to and want to commit.
I open the file by clicking it and see the file changes in the right panel.
There's a change on a line I do not want to commit, as indicated by the "TODO" comment.
By selecting the line I do want to commit the Fork app gives me context specific actions to "Stage" the changes.
I press the "Stage" button and only the selected change gets staged. The remaining unstaged file change is the line with the "TODO" comment I do not want to commit.
I click on the file in the "Staged Changes" view in the left panel. Here I see the changes the one line change I just staged and nothing else. If these are all the changes I need, I can now commit them without also committing the unstaged changes.

Conclusion

I hope this peek into how I review my changes before committing helps you make better commits. It's a process to keep being aware of. When I haven't kept a close eye on what I commit, I have included and pushed things I shouldn't have. So keep an eye on it and let's make better commits together!

Review the changes you stage so you don't accidentally include any app secrets, debugging code, personal information, or anything else that shouldn't be committed.

Try out a GUI or any of the other tools I've listed in this post and let me know which is your favorite!

Retry brittle tests until they fail

Tom de Bruijn — Wed, 02 Sep 2020 13:27:32 +0000

Brittle tests are tests that only fail some of the time. Due to this it can be difficult to reproduce the scenario that's causing it to fail.

These brittle tests can fail because of all sorts of things, some of which are:

Randomness in tests, or lack thereof. For example: the same numbers are generated twice, causing conflicts for object IDs in the database or memory.
Test execution order. For example: tests that are run first are affecting the behavior of other tests. Test state leaks into other tests. For this scenario I love using RSpec bisect.
App bugs. Actual app specific bugs that only occur some of the time. What's causing it? Only way to find out is to get a failing test and start debugging.

We want to fix those brittle tests, but the problem with debugging brittle tests is that they never fail when you want them to. A test can run without failure a 100 times, and only fail the 101th time. To manually retry it 101 times is very time consuming and not a lot of fun.

What I found myself doing is squeezing a while-statement around the command I wanted to keep retrying, but that got cumbersome and a typo was easily made. Instead, let's use a small executable for that.

Retry executable

When I'm not sure what the exact reason for the "brittleness" of a test is, I grab my "retry until fail" helper executable, until-fail for short. This is a very small wrapper around a given command that keeps repeating until it encounters a failure.

Retry a command until it fails

Usage:

  $ until-fail true
  # Will repeat forever

  $ until-fail false
  # Fails at the first iteration and breaks out of the retry loop

  $ until-fail ruby -e "rand(0..1) == 1 ? (puts 'failed'; exit(1)) : (puts 'success')"
  # Fails randomly and breaks out of the retry loop when it fails

The until-fail executable repeats the given command, in our case a test, until it runs into the failing scenario that we want to investigate further. This allows us to debug the brittle test while it's in a broken scenario. With that new information we can hopefully fix that test.

Combined with logging or print statements to provide more information about the context of the test failure, it should now be easier to debug the brittle test.

$ until-fail ruby tests/brittle_test.rb
Retry #1
DEBUG: User#accepted_terms_and_conditions == true
Success!

Retry #2
DEBUG: User#accepted_terms_and_conditions == true
Success!

...

Retry #100
DEBUG: User#accepted_terms_and_conditions == true
Success!

Retry #101
DEBUG: User#accepted_terms_and_conditions == false
Failure/Error: expect(signed_up_user_names).to include("Tom")
  expected signed up users to include "Tom"

Where to find it

The script is available in this gist.

It's a small Bash executable you can run in your shell, tested with Bash and ZSH.

Download the file and run chmod +x until-fail on it to make it executable. Move it to a location specified in your $PATH so it can be run from any location by calling until-fail and passing in the command to retry.

$ until-fail ruby some_file.rb
$ until-fail ruby tests/brittle_test.rb

See also the start this post for more information how to create your own executables.

In combination with pry

I use this automatic retry method in combination with a well placed pry statement in Ruby to open a console when the error occurs.

$ until-fail ruby tests/brittle_test.rb
...

Retry #101
From: /path/to/project/tests/brittle_test.rb:11 :

     7: it "includes newly signed up user" do
     8:   begin
     9:     expect(signed_up_user_names).to include("Tom")
    10:   rescue Exception => e
 => 11:     binding.pry
    12:     raise e # Reraise the failure so the retry script stops
    13:   end
    14: end

I use rescue Exception, because RSpec raises an Exception on assertion failures. The rescue is only temporarily, we should remove it when the brittle test has been fixed.

Now I can run the test command wrapped in the until-fail helper, and walk away from the computer. When I come back some time later, hopefully a pry console is ready for me to start debugging the scenario that fails.

I use this little until-fail helper a lot while debugging brittle tests. It has certainly made it easier reproduce brittle tests locally. I've removed randomness in tests, cleaned up state between tests, and a variety of app specific scenarios that were the cause of the brittle test.

Try it out and let me know if it has helped you!

Git is About Communication

Tom de Bruijn — Thu, 16 Jul 2020 09:47:42 +0000

An SCM such as Git is more than just a database for source code. It's not only the thing you need to interact with to get code to production, but also a log of changes on a project.

It's not just the last couple of weeks of commits that are worth looking at. Any commit remains relevant weeks, months and years later.

A commit serves multiple purposes. The first one is to explain a change during its review and the second is to explain a change to a future reader. It all revolves around communication.

Communication

Git is about talking to your team. It's about talking to yourself, from the future. What about that colleague who hasn't even joined the team yet? Yes, them too.

Git is a great resource to document why a change was made and the considerations that were part of the process of creating the commit. It's the documentation of how the project evolved.

An Example: Using git blame in Practice

Do you use git blame? Is it useful or is it a bunch of gibberish commits, including classics like: "WIP", "Fix bug", "update tests", "Move method", or my favorite: "..." (Yes, it's just three dots)?

After some digging, you've found the change that introduced a bug, but you are no further in finding why the change was made in commit "WIP". You ask the person who authored it for more information, but they may not remember why the change from a year ago was necessary or they may no longer work at the company. So you're going to have to assume things about why a change was made and work off of that. That should not be necessary.

An Example: Pull Request Reviews

When you or your team are reviewing a Pull Request, the same thing is happening, but you can ask why a change was made while it's still fresh in their memory. But you first need to ask. To make a good review, the reviewer also needs to know if and which alternative solutions (if any) were considered and why. We need to make better commits to make that possible.

Communication in Commits

Why "The why" is Important

A commit with only a subject says very little. At best it tells you which change was made, which may be useful to tell what code was changed, but the reason why the change was made is missing.

Example of a commit without a message

Explaining why a change was made is a good moment to reflect on the process that got you there. Was the change made because it was the most logical choice, or were there alternative approaches, and why were these not chosen?

If anyone else ever looks back on the commit, they can see if the restrictions a change was written under still apply or not, or how a certain bug slipped in under the assumptions a change was written.

I use writing commit messages to rubber duck problems. When I'm stuck on a change, rather than grab my rubber ducky, I explain it in writing. This way, I also immediately have my commit message for my team.

Explaining the "why"

To explain the "why" of why a commit was made, the commit message can explain the bug that's occurring that prompted a fix, along with its solution. This way, we have the reason for the commit as well as how the issue was fixed.

For example, there is a method that needs to be called with an Integer value. In the production app, an invalid argument error is reported on the line the method is called. Upon closer inspection, it is found that the method is sometimes called with a String value.

The quickest commit to write would be something along the lines of "Fix method call", but this commit without a message doesn't explain the actual problem.

A commit should explain the situation that causes the error⁠—some user input isn't cast to an Integer⁠. And how it was fixed⁠—the controller action "update" will now cast the user input to an Integer.

# Subject
Fix Ticket#set_priority being called with Strings

# The Scenario
When Ticket#set_priority was called from the TicketsController the priority
value was sometimes a String type. This caused an `InvalidArgumentError`:

InvalidArgumentError: Invalid argument type given `String`, expected `Integer`
  1: from /app/lib/models/ticket.ext:23 in `set_priority`
  2: from /app/lib/controllers/tickets_controller.ext:45 in `update`

# How
The `user_params` method will now cast all `priority` params to an Integer
before sending it to the Ticket model.

# Alternatives
<Colleague A> and I chose to cast it to an Integer in the `user_params`
method so that all other actions in the controller can also be able to use
this logic. This way, not all the methods need to implement their own casting,
making for a more fragile setup, something that's easy to miss when a new
action is added.
This allows us to clean up the `new` and `create` methods as well.

We didn't want to move the casting to the `set_priority` method because this
controller is the only place it can ever be called with a String.

# Additional sources
Fixes #1234

Communication in Pull Requests

The Format

When a Pull Request on GitHub is created with only one commit, it prefills the Pull Request title and description with the commit subject and message. This is not by accident. This way, the important information of the Pull Request is right there at the top. That's not to say that a good Pull Request only has one commit, oh no, but it does help a lot with creating smaller Pull Requests.

Example of Pull Request with a single commit

If a Pull Request has multiple commits, I advise you to use the same format. Repeating all the commits in the Pull Request's description as separate headings.

Example of Pull Request with multiple commits

I use the important commit's subject as the Pull Request title or write a brand new one if it helps identify all the combined changes.

The Content

There's very little additional context a Pull Request needs after a good title and description. Commits should contain the important bits of a Pull Request. If more context needs to be added to the Pull Request body, it should have been part of one of the commits.

This also prevents a lot of back and forth between author and reviewers to explain the changes in a Pull Request, speeding up the review time.

Keep Pull Requests Small

If a set of changes start to become large and difficult to review, send in smaller chunks as Pull Requests early to get feedback on the changes. This way you know you're on the right track.

If you find yourself refactoring code while working on a change, send in a separate Pull Request for the refactor if possible. This way, the Pull Requests are easier to review because their size is smaller and focus only on one type of change. Your team can get started reviewing the refactor while you work on finishing the other changes.

Nobody truly likes to review Pull Requests with 20 commits or 20,000 line changes. It's difficult to find the important changes between all the noise.

Also, make sure every Pull Request can be merged independently, so it doesn't have to stay open until the entire change is done.

Present a Tidy History

A reviewer doesn't need to experience the exact same process you went through to get to the final changes. If you first went for approach A and halfway through, switched to approach B, do not include both A and B, but only the final result. But do document the alternatives and reasons why approach B was chosen in the commit message.

Rebase your commits to present a tidy history with just the necessary changes. Nobody needs to see when you merged the upstream develop branch back into your branch. Or when you signed off for the day and committed all your changes into "WIP" to 'back it up' on a remote server. This is noise that hides changes in undescriptive commits, and why they were made.

# History as it was logged during development

* WIP
* Add button to edit user profile
* Merge remote-tracking branch 'origin/develop'
* Refactor button component
* Fix tests

# Tidy history ready for review

* Refactor button component
* Add button to edit user profile

Any additional commits as a result of reviews are of course fine, as they communicate the changes that were made as part of the review. Allowing reviewers to see what changes were made based on their feedback. (So no "Processed feedback" or "Update path/to/file.ext" commits.)

In Conclusion

Git is about communicating with your team. Document why you made a change in your git commits. Your team, your future self, and your future colleague will thank you.

Make sure all the context needed is present in commits and Pull Requests in terms of why a change was made, how and what alternatives there were.

Improve Pull Request review turn around time by keeping them small. And make them better by understanding why a change was necessary and review it with that thought in mind.

#to_s or #to_str? Explicitly casting vs. implicitly coercing types in Ruby

Tom de Bruijn — Tue, 25 Sep 2018 12:52:55 +0000

Type coercion is the changing of an object's type into another type, together with its value. For example, changing an Integer into a String with #to_s or a Float into an Integer with #to_i. The perhaps lesser-known #to_str and #to_int methods some objects implement do the same at first glance, but there are some differences.

In this edition of AppSignal academy, we'll dive into explicitly casting and implicitly coercing types in Ruby, while briefy touching on typecasting actors. We'll cover the differences between both methods, and discuss how they're used.

Let's first look at how we usually coerce values to different types in Ruby with explicit casting helpers.

Explicit Casting Helpers

The most common casting helpers are #to_s, #to_i, #to_a and #to_h. These are explicit casting methods. They help us easily transform a value from one type to another.

The explicit helpers come with a clear promise. Whenever #to_s is called on an object, it'll always return a string, even if the object doesn't really convert to a string well. It's like casting Michael Keaton as Batman. You'll get a batman, even if a comedy actor isn't especially suited for the role.

Ruby offers these helper methods on almost any basic object in the Ruby standard library.

:foo.to_s # => "foo"
10.0.to_i # => 10
"10".to_i # => 10

These methods, especially #to_s, are implemented on most basic types in Ruby. While the casting almost always returns a value, the result may not be what we expect.

"foo10".to_i          # => 0
[1, 2, 3].to_s        # => "[1, 2, 3]"
{ :foo => :bar }.to_s # => "{:foo=>:bar}"
{ :foo => :bar }.to_a # => [[:foo, :bar]]
Object.to_s           # => "Object"
Object.new.to_s       # => "#<Object:0x00007f8e6d053a90>"

Calling the #to_s, #to_i, #to_a and #to_h helpers forces any value to the selected type. They return a representation of the type it's coerced to regardless of what happens to the value.

Implicit Coercion Methods

Calling type casting methods on values that do not act like the type we are casting to can cause errors or loss of data. Ruby also offers implicit coercion methods which only return a value when objects act like the type. This way we can be sure that the value acts like the type we want. These implicit coercion methods are #to_str, #to_int, #to_ary and #to_hash.

Implicit coercion is like casting Leonard Nimoy as any role but Spock. They'll work if the character is close enough to Spock, but fail if they're not. The #to_str helper tries to convert to a string, but will raise a NoMethodError if the object doesn't implement the method and can't be implicitly coerced.

10.to_int                           # => 10
10.0.to_int                         # => 10
require "bigdecimal"
BigDecimal.new("10.0000123").to_int # => 10

# Unsuccessful coercions
"10".to_int             # => NoMethodError
"foo10".to_int          # => NoMethodError
[1, 2, 3].to_str        # => NoMethodError
{ :foo => :bar }.to_str # => NoMethodError
{ :foo => :bar }.to_ary # => NoMethodError
Object.to_str           # => NoMethodError
Object.new.to_str       # => NoMethodError

We can see that Ruby is a bit more strict now in what it does and doesn't coerce to the requested types. If the coercion is not possible, the #to_* method is not implemented on the object and calling it raises a NoMethodError.

When using implicit coercions, e.g. #to_str, we ask the function to return a String object, only if the original type also acts like a String. For this reason, #to_str is only implemented on String in the Ruby Standard Library.

How Ruby Uses Implicit Coercion

Other than being more precise in what we're asking for during a coercion, what else is implicit coercion useful for? Turns out Ruby uses implicit coercions itself in a fair bit of scenarios. For instance, when combining objects with +.

name = "world!"
"Hello " + name # => "Hello world!"

# Without #to_str
class Name
  def initialize(name)
    @name = name
  end
end
"Hello " + Name.new("world!") # => TypeError: no implicit conversion of Name into String

Here, we see Ruby raise a TypeError since it can't do an implicit conversion from the Name type to a String.

If we implement #to_str on the class, Ruby knows how to coerce the Name type.

# With #to_str
class Name
  def to_str
    @name
  end
end
"Hello " + Name.new("world!") # => "Hello world!"

The same works for Arrays and #to_ary.

class Options
  def initialize
    @internal = []
  end

  def <<(value)
    @internal << value
  end
end

options = Options.new
options << :foo
[:some_prefix] + options # => TypeError: no implicit conversion of Options into Array

class Options
  def to_ary
    @internal
  end
end
[:some_prefix] + options # => [:some_prefix, :foo]

But #to_ary is used in more scenarios. We can use it to destructure an Array into separate variables.

options = Options.new
options << :first
options << :second
options << :third
first, second, third = options
first  # => :first
second # => :second
third  # => :third

It also does conversion of the object into block parameters.

[options].each do |(first, second)|
  first # => :first
  second # => :second
end

There are more scenarios where the implicit coercion methods are used, such as #to_hash with **. This coerces the value to a hash with #to_hash before passing it to the parse_options method.

class Options
  def to_hash
    # Create a hash from the Options Array
    Hash[*@internal]
  end
end

def parse_options(opts)
  opts
end

options = Options.new
options << :key
options << :value
parse_options(**options) # => {:key=>:value}

Enforcing Types

Ruby also offers more resilient coercion methods when the type is of an unknown type and we want to make sure we get the correct type. There's one for every basic type (String(...), Integer(...), Float(...), Array(...), Hash(...), etc.).

String(self)       # => "main"
String(self.class) # => "Object"
String(123456)     # => "123456"
String(nil)        # => ""

Integer(123.999)   # => 123
Integer("0x1b")    # => 27
Integer(Time.new)  # => 1204973019
Integer(nil)       # => TypeError: can't convert nil into Integer

The String(...) method first tries to call #to_str on the value, and when that fails, it calls its #to_s method. Not all objects define a #to_str method, therefore checking with both the implicit coercion (#to_str) and explicit (#to_s) casting methods increases the chances that the String conversion will work and you'll get the value you want. By first calling for implicit coercion we're more likely to get a result that has the same value but is of the coerced type, and not something like "#<Object:0x00007f8e6d053a90>".

class MyString
  def initialize(value)
    @value = value
  end

  def to_str
    @value
  end
end

s = MyString.new("hello world")
s.to_s    # => "#<MyString:0x...>"
s.to_str  # => "hello world"
String(s) # => "hello world"

You should only implement the implicit casting methods for objects that act like the to be coerced type, e.g. #to_str for your own String class.

Other than first trying implicit coercion, the String(...) helper also checks the returned type. #to_str is just a method which can return any type of value, even non-Strings. To ensure we get a value of the requested type String(...) raises a TypeError if the types don't match.

class MyString
  def to_str
    nil
  end
end

s = MyString.new("hello world")
s.to_s    # => "#<MyString:0x...>"
s.to_str  # => nil
String(s) # => "#<MyString:0x...>"

Here, we can see that Ruby ignores the result of #to_str because it returned nil, which is not of the String-type. Instead, it falls back to the #to_s result.

If #to_s also returns nil and thus isn't of the correct type, String(...) will raise a TypeError.

class MyString
  def to_str
    nil
  end

  def to_s
    nil
  end
end

s = MyString.new("hello world")
s.to_s    # => nil
s.to_str  # => nil
String(s) # => TypeError: can't convert MyString to String (MyString#to_s gives NilClass)

While they may be more reliable in enforcing type coercion, note that the casting helper methods (String(...), Integer(...), etc.) are usually a bit slower as they need to perform more checks on the given value.

In Conclusion

When you want to make sure you're dealing with the right type of data for an object, type coercion is a useful process. In this post, we refreshed our knowledge of explicit casting helpers like #to_s, #to_i, #to_a and #to_h. We also looked at instances when implicit helpers like #to_str, #to_int, #to_ary and #to_hash are useful and how they're used by Ruby itself.

We hope you found this type coercion overview useful and how you found the actor typecasting analogy. As always, let us know if there's a topic you'd like us to cover. If you have any questions or comments, don't hesitate to leave a comment.

Rails Collection Caching

Tom de Bruijn — Tue, 14 Aug 2018 14:12:39 +0000

We've previously looked at fragment caching in Rails on AppSignal Academy. This greatly improves the performance of views by caching smaller pieces of them. When caching partials, we have the added benefit of being able to reuse them elsewhere in our views at little cost.

This works well for small collections, but problems quickly arise on larger collections. In this article, we'll take a look at how Rails collection caching works and how we can use it to speed up the rendering of a large collection.

Rendering a Collection

Let's start with a small controller that loads the last 100 posts for our blog's index page.

class PostsController < ApplicationController
  def index
    @posts = Post.all.order(:created_at => :desc).limit(100)
  end
end

To render these posts in the view, we loop over the @posts instance variable.

<!-- app/views/posts/index.html.erb -->
<h1>Posts</h1>

<div class="posts">
  <% @posts.each do |post| %>
    <div class="post">
      <h2><%= post.title %></h2>
      <small><%= post.author %></small>

      <div class="body">
        <%= post.body %>
      </div>
    </div>
  <% end %>
</div>

Upon requesting this page, we see the posts being fetched from the database and the view being rendered. With only 32 milliseconds spent in the view layer, this page is pretty fast.

Started GET "/posts"
Processing by PostsController#index as HTML
  Rendering posts/index.html.erb within layouts/application
  Post Load (1.5ms)  SELECT  "posts".* FROM "posts" ORDER BY "posts"."created_at" DESC LIMIT ?  [["LIMIT", 100]]
  ↳ app/views/posts/index.html.erb:4
  Rendered posts/index.html.erb within layouts/application (19.4ms)
Completed 200 OK in 37ms (Views: 32.4ms | ActiveRecord: 2.7ms)

Rendering a Collection with Partials

Next, we want to use the post element in another view, so we move the post HTML to a partial.

<!-- app/views/posts/index.html.erb -->
<h1>Posts</h1>

<div class="posts">
  <% @posts.each do |post| %>
    <%= render post %>
  <% end %>
</div>

<!-- app/views/posts/_post.html.erb -->
<div class="post">
  <h2><%= post.title %></h2>
  <small><%= post.author %></small>

  <div class="body">
    <%= post.body %>
  </div>
</div>

Started GET "/posts"
Processing by PostsController#index as HTML
  Rendering posts/index.html.erb within layouts/application
  Post Load (1.2ms)  SELECT  "posts".* FROM "posts" ORDER BY "posts"."created_at" DESC LIMIT ?  [["LIMIT", 100]]
  ↳ app/views/posts/index.html.erb:4
...
  Rendered posts/_post.html.erb (0.1ms)
  Rendered posts/_post.html.erb (0.1ms)
  Rendered posts/index.html.erb within layouts/application (205.4ms)
Completed 200 OK in 217ms (Views: 213.8ms | ActiveRecord: 1.7ms)

With 213 milliseconds spent on the view layer, you can see that the render time has increased substantially. This is because a new file (the partial) needs to be loaded, compiled and rendered for every post. Let's briefly look at how we can improve the render time with fragment caching.

Fragment Caching

As described in the fragment caching article, we'll use the cache helper in the view around the render call. In this way, we'll cache the rendering of the partial for every post.

<!-- app/views/posts/index.html.erb -->
<h1>Posts</h1>

<div class="posts">
  <% @posts.each do |post| %>
    <%= cache post do %>
      <%= render post %>
    <% end %>
  <% end %>
</div>

Started GET "/posts"
Processing by PostsController#index as HTML
  Rendering posts/index_with_partial_caching.html.erb within layouts/application
  Post Load (1.4ms)  SELECT  "posts".* FROM "posts" ORDER BY "posts"."created_at" DESC LIMIT ?  [["LIMIT", 100]]
  ↳ app/views/posts/index.html.erb:4
...
Read fragment views/posts/index.1ms)
  Rendered posts/_post.html.erb (0.1ms)
Write fragment views/posts/index.1ms)
Read fragment views/posts/index.5ms)
  Rendered posts/_post.html.erb (0.1ms)
Write fragment views/posts/index.1ms)
  Rendered posts/index.html.erb within layouts/application (274.5ms)
Completed 200 OK in 286ms (Views: 281.4ms | ActiveRecord: 2.4ms)

The first request won't be that much faster, because it still needs to render every partial the first time around and store it in the cache store.

Started GET "/posts"
Processing by PostsController#index as HTML
  Rendering posts/index.html.erb within layouts/application
  Post Load (2.2ms)  SELECT  "posts".* FROM "posts" ORDER BY "posts"."created_at" DESC LIMIT ?  [["LIMIT", 100]]
  ↳ app/views/posts/index.html.erb:4
...
Read fragment views/posts/index.1ms)
Read fragment views/posts/index.1ms)
  Rendered posts/index.html.erb within layouts/application (63.8ms)
Completed 200 OK in 78ms (Views: 75.5ms | ActiveRecord: 2.2ms)

In subsequent requests, we see that the time spent in the view is considerably lower - from 286 milliseconds down to 78 milliseconds. Yet, it's still a lot slower than what we got with our original code - it's almost twice as slow.

Note: If you're not seeing the "Read/Write fragment" lines in your logs, be sure to enable fragment cache logging in your development environment, which is set to false by default on Rails 5.1 and above:

# config/environments/development.rb
config.action_controller.enable_fragment_cache_logging = true

Collection Caching

In Rails 5, a lot of work was done to make collection caching faster. To leverage these improvements, we'll need to change our view code. Instead of calling the cache helper ourselves, we can ask Rails to render an entire collection and cache it at the same time.

<!-- app/views/posts/index.html.erb -->
<h1>Posts</h1>

<div class="posts">
  <%= render partial: :post, collection: @posts, cached: true %>
</div>

Note the render @collection, cached: true shorthand won't work for this caching speed improvement.

Started GET "/posts"
Processing by PostsController#index as HTML
  Rendering posts/index.html.erb within layouts/application
  Post Load (1.4ms)  SELECT  "posts".* FROM "posts" ORDER BY "posts"."created_at" DESC LIMIT ?  [["LIMIT", 100]]
  ↳ app/views/posts/index.html.erb:4
  Rendered collection of posts/_post.html.erb [0 / 100 cache hits] (28.2ms)
  Rendered posts/index.html.erb within layouts/application (46.6ms)
Completed 200 OK in 64ms (Views: 59.9ms | ActiveRecord: 2.0ms)

On the first request, we can already see a large improvement in time spent on the view layer. This is because Rails now prepares in advance, the partial being used for the entire collection, rather than for each post separately.

Started GET "/posts"
Processing by PostsController#index as HTML
  Rendering posts/index.html.erb within layouts/application
  Post Load (1.3ms)  SELECT  "posts".* FROM "posts" ORDER BY "posts"."created_at" DESC LIMIT ?  [["LIMIT", 100]]
  ↳ app/views/posts/index.html.erb:4
  Rendered collection of posts/_post.html.erb [100 / 100 cache hits] (19.2ms)
  Rendered posts/index.html.erb within layouts/application (26.5ms)
Completed 200 OK in 37ms (Views: 35.7ms | ActiveRecord: 1.3ms)

In subsequent requests, we see even more improvement - from 64 milliseconds down to about 35 milliseconds. A big speed improvement for the entire collection is made here by Rails optimization for collections. Instead of checking the availability of a cache for every partial, Rails checks all cache keys of the collection at the same time, saving time querying the cache store.

An added benefit of this caching helper is the summarized logging of the collection. In the first request, none of the cache keys were found [0 / 100 cache hits], but in the second request, they were all found [100 / 100 cache hits].

After updating some of the objects in the database, we can even see how many keys were stale.

Rendered collection of posts/_post.html.erb [88 / 100 cache hits] (13.4ms)

There's much speed improvement to gain with this optimized collection rendering and caching. An even bigger difference will be made when rendering larger collections. Unless you need customized views for your collection, this optimized strategy is the way to go for your Rails apps. At AppSignal, we managed to significantly speed up one of our admin views that was rendering thousands of records, in this way.

Have any questions about caching collections in Rails? Please don’t hesitate to leave a comment! If you have any comments regarding the article or if you have any topics that you'd like us to cover, then please get in touch with us.