DEV Community: Avdi Grimm

That’s So Fetch

Avdi Grimm — Fri, 06 Nov 2020 19:41:39 +0000

It’s time for another dip into the RubyTapas archives! In this third and last episode on Ruby’s #fetch family of methods, now free to all, we get into some advanced #fetch usage. Including: deep fetching, using the name of the missing key in fallbacks code, and why I never use the two-argument form of #fetch.

Director’s commentary: This was first published on RubyTapas October 26, 2012. Since then the #dig method has supplanted some of my deep-fetching needs, although it lacks a fallback block argument.

Find the original episode script and code below the video.

Introduction

This is the third and almost certainly the last episode on the #fetch method. Today I want to go over some advanced aspects of using #fetch.

#fetch beyond Hash

First of all, it’s worth noting that while up till now I’ve demonstrated it in the context of Hash objects, #fetch isn’t limited to Hashes. It’s also available on Arrays, where it behaves very similar to the Hash version, except that when the key is missing and no default block is supplied, it raises an IndexError instead of a KeyError.

a = [:x, :y, :z]
a.fetch(3)
# ~> -:2:in `fetch': index 3 outside of array bounds: -3...3 (IndexError)
# ~> from -:2:in `<main>'

You can also find #fetch the ENV pseudo-hash. I find this very useful for optional configuration values, for example enabling a port number to be customized via environment variable while still providing a default value.

port = ENV.fetch('PORT'){ 8080 }.to_i
port # => 8080

Defaults for nested hashes

Sometimes, you may want to get optional values out of a nested hash. Not only do you not know if the values will be present, but you don’t even know if the nested sub-trees will be there. For instance, here’s some configuration data:

config1 = {
  database: {
    type: 'mysql',
    host: 'localhost'
  }
}

config2 = {} # empty!

I like to handle data like this by chaining fetch statements together, with empty hashes as the default values for missing subtrees:

config2.fetch(:database){{}}.fetch(:type){'sqlite'}
# => "sqlite"

Generalized default blocks

One thing I haven’t yet shown about #fetch is that it yields the missing key that was passed in. Here’s some code that demonstrates what I mean:

{}.fetch(:foo) do |key|
  puts "Missing key: #{key}"
end
# >> Missing key: foo

One scenario where this could be useful is if we have a lot of calls to fetch which should all handle a missing key the same way. We can define the default block as a lambda taking one argument, and pass the lambda to each call to fetch. This code prompts the user for missing values.

default = ->(key) do
  puts "#{key} not found, please enter it: "
  gets
end

h = {}
name = h.fetch(:name, &default)
email = h.fetch(:email, &default)

Two-argument form of #fetch

If you’re already familiar with the #fetch method you may be wondering why I haven’t used the two-argument form in any of these videos. For those who aren’t familiar, instead of passing a block to #fetch for the default value, you can pass a second argument instead.

{}.fetch(:threads, 4) # => 4

This avoids the slight overhead of executing a block, at the cost of evaluating the default value whether it is needed or not. Personally, I never use the two-argument form. I prefer to always use the block form. Here’s why: let’s say we’re writing a program and we use the two-argument form of fetch in order to avoid that block overhead. Because the default value is used in more than one place, we extract it into a method.

def default
  42 # the ultimate answer
end

answers = {}
answers.fetch("How many roads must a man walk down?", default)
# => 42

Later on, we decide to change the implementation of #default to a much more expensive computation. Maybe one that has to communicate with an remote service before returning.

def default
  # ...some expensive computation
end

answers = {}
answers.fetch("How many roads must a man walk down?", default)

When the default is passed as an argument to #fetch, it is always evaluated whether it is needed or not. Now our expensive #default code is being executed every time we #fetch a value, even if the value is present. By our premature optimization, we’ve now introduced a potentially much bigger performance regresssion everywhere our #default method is used as an argument. If we had used the block form, the expensive computation would only have been triggered when it was actually needed.

Conclusion

OK, enough about #fetch! Happy hacking!

Re-Use SSH Config Inside Docker Containers with WSL2

Avdi Grimm — Sat, 24 Oct 2020 20:09:26 +0000

Docker and WSL have been getting more and more chummy lately. The Docker Desktop WSL2 backend integrates Docker containers with the “official” integrated Windows/Linux ecosystem (boy that still feels a little weird to write…) and gains performance perks in the process.

However, I ran into some trouble recently getting SSH to work inside of a Docker container. As I’ve written about before, I like to bind-in my credentials from my host machine, rather than proliferating credentials into containers. When a container needs to SSH out to the wider world, that means binding in my .ssh directory from the host, e.g. with a docker-compose.yml:

    volumes: 
      - type: bind
        source: ${HOME}${USERPROFILE}/.ssh
        target: /home/${DEV_USER:-abapa}/.ssh

Only one problem: volumes mounted from the Windows side are mounted via drvfs, which by default projects all files as having mode 777 on the Linux side. And SSH will refuse to use certificate files that are world-writeable.

After a bunch of research and some trial-and-error, I found a solution that’s working right now (2020-10-24). Recent versions of drvfs allow NTFS-hosted files to have fake Linux-side effective permissions attached to them, via NTFS extended attributes. And the docker-desktop distro that Docker Desktop installs has this feature enabled:

PS> wsl -d docker-desktop
# cat /etc/wsl.conf
[automount]
root = /mnt/host
crossDistro = true
options = "metadata"

Notice the options = "metadata" line in /etc/wsl.conf.

So the trick was to open a WSL terminal in that distro and set the desired effective owner and file permissions:

PS> wsl -d docker-desktop
caroline:/tmp/docker-desktop-root/mnt/host/c/Users/avdi_000/.ssh# chown -R 1000:1000 .
caroline:/tmp/docker-desktop-root/mnt/host/c/Users/avdi_000/.ssh# chmod -R u=rwX .

These aren’t “real” file attributes, since Windows has a very different permissions model from Linux. But they get saved as file attributes and treated as the effective file permissions inside WSL distros with metadata enabled.

Notes:

-R makes the change recursive through directories.
User/group 1000 are a common default user/group ID for containers.
File mode u=rwX recursively sets files to readable/writeable by owner, and directories read/write/traversable by owner.

Controlling superclass argument pass-through in Ruby

Avdi Grimm — Fri, 16 Oct 2020 15:00:13 +0000

In Ruby class-based programming, superclass versions of subclass methods are always invoked explicitly using the super keyword. There are some nuances to using super though, particularly when it comes to passing (or not passing) arguments and blocks along to the base class. In this sample from from the RubyTapas archives, we’ll talk about some of those “gotchas”.

Director’s commentary: This was originally published as RubyTapas#14 in October 2012. I used to have a habit of saying “as you know” in my scripts, I think because I was afraid my audience would see something they already knew at the start of the episode and think “oh this is nothing new”. I don’t say that kind of thing as much anymore, although I still say “as you may know” from time to time. I’ve become a lot more comfortable just stating my background material and trusting the viewer to situate themselves with regard to it.

I’m a bit mad at myself for not coming up with a more “real” example here; I think that makes it harder to follow. This was back in the era of three episodes a week (!) though, so I’ll cut myself some slack for hasty writing.

Scroll down for the video script and code.

Let’s talk about calling superclass methods.

As you know, when class Child inherits from class Parent, and both define a method #hello, the Child can reference the Parent‘s implementation of #hello, using super.

class Parent
  def hello(subject="World")
    puts "Hello, #{subject}"
  end
end


class Child < Parent
  def hello(subject)
    super(subject)
    puts "How are you today?"
  end
end


Child.new.hello("Bob") 


Hello, Bob How are you today?

If we simply want to call the parent implementation with the same arguments that were passed to the child implementation, we can omit the arguments to super. This only works if we leave off the parentheses as well.

class Child < Parent
  def hello(subject)
    puts super
    puts "How are you today?"
  end
end

This makes our code less brittle, because changes to a parent method’s parameter list won’t mean having to hunt around and update every super call that invokes it.

Sometimes we may want to force zero arguments to be passed to the superclass method. In that case, it’s important to remember to explicitly supply empty parentheses instead of leaving them off.

Here’s a version of Child that takes a special flag to indicate that it should use its default subject. When the flag is passed, it calls super with empty parentheses, forcing the superclass method to resort to the default value for subject.

class Child < Parent
  def hello(subject=:default)
    if subject == :default
      super() 
      puts "How are you today?"
    else
      super(subject)
      puts "How are you today?"
    end
  end
end


Child.new.hello(:default)


Hello, World How are you today?

There’s a catch to this, though: even with explicit empty parens, calling super will still automatically pass along any block given to the child method.

To show what I mean, let’s modify the parent class method to take a block, and then pass a block to the call to #hello.

class Parent
  def hello(subject="World")
    puts "Hello, #{subject}"
    if block_given?
      yield
      puts "Well, nice seeing you!"
    end
  end
end


Child.new.hello(:default) do
  puts "Hi there, Child!"
end
# >> Hello, World
# >> Hi there, Child!
# >> Well, nice seeing you!
# >> How are you today?


Hello, World Hi there, Child! Well, nice seeing you! How are you today?

As you can see, the output is a little mixed-up due to the block being unexpectedly passed-through despite the empty argument list to super.

In order to suppress the block being passed through, we have to use the special argument &nil:

class Child < Parent
  def hello(subject=:default)
    if subject == :default
      super(&nil) 
      puts "How are you today?"
    else
      super(subject, &nil)
      puts "How are you today?"
    end
  end
end


Child.new.hello(:default) do
  puts "Hi there, Child"
end


Hello, World How are you today?

This less-than-obvious technique eliminates the possibility of a block being implicitly passed through to the superclass method.

I have some other tricks involving the super keyword, but I’ll save them for a future episode. Happy hacking!

Ruby Singleton Objects

Avdi Grimm — Fri, 09 Oct 2020 18:31:39 +0000

Here’s another freebie from the deep RubyTapas stacks. This one is about a truth of object modeling that we don’t often talk about: not every object needs to have state. If an object has no state, there’s no need to have more than one of it. And for stateless objects, having a class just to generate a single instance may be superfluous!

Director’s commentary: In retrospect I’d tighten up the pacing on this one. And I hate that the ending monologue on the Singleton pattern has no illustration on the screen. Also, I really hope I don’t sound that bored in more recent videos.

While I was at Ruby DCamp a few weeks ago Sandi Metz asked me to write a version of Conway’s Game of Life in a semi-functional, stateless style. I decided to represent the concept of a “live cell” and a “dead cell” as two different kinds of object. The game grid would then be represented by a simple array of arrays, populated by live cells and dead cells.

I was all set to write a class for each…

class LiveCell
  # ...
end

class DeadCell
  # ...
end

…when it occurred to me: since this is a stateless implementation, there was no real need to have individual LiveCell objects for every live cell on the board. And the same goes for dead cells. Without any state, every instance would be exactly the same. And since the objects didn’t have any need for initialization, why even bother with classes?

Instead, I created the objects as singleton instances of the Object class. I assigned the instances to constants, so they would be available anywhere in the program. Then I used Ruby’s singleton class syntax to add methods to each one. Each object needed two methods: a #to_s method to represent the cell on an ASCII grid, and a method to determine what the next generation of its grid square would contain: either a live cell or a dead cell.

LIVE_CELL = Object.new
class << LIVE_CELL
  def to_s() 'o' end

  def next_generation(x, y, board)
    case board.neighbors(x,y).count(LIVE_CELL)
    when 2..3 then self
    else DEAD_CELL
    end
  end
end

DEAD_CELL = Object.new
class << DEAD_CELL
  def to_s() '.' end

  def next_generation(x, y, board)
    case board.neighbors(x,y).count(LIVE_CELL)
    when 3 then LIVE_CELL
    else self
    end
  end
end

Then I could just populate my grid with the same LIVE_CELL and DEAD_CELL objects, over and over again.

[
 [DEAD_CELL, LIVE_CELL, LIVE_CELL, DEAD_CELL],
 # ...
]

This worked quite well, but I didn’t like the fact that the singleton objects had to be created in two steps. So I decided to see if I could create the object, assign it to a constant, and define methods on it, all in a single statement.

To accomplish this feat, I took advantage of the fact that Ruby allows variables and constants to be assigned inside parenthesized sub-expressions of a statement.

class << (LIVE_CELL = Object.new)
  def to_s() 'o' end

  def next_generation(x, y, board)
    case board.neighbors(x,y).count(LIVE_CELL)
    when 2..3 then self
    else DEAD_CELL
    end
  end
end

class << (DEAD_CELL = Object.new)
  def to_s() '.' end

  def next_generation(x, y, board)
    case board.neighbors(x,y).count(LIVE_CELL)
    when 3 then LIVE_CELL
    else self
    end
  end
end

The resulting syntax was a bit obscure, but it accomplished my purpose succinctly.

A class plays two roles in an OO program:

It provides a container for behavior that’s shared by many objects.
It acts as an object factory, manufacturing new instances and ensuring they are initialized correctly.

When we have an object which does not need to share behavior with any others objects, and which requires no initialization, that renders both roles of a class superfluous. In cases like this, it can make more sense to just use a one-off singleton object.

Another way to accomplish this is to use a module as our singleton object, and only define class-level methods on the module. Let’s update the example to use this approach instead.

module LiveCell
  def self.to_s() 'o' end

  def self.next_generation(x, y, board)
    case board.neighbors(x,y).count(LiveCell)
    when 2..3 then self
    else DeadCell
    end
  end
end

module DeadCell
  def self.to_s() '.' end

  def self.next_generation(x, y, board)
    case board.neighbors(x,y).count(LiveCell)
    when 3 then LiveCell
    else self
    end
  end
end

[
 [DeadCell, LiveCell, LiveCell, DeadCell],
 # ...
]

This is a perfectly legitimate way to implement singleton objects. Personally, I feel that using a singleton instance of Object is a little more intention-revealing, since modules are usually used as namespaces and as a means of sharing behavior. But I don’t feel that strongly about it. The main thing I hope to convey in this episode is simply that sometimes a singleton object is all we need, and when that’s all we need, that’s all we should use.

I should make one last point here: I’m talking about singleton objects, not the Singleton Pattern. The Singleton Pattern describes a way to control the construction of a particular class such that only one instance is ever constructed. Singletons following this pattern have often been used to manage a great deal of centralized state, a situation that can lead to a number of problems, including thread contention and proliferation of dependencies. The singleton objects we’ve defined here are stateless and more akin to the immutable singletons that Ruby provides like true, false, and nil. As such they are somewhat less prone to abuse.

Okay, enough for today. Happy hacking!

Why and how to use Ruby’s Hash#fetch for default values

Avdi Grimm — Sun, 04 Oct 2020 18:09:21 +0000

Here’s another classic from the early days of RubyTapas. Originally published as Episode #11 in October 2012, it’s a complement to the episode on using fetch as an assertion. This episode digs into the difference between using the || operator for defaults vs. using Hash#fetch.

Director’s commentary: I can see some tiny advancements in quality here. At this point I’d realized that my original comment color was nearly unreadable on the dark background, and brightened it up a bit.

But my voiceover still sounds kind of bored. And was still content to have long sections of silence while I put new code on the screen. These days, I try to always accompany coding with simultaneous explanation.

Read on for the original script and code…

In a previous episode, we looked at how the #fetch method on Hash can be used to assert that a given hash key is present.

auth = {
  'uid' => 12345,
  'info' => {
  }
}

# ...

email_address = auth['info'].fetch('email')
# ~> -:11:in `fetch': key not found: "email" (KeyError)
# ~> from -:11:in `<main>'

But what if the KeyError that Hash raises doesn’t provide enough context for a useful error message?

Along with the key to fetch, the #fetch method can also receive an optional block. This block is evaluated if, and only if, the key is not found.

Knowing this, we can pass a block to #fetch which raises a custom exception:

auth['uid'] # => 12345
auth['info'].fetch('email') do 
  raise "Invalid auth data (missing email)."\
        "See https://github.com/intridea/omniauth/wiki/Auth-Hash-Schema"
end
email_address = auth['info'].fetch('email')
# ~> -:10:in `block in <main>': Invalid auth data (missing email).See https://github.com/intridea/omniauth/wiki/Auth-Hash-Schema (RuntimeError)
# ~> from -:8:in `fetch'
# ~> from -:8:in `<main>'

Now when this code encounters an unexpectedly missing key, the raised exception will explain both the problem, and where to find more information.

The block argument to #fetch isn’t just for raising errors, however. If it doesn’t raise an exception, #fetch will return the result value of the block to the caller, meaning that #fetch is also very useful for providing default values. So, for instance, we can provide a default email address when none is specified.

email_address = auth['info'].fetch('email'){ 'anonymous@example.org' }
email_address # => "anonymous@example.org"

Now, you may be wondering: what’s the difference between using #fetch for defaults, and using the || operator for default values? While these may seem equivalent at first, they actually behave in subtly, but importantly different ways. Let’s explore the differences.

Here’s an example of using the || operator for a default. This code receives an options hash, and uses the :logger key to find a logger object. If the key isn’t specified, it creates a default logger to $stdout. If the key is nil or false, it disables logging by substituting a NullLogger object.

This works fine when we give it an empty Hash.

require 'logger'

class NullLogger
  def method_missing(*); end
end

options = {}
logger = options[:logger] || Logger.new($stdout) 
unless logger
  logger = NullLogger.new
end
logger
# => #<Logger:0x000000030545a8
# @default_formatter=
# #<Logger::Formatter:0x00000003054580 @datetime_format=nil>,
# @formatter=nil,
# @level=0,
# @logdev=
# #<Logger::LogDevice:0x00000003054530
# @dev=#<IO:<STDOUT>>,
# @filename=nil,
# @mutex=
# #<Logger::LogDevice::LogDeviceMutex:0x00000003054508
# @mon_count=0,
# @mon_mutex=#<Mutex:0x000000030544b8>,
# @mon_owner=nil>,
# @shift_age=nil,
# @shift_size=nil>,
# @progname=nil>

But when we pass false as the value of :logger, we get a surprise:

options = {logger: false}
logger = options[:logger] || Logger.new($stdout) 
unless logger
  logger = NullLogger.new
end
logger
# => #<Logger:0x000000040bb608
# @default_formatter=
# #<Logger::Formatter:0x000000040bb5e0 @datetime_format=nil>,
# @formatter=nil,
# @level=0,
# @logdev=
# #<Logger::LogDevice:0x000000040bb590
# @dev=#<IO:<STDOUT>>,
# @filename=nil,
# @mutex=
# #<Logger::LogDevice::LogDeviceMutex:0x000000040bb568
# @mon_count=0,
# @mon_mutex=#<Mutex:0x000000040bb518>,
# @mon_owner=nil>,
# @shift_age=nil,
# @shift_size=nil>,
# @progname=nil>

That was supposed to be a NullLogger, not the default logger!

So what happened here? The problem with using || with a Hash for default values is that it can’t differentiate between a missing key, versus a key whose value is nil or false. Here’s some code to demonstrate:

{}[:foo] || :default # => :default
{foo: nil}[:foo] || :default # => :default
{foo: false}[:foo] || :default # => :default

In contrast, #fetch only resorts to the default when the given key is actually missing:

{}.fetch(:foo){:default} # => :default
{foo: nil}.fetch(:foo){:default} # => nil
{foo: false}.fetch(:foo){:default} # => false

When we switch to using #fetch in our logger-defaulting code, it works as intended.

options = {logger: false}
logger = options.fetch(:logger){Logger.new($stdout)}
unless logger
  logger = NullLogger.new
end
logger
# => #<NullLogger:0x00000003b73858>

When you want to provide default value for a missing hash key, consider carefully whether you want an explicitly supplied nil or false to be treated the same as a missing key. If not, use #fetch to provide the default value.

OK, that’s all for today. Happy hacking!

The ultimate software taboo

Avdi Grimm — Fri, 18 Sep 2020 19:35:28 +0000

I have written before about the hidden dangers of unstructured or “flat” organizations. A microcosm of the structure wars is the debate over whether Open Source software projects should adopt explicit codes of conduct. These codes, if not an introduction of true structure, are at least the imposition of a “social API” of sorts.

One objection I’ve seen raised to codes of conduct is that we shouldn’t discriminate against people who write good code but who happen to be assholes. After all, as the argument goes, not everyone is blessed with social graces. And if we exclude those people, we will miss out on their essential technical contributions.

With regard to this argument, a study out of Harvard on “toxic employees” seems apropos:

A worker in the top 1% of work productivity could return $5,303 in cost savings, while avoiding a toxic hire could net an estimated $12,489.

Studies also show that an overall negative culture has tremendously damaging impact on health, productivity, and engagement:

…a large and growing body of research on positive organizational psychology demonstrates that not only is a cut-throat environment harmful to productivity over time, but that a positive environment will lead to dramatic benefits for employers, employees, and the bottom line.

Although there’s an assumption that stress and pressure push employees to perform more, better, and faster, what cutthroat organizations fail to recognize is the hidden costs incurred.

Emma Seppälä and Kim Cameron in Harvard Business Review

These studies are from the realm of business, but it doesn’t seem like too much of a stretch to suggest that the effects extend to volunteer associations as well.

And then there’s this. A new study has found that in a medical setting, rudeness might literally kill:

…a rude comment from a third-party doctor decreased performance among doctors and nurses by more than 50 percent […] “We found that rudeness damages your ability to think, manage information, and make decisions,” said Amir Erez, an author on the study and a Huber Hurst professor of management at the University of Florida. “You can be highly motivated to work, but if rudeness damages your cognitive system then you can’t function appropriately in a complex situation. And that hurts patients.”

This is consistent with what I learned from reading Thinking, Fast and Slow: the human mind draws from a single well of energy. This energy can be sapped in many ways, and processing negativity is one of those ways. Having an asshole in the room doesn’t just raise the bar to entry. It ensures that everyone who has made it into the room won’t be working at their full potential.

Whenever I hear that it is simply impractical to exclude contributions based on social, political, or ethical objections, I think about September 27, 1983. That’s the day Richard Stallman decided that the sociopolitical structures behind the software of his day were morally untenable, and launched a project to replace every single line of code he and others made use of with Free Software.

Discarding millions of lines of existing code. Excluding all of the brilliant programmers who were working within the confines of Free-Software-unfriendly organizations. Clearly, this project was doomed from the start.

And yet here we are, decades later, and this letter was almost certainly delivered to you with the help of Free Software. Free Software powers millions of servers, phones, and devices. The GNU replacements for the classic UNIX utility stack, re-written from scratch to satisfy ethical constraints, are widely regarded as being superior to the ones that they replaced. Amazingly, counter-intuitively, rejecting existing work and existing contributors did not render the Free Software movement dead on arrival.

More recently Stallman himself has resisted codes of conduct, and has conducted himself in a way that contributes to exactly the sort of unsafe environment that codes of conduct seek to avoid. Should he be given a free pass because of all his contributions?

I argue no, he shouldn’t. If there are three things that Stallman showed the world, they are: 1) software has political and ethical implications; 2) a software movement built on ethics can survive and even thrive; and 3) eschewing people’s contributions on ethical grounds is no obstacle to progress.

I am starting to think that the greatest taboo in software development isn’t writing GOTOs, editing code in production, or even using tabs instead of spaces. No, the most dangerous, terrifying, unspeakable idea in programming is this: the suggestion that we—and all of the code we’ve written—might be replaceable.

This article was adapted from SIGAVDI#10, December 14, 2015.

Alternative Ruby Symbol Literals

Avdi Grimm — Fri, 18 Sep 2020 14:53:04 +0000

Periodically I take episodes from the RubyTapas archives and publish them for free. This episode from October 2012 is about symbol literals, and how you can use alternative quoting syntax to embed and interpolate any kind of character sequence in a symbol. And also: why you might not want to take advantage of this. Enjoy!

Video Script

As you know, a Symbol literal is a word with a colon in front of it.

:foo

You may also know that symbols aren’t limited to simple words; anything that can go in a String, can go in a symbol. But how do we get arbitrary characters, like spaces and dashes, into a symbol?One way is to start with a string and use #to_sym to convert it to a symbol:

"foo-bar (baz)".to_sym # => :"foo-bar (baz)"

But there’s a more concise and idiomatic way to do it. If we precede a quoted string with a colon, it becomes a quoted symbol.

:"foo-bar (baz)" # => :"foo-bar (baz)"

You can interpolate values into the quoted symbol just as you would into a string.

post_id = 123
:"post-#{post_id}" # => :"post-123"

And it also works for single-quotes.

:'hello, "world"' # => :"hello, \"world\""

Finally, if that doesn’t satisfy your symbol-quoting needs, there’s also a percent-quote literal syntax. Just like %q for strings, you can quote a symbol with %s followed by any delimiter you want.

%s{foo:bar} # => :"foo:bar"

Of course, all these different ways of writing symbols don’t mean you should go nuts with generating symbols. Symbols are best used for constrained vocabularies, like the names of options that may be passed to a method. The Ruby virtual machine has to keep a table in memory of all known symbols, so that it can assign a single unique integer ID to each. That means generating an unbounded set of symbols, from, say, user input can lead to memory leaks. So go easy on the symbol interpolation.That’s it for today. Happy hacking!

Rubber Duck Session: Rails Polymorphic Attributes with Karim Talek

Avdi Grimm — Fri, 19 Jun 2020 20:31:21 +0000

In this public Rubber Duck Session, my pairing partner Karim wanted to move some logic for editing message-display rules from the frontend to the backend. We settled on using JSONB and Rails polymorphic attributes as our implementation strategy.

An Investigative Debugging Toolbox

Avdi Grimm — Sun, 24 May 2020 01:29:50 +0000

Last week I ran an experimental workshop on Investigative Debugging. It was a lot of fun and the attendees seemed to get a lot out of it. Afterwards I sent out some extra reference material, including this Linux-oriented list of investigative debugging tools!

The focus here is on “surveillance tools” – utilities that give you observability into exactly what programs are really doing at runtime. It’s far from complete, but it represents a ton of research and I hope you find it helpful!

Tool	Description
BCC tools	A collection of over 70 specialized tracing and performance profiling tools built on BPF.
`bpftrace`	A Linux tool analogous to DTrace, built on BPF. `bpftrace` enables users to compose “questions” at the command-line, using a concise AWK-inspired syntax, for which they would once have had to use (or create) a dedicated tracing tool. It is often possible to recreate BCC tools using bpftrace commands. But when a dedicated BCC tool exists for a given query, using it may be easier than hand-rolling a `bpftrace` command.
Docker	A technology for isolating Linux-based services into containers, each with their own isolated filesystem, process table, and virtual network. Containers are useful from an investigative debugging standpoint because they enable us to isolate and observe how a process interacts with its environment.
`docker diff`	Get a comprehensive list of all files changed within a Docker container.
`ftrace`	A low-level, “kind of janky” (Julia Evans’ words) interface to Linux kprobe and uprobe tracing.
`HTTP_PROXY`, `http_proxy`, `HTTPS_PROXY`	These environment variables are sometimes respected by programs as a way to configure a SOCKS proxy.
`LD_PRELOAD`	An environment variable that can be used to force a program to link to arbitrary libraries before running. This can be used to substitute user-provided functions that override default behavior with e.g. tracing instrumentation.
`ldd`	List all the libraries that an ELF executable links to.
`lsof`	List open filehandles, including network connections.
LTTng	A linux tracing framework. I believe this is now one of the inputs to BPF?
`ltrace`	Trace library calls.
`mitmproxy`	An interactive HTTP proxy server that can log and inspect requests and responses. Can be configured as either a SOCKS proxy or a transparent proxy.
`nm`	List all the names (e.g. function names) in an ELF executable/library.
Postman	A workspace for experimenting with and documenting HTTP calls, which can also act as a `mitmproxy`-like request inspector for other processes.
`readelf`	A general tool for inspecting and dumping information about ELF executables and libraries.
`strace`	Trace system calls.
`strings`	Dump all the static strings found in an ELF executable or library.
`tplist`	A BCC tool for listing all USDT tracepoints available in a process.
`uflow`	A BCC tool for tracing method calls and returns in high-level languages like Python, Ruby, and Java. May be installed somewhere weird and not in the path, like `/usr/sbin/lib/uflow`
WireShark	The preeminent tool for tapping into and dissecting IP network traffic at the packet level.

Share .netrc with your devcontainers

Avdi Grimm — Sat, 23 May 2020 20:50:09 +0000

I do almost all my development inside devcontainers these days. (I take the term “devcontainer” from VS Code’s .devcontainer/devcontainer.json. I think it’s a useful term, and I’ve adopted it into my vernacular for the general case of “development-specific container”)

Some developer service clients, such as Heroku’s CLI tools, store their sensitive API keys in a ~/.netrc file. This is a semi-standardized (or at least conventionalized) file used by a number of UNIX utilities.

If you’re doing your development in a container, files like ~/.netrc can pose some problems, or at least prompt some considerations.

You don’t want to have to log in to Heroku (or whatever) every time you spin up or rebuild a devcontainer.
Even though you’re not using them for production deploys (right??), you may still want to push devcontainer images to a repository for the convenience of your teammates or CI. You don’t want to be capturing sensitive login info in an image that’s proliferating to other machines and potential public.
You may have a standard set of dotfiles that you fetch into your devcontainers (VSCode has a wonderful featurefor this), but you probably keep your dotfiles in a public git repo. Not the sort of place you want to be putting API keys.
There are lots of good reasons not to bind your entire home directory into a Docker container.

If you’re using docker-compose, you can use something like the following to bind your host ~/.netrc into your dev container’s home directory at runtime:

services:
  app:
    # ...
    volumes:
      - type: bind
        source: ${HOME}${USERPROFILE}/.netrc
        target: /root/.netrc
        # ...

Now you can log in to Heroku (or whatever) once, and reuse that login anytime you rebuild this container, as well as in any other containers you configure this way.

Note the use of trying both HOME and USERPROFILE to keep things host platform-neutral. One of the perks of using devcontainers is that you can develop on any host OS!

This example also makes the assumption that /root is the user home directory inside the devcontainer. I tend to stick with the root (admin) user inside devcontainers, rather than defining a separate unprivileged account. All the considerations that would normally push me to use unprivileged user accounts are taken care of by the fact that I’m running inside an isolated container.

(Don’t use docker-compose for your devcontainers? Consider adding it—even for an app that doesn’t depend on any other services! Just as Dockerfile is a great way to document your app’s dependencies and runtime expectations as versioned code, docker-compose.yml is a great way to document how to start your devcontainer as versioned code.)

Danger, Will Robinson!

Now, there’s one thing you need to be careful of when using this configuration: make sure ~/.netrc exists before you start your devcontainer. The reason is that if it doesn’t already exist, Docker will create it… as a directory. Oops.

So just be sure to touch ~/.netrc before you spin up your devcontainer, and you’ll be fine.

Run Rails 6 System Tests in Docker Using a Host Browser

Avdi Grimm — Mon, 11 May 2020 18:40:10 +0000

The problem: we wanted to get Rails system tests running using a browser on our development machines.

The complication: we develop our Rails project within Docker, in order to make every aspect of our development environment reproduceable.

The preference: we want to see the browser-based tests running, but we don’t want to mess around with running browser inside a container and tunneling its display to an external X-Windows server. We already have perfectly good browsers on our host machines, and we’d like to use them natively.

The outcome: we can run Rails system tests inside a Docker container. It pops up Chrome windows on the host machine, and exercises our app!

Coming up with the steps below involved a lot of trial-and-error, research, and debugging. I’m going to skip all that for this write-up, and just break down our working configuration piece by piece.

I’ve also written up an in-depth breakdown of what all these parts are, and how they work together in Rails system tests.

1. Disable the webdrivers gem

The webdrivers gem is a lovely convenience for making sure you have the right browser and WebDriver tool installed before running system tests. However, we don’t want to run the browser inside the container. We want the container to talk to a browser either on the Docker host machine, or in a dedicated selenium sidecar container.

The webdrivers gem was causing system tests to fail with “couldn’t find browser” error. So we disabled it.

# Gemfile

group :test do
  # ...
  # webdrivers
end

And then ran bundle install again to update Gemfile.lock.

2. Choose a stable port for the test server

By default Capybara picks a new random port to run the test web server on every time system tests are run. We want tools outside the container to be able to consistently communicate with the test server.

We arbitrarily chose port 3434, and documented that choice with an entry in a brand-new .env file in the project root.

CAPYBARA\_SERVER\_PORT=3434

3. Configure Docker-Compose for external browser testing

Here’s an elided version of our .devcontainer/docker-compose.yml, with the important elements commented.

# Specify a recent version of docker-compose to ensure we have fancy environment 
# variable expansion
version: '3.2'
services:
  app:
    build: 
      context: .
      dockerfile: Dockerfile

    environment:
      # Convey which port Capybara should be starting test servers on; fail loudly
      # if we've neglected to configure it.
      CAPYBARA_SERVER_PORT: "${CAPYBARA_SERVER_PORT:-3434}"
      # The test server WILL NOT BE AVAILABLE from outside the container if it binds 
      # to 127.0.0.1. It must bind to 0.0.0.0 to be exposed to the outside world.
      CAPYBARA_SERVER_HOST: "${CAPYBARA_SERVER_HOST:-0.0.0.0}"

    volumes:
      - type: bind
        source: ..
        target: /workspace
        consistency: cached

    ports:
      # Expose the test server port. Fail loudly if CAPYBARA_SERVER_PORT isn't set.
      - "${CAPYBARA_SERVER_PORT:-3434}:${CAPYBARA_SERVER_PORT:-3434}"
      # Also expose the usual app port
      - "3000:3000"

4. Configure Capybara to use the test server and port from environment

We added two lines to our test/test_helper.rb:

ENV["RAILS_ENV"] ||= "test"
require_relative "../config/environment"
require "rails/test_help"

# Use `fetch` to fail loudly if these variables aren't set. We might relax this 
# and set defaults at some point, but for the moment we want to make sure we didn't 
# miss a step.
Capybara.server_host = ENV.fetch("CAPYBARA_SERVER_HOST")
Capybara.server_port = ENV.fetch("CAPYBARA_SERVER_PORT")

class ActiveSupport::TestCase
  # Run tests in parallel with specified workers
  parallelize(workers: :number_of_processors)

  # Setup all fixtures in test/fixtures/*.yml for all tests in alphabetical order.
  fixtures :all

  # Add more helper methods to be used by all tests here...
end

5. Update the system test base class to drive a remote browser

The configuration so far makes sure that Capybara starts up the server-under-test in a way that external tools can find it and talk to it. But we still needed to tell Capybara to interact with use an external WebDriver, instead of trying to drive a browser inside the container.

We updated our test/application_system_test.rb:

require "test_helper"

class ApplicationSystemTestCase < ActionDispatch::SystemTestCase
  parallelize(workers: 1)
  driven_by :selenium, using: :chrome, screen_size: [1400, 1400],
                       options: {
                         browser: :remote,
                         url: "http://host.docker.internal:9515",
                       }
end

Notes:

Figuring out the right options to pass into driven_by was… nontrivial.
browser: :remote are the magic options to tell Capybara not to try to talk to a local WebDriver tool.
host.docker.internal is a special hostname that Docker sets up to refer to the host machine.
Port 9515 is the standard port that chromedriver listens for commands on. In a future revision I think I’d like to capture this knowledge more explicitly in another environment variable.

6. Install ChromeDriver on the development host

Our first development host happened to be a Windows machine so we used:

choco install -y chromedriver

Instructions for installing ChromeDriver on various platforms can be found at the ChromeDriver project home page.

7. Start the chromedriver

…and tell it to allow connections from anywhere. This is accomplished, unintuitively, by using the --whitelisted-ips flag without any arguments.

> chromedriver --whitelisted-ipsStarting ChromeDriver 81.0.4044.69 (6813546031a4bc83f717a2ef7cd4ac6ec1199132-refs/branch-heads/4044@{#776}) on port 9515All remote connections are allowed. Use a whitelist instead!Please protect ports used by ChromeDriver and related test frameworks to prevent access by malicious code.

Maybe later we’ll figure out how to provide a whitelist for just the dev container.

8. Run system tests inside the container

$ PARALLEL\_WORKERS=1 rails test:system

We found parallel system tests to be flakey, so we disable them with an environment variable.

9. Enjoy spooky automated browser action!

Rails 6 System Tests, From Top to Bottom

Avdi Grimm — Mon, 11 May 2020 06:27:04 +0000

A Rails 6 system test is a test that exercises your application in a way that, as much as possible, simulates a real user interacting with it via a browser. More than any other kind of tests, system tests verify that the whole app does what it’s supposed to do.

As such, they are typically used for acceptance testing: tests that verify that an app has fully implemented a specified feature. They are also useful for smoke tests: high-level tests that exercise the code just enough to verify that nothing is badly broken. As well as characterization tests: tests that capture an app’s current behavior in preparation for internal refactoring.

Technically, a system test doesn’t have to interact with an actual browser. They can be configured to use the rack_test backend, which simulates HTTP requests and parses the resulting HTML. While rack_test-based system tests run faster and more reliably than frontend tests using a real browser, they are significantly limited in their ability to simulate a real user’s experience, because they can’t run JavaScript.

For the rest of this article I’m going to be focusing on system tests that interact with your app via an actual browser.

There are a lot of moving parts involved in running a browser-based system test. Until recently, I was pretty fuzzy on how all these parts fit together. Recently I set out to get browser-based system tests running for a Dockerized Rails app, and as a result I’ve learned a bit more about them.

At a high level, a Rails system test putting an app through its paces using a browser breaks down as:

A MiniTest test case, augmented with…
Capybara testing helpers, which
- start and stop an instance of your app, and
- provide an English-like DSL on top of…
The selenium-webdriver gem, which provides a Ruby API for using the…
… WebDriver protocol in order to interact with…
A WebDriver tool such as chromedriver or geckodriver , which…
Is automatically downloaded by the webdrivers gem. The WebDriver tool automates…
A browser , such as Chrome.

The Geology of a System Test

Let’s look at each of those in more detail.

MiniTest

MiniTest is the unit testing library that Rails’ built-in testing framework is based on. It provides base classes for test cases, basic assertions like assert_equal, and a runner to run tests and report on their success and failure.

For Rails System Tests, Rails provides an ApplicationSystemTestCase base class which is in turn based on ActionDispatch::SystemTestCase:

require "test_helper"

class ApplicationSystemTestCase < ActionDispatch::SystemTestCase
  driven_by :selenium, using: :chrome, screen_size: [1400, 1400]
end

This base class augments your system test cases with Capybara abilities.

Capybara

Capybara is a Ruby library for full-stack testing of web applications. It provides your tests with a few different capabilities.

First, Capybara arranges for an instance of your app to be started before the tests start executing, and shuts it down once the tests have finished. In other words, for a Rails app it does the equivalent of running rails server before executing the tests, and hitting Ctrl-C once they are done. To keep it separate and to ensure that the tests are run against the right test instance, Capybara starts your app on a different port than the one you’d normally use to manually interact it.

Second, Capybara provides a high-level, English-style API for authoring system tests. Capybara’s DSL is what enables you to write your tests in terms similar to how you’d describe the interactions to a human:

visit "/"
click_on "Log in"
fill_in "Username", with: "avdi"
fill_in "Password", with: "xyzzy"
click_on "Submit"
# ...

In a sense, Capybara acts as a kind of stable pivot point for around which all the other aspects of testing a Ruby web application can vary. Capybara is neutral towards:

The web app framework you’re building on. Whether you’re building on Rails, Sinatra, Hanami, or raw Rack, you can test it with the help of Capybara.
The test tooling you prefer. Whether your tests are written in MiniTest, RSpec, or Cucumber, they can all incorporate the Capybara helper methods for controlling a simulated browser.
The type of browser simulation/automation. Capybara lets you switch out “drivers” to automate different kinds of browsers, real or simulated. The same Capybara tests can be run against bare Rack requests, actual Chrome, Safari, or Firefox browsers; “headless” versions of those browsers, or simulated browsers such as PhantomJS.

When Capybara is automating a real browser, it does so by translating calls to its the high-level Capybara DSL (click_on, fill_in) into invocations of a library called selenium-webdriver.

selenium-webdriver

Automatic control of a browser is typically accomplished using a WebDriver tool, which is a standalone executable for remote-controlling a particular browser. We’ll talk more about WebDriver executables in a minute.

WebDriver executables all speak the WebDriver protocol, an HTTP-based protocol for automating browsers. Capybara doesn’t speak WebDriver directly. Instead, it makes use of a library called selenium-webdriver.

What’s with the name “selenium”? The Selenium project is the umbrella under which the WebDriver protocol for controlling browsers was first developed. This protocol is now a W3C standards track spec. The selenium project still maintains various language bindings to the WebDriver protocol. The selenium-webdriver gem contains the Ruby webdriver bindings.

Here’s an example of using selenium-webdriver directly, that I swiped from the project documentation:

require 'selenium-webdriver'

driver = Selenium::WebDriver.for :firefox
wait = Selenium::WebDriver::Wait.new(timeout: 10)

begin
  driver.get 'https://google.com/ncr'
  driver.find_element(name: 'q').send_keys 'cheese', :return
  first_result = wait.until { driver.find_element(css: 'h3>div') }
  puts first_result.attribute('textContent')
ensure
  driver.quit
end

You can see how it’s a bit lower-level than the Capybara example further up. The selenium-webdriver library translates these calls into WebDriver Protocol, which it speaks to a webdriver executable.

WebDriver Protocol

WebDriver is an HTTP-based protocol for telling browsers what to do. For instance, in order to create a Chrome browser window and tell it to navigate to google.com, you would first start up a chromedriver process. Then, you’d send it a “new session” command with an HTTP post:

curl -X POST 'http://127.0.0.1:9515/session' -d '{"capabilities":{"firstMatch":[{"browserName":"chrome"}]}}'

This returns a bunch of data, including a session ID:

{ ... "sessionId":"98de922f9be6784957775f25618c60a3" ... }

Which we follow up with a “navigate to” command in the form of POSTing a new url to the created session:

curl -X POST 'http://127.0.0.1:9515/session/97039faaa396941517d800b75d0f6096/url' -d '{"url": "https://google.com"}'

This is the level of tedium that the selenium-webdriver gem is abstracting over with its Ruby bindings.

You can’t send these WebDriver commands directly to a browser, however. You have to send them to a WebDriver process.

WebDriver Tool

“WebDriver” is a protocol. A WebDriver, on the other hand, refers to a tool that speaks that protocol and controls a browser. If you want to drive a browser with WebDriver protocol, you need to first download that browser’s WebDriver tool.

WebDriver tools act as servers: when you execute them, they start a persistent process that listens for HTTP requests until it is terminated.

For every major browser, there is an associated WebDriver. Chrome has chromedriver. Firefox has geckodriver. MS Edge has edgedriver. Safari has safaridriver.

While they speak a common language, WebDriver tools aren’t all developed by the same people. Each browser’s WebDriver is a separate project, usually associated with the browser’s development team. Each WebDriver’s team typically provides binaries for every major platform that browser is available on.

Finding, downloading, and installing the appropriate WebDriver tool somewhere in your PATH is a hassle. That’s why Rails 6 projects, by default, use the webdrivers gem to automatically download an up-to-date WebDriver tool that corresponds to Capybara’s configuration.

The webdrivers gem

This part of the stack is really just a convenience. When the webdrivers gem is required, it automatically augments the selenium-webdriver gem. Now when you run selenium-webdriver tests, it automatically determines which WebDriver executable needs to be downloaded for your platform and selected browser, downloads it, and arranges for that executable to be used by selenium-webdriver.

With the WebDriver tool downloaded, your tests can drive a browser!

Browser

Once you have a WebDriver tool installed, the parts farther up the stack can use it to automate browser interactions. As system tests are run, you’ll see new browser windows appear, navigate to pages in your app, fill in forms, follow links, etc.

Conclusion

It takes a lot of layers to make a Rails 6 system test work! There are a lot of moving parts, and a lot of opportunities for things to go wrong. Personally, I’ve spent a quite a bit of time fiddling with system test configuration in order to get them running consistently. Once they are running though, it’s a particularly satisfying feeling to watch your app being put through its paces automatically.

Once you’ve managed to get system tests running in real browser windows locally, you can move on to the next step of getting them to run “headless”. Then you can have a continuous integration server such as CircleCI automatically run those slow-but-important system tests for you after every commit.

But that’s a topic for another article!