DEV Community: Jason Fleetwood-Boldt

How the Rails Procfile Works, Foreman, Overmind, Hivemind

Jason Fleetwood-Boldt — Fri, 04 Aug 2023 18:58:13 +0000

The Basic Rails Server
New Rails 7 apps come into two basic flavors: (1) By default, using Importmaps, and (2) with a Node-friendly setup like JSBundling or Shakapacker.

This has led to much confusion about how to start rails.

In the old, old days, many developers started rails using rails server. That was a bad idea. Two alternatives were always better:

1:

bundle exec rails server

2:

bin/rails server

Starting using just rails server was never a good idea, and the correct way to start rails was by using either bundle exec rails server or bin/rails server, which do the same thing.

The bin/rails script lives inside of your Rails app, and you can look at it. Here is what it does:

#!/usr/bin/env ruby
APP_PATH = File.expand_path("../config/application", __dir__)
require_relative "../config/boot"
require "rails/commands"

Note that the first thing it does is set APP_PATH, and then it requires the Rails boot file.

If you ran start rails in the old days (the naïve way, without the bin/ and without bundle exec), you were bypassing the Bundler setup of your application. Bypassing the Bundler setup of your app means that you’re not locking your dependencies to what’s in your Gemfile.lock. That’s a bad idea, because you may accidentally boot Gem versions installed globally from your Ruby instead of ensuring you have the exact Gem versions from your Gem lock file.

For this reason, the default way to start rails always involves either #1 or #2 above to make sure you’re running your code through Bundler to ensure you’re using the correct gem versions.

Foreman
Small Rails app that have no extra services (Node compiling, Sass watching, Redis, etc.). They work using the old techniques above, simply booting only the Rails server. You can upgrade to a Foreman app when you add extra services. This is how you would be set up if you started with JSBundling, Shakapacker, CSSBundling, and other setups too.

If you create a JSBundling or CSSBundling app, notice how to use Foreman to start all of your dev services at once for you in one terminal window.

Example your Procfile.dev . It might look something like this:

web: bin/rails server -p 3000
js: yarn build --watch
css: yarn build:css --watch

This is a simple Procfile.dev that has 3 services: web, js, and css. The web service is what starts the Rails server, and the js and css services are “watchers” that watch for changes to your JS or CSS and recompile them on the fly.

For JSBundling Rails apps, you will start all services using either of two methods:

1:

foreman start -f Procfile.dev "$@"

2: use the shortcut you find inside of the bin/ folder:

bin/dev

This tells Rails to start all associated services at once in a single tab. Notice that bin/dev is just starting Foreman for you:

#!/usr/bin/env bash
if ! command -v foreman &> /dev/null
then
  echo "Installing foreman..."
  gem install foreman
fi
foreman start -f Procfile.dev "$@"

When starting on a Rails app that you haven’t worked with before, be sure to understand which kind of Rails app you are working with.

bin/dev is useful even without cssbundling/jsbundling. For example, for Tailwind, so our bin/dev you would put the Tailwind watcher process into your Procfile.dev. A

And for Stripe, the bin/dev also runs the stripe webhook CLI. You can add anyting that needs a “background process,” like Sidekiq, Dart SASS, Redis, etc.

bin/dev gives us a single command to start and stop it all via a Procfile. When you CTRL-C the window running Foreman, Foreman sends the “kill” command to all of the processes it started, shutting them all down at once.

Foreman pipes all the STDOUTs to 1 screen, which is bad for debugging.

The Debugging Problem When Using Foreman
When using Foreman, you have a severe problem if you want to debug. (ruby-debug, byebug, pry, etc). When you drop into a debugger by setting a debugger, Ruby does actually halt the execution and goes into the debugger. However, the output you see in your Terminal window is confusing, and when you try to interact with the debugger, the characters you type don’t appear on the screen as you would normally expect them to.

Take a look:

• The output in your Terminal window does not look like a standard debugger, which should halt and prompt you for interactive input.

• When you type interactive output, the characters appear at the bottom of the screen, and the result printed contains confusing information.

That’s because the debugger is in the same stream output as Foreman, so you lose the ability to have a clean interface for 1) seeing what you are typing, 2) hitting enter and seeing the result. It’s very messy and is not recommended for debugging.

For debugging only, you will need to fall back to the multi-window strategy.

A normal debugging interaction looks like this:

Alternatives to Foreman
Overmind is another process manager that is similar to Foreman but with some additional features. It provides better support for managing long-running processes, such as websockets, and also supports zero-downtime deploys.

Another alternative is Hivemind. The makers of Hivemind created it for Rails applications. It can automatically detect and manage your Rails processes, including your web server, worker processes, and other background jobs.

Regarding which one to choose for your project, it depends on your specific needs. Foreman is a good choice for a simple process manager that is widely used and well-supported.

If you have long-running processes that must be managed carefully, then Overmind might be a better fit. Hivemind is a good option if you are specifically working with a Rails application and want a process manager that is tailored to that use case.

• Introducing Overmind and Hivemind — Martian Chronicles, Evil Martians’ team blog

If you are using Overmind, the setup will modify the bin/dev script to call Overmind instead of Foreman.

Then when you run bin/dev, it runs overmind s -f Procfile.dev.

Overmind uses tmux to manage the processes, which allows you to do overmind connect web or overmind connect sidekiq (or any of the other process names you defined).

This is helpful if you want a good debugging experience since Foreman doesn’t allow you to interact with the processes.

Here is a quick function to detect the kind of Rails app and start it appropriately.

startrails () {
        if [ -f bin/dev ]
        then
                bin/dev
        elif [ -f Procfile ]
        then
                bundle exec foreman start
        else
                bundle exec bin/rails server -p 3000 -b 0.0.0.0 | stdlog
        fi
}
alias sr=startrails

Ruby Arguments: Equal Signs, Colons, Splats (*), Double Splats (**) Explained

Jason Fleetwood-Boldt — Tue, 27 Jun 2023 11:48:33 +0000

originally published https://jasonfleetwoodboldt.com/courses/stepping-up-rails/ruby-arguments-equal-signs-colons-splats-double-splats-explained/

Watch the companion video on Youtube

Ruby has two basic types of arguments: positional arguments, which are specified by arity — that is, by the positioning of the variable in the method definition, and keyword arguments, which are specified by the name of the argument.

Most programmers are familiar with the basic concept of arity-based parameters: The first variable passed to the method becomes the first variable within the code of the executing method. These are called arity-based parameters.

Keyword arguments, introduced in Ruby 2.0, allow you to write code that doesn’t depend on the positioning of the arguments because every argument is explicitly named.

Positional Arguments

I will also refer to positional arguments as “arity parameters” but the formal name is positional arguments. Whatever it is called, it has a basic structure known to all programmers: the first parameter passed becomes the first variable, the second becomes the second, etc. The order you pass the variables into the calling code determine what variables get which vales inside the method.

*Example 1: Basic Positional Arguments are Ordered
*

def my_method(apple, banana, cherry)
  "#{cherry}#{banana}#{apple}"
end

puts my_method(1,2,3)
 # 3 2 1

puts my_method(1,2) 
# missing argument - illegal in Ruby

Notice that this construction makes apple, banana, and cherry all required. If you want to make them optional, you’ll want to use an = and give them a default value. Because all 3 are required, we must call the above method with three parameters (that’s why the last example is illegal syntax).

Parameter Default Values

Positional arguments will have a default value if specified with an = sign. Whatever you put after the = will be the default value.

*Example 2: Positional Arguments Can Be Optional
*

def my_method(apple = nil , banana = 'missing', cherry = 0)
    "apple=#{apple}|banana=#{banana}|cherry=#{cherry}"
end

puts my_method(4, 'found', 3 ) # apple=4|banana=found|cherry=3
puts my_method(4, 'found', nil) # apple=4|banana=found|cherry=
puts my_method(4, 'xyz') # apple=4|banana=xyz|cherry=0
puts my_method(4) # apple=4|banana=missing|cherry=0
puts my_method # apple=|banana=missing|cherry=0

*Example 3: Optional Arguments Should Come Last
*

def my_method(apple, banana, cherry = 0)
  "apple=#{apple}|banana=#{banana}|cherry=#{cherry}"
end
#
puts my_method(4, 'found', 3 ) # apple=4|banana=found|cherry=3
puts my_method(4, 'found', nil) # apple=4|banana=found|cherry=
puts my_method(4, 'xyz') # apple=4|banana=xyz|cherry=0

# ILLEGAL RUBY
# my_method(4) # wrong number of arguments (given 1, expected 2..3)
# my_method # wrong number of arguments (given 0, expected 2..3)

Positional Arguments With Default Should Come Last

More commonly used parameters typically come first because you can skip parameters from the right-to-left but not from left to right. thus, the lesser-needed parameters typically come after more used parameters when designing an API

*Example 4: Putting optional arguments first doesn’t make sense
*

def my_method(apple = "a", bannana)
  "apple=#{apple}|banana=#{bannana}"
end

puts my_method("b", 3) # apple=b|banana=3
# ILLEGAL
# puts my_method(, 1) # ILLEGAL RUBY
# puts my_method # wrong number of arguments (given 0, expected 1..2)

This demonstrates a useless design: there is no way to use the first parameter’s default value.

Rest argument (Single Splat)

Using a Rest argument is how you tell Ruby to “take the rest of the arguments passed” and turn them all into a single array. In the method definition, you specify one single variable (an array) that will take up any arguments beyond the ones already passed.

A single splat operator will receive arguments as an array to a variable or destructure an array into arguments. You can have only one splat operator in a method definition and it must come last.

A ruby method can take a single ‘rest’ argument, specified using a splat * proceeded by the name of the array to take the input.

Remember, a singular argument (always the last) in your method will group up all of the remaining arguments you pass — let’s call them “unclaimed” by the positioned arity arguments, and turn them into a single array. Ruby will make them into a single array, now named the variable name given after the asterisk (*)

*Example 5: The Rest Argument for Positional Arguments that Come After the Specified Positional Arguments (Single Splat)
*

def my_method(*money_array)
  "#{money_array.join(' ')}"
end
puts my_method(4,6,8) # 4 6 8
puts my_method(4,6) # 4 6
puts my_method # (prints nothing)

Here, the money_array takes the rest of the arguments are passed into it and turns them into a Ruby array. It is the only argument, so this example is slightly confusing because all of the arguments are “rest” arguments.

*Example 6: Rest Argument Come After Positional Arguments
*
Here, the first two positional arguments (blend and grain) are required

Ruby will crash if they are not supplied. If you supply any more positional arguments, the remaining arguments are considered the rest of the arguments, and get grouped up into an array that goes into the rest argument (in this case, sugar). In this example, you can call my_method with either 2 or more arguments. Any more than 2 (the 3rd, 4th, etc) will be grouped up into the rest argument sugar.

def my_method(blend, grain, *sugar)
   "the sugar was #{sugar.join(' and ')} using #{blend} at #{grain}"
end

puts my_method('amber', 'raw', 56, 13, 4) # the sugar was 56 and 13 and 4 using amber at raw
puts my_method('amber', 'raw', 56) # the sugar was 56 using amber at raw

puts my_method('amber') 
# ILLEGAL —–wrong number of arguments (given 1, expected 2+

A ruby method can only take one rest argument and will not compile if you specify more than one

def my_method(blend, grain, *sugar, *salt)
   // this is illegal ruby
end

If you pass a hash to a Rest argument, you get an array with a single member (the hash).
The rest arguments will always be put into an array so doing something like this is awkward:

*Example 7: Don’t get confused by trying to pass hash to a rest argument.
*

def my_method(blend, *sugar)
  "#{sugar.inspect}"
end

puts my_method("brown", {lark: 2, leper: 5}) # "[{:lark=>2, :leper=>5}]"

You have put the hash into an array with a single member, the hash.

Keyword Arguments

Ruby Keyword arguments, introduced in Ruby 2.0, give Ruby the ability for a parameter to be named. You specify its name in both (1) the method definition and (2) the calling code. You do this using a simple key: value syntax. You are freed from having to keep track of the strict positioning (arity) of your methods and are at liberty to refactor your functions with less complexity to keep in your head.
Wield this power wisely!

What is connascense?

Connascence between two software components A and B means either 1) that you can postulate some change to A that would require B to be changed (or at least carefully checked) in order to preserve overall correctness, or 2) that you can postulate some change that would require both A and B to be changed together in order to preserve overall correctness.

– Meilir Page-Jones, What Every Programmer Should Know About Object-Oriented Design

When one Ruby method has to know the order of another method’s positional arguments, we end up with a connascence of position.

That means we’ll need to change all callers of that method accordingly if we ever change the method arguments (Expensive cost of change.)

In addition, our mental model of how to use this method must change as well.

Keyword arguments have trade-offs. Positional arguments offer a more succinct way to call a method, especially if that method takes only one or two arguments.

The code clarity and maintainability gained from keyword arguments generally outweigh the brevity offered by positional arguments.

If you can easily guess their meanings based on the method’s name and there are only one or two arguments, positional arguments are still fine.

But generally speaking, codebases written after Ruby 2.0 should prefer keyword arguments.

*Example 8: One Keyword Argument
*

def my_method(apple: nil)
   "the apple is #{apple}"
end

puts my_method(apple: "mcintosh")

Keyword arguments are reversible. They can come in any order, making them more powerful for refactoring your code in the future. Although it may feel like more typing (more verbose), the payoff for you in the long run is exponential because it removes your coupling to order-based (arity-based) positional arguments.

*Example 9: Keyword Arguments Are Reversible
*

def my_method(apple: nil, banana: nil)
   "the apple is #{apple}, the banana is #{banana}"
end

puts my_method(apple: "mcintosh", banana: "yellow") 
# the apple is mcintosh, the banana is yellow
puts my_method(banana: "orange", apple: "fiji") 
# the apple is fiji, the banana is orange

*Example 10: One Positional Argument Followed by Keyword Arguments
*

def my_method(thing, apple: nil, banana: nil)
  "the apple is #{apple}, the banana is #{banana}, but the thing is #{thing}"
end

puts my_method("rock", apple: "mcintosh", banana: "yellow")
# the apple is mcintosh, the banana is yellow, but the thing is rock

puts my_method("hard place", banana: "orange", apple: "fiji") 
# the apple is fiji, the banana is orange, but the thing is hard place

puts my_method("rock")
# the apple is , the banana is , but the thing is rock

ILLEGAL RUBY
my_method("rock", "anythin else", apple: "mcintosh", banana: "yellow") 
# found extra arguments
my_method("rock", cherry: 'whodis') 
# unknown keyword 'cherry'

Keyword Arguments Can have a Default or Be Required (Ruby 2.1)
Introduced with Ruby 2.1, you can specify a keyword that is required by omitting the default.

In the examples above, we specified that if no value was passed, we assumed the value would be nil.

To require a keyword argument — cause Ruby exception to be called by any calling code that does not call the method with the required keyword— specify nothing where you would have specified a default value.

*Example 11: Keyword Arguments Can be Required
*

def my_method(apple: "x", banana: )
  "apple is #{apple} and banana is #{banana}"
end

puts my_method(apple: "a", banana: "b")
# my_method(apple: "a") # ILLEGAL: Missing arguments 'banana'

You Can Only Pass a Keyword Argument that is Specified (unless using Double Splat)

Consider for a moment this errant example:

*Example 12: Can’t Use an Unspecified Keyword (without a double splat)
*

def my_method(thing, apple: nil, banana: nil)
   "the apple is #{apple}, the banana is #{banana}, but the thing is #{thing}"
end

puts my_method("that", apple: 'golden delicious', phone: 'iphone')

No such keyword specified a keyword phone was defined in the method’s definition.

Double Splat

This is illegal Ruby because the calling code (red) tries to call a keyword argument called phone, but no such keyword argument exists. What if you wanted a way to take the unnamed keyword arguments and group them into a hash? Kind of like the rest arguments for positional arguments, there is a tool for that: The Double Splat **

Prior to Ruby 2.7, you could leave off the double splat and assume that the last argument would be turned into the hash for any unclaimed keyword arguments. However, this behavior is deprecated (no longer used), and was removed in Ruby 3.0. In Ruby 2.7, if you leave off the double splat from the last argument, you’d see a warning message like:

warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call

In Ruby 3.0, leaving off the double splat becomes illegal syntax.

In the following example, I’m using the badly named args_hash to store a hash of the unspecified arguments— that is, any keyword arguments which are not apple or banana.

*Example 13: Double Splat Groups Up Unspecified Keyword Args
*

def my_method(thing, apple: nil, banana: nil, **args_hash)
  "the apple is #{apple}, the banana is #{banana}, and the additional passed args are #{args_hash.inspect}"
end

puts my_method("that", apple: 'golden delicious', phone: 'iphone')
# the apple is golden delicious, the banana is , and the additional passed args are {:phone=>"iphone"}

puts my_method("this", apple: 'french toast', email: 'noone@nowhere.com')
# the apple is french toast, the banana is , and the additional passed args are {:email=>"noone@nowhere.com"}

puts my_method("the other", apple: 'green tomatoes', phone: 'android', email: 'person@somewhere.com')
# the apple is green tomatoes, the banana is , and the additional passed args are {:phone=>"android", :email=>"person@somewhere.com"}

puts my_method("the other", phone: 'android', apple: 'green tomatoes', email: 'person@somewhere.com')
# the apple is green tomatoes, the banana is , and the additional passed args are {:phone=>"android", :email=>"person@somewhere.com"}

Notice that in the last example above, the calling code comes in a different order from the one before it: phone, apple, email (nothing is required; apple and banana are specified keywords). In both examples, we called the method with keyword arguments phone and email (neither defined on the method), apple came in a different order. But it didn’t change the output: in both cases, apple was picked up as a properly named keyword argument, became first-class variable within the method, and args_hash became any arguments that weren’t specified in the code definition.

You May Only Have One Double Splat and It Must Come as The Last Parameter
Example 14: You can’t specify More Than One Double Splat
**
**ILLEGAL

def my_method(thing, apple: nil, banana: nil, **hello, **goodbye)
"the apple is #{apple}, the banana is #{banana}, and the additional passed args are #{args_hash.inspect}"
end
# Expected block argument or end of arguments list

You can’t have two double splats in a method definition.

Your Double Splat Must Come At the End of the Keyword Arguments
ILLEGAL

def my_method(thing, apple: nil, **hello, banana: nil)
   "the apple is #{apple}, the banana is #{banana}, and the additional passed args are #{args_hash.inspect}"
end

You must have the double-splat argument come after the keyword arguments.

*Example 15: Mixing Rest Positional and Rest Keyword is Allowed, but Confusing
*

def my_method(thing, *rest_args, apple: nil, banana: nil, cherry: 0, **hello)
  "the thing is #{thing}, apple is #{apple}, banana is #{banana}, chery is #{cherry} 
   rest positional args are #{rest_args.join(',')} and rest keyword args are #{hello.inspect}"
end


puts my_method("that", "a", "b", "c", "d",
               phone: "123-456-7890",
               apple: "golden delicious",
               email: "person@somewhere.com")

the thing is that, apple is golden delicious, banana is , chery is 0
   rest positional args are a,b,c,d and rest keyword args are {:phone=>"123-456-7890", :email=>"person@somewhere.com"}

BIG TECH WON’T SAVE US

Jason Fleetwood-Boldt — Mon, 03 Oct 2022 12:03:22 +0000

I was in the audience at the New York Tech Meetup on June 15, 2010. That was the night that Scott Heiferman smashed an iPad with a sledgehammer.

He told us he would. At first, he got a lackluster response of disbelief. So Heiferman dragged it out. He proceeded to carefully wrap the iPad in a piece of moving cloth (to protect us from the shattering glass), then paced back and forth on the stage.

He taunted us. Would he really throw away $800 just for an abstract anti-authoritarian demonstration? (He was calling out the insidiousness of tech and the magnificent position it had put him to afford to throw away $800 so casually.)

Audible shrieks and gags were heard as he brought the sledgehammer down. Inside the filled 900-seat NYU Skirball auditorium, you could feel the crowd’s palpitation. It was a hot, sticky NYC June night.

For years tickets to NYTM always sold out. In 2010 in New York City, if you had a startup, you were probably presenting it to the crowd at NYTM.

I had just moved the year prior from the tech hot bed of San Francisco. Although you might think of the SF tech scene was larger, by the count of people who show up to meetups (which was a count of the wanna bes, quite frankly), the scene I encountered through the NYTM was the largest set of tech events I’ve ever seen.

In 2009, San Francisco was filled with engineers. There, Christian Perry and Cass Phillips had launched the widely successful SF Beta also on Meetup, also a monthly networking event. Distinct to the New York Tech Meetup, SFBeta operated audience-less. (Presenters just demoed to people around tables in an open room.)

Everywhere you looked in San Francisco, people were coding in cafés in laptops. You could see Google or Apple employees who got bussed down to Silicon Valley. (That had become very fashionable a few years earlier.)

I moved for work, and as an early Rails engineer (a highly in demand and sought-after specialty), I remember how distinctly rare the job market treated me and my skill (thus, rewarding me likewise).

The main event was probably 2 hours long. By its height — about when I intersected it upon moving to New York, it was, unarguably, the biggest deal in New York City tech. Back then, to build a startup you started by writing code. First demoed it at the New York Tech Meetup and then hoped an investor would notice you.

The stage show had a keynote (usually by Nate Westheimer), and about 6 to 10 demos from New York City app developers and ‘startups.’ I put ‘startups’ here in quotation marks because these startups weren’t like the startups of today.

After a startup had presented, there would be a Q&A session. Inevitably, some uninitiated audience member would always ask how the start-up intended to make money — to monetize — their app. Nate Westheimer (the organizer) would immediately stop them. This question — how you would make money — was the ‘forbidden’ question. (I saw Nate literally strike down that question many times when asked by an audience member.) It was Nate’s belief that asking a tech venture how it was going to monetize prematurely was the wrong thing to do, and that the money-driven New Yorkers had to be kept at bay so that NYC could foster a thriving entrepreneurial community where tech could grow.

New York tech was growing, or so it felt. The old-guard monied elite of NYC were jealous of tech’s lure and wanted in. The Wall Street guys were there and asking How’s it gonna make money. Outside of Nate’s protected bubble of NYTM, it was the operative question on everyone’s lips: Is all this tech thing going to take off or is it a bubble?

Everyone was swimming in tech. The New York scene was filled and people who abstractly had other careers (usually finance bros) and wanted to break in. The after party, at a bar two blocks away — was probably the epitome of every ‘professional networking’ schmooze you’d expect. Most 20- and 30-somethings recently arrived in the city — everyone interaction felt transactional or self-promotional. Everyone had some aspiration to break into what felt like a gold rush. Nobody really thought about selling shovels.

These demos would be cool, sometimes broken (as they say is the live-demo curse), always scrappy. Their teams were small — a biz guy and a developer (almost always guys).

I can barely remember any of the companies I saw in any of these demos. (Of course, lots of big names did pass through NYTM — my intersection with it was only a few passing years over its long and large history.)

Anyone could build a startup. Facebook’s meteoric rise to the industry’s top “social media” platform was just a few years old — and it felt that at that time all you needed was a great idea, an email list, a prototype, and lots of enthusiasm.

Very few of the people I met actually were programmers.

Mostly people worked in sales, or marketing, or biz dev (this was before the formalization of what we today call ‘product management’). Part of this happened because of the Facebook effect itself.

Scoff Heiferman was the CEO and one of five co-founders of Meetup.com. New York Tech Meetup was just one of the many meetups that were spawned from the Meetup.com platform extraordinarily popular at the turn of the decade.

I will be the first to admit I was attracted to tech because of the rush it — to the feeling you’d get when you launched a product and it went viral. When you’d see your signups shoot up and you knew people were somewhere in their homes sitting on their computer screens using the software you had written. It gave me a rush: the asymmetrical power of it all. Here we were, small teams of developers, who could push out a business interaction so our website could transact with any number of people. It could scale up “infinitely” (obviously not, but it felt this way).

The word of the day — during the turn of the decade — was disruption.

It didn’t matter if you had profit. It didn’t matter if you had marketshare. What mattered was that you had a working product and that you were in a blue ocean.

(In their pivotal 2004 business book Blue Ocean Strategy Renée Mauborgne and W. Chan Kim argue that lasting success comes from creating new markets — blue oceans — as opposed to red oceans, where your business already faces a sea of bloody competition.)

So why did Scott Heiferman, the CEO & founder of Meetup.com, smash an iPad at the tech meetup during what might have been that bubble’s peek?

In the actual year 1984, Apple Computer released a Superbowl ad that would become the most iconic advertisement of all time.

Rows and rows of men sit mindlessly staring at a huge screen. They are controlled by some faceless overlord projected onto the screen.

A runner (a woman) comes into the screen and hurls a giant sledgehammer into the screen of Big Blue (IBM). Running was very popular in the early 80s.

The image referenced the book 1984 by George Orwell. In Orwellian 1984, a dystopic society is controlled by a totalitarian state led by a character referred to as “Big Brother.”

Apple Computer proclaimed in 1984 that — because of the introduction of the Macintosh computer — the future wouldn’t be like the book: a society of placated, pacified people sitting mindlessly staring at screens. They said: “See why 1984 won’t be like 1984.”

Today, when an industry becomes so powerful its controlling interests start abusing their power, we call it “Big” — like Big Pharma or Big Tech.

What Scott Heiferman was saying when he smashed a brand new iPad throwing away his $800 was: “Get off your phone!” This new tech is great, but at what cost? What were we giving up, was his question.

Is Tech actually, in fact, Big Brother?

To quote the great Carl Sagan (and perhaps my favorite Matthew McConaughey line of all time): “I don’t fear technology, I fear the men who deify it at the expense of human truth.”

As a career programmer and lifelong coder, I’ve been thinking a lot about how code just kind of ‘exists.’ It lives here on our laptop or on some ambiguous ‘server’ in the sky. As developers, we know it runs because of the artifacts it leaves us: logs, emails, database records, messages.

But code isn’t good or bad. It isn’t inherently right or wrong. Code isn’t inherently profitable or unprofitable. You might love Python or Ruby but, I hate to break it to you: It doesn’t love you back.

The primary driver of this slice of the ‘tech industry’ in America is capitalism and the relentless pursuit of control.

Sadly, I no longer think the industry is particularly driven by innovators.

When web software development was new, every software developer saw themselves as elite.

My friend Justin Falcone — someone I met at the Brooklyn Javascript meetup just before the pandemic — asked on Twitter the other day:

And this follow-up tweet:

When I started in tech we saw the future of our young profession like law or medicine — destined to be a professionalized class in society. We thought we might get a “collar color” — like blue for working-class or white for the professional class.

Instead, capitalism rushed and commodified tech.

Is anyone surprised?

Computer programmers are in this class struggle with the capitalist class — the capitalists have tried to flip the script on disruption. They’ve tried to convince the world that it is their money, and access to it, that gives them the power of disruption.

If you were a young developer today and saw all these huge tech companies and hugely funded start-ups it’s no wonder you’d see disruption in someone else’s court.

Are programmers merely the technicians operating the machinery that belongs to the disruptor class? Or is the power we have to build software itself disruptive?

The paradox — to me — is obvious.

I believe the latter, but I also believe that the delusions of the industry are laid on so thick right now it’s difficult to understand.

Why are they training legions and legions of young developers on LeetCode when they should be training them on TDD? Why are so many smart Gen Zers so concerned with data structures and algorithms when, IMHO, they should be concerned with conversion metrics and user interaction design?

The great era of tech Meetups seemed to die — and certainly, the pandemic didn’t help. Now, a significant amount of what I see on Meetup.com are coding Bootcamps offering free introductory lessons. Gone are the ‘great tech meetups’ of engaged programmers who came together around a common passion.

(No disrespect to the many good meetups still being organized today — some by me! — but attendance and engagement at these still pale in comparison from my memory of the heyday.)

All of the people I mentioned in this story are, in fact, real individuals whom I met at different Meetups.

I predict this is just a cycle. The disruptors become established. They’ll try as much as they can to hold onto that power — that’s what all capitalists do — but in their blindness for control they will lose sight of the cycle of it all. It’s called a blue ocean because your technical innovation allowed for a new ocean: one that didn’t exist before.

This is the only way disruption has ever happened.

Effective Story Writing for Software Teams

Jason Fleetwood-Boldt — Mon, 03 Oct 2022 11:57:46 +0000

This guide will help you write clear and effective user stories for software development.

Writing user stories is a delicate balancing act. A few years ago, I wrote a blog post titled How to be a Great Product Owner, which discussed several high-level concepts around being a Product Owner.

In this post, I’ll go over the key elements of story writing: What a story is, how it relates to a ‘ticket’, what writing in story form means, and the nuanced ways to think about this process.

Story form means we write a narrative — or “story” — to communicate with the engineers who will be implementing a feature.

Writing a story involves this template:

(Line #1) As a ______

(Line #2) So that I can ______

(optional) [When _____]

(optional) [And ______]

(Line #3) I should ______

You will create tickets in your project management tool like Jira, Pivotal Tracker, ClickUp, or Notion. Each ticket is a collection of many stories. Together, they encompass the smallest possible unit of work, sometimes called a ‘minimum viable product.’ It is the least amount of work we can deliver to have something that can be demoable.

Whereas most scrum teams use “1 story per ticket” I do not recommend this.

How large or small should a ticket should be?

The answer is always the same: The ticket should be as small as possible to deliver one unit of discrete work. There is a school of thought in scrum that each ticket must be infinitesimally small to force the software work to be iterative.

I do not subscribe to this ideology because splitting tickets into infinitesimally small units of work takes the iterative approach to an extreme that isn’t useful. Developers can still deliver iteratively even if the ticket isn’t written iteratively.

(Developers can — and should — break up one ticket into multiple pull requests).

Each pull request can represent one small discreet unit of work, but it still requires several pull requests to complete the whole story or complete feature. I generally prefer that approach because it lets the developers make the call about how to split up the work in the moment. The story writing process doesn’t artificially create units of work that are too small to make sense, which is the danger when the Product Owners start to break things down too small.

The flip side is that tickets can get too large. However, it is preferable to start with large ideas and then break them down naturally during the process. I recommend each ticket have between 2 and 10 stories within them. (Remember, each story is just the 3, 4, or 5 lines from the template.)

The Basics

For all features you will write, always write in story form. Each story should be 3–5 lines long. Start with these 3 lines:

(Line #1) As a ______

(Line #2) So that I can ______

(Line #3) I should ______

1 is the who; line #2 is the why, and line #3 is the what and how.

If Line #2 (“so that I can”) needs clarification regarding what the actor is doing, you may use the optional lines of “When_” and add more context with “And.”

Each line has a specific purpose, and you should focus on sticking to the purpose of the line.

Line #1: As A (The Context Actor)

In software, we talk about the context actor for the feature or feature set. A context actor simply means the customer, the admin user, the visitor, i.e., the person who is going to use the software. It does not mean the developer, the product manager, or the CEO. This is one of the most common mistakes I see: without knowing how to correctly describe the context actor, product owners use arbitrary ideas as the “who” of the story.

The context actor is very important because this is the bridge to domain context interaction, which is a fundamental part of large application design. (We actually write the code very much considering the context actor which is why getting it right is so important.)

Don’t write stories that begin with “As a developer.” That’s an abuse of story form. You shouldn’t make the context actor an external “service” or another computer program (like, “as a client app” or “as an external API”). This too is an abuse of story form. If you find yourself doing that, stop and ask yourself to interrogate the story. (See below, “When a Story Isn’t a Story”)

You should also be careful when specifying the context actor as the company’s CEO, yourself, other product owners, or other team members in the company who don’t directly use the software itself. The only time you should use members of the company as context actors is when they are directly using the software that is going to be built. (For example, “as a customer service agent” is a good use case if and only if you are building a feature specifically for customer service agents.) If you’re building a dashboard for your marketing team or executive team, it’s OK to use “as an admin” or “as a marketing manager.” You should only ever do this if that person is really the actual person who will be interacting with the feature, not merely whom the request came from.

This is the single most abused facet of story form that I see.

The “As a” line is the who.

Line #2: So That I Can ____ (The purpose for the interaction)

The purpose of this line is to force you to think about, talk about, and communicate why the actor is doing the thing they are doing.

This is not the desired outcome. This is why the person is taking the journey. If the chicken were crossing the road, the “So that I can” would be “to get to the other side.” The fact of the road’s size, the traffic, whether the the chicken sees the stop light (probably not, because he’s a chicken), and whether the cars will stop for the chicken. Those are all what and how. The “So that I can” should strictly be the why.

This too, is open for abuse. Don’t let ‘the who’ bleed into your why, and don’t put the desired outcome here in the ‘So that I can’ (that’s the I should part).

Line #3: I should _____ (The Desired Outcome)

Finally, the I should part explains the desired outcome — or the what and how. Typically you will always write things here that the user will actually see on the screen while using the software. If that’s not possible, you can use different language here to discuss the database or backend state. (Like, “I should have my changes saved.”) but that should be avoided whenever possible.

I should is the purpose for the story’s existence. After you’ve written the 3 standard, required lines, ask yourself this question:

Can I remove line #1 (As a) and line #2 (So that I can) and still basically get the idea of the story? If the answer is yes, then you’ve successfully written a good 3rd line (I should). If the 3rd line makes absolutely no sense on its own, consider adding more to it. For example, “I should be successful” is not as useful as “I should be logged in” or “I should see a message telling me my record was saved.”

You should tell us what the user is being successful at or how the software behaves. Spell it out even if you think the people you work with know it already.

That doesn’t mean you should actually leave off the 1st and 2nd lines — quite the contrary — just that the 3rd line needs to be descriptive enough to convey the what and how on its own. (Albeit, missing the context that the first two lines provide.)

When & And (Additional Info Before I should)

If you’ve written the 3 lines and you look at your story and it’s a complete idea, stop there. If, on the other hand, it is missing context, you should then add more lines before the 3rd line:

When_____

And _____

Here you will add additional context if and only if necessary. Don’t go overboard. “When” and “And” lines should be things like steps that need to be taken to get the user in the right place or screen, prerequisites that must be met, or setup information important enough to include but not the actual result or desired outcome. Include additional lines only when necessary.

Multiple Stories

Stories should be lettered or numbered: Story 1, Story 2, Story 3. Each story should describe a new facet of the implementation, but doesn’t need to repeat things for the sake of repetition. For example, if the story might apply evenly to all states of the application, you don’t need separate stories for each state. On the other hand, if different states cause different results, then you do want different stories to describe each variation in result.

By the time we get to the end of all the stories you’ve written, it should be clear what the entire feature is doing: The “happy path” (or cases when everything goes right) and the “edge cases” (or cases when things don’t) both get described. The reader (developer) has a good sense of all the things that the ticket is trying to achieve. Nothing is left hidden or missing. No additional context is needed to implement the feature — everything is described in the totality of all the stories you’ve written together (that is, all of the ones for the feature or ticket).

Full-Stack Stories (“frontend only” or “backend only”)

Story form comes from the days of full stack software development. It doesn’t make a lot of sense in siloed teams and siloed-stack engineering. For example, it doesn’t make sense to write a story for just the backend or just the frontend. The story must communicate from the customer’s perspective, and the customer doesn’t care what code lives on the frontend and what code lives on the backend.

When a Story Isn’t a Story

Sometimes you are doing simple copy changes. For those, I don’t use story form. Also, for bugs that have a stacktrace, I also don’t use story form. Instead, I prefer just the stacktrace, or if the bug is reproducible, the steps to reproduce with “Expected result” and “Actual result” to describe the bug and what should be fixed.

Sometimes you do things that have no users, so you can skip the story form.

Conclusion

The most problematic aspect of software, codebases, and development process is inconsistency. Inconsistency creates cognitive friction that costs your team time and money.

Keeping all the Product managers and Product owners consistent about story writing is one of the strongest and most important things you can do to lay the foundation for the success of your scrum team.

With a little attention to consistency and getting it right, the payoff becomes exponential. With the right mindset and a disciplined approach to doing it the right way, you will reduce friction and create joy for your team and teammates.

Enumerated Types in Rails with Postgres

Jason Fleetwood-Boldt — Wed, 27 Oct 2021 17:18:03 +0000

MEET ENUMERATED TYPES

An enum is an enumerated type. In Rails, this means that you define a specific list of allowed values up front.
Historically, you did this with an enum defined on your model, like so:
enum status: [:pending, :in_progress: :finished]

You would then create an integer field on the model. Rails will map the :pending type to the integer 0, the :in_progress key to the integer 1, and :finished to the integer 2.
You can then refer in your code to the symbol instead of the integer, which removes the dependency on the underlying integer implementation from being sprinkled throughout your code.

You can write this in your code:

Conversation.where.not(status: :pending)

Instead of writing this:

Conversation.where.not(status: 0)

That's because by writing "0" into your code, you've now created a dependency on that part of the code knowing that 0 means "pending." You've also slowed down the next developer because now they have to remember that 0 means "pending."
Using the Ruby symbol instead (:pending), you're making your code less brittle and de-complected.

So it is preferred when you can to code this way.

WHAT DOES POSTGRES INTRODUCE?

Described above is historical way of creating enums in Rails - using an integer as the underlying database field type. However, with Postgres, we can now use the native Postgres enum type. That means that we need to define the enumerated type list in Postgres itself. (Unfortunately this is an extra thing to think about in the database migration which you will see below.)

Importantly, the underlying Postgres field type will be an enum type, and we will also use the Rails enum mechanism in our Ruby model definition. The two will be mapped together, but to do so we need just a couple of steps of special setup.
Natively, Rails doesn't know about the enum database field type. That's because Rails was written to be database agnostic and other databases don't have enum. For this reason, if we add enum field types to our database.
We can actually write a migration to do it, and the first Google result on this topic suggests you do such a thin. But in fact if you go down this route, you will have some more issues without this gem.

gem 'activerecord-pg_enum'

This gem does two important things:
When Rails is building your schema file (db/schema.rb), it won't be able to create the database definition if it has enum types in it. Instead of outputting a proper schema file, it will not output the database with the enum types.

Why are we doing this?

The default integer field type is bigint, which takes up 8 bytes of memory. From the activerecord-pgenum docs:
As you will see, we're going to need to tell Postgres about our enumerated types (in schema migrations), which will be in addition to telling Rails in our model definitions. Importantly, our Rails enum definitions will no longer use arrays, instead they will use a hash. The above example will become

enum status: [:pending: 'pending', in_progress: 'in_progress', finished: 'finished']

Although this seems strangely redundant, this is the most performant and preferred way to implement enums using Postgres & Rails
Let's get started. I assume you are starting from scratch, but if you already have an app with enums in it, you may have to migrate your data to add enums to Postgres.

STEP 1: INSTALL GEM ACTIVERECORD-POSTGRES

Add to your Gemfile

gem 'activerecord-pg_enum'

Then run bundle install

STEP 2: CREATE YOUR FIRST POSTGRES ENUM

Option A: You are generating a new Model
Once the gem is installed, you can use enum as a first-class field type, like so:
rails generate model Conversation status:enum
In the migration file, add the content shown in bold below:

class CreateConversations < ActiveRecord::Migration[7.0]
  def change
    **create_enum "statuses", %w[pending in_progress finished]**
    create_table :conversations do |t|
      t.enum :status**, as: :statuses**
      t.timestamps
    end
  end
end

Note that if you fail to add
, as: :statuses

Your migration will note work.

As well, in the example above, note that the field name is status but the enumerated type itself is called statuses. Be careful when using plural/singular version of the same word to refer to similar things in your app.

Option B: Add to an Existing Model

Let's assume you already have a Conversation model. Now you would run this migration
rails generate migration AddStatusToConversation

Here, you'll edit the generate migration like so:

class AddStatusToConversation < ActiveRecord::Migration[7.0]
  def change
    create_enum "statuses", %w[pending in_progress finished]
    add_column :conversations, :status, :statuses
  end
end

What's important to note here is that statuses, which is used as the 3rd argument in the add_column above, is used to specify the type of field. Normally here you might see :integer or :string. With enumerated types, we've defined our own "type" in Postgres's terminology, so we can now refer to this as a field type itself.
We can only do this because we created the enum using create_enum in the line above, of course.

STEP 3: DEFINE YOUR MODEL

class Conversation
  include PGEnum(statuses: %w[pending in_progress finished])
end

This is the preferred way to setup your enums, although you can still use the old style hash syntax which is equivalent:
enum status: {pending: 'pending', in_process: 'in_process', finished: 'finished'}
Now, in our code we will refer to any place we have a status as a symbol (not an integer or a symbol).
To demonstrate, we can now refer to a status on a Conversation object as either :pending, :in_progress, or :finished. Rails will translate the symbols to the Postgres enums, which in turn provide the fastest and most performant database experience.

2.7.2 :001 > conv = Conversation.new
 => #<Conversation:0x000000011e9b62d8 id: nil, created_at: nil, updated_at: nil, status: nil> 
2.7.2 :002 > conv.status = :pending
 => :pending 
2.7.2 :003 > conv.save
  TRANSACTION (0.3ms)  BEGIN
  Conversation Create (0.9ms)  INSERT INTO "conversations" ("created_at", "updated_at", "status") VALUES ($1, $2, $3) RETURNING "id"  [["created_at", "2021-10-07 15:10:23.640882"], ["updated_at", "2021-10-07 15:10:23.640882"], ["status", "pending"]]
  TRANSACTION (6.5ms)  COMMIT
 => true

RENAMING AN ENUM TYPE

There comes a time in every app’s life when you’ll want to make a change to the list itself. That is, you’re not going to change the values of the records, but rather, the name of the enum type. If you change abc to apple, for example, you want the records that pointed to abc before the change to point to apple after the change.

As you may have guessed, Postgres is keeping pointers to your enumerated values, so when you make an enumerated type name change you’ll run a Rails migration to make the change and also change references in your code at the same time (in the model files, explained in Step 3). Then, you won’t have to make any changes to your database records themselves. (You haven’t changed the values, only the names of enumerated types.)

A caveat to note is that you cannot remove all of the enums and replace them. If you want to rename an enum, do it value-by-value. For the example, the syntax would be as follows (assume that we are doing this on an enum called status_type):

rename_enum_value "status_type", {from: "abc", to: "apple"}
How to rename an enum value

REMOVING THE ENUM COMPLETELY

drop_enum "status_list", %w[pending in_progress finished]
How to drop the enum entirely

You can also drop the entire enum, but this requires that there are no fields that depend on it!!!

This means Postgres is protecting you from making any records orphaned by your dropping the enum value. Instead of doing it, Postgres will give you this error:

Caused by:
ActiveRecord::StatementInvalid: PG::DependentObjectsStillExist: ERROR:  cannot drop type color because other objects depend on it
DETAIL:  column color of table alerts depends on type color
HINT:  Use DROP ... CASCADE to drop the dependent objects too.

Choice #1) Remove all fields that point to any enum you want to drop. Set all of them to another enum or remove the fields themselves.

Choice #2) Unfortunately because the activerecord-pgenum gem does not have a way to pass the CASCADE flag to the migration if you really want to do that you’ll need to write the migration yourself. This is probably a good thing because it is enforcing good hygiene on you for your data maintenance. Go with choice #1 and avoid CASCADE.

REMOVING AN ENUM TYPE

You can’t! Sorry, you can only rename.

Example App here

WEBPAGETEST.ORG

Jason Fleetwood-Boldt — Wed, 30 Jun 2021 18:08:01 +0000

July is marketing & optimization month. Every day this month Verso will introduce a new tool for marketing or web optimization. To kick things off I’m starting with the most valuable of them all: The ultimate benchmarking tool for site speed optimization.
You simply enter your URL into the bar and then wait. (Because it’s free, you are sharing the resources with everyone else. Sometimes it can take up to a minute or two to profile your site. The number of other people using the tool at the same time will determine how long it will take — like waiting in a queue while on hold on the phone.)
The first thing you’re going to see is a screen like this:

WebPageTest.org — This is one of the secret tools for web page optimizers (like me!) who make your website really fast. This will break down your page’s load time (into a “waterfall” chart) showing you which external endpoints are taking the longest, which ones are blocking the page, and how to speed up your site.
An excellent remote benchmarking tool with breakdowns of each of your external web requests, showing times for DNS, SSL Negotiation, TTFB (Time-To-First-Byte), Content Download. It includes critical five page-wide metrics that you need to pay attention to:
• First Byte
• DOM Interactive
• First Paint
• the all-important DOM Content Loaded (also known as $(document).ready in jQuery*)
• the also-important Content Download time (also known as $(window).load in jQuery*)
You’ll often want to run it a few times (in fact by default it runs twice anyway).
[*Note that jQuery is not required to be running on these websites. These are the corresponding jQuery hooks for the page rendering lifecycle associated with these events.]

This view shows you an aggregate summary. It actually runs the test three times, but usually, you get close results. In the area where you see the Network chart next to “First View,” click once.

Now you’ll come to the detailed view for the measurements.

This screen is where the magic happens. There’s a lot of information on this page, and I could teach a whole course on this subject. The two most important measurements to learn about first are First content paint (look for a light green line, not the dark green line) and the Document Complete (look for a blue line). Both are expressed as a number (and fraction) of seconds.

The First content paint is when the user will first see content, and the Document Complete is when they can interact with the page.
The First Byte time is largely dependent on your server architecture and your app and is beyond the scope of this article. If you have long First Byte times try New Relic as a performance tool to find the bottlenecks in your app and figure out how to eliminate them.

For the most part, DOM Interactive and First paint you don’t have control over, assuming your assets are correctly cached and delivered via a CDN.

The DOM Content Loaded is what we believe to be the most significant factor in the GPSI score, but keep in mind that Google will punish you if above-the-fold content is not loaded by the DOM content loaded point, or shortly thereafter. Consider there are many seconds between the 4th and 5th measurements, your images and some Javascripts are downloading. What you want to do is prioritize the above-the-fold images first, so they appear for the user as quickly as possible.

If you do this, Google will favor you. If you don’t do this and your Content Download times ($(window).load) extend out several seconds, your Google score will drop.
Next, you’ll need to understand how to load your Javascript, internal & external, using async and defer. Unfortunately, that’s beyond the scope of this post today.
Webpagetest is the gold standard for a reason: It gives you nice scores for: First Byte Time, Keep-alive Enabled, Compressed transfers, Compressed images, cached static content, and using a CDN.

Caveats: I’ve noticed that WPT tends to list many ad pixels — which inherently cannot be cached– in the “Leverage browser caching of static assets” section and thus may give a lower score here if you have too many ad pixels. I’m not exactly sure why WPT isn’t smart enough to know that these are ad pixels, maybe it would be better if the ad industry moved over to AJAX requests via JSON endpoints instead (cough, cough, antiquated technology, cough, cough). Perhaps if more did then a tool like this wouldn’t see ‘pixels’ (non-visible image files that serve only to communicate with foreign servers but aren’t actually images the user sees) as cacheable resources.

Although this optimization tool is a little complex for non-technical users, laymen marketers and site builders can easily run their site through WPT to get a sense of how fast their site is performing. Remember, it can publicly profile any website so you can run tests on your competitors’ websites too.

If you need help with this my consulting agency VERSO COMMERCE can help you with site speed, analytics, competitive research, Wix, Shopify, React, or NextJS/Node apps. Get in touch today at https://versocommerce.com

Same Time, Same Desk: Rails Conference and React Summit

Jason Fleetwood-Boldt — Fri, 23 Apr 2021 11:25:19 +0000

Last week in a virtual chat room at the React Summit Kent C. Dodds was asked an interesting question: What does he think of Rails? “I hesitate sometimes with frameworks that abstract a lot. It certainly worked out well for Rails. If you fit within the rails, pun intended, then everything’s gonna be fine. React has a totally different approach. We’re just gonna let you do this one thing really well and let you build around it.” Within that freedom greater innovation is born, he suggested. “I’ve never used Rails,” he qualified his statement.

It stands in contrast to the other industry cult-leader, David Heinemeier-Hanson (who goes by DHH), giving the keynote to the Rails Conference on the same week.

DHH was on-brand: he delivered a fire-and-brimstone polemic railing against Big Tech monopolies. He begins by throwing a Molotov cocktail at the JavaScript industry: Gmail ships with 3.5MB of JavaScript, and that Hey.com (an email platform he launched last year) ships with a mere 40 KB. Translated to laymen-speak: Hey.com has shipped a product 14 years after Gmail was first introduced to the public in 2007 with dramatically less frontend code, it loads and operates faster on modern devices and on older devices, and is easier to maintain and ship.

DHH claims that Rails 7 (the next version) will mean Rails “now has a grand unifying answer for front-end.” In one section of his talk, his anti-Node sentiments flew out like bats from a closet door one rarely opens: “Is there an opening here where developing modern web applications do not require transpiling or bundling?” He continued later, “I don’t want to F around with Node.”

The implicit assertion by DHH is that the JavaScript world (and React) has become overly complex: its patterns create huge, unmaintainable intertwined codebases. JavaScript-heavy apps and apps written in React are unnecessarily weighed down with layers that, at best, turn off new entrants into programming, and at worst, help further the monopolistic practices of Big Tech.

“Ruby had been such a bring light for me, such a revelation, such a different path…I took on a role — in a highly antagonistic fashion ([I’ll] cop to it as it is) — to evangelize Ruby in a confrontational manner. Not just: Hey this is just another church you can join. No, I was like: Hey Java programmers, look at this sh*t! Look at this! It’s all this [little] code! And, like boom! Then it’s just this much code! Then what! Then what!” Although he proclaims this like a little kid, he does have the self-awareness of a man who has led Rails for nearly two decades, sheepishly admitting this attitude was a phase of his life he had to grow out of.

On the React side of things — I was attending both conferences concurrently at my desk — things sounded quite a bit different. The ideology was less anti-capitalist, sponsored ads were broadcast on periodic schedules between talks, and the whole program seemed less focused on cult leaders. The React community moves fast. It’s more of a hodge-podge of competing ideas and everyone — like Gen Z itself — is enamored with the idea that democracy lifts up the winner among a field of competing technologies.

Years ago class-based components were how React JavaScript was written, (we’re talking about object orientation, not capitalism), but last year the React world officially changed both its syntax, implementation details, and made a major shift away from object-oriented patterns towards functional programming (FP) using what is called the “hooks pattern” or just “hooks.”

The React world sees ‘closed’ ecosystems like Rails — perhaps even DHH himself — as the oligarchy precluding innovation. Ironically, DHH sees himself as the opposite: He is the David in David vs. the Goliath of the fight against tech monopolies. Apple, Google, and their corporate power to control who codes and who does not are the true enemy.

View Post

He said, “This is the titanic fight of the century. The fight against concentrated power and money. The biggest companies, not just in the US but in the world, are tech monopolies. They are slowly squeezing the life out of people at the edges while still providing products that people enjoy. This is why this is not the same clear-cut case that you can imagine, like railroads or tobacco [were monopolies], but I think it’s going to be that.”

For myself, my career advanced during the golden age of startups (2008–2014) but was dramatically usurped by age of platformization in tech. The eight years I spent building e-commerce were rendered no longer necessary by Shopify. In short, my career was replaced by a robot.

As a 42-year-old seasoned developer on the job market, it’s challenging to make a switch. I generally think most hiring in React probably see me as too old to even be relevant. (Mr. Heinemeier-Hanson, whom I have never met but interacted with online, happens to be one month to the day older than me. Kent Dodds, whom I have also never met and only saw as an audience member at this virtual conference, is 10 years younger than DHH and I are.)

The interviews I do get are all for Rails positions and have a common theme: A large team of mid-career devs in an organization with an old Rails codebase. The codebase shows years of slipped (or non-existent) testing practices, inconsistent patterns and design systems, and rushed code abstractions problems (or, as we say in the biz, “code smells”). People throw around terms like every developer reinvents the wheel. These teams speak to a larger pattern in the tech ecosystem: A very few select Rails apps from the golden age of startups survived, and all the others failed. (If your name is Github, Shopify, or Basecamp please stand up.)

The ones that failed number in the thousands and thousands of Rails apps. They were tech startups that didn’t cross the chasm, or at least didn’t do so on Rails. This left Ruby on Rails as a paradoxically more specialized but smaller community, like the frontrunner everyone forgot about.

(In his famous 1991 book Crossing the Chasm Geoffrey Moore asserted that in the technology adoption lifecycle, a tech company could probably get its friends and family to be innovators and early adopters.Then, the critical make-or-break period is when a tech company moves, like a tortoise, across a large chasm in a desert with predators [competitors] on all sides ready to take it down.)

An experience gap in the industry has left a void: Quality standards aren’t what they should be.

Unlike 2008, today there are tech-haves and tech-have-nots: working at an early-stage startup is now seen as risky, aberrant, and probably a place where soloism is fostered. For early career entrants into software today, it seems they are all hungry to get positions at late-stage tech company (like Google, Apple, Microsoft, Amazon, Facebook, Adobe, Paypal, Shopify, Spotify, etc). When I was in the tech bubble in San Francisco in 2008, if you didn’t get in a Google, Apple, or Yahoo you could certainly be a founder of your own tech startup to gain similar cachet (I’m talking about prestige and admiration, not computer memory). Today nobody ‘goes into’ startups.

Rails was born out of a rebellion against Java. “Look at how little code we can write, Java devs. Now what!” DHH proclaimed proudly. It was always an anti-corporate framework that helped startups get from the innovators to the early adopters. But did Rails apps survive crossing the chasm to early majority?

People ask me what I predict will happen in the future of mobile, e-commerce, web development, and hardware tech. I haven’t the foggiest, I tell them, and if I knew I likely wouldn’t tell you. Keep testing and carry on.

—

Jason Fleetwood-Boldt is runs the consulting agency VERSO COMMERCE. We can help you with site speed, analytics, competitive research, Wix, Shopify, React, or NextJS/Node apps. Get in touch today at https://versocommerce.com

Cover art by IncrediblyBizzy

The WFHpocalpyse Part 3: Remote Work One Year Later

Jason Fleetwood-Boldt — Thu, 18 Mar 2021 13:33:08 +0000

[Originally published on MEDIUM, where you will get all of the hyperlinks. Sorry it was too much work to copy & paste the hyperlinks into here, and the dev.to editor doesn't paste links & styles when you paste into it.]

The day and moment will be forever with me. Nine of my friends (all gay men) from New York City in a bookstore in Rhinebeck, NY, a small enclave about 90 minutes north of the city.

Each of us is an intellectual and iconoclast in our own way. Upon arriving at Oblong Books, we went off separately down the isles to find where our inquisitive minds would take us. It was, perhaps, telling and oddly foreboding in a way.

We were all there together as a matter of a group trip: Our friend Seth had planned this trip for us to celebrate his birthday weekend.

This day is one I remember not only because I happened to have immortalized the occasion on Instagram, but because it was the afternoon of March 14, 2020 in the Empire State. News alerts came through to everyone’s phone at once: Coronavirus was discovered in New York. Governor Cuomo had issued a state of emergency ordering everyone to stay-at-home.

The cognitive dissonance of the freedom we were experiencing on our trip jarred with the change our lives would face upon returning home.

Suddenly everyone was at home. All. The. Time.

I wrote in March 2020 about how the WeWork offices I was renting were paradoxically empty and yet “open.”

Then later in the spring, I wrote another piece about remote work in the first half of my career in the tech industry. I pointed out that, prior to coronavirus, it has always the best and most competent engineers on the teams I’ve worked with who were the lone remote workers.

That is, for the majority of the 2010s, I worked with onsite teams in Manhattan NY. I remember distinctly these teams — for a period— had a remote developer who was some else. Somewhere very far away. (Like Florida, or Missouri). This employee would be a “remote employee” whereas the rest of us all lived and worked in the New York metro region.

If, for example, we had the privilege of working from home one day, that was called “WFH” not being a “remote employee.” Hands down, the ‘remote employees‘ were rockstars. And I don’t use that controversial term lightly.

What do we mean when we say “remote work”?
Are we talking about someone who lives in the same city but commutes into the office on a periodic or intermittent basis?

When I lived and worked in California for many years, it was common for most employees, especially in tech, to work from home one or two days a week.

In California, at that time (the late aughts) most people in tech pretty much drove everywhere. The relationship to your job was based on your driving commute. As well, as most people knew the roads were plagued with timing issues: Sometimes the 101 was completely free, sometimes it was bumper-to-bumper. Strategically timing your drive was an important part of finding a WFH-onsite balance in California.

In 2009 I had a full-time job in the East Bay. There was an employee who left daily at 3:30 to go pick up her child from school. I myself would work onsite all mornings and then return to my home office in the afternoon at 3:30. At 3:30, my door-to-door reverse commute across the East Bay bridge could be done in a nearly-impossible 28 minutes. At 6 pm, the drive across the bridge could take an excruciating hour and a half. (It is called a reverse commute because I lived in the metropolis and worked outside of the metropolis— the opposite of how most people commute.)

So, my point simply is that for tech workers— and especially in places like California— there’s always been a relationship with “WFH” for decades.

The privilege of tech workers to work from home was slowly eroded during the 2010s. In 2013 Marissa Mayer famously banned working from home at Yahoo. [also see this Forbes article], igniting an industry-wide trend making it more taboo and bringing engineers, management, and product teams onsite.

Fast forward to remote work in 2020.
Everybody works from home.

All. The. Time.

Zoom. Zoom on Zoom. Zoom for Zoom. Zoom app of the year. Everybody’s lives on Zoom.

But there are other options: Google Meet, Microsoft Teams. (And of course the one I’ve been using for over a decade, Skype.)

Remote work was here to stay. Or was it?
There was an announcement in May that Facebook would let employees work remotely until the end of 2020. As well, Google followed suit. In July, USA Today wrote:

U.S. companies are doing more than just allowing local employees to work from home, even once a vaccine becomes available for COVID-19 and the health crisis passes, presumably next year. They’re also seeking out new employees anywhere in the country, and even beyond, and letting them work remotely for the long term.

Throughout the summer, news coverage had headlines like “Despite Reopenings, Many Employees Will Work Remotely into 2021 and Beyond.”

In the late summer drawl of August, tech companies were announcing long-term remote work as built-in to their future vision. Zillow announced employees could work remotely ‘indefinitely.’ In October, Ford & Amazon went on record. Then in December, Apple’s Tim Cook said everybody could work remotely “until June 2021.”

June 2021 seemed to be the magical future moment business industry executives had collectively agreed the pandemic would be over. This was the month they were planning to direct everyone back into

the office.

An October Gallup poll found that in August 40% of respondents indicated they preferred working from home, but by September, only 35% did.

A Remote Work Economy
Working from home changed the economy by immediately saving commuters thousands. This ABC news report from February says

[There are even more savings while working from home according to business advisor Molly Hegeman.

“You perhaps aren’t paying a toll road fee, there’s less meals out, there’s less dry cleaning expense,” Hegeman said. “So there certainly are tangible benefits there.”

…[analysis] claims employees can save anywhere from $640 to $6,400 annually by working from home.

“There’s no question that those numbers are realistic,” Hegeman said.

ABC11 New, Working from home saves money for workers and businesses
There were some naysayers. A WSJ article published in July says “projects take longer, collaboration is harder, and training new workers is a struggle” and calls it “unsustainable.”

Microsoft published a bunch of research on this— which is of course suspicious given that they have a product to sell called Microsoft Teams for remote work. They wrote in July:

Looking at days filled with video meetings, stress begins to set in at about two hours into the day. The research suggests several factors lead to this sense of meeting fatigue: having to focus continuously on the screen to extract relevant information and stay engaged; reduced non-verbal cues that help you read the room or know whose turn it is to talk; and screen sharing with very little view of the people you are interacting with.

Microsoft, The future of work—the good, the challenging & the unknown
In early 2021 the tune started to change to “hybrid work”— some people remote and some onsite. This Entrepreneur article from February certainly is gung-ho about remote work for every business.

But now, due to the pandemic, people have finally embraced remote work, even if not everyone likes it. Business owners are now seeing the advantages to creating remote positions in their company that allow flexibility and “future-proofing” their businesses from any future catastrophe that could close their doors.

Entrepreneur, The Stigma of Working From Home Is Officially No More
But the business shift is a different question of the worker shift, and I wonder what the supply & demand dynamics will mean for which types of work will get ‘allowed’ to be remote and which won’t.

Now, in March, when a vaccinated population seems somehow within reach the conversation has turned to flexibility and flex-work.

Back in the early 2000s being onsite was also called “colocated”. Of course, to point out the decline of the word ‘colocated’ is itself a funny acknowledgment about how and where people work: It’s entirely normal for companies to have “an office here” or an “office there.”

So by logical extension, there’s always some amount of people working in different places.

The real question is: Do you need to work in the same place?

Remote employees or WFH?
The question about the shift in the economy of work is:

Are we talking about a new society in which we normalize working from home on a part-time basis?

It turns out when surveyed, most employees want the flexibility to work from home but also the ability to go into an office.

Workers in corporate America are already starting to hear the drumbeat of “back to the office we go” as society anticipates widespread vaccination. I predict what will happen is that we’ll revert back to how it was in the early 2010 and late 2000s: Most specialist, elite, and many tech engineers will be permitted some kind of part-time or permanent remote work status. Managers, corporate execs, engineering managers, or anyone who wants to move up in the company will immediately be back in the office.

It is generally accepted in the world of corporate America that putting in “face time” (that is, showing your face in the office) makes you more likely to be promoted and get raises. Do we expect America after the pandemic to be any different? Sure, many people from different industries that didn’t normally work remotely were introduced to the remote work paradigm.

But the fundamental dynamics of how people rise up in corporate America haven’t changed: corporate America likes pretty people, especially favoring white men, who show up to the office.

Although I do believe we will see a general shift towards flexible work (for example, normalizing leaving mid-afternoon to pick one’s kid from school or working from home one or two days/week), I doubt there will be a fundamental shift.

Gen Z knows it: Greuling 9-to-9 workweeks (in Japan this is call 6×9 where you work 6 days/week 9 hours/day) with late nights in the office are a thing of the past.

The 9-to-5, of course, will always be a mainstay of how we relate to our concept of work, but even that is changing. Deep work, basic work, present work. All of these kinds of work are going to spin a movement of re-thinking and re-defining work.

For the few specialized skills, companies will have to take what they can get if remote engineering becomes the norm.

Gen Z calls these people ‘simps’; the Boomers used a term I’ve decided is impolite: I’ll just call them the managerial class and corporate ladder-climbers: If that’s you, it’s probably going to be back to business in the office.

—

The How and Why of End-to-End Testing

Jason Fleetwood-Boldt — Mon, 22 Feb 2021 19:37:29 +0000

Perhaps the most significant and under-appreciated aspect of Rails and Agile software development of the last roughly 15 years is the culture and discipline around testing and test-driven development.

I’ve never come to understand why testing and TDD is often maligned by the loudest, most vocal developers: It’s too slow, it takes longer, the boss didn’t want or ask for it, they’ll say.

You don’t hear about these developers or this ideology often in professional circles, but you encounter them quickly in the wild west of freelance development.

[SEIZURE WARNING: There is an animated GIF towards the bottom of this page that flashes. Please disable animated GIFs if you are susceptible to flashing lights.]

Indeed, much of the popular and discussed rhetoric in the community of Rails is about a codebase’s test suite (a suite of tests for your whole application, this collective is called “the tests” or “the specs”): How much of your codebase is covered (as measured in %, which I discuss further below)? How easy are the tests to write? Do they use factories or fixtures? How brittle are they? Do they test the right things and are they valuable?

All of these are correct questions. Although there is no substitute for the day-in-day-out practice of this to become great at testing, I will try to offer some broad ‘best practice’ answers to these questions.

The enlightened developers don’t ask or care about whether or not the boss told us to write a tested codebase. We just know the answers to the above questions and do what’s right for the codebase: write specs.

Testing has varying degrees, varying methods, varying strengths.

In 99 Bottles of OOP, Metz, Owen, and Stankus make this interesting observation:

Belief in the value of TDD has become mainstream, and the pressure to follow this practice approaches an unspoken mandate. Acceptance of this mandate is illustrated by the fact that it’s common for folks who don’t test to tender sheepish apologies. Even those who don’t test seem to believe they ought to do so.

(Metz, et al, 99 Bottles of OOP, Second Edition, 2020. p 43)

So testing exists in a murky space: The top dev shops and teams know it is essential, but its implementation is inconsistent. Sadly, I’ve seen lots of development happen where people either just don’t write tests, write tests blindly, use tests as a cudgel, or skip end-to-end testing altogether.

Many years in this industry have led me to what seems like an extreme position. Not writing tests should be seen as akin to malpractice in software development. Hiring someone to write untested code should be outlawed.

Having a tested codebase is absolutely the most significant benchmark in producing quality software today. If you are producing serious application development but you don’t have tests, you have already lost.

Having a good test suite is not only the benchmark of quality, it means that you can refactor with confidence.

There are two kinds of tests you should learn and write:

Unit testing (also called model testing or black-box testing)
End-to-end testing (also called integration testing, feature testing, or system tests)
These go by different names. Focus on the how and why of testing and don’t get lost in the implementation details of the different kinds of tests. (To learn to do testing in Ruby, you can check out my course where I go over all the details.)

Unit Testing

Unit tests are the “lowest-level” tests. In unit testing, we are testing only one single unit of code: Typically for Rails, a model. When we talk about Unit testing in other languages, it means the same as it does for Rails, but might be applied in other contexts.

The thing you are testing is a black box. In your test, you will give your black box some inputs, tell it to do something, and assert that a specific output has been produced. The internals (implementation details) of the black box should not be known to your unit test.

This fundamental tenet of unit testing is probably one of the single most commonly repeated axioms of knowledge in software development today.

The way to miss the boat here (unfortunately) is to follow the axiom strictly but misunderstand why you are doing it.

Testing, and especially learning to practice test-driven development (that’s when you force yourself not to write any code unless you write a test first), is in fact a lot deeper and more significant than just about quality, refactoring, and black boxes. (Although if you’ve learned that much by now you’re on the right track.)

Most people think that software, especially web software, is written once and then done. This is a fallacy: Any serious piece of software today is iterated and iterated. Even if you are writing an application for rolling out all at once, on the web there should always be a feedback loop.

Perhaps one of the worst and most problematic anti-patterns I’ve ever seen is when contractors write code, it is deployed, and nobody ever looks at any error logs. Or any stack-traces. Or even at the database records. (Typically this happens less in the context of companies hiring employees because employees tend to keep working for your company on an ongoing basis whereas contractors tend to ‘deliver’ the product and then leave.)

It’s not just about “catching a bug’” here or there. Or tweaking or modifying the software once it’s live. (Which, to be fair, most developers don’t actually like to do.)

It’s about the fact that once it is live, anything and everything can and will happen. As a result, the data in your data stores might get into all kinds of states you weren’t expecting. Or maybe someone visits your website in a browser that doesn’t support the Javascript syntax you used. Or maybe this, or maybe that. It’s always something.

This is the marriage of testing & ‘real life’: You want your tests to be ‘as isolated’ as possible, yet at the same time ‘as realistic’ as they need to be in order to anticipate what your users will experience.

That’s the right balance. Your code doesn’t exist in a vacuum, and the test environment is only a figment of your imagination. The unit test is valuable to you because it is as realistic as it needs to be to mimic what will happen to your app in the real, wild world of production.

With unit testing, you aren’t actually putting the whole application through its places: You’re just testing one unit against a set of assertions.

In the wild (that is, real live websites), all kinds of chaos happens. Your assumptions that user_id would never be nil, for example, proves out not to be the case in one small step of the workflow because the user hasn’t been assigned yet. (Stop me if you’ve heard this one before.)

You never wrote a spec for the user_id being nil, because you assumed that that could never happen. Well, it did. Or rather, it might.

Many developers, especially the ones with something to prove, get too focused on unit testing. For one thing, they use the percentage of codebase covered as a badge of honor.

Percentage of Codebase Covered

When you run your tests, a special tool called a coverage reporter can scan the lines of code in your application to determine if that line of code was run through during your test. It shows you which lines got the test to run over them and which lines were ‘missed.’

It doesn’t tell you if your test was correct, that it asserted the correct thing of course. It just tells you where you’ve missed lines of code. The typical benchmark for a well-tested Rails application is about 85–95% test coverage. (Because of various nuanced factors, there are always some files that you can’t or don’t need to test— typically not your application files.)

Here I use a tool in Ruby called simplecov-rcov to show which lines (precisely, line-by-line, and file-by-file) are covered. Here in this baby little project of mine, I have an unfortunate 36.55% of my codebase covered:

As you see, the files are sorted with the least covered files shown up top. The top files are in red and say “0.00 %” covered because the test suite does not go into that file.

When I click into the file, I can actually see which lines are covered and uncovered, in red & green like so:

(Here’s a great example of that “it only happens in the wild” thing I was talking about earlier. In theory, I should never get passed a room id (params[:room]) that is not in my database [see line 4], but in practice, for some reason, while I was debugging I did. So I added a small little guard to catch for this while debugging, thus making the line of code inside the if statement uncovered by my test suite.)

Correlating the total percentage of test coverage to your code quality and/or value of the tests is often a fallacy: Look at the percentage of codebase covered, but not every day.

The problem with over-emphasis on unit testing is the dirty little secret of unit testing: Unit tests rarely catch bugs.

So why do we unit test at all then? Unit tests do catch all of your problems when you are upgrading.

You should unit test your code for the following four reasons:

(1) It helps you think about and structure your code more consistently.

(2) It will help you produce cleaner, more easily reasoned code as you refactor.

(3) Refactoring will, in turn, reveal more about the form (or shape) of your application that you couldn’t realize upfront.

(4) Your unit tests will catch bugs quickly when you upgrade Rails.

That’s it. Notice that not listed here is ‘catching regressions’ (or bugs). That’s significant because many developers think unit testing cover all of their bases. Not only do they not cover all of your bases: They don’t even catch or prevent regressions (bugs) in live production apps very often.

Testing is important. Unit testing and end-to-end testing are both important, but between the two, end-to-end testing is the most important of all.

End-To-End Testing

End-to-end testing goes by many names: System specs, integration specs, Capybara, Cypress, Selenium.

End-to-end testing for Javascript applications means the following things:

Your test starts in the database. I like factories, but fixtures are also popular.
Your test ‘hits’ the server (Rails, Node, Java, etc)
The server returns a front-end in Javascript
Your test interacts in Javascript with your web page

If you do not have all four of those components, you do not have end-to-end testing. Using Capybara, you are really doing all of these things.

If you’ve never seen a Capybara test run, here’s what it looks like:

A moving visualization showing a Selenium suite running in a Rails application.

I like to show this to people because I don’t think many people see it. Often the specs are run in headless mode, which means those things are happening just not on the screen. (But you’re still really doing invisibly them which is the important part.) While headless mode is much faster (and typically preferred by developers), using Selenium to control a real browser is an extraordinarily powerful tool— not for the development, but for evangelizing these techniques and spreading the good word of end-to-end-testing.

Most non-developers simply don’t even know what this is. I’ve talked to countless CEOs, product people, people who’ve worked in tech for years and have never even seen an end-to-end test be run. (They’ve literally never witnessed with their own eyes what I’ve shown you in the animated GIF above.)

What these people don’t even understand is that TDD, and end-to-end testing, is a practice of writing a web application development that is itself an advancement. The advancement facilitates a more rapid development process, less code debt, and a lower cost of change.

Without having actually witnessed the test runner run against the browser, it is shocking to me how many people in positions of authority are happy to hire teams of QA people to do manual testing for every new feature or release. (Disparigingly called “monkey testing” by the code-testing community.) With the easy and “inexpensive” availability of remote QA people, an industry of people are happy to keep monkey testing until judgment day. What they don’t know is that those of us who are code-testing are already in the promised land of sweet milk and honey.

My biggest disappointment personally moving from Rails to the Javascript world (Vue, Ember, Angular, React) is the lack of end-to-end-testing in Javascript. It’s not that JSers don’t ever do end-to-end testing— it’s that it might not be possible in your setup or your team.

If you are only working on the frontend, by definition you don’t have access to the database or the backend.

The fundamental issue with the shift away from Rails monoliths and towards microservices is: How are these apps tested?

I don’t know about you, but after years of being a user of microservices, I’m not entirely sold.

Don’t get me wrong: I am not categorically opposed to microservices. (Your database, and Redis, both probably already in your app, could be thought of as microservices and they work very well for us Rails developers.)

But designing applications around microservices is a paradigm ideal for huge conglomerate platforms that simultaneously want to track you, show you ads, and curate massive amounts of content using algorithms.

Most apps aren’t Facebook. I hypothesize that the great apps of the 2020s and 2030s won’t be like Facebook either.

That’s why having the power to do database migrations without involving “a DBA” (or a separate database team), or having to get the change through a backend team— something which is normal for smaller startups and Rails — has been so powerful for the last 15 years.

The social media companies are well poised for leveraging microservices, but most small-medium (even large) Rails apps are not, and here’s why: Doing end-to-end testing with a suite of microservices is a huge headache.

It’s a lot of extra work, and because it’s so hard many developers just don’t do it. Instead, they fall back lazily to their unit testing and run their test coverage reports and say they have tested code. What? The API sent a field to the React Native app that it couldn’t understand so there’s a bug?

Oh well, that was the React Native developer’s problem. OR, that was the services layer problem.

It’s a slow, creeping NIMBY (not-in-my-backyard) or NIH (not-invented-here) kind of psychology that I see more and more as I learn about segregated, siloed teams where there’s a backend in Rails, a frontend in React or another JS framework, and a mobile app — all written by segregated, separated teams who need to have product managers coordinate changes between them.

Already we see lots of major companies with websites made up of thousands of microservices. I don’t think our web is better because of it: for me, most of my experience using these websites is spinning and waiting to load. Every interaction feels like a mindless, aimless journey waiting for the widget to load the next set of posts to give me that dopamine-kick. Everywhere I look things kind of work, mostly, but every now and then just sort of have little half-bugs or non-responses. It’s all over Facebook, and sadly it seems like more and more of the web I use this degradation in experience quality has gotten worse and worse over the last few years.

It’s a disaster. I blame microservices.

I hear about everybody rushing into mobile development or Node development or re-writing it all in React and I just wonder: Where are the lessons learned by the Rubyists of the last 15 years?

Does anyone care about end-to-end-testing anymore? I predict the shortsightedness will be short-lived, and that testing will see a resurgence of the importance of popularity in the 2020s.

I don’t know where the web or software will go next, but I do know that end-to-end testing, as pioneered by Selenium in the last 10 years, is one of the most significant stories to happen to software development. There will always be CEOs who say they don’t care about tests. Don’t listen to them (also, don’t work for and don’t fund them). Keep testing and carry on.

[Disclaimer: The conjecture made herein should be thought of in the context of web application development, specifically modern Javascript apps. I wouldn’t presume to make generalizations about other kinds of software development but I know that testing is a big deal in other software development too.]

How to Be a Great Software Project Manager or Product Owner for Scrum

Jason Fleetwood-Boldt — Sun, 31 Jan 2021 21:52:32 +0000

I think of scrum and agile as always a work in progress.

That’s because different teams can practice it to different levels of strictness. That’s ok. The naysayers to scrum or agile will call it a top-heavy process, where a “Certified Scrum Master” dictates strict processes.

Sometimes I hear of teams rejecting the ideas of agile wholesale on these grounds (“we don’t need it” / “it doesn’t work for us”). “Agile” is a buzzword, which fundamentally once meant “changing as you go” but now seems less and less relevant, at least to me, since so many people use that word so poorly.

“Scrum” is a funny word. It comes from a sports play in England—you see Rugby played to this day across parks in England.

I’m not sure what it is about this particular word, or this particular software development practice —or the lineage between the two, but “scrum” as a practice, is the practice that resonates most with me in my years as a developer, and it is the one that I’ve seen worked the best on the teams I’ve been on and run.

I’m all in support of most of core concepts of “agile,” but I think the word has been watered-down. As many of my peers in the industry have pointed out to me, lots of people think they are practicing “agile development” when in fact they are still plagued by all the same mismanagement pitfalls of other options (that is, “waterfall”).

“Scrum” is a software development practice that has a lineage to some of the best minds in software development and is a set of practices that work. While “agile” has more of a value system involved in it, scrum is a little more focused on the nuts and bolts of the SDLC (“software development lifecycle”).

Scrum, also, is not a buzzword.

Sure, the heart of it is to work in short iterations, but I want to list out a few key elements, for Product managers, which often get easily missed. This post is aimed at Product managers (or “Product Owner” in scrum’s more formal terminology.)

If I could highlight one thesis, it would be this: Scrum is about optimizing the operational efficiency of your team. If you don’t get that, or you aren’t getting that from your practice of Scrum, you aren’t doing it right.

Here the three most significant anti-patterns, or problems, I see on software development teams. These are like red flags you aren’t practicing scrum.

Single queued developers. (DON’T) Developers should focus on one story at a time. Period. If you are queuing up a single developer you are doing it wrong. Each dev should finish their assignment and then come back to the team and take the next highest-priority item off the list. Ok, so I know lots of teams and devs work with a ‘back burner’ story in case you get ‘blocked’ on your first story. I get it, and I concede for your team this may not be an absolute. Nonetheless, what this principle gets at is efficiency mechanics, and the blocker for the dev on the first story represents a cog in your wheel. (This is where the Kanban principle of pulling the ‘STOP’ chain on the assembly line comes in.)
Unclear definition of done (DOD). (BAD!) Ok, so I didn’t make up “definition of done,” but I like to emphasize it. Traditionally, “definition of done” means the definition by which the team considers the story done. On teams I’ve been on and managed the concept of “definition of done” expands to each step of the SDLC (software development lifecycle), in this order:

Concept
Design/Wireframe or Spec
Execution
QA or UAT (user acceptance)
Delivery to production

This is a specific order. I didn’t make it up. I did combine some steps so your team’s process might look a little different. But the concept is the same: You work on each little piece until you’re done, and then your start again. Wash-rinse-repeat, as they say.

However your team does it is fine: Scrum is not about rigid dictation of process. Scrum is about the concept that each player will hold themselves accountable (or be held accountable) to getting the ball (the story, in software development) to the next step. That’s why DOD (“definition of done”) can and should happen at each and every step. The business owner or CEO signs off on the concept; marketing and design sign off on Design; you (the product manager) sign off Wireframe or Spec (with the developers involvement to make sure they can actually build what you want them to build). The developer writes the code, and the QA person confirms the feature works and you confirm it is acceptable to the customer, client, or will work for the end user. Finally, it is deployed to production. Get it? That’s the SDLC. That’s the whole thing. (Agile/scrum secret: scrum looks just like waterfall but you do this whole 5-step process in short, quick, iterations. More on that in another blog post)

Too much formal process? Ok, fine. Then invent an informal process. You can change the rules of scrum! That’s fine. Nobody every said scrum dictated a formal process and whoever said that shouldn’t be speaking about it. But, please, understand this process works for a reason. If you’re going to make your process more informal, know why the five steps are important and make sure your informal process at each of the steps actually works.

Get everyone to agree to this, including the stakeholder, the designers and of course the developers. Sometimes a designer or a stakeholder might say “nothing’s ever done” or “everything is a moving target.” Maybe it feels that way to you on your team. I can sympathize. But if the stakeholder or designer doesn’t want a clear “sign-off” or “thump-up” or “it’s ready now” step then that’s a sign they haven’t bought into the very heart of scrum. Go back to the beginning and start again.

(It’s rare that you have to deal with a developer not wanting a clear definition of done— good devs I’ve worked with are goal and completion-oriented by our nature. Sometimes jr. developers can drag their feet, let projects slip, and not finish things. As the PO, disciplining this kind of developer isn’t your responsibility, but be on the lookout for it and understand the DOD applies all across the whole pipeline, at each and every step. That’s why the two steps—both of which really you can and should be involved with—are so important.)

“Definition of done” (DOD) is such an important thing to scrum. Really, I didn’t make it up. Look it up; it’s a thing. Lack of discipline around “done” is the single most significant team anti-pattern I see.

Developers complain about lack of prioritization (BAD!) Ok, so this is a very common thing on many software teams. You have lots and lots of ideas and stories. New ones come up all the time. You have meetings or stand-up and developers say, “I’m looking at list of unprioritized stories.” If developers say this in your meetings, especially if they say it a lot, you’re doing something wrong.

It is the Product owner/manager’s job to represent the business’s interests, or the customer’s interest, in prioritizing the worklog, or in formal scrum known as “backlog.” This is really, really important. Yes, developers do lots of things related to prioritization of code and code debt that you (the product manager) may not understand. And yes, sometimes developers can be welcoming to prioritization (or, as is often the case, re-prioritization), and other times, developers can be very unwelcoming of re-prioritization. That’s software development. If you want to work with the best dev teams, these are the kinds of nuances you need to navigate as a successful Product manager.

Some other tips

Six more tips for Product managers. Keep in mind these are written from a developer to a Product manager, so take with a grain of salt. These are opinionated and based on my several years of working on scrum teams.

Don’t skip user stories. This one seems obvious but I’m amazed how many Product people are so quick to skip formal User Stories. They’re so easy! I’m pretty formal about them, preferring the exact style:

As a ___
When I ___
And ___
I should ___

There are a few variations on this style and any of them is OK.

Learn how to write User Stories and use them. In strict scrum, you have only 1 user story per ticket or “story.” This often doesn’t work. If you have a ‘ticket’ or ‘story’ system. I’ll give you permission now: It’s ok to have more than one User story in a single ticket.

When I write them, I just string them together, right after the other, sometimes lettering or numbering them.

STORY A
As a User,
When I type the right username & password,
And click “Login”,
I should be successfully logged in

STORY B
As a User,
When I type the wrong username & password,
And click “Login”,
I should see a message tell me my username & password are incorrect

That’s fine! So there’s a little repetition. For QA people, repetition is A-OK. By writing the stories up front you are setting up the QA step for success (more on that later).

Don’t skip the user stories. Write them, get stakeholder buy-in on them, and believe in them.

(The most common pattern is “As a __, I want to _, so that I can _”. Notice here it’s just like the one I use above except for the “I want to” / “so that I can” part. I have nothing against this pattern but I find it to be one step removed from product specifications—almost as though this pattern is written from the perspective of a business owner rather than the technical product owner, but in many ways and on many teams this is a distinction without a difference so I typically don’t dwell on it. Either “When I _” or “I want to __” is ok, the former is slightly more product-centric and the latter slightly more business-objective centric; both are preferable to no stories at all or stories written with implicit assumptions that aren’t clearly written out.)

Forcing yourself (the product owner) to go through this discipline forces you to methodically put yourself in the end user (or customer’s) perspective, and it forces out all of the little extra edge cases which need to be explicitly dealt with by the programmers. This kind of explicitness is nearly universally welcomed by developers, and more work up-front in this area typically makes the development faster and more efficient.

Don’t skip wireframes. It’s amazing to me how many Product people try to just have ‘meetings’ with developers. I know some people are better verbal communicators, but your job as Product owner is to document, document, document those meetings.

Like User Stories, wireframes drive the conversation about the software — which is most of the work! It’s the thing you didn’t think of when you just had the verbal conversation that you did think of when you put pen to paper (literally or metaphorically).

It’s that thing that your boss or other colleague thinks of after you show them the wireframe. You didn’t think of it before because some people are visual thinkers, and the feedback loop of translating the verbal conversation to the wireframe is itself the process of refining and improving the requirements themselves.

By getting all those little tidbits out up-front, you are reducing waste in the development process and speeding up the developers.

Do QA yourself. The best product people do QA themselves, even if there’s a separate “QA specialist” on the team. Yes, Product & QA are distinct skillsets. Yes, on some small teams, they are the same person. Don’t be one of those Product people who eschews doing QA yourself. If you’re not the best at it, learn to get better at it and QA the projects you are managing. It’ll be better for the team and it’ll make you a better Product manager.
Get stakeholder (or ‘exec’, as is commonly heard in startupland) buy-in early and before the devs write a line of code. Really, don’t skip this step. This is your primary job. If you are going back to the devs for revisions after revision, you aren’t doing your job right.
Think like a dev but don’t think like a dev. This one is hard. Know some things about what you think is possible with the technology you have and separate that from the technology you think you could have. Understand there’s a constant pay-it-now-or-pay-it-later tension in software development. Give just enough technical prowess without stepping on developers’ toes.
Ask for what you need. Don’t be afraid to ask the devs, but try to do so politely and without making any assumptions you know anything. Just report the facts, ask for what you need, and offer any helpful information.
Don’t ever say “but it worked before” or “but it used to work” or anything sounding like that. This is never something a developer wants to hear. If you are in a position of software development management, don’t lean on this trope.

Yes, developers are responsible for regressions that happen on their code deploys, sure. And yes, sometimes, when a developer deploys some code, it introduces a regression. But 95% I hear these words coming from non-technical people it is in fact not related whatsoever to anything any developer did. I also happen to think it is an unfortunate trope used by nontechnical people who fundamentally do not understand a concept we call in software development called software entropy. (Really I’m not making this up!).

If you really think it “used to work” just before a code change, it’s OK to report that in an evidence-based way when you report the bug to the developer. (As in, “Last successful one was at before the deploy may or may not be related.”) That’s totally cool. In fact, developers WANT you give them as much information as possible. Just do it an nice way and evidence-based fashion. Use your words, use your screenshots, and throw in “may or may not be related.” Go ahead, it’ll work like magic. I promise.

(Some really passive-aggressive version I’ve seen of this is to NOT tell the developer the key piece of information the Product manager has related to the regression thinking they will ‘test’ or ‘challenge’ the dev to find the bug. This, too, is not cool.)

As a nontechnical person, let me tell you something that the developers you work with really want to scream at you: You understand a fraction of what’s actually going on under the hood. You already know this. I don’t need to tell you this and the developers don’t need to either.

Leaning on “it used to work” consistently is an accusatory sign of an amateur manager who just doesn’t get it. If you find yourself doing this, put yourself in check and ask if this career is right for you.

Conclusion

Being a great Product manager, like all things in life, takes compassion. As a boss and mentor of mine (Rob Rubin) once told me I think smartly: the Product owner is the most leveraged individual on a Scrum team. That is, if you’re in a company and not on the Product team (like you’re the stakeholder), to get what you want out of the Engineers you should make friends with the Product people. (Thanks Rob!)

You have a great road ahead of you should you heed the discipline’s core principles. Fight them, and you may have a rocky time, especially in the areas of code debt, incorrect estimation process, missed deliverables, and mismatches between what the developers are doing and what the client or company needs.

Wherever you are at today is fine. But think about the discipline’s various approaches and think about how to take each piece of your process once small step in the direction of scrum. You might wavier, you might run into various problems, but each of them can be solved by coming back to the disciplines at each step of the chain, always bringing your team closer and closer to its most efficient state. This is why I think of it as “always a work in progress.” Teams typically don’t get it right on the first try, and have to do it inefficient ways before they can move towards scrum.

The One About the Chickens and The Pigs (aka What Stand-up Is and What Stand-up Isn’t)

Jason Fleetwood-Boldt — Sun, 31 Jan 2021 21:50:13 +0000

There’s an old adage in scrum software development about chickens and pigs at stand-up. Chickens are product managers and pigs are developers.

You don’t hear it too often anymore, probably because these days it feels a little sexist. (It’s not lost on anyone anymore how gendered the roles of product manager and developer feel in most tech companies— the product people being women and the developers being men.)

It takes a leap of faith to understand what it means, and what even is the question it’s asking anyway.

The question is fairly basic: Who participates at your morning company or engineering stand-up?

That is, I mean, really: who speaks and who does not speak. I know it sounds rigid and those of us to talk about it get called a “scrum bullwhip” (a title I proudly wear). Pigs speak at stand-up. Chickens (product managers, CEOs, and stake-holders), if they come to stand-up (and generally only product managers should) aren’t supposed to speak unless spoken to.

What? It sounds like some kind of renaissance classism like people used to say: “children shouldn’t speak until spoken to,” but to understand the chicken & pig adage is to learn something core about scrum and the stand-up meeting itself.

Standup is about managing the work, not the people.
What the F does this have to do with chickens and pigs, you might be asking? (I warned you it’s a long way around with this one.) Well, the idea is that we’re making breakfast. We’re all making breakfast together.

The end result is the breakfast. How we get there matters, but not everybody’s contribution is equal.

The scrum process forces the engineers to prioritize working on the very most important thing first (hopefully, the one task they have assigned to them).

Most product people, stakeholders, and CEOs being unfamiliar with the concept of “stand-up,” incorrectly assume or treat engineering stand up as a “management meeting” and think it’s their opportunity to talk or get what they want.

Sadly, this is, in fact, the opposite of scrum. Instead, scrum is about aligning your engineering efforts with your organizational-wide goals.

These days many of the millennials, born of the gadget generation, have grown in jobs where they can hide their high-functioning adult ADHD (Attention Deficit Hyperactive Disorder).

A high-performing engineering team works in total contrast to this ADHD, attention-switching, always-on-call mentality: The thing to work on is the one thing right in front of you, never anything else.

If that thing that you’re working on isn’t the most important thing, then the CEO or product owners haven’t correctly prioritized the backlog. When product people and CEOs come to scrum and participate it’s like a group of people trying to make breakfast when some other people trying to plan for lunch or dinner or tomorrow’s meals. The appropriate response you’ll get from the developers is: “Hey, back off, we’re making breakfast now, come back when we’re done and we’ll talk about lunch.”

The chickens lay eggs. The pigs are slaughtered. After breakfast is made, the chickens are still alive.

It’s a grotesque metaphor and one that can even be insulting to product people because it makes them feel like their contribution isn’t valuable. Well, that’s part of the crux of it too:

It isn’t that the product development contribution as a chicken (product owner) isn’t valuable, it’s that software development is a moving train.
As a developer, so that I can achieve flow, I should have the materials needed to do the ticket (story) I’m working on without a lot of back and forth with the stakeholder.

In fact, the correct amount of back and forth with the stakeholder is 0 (zero).

Each and every back and forth costs wait states — that is, times when the flow of the craft (that is, building the software) has to wait for someone else in the chain. If this is you then your process is most definitely held back by wait states.

What does this have to do with chickens not speaking at standup? It’s not that chickens aren’t actually supposed to literally be quiet, it means that they don’t have a turn when you go around each “giving” your stand-up.

Why don’t chickens have a turn? Because stand-up is about 1) what code we accomplished yesterday, 2) what we’re working on today, and 3) removing blockers.

The chickens don’t actually accomplish coding tasks. They contribute to the coding tasks (things like wireframes, mockups, designs, written user stories, business cases)— these are called artifacts. But these artifacts, although they can help the process, aren’t actually the finished result of working production-quality code. (Except, arguably, in the case of web designs where the designs are translated into working code.)

It’s a really old, sexist, and out-dated adage that comes from the 90s and in 2020 it’s probably insulting to most.
I haven’t yet thought of a good replacement, because the core of the adage (which I admit is kind of nonsense on many levels when you really try to lay it all out) is about the fact that the production of the code is what matters. Or, if you will, the end result (which in software development is working code.)

Scrum assumes and prioritizes high-performance engagement. At the same time, it shines a light on low-performing tools, processes, and people. It is the “sunlight” that will disinfect any broken engineering process.

It ain’t easy, and it ain’t for everyone, but when practiced right, it remains the most engaged and accelerated form of software discipline today.

This post was originally published on my blog here.

Universal Track Manager for tracking Traffic to Rails Websites (V0.7.0)

Jason Fleetwood-Boldt — Sun, 31 Jan 2021 21:38:26 +0000

A developer from down under (Australia) contributed to my open-source gem, Universal Track Manager. Thanks to a fellow who goes by github/sstrumph, you can now extend Universal Track Manager to track any kind of inbound field (beyond just the standard 5: utm_campaign, utm_term, utm_source, utm_ content, utm_term).

Now published: v0.7.0 of Universal Track Manager.

You can learn more about the gem, what you can use it for, if it's legal & ethical (spoilers: maybe not!) on the "README" page here on Github:
https://github.com/jasonfb/universal_track_manager

or get more info on my website
https://blog.jasonfleetwoodboldt.com/my-open-source-projects/universal-track-manager/

About
Universal Track Manager, also known as UTM, is a gem to track your visitors and their pageloads. You can use it to track Urchin Tracking Module (UTM) parameters, and the fact that these two things abbreviate to the same letters is play-on-words.

You can use Universal Track Manager to track simple information like browser, IP address, http referrer, and your inbound UTM (Urchin Tracking Module) parameters. Because UTM parameters are used most commonly with advirtising, we also refer to tracking your UTM parameters as “ad campaigns” or just “campaigns”.

In particular, this Gem can be used to solve the common first-land problem where UTM parameters are present only in the first page of a user’s visit, but not available naturally a few steps later when the event you want to track happens (see ‘UTM Hooks’)

Visits are a lot like Rails sessions; in fact, this Gem piggybacks off of Rails sessions for tracking the visit. (You will need to have a session store set up and in place.)

However, visits are not identical to sessions. More than one visit can have been created from the same session.

A session will have only one visit at a time. If a new visit event happens within an existing session, like the user returns in the same browser the following day, the old visit gets evicted from the session and a link between the newly created visit and old visit is maintained in the visits table.

Is this Ethical?
It is important to understand there are many different data points can could possibly be collected in today’s web traffic. You may use this gem at your own discretion, and you can choose either more or less data capturing, as well ore more or less data integration with your users.

The reason I underscore this point is that the safest data is anonymized data.

The general Rails best practice for the last decade has been to keep only session data with the user, but anonymize identifying data (like IP, behavior, browser), which could be used to identify individual users.

Please see the section ‘Granularity of Data Capture’ to understand the many different levels of (non-)privacy you may choose as the website operators.

Is this Legal?
In any country or region where a privacy law like the GDPR or California Consumer Privacy Act is in effect, getting informed consent to track this information is just one part of what you must do to comply with the law.

Most privacy laws regulate the usage, storage, transmission, and removal of this data once you are retaining it in your database as well.

This gem express captures this data, as described in this README document, and by using this gem you are responsible for complying with the appropriate laws and regulations subject to you.

You will note that most old privacy policies talk about much of this data being stored in “log files.” This gem takes the data retention farther and stores the data into the database. (So you should modify your privacy policy accordingly.)

Please consult a legal expert familiar with the laws of your region regarding data retention and capture if you are going to use this gem.