DEV Community: Max Chernyak

Adventures in Ruby-esque type enforcement

Max Chernyak — Sat, 13 May 2023 04:00:00 +0000

In Ruby you can kinda pretend that you have type enforcement at runtime, because Ruby is very flexible. This could be a useful-enough thing to do to organize and formalize the “guarding” of your data. As a disclaimer, I’m not actually a huge fan of this practice, because I think that if you’re going to enforce types at runtime, you may as well achieve the same result via learning how to write good constructors and immutable objects. I believe the focus should be on controlling the flow of data from source to destination, not declaring types to guard against every generic use case. Nevertheless, for many existing codebases out there, runtime-level types might be the right way to improve maintainability, so I decided to experiment with my own approach.

Again, to be clear, the above criticism is only about enforcing types at runtime, not at compile time.

Before I start, there are already libraries out there that let you declare types at runtime. They offer a bunch of fancy-named classes and methods that let you construct your own types. I disagree with their approach, because it introduces a lot of cognitive overhead. They expect me to learn all of that extra vocabulary only to end up running boolean expressions against my values. Why not just let me write those boolean expressions in the first place? This is the whole premise of my experiment: it seems easier to write a plain Ruby value check than to figure out how to build it with fancy type libraries.

A while ago, I wrote a little library called portrayal, which is a simple Struct-like object builder. It lets you declare keywords, which are just attr_accessors and a default initialize, plus some extra convenience. Using this lib as the basis, I wrote a proof of concept extension called Portrayal::Guards. In this article I show you how it works.

Leaning into boolean expressions

Let’s say we have a class Person, who has age and favorite_beer.

class Person
  extend Portrayal

  keyword :age
  keyword :favorite_beer, default: nil
  public :age=, :favorite_beer=
end

Note: Normally setters are protected, but I’m making them public above to illustrate how guards work.

Imagine that our data type requirements are as follows:

Age must be an integer between 0 and 130
Favorite beer must be nil or any string
If favorite beer is not nil, then age must be >=21

Here is one simple way to do this with Portrayal::Guards.

class Person
  extend Portrayal

  keyword :age
  keyword :favorite_beer, default: nil
  public :age=, :favorite_beer=

  guard('age must be human and beer is only for >=21yo') {
    age.is_a?(Integer) && (0..130).cover?(age) &&
      (favorite_beer.nil? || (favorite_beer.is_a?(String) && age >= 21))
  }
end

This guard can be declared anywhere in the class body. It has a single boolean expression in it. If it returns anything truthy, the guard passes. If it returns false or nil, the guard fails. The string argument serves as the error message in case it fails. With this single guard we actually solved the whole problem.

Check out how this guard protects our object:

> Person.new(age: 200)
ArgumentError: age must be human and beer is only for >=21yo

> person = Person.new(age: 5)
=> #<Person @age=5, @favorite_beer=nil>

> person.favorite_beer = 'corona'
ArgumentError: age must be human and beer is only for >=21yo

> person.update(age: 200, favorite_beer: 9)
=> {:base=>["age must be human and beer is only for >=21yo"]}

> person.update(age: 30, favorite_beer: 'corona')
=> nil

> person
=> #<Person @age=30, @favorite_beer="corona">

Three things to notice here:

This guard is guarding both initialize (.new), and writer methods.
We have a special method update, which lets you update multiple values at the same time. This helps resolve situations when you can’t assign attributes one at a time, because guards cross-check them.
Notice that the error we got from update is under a key :base. Keep it in mind for now, I will explain this later.

This was easy, it’s just a plain boolean expression that now completely guards our attributes. However, the expression is a little bit unwieldy, and the error message is not super useful for telling us what exactly is wrong. That’s okay. We can rewrite the guard into 3 separate guards.

guard('age must be an integer in human range') {
  age.is_a?(Integer) && (0..130).cover?(age)
}

guard('favorite_beer must be string or nil') {
  favorite_beer.nil? || favorite_beer.is_a?(String)
}

guard('favorite_beer is only allowed for age >=21') {
  favorite_beer.nil? || age >= 21
}

Much neater. Let’s try running the same code:

> Person.new(age: 200)
ArgumentError: age must be an integer in human range

> person = Person.new(age: 5)
=> #<Person @age=5, @favorite_beer=nil>

> person.favorite_beer = 'corona'
ArgumentError: favorite_beer is only allowed for age >=21

> person.update(age: 200, favorite_beer: 9)
=> {:base=>["age must be an integer in human range", "favorite_beer must be string or nil"]}

> person.update(age: 30, favorite_beer: 'corona')
=> nil

> person
=> #<Person @age=30, @favorite_beer="corona">

Nice, error messages are now more specific.

Just to recap, with guard and plain Ruby we can accomplish… everything. Okay, thanks, b…

But what about reuse?

Ah. Reuse is already here by default. We can have a module like this.

module ReusableTypes
  def int(name)
    guard("#{name} must be an integer") { send(name).is_a?(Integer) }
  end

  def age(name)
    int(name)
    guard("#{name} must be within 0-130") { (0..130).cover?(send(name)) }
  end

  def nullable_string(name)
    guard("#{name} must be nil or a string") {
      value = send(name)
      value.nil? || value.is_a?(String)
    }
  end
end

class Person
  extend Portrayal
  extend ReusableTypes

  keyword :age
  keyword :favorite_beer, default: nil
  public :age=, :favorite_beer=

  age :age
  nullable_string :favorite_beer

  guard('favorite_beer is only allowed for age >=21') {
    favorite_beer.nil? || age >= 21
  }
end

We put guards in module methods and call them. Nothing really changed, but we suddenly have reusable types.

In Ruby it’s a common tradition to return the name of what’s being declared. Portrayal’s keyword follows this tradition, returning the name of the keyword. If you’d like, you can put our type methods in front of keyword, and it works the same.

class Person
  extend Portrayal
  extend ReusableTypes

  age keyword :age
  nullable_string keyword :favorite_beer, default: nil
  public :age=, :favorite_beer=

  guard('favorite_beer is only allowed for age >=21') {
    favorite_beer.nil? || age >= 21
  }
end

If you don’t like the above style, you could do something else. For example, you could return name from methods in our module, and wrap the keyword names in them. Let’s also capitalize method names while at it:

module ReusableTypes
  def Int(name)
    guard("#{name} must be an integer") { send(name).is_a?(Integer) }
    name
  end

  def Age(name)
    Int(name)
    guard("#{name} must be within 0-130") { (0..130).cover?(send(name)) }
    name
  end

  def NullableString(name)
    guard("#{name} must be nil or a string") {
      value = send(name)
      value.nil? || value.is_a?(String)
    }
    name
  end
end

Which makes this possible:

class Person
  extend Portrayal
  extend ReusableTypes

  keyword Age(:age)
  keyword NullableString(:favorite_beer), default: nil
  public :age=, :favorite_beer=

  guard('favorite_beer is only allowed for age >=21') {
    favorite_beer.nil? || age >= 21
  }
end

When I said earlier that guards can be declared anywhere in the class body, I really meant it. This still works. I’m sure there are more ways you can come up with for using these guards. These are just a couple off the top of my head.

Looking at the above, you can probably already imagine how you’d be able to easily implement a type of any complexity, and Portrayal::Guards will make sure to guard your initializers and writers for you.

Okay, thanks, b…

But what about composition?

Right, we actually might need some extra features to make composition nice. After toying around with some ideas, I decided to include the following additional features into the proof of concept.

Guard chaining

One way to compose guards could be to make sure that our reusable methods return the passed-in name, like we already did above. If every declaration returns the name that it received, then we could chain guards like this:

# Type methods
def Odd(name)
  guard("#{name} must be odd") { value = send(name); value.respond_to?(:odd?) && value.odd? }
  name
end

def Int(name)
  guard("#{name} must be an integer") { send(name).is_a?(Integer) }
  name
end

# Chaining example:
Odd Int keyword :odd_number

This could be especially nice for something like Nullable, where we don’t want to create NullableString, NullableInt, etc for every possible type. So maybe if we had

def Nullable(name)
  guard("#{name} can be nil") { send(name).nil? }
  name
end

Then maybe we could write Nullable Int keyword :number?

Unfortunately, we cannot. It won’t work, because Nullable will fail anything that isn’t a nil, and Int will fail anything that isn’t an integer. They don’t mesh, because we don’t have full &&/|| capabilities across guards. The good news is that perhaps we don’t actually need them.

I’ve thought about a few ways to enable this sort of composition, and came up with what I find to be a simple solution: a pass! guard.

Special `pass!` guard

A pass! is just like a regular guard, you can have as many as you want (but you probably never need more than one), and they always run first. If a pass! returns anything truthy, then we’re done, the object is valid, no further guards are called. With this new capability we can make Nullable like this:

def Nullable(name)
  pass!("#{name} can be nil") { send(name).nil? }
  name
end

And this kind of composition works now:

Nullable Int keyword :number, default: nil
Nullable String keyword :text, default: nil

Yay!

Because a pass! always runs first, the order doesn’t matter. If a pass! sees nil, other guards won’t run. If it sees non-nil, then we proceed into int/string guards.

Unfortunately, there’s still a problem here. All the guards are mixed together, so the Nullable check for number will stop all guards from executing, even the String guard for text. That’s because we add guards into the class, but we aren’t grouping them with each other.

To solve this, I added guard grouping. But don’t worry, it’s basically nothing.

Guard grouping

Remember that :base key in the error hash you saw earlier? Here’s a reminder:

{:base=>["age must be human and beer is only for >=21yo"]}

The :base is actually a default topic for guards. And it’s super simple to group guards into other topics. Just add one more argument to the guard:

guard(:topic_name, 'error message') { boolean expression }

The new first argument :topic_name (it could be anything really) is the topic. So all guards are actually per topic. A fail or pass! in one topic won’t stop guards in another topic. This is just a more generic way to let you make guards “per attribute”. And of course it’s just what the doctor ordered for ReusableTypes module. We can now do this:

module ReusableTypes
  def int(name)
    guard(name, "#{name} must be an integer") { send(name).is_a?(Integer) }
  end

  def age(name)
    int(name)
    guard(name, "#{name} must be between 0 and 130") { (0..130).cover?(send(name)) }
  end

  def string(name)
    guard(name, "#{name} must be string") { send(name).is_a?(String) }
  end

  def nullable(name)
    pass!(name, "#{name} can be nil") { send(name).nil? }
  end
end

By the way, notice how we’re no longer returning name from each method. That’s because each guard already returns its topic, so we don’t have to do that anymore. Another small win.

With these in place we can now declare our Person this way:

class Person
  extend Portrayal
  extend ReusableTypes

  age keyword :age
  nullable string keyword :favorite_beer, default: nil

  guard('favorite_beer is only allowed for age >=21') {
    favorite_beer.nil? || age >= 21
  }  
end

Or this way if you made methods capitalized:

Age keyword :age
Nullable String keyword :favorite_beer, default: nil

Or this way, if you like to keep keyword on the left:

keyword Age(:age)
keyword Nullable(String :favorite_beer), default: nil

Or this way, if you don’t want to intefere with keywords:

Age :age
Nullable String :favorite_beer

keyword :age
keyword :favorite_beer, default: nil

Or go back to plain guard declarations. Whatever you fancy.

Keep in mind, we only learned 2 methods so far: guard and pass! (well, maybe also update if you’re pedantic). The rest is just plain Ruby.

Listing guards

Just for fun, I wanted to be able to list guards declared on a class. It’s possible with Person.portrayal.list_guards, which returns the following:

> Person.portrayal.list_guards
=> {:age=>["age must be an integer", "age must be between 0 and 130"],
 :favorite_beer=>["favorite_beer can be nil", "favorite_beer must be string"],
 :base=>["favorite_beer is only allowed for age >=21"]}

Where is this lib?

At the time of this writing the implementation is just a gist. I’m curious what people think about this before I make it into a proper gem. Let me know your thoughts. Too crazy? Or not crazy enough? :)

Rails — narrative vs model centric approach

Max Chernyak — Tue, 22 Nov 2022 05:00:00 +0000

I’ve explored DHH’s way of writing Rails applications. His introduction of CurrentAttributes and suppressors of callbacks a few years ago made me want to revisit his Youtube screencasts on Basecamp 3 and try to really appreciate this approach with an open mind.

I soon understood our fundamental difference in thinking. DHH approaches application code as an interconnected web of rich models. Each model is chock-full of concerns (modules with generalized functionality). Each model’s methods produce ripple effects far and wide across associated models, via numerous callbacks spanning many modules. This approach is somewhat graph-like. You have a graph of rich nodes, and as you activate one, it activates other nodes at various distances in all directions.

This finally made me realize that with such a dynamic way of viewing application behavior, it makes sense why one might want to suppress entire classes and categories of callbacks. Those ripple effects are nearly untraceable, and require blanket suppressors from the top. Instead of directing logic, we’re constraining logic that would otherwise spread in all kinds of surprising ways.

This also explains why it’s so difficult to avoid globals in this world. You don’t want to impede your vast ripple effects with such minutia as passing the same data across associations over and over. It’s a waste of time.

I want to say that this is a highly unusual approach, but to be frank, any consistent thought-out approach is unusual in our industry. Thoughtful codebases are unusual. So yes, it’s unusual to have a well-established approach consistently applied to the entire codebase.

I know many people disagree with DHH, but I have not heard them provide a good alternative way of viewing the entire application. I’ve seen rebuttals to individual features, but not a challenge to this entire world view. Frankly, DHH himself didn’t really conceptualize his view either.

My biggest problem with this approach is that in his videos it was really hard for me to follow the story that his code is telling. Since all ripple effects are unapologetically triggered via chains of callbacks, it was difficult to follow so many paths into so many directions, ending up with so many outcomes. I find that this doesn’t work well with how I think.

My alternative way of building apps is narrative centric. Instead of creating a web of nodes with ripple effects, I want to create small stories, each living in an entry point (whether that’s a controller action, bg job, rake task, or test). Each story has a beginning (the initiating request/call) and an end (the response), with side effects in between. I love seeing a complete story where every major plot point is clearly visible at the controller level. If groups of plot points are often repeated together, like tropes, I love the ability to group them as needed in order to keep things DRY. I love that I can look at every entry point, and decide how best to optimize it. I can decide to put various actions into a transaction, or group multiple actions into one for a more efficient SQL call. Or introduce an exponential backoff retry for something. I find that with the narrative approach I have the most clarity and flexibility.

To write beautiful short stories, I prefer to focus my attention on gifting myself the domain language necessary to write them. Both Ruby and Rails allow for very expressive routines, as long as your business logic is neatly packed away beneath clean interfaces.

Good Engineering is not Premature Optimization

Max Chernyak — Mon, 10 Oct 2022 04:00:00 +0000

The term “premature optimization” is often misused. It’s supposed to be about trading simplicity for unnecessary performance gains. Instead, it’s used as a blanket dismissal of anything unfamiliar. That’s both inaccurate and harmful. Here’s a list of when it’s not a premature optimization.

It’s not premature optimization on their part when:

They solved a problem elegantly, in a way that you didn’t think of
They solved a problem elegantly by deviating from the beaten path
They designed a clear and fitting code pattern that you didn’t come up with
They used a fitting data structure that you weren’t aware of
They used a fitting algorithm that you weren’t aware of
They achieved extra performance without sacrificing clarity, with an approach that you didn’t think of
They configured a piece of infrastructure for the required use case
They let an appropriate existing system handle the work that it’s good at handling

And they did that within the allotted time.

Respectively, it’s not premature optimization on your part when:

You propose a clean and elegant solution that they didn’t think of
You challenge an existing practice with a simpler/cleaner alternative
You introduce a new code pattern going forward, which improves codebase clarity and consistency
You recommend a minor code change to use a more fitting data structure that they weren’t aware of
You note that there’s a more fitting algorithm that they may not have seen
You explain how a small change can make the code more performant without sacrificing its clarity
You suggest an infrastructure config appropriate for the required use case
You push for letting an appropriate existing system to handle the work that it’s good at

And the time it would take to follow your advice is negligible.

I’m not saying that these are always desirable. I’m only saying that they are not premature optimization. It’s important to label other people’s work correctly (as well as your own). Lump good engineering with premature optimization, and you end up with all the consequences of bad engineering at once.

Ruby Enumerator.new(size)

Max Chernyak — Sun, 07 Aug 2022 04:00:00 +0000

Every ruby enumerator supports count. It’s a method that will iterate over every item and return their total count.

irb> enum = Enumerator.new { |yielder|
  (1..100).each do |i|
    puts "counting item: #{i}"
    yielder << i
  end
}

irb> enum.count
counting item: 1
counting item: 2
…
counting item: 100
=> 100

However, Enumerable also has size. Except, by default it’s just nil.

irb> enum.size
=> nil

A little-known feature in ruby is that you can pass a parameter to Enumerator.new to give it a shortcut “answer” to the size question.

irb> enum = Enumerator.new(100) { |yielder|
  (1..100).each do |i|
    puts "counting item: #{i}"
    yielder << i
  end
}

irb> enum.size
=> 100

No more iterating to get the count. However, there’s an even more little-known feature. You can pass a lambda to determine the size lazily, and still faster than iterating. Let’s say that you’re enumerating over products in some kind of ecommerce API.

irb> api = EcommerceApi.new('connection config')
irb> enum = Enumerator.new { |yielder|
  api.products.each.with_index do |product, index|
    puts "fetching product: #{index}"
    yielder << product
  end
}
irb> enum.count
fetching product 0
fetching product 1
…
fetching product 235
=> 236

Let’s say our API has a more efficent way of obtaining the count: total_count endpoint.

irb> api = EcommerceApi.new('connection config')
irb> enum = Enumerator.new(api.products.total_count) { |yielder|
  api.products.each.with_index do |product, index|
    puts "fetching product: #{index}"
    yielder << product
  end
}
irb> enum.size
=> 236

We no longer have to iterate over products to get the total count, but notice a new problem: we now always run total_count, even if the user of our enum never calls size. Seems like a waste. Moreover, if the products are added to the API, our size will not change. The lambda would allow us to run the API call only when requested, and always get fresh count.

irb> api = EcommerceApi.new('connection config')
irb> enum = Enumerator.new(-> { api.products.total_count }) { |yielder|
  api.products.each.with_index do |product, index|
    puts "fetching product: #{index}"
    yielder << product
  end
}
irb> enum.size # Calls -> { api.products.total_count } lambda.
=> 236

This feature also exists when using enum_for/to_enum to create the enumerator. You have to return it from the block passed into enum_for. The block arguments are any additional arguments passed to enum_for.

irb> def each_number(max = 100)
  return enum_for( __method__ , max) { |max| max } unless block_given?
  (1..max).each { |n| yield n }
end
irb> each_number(200).size
=> 200

P.S. I often forget how this Ruby feature works, and searching never brings up quick examples, so hopefully this article will help when in need of a quick reminder.

Writing Maintainable Code is a Communication Skill

Max Chernyak — Wed, 24 Nov 2021 05:00:00 +0000

Writing maintainable code is easy. Just keep methods and argument lists short, names and comments long, and follow a styleguide. Boom! Done. Unfortunately, as one famous journalist once wrote:

“For every complex problem there is an answer that is clear, simple, and wrong.”

— H. L. Mencken

It’s not style and shape that makes code hard to maintain. It’s the lack of clarity on how the code works, what it represents, and/or why it was written (this way). I’ll refer to these questions as “how?”, “what?”, and “why?” for short. The questions are straightforward, but there’s nothing straightforward about answering them. You may feel that short method bodies help with understanding the “how?” , but sometimes they make the program hard to follow. You may think that long, descriptive names always answer the “what?”, but often they add too much noise. You may feel that “wall-of-text” comments address any “why?” concerns, but now your readers TL;DR them. Every situation is different. It’s up to you, the programmer, to find an eloquent and considerate way to address the how?, what?, and why? in each particular case.

Maintainable code is code that eloquently and considerately communicates to its reader how, what, and why it implements.

How?

“I have only made this letter longer because I have not had the time to make it shorter.”

— Blaise Pascal

“How?” refers to the degree of expressiveness with which a routine or an algorithm is written.

The good news is that it’s hard to fail at answering “how?”. You’d have to write utter gibberish. The bad news is that it’s equally hard to succeed. You must break up complex algorithms into clear steps. You must seek out good metaphors that help people make sense of your abstract code. In other words, you must write code that continuously guides fellow engineers. That level of clear communication is rare, but so are great codebases. How often have you seen an algorithm expressed with such grace that it appears boringly obvious?

Another component in a successful answer of “how?” is the programming language itself. A flexible language allows you to write incredibly expressive codebases. However, your level of writing skill is all that stands between a magnum opus and a major oops. Make a few wrong moves, and your codebase is a total mess. This is why some engineering teams opt to commit to a strict programming language with guardrails. A codebase written in such a language won’t win you any poetry awards, but neither will it leave you with a magical ball of mud. Well, you might still end up with a ball of mud (engineers do have a boundless capacity to shoot themselves in the foot), but at least it won’t be magical.

As you can probably tell, there are practical business trade offs with both types of language. An expressive language better serves a small, experienced team, or a team with strong senior guidance. A strict language can support a larger and a less senior team. In the short term, both teams could accomplish the same amount of work. However, in the long term, a larger team will likely produce more code. That’s more code to support and maintain, which is certainly not ideal.

Failing at “how?”

When code fails at answering “how?”, it is often verbose, convoluted, or is again, simply utter gibberish. Much like the example below:

def r(s1, s2, s3)
  [s3.bytes, [32], s1.bytes, [10]*2, s2.bytes].map { |ba|
    ba.flat_map(&:chr).inject { |v, a| "#{a}#{v}" }.reverse
  }.inject(&:+)
end

In the above, we didn’t take the time to find a more considerate representation of the desired behavior.

Succeeding at “how?”

In the example below, we’ve put in the effort to make the code easy to follow. You can see that strings are being concatenated.

def r(s1, s2, s3)
  s3 + " " + s1 + "\n\n" + s2
end

Now, while the above code clearly answers “how?”, we still don’t know what business function it accomplishes.

What?

“Isolating complexity in a place where it will never be seen is almost as good as eliminating the complexity entirely.”

— John Ousterhout

If you have succeeded at what? it means that a new maintainer understands the goal of every piece of your code. In order to ensure that those goals are clear, you must figure out 1) what to abstract and encapsulate and 2) what to name it.

On one hand, the “what?” can be used to cover up the problems of “how?”. You can write awful code as long as your function is well isolated, well tested, and well named. Once the goal of your function is clear, nobody will ever need to look inside of it. Congrats, you saved some time now, and someone could simply swap out the whole thing later. Sounds like a win-win, but there is a catch. You better make damn sure you get the abstraction right, because the stakes are high. If you get it wrong, then someone will have to dig into your messy function and tease it apart. They will not enjoy that. The moral of the story is, if you have any doubts about your choice of abstraction, then definitely put some extra time towards a clean implementation.

On the other hand, it’s possible to take “what?” too far. For example, you might feel the need to blindly fixate on consistency in naming, or include the greater context in every name, length be damned. Maintainable code is not about communicating consistently or exhaustively. It’s about communicating the right amount of information at the right time (i.e. eloquently). Long names work well in high-level interfaces that mimic business terminology. However, they can be distracting in low level code, where it’s easy to lose sight of data transformations in a forest of names.

Much like a novelist’s prose, it takes years to develop good taste for eloquent and considerate naming. My advice is to get comfortable reading other people’s code. Put yourself in the shoes of your audience.

Failing at “what?”

Short names typically send a signal that they are contextually self-explanatory. Long names signal that we’re breaking away from current context, and we should pay special attention. Moreover, long names are harder to tell apart. Use them sparingly.

In the below example, the names are too long and redundant given their context.

def render_email_with_a_greeting(email_recipient_name_string_for_rendering_email, email_body_string_for_rendering_email, email_greeting_string_for_rendering_email)
  email_greeting_string_for_rendering_email + " " + email_recipient_name_string_for_rendering_email + "\n\n" + email_body_string_for_rendernig_email
end

Succeeding at “what?”

def render_email(recipient_name, body, greeting: 'Hello,')
  greeting + " " + recipient_name + "\n\n" + body
end

Here we understand what’s being done, and begin to form some valid “why?” questions.

Why?

“Give light and people will find the way.”

— Ella Baker

Some schools of thought consider all code comments to be failures of code expression. I tend to agree with this for “how?” and “what?”, but not “why?”. Trying to cram all business context into names of variables and functions is bound to make the code more confusing. The code already has more than enough to deal with answering the “how?” and “what?”. Let’s give it a break by answering “why?” in the comments.

That said, code is the most dependable source of truth, and unfortunately comments are a distant second. They tend to lie. This means that we should not overuse them. To excel at “why?”, it’s important to learn to:

Pinpoint which decisions actually need context.

Usually, decisions that need to be explained derive from one of four circumstances: 1) there is a non-obvious business reason for your decision 2) you did a significant amount of research to arrive at a decision 3) you were on the fence about your chosen solution or 4) you were asked a question in a code review. In each of these situations, it is probably a good idea to leave a clarifying comment.
Identify the level of detail needed for your audience.

The people who will read your code comments are likely to be experienced programmers who are familiar with your company’s internal terminology and processes. Lean on your shared knowledge to communicate efficiently.

Failing at “why?”

# Keyword argument `greeting` has a default value.
def render_email_to_send(recipient_name, body, greeting: 'Hello,')
  # Emails can be plain and html, and while most email
  # clients support html, it's a good practice to add plain
  # text versions as a fallback.
  greeting + " " + recipient_name + "\n\n" + body
end

Here we see multiple pitfalls: addressing an unlikely audience, going into arbitrary levels of detail, adding redundant information, and failing to answer likely questions. The outer comment is redundant. The inner comment neither seems relevant to the code, nor does it consider the audience it’s most likely addressing.

Succeeding at “why?”

def render_email(recipient_name, body, greeting: 'Hello,')
  greeting + " " + recipient_name + "\n\n" + body
end

When looking at the above code we can assume familiarity with the basics and identify a couple of potential questions:

Why do we need to support a custom greeting?
Since we use \n, is this function only used for plain text emails?
(For rubyists out there) why are we concatenating with + instead of "#{interpolation}"?

Here’s one way to address them.

# We allow custom greetings because marketing wants to be able to
# personalize them by time of day, e.g. "Good Afternoon, Person".
def render_plain_text_email(recipient_name, body, greeting: 'Hello,')
  # We avoid interpolation because we want nil values to error out.
  # Helps prevent missing content in sent emails.
  greeting + " " + recipient_name + "\n\n" + body
end

There are now comments explaining why we allow custom greetings and avoid interpolation. We also clarified our use of \n by adding _plain_text_ into the method name.

Alternatively, we could consider eliminating the top comment by renaming greeting to personalized_greeting as follows:

def render_plain_text_email(recipient_name, body, personalized_greeting: 'Hello,')
  # We avoid interpolation because we want nil values to error out.
  # Helps prevent missing content in sent emails.
  personalized_greeting + " " + recipient_name + "\n\n" + body
end

Useful Framing

“If I had an hour to solve a problem and my life depended on the solution, I would spend the first 55 minutes determining the proper question to ask for once I know the proper question, I could solve the problem in less than five minutes.”

— Albert Einstein

When we work with fellow engineers and stakeholders, we engage in three of the most difficult kinds of communication: 1) giving feedback (in code reviews) 2) negotiating (in estimations) and 3) conveying abstract concepts (in code). These conversations can be anxiety inducing and we have them multiple times a day! The “how?”, “what?” and “why?” framework can help us organize our thoughts.

When conducting code reviews, you could be more specific in pointing out a problem:
- “I see what you’re doing but have trouble understanding how it works under the hood.”
- “I see how this works and why we need this, but extracting a method would make it easier to understand what this piece is doing.”
- “I see what is being accomplished, and how it’s done, but I am unclear why we made this particular choice.”
When negotiating refactoring deadlines, you now have language that can help stakeholders understand exactly what you’re trying to achieve:
- “It’s hard to understand how this code works under the hood. We need to do a refactor before we can confidently change it.”
- “This code needs to be broken up so we can more easily follow what it’s doing.”
- “A lot of our reasons why were never written down, so we’d like to try and add some context to the codebase before we forget.”
And finally, it provides a checklist to reflect on your own work before you share it with the team:
- “Will a reader easily understand how my code works?”
- “Do my names clearly convey what my code accomplishes?”
- “Have I given the proper amount of context to convey why I wrote the code this way?”

It would be interesting to adopt a code quality standard along the lines of: “all new code must successfully convey how, what, and why, to at least 2 of your colleagues.” If you were to conduct such an experiment, I would love to know how it goes.

Mindful Code Reviews

Max Chernyak — Mon, 04 Oct 2021 04:00:00 +0000

Code Reviews are First-Class Citizens

Code reviews are an integral part of our daily work as engineers. They help us reduce bugs, share knowledge, collaborate asynchronously, build rapport, feel recognized, and most importantly, keep software maintainable. Diligent code reviews can save the team from insidious architectural mistakes that may hinder all future development. So why do we often treat them as second-rate citizens, a distraction in the way of shipping? Why is it wrong for a good day of work to consist entirely of leaving PR feedback? Why do we try to sneak reviews past stakeholders, and outright skip them in the face of movable deadlines? There are definitely reasons for it, but whatever they are, it might help to take a deeper look at your engineering culture. I believe that in a growing (and especially geographically distributed) company, engineering success is predicated on embracing code reviews as first-class citizens, with full stakeholder buy-in.

Writing Under Pressure 👂📣

To members of the computer generation it’s no surprise that it is easy to accidentally come off dry, dismissive, judgmental, or worse in text. This is exacerbated by constantly putting out fires and missing deadlines while trying to leave feedback. On the receiving end, people most likely take pride in their work, and are attuned to listen carefully. Reading PR comments then becomes sort of like putting your ear directly onto an active megaphone. This is why I believe that rushed code reviews are harmful to engineering culture. All this pressure makes being kind and considerate a more difficult challenge than it has to be. If your company is a burning tornado, fixing that could be a crucial step towards mindful… everything, let alone code reviews.

Practices

Over the years in the industry I’ve compiled a list of my favorite code review practices. Here they are in no particular order.

1. Advocate for the Reviewee

Approach each comment from the position of respect for author’s work and decisions

Even when some of the author’s decisions appear to be “clearly suboptimal”, or straight up mistakes, assume the best intentions on the their part. Spend some time advocating in your mind for the code you’re reading, challenging your own assumptions. Always start by acknowledging the thought and work that the author put into the code, before giving them any further feedback. If you understand where the author is coming from, take time to acknowledge it before providing counterarguments.

2. Objectivity > Subjectivity

Seek out objectivity in all arguments

A comment asking for a change should make an objective, respectful case for it. When making a case, dig past personal preferences all the way down to objective underpinnings of your argument. A tiny nugget of strict objectivity is miles more effective than a 500-word opinion piece. The only acceptable kind of subjectivity is when the code appears confusing. In such a case, ask if we can find a way to make the code easier to follow. A healthy team always strives to address confusing code.

3. Conversation > Silence

Subjectivity is welcome as long as it’s a discussion

Sometimes we can’t help but voice a subjective opinion. In doing so, we must acknowledge that we have been unsuccessful in finding an objective argument, and are asking the author to indulge us for a moment. This is ok, as long as we present these opinions as topics for discussion, and not as something we insist on being implemented. Use the discussion as a tool to figure out the objective underpinnings behind your opinion. The team should support this exploration, and try to learn from it. Of course, be willing to accept that the discussion will not always result in your opinion making its way into the code.

4. Ask, Don’t Tell

Turn statements into non-leading questions

Every time you are making a statement, try to turn it into a non-leading question. Let’s say you think code should be moved to another function.

The original thought: “This code should be moved to function X.”
The question form: “Could you move this code to function X?”
The non-leading question form: “What are your thoughts on moving this code to function X?”

In the end, be genuinely willing to embrace and accept counterarguments, or open up a healthy/constructive discussion.

5. Care About Details

It is not a waste of time to discuss a detail in depth

We have a term in our community: “bikeshedding”. Let’s unlearn this term in code reviews. Details can matter because technical debt tends to be a “death by a thousand cuts”. Besides, a big discussion over a small detail can often be useful for other things, like establishing a rapport with someone. A self-conscious fear of wasting time could cause more damage than actually staying on topic. The fact that 2 intelligent people are genuinely interested in finding a good outcome is enough of a justification for having a discussion, and you shouldn’t be worried as long as it stays constructive.

6. Specific Examples > Generalizations

Try to propose a concrete solution

If possible, use pseudo-code or real code (untested is ok) to illustrate your points. If writing the code is not feasible, take time to thoroughly articulate your thoughts to increase the likelihood of the other person being able to follow your comment. This is especially important when collaborating across time zones. Keep in mind that despite your best efforts they might still not follow it, and this is ok, we are not trying to rush through this. If articulating your thoughts is extra difficult, feel free to collaborate on a problem by writing up and submitting a pull request with your suggestions into the branch that you’re reviewing. Be willing to agree not to merge it, since this is just a part of having a discussion.

7. Working Code > No Code

Always respect working code

If a fellow engineer submits a PR with a working and tested implementation, but you find that it could use a better architectural approach, this is a great problem to have. Now we can focus on refactoring this PR without worrying about implementation details, since they are already working and tested. This actually frees us to collaborate on reshaping the code’s architecture while maintaining the same logic.

8. Advocate for the Reviewer

A code review itself is an original work

When you are on the receiving end of a code review, treat the review itself as a fruit of the labor of its author. Even though they’re reviewing your code, their review is their original work. Instead of only focusing on the changes you’ve been asked to make, try to approach the reviewer with empathy and appreciation for the ideas they’ve put forward. It never hurts to let people see that you appreciate their work.

9. Use Complete Thoughts

Fight the instinct to leave a quick one-liner

One-liners are notoriously bad when it comes to unintended signaling. Here are some of the ways they can be misinterpreted:

my work is not worthy of your proper consideration
you don’t care whether I agree or disagree
you don’t take code reviews seriously
your time (writing) is more valuable than my time (unpacking what you mean)
you’re rubbing my nose in how immediately obvious the issue was to you

For tiny issues, writing a complete sentence might feel like an overkill. However, regardless of topic, a well written comment is never a waste of time, since it’s meant for a human to read. Even if you think you know the recipient well, anyone can have a bad day and take things close to heart. They may not show you any signs, since nobody likes to come off difficult, but the overall perception can wear one down over time. The onus is on reviewer to keep it in mind.

10. Practice

These aren’t rules to be followed perfectly from day one

These practices aren’t meant to be a checklist. As long as you follow these practices in spirit, it’s ok to make your own judgment calls based on specific situations. The more you practice, the easier it gets.

11. Have fun!

Enjoy geeking out on technical discussions with your colleagues

Code reviews are places where we get to unapologetically talk deep programming, so let’s take advantage of it, and have fun!

P.S. Emojis, gifs, memes, and puns are welcome and encouraged.

Special thanks to the awesome Philip Szalwinski for suggestions and contributions.

Don't Build A General Purpose API To Power Your Own Front End

Max Chernyak — Mon, 13 Sep 2021 15:58:36 +0000

TLDR; YAGNI, unless you're working in a big company with federated front-ends or GraphQL.

It's popular in web dev nowadays to build a backend that serves JSON, and a frontend that renders the app. This is fine. I'm not the biggest fan, but it's really okay. Except it's not okay if you think that your backend needs to be designed like a generic public API. This will not save you time.

Why not?

When you design a general purpose API, you have to figure out a bunch of annoying stuff.

How to predict and enable all possible workflows
How to avoid N+1 requests for awkward workflows
How to test functionality, performance, and security of every possible request
How to change the API without breaking the existing workflows
How to prioritize API changes between internal and community requirements
How to document everything so that all parties can get stuff done

And on the front-end side, there's a bunch more:

How to collect all the data needed to render a page
How to optimize requests to multiple endpoints
How to avoid using API data fields in unintended ways
How to weigh the benefit of new features against the cost of new API requests

Do these really have to be your problems if you're just making a backend for your frontend? Do you have to imagine every possible workflow, avoid N+1 request issues, test every request configuration, or deny yourself features when you know exactly what each page needs to look like? You can probably see where I'm going with this.

So what do you suggest?

I suggest you stop treating your frontend as some generic API client, and start treating it as a half of your app.

Imagine if you could just send it the whole "page" worth of JSON. Make an endpoint for /page/a and render the whole JSON for /page/a there. Do this for every page. Don't force your front-end developers to send a bunch of individual requests to render a complex page. Stop annoying them with contrived limitations. Align yourselves. 🧘‍♂️

And in that JSON, actually render the page. Don't render abstract models and collections. Render concrete boxes, sections, paragraphs, lists. Render the visual page structure.

{
  "section1": {
    "topBoxTitle": "Foo",
    "leftBoxTitle": "Bar",
    "linkToClose": "https://…"
  },
  "section2": {
    …
  }
}

This is similar but not quite the same as Server Driven UI¹. Perhaps we could call it Server Informed UI.

How is that better exactly?

Have you seen that list of annoying decisions up there? For one, they are gone now. 💨

For two, you are now free to decide "I want page a" and then implement “page a” in the backend, and in the frontend. Super straightforward. ✅

No more "what API workflows do we need to introduce to sort of make this page possible almost? 🤔". You can keep "page a" dumb to only do what it needs to do. You test the crap out of "page a" for bugs, security, performance. You can even fetch everything for "page a" in a single big SQL query. You can cache the entire JSON payload of "page a".

Frontend knows exactly what each field in "page a" payload is for. There are no discrepancies in field meanings. They represent exactly what frontend needs.

When a stakeholder tells you to change "page a" you will be able to literally go ahead and change "page a", instead of spending meetings figuring out how your backend API could accommodate the change in "page a". It's not a choreographed conglomeration of API requests. It's just "page a". You have freed yourself from self-imposed limitations of your API.

Your business logic has now moved from being haphazardly split between frontend and backend into just backend. Your frontend can finally focus on presentation and UI. Your backend can finally focus on implementing exactly what's needed. Kinda the goal, no?

Have you actually tried this?

Yes, I have tried this on a couple of production projects so far. One of them was personal, the other was a consistent multi-year refactoring effort in an existing company. The whole team was bought in, and it worked out well. The only problem we've encountered was that the front-end team has gotten increasingly bored. Nearly all business logic was taken away from them. At the same time, no "excitement" was added to the back-end team. It's just gotten kinda boring all around. Somehow we all ended up talking more about the business than the code.

Feel free to stop reading here if you're convinced. Next part is just responding to various rebuttals I keep hearing.

But I want my front-end team to have freedom! (Or, I want my front-end to be decoupled!)

Let's be honest, your frontend doesn't really have freedom. When they send you 7 requests to render a single page, that's not freedom. It's jumping hoops to meet basic requirements. As soon as requirements change, you probably going to need to change the backend anyway to accommodate it. The freedom is all accidental and mostly in the wrong places.

If you really want to give your front end team freedom, install them a GraphQL wrapper directly on top of Postgres and quit. 😛

But we actually want a general purpose API anyway, so this is 2 birds with 1 stone, no?

No, you would not actually want to make this API public. You think you would, but when time comes, you'd be like "crap, maybe I shouldn't". These 2 APIs have very different reasons to change. Public API needs to enable the workflows of your clients. Private backend needs to enable the next whim of your product manager. Stop jamming sticks into your own bicycle wheels.

But how will I reuse the logic when building JSON for pages? I reused so much logic in my CRUD controllers!

If your programming language lets you reuse logic (it does), then you can reuse logic. Use mixins, composition, inheritance, whatever you got to work with. If you make yourself some good abstractions, then you will have an amazing time putting together pages from your LEGO blocks.

But we can reuse this API for the mobile app too!

Your mobile app has a different set of pages with different info, structures, and reasons to change. You'll save more time and sanity making another backend specifically for it. But hey, you can reuse a lot of your logic (see the previous paragraph).

But what if a page needs a partial XHR update? Am I supposed to always return an entire page?

No, it's okay to make an endpoint that returns just something specific. You have my permission. Make endpoints for snippets of data for specific page sections or whatever. It's okay. Render your React components from initial payload, then update them from XHR calls to these endpoints. But only introduce these endpoints when you need them on certain pages. These are exceptions, not the default.

But my frontend is a SPA, so it almost always needs data snippets, not entire pages

Those data snippets could still be provided as partial page structures, not generic resources. As long as your backend only serves the exact needs of your frontend, you're good. 😇

But I'm building a site builder, so my frontend is dogfooding the site builder API

🗡 I dub thee a legitimate use case haver, congratulations!

Do you have data to support your claims?

I wish. It's pretty hard to measure these kinds of things in our industry. Who's gonna maintain 2 architectures for the same software for 3 years, and compare productivity between them? All I got is a mixed bag of personal experiences. Feels inductively justifiable. 🤷‍♂️

There has already been some experimentation with this approach. A Server Driven UI is when the API tells the client which components to display and with which content. That said, most SDUI implementations take this idea all the way. They treat API payloads as a kind of declarative UI language. The front-end then acts as an interpreter, and dynamically renders the declared components. I don't think this level of generalization is necessary for most apps, but it's a fun approach to explore. ↩

3 Reasons Not To Implicitly Memoize

Max Chernyak — Sat, 21 Mar 2020 04:00:00 +0000

The other day I was listening to this Bikeshed podcast episode, where the hosts were discussing when is it a good idea to memoize values using ||= ruby idiom. Since this is a common question even among seasoned developers, I decided to write up my take on it. The short answer is: never.

Problem

Let's take a look at this example. We query the database to find the user by id, then use their email to make an API call to download a profile and grab the name. While this example is indeed contrived, it's fairly common to see variations on this theme in the wild.

def name
  @name ||= @api.fetch_profile(User.find(@id).email).name
end

Now, just to get it out of the way, there are various problems with this code. However, in this post, let's just view it from the angle of memoization. So, what are the 3 reasons not to memoize like this?

Reason 1: Caller is misled about the real impact of making this call.

Typically, doing this sort of memoization goes hand-in-hand with naming your method with a noun. Since the method is named so inconspicuously (name), we're signalling that a caller doesn't have to worry what happens under the hood. We perpetuate the practice of calling this method mindlessly, with no regard for the fragile sequence of interdependent network operations that it takes to fulfill the request. I get it, we want to encapsulate the plumbing, but couldn't we do it without misleading the caller?

Reason 2: Caller has no say in cache invalidation.

This memoization style assumes that caller will never want another fresh value. For web apps, it probably comes out of another assumption that we're always living within a web request, and we never want to fetch any data twice. Unfortunately, each such memoization slowly eats away at our understanding of how data flows through our application, making it much harder to debug problems, or implement anything else on top of the same codebase.

Reason 3: Caller has no way of stopping redundant work.

In our example, if a caller already has a user available, the method will fetch it again anyway. In a well architected system we should be able to inject that dependency, especially if it took something as error-prone as network or database roundtrips to obtain it.

Solution

How would we avoid all 3 of the above problems? It's not that difficult, but with a caveat that you didn't already overcommit to bigger architectural mistakes. Still, it's never too late to stop making things worse. So without further ado, here's the code free of all of the above problems.

def retrieve_name email: User.find(@id).email, api: @api
  api.fetch_profile(email).name
end

You might've just done a double-take: wait, how is this the solution? We just removed caching and added some useless arguments. Bear with me, let's talk through this real quick.

Note that arguments are optional, so the method can still be called without passing anything. Let's go back and see if we've addressed the problems with the original code.

1. Is caller still misled about the real impact of calling this?

No. The fact that this method name is now a verb retrieve_name makes it clear that when you call it, it will do things. That's all it takes to send the correct signal.

2. Can the caller control cache invalidation?

Yes.

name = retrieve_name

# Name is now cached, feel free to reuse it.
do_something_with(name)
do_something_else_with(name)

# Get a fresh name whenever you want.
fresh_name = retrieve_name

3. Can the caller stop redundant work from happening?

Totally.

my_user = User.find(123)
name = retrieve_name(email: my_user.email) # Saves a database call.

In case it's not obvious, we couldn't accept arguments the same way in the original version, because we're only caching one value, and even if we then passed a different user, we would still get back the first cached value.

Ultimately, with very little effort, we just gained 3 significant advantages in maintainability, reusability, and performance of our code.

FAQ

What if I need to call this method from different places, so I don't have a variable to reuse?

I feel your pain. Unfortunately, if you must depend on this caching technique because you cannot assign a variable once, and pass it around, I have some bad news for you. Your abstractions need rethinking. There should be a top level routine in your code that tells the story of a particular transaction. Values that are reused need to be floated up into that context and passed into whatever needs them. In a vanilla Rails world the place like this would be your controller actions. If doing this makes your actions too long, you're missing intermediary objects that give you a clean abstraction to write your routine. That said, this is a pretty big topic best left for future blog posts.

Don't use docker to run your app in development

Max Chernyak — Sat, 04 Aug 2018 00:00:00 +0000

Using docker in development can be very convenient, but running your actual app (you know, the one you’re coding) in docker introduces various headaches.

Mounted volumes are slow and error-prone
You need hacks and shortcuts to run any console/debug commands in containers
Live updates on code changes are unreliable in docker
Runtime is slower in docker
Dependency updates are slower in docker
Networking is more complicated with docker

What genuinely surprises me, is that often teams don’t consider the obvious: just run your app directly on your machine. I find it to be the sweet spot of dev setup. Let’s see what it would look like.

1. Use docker-compose for databases and external services

Create a docker-compose.yml file in your app’s root and only declare your databases in it. For example, this file gives you

Postgres on localhost:5432
Redis on localhost:6379
Fake S3 on localhost:9000

version: '3'

services:
  postgres:
    image: postgres:10.3-alpine
    ports:
      - "5432:5432"
    volumes:
      - postgres-data:/var/lib/postgresql/data
  redis:
    image: redis:3.2.11-alpine
    ports:
      - "6379:6379"
    volumes:
      - redis-data:/data
  minio:
    image: minio/minio
    volumes:
      - minio-data:/data
    ports:
      - "9000:9000"
    entrypoint: sh
    command: -c "mkdir -p /data/dev /data/test && /usr/bin/minio server /data"
    environment:
      MINIO_ACCESS_KEY: access_key
      MINIO_SECRET_KEY: secret_key

volumes:
  postgres-data:
  redis-data:
  minio-data:

2. Use asdf for language runtimes

Create a .tool-versions file in app’s root. Here’s an example for elixir and node setup.

elixir 1.6.4-otp-20
erlang 20.2.4
nodejs 10.8.0

Asdf is like rvm, nvm, and other version managers combined. It has an extensive list of things it can manage.

3. Setup everything

Now you can bootstrap the application by running

asdf install
docker-compose up

and in another terminal you run the app itself:

mix phx.server

That’s it. Now you have the benefit of quick and simple dev setup without giving up all the convenience of interacting with your app directly, without containers in the middle.

Bonus: How to make local Rails work with dockerized Postgres?

The cool part is that your database.yml can be committed to the repo, it will always look the same:

default: &default
  adapter: postgresql
  username: postgres
  host: localhost

development:
  <<: *default
  database: myapp_dev

test:
  <<: *default
  database: myapp_test

production:
  <<: *default
  database: myapp

However, there’s a minor issue when using this setup in Rails. You might get an error when trying to install the pg gem or run a rake db:structure:dump command. Both of these actions rely on postgres being installed locally. To work around it simply add postgres to your .tool-versions — asdf supports it. You will not be actually running this postgres, only using its cli as a client, and satisfying pg’s dependencies.

Linux permissions cheatsheet

Max Chernyak — Sun, 15 Jun 2014 00:00:00 +0000

chmod [a]bcd

bit	scope	description
a		sticky:1, setgid:2, setuid:4 (optional, default: 0)
b	owner	x:1/w:2/r:4 - xw:3/xr:5/wr:6/xwr:7
c	group	x:1/w:2/r:4 - xw:3/xr:5/wr:6/xwr:7
d	everyone	x:1/w:2/r:4 - xw:3/xr:5/wr:6/xwr:7

Note: only file/dir owner can chmod it
Note: scripts need both x and r permissions to execute(that’s because scripts are read into interpreter) (only r is enough if ran via ruby script.rb, sh script.sh)

files

bit setting	meaning
sticky on files	no effect
setgid on execable binaries	no matter who executes, process runs as file’s group
setuid on execable binaries	no matter who executes, process runs as file’s owner
setuid/setgid on scripts	ignored due to security issues
setuid/setgid on non-execables	no effect¹

Warning: setuid is dangerous

directories

bit setting	meaning
x on dirs	`cd`, `stat` (e.g. `ls -l`), inode lookup (access files)
w on dirs	add/delete/rename files (requires `x` for inode lookup)
r on dirs	`ls`

Note: having xw on a dir is enough to delete any file in it(unless it has sticky bit)

sticky on dirs

only used when writable by group/everyone
files in dir can only be edited/deleted by their owner (think /tmp)
symlinks only work if target is within this dir

setgid on dirs

all files/subdirs created by anyone in this dir inherit its group
all subdirs inherit this bit when created

setuid on dirs

no effect

sources

There is an exception. See “SUID and SGID on non-executable files” on this page.

CMS Trap

Max Chernyak — Tue, 26 Nov 2013 00:00:00 +0000

Parable

Many years ago there was a Rails app. It started with things. These things were actually blueprints for other things. The other things needed many associated parts, and parts of parts. How many? The blueprints knew. The blueprints absolutely had to have an admin interface, but changing the blueprints would cause a chain reaction on things and parts. Every modification to the things and their blueprints permeated throughout the coupled network of various models. The admin UI complexity quickly skyrocketed as parts continued to branch out into more entities. It got to the point where blueprints had to have serializable, persistable snippets of logic. At that point every feature has become subject to a very difficult implementation, and thus the app degraded into the state of utter unmaintainability. It felt as if there was a _content management system_standing in the way of getting things done, imposing itself as the middle man between the feature and its implementation. It was like the system actually forced all the business logic to be reframed in terms of this higher level of abstraction.

The worst part? This was a minimal viable product for a newly-born startup.

Accidentally CMS

Image¹ by nerovivo

The programmer’s nature encourages us to indulge ourselves in solving puzzles and modelling abstract concepts. It’s the passion that makes us lose sight of the danger looming ahead, the trap we’re edging towards thanks to our subjective assumptions and vague speculation, the trap of building a overdesigned and overcomplicated system for its own sake. A CMS trap. We suffer various consequences, ranging from burnouts, and loss of enthusiasm, to missed deadlines, and failed businesses, yet we never seem to speak of this mistake directly. Somewhere by a water cooler, an experienced colleague casually points out that you might be overcomplicating things. Somewhere in an IRC chat you get ridiculed for asking questions about a complex object model for a project that will most likely never see the light of day. Yet nobody can clearly explain exactly what is the underlying thought process. These casual remarks is all the education we get on the subject, and people end up learning this the hard way. That’s why I’d like to shine some light on this phenomenon. To start, here is my best shot at defining the CMS trap the way I see it.

A CMS Trap is a state of a web-application in which the development of content management systems is obstructing the development of the content.

If you are building a startup like me, you should know that this trap is especially dangerous in the early stage. Only a small percentage of companies get to play the long game, and by that time their problems have shifted onto a entirely different plane of existence. While these companies may also be subject to falling into the CMS trap, they would probably be able to afford it, if not even pursue this direction intentionally. Here, I’d like to focus on the much more abundant variety: the small companies. The problem would become apparent as soon as you’ve opened your doors to an influx of customers, who’d begin using your project, and providing you with real analytics and feedback. At this point your project would no longer be driven by your gut feeling, rather you’d have real data suggesting how to proceed, dictating which features to implement next. This would be the time when all of your initial architectural assumptions are being tested, and reality is beginning to set in. Reality has no tact, it doesn’t spare you any painful truths when dawning upon your hopeful application design. You’d wish you could refactor, but it’d be too late, as you’d be forced to keep up with new features instead, and implementing them would only be getting harder in this downwards spiral of dwindling productivity.

I’ll thank me later

“Most of our assumptions have outlived their uselessness.”

Marshall McLuhan

Simply put, we love designing systems. As soon as we form some understanding of a problem, we rush to our /(?:whiteboards|moleskines|mindmaps|editors)/ and start passionately defining entities and their interactions. It feels good, it’s what we do best. We tackle some of the most fundamental decisions about the project. Then, having carefully outlined our assumptions, we commit to them. We like to think that we sow wisdom and flexibility with our early decisions, and_we will thank ourselves later_. With all of those useful points of extension and well-represented entities, what could possibly go wrong? The reality is, that most likely these early assumptions will restrict our future, not expand it. The day comes when we meet our old friend, the innocent “past self” staring back at us from the editor, smiling proudly. This well-meaning person spent hours, days, and weeks diverting our efforts into the abyss of speculative architecture, while having barely any idea about the real problems we’ll be facing. We are now stuck with all of that “helpful” code. It’s as if you decided to cook some salad, but instead of having separate ingredients laid out in front of you, all you have is another fully cooked salad given to you by a stranger, which you are now forced to dig through in hopes of fetching some of the pieces you need.

In programming, your past self is nothing but a stranger with boundary issues.

In the same spirit, imagine you come back to your computer only to find your app reorganized by some clueless stranger in ways that have little to do with reality. This isn’t very different to how we find ourselves looking at a system we’ve over-modelled in the past. Wouldn’t you wish that you didn’t have to deal with any of this garbage, and could instead simply greenfield your way ahead as dictated by your business needs?

To bring this back to my personal story, I eventually realized that with every new business feature I spent more time figuring out how to fit it into the existing framework I imposed on myself than actually designing the feature. As you may have guessed, I was thanking myself profusely for being so considerate.

C < RUD

“It’s harder to read code than to write it.”

Joel Spolsky

Talking about architecture is a lot like talking about code itself. It’s not exactly that we can never be mindful of the future, it’s just that the odds are not in our favor. Code is easy to write and hard to change or remove. Every line we light-heartedly throw into the mix will eventually be taunting us with the timeless question: “guess what’s going to break if you touch me? ”. Architectural decisions, just like code, are easy to make and very hard to unmake. While in code this problem is alleviated with testing, in architecture we don’t have testing. The only measure of quality we have is really the measure of pain we feel when working on a new feature, and by that time it’s often too late. Bad architecture can suffocate your business even while your code is sporting 100% test coverage.

Alarm triggers

Image¹ by Fey Ilyas

As with most traps, there is no specific way of knowing when you are walking into one. The best you can hope for is to have some sort of “tells” that warn you of an upcoming danger. Below I list some of these tells from my own experience. Seeing these things in your early stage project should at least make you suspicious.

The early onset conservatism

“A state without the means of some change is without the means of its conservation.”

Edmund Burke

Say you are faced with a new feature, and you find it to be a real yak shave. You realize that it will take a huge refactor, and you are arguing for ways to just avoid it. There is a fine line between negotiating feature requirements for reasons of efficiency, and negotiating them because you are stuck in the accidental buildup of legacy architecture. Could it be that your early speculative design decisions are starting to get in the way of today’s real business needs? Have you perhaps built too much too soon, and is it only a matter of time before the trap snaps, leaving your project effectively paralyzed? Don’t get me wrong, more often than not being conservative is a healthy defense against unnecessary complexity, it’s a standard practice of an experienced developer. The problem is when there is too much defense too early in the project’s life. It should definitely cause some suspicion.

The Drupal syndrome

“Hm, our pricing rules are different for various products, so we’ll need to find a way for an admin to define these rules in the admin panel. Maybe we should store code in the database and eval it?”

This is a classic sign of walking into the CMS trap. You are trying to come up with ways to let admin program some logic, which should then be saved to the database. If it wasn’t for the admin interface, it could’ve been done with only a few lines of code. However, now we are talking about creating price models associated with rules, and all the complexity emerging from this. Any further extensions to pricing capabilities, which could’ve been implemented with a line or two of code, would now have to take the shape of database migrations, forms, validations, and everything else down this rabbit hole. Do you really need admin UI for pricing rules at this point?

The seed is weak

How do you implement 10 categories to place your products into? Typical answer involves creating a Category model and then writing a script that will seed the 10 prescribed categories, which will be assigned to products. Then you’d make sure every developer runs this seed file. Also, don’t forget about running it in production of course. On every deploy. And on every pull. And when setting up a new machine. And when running tests. And naturally, if you change something in the seed file.

If your early-stage application relies on a lot of seed data, you are on a slippery slope. Things that can be assumed constant mustn’t need to be modelled as database-backed entities at this point yet, but I’ll get back to this later.

Every road leads to Mordor

“One does not simply implement business logic”

Boromir

This is somewhat similar to the early onset conservatism, yet there is a difference. Ever found yourself intimidated by a trivial task? Ask yourself this: would this task be intimidating if it was to be implemented in isolation, without the rest of the app surrounding it? If the answer is yes, look at your feet, because you might be caught in the trap. Implementing a feature in a well architected system shouldn’t be any more difficult than implementing it in isolation.

The phantom pain

Sometimes a CMS trap can be recognized by the presence of phantom pains that stem from hidden implications of an emerging CMS. For example, in reality you would never need to delete your categories, but because you built them as admin-editable database-backed records, suddenly you are thinking about the non-existent scenario of having them deleted. Your architecture took the liberty of making you contemplate a scenario that isn’t real. You end up dealing with fake pains, the phantom pains.

Prevention

Image¹ by Henry Burrows

All of the above symptoms have something in common. They are all a product of early assumptions that lead to a complex system. At this point it’s useful to answer two questions: “what’s a complex system?” and “how do you program without making assumptions?”.

Well, for the purposes of this essay let’s say that a complex system is a system of networked nodes which consists of more nodes and connections than you can generally track in your head. Obviously, to get a low-complexity system you need to reduce the number of nodes and connections. As for the latter question, that’s what brings me to the main point. In order to program without making early assumptions, you must avoid doing things at runtime.

Let me elaborate. Having been scarred by over-modelling, I found that there is a principle that should become fundamental in all decision making. Let’s call it:keep it static, stupid, which seems appropriate because it’s really nothing more than a slightly more architecturally-aware riff on keeping things simple.

Making things static is the architectural equivalent of avoiding premature optimization.

The beauty of this principle is that it’s applicable on every abstraction level, regardless of whether you are talking about views, database, or code. The idea itself is simple: if in doubt, do it statically. It’s easier to understand what this means by looking at some concrete examples on various levels of a typical Rails app.

Can it be solved with a class?

Earlier in the post I mentioned pricing rules. This is a common problem where each product might abide by a different pricing algorithm. Price could depend on quantity, current user (think loyalty programs), order history, coupons, and various other things. To avoid the CMS trap I urge you not to allow constructing these kinds algorithms at runtime at an early stage. Write a pricing scheme class. Use strategy pattern. Make the pricing algorithm swappable at the code level. Define pricing rules via your programming language, this way the complexities of this logic can be mapped directly to code, and not warrant a whole layer of abstraction.

Programming languages already come with many wonderful tools, such as conditions and loops. Why reimplement them at a higher level of abstraction? These tools are more than enough to allow you to build complex pricing logic by writing code, directly. Once you have multiple pricing algorithms written as pluggable objects, feel free to let admin choose one, perhaps even “fill in the blanks” by plugging in factors and key values into your algorithm, but evolve this functionality gradually, as needed. Build out your admin UI with time, injecting more and more runtime flexibility into your strategy objects. Remember, you can always make static/hardcoded things dynamic, but not so much the other way. Everything that you make adjustable at runtime introduces complexity into every decision you make from that point on, and increases chances of bugs you cannot foresee, even in seemingly unrelated parts of your app.

Can it be solved with a static page?

Say you are listing things on a page for customer to see. These things may very well be products, photos, or files, whatever else it is that you are doing. Now, you have probably decided that there would be a title, a description, a picture, perhaps author or brand on each of those elements. You’ve split up your entities into these data fields and you decided to build database-backed models. This is where I’d suggest to stop and consider whether you have any good reason for why it can’t be a static page. A static templated view means that in order to change things, you have to edit the view and deploy, yes, but it also means that you don’t have controllers, models, migrations, forms, admin UI, or anything else. In fact, you might kind of still have an admin UI if you’re using Github. It’s not as real time as it could’ve been, but decent nonetheless. People can edit views on Github directly without much issue.

This becomes more of an issue if the things you list are categorized and otherwise laid out based on certain rules. In the dynamic approach, this would immediately force you to create a network of associated models just to render this sort of a page. Consider how little you know at this point about your future needs, and how constrained you will be having speculated your way towards that future. Consider also how quick and easy it would be to just sit down and hardcode this page. Just like the case with strategy pattern, you can always inject dynamic content into this page going forward, when the real needs arise. If you build out a dynamic system right away, you will likely end up constrained by it. Err on the side of static.

Can it be solved with a constant?

Getting back to the seed data issue, this example is fairly simple. You are creating categories. These categories are predefined. Instead of adding models, tables, and seed data, why not simply make a constant with an array? Code allows you to carry static data without involvement of a database. Use that, and wait until you really need the editing of categories at runtime. When that time comes, you could always extract data from the constant into the seed file, without any issues. Moreover, even that isn’t necessary. If you have some categories that never change, and some that should be manipulated in admin panel, you shouldn’t even seed the former ones. You could leave them in the constant, always read them from there, and this way avoid seed data altogether. It’s actually a little secret of mine. I don’t like seed data. It’s been years now, and our app works right out of the box on any new developer’s machine. If you pull our code into your dev machine, the app will just run. This is why I say: when in doubt, hardcode.

Can it be solved with a string in the database?

Say, at this point the app is working just fine, and you have your necessary database-backed models. You need to display a free-form text that may differ from one entity to another (e.g. different per product), yet it might contain certain values interpolated from elsewhere. As per the principle, you should not think about modelling this text via classes. First, ask whether you could simply get by with letting admin type the text as a string. But wait, you’d say. If this text has values plugged in from elsewhere, why should admin be typing them by hand? Would she have to look them up every time? Seems wrong. Well, relax a bit and consider canned snippets. That’s right, perhaps you can simply setup a free form text field while providing some pre-written text for admin, which has appropriate values already plugged in. When you think you need structured data for storing something highly flexible, consider instead using a plain string with canned snippets.

Wrapping up

“For every complex problem there is an answer that is clear, simple, and wrong.”

H. L. Mencken

While the above text is a good general principle, it cannot exactly apply to problems that are clearly asking for CMS-like solutions. When you are tasked with building a highly-flexible CMS, that’s what you do, naturally. When you are asked to build an app with something like Drupal, you are in a whole different realm, where the CMS trap is pretty much your perpetual state of being. However, even in those special cases questions will arise whether to make something more or less dynamic, and I encourage every developer to always lean towards static. You will be doing a service not only to your future self, but also to the next developer, who would much rather slice up a piece of static html and inject some dynamic content than attempt to understand a steaming pile of speculative architecture with many moving parts.

It’s also important to note that I’m not advocating entirely against architecting up front. It’s good to a healthy extent, yet there is a line we draw in the sand on a case by case basis. I encourage you to think carefully about where to draw that line every time you implement something.

Speaking of my story, it ended with a year-long stagnation and a very reluctant revamp of the entire app. In the end, the aforementioned “blueprints” have been downgraded to hardcoded classes, and over time they have become very declarative, thanks to a naturally-evolving internal DSL. Seeing these files today and imagining how I’d proceed implementing runtime admin UI for all the moving parts is nightmarish. Even though it ate away a year, I’m still glad that we bit the bullet and refactored. It was painful, but now this mistake is far behind us.

Unless you know exactly what you’re doing (which is unlikely), stay static. Try to put extra effort into determining which parts of your business can be left hardcoded. If in doubt, hardcode. While doing that, make sure you follow best practices: never put the same conditions in two places, never repeat constant data, use composition, dependency injection, inheritance, whatever you need to make sure you abide single responsibility principle, and maintain singular authority.

Most importantly, don’t get yourself tangled in too much speculation, let the story unfold naturally.

Used under CC BY-SA 2.0 / cropped, shadow added. ↩¹ ↩² ↩³