DEV Community: Riccardo Odone

How would I do it in Haskell?

Riccardo Odone — Thu, 26 Nov 2020 08:47:41 +0000

You can keep reading here or jump to my blog to get the full experience, including the wonderful pink, blue and white palette.

Since I started playing with functional programming, not only I've dot-chained the hell out of object-oriented code, but I often catch myself thinking: how would I do this in Haskell?

Here's a couple of examples that come to mind where I contaminated Ruby with functional intuitions.

Nested Loops and Filtering

During the Global Day Of Code Retreat, together with Joanna, we were writing a method to find the eight neighbors of a cell. In other words, given [ 1, 1 ] as an input we wanted to generate the following output:

[ [ 0, 0 ], [ 0, 1 ], [ 0, 2 ]
  [ 1, 0 ],         , [ 1, 2 ]
  [ 2, 0 ], [ 2, 1 ], [ 2, 2 ]
]

We started with a simple implementation:

def neighbors(cell)
  [ -1, 0, 1 ].map do |y|
    [ -1, 0, 1 ].map do |x|
       [ cell.first + y, cell.last + x ]
    end
  end
end

# [1, 1]
# [[[0, 0], [0, 1], [0, 2]], [[1, 0], [1, 1], [1, 2]], [[2, 0], [2, 1], [2, 2]]]

To remove the double nesting in the output, we reached for flatten:

def neighbors(cell)
  [ -1, 0, 1 ].map do |y|
    [ -1, 0, 1 ].map do |x|
       [ cell.first + y, cell.last + x ]
    end
  end.flatten # <=
end

# [1, 1]
# [0, 0, 0, 1, 0, 2, 1, 0, 1, 1, 1, 2, 2, 0, 2, 1, 2, 2]

Oops, that was one flatten too much. We tried again with flat_map:

def neighbors(cell)
  [ -1, 0, 1 ].flat_map do |y| # <=
    [ -1, 0, 1 ].map do |x|
       [ cell.first + y, cell.last + x ]
    end
  end
end

# [1, 1]
# [[0, 0], [0, 1], [0, 2], [1, 0], [1, 1], [1, 2], [2, 0], [2, 1], [2, 2]]

Yay! The last piece was to remove the current cell from the output:

def neighbors(cell)
  [ -1, 0, 1 ].flat_map do |y|
    [ -1, 0, 1 ].map do |x|
       [ cell.first + y, cell.last + x ]
    end
  end.reject { |neighbor| neighbor == cell } # <=
end

# [1, 1]
# [[0, 0], [0, 1], [0, 2], [1, 0], [1, 2], [2, 0], [2, 1], [2, 2]]

There was something inelegant about that code, though. To hell with the exercise, we wasted the rest of the session refactoring the method.

The nested loop looked ugly. Not to count we were generating an invalid neighbor, only to filter it out later. Here's what we came up with:

def neighbors(cell)
  [ -1, 0, 1 ]
    .repeated_permutation(2)
    .reject { |permutation| permutation == [ 0, 0 ] }
    .map { |y, x| [ cell.first + y, cell.last + x ] }
end

# [1, 1]
# [[0, 0], [0, 1], [0, 2], [1, 0], [1, 2], [2, 0], [2, 1], [2, 2]]

A Semigroup in the Wild

A few days ago, I stumbled upon the following code in a production codebase:

def group_by_product(items_grouped_by_template)
  items_grouped_by_template.select { |item| item.fetch(:variation_type) == 'quantity' }
                           .group_by { |item| item[:product_id] }
                           .each_with_object([]) do |(_product_id, order_items), array|
    array << item_grouped_by_product(order_items)
  end
end

def items_grouped_by_template
  @order_items_grouped_by_template.each_with_object([]) do |(template, order_items), array|
    array << item_grouped_by_template(template, order_items)
  end
end

def item_grouped_by_template(template, order_items)
   {
     id: template.id,
     variation_type: template.product_template.template_type,
     variation_name: template.product_template.name,
     product_id: template.product.id,
     product_name: template.product.name,
     total_quantity: product_template_total_quantity(template, order_items),
     reporting_category: reporting_category(template),
   }
end

def item_grouped_by_product(order_items)
   {
     id: order_items.first.fetch(:id),
     variation_type: order_items.first.fetch(:variation_type),
     product_id: order_items.first.fetch(:product_id),
     product_name: order_items.first.fetch(:product_name),
     total_quantity: product_total_quantity(order_items),
     reporting_category: order_items.first.fetch(:reporting_category),
   }
end

If you don't understand it, don't worry, I'm there with you. I speculate the author didn't understand it, either.

However, after massaging the code for a while, I noticed one thing: there's a hidden data structure that gets combined with itself: I found a semigroup!

I created a Summary that can be concated with another one:

Summary = Struct.new(
  :id,
  :variation_type,
  :variation_name,
  :product_id,
  :product_name,
  :total_quantity,
  :category,
  keyword_init: true
) do
  def concat(other)
    Summary.new(
      id: id,
      variation_type: variation_type,
      variation_name: "#{variation_name} + #{other.variation_name}",
      product_id: product_id,
      product_name: product_name,
      total_quantity: [self, other].map(&:total_quantity).reduce(:+),
      category: category
    )
  end
end

With that in place, I could refactor to something similar to what follows:

product_product_templates
  .map { |product_product_template| to_summary(product_product_template) }
  .reduce { |acc, summary| acc.concat(summary) }

def to_summary(product_product_template)
  Summary.new(
    id: product_product_template.id,
    variation_type: product_product_template.product_template.template_type,
    variation_name: product_product_template.product_template.name,
    product_id: product_product_template.product.id,
    product_name: product_product_template.product.name,
    total_quantity: product_template_total_quantity(product_product_template),
    category: category(product_product_template)
  )
end

Still ugly, but at least I can reason about it.

Get the latest content via email from me personally. Reply with your thoughts. Let's learn from each other. Subscribe to my PinkLetter!

Make Your Tests Fail Randomly (and Profit)

Riccardo Odone — Thu, 19 Nov 2020 09:23:51 +0000

You can keep reading here or jump to my blog to get the full experience, including the wonderful pink, blue and white palette.

Recently, I started experimenting with random values in automated tests. It improves the readability of my code and the likelihood of failures in the presence of bugs or wrong mental models.

A Story From Last Week

I was tasked with a new feature in the middle of the worst code you can imagine. I identified a seam where I could introduce the logic, and I was glad to see the surroundings were covered with tests.

However, as soon as I opened the specs, my heart missed a beat. Some tests were as dry as beef jerky:

it "works for a business owner" do
  perform_test(@business.owner)
end

Other tests were soaked wet:

describe 'for a wholesaler' do
  it 'with an existing payment card' do
    build_fixtures(cart_owner: nil)

    wholesaler = create(
      :wholesaler_org,
      :with_payment_card,
      business: @business
    )

    @business_cart.parent = wholesaler.owner

    user_session = build_auth_session_for_user(wholesaler.owner)
    user_session.sign_in

    order_create_params = {
      wholesaler: true,
      business_cart_id: @business_cart.id,
      fulfillment_type: 'shipping',
      fulfillment_date: (Time.zone.now + 1.week).iso8601,
      order_customer_id: wholesaler.owner.id,
      order_customer_email: wholesaler.email,
      order_customer_name: wholesaler.name,
      order_customer_first_name: wholesaler.owner.first_name,
      order_customer_last_name: wholesaler.owner.last_name,
      order_customer_phone_number: wholesaler.phone_number,
      fulfillment_location_type: 'Address',
      fulfillment_location_id: wholesaler.owner.default_address.id,
      payment_card_attributes: {
        payment_type: 'credit_card',
        id: wholesaler.payment_cards.first.id,
      },
    }

    user_session.post resource_path(@business.id), params: order_create_params

    expect_model_response(Order.last)
    expect_json(
      parent_type: 'WholesalerOrg',
      parent_id: wholesaler.id,
      order_customer_email: wholesaler.email,
      order_customer_name: wholesaler.name,
      order_customer_phone_number: wholesaler.phone_number,
      fulfillment_type: order_create_params[:fulfillment_type],
      fulfillment_date: Order.last.fulfillment_date.iso8601(3),
      fulfillment_location_type: Address.to_s,
      fulfillment_location_id: Address.last.id,
      total_price: @business_product.business_product_templates.first.unit_price + 0,
      total_product_price: @business_product.business_product_templates.first.unit_price,
      total_tax_price: 0,
      payment_source_id: wholesaler.payment_cards.first.id
    )
    expect_json_sizes('order_items', 1)

    expect(@business_cart.reload.total_price).to eq 0
  end
end

Both completely unreadable: it was not clear to me what was under test or how they even worked. So I created a brand new file:

it 'with wholesale order it does not create tax records' do
  owner =
    User.create!(
      first_name: random_string,
      last_name: random_string,
      email: random_email,
      password: random_string,
      type: 'business'
    )
  business =
    Business.create!(
      owner: owner,
      name: random_string,
      phone_number: random_string
    )
  wholesaler =
    User.create!(
      first_name: random_string,
      last_name: random_string,
      email: random_email,
      password: random_string,
      type: 'wholesaler'
    )
  business_order = create_business_order!(business: business, recipient: wholesaler)

  CalculateTaxesAndUpdatePrice.new.call(business_order: business_order)

  expect(BusinessOrderTaxSummary.count).to eq(0)
  expect(BusinessOrderTaxLineItem.count).to eq(0)
end

I made noise explicit by substituting hard-coded values with random ones. The second step was to extract some builders and just leave pure signal in the spec itself:

it 'with wholesale order it does not create tax records' do
  business = create_business!
  wholesaler = create_user!(type: 'wholesaler')
  business_order = create_business_order!(business: business, recipient: wholesaler)

  CalculateTaxesAndUpdatePrice.new.call(business_order: business_order)

  expect(BusinessOrderTaxSummary.count).to eq(0)
  expect(BusinessOrderTaxLineItem.count).to eq(0)
end

A tad more readable, huh? It's clear that type: 'wholesaler' makes the difference and what the test is assessing.

I made sure the suite was green and pushed to CI. A couple of minutes later, I realized the build was red: the new test failed. I tried again on my computer, green.

I joked with myself that by introducing random values, I introduced random failures. Actually, that was exactly what happened, but it turned out great. Thanks to that, I discovered that wholesale orders are tax-exempt.

I discovered a business rule.

Sure, in this case, I could have written two tests: one for retail and one for wholesale. But there's a couple of problems: I couldn't test all combinations and I would write tests according to my bias. Random values defuse both of those issues.

I decided to introduce another element of randomness: instead of hardcoding one order-item per order in create_business_order!, I let the code generate an array of length 0, 1, or 2.

def create_business_order!(business:)
  BusinessOrder.create!(
    order_items: many { build_order_item },
    # ...
  )
end

def many
  Array.new([0, 1, 2].sample).map do
    yield
  end
end

Boom: another failure. Tax calculations assumed there would always be at least one order-item and started failing with 0.

I uncovered a bug in the code.

To summarize, random values allowed me to upgrade my mental model and find a bug. But this is no silver bullet, at least not in this form and shape.

Firstly, it's impossible to reproduce deterministically a failing test run: random values change every time.

This can be solved easily, at least in Ruby. The rspec test runner uses a seed value to randomly generate the order to run specs. The following line primes the Ruby's pseudo-random number generator with the same value:

srand(RSpec.configuration.seed)

With that in place, executing rspec --seed 1234 twice will both run tests in the same order and generate the same "random" values.

Secondly, it would be best if a test either always passed or not. With random values, it's not possible to guarantee that level of determinism. But running the tests multiple times gets it close enough. Remember, even when the test is run only once, it still covers more than a test with hardcoded values; it just takes more time.

In rspec it could be achieved with:

100.times do
  it "tests something with random values" do
  end
end

This is an excellent idea; you are a genius!

I wish I could take credit for it. I stole this technique from Romeu Moura after attending his Domain Invariants & Property-Based Testing for the Masses. And I'm guilty as charged since I pop up in the recording several times. I had short hair and wasn't as pink at the time, so bonus points if you can spot me.

Should you wish to explore this topic more in-depth, I suggest you look into Property-Based Testing. I also wrote Property-based testing (with a sprinkle of JavaScript) and Diamond kata via property-based TDD in JavaScript. Both featured on HackerNoon, so you bet they are good reads!

Get the latest content via email from me personally. Reply with your thoughts. Let's learn from each other. Subscribe to my PinkLetter!

How to Terminate Legacy Code without Getting Stuck

Riccardo Odone — Wed, 11 Nov 2020 08:14:31 +0000

You can keep reading here or jump to my blog to get the full experience, including the wonderful pink, blue and white palette.

Legacy and greenfield projects are two different beasts. Both require coding, but different rules apply. Failing to make the mental switch is an excellent recipe for getting stuck.

Some developers are perfectionists. But all of us have a minimum level of decency. We would never push bad code according to our definition of quality and skills.

Having an eye for good code is great in greenfield situations where the lack of constraints allows crafting software with finesse.

However, legacy code sits way below our minimum decency level and would require too much work to rewrite. We cannot undo months of accrued technical debt in a few days. And we shouldn't even try.

In this situation, we should embrace the inversion principle. Instead of trying to reach success, we should strive to avoid failure. Rather than repaying all the technical debt, we should just make it less awful.

Legacy code is not created all the same. Some never change and should be left alone. Some require multiple interventions. Don't try to make it perfect; just make it less awful for the next round of changes.

You will be back.

Get the latest content via email from me personally. Reply with your thoughts. Let's learn from each other. Subscribe to my PinkLetter!

Vanilla JavaScript vs. RxJs

Riccardo Odone — Thu, 05 Nov 2020 09:23:52 +0000

You can keep reading here or jump to my blog to get the full experience, including the wonderful pink, blue and white palette.

This post compares vanilla JavaScript with RxJs. My intent is not to demonstrate whether one or the other is the best approach. As always, it depends.

But I want to highlight the importance of tackling a problem from different angles. In this case, it's imperative against declarative, or "push" vs. "pull."

Also, different mental models provide insights that can be exploited in the solution, regardless of the paradigm chosen. In this article, the imperative approach helps exploring the problem, the declarative one distills the solution: both have their merits.

It's Monday Morning

While you wait for the browser to load the to-dos, you wonder about the feature you will be working on today.

Maybe you will work in Elm-land where run-time exceptions never show up, or you will be modeling new domains in Haskell where impossible states don't compile.

Nope, it's JavaScript. You need to add an input field to enable users to fetch data.

Damn.

You believe in small steps and short feedback loops, so this is your first move:

<input type="text" id="query" />

const callback = value => console.log(value)

const queryElement = document.getElementById("query")
queryElement.addEventListener('input', event => callback(event.target.value))

A glance at the browser confirms that typing in the input field logs the value in the console. Great!

Time to fetch:

-const callback = value => console.log(value)
+const callback = query =>
+  fetch(`https://httpbin.org/get?query=${encodeURIComponent(query)}`)
+    .then(response => response.json())
+    .then(response => console.log(response))

Another quick manual test confirms that the requests work.

You spend the rest of the day making things pretty and replacing the console.log() with the appropriate function to fill the DOM. Then, you move the ticket to done full of pride.

That was slick!

Unfortunately, the next day you get an email from the devops team with the following subject: URGENT!1!. After your deploy, servers started receiving a ton of requests.

You open the application and type "holy moly!" in the text field. Your heart skips a bit when you notice it generated 10 network requests:

"h"
"ho"
"hol"
"holy"
"holy "
"holy m"
...

Holy moly! indeed, I forgot to debounce!

+const DEBOUNCE_MILLISECONDS = 300
+let scheduled
+
 const callback = query =>
   fetch(`https://httpbin.org/get?query=${encodeURIComponent(query)}`)
     .then(response => response.json())
     .then(response => console.log(response))

+const debounce = fnc => arg => {
+  clearTimeout(scheduled)
+  scheduled = setTimeout(() => fnc(arg), DEBOUNCE_MILLISECONDS)
+}
+
+const debouncedCallback = debounce(callback)
+
 const queryElement = document.getElementById("query")
-queryElement.addEventListener('input', event => callback(event.target.value))
+queryElement.addEventListener('input', event => debouncedCallback(event.target.value))

To make sure not to piss the ops team again, you get deeper into manual testing. The debouncing works, but there is something strange: sometimes, the application displays data for an old query.

Aha, the responses are coming out of order.

To make it more visible you introduce a random delay in the fetch:

+const throttledFetch = (url, options) => {
+  return new Promise((res, rej) => {
+    const throttleBy = Math.random() * 10000
+    console.log(`throttledBy ${throttleBy} milliseconds`)
+    fetch(url)
+      .then(x => setTimeout(() => res(x), throttleBy))
+      .catch(x => setTimeout(() => rej(x), throttleBy))
+  })
+}
+
 const callback = query =>
-  fetch(`https://httpbin.org/get?query=${encodeURIComponent(query)}`)
+  throttledFetch(`https://httpbin.org/get?query=${encodeURIComponent(query)}`)
     .then(response => response.json())
     .then(response => console.log(response))

Luckily, you can abort the previous fetch before executing the next one:

+let controller = new AbortController()

 const throttledFetch = (url, options) => {
   return new Promise((res, rej) => {
     const throttleBy = Math.random() * 10000
     console.log(`throttleBy ${throttleBy} milliseconds`)
-    fetch(url)
+    controller.abort()
+    controller = new AbortController()
+    fetch(url, { signal: controller.signal })

It's almost the end of the day, and you are staring at this code:

const DEBOUNCE_MILLISECONDS = 300
let scheduled
let controller = new AbortController()

const throttledFetch = (url, options) => {
  return new Promise((res, rej) => {
    const throttleBy = Math.random() * 10000
    console.log(`throttleBy ${throttleBy} milliseconds`)
    controller.abort()
    controller = new AbortController()
    fetch(url, { signal: controller.signal })
      .then(x => setTimeout(() => res(x), throttleBy))
      .catch(x => setTimeout(() => rej(x), throttleBy))
  })
}

const callback = query =>
  throttledFetch(`https://httpbin.org/get?query=${encodeURIComponent(query)}`)
    .then(response => response.json())
    .then(response => console.log(response))
    .catch(error => console.log(error))

const debounce = fnc => arg => {
  clearTimeout(scheduled)
  scheduled = setTimeout(() => fnc(arg), DEBOUNCE_MILLISECONDS)
}

const debouncedCallback = debounce(callback)

const queryElement = document.getElementById("query")
queryElement.addEventListener("input", event => debouncedCallback(event.target.value))

The throttling code needs to be removed. Still, the software crafter inside your head is in pain. You shouldn't have to tell JavaScript what to do line by line.

Instead of "pushing" information around, you want to "pull" and react to it. It should be as declarative as a spreadsheet.

It's too late to conjure that thought, your fingers are already typing yarn add rxjs:

const queryElement = document.getElementById("query")

fromEvent(queryElement, 'input').pipe(
  debounceTime(300),
  map(event => event.target.value),
  switchMap(query => fromFetch(`https://httpbin.org/get?query=${encodeURIComponent(query)}`)),
  flatMap(response => response.json()),
  catchError(error => console.log(error))
)
.subscribe(response => console.log(response))

Not only this achieves the same result, but also it's shorter and declarative. Not to count the additional insight you notice from this new angle:

const queryElement = document.getElementById("query")

fromEvent(queryElement, 'input').pipe(
  debounceTime(300),
  map(event => event.target.value),
+ distinctUntilChanged(),
  switchMap(query => fromFetch(`https://httpbin.org/get?query=${encodeURIComponent(query)}`)),
  flatMap(response => response.json()),
  catchError(error => console.log(error))
)
.subscribe(response => console.log(response))

You make sure nobody else is looking, you sneak in the additional dependency, and you deploy.

Now, it's the end of the day!

Get the latest content via email from me personally. Reply with your thoughts. Let's learn from each other. Subscribe to my PinkLetter!

The Simplest Most Powerful Trick–Verification Steps

Riccardo Odone — Thu, 29 Oct 2020 09:26:19 +0000

You can keep reading here or jump to my blog to get the full experience, including the wonderful pink, blue and white palette.

Manual testing is an essential piece in the software development puzzle. Still, it's often not given the respect it deserves. Sometimes, teams work without any quality assurance and don't test at all. Sometimes, it's the author of the feature who half-asses manual testing. Sometimes, it's the client that clicks around the interface without any guidance.

Sure, it would be best to have an expert tester on the team. Still, not having one should not mean disregarding manual testing. If you are reading this article, I bet you are not a chef, but you manage to eat every day. Instead, we should enable everybody to perform manual testing. Even if the team can count on a dedicated tester.

Some years ago, I was exposed to an equally simple and powerful idea: verification steps.

Verification steps provide guidance on how to verify a feature implementation. They look similar to a description of how to reproduce an issue in a bug report. They don't come with a 100% save-ass guarantee, but the list of benefits is long, and it's not limited to finding bugs:

Summary. Ticket descriptions tend to be in the form of a conversation, not to count all the back-and-forth in the comments. Verification steps work as a TL;DR.
BDD. As a developer, I love diving into code right away. Sometimes it means I get lost in the details and lose sight of the big picture. Having a clear step-by-step breakdown in the ticket helps.
Definition of done. In some cases, it's not clear when to stop working on a feature. By following the steps, it's easy to verify that.
Empathy. Verification steps are a journey, the same one users will embark on. By following the same path, it's easier to get in their shoes.
Keep the team in sync. When everybody is working on their stuff, silos are born. By encouraging the team to test tickets via the verification steps, knowledge spreads.
Ensure successful deploys. Ever git pushed and realized, hours later, it has been rejected because your local branch was behind? Or maybe forgot to toggle a feature flag? I cannot count the number of times I deployed a feature when I actually hadn't. Luckily, anybody can catch it by following the verification steps.
Build things right. When focused on coding, developers embed their own bias into the software. A list of steps can not only shake programmers out of their own bubble, but they also allow the verifier to test according to their own bias. And yes, the more diverse the team, the better.
Build the right things. It's challenging to feel a feature by reading a ticket description. Verification steps make it more apparent what the final user experience would be. Also, designers can make sure the steps represent what they envisioned.
Doing one thing at a time. Any writer would say that drafting and editing don't go hand-in-hand. The same is valid for coding and manual testing. When the author of the code and the tester are the same person, verification steps help to enforce the boundaries.
Serendipitous verification. I had clients, designers, and testers providing better technical solutions than whatever I came up with in the past. By enabling everybody to chime in with their expertise, bias, and freshness, better solutions surface.

Get the latest content via email from me personally. Reply with your thoughts. Let's learn from each other. Subscribe to my PinkLetter!

How to Deal with (Your) Unsatisfying Code

Riccardo Odone — Thu, 22 Oct 2020 07:50:57 +0000

You can keep reading here or jump to my blog to get the full experience, including the wonderful pink, blue and white palette.

Shit happens.

From time to time, I craft software that is not satisfying. Maybe it's messy, verbose, or namings are off. Maybe it's over-tested or under-tested. Maybe it's not correct, or maybe it's over-engineered. Maybe it just sucks, but I don't know why.

This happens when there's a gap between what I developed and what perfect could look like. It's also known as technical debt. The more of it, the more time it will take to understand the code in the future.

In the past, I committed the crime of adding a Refactor Feature X ticket to the cemetery (aka backlog). In other words, I handed off the problem to somebody else in the future (sometimes myself). It didn't work well, but, luckily, there are better ways.

Technical Debt is Not All the Same

Let's say there's a box somewhere stuffed with spaghetti code. It works correctly and never requires changes. Should you rush to refactor it? Of course not; it's a debt that doesn't need to be repaid.

However, with bugs or spec changes, you have to open that box and understand what's going on.

If you had a choice, which one of the following parallel universes would you choose?

Parallel universe number one:

class SomeClass
  # ...
  # 300 lines of code
  # ...
end

Parallel universe number two:

class SomeClass
  def go_for_it
    data = FetchDataFromDatabase.go_for_it
    result = PerformSomeCalculations.go_for_it(data)
    SaveDataToDatabase.go_for_it(result)
  end
end

class FetchDataFromDatabase
  # ...
  # 100 lines of code
  # ...
end

class PerformSomeCalculations
  # ...
  # 100 lines of code
  # ...
end

class SaveDataToDatabase
  # ...
  # 100 lines of code
  # ...
end

I would pick number two.

The chaos that can be fit in a box is proportional to its volume: three smaller boxes are better than a big one. Most importantly, in the second universe, SomeClass works as an index. It gives a high-level overview of the algorithm. Also, it suggests what boxes to open to change a specific aspect of the feature. And there's a chance it's only one class, not all of them.

If you lost the keys to your car, would you rather know they are somewhere in the apartment or in one of the drawers close to the entrance?

So What to Do with Unsatisfying Code?

If the specifications are stable and the code works, better to leave it alone.

If it's likely to change, try to make sure it works correctly, and it reveals intentions. In other words, on the one hand, make a change less likely, and, on the other hand, make sure the future reader will be guided throughout the code.

You can structure your process to (almost) guarantee code that is both correct and intention revealing. Should you accept the challenge, here's some food for thoughts:

Get the latest content via email from me personally. Reply with your thoughts. Let's learn from each other. Subscribe to my PinkLetter!

From Zero to RxJs via Knowledge Transfer

Riccardo Odone — Thu, 15 Oct 2020 07:12:42 +0000

You can keep reading here or jump to my blog to get the full experience, including the wonderful pink, blue and white palette.

A couple of weeks ago, I talked about going From Temporary Knowledge to Permanent Knowledge. Last week, I shared 10 Knowledge Transfers that Make me Thrive with Legacy Code. Today, I want to ride the momentum and focus on how permanent knowledge got me up to speed with RxJs.

RxJs is a complex library. It's confusing because it requires solving problems differently. In particular, it makes heavy use of four concepts that are not straightforward by themselves: async programming, streams, event-driven modeling, and functional programming.

RxJs is incredibly powerful and declarative, though. Let me shamefully steal some code from rxjs.dev to show why:

// Vanilla JavaScript

let count = 0;
const rate = 1000;
let lastClick = Date.now() - rate;
document.addEventListener('click', event => {
  if (Date.now() - lastClick >= rate) {
    count += event.clientX;
    console.log(count);
    lastClick = Date.now();
  }
});

// RxJs

fromEvent(document, 'click')
  .pipe(
    throttleTime(1000),
    map(event => event.clientX),
    scan((count, clientX) => count + clientX, 0)
  )
  .subscribe(count => console.log(count));

At the cost of increased complexity, RxJs subscribes to a stream of click events, employing local state, and composing declarative code with functional programming trickery. Said differently, the vanilla JavaScript code models the evolution of a value over time by mutating (global) state.

I would posit, complexity is there regardless; in one case, it's in using RxJs; in the other, it's embedded in the vanilla JavaScript code. But I digress.

Hopefully, the example justifies the need for a different approach when writing RxJs code. However, I quickly managed to get up to speed by transferring knowledge from the past and filling in the blanks.

It's similar to those RPG games where the map is black until the area is explored. I had already visited most of the places surrounding the principles RxJs builds upon. These mental models allowed me to approach the library with ease. Let's see how.

Streams

In RxJs, everything is a value that evolves over time. It's like an array of elements that are appended eventually. That is why, in the example above, it's possible to map to transform each value, including the future ones!

In other words, arrays support a pull use-case while streams a push use-case. If we needed to see the room temperature in real-time, the former would require polling and state mutations; the latter would just work.

Great, modulo some quirks streams can be treated like arrays.

Async Programming

I still remember the confusion when I started working with promises (or futures). I could not reconcile sync and async programming:

var result = fetch(...)
console.log(result) // WRONG: it logs the promise

Then, I realized when you go async, you cannot go back:

fetch(...).then(result => console.log(result))

If a promise produces a value eventually, streams produce multiple values eventually. But everything is async, so the former requires then, the latter provides several operators (see the initial example).

Awesome, streams are like promises.

Event-Driven Modelling

Visualizing a problem in terms of events is strange. Luckily, I was exposed to DDD and Event Storming early in my career. However, we all know a perfect example of event-driven programming: spreadsheets.

In particular, to calculate in C1 the addition of A1 and A2, it's enough to use =A1+A2. That is, whenever A1 or A2 changes, update the cell to their sum. Notice that nobody "pushes" values to C1; it's C1 that "pulls" from A1 and A2. I find it elegant and clean.

Nice, event-driven programming is like using spreadsheets.

Functional Programming

RxJs employs functional programming for flow control, composition, and decomposition. On top of that, it adheres to many interfaces (typeclasses) that make me feel at home.

By looking at the PureScript bindings for RxJs, it's clear that an Observable is a Functor, a Monad, a Semigroup, and many others. That is enough to know I can use map, flatMap, concat, and more.

Hell yeah, learning monads was not time wasted!

Outro

Hopefully, this explains how knowledge transfer allowed me to rock and roll with RxJs.

Notice that permanent knowledge compounds. When I'm exposed to a new concept close to a dense area of my knowledge graph, I can approach it from multiple angles making sense of it with different mental models.

Get the latest content via email from me personally. Reply with your thoughts. Let's learn from each other. Subscribe to my PinkLetter!

10 Knowledge Transfers that Make me Thrive with Legacy Code

Riccardo Odone — Fri, 09 Oct 2020 11:33:26 +0000

You can keep reading here or jump to my blog to get the full experience, including the wonderful pink, blue and white palette.

It's a sunny day, like many others. You just got to the trash room to throw away the rubbish. Something strange jumps to your eyes as soon as you open one of the bins. It's a metal box. Lifting it up, you notice it ticks the seconds, and it displays a countdown on a red LED screen. Currently, it's marking 01:14 and going down. A jolt of adrenaline starts circulating in your blood: it's a bomb.

You are in luck; it's the perfect day to find explosives. Yesterday, you spent the entire evening on a Mission Impossible marathon. You manage to calm down and recall what needs to be done: cut the blue wire. You lift the box with a smooth movement, lay it down, and open the lid with care.

Cut the blue wire. Cut the blue wire. Cut the blue wire. To your dismay, as soon as the circuit is exposed, you notice wires of all colors but blue.

Legacy software is that bomb. Every time, it's different in its own dramatic way: there are the challenges of greenfield code and all the constraints coming from choices made in the past. As they say, it's easy to perform well when things are predictable and straightforward. Unfortunately, you don’t know what to expect when dealing with legacy code until you open the ticking box.

I'm currently rescuing such a codebase. It's not so awful because I've seen more spaghetti in my life. Trust me, I'm Italian. However, it's problematic enough to have discouraged developers in the past.

After several weeks, I'm proud to say that not only I overcame the mess, but also I've been thriving in this project. I credit investing in transforming temporary knowledge into permanent knowledge. In other words, I didn't have to relearn anything from scratch; I only transferred knowledge over from past experiences and filled the blanks.

Here are the ten knowledge transfers that saved my ass:

1. Redux

In the past, I've dealt with several React applications that made use of Redux to manage state. But I didn't stop there. I decided to learn Redux's principles instead of just learning how to use it in a specific context. In other words, I laddered up from API bindings to architecture.

The project I'm working on employs NgRx, which, in turn, relies on Redux. I didn't have to start over with reducers, selectors, and so on. I took a quick glance at the NgRx's docs, and I was good to go.

2. Confidence

Legacy code is an emotional rollercoaster. Every up is matched by a down. Some days you think you've killed it just to realize later you screwed it up.

In the past, I had several experiences like that. This is why I'm confident that, as long as there is a roadmap, things will turn out ok; it's just a matter of time. Hell, it took us two years to remove Angularjs from AirCasting.

Another superpower I embrace is being at ease with ignorance. There is no way to know each and every corner of a codebase. It's quite dangerous to delude oneself of the contrary. I'm confident in being ignorant because it actually makes a difference.

3. Refactoring and Code Design

The constraints of a legacy codebase require expertise and flexibility. The strategies and tactics you apply by the book in a greenfield situation won't work. And thinking otherwise will result in being overwhelmed.

Most of the time, clean red, green, refactor cycles won't be an option. Also, you won't get to the perfect architecture on the first attempt. Legacy code grows messy over a long period, we should not expect to fix it in one take.

Nowadays, I learned to obtain feedback, not necessarily via TDD, to refactor with confidence. At the same time, I developed an eye for design (object-oriented and functional). Without it, the only refactoring one can afford is small in scope, like renaming a method or extracting a function.

I believe that code quality is free when done right, and in coding simple solutions to complex problems by design.

4. Being a Team Player

I'm astonished by the miracle that a team worked when I look at delightful software. There are so many things that could go wrong. The most challenging part is the coordination, though. Designers and developers need to understand each other. The team needs to foster feedback from the customers and analyze all the nuances. Everybody needs to agree on tabs vs. spaces.

I'm a fan of creating a team playbook, writing livable code, and focusing on generativity (rather than productivity).

5. Having a Product Mindset

A delightful product is not born by just moving tickets to done, deploying the latest bleeding-edge technology, or solving the wrong problems.

Everything is a hypothesis until it's validated. We need to uncover assumptions, ask silly questions, and find and treasure the most controversial disagreements.

6. Angular v4

This project could have been built in Rails. Unfortunately, it was overengineered. Every time I'm changing a form, I cannot help but think I'm doing double the work, with double the risks, without any gain.

My only experience with Angular has been with Angularjs in AirCasting. However, combined with several other projects on the frontend, it jumpstarted my understanding of Angular v4. Two-way data bindings, components' lifecycle, automatic dependency injection, routing, among other things, are common concepts across the board.

7. RxJs

Angular v4 makes extensive use of RxJs. Here, I found myself at home, thanks to my experience with Haskell and PureScript. As soon as I understood I was dealing with composition, functors, monads, and the like, I just did the functional programming thing: decomposed to simple problems and recomposed with the correct operators.

8. TypeScript

In Angular v4, types are a thing. To be honest, it was a let down to discover that, by default, any and null are allowed in TypeScript. The project I'm working on has 546 occurrences of anys, and makes extensive use of nulls.

It's a frustrating experience coming from Elm where runtime exceptions don't exist. Not all is lost, though; many of the techniques I learned in Elm to make programs correct by design can still be employed.

9. Tweaking Performance

Rescuing the codebase, we realized it's slow and breaks when there is too much data to send to the frontend. Who would have guessed pagination in a CRUD application was a requirement?!

Performance work is not my best suit, but permanent knowledge helped. I haven't stopped at optimizing Ruby and Rails code in the past. I learned to make things reproducible, measure them, and tweak the most significant bottlenecks first.

10. Solving Problems

As knowledge workers, we should go all-in to cultivate the ability to perform cognitively demanding work without interruptions. In a disrupted world, it's becoming rarer and rarer to find people who can sustain the discomfort of long and deep sessions of work.

Just yesterday, I opened a file, looked at it from multiple perspectives, and stared some more to understand how to tackle it. Had I given in to distractions, it would have taken more time, and the solution would have been inferior. Not to count the frustration of reloading information in my mental RAM at each context switch.

Get the latest content via email from me personally. Reply with your thoughts. Let's learn from each other. Subscribe to my PinkLetter!

From Temporary Knowledge to Permanent Knowledge

Riccardo Odone — Thu, 01 Oct 2020 08:31:34 +0000

You can keep reading here or jump to my blog to get the full experience, including the wonderful pink, blue and white palette.

My professional career started under the wings of two exceptional people: Ania and Marcin. They taught me something that stayed with me ever since.

I picture three elements of doing something: the WHY, the HOW, and the WHAT. We start with a goal, we rely on strategies and tactics, and execute.

During my internship, my mentors kept insisting on the WHY. I'm glad they entrusted me with figuring out the HOW and the WHAT. Because it's the WHY where I needed guidance.

They gifted me with an unfair advantage: extrapolating permanent knowledge from temporary knowledge.

Temporary Knowledge

Temporary knowledge expires. There's a long list of frameworks and technologies that were popular just a few years ago and are already long forgotten.

Recently, I've read the following:

Because of the temporary nature of the knowledge capital, computer programmers quickly reach a stage in their career when their old knowledge capital becomes worthless at the same rate as they acquire new knowledge capital. Their total knowledge capital is no longer increasing, so neither does their salary increase. They have reached the dead end plateau of their career, and it happens after less than ten years in the field.

To which somebody replied with:

One difference I've seen from young developers and more mature developers is that more mature developers tend to have a better understanding of how users will operate and will help mitigate issues before they arise. They have wisdom and that certainly isn't "temporary knowledge capital".

As a developer, I wish the latter was true. However, I cannot help but think there's some merit to the first take. I posit both opinions are valid through the proper lens, and we better be aware of it. Let me explain.

The WHATs of anything are bound to its context and lifecycle. For example, the syntax for records in Haskell is relevant only to Haskell code. However, the HOWs and the WHYs are part of the nature of software to a more significant extent. They can be transferred to other languages.

Don't get me wrong, temporary knowledge is essential. It's a bit like shallow and deep work. Our jobs entail both. We need the WHATs, in other words, the concrete implementations, to perform a task. At the same time, we should be contemplating the HOWs and WHYs behind them.

One can write code without critical or creative thinking, but it doesn't mean they should do it. Copy-paste from StackOverflow, anybody?

The problem is, by disregarding practices and principles, we are condemned to start from scratch and re-learn at every context switch.

I fear not the man who has practiced 10,000 kicks once, but I fear the man who has practiced one kick 10,000 times.

–Bruce Lee

Permanent Knowledge

Permanent knowledge does not expire because it's not bound to the current context. But it has a price. Given the higher level of generalization, it's often more challenging to acquire. It's a bit like in code, its easier to start with a concretion and, later, extract the abstraction.

Thus, one way to develop permanent knowledge would be laddering up from temporary knowledge. Instead of throwing arrays at problems, understand HOW to use them, most importantly, understand WHY. Syntax changes from language to language, but an array's nature is the same across the board. Hell, you could follow the abstraction to math, then it would apply to everything, not only software.

Recently, I started working on a legacy project that is over-engineered with all sorts of technologies. What saved me was transferring knowledge:

Haskell -> RxJs
Redux -> NgRx
Elm -> TypeScript

It's not knowing Haskell, Redux, and Elm that enabled me to thrive in this project. It's the fact that I understand the principles behind those technologies. They are just an incarnation of category theory, reactive functional programming, and strongly-typed functional programming.

At the end of the day, we need a concrete implementation (temporary knowledge). But we cannot do a good job if we are not guided by permanent knowledge. I can easily search the details of an algorithm on a tree if I know it exists. If I don't, I'm going to re-invent the wheel, and possibly screw it up.

I feel I should close this post with a joke. Something about a JavaScript developer trying to convince a Ruby developer that the data structure they need is an object, not a hash. But I fear it would reflect the reality of the silos created by temporary knowledge around us.

Get the latest content via email from me personally. Reply with your thoughts. Let's learn from each other. Subscribe to my PinkLetter!

Making an Endpoint 13 Times Faster

Riccardo Odone — Wed, 23 Sep 2020 07:35:56 +0000

You can keep reading here or jump to my blog to get the full experience, including the wonderful pink, blue and white palette.

Last week I prepared a script to reproduce performance issues. The goal was to populate the database with realistic data to test speed and UX on development and staging. As a result, we realized some screens in the application not only are slow, but they also break when displaying a big dataset.

We started working on the page that lists all the orders submitted by customers. The script inserts in the database an incremental number of them. Since in the screen it's possible to filter by date, this setup allowed us to check how fast the application loads compared to the number of items:

25 orders took about 8 seconds;
50 orders took about 12 seconds;
100 orders took about 18 seconds;
around 1000 orders, the application crashed with "Memory quota vastly exceeded".

As a developer, my first instinct was to jump in and make it fast. But there were two issues with that approach: identifying what problem to solve, and defining what fast means.

Even if it were technically feasible loading thousands of items fast enough, would it even make sense? In our case, we don't think so. Given our understanding of the users, we believe they would interact with the list by using the search field to drill down to one specific order. In other words, they would rarely scroll the entire list, especially when hundreds of items were displayed.

By taking a look at analytics, we identified where the application spends time rendering the list:

about 85% on the backend (more than 90% time is spent in the JSON serializer);
about 15% on the frontend.

Given this is a rescue project, we are aiming for good enough, not perfect. But what does fast enough mean in our context?

We went back to staging and loaded different amounts of orders on the screen to get a feel for it. There are more scientific ways of defining a performance budget, but we didn't want to overengineer.

With the user's need in mind, understanding the performance issues, and a definition of fast, I decided to explore solutions by spiking code. I had assumptions on what and how to optimize, but I wanted to validate the solution before going all-in.

I coded the JSON serializer from scratch, removed the unneeded data sent to the browser, paginated the endpoint, and removed the n+1 queries. This got us to 900 milliseconds, regardless of the search criteria. Since each page contains 50 items, we are more than 13 times faster in the worst case (i.e., 12000/900) and infinitely faster in the best case (i.e., the application does not crash).

The spike was successful. Now it's time to save the git diff, reset hard, and rewrite from scratch with the proper discipline.

Get the latest content via email from me personally. Reply with your thoughts. Let's learn from each other. Subscribe to my PinkLetter!

How to Investigate Performance Issues in a Web App with a Simple Script

Riccardo Odone — Wed, 16 Sep 2020 07:44:51 +0000

You can keep reading here or jump to my blog to get the full experience, including the wonderful pink, blue and white palette.

What happens when you build for several months without testing with real users and real data? An application that doesn't work with real users and real data happens.

This is what I wrote in my rant about 99 Percent Done. But I'm tasked with being part of the solution not part of the problem. So let's make performance right.

The first step to solve any coding issue is to be able to reproduce at will. In this case, the endpoints are slow, and I'm ready to bet all-in that it's due to database calls and JSON rendering. Not only that, the application monitoring service we employ draws the same conclusions. Thus, I could jump right in, gold master test, and rewrite queries and serializers. However, there are a couple of problems with this approach.

First of all, I would need to populate the database with enough data to perform a meaningful performance (and UX) test. It's not feasible to fill thousands of forms by hand to reproduce every time. A small, unrealistic dataset would not work either for obvious reasons.

Secondly, I don't want to optimize the entire application. It would take forever, and, at the moment, we are validating the product, not perfecting it. Said differently, I want to fix performance problems in places where users will spend most of their time, not optimize a screen they will rarely use.

Lastly and most importantly, I need to share performance issues with the rest of the team. We have to experience features to decide when they are both fast enough and usable. Technically the application could render thousands of things on the screen, but the user experience would likely suffer.

Given the shake list mentioned above, I got a bunch of options:

Tweak performance without any feedback. Nope.
Manually fill thousands of forms. Hell, no!
Some sort of database seeds on development and staging. This would yield the most fine-grained control. In fact, too much. I would have to understand the complex network of records each form submission generates and recreate it manually. Too error-prone and time-consuming.
Call controllers behind the endpoints directly. This seems sexy because I wouldn't need to get acquainted with implementation details, but mingling with the web framework's internals feels hacky.

If only I could simulate a user but faster. Wait a second! A person clicking things in a browser just sends HTTP requests to the server.

I can copy the requests from the network tab in the dev tools, translate them into code, and wrap each request in loops and conditionals. Moreover, there are services to translate cURL commands into another language (search "curl to MY_LANGUAGE"), and copying cURL requests is one right-click away:

Too good to be true, let's analyze the cons:

Coarser-grained control. I can either call an endpoint or not; there's no middle ground. This is a plus because it's what actual users do. Also, if I need to interact with the database, I could connect to it without a problem.
Automated emails. Luckily the framework allows disabling all deliveries with a boolean flag.
Third-party API calls (e.g., payments and tax calculations). I'm in luck again because orders can be marked as tax-free and paid in cash to avoid third-party service requests. This makes things a bit less realistic, but we could manually test those specific cases or develop more sophistication.
Authentication. The application uses tokens, so with a couple of additional requests, I can authenticate the script.

Great, there are no blockers. On the plus side, there are a ton of pros:

Easy to add new requests. Open the browser, simulate the user, copy the cURL, paste them in the script. We could even ask users to send that along as a way to reproduce things.
Test memory leaks. Among other issues, this application leaks memory, and the script can help investigate that problem.
Test concurrency. Requests could be performed in parallel to load test the application as described in Facing Concurrency in Web Apps Guided by Tests.
Measure. I could time each request to verify the performance tweaks do work.
Feedback. Anyone on the team can navigate to a screen and feel for its speed and usability.
We are the users. Performance can be measured into a single number, but UX is interaction, not math, at the end of the day. By putting ourselves in the shoes of the customers, we can make better decisions.

This looks promising. I'll let you know in the newsletter whether I crashed and burned or if I can move to rant about the next legacy issue in this codebase.

Get the latest content via email from me personally. Reply with your thoughts. Let's learn from each other. Subscribe to my PinkLetter!

99 Percent Done

Riccardo Odone — Thu, 10 Sep 2020 07:11:14 +0000

You can keep reading here or jump to my blog to get the full experience, including the wonderful pink, blue and white palette.

It's 99% done

This is what I've heard on the first day of a new project months ago.

I'm happy to report that, today, after several weeks of work, we are still 99% percent done. Or maybe I should say 99.9% done. Math is awesome, there's always room to add yet another nine to make it right.

I have enough experience to know that when 99% or almost are put in front of the word done, they actually mean nowhere close. The issue is that done does not mean anything until it's given a definition people can refer to.

When hard things surface, having a blurry definition (or none at all) justifies doing the easy work instead of the right one. Moving a ticket to a column named Done does not make the project one ticket more successful.

We need a definition of done, otherwise...

...There's No Finish Line

This product was supposed to be launched years ago. But without a definition of done, it was always not done, by definition.

Being a commercial endeavor, profit should have been the success metric guiding the effort. Instead, without boundaries, the product grew to a complicated aggregate of features that accrued bugs and welcomed scope creep.

...Developers Just Translate Specifications

Don't get me wrong, coding is awesome but I'm tired of building things that have zero positive impact. When developers are tasked with translating specifications into code, they will just do that. Or worse.

If people are not given the incentive to care about the product, they will have fun elsewhere. For example, by introducing accidental complexity. Or at least that's what I did in the past.

...We are Building the Wrong Thing

A ticket containing just a specification without any additional context can be moved forward in two ways: by doing a bad job or by doing a perfect one. Since the latter is unattainable, the result is destined to suck.

If we don't know who's going to use the feature and how, the only way to develop it is to cover all possible cases, many of which conflicting. Good luck with that!

...We are Building the Wrong Thing Wrong

What happens when you build for several months without testing with real users and real data? An application that doesn't work with real users and real data happens.

But it works on my machine. Yes, in development, with one user, and three products. It's going to be fun to fix months of work done on top of a slow infrastructure that leaks memory.

Have a Definition of Done in the Ticket

I have a wild idea. Maybe we should keep the definition of done in the ticket and let everybody know what we are trying to achieve.

Yes, that means we need to figure out what we are trying to do before we do it. But it doesn't seem such a bad idea.

Get the latest content via email from me personally. Reply with your thoughts. Let's learn from each other. Subscribe to my PinkLetter!