DEV Community: Anton Ivanopoulos

Detecting new and shifting public holidays with Ruby

Anton Ivanopoulos — Sat, 24 Sep 2022 06:38:04 +0000

Our app has a few features that involve patient communications, specifically around not sending those communications out when there’s a public holiday (when a clinic might be closed and unable to handle incoming calls).

Countries all have different holidays, but that also applies to regions and cities. Because people from all across the country use our app, we account for that. We’ll detect if there’s a holiday or not for a given set of users based on location, which will affect those comms going out.

In the last couple of years, some interesting cases had confused customers because holidays started behaving weirdly, sending out things when we weren’t expecting to, or not sending things when we should.

We use the holidays gem for detecting holidays on given dates, so I’ll go over what we ran into and how some previously-unused features of the gem helped us.

tl;dr

We had a case of an existing public holiday shifting from when it would usually occur, and we needed to account for the new date.
We had a case of a new, one-time public holiday that we needed to define ahead of time.

The basics

The holidays gem uses a series of yaml definition files that list all of the holidays in a given region. An example of one of these definitions is this one for New Year’s Day:

months:
  1:
    - name: New Year’s Day
      regions: [au, au_nsw, au_vic, au_act, au_sa, au_wa, au_nt, au_qld]
      mday: 1
      observed: to_monday_if_weekend(date)

There are a few parts here:

You specify the month the holiday occurs.
You can define more granular regions observe the holiday. In this case, New Year’s Day applies to all states in Australia as well as whole AU region.
You can define the day of the month it occurs on (you can also specify the week number it occurs in and the day of that week)
You can also define functions to handle the definition. In this case, handling moving the observed date to the following Monday if the holiday falls on a weekend. You can also just use a function to handle everything if things are particularly complex.

Then when you want to check if there’s a holiday in a certain date, you can check it in the following way:

Holidays.on(Date.civil(2008, 4, 25), :au)
=> [{:name => ‘ANZAC Day’,…}]

Holidays that don’t want to sit still

One of the interesting gotchas from the past couple of years with COVID is having to handle holidays that shift around due to extenuating circumstances.

In this particular case, we had support tickets from confused Brisbane customers asking us why things weren’t being sent out as expected on a random Wednesday in August. It turns out this holiday had a series of interesting cases from 2020–2021. In 2020, it got moved to a Friday when it usually occurs on a Wednesday, and in 2021 it was moved to a later date (October some time). On this most recent occurrence in 2022, it was a case of the holiday being detected by holidays on the wrong Wednesday in August.

Taking a bit of a closer look, we found the holiday defined as follows:

8:
  - name: Ekka
    regions: [au_qld_brisbane]
    week: -3
    wday: 3

It turns out that this mostly works, but, is actually a little too simple for how the holiday actually occurs. As taken from the QLD public holidays page:

The Royal National Agricultural (RNA) Show Day (Brisbane only) is held on the Wednesday during the RNA Show period. The RNA Show commences on the first Friday in August, unless the first Friday is prior to 5 August, then it commences on the second Friday of August.

So the holiday has a bit of a dynamic aspect not currently being taken into account. On a recent post, I had someone comment that they had used my post as a launchpad for making a change in an open source project, and that inspired me to go and sort this out myself. It ended up being a pretty simple change (my first proper OSS contribution, hell yeah), but essentially changes the holiday definition to:

- name: Ekka
  regions: [au_qld_brisbane]
  function: qld_brisbane_ekka_holiday(year)

With an accompanying function (I found writing this ruby function in yaml to be pretty funky but we got there):

qld_brisbane_ekka_holiday:
  # https://www.qld.gov.au/recreation/travel/holidays/public
  # The Ekka holiday occurs on the Wednesday during the RNA Show perido, the RNA show occurs on the first Friday of August, unless that’s prior to August 5, then it occurs on the second.  arguments: year
  ruby: |
    first_friday = Holidays::Factory::DateCalculator.day_of_month_calculator.call(year, 8, :first, :friday)
    if first_friday < 5
      second_friday = Date.civil(year, 8, Holidays::Factory::DateCalculator.day_of_month_calculator.call(year, 8, :second, :friday))
      second_friday + 5 # The next Wednesday
    else
      Date.civil(year, 8, first_friday) + 5
    end

Something I found pretty cool while putting this together is how the tests are done, they’re also in the yaml file. The development flow has you changing the branch that the holidays/definitions submodule under holidays/holidays is using, running some make commands to generate the functions and tests, then running those tests

- given:
    date: ‘2019–08–14’
    regions: [“au_qld_brisbane”]
  expect:
    name: “Ekka”
- given:
    date: ‘2022–08–10
    regions: [“au_qld_brisbane”]
  expect:
    name: “Ekka”

I added some extra tests for 2023/24, so we’re ready to rock when next year rolls around.

New public holidays

With the recent passing of Her Majesty Queen Elizabeth II, it was announced that Australia would have a Nation Day of Mourning on September 22nd and that it would be a public holiday.

The holidays gem allows you to load in custom holidays, so we opted to go that route in the short term. We added a new initializer to load in a custom holidays.yml file that contained the definition (shout out to Alex for putting this together):

# config/initializers/holidays.rb
Holidays.load_custom(“#{__dir__}/holidays.yaml”)

And then the definition:

# Custom holiday definitions for the holidays gem (https://github.com/holidays/holidays)
#
# See https://github.com/holidays/definitions/blob/master/doc/SYNTAX.md— -months:
  9:
    - name: National Day of Mourning for Queen Elizabeth II
      regions: [au]
      mday: 22
      year_ranges:
        limited: [2022]

Moving forward

This was all pretty straightforward stuff and I definitely don’t expect us to be needing to do these kinds of changes often (the definition for the Day of Mourning is already merged into the gem itself), but I really enjoyed getting a better look at the internals of this gem. Holidays are a bit of a hairy beast (especially when you have new ones popping up), but it was interesting to poke at some of the features on offer here that we hadn’t really had to play with before.

Some final questions I’m left with:

Can you override the definition of an existing holiday with something new?
Can you exclude holidays from the bundled definitions? (It seems like an open point of discussion in this issue)

Technical leadership during large refactors

Anton Ivanopoulos — Wed, 11 May 2022 09:16:49 +0000

A Long Time Ago™, we started a project at work to move our API serialisers from ActiveModelSerializers to Blueprinter. This kind of thing is hard. Hard to migrate a bunch of existing serialisers in prod on high traffic endpoints and be confident nothing will blow up in your face. Hard to get people motivated to migrate individual files and get things done. Hard to be sure that you've communicated the change correctly and that no one is left introducing more technical debt while you migrate.

These are all issues that we had to deal with doing our migration. Our migration languished for a long time in a kind of limbo state where we had started using the new library. However, an extensive collection of legacy serialisers sat gathering dust, not migrated as they were in critical or commonly-used paths. There was a general trepidation around getting in there and changing things over.

I chatted to team members who had contributed to the migration already, reflected on my own experiences, and identified some common concerns. In this article, I'll go through how we address the main hurdles we had with getting the project completed:

It wasn't easy to maintain momentum because it wasn't clear what the end goal was.
Even with the existing tests, people weren't confident that their changes would go out to production and everything would be fine.
It was unclear where to start or how to get involved.

Setting goal posts and building momentum

So there were two immediate issues I wanted to deal with before trying to start approaching the problem of actually migrating any files over.

How do we know when we've finished the work?
How do we know people aren't adding more technical debt with the old library while we're migrating?

Both of these were important problems to solve. It's hard to motivate folks to finish a job when they don't know how long is left to go, what the end state looks like, or in this case, how do we know when we're safe to remove the old library? And we want to remove as much friction as we can for other developers having to write serialisers during this period. No one wants to be confused about which bit of tech to use when starting something, and they don't want to have it pointed out to them in a code review after they've just finished doing their work.

I was reminded of some work I'd done recently in another project where I used a custom Rubocop class to solve similar issues. They're handy because you can enforce a certain direction in the codebase on an ongoing basis, preventing anyone from doing the thing you're trying to avoid, while also providing more context in the error messages they'll receive. On top of that, the list of offences Rubocop would generate for the existing files will reflect the body of work we need to get through before we can consider the project completed.

Here is what we ended up with:

# frozen_string_literal: true

require 'rubocop'

module RuboCop
  module Cop
    module Serializers
      class PreferBlueprinter < Base
        BASE_PATTERN = '(const (const nil? :ActiveModel) :Serializer)'
        MSG = 'Use Blueprinter, please!'

        def_node_matcher :class_definition, <<~PATTERN
          (class (const _ _) #{BASE_PATTERN} ...)
        PATTERN

        def on_class(node)
          class_definition(node) do
            add_offense(node.children[1])
          end
        end
      end
    end
  end
end

I'm still getting used to writing these. Still, this article from Evil Martians has been a big help. The rubocop-rails codebase also had some cops similar to what I wanted to put together. The cop we've put together checks if the class inherits from ActiveModel::Serializer and adds an offence to that line.

Now to get our todo list of classes we need to migrate. Running bundle exec rubocop --format files --only Serializers/PreferBlueprinter will generate a list of files that cause offences for our new cop. We can add that to our rubocop.yml file as an exclude list for that cop.

Serializers/PreferBlueprinter:
  Exclude:
    - 'app/serializers/old_serializer_a.rb'
    - 'app/serializers/old_serializer_b.rb'
    - 'app/serializers/old_serializer_c.rb'

That list is now how we're going to measure how close we are to being done. Each contribution to the migration will whittle that file down until we don't have anything left on the ignore list. After that, we can do a final PR to remove the cop and old library from our codebase.

Raising Confidence

For each desired change, make the change easy (warning: this may be hard), then make the easy change.
— Kent Beck

Okay, but we still haven't dealt with the root issue of people's low confidence and fear around changing these load-bearing serialisers. From chatting with the team, I learned two things:

People were less concerned about completely breaking things. There were enough tests to catch issues, and while annoying, we'll get alerts about that kind of thing and can roll back.
They were more concerned about subtler serialisation issues. How can we be sure that we're not introducing some small difference into the new serialiser that might cause issues?

So what can we do? I thought we could add some form of safety net so that if things did go awry with our changes, we'd a) know so we could fix them and b) not adversely affect production. I landed on a version on dark reading, except here, we're comparing the results of two different serialisers rather than reading from two databases. If we detect any differences, we'd throw away the new one and fall back to the old JSON.

The below helper class is a generalised version of what we ended up with (with extra comments around some of the decisions):

class BlueprinterMigrationFallback
  class BlueprinterValidationMismatch < StandardError
    def initialize(message, diff)
      @diff = diff.map do |operator, field, _|
        "#{operator} #{field}"
      end

      super(message)
    end

    attr_reader :diff
  end

  # For the original serialisation, we can just pass in the JSON output as is,
  # but, I chose to use a proc for the new serialisation so that we don't go through
  # the work of serialising it for requests that we don't need to.
  def self.validate(ams_json:, blueprinter_proc:, ignore: [])
    sample_rate = Rails.env.production? ? 0.1 : 1.0
    sampled = Random.rand < sample_rate

    return ams_json unless sampled

    blueprinter_json = blueprinter_proc.call

    # We found that some serialised fields _could_ be non-deterministic for
    # whatever reason, so we added support to ignore fields from a diff as
    # they would be different with each serialisation.
    diff = Hashdiff.diff(ams_json, blueprinter_json).reject { |_, field, _| ignore.include?(field) }

    # Raise our custom exception if there's a diff between the two versions.
    raise BlueprinterValidationMismatch.new('Resource failed to serialise with Blueprinter', diff) unless diff.empty?

    # Everything's all good, use our new serialisation!
    blueprinter_json

  # Rescuing from StandardError here in case the proc we pass in throws some
  # error we're not expecting.
  rescue StandardError => exception
    # Log the exception + the keys and operators involved in the diff. In our
    # app, this is where we capture the exception in Sentry so we can follow up later.

    raise exception if Rails.env.development? || Rails.env.test?

    # Return the original JSON.
    ams_json
  end
end

It's a pretty simple class, but the safety net of not using the new serialiser unless it matches the old one should raise confidence in making the changes. The tradeoff is that you're doing the work twice for a subset of requests, and that's a potential gotcha depending on what you're serialising and how heavy it is.

The other potential gotcha is the diffing itself, which can be a potentially long-running step depending on how big the payload is. We got bitten by that in one case where each record on the index action was also serialising thousands of associated records.

Getting people on the same page

Having added the helper class to the codebase and gone through a couple of migrations myself, I had a body of work I could point to as examples for others to follow.

I compiled that info into a document that outlined the steps one would take to migrate one of the old classes, why we approached the problem this way and some additional use cases that other people might run into. The document also reiterated the problem statement, why we were changing serialisers in the first place and what we were hoping to get out of it. The project had been in limbo for long enough that it was worth communicating that info that might have been lost to the depths of Slack.

Why are we even doing this?
If you've read my last post about the JSON module redefinition the Oj gem performs if you use it's .optimize_rails method, I went into some detail around the potentially unexpected behaviour that that introduces, particularly in places you're not intending for it to happen (i.e. leaking out into another gem).

This project started out for other reasons, but, this framing helped to give it new life.

How did we go?

None of these changes meant that the work was instantly completed overnight, but they provided a framework for getting the job done. The project is moving along again, and we've made our way through a third of the files we need to migrate (50 of a set of 150).

I'm really pleased with how this has turned out so far, and the process has given me more tools to approach managing a large cross-team change. I'll be sure to report back with anything new I learn as we make our way to the finish line!

Just to wrap up, these were some of the key points we looked at as ways to better manage a large cross-team change:

Ensure you can measure and report on progress. You need an end goal in sight and reporting on how you're moving can help keep the momentum up.
Minimise risk. Make it easy for people to fail, they'll be more inclined to help out and get involved if they have fewer things to worry about.
Document the change. Make sure you're able to articulate what you're doing and why, and what the benefits are.

Let me know if you've had your own experiences or challenges going through change management in a way that gets people on board or makes it easy for others to get involved.

Hunting down spooky JSON module redefinition via Oj

Anton Ivanopoulos — Tue, 26 Apr 2022 09:16:11 +0000

I've been spending some time recently to get on top of the dependabot alerts in our repo that have fallen by the wayside a little.

Many of these are often a quick look over the changelog and approve, assuming the tests all pass; there might be some QA involved if anything is particularly spooky-looking or load-bearing. Then some upgrades need a bit of a closer look because the PR bumps several versions, and there are some breaking changes in the API, or the behaviour is now different, and things break in the app, not usually a massive drama.

A new mystery

Something more interesting than I ran into recently came up while upgrading the version of fhir_client that we're using, from 4.0.3 to 5.0.3. Most of the changes looked relatively innocuous, except our test suite was now failing in a whole bunch of places with the following error:

NoMethodError:
  undefined method `has_key?' for nil:NilClass

There was no specific place that was causing issues; it seemed to me whenever we were making a request via the client we had set up to hit an external API. Taking a quick trip to the issues for the project turned up an issue that looked suspiciously similar but with no actual outcome at the end of the thread barring some workarounds.

So it seems like I was on my own here. Let's take a look at the full stack trace from the failures:

NoMethodError:
  undefined method `has_key?' for nil:NilClass

# /usr/local/bundle/gems/fhir_models-4.2.1/lib/fhir_models/bootstrap/json.rb:10:in `pretty_generate'
# /usr/local/bundle/gems/fhir_models-4.2.1/lib/fhir_models/bootstrap/json.rb:10:in `to_json'
# /usr/local/bundle/gems/fhir_client-5.0.3/lib/fhir_client/client.rb:397:in `request_payload'
# /usr/local/bundle/gems/fhir_client-5.0.3/lib/fhir_client/client.rb:527:in `post'
# /usr/local/bundle/gems/fhir_client-5.0.3/lib/fhir_client/sections/crud.rb:207:in `base_create'
# /usr/local/bundle/gems/fhir_client-5.0.3/lib/fhir_client/sections/crud.rb:177:in `create'

Cracking open the fhir_models gem with code $(bundle show fhir_models) on the update branch to get a look at the changes, I can see that the module having issues is pretty simple:

require 'JSON'

module FHIR
  module JSON

  ...

  def to_json(opts = nil)
    JSON.pretty_generate(to_hash, opts)
  end

  ...
end

I could confirm with pry that trying to generate the JSON with to_hash and opts (which was nil) was raising that error, and via rails console that trying to pass other random hashes of data to JSON.pretty_generate would also throw the same error when opts was nil, but would be fine if opts was an empty map.

The mystery arose when looking at the source for JSON.pretty_generate, where the definition of that method uses nil as the default value of opts, and the handling should very much take care of a nil value.

Things became a little clearer when trying to do the same test via irb:

$ irb
irb(main):001:0> require 'json'
=> true
irb(main):002:0> JSON.pretty_generate({a: 'b'}, nil)
=> "{\n  \"a\": \"b\"\n}"

No error. I began to wonder if JSON was even the same thing here. My next lead was that there was something off in our app.

A case of mistaken identity

I knew that our app was using Oj for JSON serialisation. But I didn't initially notice anything spooky about pretty_generate. A colleague of mine (shoutout Iain) had keener eyes and pulled up the C file that dealt with something spooky sounding in mimic_json.c.

Reading up a bit more about the Rails setup for the Oj gem, the documentation suggests adding the following to an initialiser in your app to have the gem "take over" (quotes theirs) many methods of the JSON gem. That info, partnered with the C file, illuminated us to the fact that the OJ.mimic_JSON method redefines the JSON.pretty_generate method (among others) with a different implementation, one that either doesn't handle a nil value for opts or changes the handling to be focused on an empty hash.

Taking a look back at our initialisers, I found one for Oj, one where we're calling Oj.optimize_rails, of which mimic_JSON is one of the steps.

Moving forward from a drop-in replacement

We find ourselves here because we broadly do a lot of JSON generation as part of our API. We use active_model_serializers, and the easiest way of getting the benefits of Oj's speedier JSON generation is to let it take over everything in one step making all of your serialisers more performant.

On the other hand, the problem is that you're now rolling with all of those redefined methods any time you use the JSON module, even when you're not a) aware of it or b) intending to use them.

In a happy coincidence, we're knee-deep in the middle of a migration of our serialisers to use Blueprinter instead of AMS. One benefit, in this case, is that it lets you explicitly set what generator you're using, like so:

require 'oj'

Blueprinter.configure do |config|
  config.generator = Oj # default is JSON
end

This whole scenario has fast-tracked our plans a little to finish off our migration so that we can remove gotchas like this and unblock our upgrade of the fhir_client gem that we were looking for at initially.

I always enjoy a bit of a spelunk like this. I'd be keen to hear of any other interesting scenarios that you might have run into that are similar.

TIL: Timecop has a safe mode

Anton Ivanopoulos — Wed, 13 Apr 2022 23:20:18 +0000

We use Timecop at work as a means of easily mocking the current date and being able to traverse around to simulate shifts in time. It's been an indispensable tool for testing out some particularly complex, time-sensitive components of our product.

Something a little inconsistent throughout the codebase has been people's preference between using Timecop's block or not. The latter relies on the developer to put time back to normal by calling Timecop.return after they’re done.

Lost in Time

Given that both #freeze and #travel mess with the app's understanding of what Time.now is at any given moment, what happens if you don't return time to normal?

This recently popped up as the cause of a few flakey tests that would rear their head in CI. We have a lot of tests in different areas that are sensitive to time. One, in particular, was popping up as a flake that was testing that a timestamp representing the last time a record was accessed was being updated. The flake that was popping up was failing intermittently because the timestamp was not changing, even though after some initial poking around it seemed like the actual functionality was behaving just fine otherwise.

It stood out to me that the failed assertion was seeing a very event time not being set, one that didn't involve any seconds or milliseconds. Having a quick search for that DateTime, I found another unrelated set of tests that were set up something like the following:

context 'context 1' do
  before { Timecop.freeze(some_time) }
  after { Timecop.return }
end


context 'context 2' do
  before { Timecop.freeze(some_time) }
end

So we had a context being set up in our test suite that was freezing time, but not putting it back to normal so that frozen time was leaking out and impacting other tests.

Enter Safe Mode

With things from my last post in mind, I really didn't want this to become another thing where we were like, "you need to know about this gotcha forevermore whenever you're going to use Timecop". Luckily, a quick look at the docs showed that the library has a safe mode feature that enforces block syntax use. You can add this in your spec helper with the below:

Timecop.safe_mode = true

Adding this raised several errors in the test suite in places that were set up like the above contexts:

Timecop::SafeModeException:
       Safe mode is enabled, only calls passing a block are allowed.

With the tests failing, we now had a list of tests that we needed to go and amend and use the block syntax where necessary. A basic version of what that might look like is:

it 'does something time-sensitive' do
  Timecop.freeze(some_time) do
    # Some test setup and assertions
  end
end

At the end of your test, time will be back to the way it was without having to call Timecop.return. This can get a little monotonous if you have a lot of tests in one area that is all time-sensitive, but you can still achieve something similar to context 1 above using the setup hooks.

context 'context 1' do
  around do |example|
    Timecop.freeze(some_time) { example.run }
  end
end

I was pretty happy with how this turned out, a rather insidious cause for flakey tests had been ironed out, and the issue of consistency between the two ways of using the library was solved by being forced to use one of them.

TIL: Queuing Sidekiq workers safely with the help of Isolator

Anton Ivanopoulos — Tue, 16 Nov 2021 07:58:32 +0000

Listening to a recent episode of the thoughtbot podcast, The Bike Shed, one of the hosts talked about a couple of gems that they'd come across that seemed pretty handy to me.

The first is isolator, a gem by the folks over at Evil Martians that detects non-atomic interactions within a database transaction. What do we mean by that? A simple example taken from the documentation that uses background jobs is something like the following:

User.transaction do
  user.update!(confirmed_at: Time.now)
  UserMailer.successful_confirmation(user).deliver_later
end

Some variation of the above is relatively common in many Rails codebases that I’ve worked with, more so when you start following the command pattern where you might have some service objects doing some work in a transaction.

Another reasonably common sight is to defer work in a Rails model's hooks, usually after_create, after_update, after_save, or even some combination of those. This particular set of hooks occur while the database transaction is still active. An example:

class Comment < ApplicationRecord
  after_create :notify_author

  private

  def notify_author
    CommentMailer.comment_created(self).deliver_later
  end
end

Isolator will throw errors at runtime whenever this kind of work happens inside a transaction.

Why should we avoid this?

Isolator appealed to me because I was in the middle of doing some of the plumbing behind migrating from DelayedJob to Sidekiq. The main difference here is that Sidekiq uses Redis as its backing data store. In contrast, DelayedJob uses a table in your database (I won't go into the pros and cons of either here).

Using Redis as its data store, Sidekiq processes jobs very, very quickly. This is where we start running into problems when queuing workers mid-transaction.

When enqueuing with Sidekiq (and Redis), there is a non-trivial chance that Sidekiq will pick up the job and start processing it before the transaction that queued the job has been committed to the database. In the case of the Comment model, it might mean that when the delayed mailer runs to notify the author about a new comment, that comment doesn't exist yet when the job starts processing.

How does isolator help here? The immediate benefit is knowing that this is happening in your dev or test environments. If we were to install the gem and create a new comment, we'd get an error something like the following:

Isolator::BackgroundJobError:
You are trying to enqueue background job inside db transaction.
In case of transaction failure, this may lead to data inconsistency and unexpected bugs.

To solve this, we want to change our model to queue up the mailer in a transactional callback like after_commit instead (we'll get to ad hoc transactions in a bit). We could rewrite the Comment model above like so:

class Comment < ApplicationRecord
  after_create_commit :notify_author

  ...
end

In my situation, isolator helped quite a lot because, with the relatively robust test coverage that we have, I immediately was able to identify several potential gotchas that would've happened if I'd just flipped over to Sidekiq as is.

I can see a couple of other potential benefits by having this gem in our project. I find having a new, concrete error around that particular pattern makes for a more straightforward explanation about what's happening and why doing this work within a transaction is a potential foot-gun, as opposed to starting at "my background job behaves weirdly in production sometimes". Having the errors helps a lot when it comes to code reviews as well. In the same vein as gems like strong_migrations, the author of the changes will get messages about these patterns ahead of time, and reviewers have one less thing to watch out for and can focus on other things.

What about those ad hoc transactions?

So changing the type of model hook solves our issue if we're queuing jobs up there, but what about that first example where we have a transaction we've started up manually in some other part of the code?

Well, the Evil Martians folks have something else for that problem. The after_commit_everywhere gem allows you to use similar transactional callbacks outside of your models, so something like this:

ActiveRecord::Base.transaction do
  after_commit { puts "We're all done!" }
end

I've not had a real opportunity to use this yet; I'm currently using the gem because it replaces the after_commit hooks provided by aasm, where we're queuing jobs on state changes. But, I like the syntax in theory; I think it reads pretty well, and if you're doing a bit of work inside a transaction, having that block at the bottom would express the intent clearly.

I'm keen to see how this gem shakes out post-migration and how helpful we find it having the safety net there while folks get used to the patterns. Let me know if you've used either gem or if you have any other tools that help you with similar gotchas.

Refocusing with smaller retros

Anton Ivanopoulos — Tue, 09 Nov 2021 05:37:26 +0000

I've had success with retros since I was introduced to them back in my first role as an engineer. I've always found them helpful from the perspective of improving on the more existential crises that teams I've been a part of run into and a good space for recognising the team's accomplishments as a whole.

One problem I've tried to tackle more recently is when trying to reflect on issues specific to a smaller (or sub) team when doing retros with a broader team. When trying to raise issues, it can be hard to get the engagement of everyone present because they weren't running into those problems or lacked context. The lack of context was also a hard one to tackle because there was so much context to give in the first place and that usually came with a lot of past baggage that came kicking and screaming with it.

Because I had a fair amount of autonomy in my position, I thought this marked a good time to introduce retros that focused on our product.

Finding the right people

Right off the bat, I had a small group of people interested in meeting and reflecting on the work this small team was getting done. My direct report and I were the engineering team behind things, and we had a UX designer that had worked closely with us for two years at that point, and we had a breadth of experience to look back on that we'd not done at that point.

We also had the idea of "product captains", members of the CX teams that would essentially be the experts on individual products we were offering. The product we owned had two, as there were two components to the product that, while similar, tackled different problems and had a lot of nuances that came with that.

Having the product captains come along was one of my main goals for these smaller retros. They were knowledgeable about the product itself, but because of their customer-facing roles, they had a breadth of knowledge about things customers were running into that we might not be privy to yet.

Not part of the formal proposition I was putting forward about starting these meetings up, but doing these smaller retros ticked a box for me as a manager. We'd recently gone remote at the start of the pandemic, and my direct report had only been with the team for a handful of months at that point. I found that introducing these retros to be a relatively small step we could take to get some more face time with other people in the business and some exposure to how they work.

Benefits and new reflections

We now had a space for our small team to reflect on the work we were doing. Great!

One thing that jumped out pretty early on, though (and I hoped this would be the case), is that this was not just a space for reflections about our engineering work. The product captains were bringing a lot of good material to us from their constant chats with customers. That brought about a lot of new conversations that wouldn't have come up in our regular engineering retro.

Those conversations gave a lot of insight into how people were using the product, what they were missing, and if we missed the mark on things we released. It also gave us a new way to celebrate accomplishments because we now had a customer-centric source flavour to the feedback we were getting.

On the flip-side, it gave the product captains a new sense of engagement and transparency that might have been lacking previously. They were now getting more actively involved in the product with a platform where they could raise their ideas and concerns, and they could get a feel for where we were at in terms of delivering the things we were working on, with a rough sense of how far away we might be from launching new features.

Overall, the new meetings breathed some new life into the work this team was doing when we might have been feeling more disconnected.

Retro'ing the retros

We were learning as we went with these new retros, and we found a few things early on that helped us be more effective:

Make sure everyone is on the same page around the purposes of the retro and what we're trying to get out of them. Not everyone had had experience doing retros before, so it helped get everyone familiar with the process. Interestingly this was one of my first times doing retros outside of an engineering org, and the level of engagement and the number of things raised was exciting.
Talk about what you're trying to achieve by meeting regularly. I didn't want to throw everyone into a new recurring meeting because it solved some of my issues. But, I saw an opportunity to shape how we worked cross-functionally from now on, and communicating that was important. Framing the retros in that light led to some exciting conversations about team goals, how we measure our success and what aspirations we should have.

On the whole, though, I think doing small-scale retros helped our team by giving us space to talk about our specific problems, and it gave us some new directions to explore now that we were getting different perspectives.

I'd be keen to hear if anyone's tried something similar and what you got out of it, what worked and what didn't, and if you're still doing them.