DEV Community: Carsten Zimmermann

Don't call it `*_id`!

Carsten Zimmermann — Sun, 07 May 2023 10:08:22 +0000

In a distributed system, we often need to store data that we do not own. We might use it as a unique identifier across domains or the system we do own needs to proxy it to yet another service.

In many cases, the origin system exposes it in the same way it is represented internally, for instance: a foreign key name of a relational database:

Fig. 1: local relational table, rev. 1

+----+--------------+----------------------+
| id | name         | backend              |
+----+--------------+----------------------+
| 42 | Marley Spoon | elixir, ruby, python |
+----+--------------+----------------------+

Translated to JSON, we could send it over the wire as follows:

Fig. 2: example JSON payload

{ 
  "id": 42, 
  "name": "Marley Spoon",
  "backend": ["elixir", "ruby", "python"]
}

A service consuming the information may want to add prefixes to scope it for "companies" and store/cache it like so:

Fig. 3: local relational table, rev. 2

+------------+--------------+----------------------+
| company_id | company_name | backend              |
+------------+--------------+----------------------+
| 42         | Marley Spoon | elixir, ruby, python |
+------------+--------------+----------------------+

But here's the problem: as engineers, we look at _id fields and immediately think of it as integers. However, the consuming service has no control over the data it receives and the data type is only assumed.

If you use that distributed ID field as a local foreign key: some external system controls the value and an unforeseen change might break our setup.

Identification

It has proven valuable to us to use the pattern *_identifier to indicate that…

it is some kind of a unique identifier
some other system has control over it

If the *_identifier value is to be stored, it should always be saved as a string type. Almost anything can be coerced into a string, and that way we guarantee that the origin system can choose whatever they want for their unique identifier.

This is particularly true if the origin system decided to move to using UUIDs. A final version of the local relational table above could look like this:

Fig. 4: local relational table, rev. 4

+--------------------------------------+--------------+----
| company_identifier                   | company_name | …
+--------------------------------------+--------------+----
| 328129ae-df4e-4168-94d3-2572b4b343ef | Marley Spoon | …
+--------------------------------------+--------------+----

Payload

If the system exposing the data is controlled by your organisation, we can support this at the source.

It is a common pitfall of API designs, especially RESTful APIs, to expose a resource exactly like you represent it in your database. This makes sense to reduce the cognitive load of the team maintaining the API. However, the data layer will inevitably change, rendering this point moot: the DB representation has to be translated to maintain a stable contract. Why not abstract from the data layer to begin with and name the keys in the payload in a system-agnostic way?

Fig. 5: JSON payload abstracted from persistence

{ 
  "company_identifier": "328129ae-df4e-4168-94d3-2572b4b343ef",
  "company_name": "Marley Spoon",
  "backend": ["elixir", "ruby", "python"]
}

Summary

Use *_identifier instead of *_id fields for externally owned data
Prefer a string type over integer for *_identifier values
Avoid a 1:1-map of your persistence model to your external API

Constant Doubts

Carsten Zimmermann — Thu, 23 Mar 2023 12:58:47 +0000

I always had mixed feelings towards Ruby constants. First, they're not that constant to begin with. You can reassign a constant at runtime freely and all you get is a warning:

irb(main):001:0> FOO = :bar
irb(main):002:0> FOO = :lol
(irb):2: warning: already initialized constant FOO
(irb):1: warning: previous definition of FOO was here

Contrary to popular belief, a #freeze won't help, either. Yes, it makes the object you're assigning immutable, but it doesn't prevent another constant assignment:

irb(main):001:0> FOO = "bar".freeze
irb(main):002:0> FOO = "baz".freeze
(irb):2: warning: already initialized constant FOO
(irb):1: warning: previous definition of FOO was here
irb(main):003:0> # 🤷

(It would be nice to be able to change warning into an error, but there's certainly no RUBYOPT flag that I know of.)

But most importantly: unless they're defined in the top level object space, I consider them implementation details. If an instance of A::Nested::Class accesses Some::Other::CONST, it crosses 5-6 boundaries (depending on where you start counting). That's not actually respecting that module's privacy and also a violation of the Law of Demeter.

`private_constant` all the things!

By default, constants are public. Most software engineers are very eager to work with the private keyword to limit the public API of their instances, but it's rarer to see that same rigor applied to class or module-level constants.

Ruby has private_constant since basically forever (MRI v1.9.3 to be precise). It accepts one or more symbols referring to defined constants in its scope:

module MyModule
  FOO = "bar".freeze
  LOL = :wat?

  private_constant :FOO, :LOL

  def self.foo
    FOO
  end
end

Any attempt to access MyModule::FOO from outside MyModule will raise a NameError ("private constant MyModule::FOO referenced").

Please note that MyModule.foo still works (and returns the frozen string "bar") as it only accesses the private constant from within its defining scope.

(Singleton) Methods Over Constants

I mentioned I consider constants implementation details. A named identifier for a magic value, maybe something configurable and set at load time. And sometimes, you want to expose that to other components in your applications.

As shown in the code snippet above, you can always create a singleton method / module function that wraps around a private constant.

In my opinion, methods are in all cases superior to CONSTANTS:

you can defer (lazy load) the assignment
you can memoize (@class_var ||= ...) what's being assigned
you can delegate the method call
it's easier to stub a method than a constant in your tests
it pairs well with making class/module-level value configurable on the application level, e.g. using a well-known MyModule.configure(&block) format or by using dry-config
it feels much more OOP to send a message to an object than working with its constants (also, I always feel that CONSTANTS YELL AT ME in the source code)

Points 1..5 all give you a great forward compatible way to refactor how your magic value is used, all for the small price of making your constants private and adding a getter singleton method around it if you really need to expose it to the outside world right away.

Enforce Explicit Constant Visibility

Unfortunately, there is no way to make all constant private by default, but RuboCop includes a RuboCop::Cop::Style::ConstantVisibility cop to at least make the constant scope explicit.

I like that it makes you stop and think about what you're doing.

Named Classes Are Constants

Now, constants aren't only referred to by UPPERCASE identifiers: all class and module names are in fact constants, so you could argue that my criticism about accessing Some::CONSTANT directly must also extend to a form like Nested::Class.new.

And indeed it does, but that will be the topic of the next article. 😀

Cover image credit: DALL·E 2.

What's the point? The 3 Es.

Carsten Zimmermann — Tue, 05 Apr 2022 12:00:54 +0000

Musings on the arbitrary number we attach to tasks & stories

Once per moon, we find ourselves discussing both what dimensions we should consider when we’re doing our Backlog Refinement and what types of work should receive Story Points to begin with. Occasionally, the term →“Business Value” is tossed into the conversation, but for my heavily backend-focussed team, it is very difficult to attach that particular metric to a highly technical task – and we currently have a lot of them.

This article originally aimed to conclude that conversation, a publicised Working Agreement if you will. It may serve as inspiration to others who wish to look at their pointing strategy through the lens of a Backend team. But as with most things on the internet, YMMV.

Issue Types

We work with Product Backlog Items (PBI) that can be categorised as one of the following:

Story

Captures work from an end user’s perspective, with varying definitions of what an end user is (for us, it can be another system consuming our public API). They typically follow the format of “As <who> <when> <where>, I want <what> because <why>”.

Stories are pointed.

Task

Describes an often technical aspect of an overarching goal. This could be a necessary refactoring, foundational work, paying back technical debt, or general housekeeping chores.

We point Tasks.

Bug

Represents a defect, something that isn’t working as intended. Bugs have varying priorities: some have a very low impact with known workarounds, the more urgent ones affect production and need to be addressed right away.

Some Bugs we point, some Bugs we don’t:

High-priority bugs that need to be added to current Sprint probably represent rework – a missed edge-case or something that slipped through QA. As such, it should not count towards the team’s →Velocity as that work should have gone into previous Sprints… so to speak
Bugs for which we have workarounds or whose impact is considered low should be part of the normal prioritisation process. Attaching a number to it will help weighing against other items in the Backlog.

Spike

Work to explore a new technology, test new processes, eliminate unknowns (see below, →Entropy). The goal is not to ship production-ready code, but rather to provide a proof of concept. Code produced while working on a Spike might deliberately forgo tests or sad path handling.

The main output of a Spike is information. The information is shared with the team. We currently do not point Spikes, but we time-box them (~2 days).

Epic

A container for related issues of various types. Ideally, they have a defined end-date and all its member issues (Stories & Tasks, mainly) are defined & scoped together to avoid too much context switching.

Possible anti-pattern: using an Epic to group related tickets in such a way that the Epic grows and grows.

Interpreting Points: The Three Es (3Es)

I understand story points to be one of the following:

Entanglement (as in: how many connections / interactions / dependencies)
Effort (as in: how time consuming is the actual work)
Entropy (as in: state of disorder, The Unknown)

Entanglement

Entanglement reflects how many moving parts have to be considered, the literal complexity of the feature. Higher complexity will translate to more diligence considering side-effects or edge-cases, as well as requiring a more thorough QA process.

Effort

Effort can be a measurement for how laborious a task is, even if it’s not rocket science. An example would be restructuring a code base where the goal is clear, but it will involve renaming many files and module names.

Entropy

Entropy expresses the uncertainty around a given task. It can cover blind-spots about the implications of a change, lack of experience with the domain or technology, a placeholder for the unknown Unknowns that might be uncovered while working on the task.

Note: if there’s too much entropy, creating a →Spike first is advisable.

Business Value

Now, the Three Es don’t include the hallowed Business Value. After all, everything in agile software development is about continuously delivering value, right?

Unfortunately, there is no objective scale by which we can measure Business Value. For some cases, “revenue added / cost saved” would be a good unit of measure, but “customer satisfaction” is nigh impossible to translate to money, whereas “better fault tolerance” requires a lot of effort (and guesswork) to put a € sign next to it.

Business Value is also time-dependent: it exists within the context of the organisation’s current situation and direction. Whatever arbitrary value we assign as Business Value for a story today might receive a completely different rating next month when re-evaluated.

Business Value is not a metric to feed into a team’s Velocity. It’s a means to prioritise work!

Further reading: Business Value by Erik de Bos.

Velocity & Capacity

In Agile software development processes, the Sprint Velocity is a metric for the output of a team, measured in Story Points completed. Cynically, one could argue that it’s also a metric on how good the team is at estimating, but let’s assume that outliers in estimation are inevitable and are evened out over time.

A (somewhat) deterministic pointing system where units of work (represented in PBIs) can be compared is key to assess the output of a team. Hopefully, the 3Es above help with that.

Now, Velocity as a metric for a team’s output (i.e. the sum of completed story points) is meaningless if it’s not correlated to the input of the Sprint: the person-days that were available. It’s the Story Points / Available Capacity that really matters. The rationale being: one Sprint might have lower output because ⅓ of the team was recharging their batteries on well-deserved PTO (reduced input).

If you want to track the relative efficiency of your Sprint, the relation of completed Story Points to Days Available provides the best insight and it also leads to more precise capacity planning: by calculating, say, a 4-Sprint-average of that ratio, you’ll end up with a multiplier to give you a good idea about how many Story Points you might complete in the next Sprint.

Sprint	1	2	3	4
A: Points completed	20	22	28	18
B: Available Days	40	35	42	22
Capacity Modified (A/B)	0.5	0.63	0.67	0.82

The 4-Sprint-average is (0.5 + 0.63 + 0.67 + 0.82) / 4 = 0.655. If you have 36 person-days available in your upcoming Sprint, you should aim for ~24 Story Points.

It’s controversial if you want to count unfinished business towards your delivered output. I get why you wouldn’t want that – a User Story that isn’t fully done doesn’t provide value – but we also established that we won’t look at value.

Being able to accurately plan what can be delivered in a Sprint is much more tangible than the incorporeal value and to that end, we split issues based on what was completed and what will roll over to the next Sprint.

Conclusion

In general, we already have a good grasp on how how many points to attach to a PBI – being primarily an #Elixir team, we have mentally assigned @default_points to 2 and just use the ~~module~~ brain attribute in Scrum Poker. This is a good sign, as we're apparently doing ok at splitting work into bite-sized chunks already. But for more complex issues and when the estimates given differ a lot, it's good to do a sanity check by looking at the three dimensions Entanglement, Effort, and Entropy separately.

Additionally, finally having resolved that considering Business Value as something that should go into a PBI's story point estimation was a huge win.

Header photo credit: PxHere, CC0 Public Domain