DEV Community: Mike Lowen

Test Data Factories in Javascript

Mike Lowen — Sat, 14 May 2022 09:10:25 +0000

Originally published on my blog.

Friday the 13th was a quiet evening and when the rest of the family were in bed or watching TV I chose to play around with a bit of the thought experiment. This time unlike others I wrote up what I was doing as I went along, what follows is a tidied up and expanded version of the notes that I took. While I was doing this I was primarily using two tools - RunJS for my javascript REPL playground and Typora for my notes.

In the ruby world I'm a fan of factory_bot for generating objects that will serve as test data in my unit tests. I had a bit of time to kill this evening and thought it would be fun to implement the subset of the functionality that I use in factory_bot in javascript to see how I would approach it.

For those unfamiliar with factory_bot is an implementation of the factory pattern to generate objects as alternative to using fixtures in tests. This approach allows you to generate your test data in a known and expected way will allow you to overwrite any properties of the object as appropriate for the test. This approach allows you to remove a bunch of duplicate code from your tests. As an example of how I would use this in my current project I have a front end driven by APIs below is a sample payload from one of the APIs

    {
  "_links": [
    {
      "href": "/clients/898f3b58-6c16-40f7-bf29-c9dac868bb2c",
      "rel": "self"
    },
    {
      "href": "/clients/",
      "rel": "up"
    },
    {
      "href": "/organisations/1f7c161a-8de1-44ff-9ac3-e36ecc01dfdc",
      "rel": "organisation"
    }
  ],
  "_type": "urn:emboss:client",
  "name": "Testing",
  "oauth": {
    "id": "738542d0-e1fb-40c3-8786-8b47119d9151",
    "secret": "************************f8c644399992"
  }
}

Rather than have multiple copies of this object in the codebase each just slightly different to the last, what I'd like to do is have something like the following:

const payload = build(payloadFactory, { name: "Development" });

Where the second argument contains any overrides you want to make to the base object. I've got a bit of a quiet evening so lets play around with this idea, first lets define what we want a factory to look like. Considering that I'm restricting the functionality to only building objects what's wrong with what was defined in the example payload? It's simple and it works, we can assign it to variable call it our factory and pass it into a build function with some overrides to modify some of the properties to create a resulting object. Following that line of thinking the first version of our build function is as simple as:

function build(factory, overrides = {}) {
  return { ...factory, ...overrides };
}

As a generalisation this works great until we want to have more than one instance of our object, the simple solution to that is you overwrite more data to make each unique but you end up finding yourself having to do that a lot, wouldn't it be neat if we could build in some automated uniquness into our factories? To acheive this what I'll do is extend our definition of a factory such that if any property of the factory is an function then it will be called to generate the value of the field in the resulting object. A simple example would be this factory:

const factory = { foo: 1, bar: () => true };

would result in

{ foo: 1, bar: true }

This would mean that should we actually want a property of the resulting object to be a function we'd have to do something like:

const factory = {
  foo: 1,
  bar: () => { 
    return () => console.log("I'm a method on the object!");
  }
}

What would that mean for our build method though? That gets a little more complicated now, first we want to generate the appropriate value for any field, which would take the key of the field, the factory which defines it and finally the overrides the method to do that would look like:

function populate(key, factory, overrides) {
  if (key in overrides) { return overrides[key]; }
  if (factory[key] instanceof Function) { return factory[key](); }

  return factory[key];
}

Next we want to construct the object from the factory based on the properties in the factory like so:

function build(factory, overrides = {}) {
  return Object.keys(factory).reduce((obj, key) => {
    return {
      ...obj,
      [key]: populate(key, factory, overrides)
    };
  }, {});
}

This will do what we want, it will allow you to use a library like Faker to generatedata to populate your object, it may not be the fastest way to do it but I think it's pretty legible. A nice side effect of this approach is this allows our factories to use other factories to define some of their properties, like so:

const factory1 = {
  foo: 1,
  bar: () => true
};

const factory2 = {
  tar: "string",
  baz: () => build(factory1)
}

The next step in complexity for our factories is to introduce dependencies, there are times when you want to populate data in your factory in a specific order, a simple example of this could be a factory like:

const factory = {
  firstName: () => faker.name.firstName(),
  lastName: () => faker.name.lastName(),
  fullName: () => '??'
}

We don't have a way to set it up so that fullName will be populated with firstName + ' ' + lastName, my thoughts around this would be to declare the dependencies on the method like so and pass the in progress object in like so:

const factory = {
  firstName: () => faker.name.firstName(),
  lastName: () => faker.name.lastName(),
  fullName: (obj) => `${obj.firstName} ${obj.lastName}`
}

factory.fullName.dependencies = [ 'firstName', 'lastName' ];

It would also be nice if that was wrapped in a helper function so the factory looks (IMO) cleaner:

const factory = {
  firstName: () => faker.name.firstName(),
  lastName: () => faker.name.lastName(),
  fullName: dependantProperty((obj) => `${obj.firstName} ${obj.lastName}`, [ 'firstName', 'lastName' ])
}

This however breaks our current build function, first we don't pass the object into the populate method and secondly we have no gaurantee on the order in which the properties will be populated. To solve this I'm going to order the keys to populate them in the appropriate order (though I'm not handling circular dependencies). Then to make sure that the population methods have the appropriate context to populate the object so it will now pass the object into the method when populating.

function populate(key, obj, factory, overrides) {
  if (key in overrides) { return overrides[key]; }
  if (factory[key] instanceof Function) { return factory[key](obj); }

  return factory[key];
}

function build(factory, overrides = {}) {
  const order = (prev, current) => {
    if (prev.includes(current)) { return prev; }
    if (factory[current] instanceof Function && factory[current].dependencies) {
      return [ ...factory[current].dependencies.reduce(order, prev), current ];
    }

    return [ ...prev, current];
  }

  return Object.keys(factory).reduce(order, []).reduce((obj, key) => {
    return {
      ...obj,
      [key]: populate(key, obj, factory, overrides)
    };
  }, {});
}

Again I've prioritised readability over performance, this was actually my second attempt my first attempt combined everything into a single function and felt very messy. Lastly I with the build method only populating the resulting object with the fields present in the factory I thought it gives us the oppotunity to pass additional context data into the factory via the overrides parameter and then pass that along to the property population method, allowing you to write population methods like so:

const factory = {
  name: dependantProperty(
    (obj, context) => context.nickName ?? `${obj.firstName} ${obj.lastName}`, 
    [ 'firstName', 'lastName' ]
  ),
  firstName: () => faker.name.firstName(),
  lastName: () => faker.name.lastName(),
}

build(factory, { nickName: 'Bob' });

Another neat aspect of the approach taken to defining factories is that you get extending factories for free by using the spread operator. If we take the above factory and wanted a new factory that extends it to includ an email address we could achieve it like so:

const subfactory = {
  ...factory,
  email: () => faker.internet.email()
};

This was a fun thought experiment about how I could bring some functionality from a library I like in a different language into javascript. It isn't however a 1-1 port of the functionality of factory_bot just the pieces that I found interesting to play around with in an evening. I've setup the code in a git repository if you want to have a look at it in its entirety, given time I may come back and further tune it and publish it to NPM.

Skills to focus on for staff+ roles

Mike Lowen — Sun, 10 Apr 2022 11:01:26 +0000

Originally posted on my blog.

Recently a senior developer at work who is on the individual contributor (IC) path asked for advice as what they should be focussing on as they progress towards a staff+ role. Reflecting on the feedback I realised that with a bit of generalisation it would be useful for anyone looking to step up into a staff+ style of role. With the permission of the developer who requested the guidance here is an edited and slightly expanded version of the feedback I gave.

A bit of context around the advice given the company I work for is a SaaS product company currently going on a journey to transform our monolith into a service based architecture, this does influence some of the advice and recommendations that have been given.

I'm the sort of person whose learning style suits asorbing information from books and then using little projects (when I can) to apply that knowledge. As such I make recommendations of a few books throughout this post, these are all books that I have myself.

As you progress your technical skills while important become less relevant in many respects. In my mind what differentiates an intermediate from a senior from a staff+ engineer is increasing less about technical skill as you go along and much more about what are traditionally called "soft skills", though I've never been a big fan of the term as I think these are the skills tend to be more difficult to master. As a generalisation communication is always a good thing to keep working on as it will become probably your core tool in your toolbox. The IC path is one that is centered around influence without authority and most of what you end up doing is trying to convince people to do things in the way you suggest.

In terms of specific skill sets that become more prominent when working in staff+ roles it's my opinion that you would want to focus on the following.

Strategic thinking

As you progress down the IC path you will have more and more influence on company and engineering strategy so it is important to get your head in that space. This can range from how do you ensure you are supporting the broader strategy and how do you foster and create the appropriate strategies and vision for engineers informed by what the company is doing. Part of the role of a staff+ engineer is to provide the guiding light to rest of the engineering capability. A good write up around formulating strategies can be found at StaffEng, a book mentioned in there that I have also had recommended to me by others is Good Strategy / Bad Strategy, I must admit this is one on my to read list but I've had enough people I respect talk well about it that I got a physical copy.

Product thinking

You will find yourself making more and more product decisions or at the very least in assisting in them. While a product manager is ultimately accountable for a product roadmap or strategy as a staff+ engineer I believe it is a shared responsibility between the two roles to develop those artefacts. You'll also find yourself being asked more and more product orientated questions by engineers as well so it is important that you build up that product skillset especially working if you continue working for product companies. A few books I might recommend are:

Tradeoff analysis

This again is another key skill set that will come into play, to quote Neal Ford

There are no right or wrong answers in architecture—only trade-offs.

Understanding the tradeoffs is also about how do you influence where you system is going in the future. Many times this may be unintentionally setting the direction for the architecture for the system. Where possible you should avoid concreting over a door and making a decision reversible if possible. When making faced with deciding between tradeoffs and making decisions a good question to ask yourself is "do I need to make this choice now?". Sometimes it's better to wait to gather more information and make an informed decision and wait for the last responsible moment.

One of the core responsibilities of a staff+ engineer in my opinion is balancing emergent vs intentional architecture. Intentional architecture is what you set out to build initially and emergent is what presents itself as you begin to actually build the system. In my experience most of the time the emergent architecture is usually the preferred option, it usually presents itself due to the discovery of new information. At the heart of this balance is making sure you understand the tradeoffs of following either path.

During delivery this quite often presents itself as when do we take on technical debt, which is always a tough decision and one where the manta "Perfection is the enemy of good enough" comes into play. At the end of the day the most important thing is deliverying business value, the code and architecture can be perfect but if it's not out in the world being used it doesn't really matter. You can always come back and fix up incurred debt (it's a different problem if not given that opportunity). In my experience and opinion the systems that usually have the higher levels of tech debt are new systems which brings to mind a quote by Reid Hoffman:

If you are not embarrassed by the first version of your product, you’ve launched too late.

Now all this is not to say you should always take the option to incur technical debt but it's important to understand the tradeoffs when making any decision. I think it was Neal Ford or Sam Newman who also said that (and I'm paraphrasing):

Every decision has trade offs, if you think you've found one that doesn't it just means you haven't found it yet.

Systems Thinking

Perhaps of all of the areas of focus here this is the one that comes most naturally to a developer, the key here is lifting up the level of abstraction which you are working at. Where you may be used to working at the class/function/story level you are having to hoist yourself up to understanding the impact on the entire system. I'd go so far that you need to make sure that you are operating at a level of abstraction where you understand the impacts on the people and process of either the company or the customer.

Stakeholder Management

Thinking on all of the above another of the key responsibilities of a staff+ engineer is stakeholder management. You'll be wanting to understand who they are balancing their various needs and wants and how that will impact the system that is being built. This also includes making sure they understand the impacts that technical strategies and decisions will have on them. One of the things I see people new to the role overlook is that the development teams are also stakeholders that you need to account for.

General Advice

From here I would like to offer more generalised advice from my own journey that I hope you find valuable. Starting with some advice that I got when I first moved into an IC leadership role regarding the developers in my team (I had just stepped into a technical lead role):

Just because it's not the way you'd do it, doesn't mean that it's wrong.

This is something that I still struggle with at times (I think it comes from being overly opinionated when the mood takes me), it is something that I've needed to loosen up on. Though loosening up can also be detrimental at times, where teams may go too far off the reservation, which is likely more to do with a lack of guard rails for the team to operate within. I've also in my time come to value a hunger to learn and adapt over straight skill sets when it comes to working with developers. Skills can be taught but that type of mindset it a lot harder to embed in someone who doesn't have it.

An architects or staff+ engineer (I consider the roles to be largely analogous) is a very amorphous role and will generally change to fill the "gaps" in an organisation this has also been referred to as being glue. While your focus will be on the more technical side of things, the people side should not be ignored. Much like DevOps can be described as the intersection of people, process, and technology the same can very much be said about architecture - after all that's why Conway's law is a thing. You will find yourself involved and helping to guide and define organisational change it can be helpful to have a view on how these sort of things could work and the impact it will have on how you build your systems, I've found team topologies to be one that works well for my mental model.

Now I know I said that I don't think that technical knowledge isn't as important it is not something that should be neglected, here are the list of technical books that I've found to be useful in my role/career:

And because I do love me some dead trees on my shelf here are a set of books I have on my backlog that I'm working my way through slowly but surely that I think could be applicable/interesting.

Staff Engineer (Currently reading)
EDGE: Value Driven Digital Transformation
Project to Product
Building Event Driven Microservices
Coders
An Elegant Puzzle
Lovability
The Mom Test

The Other API Version

Mike Lowen — Tue, 28 Dec 2021 20:27:15 +0000

Originall published on my blog

Versioning RESTful API endpoints kind of sucks, there are a bunch of ways to do it all with different tradeoffs which are largely semantic and no real consensus on the "right" way to do it. Thinking about versioning endpoints brings to mind that Winston Churchill quote about democracy:

democracy is the worst form of government – except for all the others that have been tried.

Where you can replace democracy & government with your preferred style & versioning respectfully. I think this largely comes from the fact that there isn't an ordained way to go about making breaking changes to your API outside of either don't or version it.

Endpoints though are not the only thing that should be versioned in an API, you should also be thinking about how you want to version your data. If you aren't versioning the data in your API and taking the default naive approach you are left in a place where the last write wins where whoever sends their request last will potentially overwrite any previous changes. You can visualise the problem like so:

In our hypothetical example both Client A and Client B have fetched and modified the same resource from the API but because Client A submitted their change last they have overwritten the changes made by Client B and none are the wiser Client B doesn't know their changes have been overwritten and Client A didn't know they weren't working on the latest version of the resource. In the worst case scenario from there those two clients may continue working working on the increasingly divergent resource each time overwriting the others changes.

This issue is most prevalent when updating an entity using the PUT method where you are replacing the entire resource (as opposed to PATCH which is only a partial modification). You could find yourself changing fields back to a previous state that someone else had modified.

Unlike endpoint versioning HTTP does provide us a couple of "right" ways to version the data in our APIs utilising standard headers. The first of these is If-Unmodified-Since this header accepts a datetime as a value that should correspond to the the datetime which the client retrieved the resource, when updating the resource if the last changed time on the server is more recent than that supplied in the header the change is rejected.

While this is likely this is the easier of the two options to implment (IMO) it doesn't completely mitigate the last write wins problem. With this approach we are reliant on clients to track when they retrieved the resource, if the client so chooses they could just specify the current datetime when making the update and the API would be none the wiser.

The second approach utilises two HTTP headers ETag and If-Match. The ETag header is supplied by an API when returning a resource and the value is intended to represent the version of that resource. What you use to represent that version is up to you as an API implementor, I often use a UUID that I store with the resource in the database but more commonly you will see APIs use a hash of the content or a revision number. The If-Match header is the other half of the solution, the client uses this header when sending the update to the API and it should contain the version of the resource (received via the ETag) that the client has modified. If the version supplied in the request does not match what the API is expecting then it will reject the update.

My preference is to use the ETag / If-Match combination as while it is a little more effort to implement it provides better protection against overwritting other clients changes to resources. Regardless of how you approach versioning of data in your API I think that it's important that you do version the data. It provides clients integrating against your API a consistent and expected experience where they can rely on any data they submit not being accidentally overwritten.