DEV Community: Jonathan

There’s no rule that Agents, Skills, Tools or Commands must be LLM-based. Some use cases require 100% accuracy and determinism – for those cases, we can apply these patterns to regular code.

Jonathan — Fri, 13 Mar 2026 23:43:31 +0000

Dealing with flaky tests

Jonathan — Tue, 10 Mar 2026 23:21:54 +0000

Flaky tests are tests that produce inconsistent and non-deterministic results, sometimes passing and sometimes failing. They can undermine the reliability of testing processes and complicate software development by masking real issues and wasting time.

Flaky tests are particularly difficult to debug and fix because of their non-determinism. We cannot simply go through the usual development cycle of test → modify → re-test. This is because we cannot reliably reproduce the error on each re-test, and thus, cannot know whether any single modification has corrected it.

Rather than going all in on one tactic, for flaky tests, I prefer to have a grab-bag of techniques at my disposal. I'll pick and choose one or tactics from this grab bag, based on the situation and context.

In this article I'll share my grab-bag of techniques. These are tactics I've used myself or seen used by others with success.

First we try to reproduce and diagnose the flakiness.

Then, once we have a (hopefully firm) notion of the cause, we can apply solutions or mitigations.

Solutions: actually fixing the flaky tests
Mitigations: minimising harm, impact, cost, etc. for flaky tests we cannot fix

The examples are in Jest and Playwright, as that is what I use in most of my work environments, but similar principles likely apply to other tools.

But before diving into tactics, let's take a brief step back and look at first reproducing the flakiness.

Reproducing flakiness

Fundamental to addressing any kind of software bug is reproducing it.

But how do you reproduce a flaky test? As discussed above, flaky tests are difficult to reproduce consistently because their failure is non-deterministic: sometimes they fail sometimes not.

There are a couple of options here:

Running the test repeatedly to generate a mass of failures
Observing prior test failures in logs

Running the test repeatedly to generate failures

We cannot reproduce the failure on a single run but we might have a chance on multiple runs.

Assuming Jest and a test script, we can use a command like this to repeatedly run a test:

for i in {1..20};
do (
  npm run test -- '{test_file}' --no-watch ||
  (echo "Failed after $i attempts" && break)
);
done

(Replace {test_file} with your test path and filename).

Some test frameworks provide this re-running capability out-of-the-box. Here's how to do it with the Playwright CLI:

npx playwright test {test_file} --repeat-each=20 --fail-on-flaky-tests

To increase the failure rate for reproduction purposes, we can simulate failure conditions.

For example:

Simulating slower CPU and/or fewer cores
Simulating lower available memory
Simulating slower network speeds

These failure conditions will give us more failures, which may hopefully resolve in faster diagnosis of the cause.

Some technologies for enacting these simulations include:

Virtual machines: Running the tests in a Virtual Machine with slow configuration.
Containers: Running the tests in a container, such as a Docker container, with slow configuration.
Test runners: Configuring the test runner itself to run tests slower.

Virtual machine

Using VirtualBox or similar, we can configure limited resources. VirtualBox allows limiting CPU count and processing cap.

Container

Using Docker or similar, we can configure limited resources. Docker allows this via CPU arguments.

For example, we could create a Dockerfile for our app:

FROM node:22.12.0-alpine
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .

And use it run our tests, with constrained CPU, like this:

docker run --cpu-shares 2048 tests npm run test

Test runner

In Jest, we can try one or more of:

Turning off the cache using --no-cache
Turning off multiple workers using --runInBand
Increasing the worker count, using --maxWorkers

Aside: By running a higher number of workers than CPU cores, we effectively add more workload to our machine, causing it to run slower. This can help create conditions that will reproduce a test's flakiness.

Observing prior test failures in logs

Alternately/additionally, we can try to gather information about the failures we have had so far.

For example, if a test is being flaky in CI, we can gather log output from the CI environment and examine it to search for clues.

Maybe we can learn something about the cause of the failures by observing phenomena related to the failures.

👀 Phenomena	🚨 Implications
Test fails at a certain time of day	Issue with date/time logic Issue with resource in a particular availability zone
Test fails only when modified	Issue with caching
Test fails when the CI server is being heavily utilised	Issue with test being vulnerable to resource availability

Aside: Tip: Structured logging with detailed information about resource usage, date/time and other data can help to expose more information about test failure in logs. Such info can be logged on test failures. For example, in Jest, we can handle the afterEach hook in a setupFile and inspect [testInfo.failed]. Structured logs can be generated using a library such as structured-log.

Diagnosing flakiness

Once we are able to reproduce flakiness, we can move to diagnosis, to uncover the root cause.

Similar to diagnosing regular bugs, we can diagnose flaky tests by making small changes and measuring the results. With flaky test rates, rather than a single pass/fail, we measure the overall pass/fail rate. A significantly lower percentage of failures can be correlated with a code change to help uncover the cause of the failure.

Aside: Warning: It's important here to emphasise failure of the test to run properly rather than just failure of the test. A flaky test might give false-positives, passing intermittently when it should be failing.

Some of the usual diagnostic techniques can be applied:

Debugging: using console logging statements to observe behaviour of the test and/or application code during test execution
Comparing recent versions: using git bisect or similar to compare results between recent versions of the branch
Process of elimination: removing parts of code and measuring results, to "eliminate" irrelevant parts and identify parts that are actually causing flakiness

Solving flakiness

This section covers possible solutions to flaky tests.

Some solutions might become evident from examining output after reproducing flakiness. In other cases, it might be worth experimenting with various solutions in a "try-and-see" approach.

Await your elements
Keep your promises
Reduce test size
Reduce test file size
Reduce the number of workers
Clean up at the end
Optimise the application under test
Disable resource-intensive application features when running in tests
Increase timeouts

Await your elements

Problem: Operations occur before the DOM has completed loading. For example, the test tries to click a button inside a dialog before the dialog has loaded.

Solution: Wait until elements have been rendered before performing operations that depend on them.

❌

const deleteButton = screen.getByText("Delete");
await userEvent.click(deleteButton);

const confirmButton = screen.getByRole("button", { name: "Confirm" });
await userEvent.click(confirmButton);

✅

const deleteButton = screen.getByText("Delete");
await userEvent.click(deleteButton);


await screen.findByRole("dialog");

const confirmButton = await screen.findByRole("button", { name: "Confirm" })
await userEvent.click(confirmButton);

Keep your promises

Problem: Test operations are being done before Promises on which they rely have been completed. For example, an async API call is made, but the test runs an operation that depends on the API result before the promise has completed.

Solution: Wait for calls to have been made, using, say, a "completed" flag.

❌

jest.spyOn(accountsApi, "getAccounts").mockResolvedValue([{
  accountId: 1234,
  isPrimary: true,
}]);
jest.spyOn(accountDetailsApi, "getAccountDetails").mockResolvedValue({
  accountId: 1234,
  balance: 10,
  accountName: "Jack's Primary Account",
});

render(<PrimaryAccountDetails />);

const heading = screen.getByRole("heading", { level: 2 });
expect(heading).toHaveText("Jack's Primary Account");

✅

jest.spyOn(accountsApi, "getAccounts").mockResolvedValue([{
  accountId: 1234,
  isPrimary: true,
}]);
jest.spyOn(accountDetailsApi, "getAccountDetails").mockResolvedValue({
  accountId: 1234,
  balance: 10,
  accountName: "Jack's Primary Account",
});

render(<PrimaryAccountDetails />);

await waitFor(() => {
  expect(accountsApi.getAccounts).toHaveBeenCalled();
});
await waitFor(() => {
  expect(accountsApi.getAccountDetails).toHaveBeenCalled();
});

const heading = screen.getByRole("heading", { level: 2 });
expect(heading).toHaveText("Jack's Primary Account");

Reduce test size

Problem: A test is very long and, thus, times out before being completed.

Solution: Reduce test size.

Even if we prefer longer integration-style tests, as recommended by Kent C. Dodds in [Write fewer, longer tests], there can still be ways to reduce the size of our tests while preserving their scope.

For example:

Remove operations that are incidental and not really required for the test to exercise the code.
Remove operations that are already covered by other tests.
Simplify operations. For example, retrieve a DOM element directly where possible, rather than traversing multiple parent/child elements unnecessarily.

Reduce test file size

Problem: Test files are large. Processing each large file ties up system resources (especially processor usage), causing other tests to time out.

Solution: Reduce test file size.

One way is to split up test files by function or component.

orders-pending.test.tsx

orders-delivered.test.tsx

orders-cancelled.test.tsx

orders-previous.test.tsx

Or we could use numbering or lettering system.

small-test-01.test.tsx

small-test-02.test.tsx

small-test-03.test.tsx

small-test-a.test.tsx

small-test-b.test.tsx

small-test-c.test.tsx

Reduce the number of workers

Problem: Test runners such as Jest may be greedy and fail to balance resource usage between tests when running many tests in parallel on resource-constrained environments such as CI

Solution: Check the CPU configuration of the CI if possible. Try to reduce the number of simultaneously running tests by configuring the test runner.

Contrary to our human intuition ("more is better") it may be better to reduce the number of simultaneously running tests. This is because test runners can be greedy and consume as many resources as possible at any given time (CPU, memory, etc). Running multiple tests at once can cause resource usage to become imbalanced, as tests compete with eachother for resources.

The solution may be to reduce the maximum number of workers. Jest allows this to be configured via the maxWorkers setting. However, if this exceeds the number of CPU cores on the machine, the processor may be forced to split execution time between multiple threads. This may cause tests to take longer than expected to execute, resulting in timeouts. This problem may only occur on CI environments, where CPU resources may be more limited, making it tricky to identify. Try reducing the maxWorkers setting or even eliminating it. (Jest defaults to the number of cores available on the machine, which is usually the safest bet.)

Clean up at the end

Problem: Tests leave behind "uncollected garbage", such as memory usage, threads, promises, etc. This slows down the test suite as a whole, making some tests flaky.

Solution: Cleaning up after each test reduces resource demand on the test runner, which reduces the occurrence of flakiness.

Note that the flaky test itself might not be the culprit here, but rather, some or all other test(s) as a whole generating garbage. This uncollected garbage might only be noticeable when all tests are run together in CI, not when running an individual test on its own. This can make the "uncollected garbage" issue tricky to detect. It might only be detectable by trial-and-error – say, observing resource usage on the whole test suite over repeated runs.

Increase timeouts

Most test frameworks provide allow timeouts to be configured.

Increasing the timeout allows tests to run longer without failing, which may solve flakiness.

Jest has testTimeout configuration and the timeout parameter
Playwright has timeout configuration and test.slow() (called within a test)

Downside: Increasing timeouts too much or globally might allow performance issues to creep into the application. Timeouts should be increased only on flaky tests if possible, and we should find ways to enure those features continue perform adequately for end-users.

Optimise the application under test

Problem: The application we're testing is itself buggy or just slow. If the application itself is slow, then the automated tests that exercise it will probably also be slow, leading to flakiness.

Solution: Find slow points in the application under test and optimise their performance.

To find slow points, we can add timer statements to different parts of the test or application.

console.time("Fetch user details");
const userDetails = await fetchUserDetails();
console.timeLog("Fetch user details");

We can also try rigorous manual testing, combined with performance tooling, such as Chrome Devtools Performance tab.

Techniques to improve performance can then be applied – see: improving performance in React apps.

Mitigating flakiness

So maybe we've tried all the above and nothing has worked. In that case, we can consider mitigation – approaches that reduce the impact of the problem without solving it entirely.

These might be used temporarily as an emergency resort or permanently if considered a reasonable compromise.

Disable resource-intensive application features when running in tests
Reconfigure test runners
Tag known flaky tests and configure accordingly
Use a different kind of test
Use a different method of verification

Disable resource-intensive application features when running in tests

Problem: Some features of our application may be resource-intensive, causing flakiness, while not offering much value in an automated testing context.

Mitigation: Disable resource-intensive features for test environment only.

Certain application features may be inherently resource intensive and not needed to verify correctness for a given automated test.

Common examples:

Animations (even when implemented with CSS transitions only can create drag)
Graphics (large complex DOM-heavy graphics and loaded as part of a page, such as 3D sprites in Canvas or complex SVGs)
Event Subscriptions (say, to backend events via WebSockets)

These features can be disabled only for test execution, via, say, feature flags.

Reconfigure test runners

Problem: Flakiness produced by resource-constrained environments is not worth the cost savings of the resource constraints.

Mitigation: Increase resources to get better value for investment, such as higher developer productivity during a critical period.

Depending on the cause, test flakiness might be drastically reduced in the short-term by simply beefing up resources on the test runners. Depending on the organisation, business context, timeframe, etc. this might be an optimal approach.

For example, suppose a legacy system is scheduled to be decommissioned in a few weeks, with a newer, totally re-written version already performing well in canary testing and ready to be rolled out next week. If the legacy system has a lot of flaky tests, blocking developers from deploying changes during that short space of a few weeks, it might make sense to increase resources just to unblock developers. Engineer time is more valuable and costly than brute resource usage.

Or suppose the business context is seasonally sensitive, such as an online retailer experiencing very high demand during holiday periods. During this period there is a high velocity of new feature releases, requiring a large number of automated tests of varying quality to run smoothly. Here, trading off resource cost for feature velocity might be worthwhile, at least during the peak period.

Tag known flaky tests and configure accordingly

Problem: Flaky tests block the whole pipeline, interfering with delivery velocity.

Mitigation: Separate flaky tests from non-flaky tests, to ensure that they run correctly or at least do not disrupt other tests.

Many test frameworks allow tags to be applied to tests, allowing those tests to be grouped and treated as a unit, for separate execution, separate configuration, or some other kind of separation.

Flaky tests, once identified, can be grouped in this way for special treatment.

In Playwright, test tags can be included in the test name:

test('test full report @flaky', async ({ page }) => {
  // ...
});

In Jest, a similar effect can be achieved by passing a carefully written regex to the testNamePattern config setting:

package.json:

"scripts": {
  ...,
  "test": "jest --testNamePattern='^(?!.*\@flaky).*$'",
  "test-flaky": "jest  --testNamePattern='^(.*\@flaky).*$'",
  ...,
},

Once separated, flaky tests might be treated in various ways:

Separate environment: for example, run flaky tests on an instance with more resources (CPU, memory), faster network connections, etc.
Separate lifecycle: for example, run flaky tests periodically, so that they are still useful but do not block non-flaky tests.
Separate execution style: for example, re-try the flaky test more times than other tests so that they don't fail

Use a different kind of test

Full end-to-end browser tests are known to be more flaky than traditional unit or unit-style integration tests. This is due to the performance overhead of loading a whole browser, loading the whole application at once, triggering interactions with whole DOM elements and waiting for feedback.

We could instead shift some of these tests to integration-style unit tests. Described in Kent C. Dodd's famous article Static vs Unit vs Integration vs E2E Testing for Frontend Apps, these tests can cover entire user flows (such as logging in) while mocking the calls that could otherwise call flakiness, such as server-side API calls.

Another option, for tests target intermittent but approximately deterministic behaviour, is to use fuzzy logic to verify that behaviour. For example, suppose we need to exercise some behaviour that operates on the current date and time, but for some reason cannot control the current date and time by mocking. If the test assertion does not need to have millisecond-level precision, perhaps we could instead assert against a range considered correct.

❌

  expect(timeAfterClickPause.getTime())
    .toEqual(new Date(2026, 3, 1, 13, 1, 1).getTime());

✅

  expect(timeAfterClickPause.getYear()).toEqual(2026);
  expect(timeAfterClickPause.getMonth()).toEqual(3);
  expect(timeAfterClickPause.getDate()).toEqual(1);
  expect(timeAfterClickPause.getHours()).toEqual(13);
  expect(timeAfterClickPause.getMinutes()).toEqual(1);
  expect(timeAfterClickPause.getSeconds()).toEqual(1);

Use a different method of verification

If our automated test is trying to exercise something that is inherently prone to intermittent failure, within no acceptable margin of error, perhaps we need a different method of verification altogether.

For example, there is probably no good way to write an automated test for generating the next Bitcoin hash on the official fork. (Until/unless we get quantum computing in the cloud, in which case, any crypto-based business model might be in jeopardy!) For this case, we would probably need to wait until we have a large and engaged enough user base and then apply observability.

Various methods of verification that might fit the scenario:

Local browser tests: tests that are run locally and manually by engineers, not in CI.
Monitoring / observability: simply provide observability into a feature without necessarily testing it. Failures can be surfaced in a dashboard or alert. Downside of this method: we risk catching a failure too late, after it has already affected a significant number of users before being discovered.
Manual testing: periodically manually test a feature in a prod-like environment.
Visual diff testing: automatically capture screenshots of the application or component, raising alerts when differences are detected. Causes of differences can be sought by, say, comparing versions of the code base (using git bisect or similar).

Conclusion

Flaky tests undermine testing processes, developer morale and ultimately product reliability. So it's important to address them. Unfortunately fixing flaky tests can be more difficult than consistently failing tests, due to their non-determinism.

Difficulties reproducing flaky tests can be addressed by:

Running the test repeatedly to generate failures
Observing prior test failures in logs

Flaky tests can be dealt with by:

Solutions: Optimising async or time-sensitive code, optimising test or file size to smooth test runner execution, configuring the test runner itself (workers, timeouts) or optimising the application under test.
Mitigations: Disabling application features that tend to induce flakiness, increasing test runner resources (temporarily or permanently), separating flaky tests (say, by tagging) or opting for a different kind of test or verification.

How I built my designerly CV

Jonathan — Sat, 28 Feb 2026 05:43:00 +0000

index.html

<p>hello</p>

As a Front End Engineer – a weird yet oddly common cross between software engineer and designer – I made three self-discoveries early on:

I am often looking for work
I care about aesthetics
I can hack almost anything

Thus, necessity proved the mother of invention, and I found myself striving to produce a CV worthy of my profession.

Selecting a word processor

This might seem like a problem of the 90s, but even now in an age of free browser-based apps, I still found it a non-trivial question.

My requirements:

Free & open-source (I'm well off enough, but just allergic to subscriptions to big tech firms)
Minimal dependency on web connectivity (I want to be able to edit my CV at short notice with limited Internet connectivity)
Able to save as a regular file in the file system and share easily (say, as an email attachment, in a LinkedIn message, etc.)
Able to export to common formats – both PDF and Word (in case a recruiter needs to edit in their logo)
Compatible with Mac and Windows (I sometimes work on a Windows box)

With a very small learning curve LibreOffice ticked all the boxes nicely.

Layout design

Before jumping into the word processor, I first developed a very simple pen & paper sketch, followed by a simple block diagram in Draw.io. This allowed me to experiment and tweak the design at a high level until I reached something I was comfortable with.

I ended up with a vertically stacked set of blocks for the jobs, with each block having a two-column layout with duties on the left and details (such as start/finish dates, tech and testimonials) on the right.

The design also includes a big header with some upfront features such as the tech I mainly focus on and my formal education. And an end section with side projects and short courses.

Implementation

Tables are supposed to be anathema to web developers, aren't they? Well, this design is intended for a static document format (Word, PDF) or printed paper. It is not intended for the browser. Those who prefer or need to use a browser are welcome to peruse my Work section on conwy.co or my LinkedIn profile.

It was quite easy to use the native Table, Paragraph, Bullets and Style & Formatting facilities of LibreOffice, which play nicely with Word and PDF. Margins, padding and bullet style are all fully configurable, so I was get a close to perfect design.

To be sure, as I was building the CV, I frequently exported to PDF and Word to check the results.

Dancing with content

Content writing for static documents is surprisingly different than for web, especially when working with a carefully crafted layout. You can't rely on the user's scroll bar so much, so text needs to be carefully crafted to fit inside the container.

I found myself going back and forth between Sublime Text and LibreOffice – the former to craft source material, the latter to edit it down to fit within the "real estate" of a formal document.

A few tricks in no particular order:

Shortening month names (~~Jan~~ → Jan, etc)
Abbreviating date intervals (~~year~~ → yr, etc)
Reducing font size on whole lines (like testimonial author and date)
Combining multiple details onto one line (like start/finish dates, period, industry)
Removing unnecessary verbosity and non-essential grammar (trading off minor grammatical correctness for visual appeal)

Finishing touches

For a finishing touch, I wanted to use a nice fancy font for headings.

After some exploration I settled on Sweden Sans, kindly made available in the public domain by the generous tax payers of one of the most liveable countries on Earth.

Web developers have it easy when it comes to interesting non-standard fonts. We just package the WOFF/TTF/ODF with our website and link to it. Or better yet, use Google Fonts and just grab a link there.

A quirk of word processors like LibreOffice, when exporting an editable format like Word/DOCX, is that you cannot rely on the end-user having installed all the required fonts.

But there's a hack! You can embed fonts in Word files. And it turns out LibreOffice fully supports font embedding. Simply go to File → Properties → Font tab and check Embed fonts in the document. This worked nicely, both for exported Word/DOCX and, of course, PDF files.

Download my template

In the spirit of Christmas I thought I'd share my CV, accompanied by a re-usable template.

My CV • Download: conwy-cv.zip
Designerly CV Template Package • Download: conwy-designerly-cv-template-package.zip

Generic HOC creator pattern for React

Jonathan — Fri, 24 Jan 2025 02:11:00 +0000

I've recently developed a pattern that allows me to compose two React components together in a highly decoupled way, using a generic HOC creator.

In this article I'll motivate and outline the pattern, provide an example, and discuss my own implementation, createWithHOC.

Motivation

Suppose we have an extremely general component, which we want to widely re-use by connecting it to other components in our component library.

This component is so general that most of its end-consumers will not know it even exists.

However, its end-consumers will need to customize its behaviour, by means of props.

Pattern

The pattern here is to create a HOC (Higher-Order Component) creator function, which can be applied to the general component by the combined components and exported for re-use by the consumer components.

The HOC creator function is called and passed the general component as a parameter, along with a string descriptor (more on that later).
Its result is a HOC, which is exported. This exported HOC allows the general component to be combined with another component.
One or more other components call this HOC, to combine themselves with the general component.
Its result is a component which has almost identical props as the combined component, except that it includes one additional prop. This additional prop, named by the string descriptor from step 1, contains the props to be passed to the general component.

Example

Let's look at a realistic example to see how this pattern can be applied.

Suppose we have a Tooltip component, which takes a children prop and a contents prop. The children prop will have the component that triggers the tooltip. The contents prop will have the component that should be shown inside the tooltip.

interface TooltipProps {
  readonly children: ReactNode;
  readonly contents: ReactNode;
}

export function Tooltip(props: TooltipProps) {
  // ...
}

Now in our component library, there are several different components that might all need to have a tooltip.

For example:

Button
Checkbox
Image

For each of the above three components, we want to allow their consumers to provide an optional tooltip. But we don't want to couple these three components too closely to the Tooltip component. And we want the re-use of the Tooltip component to be as easy and straight-forward as possible.

First, in our tooltip.tsx file, lets create and export an HOC that exposes Tooltip in a re-usable manner:

export const withTooltip = createWithHOC(Tooltip, "tooltip");

Notice that createWithHOC doesn't know much about Tooltip. It just receives its definition and a string descriptor - "tooltip". However, what it returns - withTooltip - is very useful.

Here's a simple implementation of our Button component:

interface ButtonProps {
  readonly children?: ReactNode;
  readonly onClick: VoidFunction;
}

export const Button_ = (props) => {
  return (
    <button onClick={props.onClick}>
      {props.children}
    </button>
  );
};

export const Button = withTooltip(Button_);

If we were to consume Button directly, without the withTooltip wrapper, it would have the children and onClick props, as we would expect.

However, with the withTooltip wrapper call, Button is now augmented with an additional tooltip prop. (Named by the string "tooltip", which we originally passed to the createWithHOC call in our tooltip.tsx file).

So now the props of Button_ look like this, from a consumer's point of view:

// Note: Fictitious interface name.
// This is just the un-named type of the props parameter of the
// `Button` component exported in the previous code sample.
interface ButtonProps { 
  readonly children?: ReactNode;
  readonly onClick: VoidFunction;
  readonly tooltip?: TooltipProps;
}

So when we consume Button, our code can look like this:

const Item = () => {
  return (
    <Button
      onClick={handleDeleteClick}        // <-- <Button_> props
      tooltip={{ contents: "Delete" }}   // <-- <Tooltip> props
    >
      🗑️
    </Button>
  )
}

This will have the same net effect as if we had consumed Button_ directly, wrapped in Tooltip, passing the appropriate props to each.

const Item = () => {
  return (
    <Tooltip contents="Delete">
      <Button_ onClick={handleDeleteClick}>
        🗑️
      </Button_> 
    </Tooltip>
  )
}

Advantages

I found three advantages of using this pattern, where applicable:

Smaller consumer code
Aesthetically pleasing consumer code
Ability to constrain re-use of general components

Smaller consumer code

Consumer code is significantly smaller and less indented, as we are relying on props rather than nesting. This benefit can add up quickly when, say, rendering multiple instances of the same component type side-by-side.

Here's an example of multiple buttons with tooltips:

const Item = () => {
  return (
    <>
      <Button onClick={handleDeleteClick} tooltip={{ contents: "Delete" }}>
        🗑️
      </Button>
      <Button onClick={handleEditClick} tooltip={{ contents: "Edit" }}>
        📝
      </Button>
      <Button onClick={handleCopyClick} tooltip={{ contents: "Copy" }}>
        📑
      </Button>
    </>
  )
}

Aesthetically pleasing consumer code

Consumer code is also more aesthetically pleasing, when relying on props. It presents as a cleaner, more compact and concise list of components.

Here's an example, in which we render different kinds of form controls side-by-side.

Using the generic HOC pattern, each form control is passed a prop to configure one or more of:

A label
A default
Validation logic

const Item = () => {
  return (
    <>
      <TextInput
        name="firstName"
        label={{ contents: "First name" }}
        validation={{ required: true }}
      />
      <TextInput
        name="lastName"
        validation={{ required: true }}
      />
      <DateInput
        name="dob"
        label={{ contents: "Date of birth" }}
        validation={{ required: true, maxDate: nowMinus18Years }}
        default={{ value: nowMinus20Years }}
      />
      <Checkbox
        name="registerForUpdates"
        label={{ contents: "Register for updates" }}
      />

      <Button type="Submit">
        Submit
      </Button>
    </>
  )
}

All controls appear at a consistent level of indentation and so do their props. This is easier to read and more aesthetically pleasing than if each control was nested in a different set of containers (validation, etc) and appeared at inconsistent levels of indentation.

Aside: This could be viewed as a variant of the principle of "cleaning up irregularity", discussed in Dustin Boswell's book The Art of Readable Code, under Aesthetics (Chapter 4, pp. 37).
The code follows a regular pattern, making it easier to visually scan and add/remove similar items (in our example, form controls).

Ability to constrain re-use of general components

An additional advantage of this pattern is that we could, if desired, constrain the re-use of our general component.

For example, we could keep the Tooltip itself private to its module folder and only export the withTooltip HOC. Then we could only apply withTooltip to components that we are confident will work well with the Tooltip.

As a consumer, it would be easy to determine whether a given component supports the tooltip or not – we could simply examine the available pops via our IDE's auto-suggest feature.

Here's an example with Visual Studio Code IntelliSense:

Implementation

Here's the createWithHOC function I developed in Typescript for creating the with* HOC creator.

import { get, omit } from "lodash";
import { ComponentType } from "react";

export function createWithHOC<THOCProps, THOCName extends string>(
  HOC: ComponentType<THOCProps>,
  hocName: THOCName,
) {
  return function withHOC<TLOCProps extends JSX.IntrinsicAttributes>(
    LOC: ComponentType<TLOCProps>,
  ) {
    return function ComponentWithLOC(
      props: TLOCProps & Partial<Record<THOCName, THOCProps>>,
    ) {
      const hocProps = props[hocName] as THOCProps;
      if (hocProps) {
        const hocPropsWithoutKey = omit(hocProps, "key") as THOCProps;
        const hocKey = String(get(hocProps, "key"));
        const locPropsWithoutHocProps = omit(props, hocName) as TLOCProps;
        return (
          <HOC {...hocPropsWithoutKey} key={hocKey}>
            <LOC {...locPropsWithoutHocProps} />
          </HOC>
        );
      } else {
        return <LOC {...props} />;
      }
    };
  };
}

I also published it as a Github Gist: create-with-hoc.ts.

Please feel free to re-use, tweak and/or share as desired. I hope someone out there finds it useful!

Diagramming React code

Jonathan — Sat, 18 Jan 2025 10:37:00 +0000

After working on a Typescript diagram format I wanted to focus on a React equivalent.

Diagrams can be useful for various purposes:

Quickly sketching ideas with pen & paper or a diagramming too such as Draw.io
Designing a solution at a high-level before writing any code
Comparing different designs with eachother to decide which one is best
Understanding an existing code-base by diagramming it
Working out a refactoring strategy

In this article I'll describe a UML-influenced diagramming format for React.

Overview

The format consistent of the following elements:

Component - rectangle with title and list of props
Component call - caller to callee connected with solid-arrow-terminated line
Component call with props - caller to props and props to callee connected with solid-arrow-terminated line
Component render props - render prop rectangle connected to component with dot-terminated line
Function - rectangle with title and list of parameters
Type or interface - rectangle with title and list of props

Component

A component is depicted with a depicted with a rectangle with <<Component>> descriptor and title at the top and, optionally, props underneath.

Component call

A component can render another component – here this is referred to as a "component call".

A component call is depicted with a line from the caller component rectangle to the callee component rectangle, terminating in a filled arrow symbol.

Component call with props

A component can pass props to another component – here this is referred to as a "component call with props".

A component call with props is depicted with a line from the caller component rectangle to a props rectangle and another line from the props rectangle to the callee component rectangle, terminating in a filled arrow symbol.

The props are depicted in a props rectangle, in which each prop has its own rectangle. This allows any individual prop to be linked to a type, function or component rectangle.

Component render props

Render props are props for which we pass a React component, a function which renders a component or a React node.

A render prop is depicted with a line from the prop box to a Component or Function rectangle, terminating in a dot symbol.

Function

Same as in the Typescript diagram format, a function is depicted with a rectangle with <<Function>> descriptor and title at the top and, optionally, parameters underneath.

Type or interface

Same as in the Typescript diagram format, a type or interface is depicted with a rectangle with <<Type>> or <<Interface>> descriptor and title at the top and, optionally, fields underneath.

A composition relationship between types or an inheritance relationship between interfaces is depicted with a line from the composer/inheritor to the composed/inherited type/interface, terminating in an empty arrow symbol.

A reference relationship between two components, functions, types or interfaces is depicted with a line from the referencer to the referenced, terminating in an arrow symbol.

Example: contacts list

Here's an example of a React diagram depicting components that make up a contacts list.

ContactsList component
ContactsList -> ContactListItem component call with render props
ContactsListItem component
ContactPhone component
ContactEmail component
Contact interface
getContacts function

Downloads

To make it easier to use this format, I've implemented downloadable templates for common diagramming tools:

Mermaid • Download: diagramming-react.mermaid.md
Draw.io • Download: diagramming-react.drawio
Figma • Download: diagramming-react.fig

AI for developers

Jonathan — Thu, 09 Jan 2025 03:00:00 +0000

Over the last couple of years, I've sprinkled AI usage through various typical software engineering activities.

The most common tools (and what I tend to use) have been Copilot and Copilot Chat, but some free tools such as Llama Coder are beginning to become more mainstream.

In this article I'll summarise the scenarios where I found AI useful.

Writing code faster with autocomplete
Getting help with error messages
Getting help with technical problems
Writing bash scripts
Generating unit test code
Summarising conversations
Code reviews

Writing code faster with autocomplete

AI-powered autosuggest was a great help when I had to write a chunk of code that's necessarily verbose, but not unique to my problem.

A perfect example is given on Copilot extention homepage.

Getting help with error messages

Pasting an error message into Copilot Chat, prefixed with /fix, can provide good guidance on the cause and fix for a given error.

When the responses given aren't suitable, follow-up prompts such as "can you suggest some other alternatives" can help.

Getting help with technical problems

Similar to error messages, I found that expressing a technical problem in an AI-powered chat can yield some good ideas and solutions.

This worked even better when I followed up the initial question with clarifications and/or challenged the AI to think of additional ideas or alternatives.

Writing bash scripts

Bash scripts can perform many kinds of general tasks. Once you have a script that does what you want, you can re-use it any number of times.

A specific need I had was to see all the authors who had edited a particular file in a Git repository. With the help of ChatGPT I was able to write a script that achieved what I wanted.

Here's the prompt I used:

Copyable: Please write a bash script which finds and lists all the unique authors who have edited the given file in the current Git repository. The given file will be specified as the first parameter to the bash script.

I've published the resulting script as a Github Gist: list_authors.sh.

The script takes as an argument the path to a file in that repo. It then uses git log on the current repository to find all the authors who have edited the file, sorts and uniqs them and outputs them to stdout.

Here's a sample of the output from running it on a file in the NextJS repository:

$ list_authors packages/next/src/shared/lib/head.tsx
Authors who have edited 'packages/next/src/shared/lib/head.tsx':
Adam Stankiewicz <sheerun@sher.pl>
Filipe Medeiros <filipesilvamedeiros@gmail.com>
Gerald Monaco <gerald@gmonaco.me>
JJ Kasper <jj@jjsweb.site>
...

Generating unit test code

I sometimes use GitHub Copilot to generate unit tests for a particular code file I am working on.

This can be done by opening the file under test, pressing Cmd+I and asking Copilot to write a unit test for the current file.

I would use a prompt like: "Please write a unit test for this file".

I use this is mainly to set up the scaffolding, such as describe and it blocks and basic assertions. Usually I need to tidy up a few things here and there, but the AI is surprisingly good at generating decent test code.

Another technique is to provide just the code for one function or code block and ask AI to generate test code for it, listing a few test cases.

Here's an example of a prompt I might use:

Copyable: Please write a Jest unit test for a Typescript function.

The unit test should have the following it blocks:

returns a list of dates between the two dates passed

returns an empty array if two identical dates are passed

throws an error if null is passed

Please write tests that satisfy the above it blocks.

Here is the Typescript function:

\`

function calculateDaysBetweenDates (begin, end) {

var beginDate = new Date (begin) ;

var endDate = new Date(end) ;

var days = Math. round (endDate - beginDate) / (1000 * 60 * 60 * 24) ) ;

return days;

}

`\

Summarising conversations

Some colleagues and I had a long back-and-forth about code reviews in a Github discussion.

When it came time to summarise all these in a meeting, I found that AI-powered chat was a great way to summarise the discussion and get everyone's thoughts on what we had discussed.

It looks like ChatGPT now has a quick Summarise button for this purpose.

Code reviews

When reviewing pull requests I've been experimenting with various ways of asking AI help me.

One way I found useful was to have AI review and summarise the changes and overall quality.

Here are the steps I took for doing this with Copilot Chat:

Generate a diff between the pull request source and destination branches.
- git checkout <source-branch> to checkout the branch
- git diff <destination-branch> | pbcopy to run the diff and copy the output
Paste the whole diff into a new empty tab in VS Code
Press Cmd+I and prompt Copilot: "Please review these code changes"

Here's a sample of the kinds of results I tended to get (anonymised):

The code changes introduce several new features and modifications to the healthcare application. Here are the key points:

Localization Updates:

Added new localization strings for transfer confirmations in en-AU.json.

Removed Mock Initialization:

Removed the conditional import of mock initialization in layout.tsx.

New Patient Transfer Form Submission Logic:

Created a new file submit.ts to handle the patient transfer form submission logic.

This includes creating a Ward object, making a patient transfer request, and handling different states of the transfer (success, delay, error, no-response)

Transfer Form Tests:

Updated transfer-form.test.tsx to reflect changes in patient name and ward number and added tests for the new patient transfer form submission logic.

Overall, these changes enhance the patient transfer functionality by providing detailed feedback on the transfer status and improving the test coverage for various scenarios.

Not as good as a human reviewer, but not useless either.

And not bad given the tiny effort on my part!

Conclusion

While I feel AI might have been a little over-hyped, I think there's still merit in becoming familiar with it and using it where it fits.

It seems likely that AI adoption among software engineers will grow, so it will be advantageous to be familiar and conversant with the tooling.

AI user interface patterns

Jonathan — Fri, 04 Oct 2024 03:01:52 +0000

As AI capabilities become increasingly available to everyday development teams, it might be useful to explore way to surface these capabilities to end-users.

This article describes some user interface patterns for surfacing AI.

Using "treat it like a person" as a core principle, I provide patterns, examples (including in-the-wild cases) and how they mimic humans.

Principles

As the definition of "AI" is still somewhat in flux, I want to assume a definition for the purposes of this article:

The capability of software/machines to do things that are normally thought of a human, in a rational way.

The fact AI mimics human thinking/behaviour suggests that we user interfaces should expose AI capabilities in a human way.

As Ethan Mollick puts it in the book Co-Intelligence (bold mine):

AI doesn’t act like software, but it does act like a human being. I’m not suggesting that AI systems are sentient like humans, or that they will ever be. Instead, I’m proposing a pragmatic approach: treat AI as if it were human because, in many ways, it behaves like one. This mindset, which echoes my “treat it like a person” principle of AI, can significantly improve your understanding of how and when to use AI in a practical, if not technical, sense.

The patterns I've encountered are:

Inline suggestion
Single-agent chat
Multi-agent chat
Prompting and nudging controls

Let's look at them in detail...

Pattern: Inline suggestion

While performing a task, relevant ideas are displayed nearby in text or even imagery.

The user might be invited to provide input, such as selecting between variants or filling in a blank. (See Fill-in-the-blanks pattern)

Example: As a product owner enters a story into a task tracking system, the AI suggests edge cases they didn't yet consider.

In the wild: Code assistance plug-ins in IDEs (such as Genie AI, Amazon Q, CodeGPT, Codeium and Llama Coder) and dedicated IDEs (such as Cursor). Email and text message completion, as seen in Gmail, LinkedIn messaging and Apple Messages.

Mimics: 🧍 Colleague, mentor, friend, etc. sitting nearby verbally offering a suggestion and/or physically pointing to a part of the screen.

Pattern: Single-agent chat

A specialised chat bot is available for back-and-forth discussion.

Example:: Customer on an e-commerce website starts entering a question about a product. A specialised chat-bot replies with a detailed response. An input box allows the customer to respond with a follow-up question.

In the wild: "Have a question?" feature on Hotels, "QnaBot" on Amazon.

Mimics: 💬 Colleague, mentor, customer service, etc. communicating with the user via chat.

Pattern: Multi-agent chat

Multiple chat bots appear in the same chat window. Each bot has a different persona and perspective, and only contributes where applicable.

By splitting AI responses among multiple bots, rather than just one, it's easier for the user to mentally divide the AI output into different "buckets".

Also, because this is analagous to real human-human team-work, it's intuitive for people.

Users can address individual bots by name, to ask for further assistance on a specific topic or aspect covered by just that bot.

Example: Financial advisor tool for recommending products to customers. Agents representing analysts and compliance each offer a perspective. The advisor uses these insights to prepare for a meeting with the client.

In the wild: Slack automation bots such as Trello for Slack, ThreadReaderApp on Twitter and dedicated chat platforms such as Symphony.

Mimics: 👭 Group of people working together, such as a team meeting.

Pattern: Fill-in-the-blanks

A stencil is displayed, with some areas for user input and some ares for AI generated content.

As the user fills in the inputs, the AI uses contextual information to generate more of the content. User and AI both work together until the full output has been generated.

Example: Writing a CV for a job. You start to fill in work history items. The AI suggests additional bullet points, which you accept or refuse. The AI suggests shorter more focussed descriptions and word removal, which you accept or refuse.

Mimics: 📈 Collaborative white-boarding with colleagues (virtually or physically), collaborative card sorting exercises with a team.

Pattern: Nudging controls

A "work in progress" is displayed in the center while command-buttons for "nudging" are displayed around the edges or off to the side. By clicking the buttons, you can ask the AI to change the work along some dimension.

Example: 3D image manipulation. We ask the AI to make the shape more or less rounded, more or less flat, etc.

In the wild: Dall-E image generator.

Mimics: 💺 Pairing with a designer, where the designer is tweaking this or that based on your input.

Structured manual testing - benefits and techniques

Jonathan — Fri, 13 Sep 2024 12:10:29 +0000

While automated testing methods have been established for a long time in the software development process (e.g. unit, integration and end-to-end tests), relatively less attention has been paid to manual testing.

However manual testing is far from "dead". Software developers still routinely verify their work by using products manually. Further, developers are usually required to take responsibility for the end-to-end functioning of their software, not just writing quality code and passing unit tests. They are usually encouraged not lean too heavily on QA teams. (In fact, some companies don't even have QA teams!)

In this article, I will:

Review the distinct benefits of manual testing.
Present a real-life scenario where manual testing adds value.
Provide guidance for structuring your manual testing, so you can get maximum benefit from it.

Benefits of manual testing

Manual testing allows you to achieve certain specific goals which may not be available through automated testing:

Discover actual behaviour – how the system currently behaves at runtime. (This information is not always readily available by other means.)
Determine intended behaviour – how the system is expected to behave. (Also not always readily available other than by manually testing the system.)
Perform testing immediately – without setting up test frameworks, etc.
Perform testing in any environment you can access – not just local or QA environments.
Put yourself in the end-user's shoes – experience the system as they would.
Verify complex, lengthy workflows – which might be too difficult to automate. For example, complex interactions between an end-user and the system or complex data processing activities on the backend.

As with any activity, manual testing can offer maximal benefit when performed in a structured manner.

In my experience this involves:

Writing clear, structured test cases - e.g. heading, description, numbered steps with actions and expected results
Organising test cases for rapid retrieval, by using consistent naming and tags
Sharing test cases within the team/organisation, so others can leverage them

Example of a test case

Here's a simple example of a test case involving a user logging in:

user_can_login.md:

# User can login

Users who have an account should be able to log in.

## Steps

1. Go to the homepage
2. Click the login button
3. Expect that the login screen is shown
4. Enter username
5. Enter password
6. Click login screen submit button
7. Expect that you are shown logged in, in the header section

Notice we have a brief heading and description, followed by neatly numbered steps.

Steps can be:

Actions (e.g. "click the login button") or
Expectations (e.g. "expect that the login screen is shown")

This format allows us to quickly follow the steps of the test case (actions) and know what to look at to determine whether the test passed or failed (expectations).

Scenario: critical fixes for a startup

A realistic scenario might make it easier for you to see how manual testing can help.

Imagine you begin work as a software engineer at a rapidly growing startup, building a complex product with many user flows.

You are assigned to work on the sign up experience. Users provide various personal details such as their country of residence. Based on these, the system provides various prompts then accepts payment.

You are given your first development task:

"Please fix the flow for Japanese customers. They are getting stuck at the point where they submit their personal details, but before they have paid for the product."

This is based on direct customer contact. No one in the company can tell you exactly what "stuck" means or in exactly which part of the flow this is occurring.

There is also minimal unit test code, code quality is not good and there's little documentation. Remember, it's a fast-growth startup – they don't have the same time and resources as a more mature company.

How would you go about solving this? Your approach might look like this:

You go through the flow manually, simulating a Japanese customer (perhaps setting your browser location to Japan).
As you go, you write down the steps you are taking, such as which details you entered, which buttons you clicked, etc. (This makes it easier to keep track of what you're doing, in case you need to restart the process).
You find the exact point where the system is stuck - the submit button on screen 5 doesn't do anything.
Examining the requests/responses, you discover that the system skipped the collection of the user's driver licence details, which are required for customers in certain countries, including Japan, causing an underlying API call to fail if not provided.
Your verify this requirement with Backend engineers and the Product owner. Now you know what the fix is: you need to enable drivers licence details collection for Japanese customers.
You make the fix in the relevant part of the code-base.
Testing your work manually, you realise this data can be collected earlier in the sign-up flow, with a skip option given for customers who don't have the details on hand. This will be a nicer user experience and increase the number of potential customers in the sales funnel.
You complete all your changes, cover them with a unit test, save your manual testing steps in a markdown document (linked to from the pull-request) and push your changes.
Once in Prod, you do a quick verification and see that everything works as expected.
You can now report to the team that your task is completed with (hopefully) zero bugs!

Notice how documented manual testing helped you to solve this problem:

You found the actual error by manually going through the flow (step 1).
You kept track of your testing by writing down the steps, allowing you to quickly and efficiently repeat your test efforts whenever needed (steps 2, 7, 9).
You easily verified your work in Prod (step 9).
You empathised with the end-user and even found an opportunity to improve their experience as well as the onboarding rate (step 7).
You added value to the team by documenting your manual testing steps (step 8).

As we'll soon see, this is only the beginning of the benefits!

Tagging your test cases

Tagging can be a powerful way of making your whole test case collection searchable.

Suppose every time you refer to the login screen in your Markdown files, you use the exact phrase: "login screen". Perhaps wrap it in brackets: "(login scren)".

Now this exact phrase is searchable, via a simple find-in-files in your text editor. By searching for the string "(login screen)" you can find every test case involving that screen.

For example, your search might yield the following results:

user_can_login.md
user_can_recover_forgotten_password.md
user_cannot_login_with_wrong_credentials.md
user_can_login_from_another_country.md
user_can_login_with_a_linked_google_account.md

This gives you powerful new capabilities such as:

Regression-testing - checking various test cases, in case your change might have broken something.
Exploratory-testing - observing how the application behaves in various scenarios, generating ideas for improvement or uncovering hidden bugs.
Determining which unit tests to write - to boost test coverage in a critical area of the application.

Test data

Suppose a feature you want to test relies on certain data existing in the system beforehand.

For example, you might need a certain kind of user account, such as a user who has their country set to Japan.

You could create a test user in your testing environment - hiroshi@yompail – and save it in your test case under a "Test data" heading.

user_can_login.md:

# User can login

## Steps

1. Go to the homepage
...

## Test data

- User: hiroshi@yopmail.com / P@ssw0rd

Results and artifacts

It can be very useful to know the full list of dates/times when you ran your test and what the result was on each run.

These can be added to a "runs" section of the test case file.

user_can_login.md:

# User can login

## Steps

1. Go to the homepage
...

## Runs

| Date/time               | Result    |
| ----------------------- | --------- |
| 2024-10-01 9:00 AM      | Succeeded |
| 2024-09-04 10:00 AM     | Failed    |

How might this be useful?

Spotting a pattern in failures can indicate a systemic problem, such as insufficient compute resources or code quality issues with a particular part of the code base.
Correlating failures with code changes can narrow your version control system search. For example, if you know the failure happened within the last week, you can limit your search changes made within that timeframe.

When with manual testing, it is common for engineers to capture artifacts of their work, such as screenshots, screen recordings and copies of log output. These serve to demonstrate work done, prove that things worked correctly at a certain date/time and capture additional information that could help identify additional problems or improvement opportunities.

Artifacts from manual tests can be organised alongside test cases, using a structured folder naming system.

I have found it best to keep artifacts in folders named after the test cases and test run dates from which they were generated.

Here's an example:

/test_cases
- user_can_login.md
- user_can_recover_forgotten_password.md
- ... etc
/test_artifacts
- /user_can_login
- /2024_10_01_9_00_AM
  - Screen Recording 2024-10-01 at 9.01.55 am.mov
  - Untitled2.png
- /user_can_recover_forgotten_password
- ... etc

Manual testing workflow

You can make manual testing a regular, consistent part of your workflow. As you strengthen this habit, your work quality and overall knowledge of the system should improve.

Here are some ideas:

Write a test case at the beginning of working on a major feature.
Include or link to the test case in the task tracking system.
Perform a test case of every major feature you deliver, writing a test case if one doesn't already exist.
Include or link to your test case from every pull request you submit.
Keep all your test cases in a shared knowledge system, such as your project's wiki.
Copy relevant parts of a test cases in chat conversations about a feature or bug.

Manual testing tools

There are a range of software tools to help you write and manage test cases.

Testmatic. Shamless plug – I built this! It includes a web-based UI and CLI and saves everything to Markdown.
Azure Test Plans. This one has a nice web-based UI and integrates with the Azure suite.
DoesQA. An interesting product that apparently allows you to write "codeless" but runnable tests.

Conclusion

Manual testing offers distinct and powerful benefits, not offered by automated testing, such as understanding and representing current and desired system behaviour, making fast progress in challenging environments with limited documentation and test coverage, verifying changes in multiple environments, verifying complex workflows and empathising with end-users.

Structuring your manual test efforts compounds these benefits: you can quickly locate related tests (enabling regression and exploratory testing), ease your test efforts (using test data) and keep track of test results (helping you identify patterns in failures or find the root cause of an issue).

Streamlining code reviews

Jonathan — Tue, 08 Aug 2023 04:58:00 +0000

For myself and others, I've found that code reviews can be a challenging part of being a software developer.

There may be many code changes to go through, many possible mistakes to be found and limited time to find and communicate them. Even before considering mistakes, the sheer amount and complexity of the code and changes can feel overwhelming – you don't know where to begin. It often feels like code review is chaotic and random, and it's down to luck whether or not it will be constructive.

In this article I want to share my code review process, which I've evolved over time, in response to the pressures mentioned above.

These are not necessarily based on hard evidence such as data and statistics, but more of a grab-bag of potentially useful ideas sampled from a range of different work environments, development stacks and teams, over many years.

Broadly, there are three practices I follow, in order, for each code review:

Context – gain a high level understanding of the context surrounding the change, by checking the description, commit messages, task and any other documentation, and/or by asking the author
Scan – scan all of the code in the change, observe any questions or issues that come to mind, try to answer them on my own, otherwise leave a comment
Checklist – make a final pass of all the code, this time against a checklist, and call out minor and major issues

Let me go into a bit more detail on how I perform each of these.

Context

It's difficult to meaningfully assess a code change without understanding its context.

Context includes: what problem or requirement the code change intends to solve and how it fits into the broader context of the system.

Without context, the code may either look superficially correct, leading you to approve it too quickly, or it may look very odd, generating innumerable questions to the author. Conversely, when you do understand the intention and context, a code change becomes much easier to understand and more intelligible.

So I've found it useful to learn more about the context if I'm unclear.

Here are some methods I use to learn more about the context:

Read descriptive notes and commit messages for the change, if any.
Look up the task associated with the code change and read description and comments there.
Look up code files related to the change and examine the code there.
Look up the history of the files associated with the change, look up the tasks associated with that history, read the descriptions and comments.
Search internal information sources (chat channels, wiki, etc.) using keywords found in the code.
Ask the author of the change directly for context.
Look up the authors of the files associated with the change in version control history and ask them directly for context.

While the above might seem time-consuming, I've found it possible to fairly quickly improve my knowledge, even within minutes, by just picking a few of the relevant methods and applying them.

For example, if I already work closely with the author involved, a simple message asking for context often gets a quick reply. Or if the code changes are attached to tasks, e.g. via task codes, it's usually possible to access the task with just a couple of clicks, and then read it within a few minutes.

I believe this "context hunting" is usually worth the effort. It's not only about understanding the code change you're looking at. The benefits of contextual knowledge compound over time. Initially you might spend, say, 10 minutes reading and digesting contextual information, but eventually the time spend can approach 0, as you develop a systematic understanding of the whole system. That systematic understanding is highly valuable in all kinds of ways, not only for code review. It can help you to succeed in your own projects within the organisation and even help you to make the case for new projects and initiatives.

Scan

After gaining context, the next step is to scan the code, getting a "big picture" view of how it hangs together.

I'll often check out the change locally, open some of the files in my IDE, and use the navigation tools in the IDE to figure out what sequence of calls are being made, what data are being passed, etc. If it's a complex network of calls, I might spend a few minutes sketching an execution flowchart.

I also survey the content of new code added, looking at key variables, control-flow structures (conditionals, switches, loops, etc), and getting an overall grasp of what the code does.

At this point I may already have questions or issues for the author, and if so, I won't hesistate to leave a few comments.

My comments will usually be in the following form:

[Priority]: (Message)

This helps the author to understand my intent and prioritise which comments to reply to.

For example, if it's a minor issue, which shouldn't necessarily block merging, I'll write something like this:

[Minor]: `.forEach(expandSection)` might be more concise here.

But if it's a question, I'll write:

[Question]: Should this section be hidden for users without permissions?

And if it's a major issue, I'll write:

[Major]: Should include an authorization check here.

Checklist

By this point I'll have a pretty solid understanding of the change.

Now it's a good time to run through a code review checklist and see if I missed anything significant.

Because I broadly understand the change as a whole, even with a large checklist of 50 items, it's possible to quickly scan the checklist and pick out only the items that apply to the code.

For example, suppose one of the items in my checklist is:

Query keys should be appropriately unique

After scanning the change, I will already know whether or not the change includes any query keys at all. If it does not, then I can immediately skip this step.

On the other hand, if the change does contain enums, then I will know to check the following item in my checklist:

Enum values should match keys

Where does this checklist come from?

I usually build a unique checklist for each project I work on. As initial inputs to the checklist, I analyse the codebase I'm working on and read any technical documentation, such as coding standards.

Subsequently I will add new items to the checklist, based on comments others leave on my change submissions, technical discussions with team members and general observations.

Additionally, I've built up a pool of coding standards and best practices over my time as a developer. Some of these you can find documented in my article, Towards zero bugs. I plan to publish a comprehensive list of them in a future blog post.

This checklist isn't only useful for reviewing others' work – I use it on my own changes as well. By anticipating feedback and addressing it earlier, my code will already be of higher quality by the time it reaches the screens of others. This reduces the review workload on other engineers and improves my reputation within the team.

Conclusion

Code reviews are an integral part of modern software engineering.

At a team level, they're a great way to maximise code quality and ensure a shared understanding and knowledge of the code and systems.

At an individual level, they're useful for understanding as much of the code as possible, both at a high level and a detailed level. This improves the quality of my own work and increases the likelihood of success in my current work and new initiatives within the organisation.

Having a normalised process for performing code reviews helps make them easier and more fun. It also improves the quality of the feedback and, long-term, the code base.

Should we run unit tests in the cloud?

Jonathan — Sat, 17 Jun 2023 14:37:07 +0000

Imagine if you could push your current branch to a test server and have it automatically run all of your unit tests!

You could get feedback on your entire test suite in a few minutes, rather than waiting for a lengthier CI process, which might take up to ~20 minutes (depending on the infrastructure and setup in your organisation).

Several automation test platforms now exist, to perform cloud testing on longer-running end-to-end tests. These include Selenium Grid, TestingBot and BrowserStack.

Why not extend this kind of service also to unit tests?

Advantages of cloud-based unit test execution over CI:

CI can be time-consuming, if it runs a full build, linting, type-checking and other operations. Cloud-based unit testing lets us isolate just the tests.
CI tends to be monitored on a separate web-page. Cloud-based unit testing (in theory) could be integrated directly into the developer's IDE, so it would feel similar to running the tests locally.

Advantages of cloud-based unit test execution over local:

When working with a very large code base, with a large number of tests, and not well de-coupled, we might want to run all the tests. An optimised cloud environment might be able to do this faster than a dev box, and without consuming resources which could cause our IDE and other tools to slow down.

So far I don't know of any platforms that actually do this, but perhaps the aforementioned automation testing platforms could be configured/tweaked to enable it.

What do you think?

Visualising execution flows

Jonathan — Sat, 10 Jun 2023 15:12:38 +0000

Anyone who has spent some time developing software knows writing new code is but a small part of the job. At least as big, perhaps bigger, is understanding the existing code. And that includes understanding the runtime behaviour of that code!

I often found myself having to understand a complex cluster of code modules, entailing many function calls being made and many data types being passed and returned.

To properly understand the behaviour of the code, I needed to see a whole flow together at once so I could reason about it. I needed to somehow visualise it, e.g. by listing out the function calls in a text editor or maybe drawing a diagram on a piece of paper or in a diagramming application.

After doing this quite a few times, I have started to evolve a more consistent and powerful format, one which is text-based (and so, easy to work on in a standard text editor) but can also be converted to a visual flowchart using a tool called Mermaid.

In this article I want to describe this format and the reasoning behind it.

Execution flow notations can be useful in understanding an existing code-base, troubleshooting bugs, communicating with other team members and for solution design.

But first some background...

What do I mean by "execution flow"?

It's helpful to define the concept of "execution flow".

I'm referring to the path that the runtime will take through the code as it executes the code during a real-life use case.

You should not confuse this with a more specific term: call stack. Since a flow can include multiple function calls in sequence, each producing its own distinct call stack, a flow can include multiple call stacks. Much of the complexity of an execution flow is precisely that calling of multiple functions and the passing of data to them and returning of data from them. So "call stack" is too narrow a term to cover what I'm trying to describe.

On the other hand, you should also not confuse this with a more general term such as abstract syntax tree or "code structure". We are not describing the code as a whole, but just one path of possible execution of the code. Any piece of code that has one or more conditionals (e.g. if or switch statement, etc.) will execute differently depending on how those conditionals evaluate. For the same code, different lines might execute depending on the situation (e.g. depending on external state of some kind such as a database, web-service, system clock, etc.). Thus one code base can support multiple execution flows.

Example of an execution flow

Let's use a hypothetical example – handling a user login on a Java backend.

1 class LoginResource {
2   ...
3   private Response login(String username, String password) {
4     if (this.userAuthProvider.isValidUser(username, password)) {
5       this.sessionProvider.setCurrentUser(username);
6       return new Response(200, "Login succeeded.");
7     } else {
8       return new Response(401, "User credentials are invalid.");
9     }
10   }
11 }

Can you spot the two execution flows in this code?

Flow 1 - Logging in successfully

First, we have flow when the user's credentials are valid - that's lines 4-6:

4     if (this.userAuthProvider.isValidUser(username, password)) {
5       this.sessionProvider.setCurrentUser(username);
6       return new Response(200, "Login succeeded.");

Inside login(), the if condition calls UserAuthProvider::checkUserCredentials, passing user credentials.
UserAuthProvider::checkUserCredentials returns true.
Execution proceeds into the then block.
We call SessionProvider::setCurrentUser, passing user credentials.
We return Response, passing success parameters.

Notice that this isn't just a single call-stack, as there are actually two method calls in this flow, each of which will generate its own call stack.

UserAuthProvider::checkUserCredentials
SessionProvider::setCurrentUser

Flow 2 - Failure to log in

What if the user credentials are not valid and isValidUser returns false?

That would be a separate execution flow - lines 4 and 7-8:

4     if (this.userAuthProvider.isValidUser(username, password)) {
...
7     } else {
8       return new Response(401, "User credentials are invalid.");

Inside login(), the if condition calls UserAuthProvider::checkUserCredentials, passing user credentials.
UserAuthProvider::checkUserCredentials returns false.
Execution proceeds into the else block.
We return Response, passing failure parameters.

IDE tools for determining execution flow

So how to we figure out how our code flows in the first place?

We can, of course, just read the code, open various files as needed, and try to follow along in our head.

Thankfully we also have automated tools to help reduce some of the tedium. You'll likely be familiar with these:

Go to definition - we can select a reference (function, class, variable, etc) and be taken to its original definition
Find references - we can select a definition (function, class, variable, etc) and pull up a list of all points in the codebase which reference the definition

Different IDEs name these differently, but most mainstream IDEs have them in one form or another, including IntelliJ IDEA, VSCode, Visual Studio and xCode.

Screencast of a developer using Go to definition tool in IDEA

For example, in the code sample given previously, we might use Go to definition to locate the class whose login method is being called.

Go to the LoginResource class and its login method.
Right-click the isValidUser call and select "Go to definition".
Observe that it is defined in the UserAuthProvider class and its isValidUser method.
Go back the LoginResource class and its login method.
Right-click the setCurrentUser call and select "Go to definition".
Observe that it is defined in the SessionProvider class and its setCurrentUser method.
Go back the LoginResource class and its login method.
Observe that a new Response object is constructed.

We might want to find out where else the UserAuthProvider::isValidUser method is called.

Supposing there was a RegisterResource class having a register method, as shown below:


class RegisterResource {
  ...
  private Response register(String username, String password) {
    if (this.userAuthProvider.isValidUser(username, password)) {
      this.sessionProvider.setCurrentUser(username);
      return new Response(200, "Login succeeded.");
    } else {
      this.userAuthProvider.registerUser(username, password);
    }
  }
}

Then we might locate this piece of code by using the Find references tool:

Go to the UserAuthProvider class and its isValidUser method.
Right-click the login method and select "Find usages".
Observe that it is called in the LoginResource class, in its login method.
Observe that it is also called in the LoginResource class, in its register method.
Observe that a new Response object is constructed.

Describing execution flows with text

Suppose we wanted to make some notes of the execution flows we discovered. Maybe there are too many for us to easily memorise. Perhaps we want to see them all in one view rather than scattered among many files.

Let's start with the first flow – successful login:

LoginResource::login
  --->|userName,password| UserAuthProvider::isValidUser
  ---|true| LoginResource::login

  --->|userName,password| SessionProvider::setCurrentUser
  --- LoginResource::login

  --->|200,'Login succeeded.'| Response::constructor

Then the second flow – successful login:

LoginResource::login

  --->|userName,password| UserAuthProvider::isValidUser
  ---|false| LoginResource::login

  --->|401,'Login failed. Invalid credentials.'| Response::constructor

And the final flow – register:

LoginResource::register
  --->|userName,password| UserAuthProvider::isValidUser
  ---|true| LoginResource::register

  --->|200,'Login succeeded.'| Response::constructor

Now we can step back and look at all these flows together and see the bigger picture, e.g. how login and register both check user validity using UserAuthProvider, and how both instantiate the Response class with various constructor parameters.

Aside: Sketching execution flows

Observe that we don't have to cover the flows exhaustively, nor do we have to limit our coverage. We can cover just the parts of code that we are concerned with, based on our current goal, e.g. to solve the current bug or to learn more about a specific part of the code-base. We can make a kind of "partial sketch" of the parts of the execution flows that interest us.

For example, we don't cover how login and register are called, and we don't cover which further calls are made by UserAuthProvider, SessionProvider or Response, if those parts of the code don't interest us.

And if we are dealing with a bug in which isValidUser incorrectly returns false, we can focus more on isValidUser and which methods it calls.

Format for notating execution flows

Did you notice the textual format used in the previous section to notate the execution flows?

Let's deep-dive into that.

Class::methodCalling
  --->|parameters| Class::methodBeingCalled
  ---|return values| Class::methodCalling

Class::methodCalling - the caller
--->|parameters| - execution flowing from caller to callee, with the parameters being passed in the call
Class::methodBeingCalled - the callee
---|return values| - execution flowing from callee back to caller, the value returned from the callee
Class::methodCalling - the caller (again)

We can chain these together to notate a sequence of consecutive calls.

For example:

Class1::method
  --->|parameters| Class2::method
  --->|parameters| Class3::method
  ---|return values| Class2::method
  ---|return values| Class1::method

Representing closures and indirect calls

Thusfar we've use the Class::method format to reference the callers and callees. This should work reasonably well for classical OO code-bases written in Java, C#, Swift etc.

But what if we want to reference code in other ways, such as named closures, for languages written in Javascript, Typescript, etc.?

Here are some notations that could allow such structures to be referenced:

Nested closure

foo/bar - Reference a closure witin another closure.

For example: retry/handleTimeout references a handleTimeout function nested inside a retry function in Javascript.

Indirect call

-.-> - References an indirect method call - a call which our code doesn't make directly, but causes to be made, such as calling setTimeout on a function in Javascript. It looks like a dotted line.

Example in Javascript

Let's use an example – a recursive Javascript function – to put all these ideas together.

function retry(action, times, count = 1) {
  const timeout = Math.pow(count, 2) * 1000;
  setTimeout(function handleTimeout() {
    if (!action() && count <= times) {
      retry(action, times, count + 1);
    }
  }, timeout);
}

retry is a recursive function, which calls setTimeout, passing a closure. That closure executes. Depending on the number of times retry has called itself already (time), it may call retry again or simply do nothing, halting the recursion.

We can notate this execution flow, including the closure, using the nested closure, multiple calls, row/column and indirect call notations given above, in the following manner:

retry
  -.->|action, times=3| retry/handleTimeout
  ---> action
  ---|false| retry/handleTimeout
  --->|action, times=3, 2| retry
  -.->|action, times=3, count=2| retry/handleTimeout
  ---> action
  ---|true| retry/handleTimeout
  ---retry

Visualising flows with Mermaid

Now the juicy part – lets look at how this format can be instantly converted into a visual flowchat using Mermaid!

Mermaid is a free, open-source tool, which takes code written in a specific syntax and converts it into a diagram.

You can run Mermaid in the browser using Mermaid Live, or if you prefer, you can download and run it locally using the instructions on the mermaid-live-editor GitHub profile.

Note: We'll need to add the keyword graph to the top of the text.

Also, in these examples, we add numbered circular nodes (e.g. ---n1((1))) to indicate the order of execution.

The following is how our two earlier Java examples – login success and login failure – render in Mermaid:

And here's the Javascript example:

Javascript recursive function example rendered by Mermaid

Notice that we've added small numbered circles, indicating the order in which the calls occur. This makes the flow a bit easier to navigate.

Imagine this appearing in a Slack conversation.

It could potentially be easier to read and follow an execution flow diagram than to read paragraphs of text trying to describe in plain language the complex sequence of calls.

Aside: Asynchronicity and concurrency

Though we touched on async in the Javascript example with the setTimeout call, we haven't fully addressed the issue of describing asynchronicity or concurrency in execution flows.

This is probably a fairly deep topic that deserves a dedicated article. However I have no doubt it can be represented diagrammatically, as long as a strict convention is adhered to.

Isn't this just a flowchart?

Yes, but it's a specialised form of flowchart, focussed on representing execution flow.

The flowchart directly maps to the code it represents, so it accurately and unambiguously conveys information about that code. At the same time, because it's not actually code, but a diagram, it allows us to more easily view and reason about the code in terms of execution flows specifically. We don't have to jump around between files, scroll up and down, etc. but can see a whole execution flow in one screen.

Also by establishing and adhering to a convention in how we represent callers, callees, parameters and return values, etc. this flowchart technique is re-usable across programming languages, codebases, business domains, etc. A similar versatility is found in UML, sequence diagrams and other kinds of specialised diagram formats.

Why not use sequence diagrams?

You might have seen diagrams similar to those described here, but laid out as sequence diagrams. Execution flows can certainly be visualised as sequence diagrams. A sequence diagram is arranged as a set of vertical columns connected by arrows, where each column represents a method and each arrow represents a call.

There are weaknesses of sequence diagrams, however.

They present each method in a column, so we may soon run out of horizontal space, whereas flowcharts can flow down and across. Also, even for lengthy flowcharts, scrolling up and down is easier on most devices than scrolling side-ways.
They may position the caller and the callee very far apart, so that the eye has to scan back and forth over a large distance to see the call, whereas flowcharts can more position the caller and callee closer together, making scanning easier.

For these reasons, I find the flowchart format more appealing.

Are there tools that generate execution flowcharts automatically?

Surprisingly, not really.

For dependency visualisation, I found a few interesting plugins for VSCode, and also experimented with IDEA's dependencies analysis tool.

However, all of these tools are focused on reporting compile-time dependencies, which are a different kind of thing to execution flows.

Dependency graphs of course help us to understand how code is structured, but they don't give us the full picture of which parts of that code execute in which order at runtime. For that, we really need execution flows.

Theoretically any tool that could automatically report execution flows would need to be able to analyse the code in terms of its expected execution at runtime. The tool might, like a debugger, execute the code, in order to determine the flow of control, e.g. where the flow of control depends on some state which can only be discovered at runtime. Or it could statically analyse the code to determine all possible flows and generate a report of all of them.

It's beyond the scope of this article to look into how such a tool could be developed, but it's something I'm interested in looking into and perhaps even undertaking myself.

Update on ChatGPT (11/06/2023)

Experimentation with ChatGPT yielded promising results.

The LLM (Large Language Model) tool was able to generate a flowchart with labels in plain-English in both ASCII and Mermaid formats.

The flowchard did accurately follow the flow of the code. However, it did not use the format I described above, which is intended to directly map to elements in the code (function names, variable names, etc).

With some more training of ChatGPT, more detailed prompts or a more customised LLM tool than ChatGPT, perhaps it will be possible in the near future for a chat-bot to generate execution flows automatically. That would be cool!

Conclusion

This article has outlined a format for describing execution flow, which can be used to visualise and understand how parts of a codebase execute at runtime (and generate diagrams).

This understanding can help to diagnose bugs/errors, determine the best points at which to change the code, estimate how long changes might take, and no doubt many other use cases.

I hope you find it useful!

Increase your test coverage with combinatorial testing

Jonathan — Sat, 15 Oct 2022 09:24:07 +0000

Have you recently tried to unit test a function which has many combinations of possible inputs and expected outputs?

An increasingly common way of writing such a test is to utilize a data-driven test. The problem with data-driven tests is that they can quickly grow to be large and unwieldy.

In this article, I want to introduce a technique for generating data-driven tests without having to spell out every individual combination of inputs/outputs in code.

But first, a quick refresher on data-driven tests...

Data-driven tests and their limitations

You may be familiar with data-driven testing. Basically you write a table of combinations of inputs and outputs in which each test case is inputted as "row" of data. Data-driven testing is now supported by many popular test frameworks (Jest, JUnit, NUnit to name a few).

One problem with data-driven testing is: sometimes we have so many combinations to cover that a comprehensive data-driven test would be very lengthy and difficult to read or maintain.

Imagine, for example, trying to write a data-driven unit test for a function which returns the number of days in a given month.

The function takes two parameters: year and month and returns one day value. It has to deal with a range of values for each parameter. Multiplying all values that need to be tested by two parameters yields a large number of combinations.

it.each([
  { year: 2021, month: "jan", expectedDays: 31 },
  { year: 2021, month: "feb", expectedDays: 28 },
  { year: 2021, month: "march", expectedDays: 31 },
  { year: 2021, month: "april", expectedDays: 30 },

  // and so on and so on ... 😓
]);

That number of combinations, though easy for a computer to process, is not easy for us to wrap our human minds around!

Perhaps the solution is to express the combinations in a more concise manner – as grouped ranges of values – rather than spelling out every single combination.

Combinator function to the rescue!

What is a combinator?

In the world of functional programming, the term "combinator" informally refers to a pattern...

"where complex structures are built by defining a small set of very simple 'primitives', and a set of 'combinators' for combining them into more complicated structures"

– Combinator Pattern • wiki.haskell.org/Combinator_pattern

The combinator I present in this article is more specific. It takes as input an object whose properties each have a value that is an array. Then it combines each value of each array. All of the objects generated by this means are then returned to the caller.

For example, suppose we provide an input object having a single property, "color", whose value is an array containing elements "red" and "blue":

{
  color: ["red", "blue"];
}

The combinator will return us an array having the following objects:

an object having a "color" property whose value is "red" and
an object having a "color" property whose value is "blue"

[
  {
    color: "red",
  },
  {
    color: "blue",
  },
];

Suppose we provide an additional property in our input object, "brightness", whose value is an array containing elements 100 and 200:

{
  color: ['red', 'blue'],
  brightness: [100, 200]
}

The combinator will return us an array having every specified combination of "color" and "brightness":

an object having a "color" property whose value is "red" and a property "brightness" whose value is 100 and
an object having a "color" property whose value is "red" and a property "brightness" whose value is 200 and
an object having a "color" property whose value is "blue" and a property "brightness" whose value is 100 and
an object having a "color" property whose value is "blue" and a property "brightness" whose value is 200

Like this:

[
  {
    color: "red",
    brightness: 100,
  },
  {
    color: "red",
    brightness: 200,
  },
  {
    color: "blue",
    brightness: 100,
  },
  {
    color: "blue",
    brightness: 200,
  },
];

Providing a definition object as input, we can get a large set of results as output.

Here's a high-level diagram:

Let's apply this combinator to a slightly more "real world" example.

Applying a combinator to an example: days in a month

For historical reasons, determining the number of days in a month in the Western calendar is complicated.

The following short rhyme tries to summarize the rules in a memorable way:

Thirty days have September,

April, June and November.

All the rest have thirty-one,

except February alone, which has

twenty-eight days each year

and twenty-nine days each leap-year

Suppose we wanted to unit-test a function, getDaysInMonth, which takes month and year as input and returns a number of days.

We could simply input every possible date into the unit test and assert on the month of each. As mentioned above, that could involve quite a lot of fiddling in Excel and would result in a very long and not very human-readable test file.

Instead, let's try to tackle this problem with a combinator.

Starting with the first two lines of the rhyme:

Thirty days have September,

April, June and November.

We can express this "thirty days" combination set programmatically, like this:

const thirtyDays = combinate({
  year: range(2020, 2023),
  month: ["april", "june", "september", "november"],
  expectedDays: [30],
});

The result can easily be passed into a data-driven test in Jest:

it.each(thirtyDays)(
  "$month in $year should have $expectedDays days",
  ({ month, year, expectedDays }) => {
    expect(getDaysInMonth(month, year)).toBe(expectedDays);
  }
);

On running the unit test, the following test cases will be generated and executed:

✓ april in 2020 should have 30 days (3 ms)
✓ june in 2020 should have 30 days
✓ september in 2020 should have 30 days
✓ november in 2020 should have 30 days
✓ april in 2021 should have 30 days
✓ june in 2021 should have 30 days (1 ms)
✓ september in 2021 should have 30 days
✓ november in 2021 should have 30 days (1 ms)
✓ april in 2022 should have 30 days
✓ june in 2022 should have 30 days
✓ september in 2022 should have 30 days
✓ november in 2022 should have 30 days
✓ april in 2023 should have 30 days
✓ june in 2023 should have 30 days
✓ september in 2023 should have 30 days
✓ november in 2023 should have 30 days

Notice how we can use a small amount of code (in this example, 5 lines for the combinate call) to generate a much larger set of test cases (16). This gives our test code more leverage.

Covering the remaining lines of the rhyme:

All the rest have thirty-one,

const thirtyOneDays = combinate({
  year: range(2020, 2023),
  month: ["january", "march", "may", "july", "august", "october", "december"],
  expectedDays: [31],
});

The following data will be generated:

✓ january in 2020 should have 31 days (2 ms)
✓ march in 2020 should have 31 days (1 ms)
✓ may in 2020 should have 31 days (1 ms)
✓ july in 2020 should have 31 days (1 ms)
✓ august in 2020 should have 31 days
✓ october in 2020 should have 31 days
... etc ...

except February alone, which has
twenty-eight days each year

const februaryDays = combinate({
  year: [2023],
  month: ["february"],
  expectedDays: [28],
});

✓ february in 2023 should have 28 days (2 ms)

and twenty-nine days each leap-year

const februaryLeapYearDays = combinate({
  year: [2024],
  month: ["february"],
  expectedDays: [29],
});

✓ february in 2024 should have 29 days (2 ms)

Finally, putting it all together, here is the complete unit test:

describe("getDaysInMonth", () => {
  const thirtyDays = combinate({
    year: range(2020, 2023),
    month: ["april", "june", "september", "november"],
    expectedDays: [30],
  });

  const thirtyOneDays = combinate({
    year: range(2020, 2023),
    month: ["january", "march", "may", "july", "august", "october", "december"],
    expectedDays: [31],
  });

  const twentyEightDays = combinate({
    year: [2023],
    month: ["february"],
    expectedDays: [28],
  });

  const twentyNineDays = combinate({
    year: [2024],
    month: ["february"],
    expectedDays: [29],
  });

  it.each([
    ...thirtyDays,
    ...thirtyOneDays,
    ...twentyEightDays,
    ...twentyNineDays,
  ])(
    "$month in $year should have $expectedDays days",
    ({ month, year, expectedDays }) => {
      expect(getDaysInMonth(month, year)).toBe(expectedDays);
    }
  );
});

Notice that we can assign meaningful names to each of the variables, increasing the readability of the test code.

I'm sure you would agree that this test code, using a combinator, is more concise and readable than a large table of numbers and strings!

In closing, I encourage you to use combinatorial testing to shorten and sweeten your data-driven tests, thus testing your software thoroughly and making it maximally robust.

Introducing combinator-util

If you'd like to add a little combinatorial goodness to your unit tests, please check out this re-usable, open-source NPM package:

https://github.com/jonathanconway/combinator

Contributions welcome!

DEV Community: Jonathan

There’s no rule that Agents, Skills, Tools or Commands must be LLM-based. Some use cases require 100% accuracy and determinism – for those cases, we can apply these patterns to regular code.

Dealing with flaky tests

Reproducing flakiness

Running the test repeatedly to generate failures

Virtual machine

Container

Test runner

Observing prior test failures in logs

Diagnosing flakiness

Solving flakiness

Await your elements

Keep your promises

Reduce test size

Reduce test file size

Reduce the number of workers

Clean up at the end

Increase timeouts

Optimise the application under test

Mitigating flakiness

Disable resource-intensive application features when running in tests

Reconfigure test runners

Tag known flaky tests and configure accordingly

Use a different kind of test

Use a different method of verification

Conclusion

Further reading

How I built my designerly CV

Selecting a word processor

Layout design

Implementation

Dancing with content

Finishing touches

Download my template

Generic HOC creator pattern for React

Motivation

Pattern

Example

Advantages

Smaller consumer code

Aesthetically pleasing consumer code

Ability to constrain re-use of general components

Implementation

Diagramming React code

Overview

Component

Component call

Component call with props

Component render props

Function

Type or interface

Example: contacts list

Downloads

AI for developers

Writing code faster with autocomplete

Getting help with error messages

Getting help with technical problems

Writing bash scripts

Generating unit test code

Summarising conversations

Code reviews

Conclusion

Further reading and viewing

AI user interface patterns

Principles

Pattern: Inline suggestion

Pattern: Single-agent chat

Pattern: Multi-agent chat

Pattern: Fill-in-the-blanks

Pattern: Nudging controls

Further reading

Structured manual testing - benefits and techniques

Benefits of manual testing

Example of a test case

Scenario: critical fixes for a startup

Tagging your test cases

Test data

Results and artifacts

Manual testing workflow

Manual testing tools