DEV Community: Jon Ashdown

TDD - Avoiding the Clangers

Jon Ashdown — Wed, 16 Jul 2025 20:31:56 +0000

Test-Driven Development, when done correctly, will provide the following benefits to any development team and organization:

Direction in coding
Documentation
Ongoing proof that software continues to work
Succinct, high-quality code
Fewer bugs
Confidence in the codebase

If done incorrectly, we could have a clanger, which could manifest as a bug in the future.

Basic Theory

Write a failing test, write enough code to make the test pass, refactor, and repeat.

The key thing to note here is the failing test. If the test passes, then either the test is wrong, or the requirement of the test has already been met. At this point, we should check that there isn't an existing test that covers the requirement and make a decision to keep or discard the new test; however, we don't need to write any code.

This can be visualized in the TDD Cycle:

Advanced Theory

There are several levels of testing, most of which can be automated. Whilst unit tests cover the code, higher-level tests might cover business logic, accessibility, or data flows across large systems.

The Pyramid of Testing

The pyramid of testing conceptually arranges these testing levels with manual testing at the top of the pyramid, end-to-end testing (where we test the full system), followed by integration testing (where we test some of the system and mock dependencies between services), and finally, unit testing (where we test the code and mock clients to services).

As we go up the pyramid, any bugs that we find become more expensive. A bug found at the unit-test level might take an hour or so of developer time to fix, whilst bugs found at the very top of the pyramid could result in lost sales, lost trust, or, if we breach GDPR laws, £20 million fines and prison sentences.

Clangers

Write Code with No Tests

Good Luck! This is a very quick way to deliver overly complex solutions, introduce bugs and other tech debt, and have unmaintainable code.

Write Code and Then Write Tests

This is not much better than writing code with no tests. It doesn't really consider either the business logic or the existing code. It only proves that the code that has just been written works, but there is no guarantee.

Often, when this happens, a new (cancelable) piece of work is created to write the tests.

Treat BDD and TDD as Opposing Methods

TDD without BDD proves that the code works and will help keep the codebase simple, but there is no proof that we have built the right thing.

BDD without TDD proves the feature is correct, but debugging will be difficult.

BDD and TDD should be considered complementary, with BDD describing the feature (what) and TDD describing the implementation (how).

Test for the Presence of a Function

// Useless test
test('there is a function called area', () => {
  expect(typeof area).toEqual('function');
});

// Useful test
test('area returns 0 when the first argument is 0', () => {
  expect(area(0, 1)).toEqual(0);
});

If we are extending classes or implementing an interface, then either the IDE, the linter, or any type checker will pick this up.
The tests that we really care about are those that exercise the functionality of area. If the function does not exist or gives an incorrect result, we will get a test fail!

// Useful test
test('area returns 0 when the first argument is 0', () => {
  expect(area(0, 1)).toEqual(0);
});

Tests That Test Themselves

test('area returns 1 when the arguments are both 1', () => {
  const expected = 1;

  const result = 1;
  area(1,1);

  expect(result).toEqual(expected);
})

This is not always easy to spot in review. However, by adopting a test-first approach, we will get a passing test before we write code, which will highlight the problem.

Mocking the Code Under Test

// Production code
export const area = (x: number): number => {
  return 4 * (x * Math.pow(2, 0.5) * 0.25) * (x * Math.pow(2, 0.5) * 0.5)
}

// Tests
test('area() calculates and returns the correct value', () => {
  areaMock = mock(area)
  areaMock.returnValue(4)
  expect(areaMock(2)).to.equal(4)
})

test('area() calculates and returns the correct value', () => {
  areaMock = mock(area)
  areaMock.mockImplementation((x) => {
    return x * x;
  })
  expect(areaMock(2)).to.equal(4)
})

In both the above tests, area has been replaced with mocked functionality, meaning that the function is not actually tested, and any bugs in the implementation might never be found. Again, this is hard to spot in code review, but a test-first approach will uncover this.

// Tests
test('area() calculates and returns the correct value', () => {
  const result = area(2);
  expect(result).to.equal(4);
});

// Production code
export const area = (x: number): number => {
  return 4 * (x * Math.pow(2, 0.5) * 0.25) * (x * Math.pow(2, 0.5) * 0.5)
}

Mocking the Wrong Thing

// Production code
const httpClient = new HttpClient()

export const goAndGetAResource = async () => {
  result = await httpClientWrapper()
  if(!result.success) {
    console.error('we got a 404')
  }
 }

const httpClientWrapper = async (): Promise<{ success: boolean, resource: unknown }> => {
  try {
    const resource = await httpClient.get('https://example.com/randomThing');
    return { success: true, resource };
  } catch (e) {
    return { success: true };
  }
}

// Tests
test('it logs when we get a 404', () => {
 const consoleMock = mock(console)
 httpClientWrapperMock = mock(httpClientWrapper)
 httpClientWrapperMock.resolvedValue({ success: false })
 await goAndGetAResource()
 expect(consoleMock.error).toBeCalledWith('we got a 404')
});

Here, mocking httpClientWrapper is not the correct thing to mock, as it is an implementation detail of goAndGetAResource. This has resulted in unnecessarily complex code, and the mock is actually hiding a bug; httpClientWrapper will never return { success: false }.

The correct thing to mock is the httpClient, in particular, the get method.

// Tests
const getMock = mock();
const HttpClientMock = mock(HttpClient);
HttpClientMock.returnValue({
  get: getMock
})

test('it logs when we get a 404', () => {
 const consoleMock = mock(console)
 const error = new Error();
 error.status = 404;

 getMock.rejectedValue(error);
 await goAndGetAResource();

 expect(consoleMock.error).toBeCalledWith('we got a 404')
});

// Production code
const httpClient = new HttpClient()

export const goAndGetAResource = async () => {
  try {
    result = await httpClient.get('https://example.com/randomThing');
  }
  catch (error) {
    if (error.status === 404) {
      console.error('we got a 404');
    }
  }
 }

Not Testing Side Effects

const renderPersonalData = async (name) => {
  try {
    result = await httpClient.get(`https://example.com/personalData/${name}`)
    // Side effect - where do the logs go, should anything be redacted?
    console.log({result})
    return render(result)
  }
  catch (e) {
    console.error(e)
  }
}

In an ideal world, we should avoid side effects wherever possible. If we must have side effects, then having tests for them indicates that they are intentional and handled correctly. In this example, we should assert that when either console.log or console.error are called, that any sensitive data is redacted.

Not Resetting Mocks

// Production code
const area = (x: number): number => {
  console.log(x);
  result = x * x;
  console.log(result);
  return result;
}

// Tests
const consoleMock = mock(console);

test('It logs the input', () => {
  area(4);
  expect(consoleMock.log).toBeCalledWith(4);
});

test('It logs the result', () => {
  area(2);
  expect(consoleMock.log).toBeCalledWith(4);
});

In this case, we have asserted that console.log has been called with the number 4, but is it the input of the first test or the result of the second test? Is it both?

// Tests
const consoleMock = mock(console);

beforeEach(() => {
  resetAllMocks();
});

test('It logs the input', () => {
  area(4);
  expect(consoleMock.log).toBeCalledWith(4);
});

test('It logs the result', () => {
  area(2);
  expect(consoleMock.log).toBeCalledWith(4);
});

// Production code
const area = (x: number): number => {
  console.log(x);
  result = x * x;
  console.log(result);
  return result;
}

Test Bleed

This is where the outcome of one test affects the next. Whilst not resetting mocks is an example, there are other types of test bleed, such as a shared result object across tests that is being modified.

// Production code
const addUserRole = (user: User, role: string) => {
  user.roles.push(role);
}

const newUser = (name: string):User => ({
  name,
  roles:['reader']
});

// Tests
// Shared state that can be mutated
const user = newUser(Jon);

describe('user management', () => {
  test('should be able to add a role to a user', () => {
    addUserRole('author');
    expect(user.roles).toContain('author');
  });

  test('a new user should only have the reader role', () => {
    expect(user.roles).toEqual(['reader']);
  });
});

The second test will fail as the first test is mutating shared state. This can be identified by running the tests in a random order.

// Tests
describe('user management', () => {
  test('should be able to add a role to a user', () => {
    const user = newUser('Bob');
    const { roles } = addUserRole('author');
    expect(roles).toContain('author');
  });

  test('a new user should only have the reader role', () => {
    const user = newUser('Jon');
    expect(user.roles).toEqual(['reader']);
  });
});

// Production code
const addUserRole = (user: User, role: string): User=> {
  const { roles } = user;
  roles.push(role);
  return {...user, roles}
}

const newUser = (name: string):User => ({
  name,
  roles:['reader']
});

Synchronous Testing of Asynchronous Code

// Production code
const httpClient = new HttpClient()

export const goAndGetAResource = async () => {
  try {
    result = await httpClient.get('https://example.com/randomThing');
  }
  catch (error) {
    if (error.status === 404) {
      console.error('we got a 404');
    }
  }
}
// Tests
test('should fetch a resource', () => {
  const getMock = mock();
  const HttpClientMock = mock(HttpClient);
  HttpClientMock.returnValue({ get: getMock });

  goAndGetAResource();

  expect(getMock).toBeCalledWith('https://example.com/randomThing');
});

What happens here is what usually happens when we call asynchronous functions in a synchronous way. That is, the next synchronous function will be called before the asynchronous function completes. As this is a test, the test run will complete before the assertion is checked.

// Tests
test('should fetch a resource and wait for it to resolve', async () => {
  const getMock = mock();
  const HttpClientMock = mock(HttpClient);
  HttpClientMock.returnValue({ get: getMock });

  await goAndGetAResource();

  expect(getMock).toBeCalledWith('https://example.com/randomThing');
});

// Production code
const httpClient = new HttpClient()

export const goAndGetAResource = async () => {
  try {
    result = await httpClient.get('https://example.com/randomThing');
  }
  catch (error) {
    if (error.status === 404) {
      console.error('we got a 404');
    }
  }
}

Unit Testing Infrastructure as Code

// Production Code
export const createLambdaWithDynamoDBConfig = (functionName: string, tableName: string) => {
  const lambdaConfig = {
    FunctionName: functionName,
    Runtime: 'nodejs20.x',
    MemorySize: 128,
    Timeout: 30,
    Handler: 'index.handler',
    Role: `arn:aws:iam::123456789012:role/${functionName}Role`, // Reference to the IAM role
  };

  const dynamoDBTableConfig = {
    TableName: tableName,
    AttributeDefinitions: [{ AttributeName: 'id', AttributeType: 'S' }],
    KeySchema: [{ AttributeName: 'id', KeyType: 'HASH' }],
    ProvisionedThroughput: {
      ReadCapacityUnits: 1,
      WriteCapacityUnits: 1,
    },
  };

  const iamRoleConfig = {
    RoleName: `${functionName}Role`,
    AssumeRolePolicyDocument: {
      Version: '2012-10-17',
      Statement: [
        {
          Effect: 'Allow',
          Principal: { Service: 'lambda.amazonaws.com' },
          Action: 'sts:AssumeRole',
        },
      ],
    },
    Policies: [
      {
        PolicyName: `${functionName}DynamoDBWritePolicy`,
        PolicyDocument: {
          Version: '2012-10-17',
          Statement: [
            {
              Effect: 'Allow',
              Action: ['dynamodb:PutItem', 'dynamodb:UpdateItem', 'dynamodb:DeleteItem'],
              Resource: `arn:aws:dynamodb:us-east-1:123456789012:table/${tableName}`,
            },
          ],
        },
      },
    ],
  };

  return {
    lambda: lambdaConfig,
    dynamoDB: dynamoDBTableConfig,
    iamRole: iamRoleConfig,
  };
};

// Tests
test('should create the exact lambda and dynamodb configuration', () => {
  const functionName = 'my-data-processor';
  const tableName = 'my-data-table';

  const expectedConfig = {
    lambda: {
      FunctionName: functionName,
      Runtime: 'nodejs20.x',
      MemorySize: 128,
      Timeout: 30,
      Handler: 'index.handler',
      Role: `arn:aws:iam::123456789012:role/${functionName}Role`,
    },
    dynamoDB: {
      TableName: tableName,
      AttributeDefinitions: [{ AttributeName: 'id', AttributeType: 'S' }],
      KeySchema: [{ AttributeName: 'id', KeyType: 'HASH' }],
      ProvisionedThroughput: {
        ReadCapacityUnits: 1,
        WriteCapacityUnits: 1,
      },
    },
    iamRole: {
      RoleName: `${functionName}Role`,
      AssumeRolePolicyDocument: {
        Version: '2012-10-17',
        Statement: [
          {
            Effect: 'Allow',
            Principal: { Service: 'lambda.amazonaws.com' },
            Action: 'sts:AssumeRole',
          },
        ],
      },
      Policies: [
        {
          PolicyName: `${functionName}DynamoDBWritePolicy`,
          PolicyDocument: {
            Version: '2012-10-17',
            Statement: [
              {
                Effect: 'Allow',
                Action: ['dynamodb:PutItem', 'dynamodb:UpdateItem', 'dynamodb:DeleteItem'],
                Resource: `arn:aws:dynamodb:us-east-1:123456789012:table/${tableName}`,
              },
            ],
          },
        },
      ],
    },
  };

  const actualConfig = createLambdaWithDynamoDBConfig(functionName, tableName);
  expect(actualConfig).toEqual(expectedConfig);
});

This is an extremely brittle test, which is testing the JSON output of our IaC function, not what is deployed to the cloud, nor the functionality we want (can the lambda write to the table?), and if any aspect of the infrastructure changes, this test will fail!

It would be better to simply delete this test and have an end-to-end test that asserts data is written to the DynamoDB table when the lambda is invoked.

Avoiding the Clangers

Start at the top of the pyramid of testing and work down.
Work with stakeholders to define the requirements of a feature.
Write requirements as tests.
Make sure new tests fail before writing code.
Consider writing tests as part of delivering a feature.
Write unit tests only to implement requirements.
Keep logic out of tests.
Keep tests isolated.
Express bugs as tests.
Review tests.

Testing With The Builder Pattern

Jon Ashdown — Fri, 11 Jul 2025 22:09:18 +0000

A Story about the Mystery Guest

I remember the conversation well, sitting down with my colleague for a pairing session on my first day working on an unfamiliar code base, opening up a test file and looking at the tests around the feature we where tasked with modifying.

Me:

Why is this test asserting this value?
where is the data coming from?

Colleague:

Oh, its coming from the 21st fixture file, line 453

Me:

Wow, good memory!
and how is it being loaded into the test file?

Colleague:

I don't know. I think there is some boot-strapping done by the test runner.

The conversation continued along the lines of how the fixtures where periodically generated and how inconvenient they where to maintain.

A classic example of The Mystery Guest anti-pattern. A pattern where we don't know where the test data is coming from and how it is affecting the outcome of the test. A pattern that adds fragility and breaks the concept of tests as documentation.

How can we write good tests?

A good test follows these principles:

Documents the behavior of a feature.
Introduces no logic other than what is contained in the feature under test.
Describes clearly what, given the input, the output should be.
Independent of other tests.
Responsible for it's own setup and teardown.

These last 3 points are what this article focuses on.

We could simply write tests that use raw data, and meet the requirements of a good test. However with complex data, this will soon get unwieldy and could cause a maintenance nightmare should the shape of the data change.

export interface CartItem {
  name: string;
  price: number;
  quantity: number;
}

export interface Cart {
  items: CartItem[];
  taxRate: number;
}

export function calculateTotalPrice(cart: Cart): number {
  const subtotal = cart.items.reduce((total, item) => total + item.price * item.quantity, 0);
  return subtotal * (1 + cart.taxRate);
}

// Test for calculateTotalPrice

test('should calculate the total price with tax', () => {
  // Arrange: Create the test data directly in the test
  const cart: Cart = {
    items: [
      { name: 'Apple', price: 1.0, quantity: 2 },
      { name: 'Banana', price: 0.5, quantity: 3 },
    ],
    taxRate: 0.1, // 10% tax
  };

  // Expected Result
  // (2 * 1.0) + (3 * 0.5) = 2.0 + 1.5 = 3.5
  // 3.5 * 1.1 = 3.85
  const expected = 3.85
  // Act: Call the function under test

  const totalPrice = calculateTotalPrice(cart);

  // Assert: Check the result
  expect(totalPrice).toBe(expected);
});

Introducing the Builder Pattern

The builder pattern allows for a way of meeting the requirements of a good test and provides the following:

Sensible defaults
Future proofing - if the shape of the data changes the tests don't have to
Required data guard - If the wider functionality requires specific data to be present, whilst the test doesn't, then the test wont fail

There are a number of ways of implementing the builder pattern in the JavaScript/TypeScript eco-system. However the implementation that works for my use case takes advantage of both the object oriented paradigm and the code generation that my IDE gets from using TypeScript.

export abstract class AbstractDataBuilder<T> {
  private data!: Partial<T>;

  with<K extends keyof T>(key: K, value: T[K] | undefined) {
    this.data[key] = value;
    return this;
  }

  get build(): Partial<T> {
    return Object.freeze(this.data);
  }
}

When this abstract class is extended, an interface or type T must be specified, this allows for type hinting and code completion in the IDE. The Partial allows testing data where some attributes might be missing. The build method allows for the data to be returned and used without pollution from the instantiated builder class. The with method returns this so we can chain calls to the builder.

Implementing a builder for the interfaces/types we are testing against looks like:

import { AbstractDataBuilder } from '.https://raw.githubusercontent.com/jonashdown/Blog/main/_pngs/libs/abstract-data-builder'
import { Cart, CartItem } from 'https://raw.githubusercontent.com/jonashdown/Blog/main/_pngs/cart.types'

export class CartItemBuilder extends AbstractDataBuilder<CartItem> {
  constructor() {
    super();
    this.data = {
      //keys generated by the IDE
      name: 'some item',
      price: 0.0,
      quantity: 0,
    };
  }
}

export class CartBuilder extends AbstractDataBuilder<Cart> {
  constructor() {
    super();
    this.data = {
      //keys generated by the IDE
      items: [],
      taxRate: 0.1
    };
  }
}

Now we can change our test code:

import { Cart, CartItem } from 'https://raw.githubusercontent.com/jonashdown/Blog/main/_pngs/cart.types';
import { CartBuilder, CartItemBuilder } from 'https://raw.githubusercontent.com/jonashdown/Blog/main/_pngs/cart.builders';
import { calculateTotalPrice } from 'https://raw.githubusercontent.com/jonashdown/Blog/main/_pngs/cart';

// Test for calculateTotalPrice
test('should calculate the total price with tax', () => {
  // Arrange: Create the test data using the builder
  //We could have production scenarios where data is missing.
  //Inform TypeScript that the Partial data we generated for the test is complete.
  const item1 = new CartItemBuilder()
    .with('price', 1.0)
    .with('quantity', 2)
    .build as CartItem;

  const item2 = new CartItemBuilder()
    .with('price', 0.5)
    .with('quantity', 3)
    .build as CartItem;


  const cart = new CartBuilder()
    .with('items', [item1, item2])
    .with('taxRate', 0.1)
    .build as Cart;

  // Expected Result
  // (2 * 1.0) + (3 * 0.5) = 2.0 + 1.5 = 3.5
  // 3.5 * 1.1 = 3.85
  const expected = 3.85

  // Act: Call the function under test
  const totalPrice = calculateTotalPrice(cart);

  // Assert: Check the result
  expect(totalPrice).toBe(expected);
});

N.B We no longer include the item names as we don't need them to get the result, and there is a default value for name, any guard that checks for the presence of name wont cause the test to fail. Here we are explicitly changing the values that we care about.

Problems

Whilst this way of using the builder generally works, there are some issues:

Not Future Proof

The CartBuilder is expecting an Array of items. What if the underlying data structure was changed to a Map or Set? This means our tests are not future proof.

This issue can be handled by adding an extra function withItem to the concrete class, allowing the data structure to change without changing the tests.

export class CartBuilder extends AbstractDataBuilder<Cart> {
  constructor() {
    super();
    this.data = {
      //keys generated by the IDE
      items: [],
      taxRate: 0.1
    };
  }

  //expose a method to add items to attributes with complex types
  //allowing the complex type to change in the future.
  withItem(item: CartItem){
    this.data!.items!.push(item);
    return this;
  }
}

// Test for calculateTotalPrice
test('should calculate the total price with tax', () => {
  // Arrange: Create the test data using the builder
  const item1 = new CartItemBuilder()
    .with('price', 1.0)
    .with('quantity', 2)
    .build as CartItem;

  const item2 = new CartItemBuilder()
    .with('price', 0.5)
    .with('quantity', 3)
    .build as CartItem;

  const cart = new CartBuilder()
    .withItem(item1)
    .withItem(item2)
    .with('taxRate', 0.1)
    .build as Cart;

  // Expected Result
  // (2 * 1.0) + (3 * 0.5) = 2.0 + 1.5 = 3.5
  // 3.5 * 1.1 = 3.85
  const expected = 3.85
  // Act: Call the function under test

  const totalPrice = calculateTotalPrice(cart as Cart);

  // Assert: Check the result
  expect(totalPrice).toBe(expected);
});

Explicit vs Implicit undefined

What if we need to have missing data for a test? Currently we would need to use

const item = new CartItemBuilder()
    .with('price', 0.5)
    .with('quantity', 3)
    .with('name', undefined)
    .build;

this currently will return an object with an explicit undefined:

{
  "name": undefined,
  "price": 0.5,
  "quantity": 3
}

Some test frameworks treat this differently to an implicit undefined, giving unexpected and hard to debug results:

{
  "price": 0.5,
  "quantity": 3
}

We can resolve this by adding a delete method to the AbstractDataBuilder:

export abstract class AbstractDataBuilder<T> {
  private data!: Partial<T>;

  with<K extends keyof T>(key: K, value: T[K]) {
    this.data[key] = value;
    return this;
  }

  delete<K extends keyof T>(key: K) {
    delete this.data[key];
    return this;
  }

  get build(): Partial<T> {
    return Object.freeze(this.data);
  }
}

Same default data

Whilst running TDD tests with Data Builders in this way we probably wont see Test Bleed, a problem where the state of one test affects the state of another. We may see it when running integration tests, breaking the Independent of other tests principle.

To fix this we can add random data to the concrete builder classes. In this case using faker.

import { faker } from '@faker-js/faker';
import { AbstractDataBuilder } from '.https://raw.githubusercontent.com/jonashdown/Blog/main/_pngs/libs/abstract-data-builder'
import { CartItem } from 'https://raw.githubusercontent.com/jonashdown/Blog/main/_pngs/cart.types'

export class CartItemBuilder extends AbstractDataBuilder<CartItem> {
  constructor() {
    super();
    this.data = {
      //keys generated by the IDE, values are random
      name: faker.food.fruit(),
      price: faker.number.float({ min: 0, max: 100, multipleOf: 0.01 }),
      quantity: faker.number.int({ min: 0, max: 100 }),
    };
  }
}

Where should Data Builders Live ?

Ideally Data Builders should be stored alongside the Classes, Interfaces and Types they represent. This allows for quick updates without having to navigate through multiple folders, and allowing for Seperation of Concerns.

The examples shown use this folder structure

├── package.json
├── package-lock.json
├── Readme.md
├── tsconfig.json
└── src
    ├── libs
    │   └── abstract-data-builder.ts
    └── cart
      ├── cart.ts
      ├── cart.builder.ts
      ├── cart.types.ts
      └── cart.test.ts

Exporting Non-JSON data from Builders

All the examples so far are exporting data as JSON, the default for the JavaScript eco-system. What if we wanted some other format? Potentially we might want to test grahpql, string, xml, yaml or some other JSON structure.

export class CartItemBuilder extends AbstractDataBuilder<CartItem> {
  constructor() {
    super();
    this.data = {
      //keys generated by the IDE
      name: 'some item',
      price: 0.0,
      quantity: 0,
    };
  }

  get string():string {
    return JSON.stringify(this.data);
  }

  get xml():string {
    //custom xml generator
  }
}

Conclusion

The Builder Pattern, if used correctly, provides a clean, powerful, extensible and reusable way of generating test data, allowing us to follow the principles of good testing.

It can be used effectively across all levels of automated testing, and it can be used to generate partial data.

Most importantly by using this pattern, we can eliminate the Mystery Guest anti-pattern and the overhead of maintaining large collections of fixtures. Should the structure of the data change, the blast-radius is kept to a minimum.

A (Future) Story about the Builder Pattern

Me:

Why is this test asserting this value?
where is the data coming from?

Colleague:

It's the behavior of the function we are testing.
The data is coming from an instantiation of a Builder Class, where we use sensible defaults and override the data that we care about.

Me:

Wow, thats impressive!
and how is it being loaded into the test file?

Colleague:

The Builder Class is imported just like any other dependency, and is stored and updated alongside the schema it represents.
The instantiation and overrides happen within each test, and as the defaults are randomized, we have mitigated against Test Bleed

The conversation continued along the lines of how the builders where easy to maintain, and when the data structures change the affects are minimal.

How To Guarantee Race Conditions When Integration Testing

Jon Ashdown — Thu, 10 Jul 2025 15:56:48 +0000

After some recent outages with Github, my team found that a particular repo had a large number of flaky integration tests, which where uncovered when the team did some minor maintenance and highlighted by the outages.

On checking both the main branch and running the builds from developers machines, it became obvious that the flakiness was caused by race conditions within the test suites, as different tests where failing on each build run.

Some Definitions

Race conditions occur when several processes compete for a resource, and flaky integration tests are often a manifestation of that.

The team's integration tests are automatically run on an environment like live (in this case an AWS account that has been set up to replicate the live environment) and exercise individual components in a larger microservice architecture. Typically they either write to data stores or trigger events and assert that the expected processing has taken place by interrogating other stores or capturing events from the processing.

e.g

How can race conditions occur?

Truncating databases whilst tests are still running

Tidying up test data is good practice as it allows for stacks to be torn down efficiently without leaving any artifacts. However when each test suite attempts to truncate the databases it is using, race conditions are assured as other test suites may not have finished and may find themselves asserting against non-existent data.

If the test is responsible for deleting the test data, any failure to delete will fail the test.

Not allowing processes to complete before assertion

Data processing takes time. By inputting data and immediately asserting against the output will cause race conditions. A slow internet connection will likely pass an integration test, but a fast one will fail.

Reusing/Copying data between tests

Copying test data between tests is guaranteed to get race conditions. Is test 1's data causing test 2 to pass? Is test 2 failing because test 1 is passing? Is test 1 passing because test 2 is? No-one really knows, but our ctl-c/ctrl-v muscles have had a good workout.

How can race conditions be prevented?

Truncate databases after all tests have finished

Have a separate clean up step to truncate test data, and ensure that all tests are in a finished state.

This also means that failure to delete is not a test failure, and deletion is still possible should a test fail.

Allow processes to complete before assertion

A simple sleep between writing input data and asserting against output data, whilst primitive, might work, however the sleep time will need to be tweaked to ensure processing has finished.

However a better approach exists. Polling the output and asserting over a sensible period of time will allow the processing to take place. Tools like WaitForExpect provide a way of polling and will pass a test the moment the assertion is met, and will fail on a timeout.

Use unique data

Using unique, or better yet random data, will ensure that there is no bleed between tests and keep the assertions clean. Using the builder pattern with random defaults will provide a reusable method of specifying unique data.

How To Guarantee Race Conditions When Integration Testing

Jon Ashdown — Thu, 10 Jul 2025 15:50:50 +0000

Some Definitions

Race conditions occur when several processes compete for a resource, and flaky integration tests are often a manifestation of that.

e.g

How can race conditions occur ?

Truncating databases whilst tests are still running

If the test is responsible for deleting the test data, any failure to delete will fail the test.

Not allowing processes to complete before assertion

Reusing/Copying data between tests

How can race conditions be prevented?

Truncate databases after all tests have finished

Have a separate clean up step to truncate test data, and ensure that all tests are in a finished state.

This also means that failure to delete is not a test failure, and deletion is still possible should a test fail.

Allow processes to complete before assertion

A simple sleep between writing input data and asserting against output data, whilst primitive, might work, however the sleep time will need to be tweaked to ensure processing has finished.

Use unique data

Automating my Blog

Jon Ashdown — Sat, 11 Nov 2023 18:27:33 +0000

I use Github to store the code that I write, and Github Actions to automate my workflows. My blog is hosted on a number of platforms including Dev.to, which accepts markdown as an input format and has an api.

If I could store my blog in Github, I could do the following

Write blog articles off line and use git to keep them upto date.
Push completed articles to Github.
Take advantage of Github Actions to:
- Publish to various blogging platforms
- Grammar and spell check
- Generate and store diagrams
- Inform social media platforms of new/updated articles
- Display latest articles on my Github profile.

Publishing to Dev.to

A quick search of the Github marketplace found Publish to dev.to maintained and owned by sinedied which seems to fit my needs.

After following the config steps for this action, I was able to see articles being published to my blog. The default config worked fine for the way I work, which is to always create a Pull Request before I merge to main.

Generating Diagrams

Mermaid.js seemed like the obvious choice as it integrates with Markdown and Github, and importantly has a commnand line interface, which means it can be used in a workflow file.

Any generated images can use the github raw urls, so a bit of sed magic will convert the local urls that mermaid generates into a full url. N.b to take advantage of Github as a hosting service, the repo will have to be set as public otherwise, there will be a need to play with tokens and their expiry. I will write about how the repo is secured in another post.

I will also need ot consider how the generated images are added to the repo.

AWS Gotchas

Jon Ashdown — Wed, 17 Nov 2021 14:36:25 +0000

AWS Gotchas

I will highlight AWS gotchas as I find them

Dynamodb

TTL Fields are in Seconds

TTL Fields on dynamodb are great, as they allow a grace period between marking a record for deletion and the actual deletion. Meaning that when you have finished with a record, you can set a TTL field to a time in the future and AWS internal mechanisms will remove the record at some point soon after the that time has passed.

Gotcha:

The TTL field is an integer in seconds from the Unix epoch, all other dates, including explicit date fields in dynamodb are stored as ISO dates, which are stored under the hood as microseconds from the Unix epoch. This is documented at Using DynamoDB Time to Live (TTL). However, it is easy to forget hence the reason for this gotcha.

Workaround:

Calculate the TTL using your standard date library, convert to integer and divide by 1000 to convert milliseconds to seconds.
e.g

import { DocumentClient } from 'aws-sdk/clients/dynamodb'
const dynamoDb = new DocumentClient()

const hoursInTheFuture = 24

const getFutureTimeInSeconds = () => {
  const time = new Date()
  const hours = time.getHours() + hoursInTheFuture

  time.setHours(hours)

  return Math.round(time.getTime() / 1000)
}

export const deleteInTheFuture = async (id) => {
  const result = await dynamoDb.update({
    TableName: 'myDynamoDbTable',
    Key: {
      id
    },
    UpdateExpression: 'SET ttl = :t',
    ExpressionAttributeValues: {
      ':t': getFutureTimeInSeconds() //record will be deleted 24 hours from now
    }

  }).promise()

  return result
}

AWS Gotchas

Jon Ashdown — Wed, 17 Nov 2021 14:32:35 +0000

AWS Gotchas

I will highlight AWS gotchas as I find them

Dynamodb

TTL Fields are in Seconds

Gotcha:

Workaround:

Calculate the TTL using your standard date library, convert to integer and divide by 1000 to convert milliseconds to seconds.
e.g

import { DocumentClient } from 'aws-sdk/clients/dynamodb'
const dynamoDb = new DocumentClient()

const hoursInTheFuture = 24

const timeInSeconds = () => {
  const time = new Date()
  const hours = time.getHours() + hoursInTheFuture

  time.setHours(hours)

  return Math.round(time.getTime() / 1000)
}

export const deleteInTheFuture = async (id) => {
  const result = await dynamoDb.update({
    TableName: 'myDynamoDbTable',
    Key: {
      id
    },
    UpdateExpression: 'SET ttl = :t',
    ExpressionAttributeValues: {
      ':t': timeInSeconds() //record will be deleted momentarily from now
    }

  }).promise()

  return result
}

Starting a technical blog

Jon Ashdown — Thu, 28 Oct 2021 17:13:39 +0000

Starting a Blog

I have decided to start a technical blog. This will be a combination of tutorials, opinions and anecdotes of my experience of 20 years working in software.
Hopefully the content will be of use to someone.

Jon AshdownFollow

Software Consultant