Test Doubles in Automated Testing

Introduction: Why do Test Doubles Matter?

• Real collaborators like databases and third-party APIs can make tests slow and hard to control. Test Doubles replace them with simpler alternatives when needed, but knowing when to use them and when to use real objects is just as important.
• Nomenclatures and concept definitions follow Gerard Meszaros and Martin Fowler. References are at the end of the article.

First: General Test Nomenclatures

• SUT: System Under Test. It is the main (class, object, function, etc.) being tested.
• Collaborators: "Secondary objects", they are not the main object like the SUT, but are necessary to test the SUT.
• Observation Point: Provides the ability to analyze the interaction between the SUT and other parts of the system after exercising the SUT.
• Indirect Input: When the behavior of the SUT depends on values returned by another component whose services it uses.
• Indirect Output: Calls the SUT makes to its collaborators (e.g. saving to a repository) that are not visible in the SUT's return value and can only be observed by inspecting the collaborator.

What are Test Doubles?

• Any kind of pretend object used in place of a real object for testing purposes.
• They need to make the SUT believe it’s talking to its real collaborators

Important:

• When talking about Test Doubles we are mainly talking about unit tests and sometimes integration tests, but rarely for e2e tests. Unit tests alone will never guarantee reliability for future software changes during maintenance or development of new features, so e2e tests are highly recommended.
• The quote below argues in favor of this position:
  • "It's at this point that I should stress that whichever style of test you use, you must combine it with coarser grained acceptance tests that operate across the system as a whole. I've often come across projects which were late in using acceptance tests and regretted it." - Martin Fowler

One exception to using Test Doubles in E2E tests

• Mainly applies to third-party services.
• Testing third-party services in E2E tests can be tricky and non-deterministic, failing due to network latency, timeouts, or other factors. Flaky tests are undesirable.
• While some argue it helps detect failures, real downtime is usually noticed by customers first. Monitoring tools are a better way to track service issues.
• For deterministic E2E tests, it is often preferable to use Test Doubles for third-party services. As Martin Fowler explains:
• "The other area where these tests don't cover the full breadth of the stack lies in connection to remote systems. Many people, including myself, think that tests that call remote systems are unnecessarily slow and brittle. It is usually better to use TestDoubles for these remote systems and check the doubles with ContractTests." - Martin Fowler

For the upcoming examples we will consider the code below:

function validateUser(user: User): ValidationResult {
  if (!user.name) {
    return { errorMessage: "required name", valid: false };
  }
  if (user.name.length < 1 || user.name.length > 30) {
    return { errorMessage: "name length between 1 and 30", valid: false };
  }
  return { valid: true };
}

// using dependency injection to facilitate tests (dbRepository)
export async function createUserUsecase({
  user,
  dbRepository,
}: CreateUsersUsecasePayload): Promise<User> {
  try {
    const validationResult = validateUser(user);
    if (!validationResult.valid) {
      throw new Error(validationResult.errorMessage);
    }
    const existing = await dbRepository.findById(user.id);
    if (existing) throw new Error("user already exists");
    const savedUser = await dbRepository.save(user);
    return savedUser;
  } catch (error: unknown) {
    if (error instanceof Error) console.error(error.message);
    throw error;
  }
}

export async function listUsersUsecase({
  dbRepository,
}: ListUsersUsecasePayload): Promise<User[]> {
  try {
    const users = await dbRepository.list();
    return users;
  } catch (error: unknown) {
    if (error instanceof Error) console.error(error.message);
    throw error;
  }
}

Dummy

• Object passed only to fill parameter lists but never really used.

  describe("createUserUsecase with Dummy", () => {
    const dummyDbRepository = {} as DbRepository<User>;

    test("Given empty username should throw error", async () => {
      await assert.rejects(
        async () =>
          createUserUsecase({
            user: { id: 1 } as any,
            dbRepository: dummyDbRepository,
          }),
        /required name/
      );
    });

    test("Given username too long should throw error", async () => {
      await assert.rejects(
        async () =>
          createUserUsecase({
            dbRepository: dummyDbRepository,
            user: { id: 3, name: "a".repeat(31) },
          }),
        /name length between 1 and 30/
      );
    });

    // we can't make this one with dummy...
    // test("Given valid username should save user", async () => {});
  });

Fake

• Object with a real working implementation, but with a shortcut that make it not a good option for production, but perfect for tests.

  class UsersInMemoryRepositoryFake implements DbRepository<User> {
    users: User[] = [];

    async save(input: User): Promise<User> {
      this.users.push(input);
      return input;
    }

    async list(): Promise<User[]> {
      return this.users;
    }

    async findById(id: number): Promise<User | null> {
      return this.users.find(u => u.id === id) ?? null;
    }

    clean() {
      this.users = [];
    }
  }

  describe("createUserUsecase with Fake", () => {
    const usersInMemoryRepositoryFake = new UsersInMemoryRepositoryFake();

    beforeEach(() => {
      usersInMemoryRepositoryFake.clean();
    })

    test("Given empty username should throw error", async () => {
      await assert.rejects(
        async () =>
          createUserUsecase({
            dbRepository: usersInMemoryRepositoryFake,
            user: { id: 1 } as any,
          }),
        /required name/
      );
    });

    test("Given username too long should throw error", async () => {
      await assert.rejects(
        async () =>
          createUserUsecase({
            dbRepository: usersInMemoryRepositoryFake,
            user: { id: 3, name: "a".repeat(31) },
          }),
        /name length between 1 and 30/
      );
    });

    test("Given valid username should save user", async () => {
      const user = { id: 2, name: "John" };
      const savedUser = await createUserUsecase({
        dbRepository: usersInMemoryRepositoryFake,
        user,
      });

      assert.deepStrictEqual(savedUser, user);
      assert.deepStrictEqual(await usersInMemoryRepositoryFake.list(), [user]);
    });

    test("Given duplicate id should throw", async () => {
      // Because fake has real logic (unlike Stub)
      const user = { id: 1, name: "John" };
      await createUserUsecase({ dbRepository: usersInMemoryRepositoryFake, user });
      await assert.rejects(
        async () => createUserUsecase({ dbRepository: usersInMemoryRepositoryFake, user }),
        /user already exists/
      );
    });
  });
  // Could test the list use case here too, but to exemplify this is enough.

Stub

• Provide fixed responses for what is required during the test, but can't respond anything outside the context of the test.
• We never verify Stub's state or behavior, we are only using the stub with a fixed response to test what we need.
• There are Stub variations; I'll mention the ones I consider most common:
  • Responder: A stub that injects valid indirect inputs into the SUT. Generally used in "happy path" tests.
  • Saboteur: A stub that injects invalid indirect inputs into the SUT. Used to test how the SUT behaves with incorrect indirect inputs.
  • Hard-Coded: Responses baked into the implementation. Not all stubs need to be hard-coded — a Configurable Stub gets its response injected at setup (e.g., via constructor).
  • If you always hard-coded stubs and wondered if anything else was a Fake: the difference is that Fakes mirror real production logic but take shortcuts, while stubs just return canned responses (even if they store some state to do so).
• Example of SUT:

  class UsersStubRepository implements DbRepository<User> {
    async save(input: User): Promise<User> {
      return input;
    }

    async list(): Promise<User[]> {
      return [{ id: 1, name: "joao" }, { id: 2, name: "john" }];
    }

    async findById(_id: number): Promise<User | null> {
      return null;
    }
  }

  describe("listUsersUsecase with Stub", () => {
    const usersStubRepository = new UsersStubRepository();

    test("Given existing users should return users list", async () => {
      const users = await listUsersUsecase({ dbRepository: usersStubRepository });
      assert.deepStrictEqual(users, [{ id: 1, name: "joao" }, { id: 2, name: "john" }]);
    });
  });

  describe("createUserUsecase with Stub", () => {
    const usersStubRepository = new UsersStubRepository();

    test("Given valid user should save user", async () => {
      const user = { id: 1, name: "John" };
      const result = await createUserUsecase({ dbRepository: usersStubRepository, user });
      assert.deepStrictEqual(result, user);
    });

    test("Given same id called twice should save both times without throwing", async () => {
      const user = { id: 1, name: "John" };
      await createUserUsecase({ dbRepository: usersStubRepository, user });
      const result = await createUserUsecase({ dbRepository: usersStubRepository, user });
      assert.deepStrictEqual(result, user);
    });
  });

Spy

• Stubs that also record some information based on how they were called. For example, an email service that records how many messages it was sent.
• It can save any information on how they were called: number of calls, arguments passed, order of calls, timestamp, return values...

  class UsersSpyRepository implements DbRepository<User> {
    private _saveCallCount: number = 0;
    private _lastSavedUser: User | null = null;
    private _findByIdCallCount: number = 0;
    private _lastFindByIdArg: number | null = null;

    async save(input: User): Promise<User> {
      this._saveCallCount++;
      this._lastSavedUser = input;
      return input;
    }

    async list(): Promise<User[]> {
      return [{ id: 1, name: "joao" }, { id: 2, name: "john" }];
    }

    async findById(id: number): Promise<User | null> {
      this._findByIdCallCount++;
      this._lastFindByIdArg = id;
      return null;
    }

    getSaveCallCount() { return this._saveCallCount; }
    getLastSavedUser() { return this._lastSavedUser; }
    getFindByIdCallCount() { return this._findByIdCallCount; }
    getLastFindByIdArg() { return this._lastFindByIdArg; }

    clean() {
      this._saveCallCount = 0;
      this._lastSavedUser = null;
      this._findByIdCallCount = 0;
      this._lastFindByIdArg = null;
    }
  }

  describe("createUserUsecase with Spy", () => {
    const usersSpyRepository = new UsersSpyRepository();

    beforeEach(() => {
      usersSpyRepository.clean();
    });

    test("Given valid user should call findById with user id then save the user", async () => {
      const user = { id: 42, name: "John" };
      await createUserUsecase({ dbRepository: usersSpyRepository, user });

      assert.strictEqual(usersSpyRepository.getFindByIdCallCount(), 1);
      assert.strictEqual(usersSpyRepository.getLastFindByIdArg(), 42);
      assert.strictEqual(usersSpyRepository.getSaveCallCount(), 1);
      assert.deepStrictEqual(usersSpyRepository.getLastSavedUser(), user);
    });

    test("Given invalid user should not call findById or save", async () => {
      await assert.rejects(
        async () => createUserUsecase({ dbRepository: usersSpyRepository, user: { id: 1 } as any }),
        /required name/
      );
      assert.strictEqual(usersSpyRepository.getFindByIdCallCount(), 0);
      assert.strictEqual(usersSpyRepository.getSaveCallCount(), 0);
    });
  });

Mock

• A pre-programmed object/function that has expectations about how it should be used or called, and which will verify that the expected actions occurred.
• In general tests we do state verification using real instances of our classes and checking how their states were impacted after the SUT exercise, this state verification is made using asserts in the collaborators.
• Mock objects allows us to do behavior verification checking what calls were made to the mock. Unlike Spy, there are no external state asserts on the mock in the test body; the mock encapsulates its own verification internally (assertions run inside the mock methods and in verify()).

class UsersRepositoryMock implements DbRepository<User> {
  private _expectedFindByIdArg: number | null = null;
  private _expectedSaveArg: User | null = null;
  private _expectedFindByIdCalls = 0;
  private _expectedSaveCalls = 0;
  private _actualFindByIdCalls = 0;
  private _actualSaveCalls = 0;

  expectFindById(id: number, times = 1) {
    this._expectedFindByIdArg = id;
    this._expectedFindByIdCalls = times;
    return this;
  }

  expectSave(user: User, times = 1) {
    this._expectedSaveArg = user;
    this._expectedSaveCalls = times;
    return this;
  }

  async findById(id: number): Promise<User | null> {
    this._actualFindByIdCalls++;
    assert.strictEqual(id, this._expectedFindByIdArg);
    return null;
  }

  async save(input: User): Promise<User> {
    this._actualSaveCalls++;
    assert.deepStrictEqual(input, this._expectedSaveArg);
    return input;
  }

  async list(): Promise<User[]> { return []; }

  verify() {
    assert.strictEqual(this._actualFindByIdCalls, this._expectedFindByIdCalls);
    assert.strictEqual(this._actualSaveCalls, this._expectedSaveCalls);
  }
}

describe("createUserUsecase with Mock", () => {
  test("Given valid user should call findById and save with correct arguments", async () => {
    const user = { id: 1, name: "John" };
    // assertions configured on arrange part
    const repositoryMock = new UsersRepositoryMock()
      .expectFindById(user.id)
      .expectSave(user);

    // asserts on the mock are executed during the exercise of the SUT -> testing behavior
    // mock is checking the passed args
    await createUserUsecase({ dbRepository: repositoryMock, user });

    // verify if it was called enough times
    repositoryMock.verify();
  });

  test("Given invalid user should not call findById or save", async () => {
    const repositoryMock = new UsersRepositoryMock();

    await assert.rejects(
      async () => createUserUsecase({ dbRepository: repositoryMock, user: { id: 1 } as any }),
      /required name/
    );

    repositoryMock.verify();
  });
});

Use Cases Quick Summary:

• Dummy:
  • Use when: the collaborator is required but never called in that test path
  • Avoid when: the SUT will actually invoke the collaborator
• Fake:
  • Use when: you need a real working collaborator with simplified internals (e.g. in-memory DB)
  • Avoid when: a fixed response is enough -> a Stub is simpler
• Stub:
  • Use when: you need to control what the collaborator returns (indirect input)
  • Avoid when: you also need to verify how many times or how the collaborator was called
• Spy:
  • Use when: you need to verify indirect output by asserting on the collaborator's recorded state after the SUT runs (e.g. callsToSave === 1)
  • Avoid when: you only care about the SUT's return value -> a Stub is enough
• Mock:
  • Use when: you need to verify the interaction itself (what was called, with what args) without caring about the collaborator's internal state (behavior verification)
  • Avoid when: the collaborator needs real stateful behaviour, use a Fake instead

When and How to Use Test Doubles

• The quote below shows that it depends on the testing philosophy that you and your team prefer:
  • "The classical TDD style is to use real objects if possible and a double if it's awkward to use the real thing. So a classical TDDer would use a real warehouse and a double for the mail service. The kind of double doesn't really matter that much." - Martin Fowler
• Another important quote from Martin Fowler is about cases we find during software development, such as a cache, where state verification proves to be unfeasible in some situations, and for that mock objects are a good fit for behavior verification.
  • "Occasionally you do run into things that are really hard to use state verification on, even if they aren’t awkward collaborations. A great example of this is a cache. The whole point of a cache is that you can’t tell from its state whether the cache hit or missed - this is a case where behavior verification would be the wise choice for even a hard core classical TDDer. I’m sure there are other exceptions in both directions." - Martin Fowler

Risks of Overusing Test Doubles:

• You should not overuse Test Doubles, since your SUTs will be using real implementations it's important to test using them too.
• If we overuse Test Doubles we will have what is called Fragile Test
  • "We must be careful when using Test Stubs because we are testing the SUT in a different configuration from that which will be used in production. We really should have at least one test that verifies it works without a Test Stub. A common mistake made by test automaters new to stubs is to replace a part of the SUT that they are trying to test. It is therefore important to be really clear about what is playing the role of SUT and what is playing the role of test fixture. Also, note that excessive use of Test Stubs can result in Overspecified Software." - Gerard Meszaros

Conclusion

• Although we've discussed nomenclature and the usage of Test Doubles throughout the article, developers normally don't follow this naming strictly. It's important to understand the role of each one, but in practice you just need to know these tools to write better tests.
• In the majority of testing libraries, people tend to call everything a mock. Is it a problem when we use the same name for multiple different things? Absolutely.
• But if you understand the concepts and know when you need each one, you are able to simply use a dummy, stub, or fake for the majority of easier test cases, and spy on what is needed to test indirect output. Mocks help to better follow object-oriented design, in which you want to "tell, don't ask". Example:

  // BAD
  class OrderService {
    register(order) {
      const user = db.getUser(order.userId);

      if (user.isActive) {
        emailService.send(user.email); // this logic is outside the domain object (user)
      }
    }
  }

  // GOOD
  class OrderService {
    register(user) {
      user.notify();
    }
  }

  test("should notify user", () => {
    const user = {
      notify: jest.fn(),
    };
    const service = new OrderService();
    service.register(user);
    expect(user.notify).toHaveBeenCalled();
  });

References

• https://martinfowler.com/bliki/TestDouble.html
• https://martinfowler.com/articles/mocksArentStubs.html
• http://xunitpatterns.com/Test%20Double.html
• http://xunitpatterns.com/Test%20Stub.html
• http://xunitpatterns.com/Test%20Spy.html
• http://xunitpatterns.com/Mock%20Object.html
• https://martinfowler.com/bliki/InMemoryTestDatabase.html
• Github Repository for this article