Reverse Engineering - understanding Spies in Testing

Follow me on Twitter, happy to take your suggestions on topics or improvements /Chris

We use a spy to not only mock a response from a dependency but to ensure that our dependency has been correctly called. With correct we mean we mean the number of times, the correct type and number of arguments. There is a lot we can verify to ensure our code behaves correctly. This exercise is about understanding Spies in Jasmine, what goes on under the hood

In this article we are looking to explain:

• WHY, Understand WHY we use Spies and what they are good far
• WHAT, Explain what Spies can do for us
• HOW, uncover how they must be working under the hood but attempt to reverse engineering their public API

TLDR If you just want to see the implementation and don't care for reading how we got there then scroll to the bottom where the full code is. :)

 Why Spies

Let's set the scene. We have a business-critical function in which we want to ship an order to a user. The application is written in Node.js, that is JavaScript on the backend.

It's imperative that we get paid before shipping the order. Any changes to this code should be caught by our spy that we are about to implement.

The code looks like this:

async function makeOrder(
  paymentService, 
  shippingService, 
  address, 
  amount, 
  creditCard
) {
  const paymentRef = await paymentService.charge(creditCard, amount)

  if (paymentService.isPaid(paymentRef)) {
    shippingService.shipTo(address);
  }
}

We have the function makeOrder(). makeOrder() gets help from two different dependencies a shippingService and a paymentService. It's critical that the paymentService is being invoked to check that we have gotten paid before we ship the merchandise, otherwise it's just bad for business.

It's also important that we at some point call the shippingService to ensure the items gets delivered. Now, it's very seldom the code is this clear so you see exactly what it does and the consequences of removing any of the below code. The point is we need to write tests for the below code and we need spies to verify that our code is being called directly.

In short:

Spies are about asserting behavior over asserting on results

 What

Ok so we've mentioned in the first few lines of this article that Spies can help us check how many times a dependency is called, with what arguments and so on but let's try to list all the features that we know of in Jasmine Spies:

• Called, verify it has been called
• Args, verify it has been called with a certain argument
• Times called, verify the number of times it has been called
• Times called and args, verify all the number of times it was called and all the arguments used
• Mocking, return with a mocked value
• Restore, because spies replace the original functionality we will need to restore our dependency to its original implementation at some point

That's quite a list of features and it should be able to help us assert the behavior on the above makeOrder().

The HOW

This is where we start looking at Jasmine Spies and what the public API looks like. From there we will start to sketch out what an implementation could look like.

Ok then. In Jasmine we create a Spy by calling code like this:

const apiService = {
  fetchData() {}
}

Then we use it inside of a test like this:

it('test', () => {
  // arrange
  spyOn(apiService, 'fetchData')

  // act
  doSomething(apiService.fetchData)

  // assert
  expect(apiService.fetchData).toHaveBeenCalled();
})

As you can see above we have three different steps that we need to care about.

1. Creating the spy with spyOn()
2. Invoking the spy
3. Asserting that the spy has been called

Let's start implementing

Creating the Spy

By looking at how it's used you realize that what you are replacing is one real function for a mocked function. Which means WHAT we end up assigning to apiService.fetchData must be a function.

The other part of the puzzle is how we assert that it has been called. We have the following line to consider:

expect(apiService.fetchData).toHaveBeenCalled()

At this point we need to start implementing that line, like so:

function expect(spy) {
  return {
    toHaveBeenCalled() {
      spy.calledTimes()
    }
  }
}

WAIT. You just said that apiService.fetchData is a function. Yet in expect() you send it in and call calledTimes() on it like it was an object. I'm lost :(

Ah, I see. You probably have a background from an OO language like C# or Java right?

How did you know?

In those languages you are either an object or a function, never both. But we are in JavaScript and JavaScript state that:

Functions are function objects. In JavaScript, anything that is not a primitive type ( undefined , null , boolean , number , or string ) is an object.

Which means our spy, is a function but it has methods and properties on it like it was an object..

Niiice. and weird..

Ok then. With that knowledge, we can start implementing.

// spy.js

function spy(obj, key) {
  times = 0;
  old = obj[key];

  function spy() {
    times++;
  }

  spy.calledTimes = () => times;

  obj[key] = spy;
}


function spyOn(obj, key) {
  spy(obj, key);
}

module.exports = {
  spyOn
}

spyOn() calls spy() that internally creates the function _spy() that has knowledge of the variable times and expose the public method calledTime(). Then we end up assigning _spy to the object whose function we want to replace.

Adding matcher toHaveBeenCalled()

Let's create the file util.js and have it look like so:

// util.js

function it(testName, fn) {
  console.log(testName);
  fn();
}

function expect(spy) {
  return {
    toHaveBeenCalled() {
      let result = spy.calledTimes() > 0;
      if (result) {
        console.log('spy was called');
      } else {
        console.error('spy was NOT called');
      }
    }
  }
}

module.exports = {
  it, 
  expect
}

As you can see it just contains a very light implementation of expect() and it() method. Let's also create a demo.js file that tests our implementation:

// demo.js

const { spyOn } = require('./spy');
const { it, expect } = require('./util');

function impl(obj) {
  obj.calc();
}

it('test spy', () => {
  // arrange
  const obj = {
    calc() {}
  }

  spyOn(obj, 'calc');

  // act
  impl(obj);

  // assert
  expect(obj.calc).toHaveBeenCalled();
})

We have great progress already but let's look at how we can improve things.

Adding matcher toHaveBeenCalledTimes()

This matcher have pretty much written itself already as we are keeping track of the number of times we call something. Simply add the following code to our it() function, in util.js like so:

toHaveBeenCalledTimes(times) {
  let result = spy.calledTimes();
  if(result == times) {
    console.log(`success, spy was called ${times}`)
  } else {
    console.error(`fail, expected spy to be called: ${times} but was: ${result}`)
  }
}

Adding matcher toHaveBeenCalledWith()

Now this matcher wants us to verify that we can tell what our spy has been called with and is used like this:

expect(obj.someMethod).toHaveBeenCalledWith('param', 'param2');

Let's revisit our implementation of the spy():

// excerpt from spy.js

function spy(obj, key) {
  times = 0;
  old = obj[key];

  function spy() {
    times++;
  }

  spy.calledTimes = () => times;

  obj[key] = spy;
}

We can see that we capture the number of times something is called through the variable times but we want to change that slightly. Instead of using a variable that stores a number let's instead replace that with an array like so:

// spy-with-args.js

function spy(obj, key) {
  let calls = []

  function _spy(...params) {
    calls.push({
      args: params
    });
  }

  _spy.calledTimes = () => calls.length;
  _spy._calls = calls;

  obj[key] = _spy;
}

As you can see in thee _spy() method we collect all the input parameters and adds them to an array calls. calls will remember not only the number of invocations but what argument each invocation was done with.

Creating the matcher

To test that it stores all invocation and their argument lets create another matcher function in our expect() method and call it toHaveBeenCalledWith(). Now the requirements for it is that our spy should have been called with these args at some point. It doesn't say what iteration so that means we can loop through our calls array until we find a match.

Let's add our matcher to the method it() in our utils.js, like so:

// excerpt from util.js
toHaveBeenCalledWith(...params) {
  for(var i =0; i < spy._calls.length; i++) {
    const callArgs = spy._calls[i].args;
    const equal = params.length === callArgs.length && callArgs.every((value, index) => { 
      const res = value === params[index];
      return res;
    });
    if(equal) {
      console.log(`success, spy was called with ${params.join(',')} `)
      return;
    }
  }
  console.error(`fail, spy was NOT called with ${params} spy was invoked with:`);
  console.error(spy.getInvocations());

}

Above you can see how we compare params, which is what we call it with to each of the arguments in our invocations on the spy.

Now, let's add some code to demo.js and our test method invocation, so we try out our new matcher, like so:

// excerpt from demo.js

it('test spy', () => {
  // arrange
  const obj = {
    calc() {}
  }

  spyOn(obj, 'calc');

  // act
  impl(obj);

  // assert
  expect(obj.calc).toHaveBeenCalled();
  expect(obj.calc).toHaveBeenCalledWith('one', 'two');
  expect(obj.calc).toHaveBeenCalledWith('three');
  expect(obj.calc).toHaveBeenCalledWith('one', 'two', 'three');
})

Running this in the terminal we get:

We can see that it works like a charm. It succeeds on the two first ones and fails on the last one, as it should.

Reset, the final piece

We got one more piece of functionality we would like to add, namely the ability to reset our implementation. Now, this is probably the easiest thing we do. Let's visit our spy-with-args.js file. We need to do the following:

1. Add a reference to the old implementation
2. Add a method reset() that points us back to our original implementation

Add a reference

Inside of our spy() function add this line:

let old = obj[key];

This will save the implementation to the variable old

Add reset() method

Just add the following line:

_spy.reset = () => obj[key] = old;

The spy() method should now look like so:

function spy(obj, key) {
  let calls = []
  let old = obj[key];

  function _spy(...params) {
    calls.push({
      args: params
    });
  }

  _spy.reset = () => obj[key] = old;
  _spy.calledTimes = () => calls.length;
  _spy.getInvocations = () => {
    let str = '';
    calls.forEach((call, index) => {
      str+= `Invocation ${index + 1}, args: ${call.args} \n`;
    });

    return str;
  }

  _spy._calls = calls;

  obj[key] = _spy;
}

Summary

We've come to the end of the line.
We've implemented a spy from the beginning. Additionally, we've explained how almost everything is an object which made it possible to implement it the way we did.

The end result is a spy that stores all the invocations and the parameters it was called with. We've also managed to create three different matchers that test whether our spy was called, how many times it was called and with what arguments.

All in all a successful adventure into understanding the nature of a spy.

We do realize that this is just a starter for something and taking it to production means we should probably support things like comparing whether something was called with an object, supporting, mocking and so on. I leave that up to you as an exercise.

As another take-home exercise, see if you can write tests for the function makeOrder() that we mentioned in the beginning.

Full code

Here is the full code in case I lost you during the way:

util.js, containing our matcher functions

Our file containing our functions it() and expect() and its matchers.

// util.js

function it(testName, fn) {
  console.log(testName);
  fn();
}

function expect(spy) {
  return {
    toHaveBeenCalled() {
      let result = spy.calledTimes() > 0;
      if (result) {
        console.log('success,spy was called');
      } else {
        console.error('fail, spy was NOT called');
      }
    },
    toHaveBeenCalledTimes(times) {
      let result = spy.calledTimes();
      if(result == times) {
        console.log(`success, spy was called ${times}`)
      } else {
        console.error(`fail, expected spy to be called: ${times} but was: ${result}`)
      }
    },
    toHaveBeenCalledWith(...params) {
      for(var i =0; i < spy._calls.length; i++) {
        const callArgs = spy._calls[i].args;
        const equal = params.length === callArgs.length && callArgs.every((value, index) => { 
          const res = value === params[index];
          return res;
        });
        if(equal) {
          console.log(`success, spy was called with ${params.join(',')} `)
          return;
        }
      }
      console.error(`fail, spy was NOT called with ${params} spy was invoked with:`);
      console.error(spy.getInvocations());

    }
  }
}

module.exports = {
  it, 
  expect
}

spy implementation

Our spy implementation spy-with-args.js:

function spyOn(obj, key) {
  return spy(obj, key);
}

function spy(obj, key) {
  let calls = []
  let old = obj[key];

  function _spy(...params) {
    calls.push({
      args: params
    });
  }

  _spy.reset = () => obj[key] = old;
  _spy.calledTimes = () => calls.length;
  _spy.getInvocations = () => {
    let str = '';
    calls.forEach((call, index) => {
      str+= `Invocation ${index + 1}, args: ${call.args} \n`;
    });

    return str;
  }

  _spy._calls = calls;

  obj[key] = _spy;
}

module.exports = {
  spyOn
};

demo.js, for testing it out

and lastly our demo.js file:

const { spyOn } = require('./spy-with-args');
const { it, expect } = require('./util');

function impl(obj) {
  obj.calc('one', 'two');

  obj.calc('three');
}

it('test spy', () => {
  // arrange
  const obj = {
    calc() {}
  }

  spyOn(obj, 'calc');

  // act
  impl(obj);

  // assert
  expect(obj.calc).toHaveBeenCalled();
  expect(obj.calc).toHaveBeenCalledWith('one', 'two');
  expect(obj.calc).toHaveBeenCalledWith('three');
  expect(obj.calc).toHaveBeenCalledWith('one', 'two', 'three');
})

Comments

[Rocco Gränitz]

Hi Chris. Thanks for sharing this great article. I`d like to suggest one improvement. From a security perspective your initial example could be misleading. The decision about shipment is done in client code and could be easily bypassed by a malicious user. I try to make our devs aware of such design weaknesses. Maybe you add some comments to also doublecheck payment on server side or find another example? Best, Rocco

[Chris Noring]

hi Rocco. This is not client code. It's Node.js i.e server side. I can add a comment though cause I agree with you generally, thanks :)