Application testing, the good, the bad and the ugly: from a senior programmer's point of view

What framework do you use for testing?

This is a commonly asked question, but for me, it's a superficial and potentially hazardous inquiry.

Unit tests alone cannot guarantee that a particular feature will function correctly in production.

How many times I saw code being committed to production, just to witness the system collapse moments after deployment. 🔥

In fact, that is so common that I once prohibited any new releases from being made on Fridays or overnight.

The truth is, a component may work flawlessly in a test environment, but there are other crucial factors to consider such as its performance, safety, target devices, and compatibility with production dependencies.

To illustrate, let's say a junior developer (not you! 😅) creates a function that grows exponentially with the number of requests, an O(2ⁿ) of the worst kind.

This function could pass a unit test despite its suboptimal performance and could automatically make its way into production with an automated CI/CD pipeline.

To prove my point, I'll provide an example below.

If you prefer, you can skip the math and go straight to the result, or simply take my word for it.

You can Skip the math part and go straight to the conclusion

++++++++++++++++++++++++++++++++++++++++++++++++++

Calculate the nth term of a Fibonacci sequence

The Fibonacci sequence, has the numeral 1 as the first and second term of the order, and the following elements are originated by the sum of its two predecessors, so:

1
1 + 1 = 2
1 + 2 = 3
3 + 2 = 5

And so on…

1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610, 987, 1597, 2584, 4181...

In this solution, we use a function that already returns 1 if n is less than 2 (because we already know that the first two terms of a Fibonacci sequence are 1) then we go backwards calculating recursively

function fib(n) {
  if (n <= 2) return 1
  return fib(n - 1) + fib(n - 2)
}

If we run this function in Jest, it will return as correct.

fib(5) must be 5

fib(39) should be 63245986

fib(40) must be 102334155

“There are no failing tests, congratulations!”

Click here to preview in CodeSandbox

The test may not reveal that, while the function produces the expected output, it also leads to an exponential increase in the required computations to solve the problem.

Simplifying this in BIG-O notation, we get an O(2ⁿ) complexity, which is terrible.

If we run fib(n) we get the following:

in fib(5) the calculation time was 0 ms

in fib(39) the time passed to 489 ms

in fib(40) the time passed to 779 ms

Notice that it didn't take much for the running time to double (from 39 to 40).

end of the boring math part
++++++++++++++++++++++++++++++++++++++++++++++++++

And what to do then to minimize errors in production?

It depends!

What do you mean??? 🤯

The testing approach to use depends on various factors, such as the availability of time/money, the complexity of the component, and the risks involved.

Common sense and experience play a significant role in deciding the level of testing required.

For instance, if the project is small and low-risk, there is no need to perform complex tests.

However, if the project involves high-risk factors like money, security, or high volume of requests, it is advisable to use a comprehensive testing arsenal.

Some strategies to reduce risks in production include:

Run performance tests on every commit. By doing this, you decrease the risk of performance regressions early on. The earlier you detect it, the easier it will be to revert.
Do canary tests whenever the functionality has a high impact or risk for the user or the application. Testing the application within a smaller, controlled audience will eliminate a lot of risk without taking away external variables. Netflix, for example, uses its employees' personal accounts to test functionality changes before releasing them to the rest of the public.
Use external error tracking tools. I always recommend Sentry (https://sentry.io/) as it is excellent at detecting errors in real time. Whenever an error happens to a user, you receive a complete report of what happened, the type of browser used and many other insights that will help you to isolate and resolve the bug.
Evaluate not only the inputs and outputs of your components. Make an analysis of the algorithms (I'm a fan of Big O Notation 💘) and apply, whenever possible, software design patterns, the famous design patterns.

Before signing off, I want to share an O(n) solution that is more performant than the example presented initially.

Cheers!

Renato

Bonus:

Correcting the example function through memoization

function fib(n, memo = []) {
  if (memo[n] !== undefined) return memo[n]
  if (n <= 2) return 1
  var res = fib(n - 1, memo) + fib(n - 2, memo)
  memo[n] = res
  return res
}