loading...
Cover image for 100% test coverage is not enough...
eBay Tech Berlin

100% test coverage is not enough...

destro_mas profile image Shahjada Talukdar Originally published at ebaytech.berlin ・Updated on ・6 min read

Even if your unit tests cover everything and pass at build time, your app can still be completely broken in production. πŸ§πŸ€”

In web application development, testing is crucial.Β 
You can't build a high-quality app without proper testing.

So today we will talk about TESTING. Hello, Mic, Testing 1,2,3,4 … 🎀

We test in one of two ways:

  • Manual

  • Automated

Manual Testing

As the name suggests, the testing is done manually in this case, by humans.
And we can't deny it, we all do it. But that's not what we'll talk about today.

We will talk about Automated Testing πŸ€–

Automated Testing

There are three different methods of Automated Testing:

  • Unit

  • Integration

  • End-to-End

Let's take a closer look at each approach.

Unit Testing
  • Unit tests take a piece of the product and test that piece in isolation.

  • Unit testing should focus on testing small units.

  • Units should be tested independently of other units.Β 
    This is typically achieved by mocking the dependencies.

Integration Testing
  • Integration testing is when we integrate two or more units.

  • An integration test checks their behavior as a whole, to verify that they work together coherently.

End-to-End Testing
  • End-to-end testing is a technique used to test whether the entire application flow behaves as expected from start to finish.

  • Tests that simulate real user scenarios can easily help to determine how a failing test would impact the user.

Wow, now we know a bit about what those three mean πŸ‘

Now let's dig into Unit Testing and the associated tools.

In the last few years, several tools have been popular, including:

What is Jest? (From its official site)

Jest is a delightful JavaScript Testing Framework with a focus on simplicity.
It works with projects using: Babel, TypeScript, Node, React, Angular, Vue and more!

For this article, we use React examples, but the approach would be similar in other frameworks.

First, let's see how Jest works. We have a module called sum.js -

function sum(a, b) {
  return a + b;
}
module.exports = sum;

We can test this using Jest in sum.test.js like this:

const sum = require('./sum');

test('adds 1 + 2 to equal 3', () => {
  expect(sum(1, 2)).toBe(3);
});

And wow, it will pass. πŸ‘ awesome!!!

But what happens, if we use null or "1" -

sum(null, 2);
sum("1", 2);
// etc...

You might try to add all the possible cases to test it and make it solid. But let's be honestβ€Š-β€Šmost companies don't research all the test cases for any unit.

But for the sake of simplicity, let's assume this function is tested enough to work perfectly. So, we need to find out all the units of the app and test them in the same manner.

And then again, do we really know what the Units of the app are?

For example:

We don't know the value of a and b for the sum function. They come from 2 separate modules.

function getA() {
  // calculations or Remote API call
  return a;
}
module.exports = getA;
function getB() {
  // calculations or Remote API call
  return b;
}
module.exports = getB;

So, to do the sum operation, we need to do something like this:

const a = getA();
const b = getB();
// then the sum op
sum(a, b);

But we are really motivated developers and want to prepare unit test for getA and getB as well as sum, so we think we have 100% Unit test coverage.

All our tests pass separately as unitsβ€Š-β€Šbut when we run the whole thing together in production, it still doesn't work as expected.

Then we think of creating a function named doEverythingAndReturnSum which (as the name suggests) does everything and returns sum:

function doEverythingAndReturnSum() {
  const a = getA();
  const b = getB();
  // then the sum op
  return sum(a, b);
}

Wow, nice.
--But hat is the Unit here?

getA ? getB ? sum? or doEverythingAndReturnSum ? πŸ’­

Alt Text

If we now test all the units separately and feel happy that our app has 100% test coverage, the result might look something like this πŸ™ˆπŸ’₯

So far, we've been looking mostly at the JavaScript code, not the user interface. But as we know, UI testing is even more challenging, as there are multiple layers of code involved:

  • DOM
  • styles
  • events
  • Data coming from Remote APIs
  • Browsers, and so on

When we talk about UI testing for React apps, the first thing that comes to mind is "Components".
Components are the building blocks for a React app.Β 

  • So, of course, we need to test the components.

One of the most popular tools for testing React components is Enzyme.

Enzyme is a JavaScript Testing utility for React that makes it easier to test your React Components' output. You can also manipulate, traverse, and in some ways simulate runtime given the output.

Let's look at an example of how we can unit-test a component with Jest and Enzyme.

We have one component called MyComponent, and we pass a <div> element to it with a class named unique

To test it, we assert that after rendering, the component should include the <div> with the unique class:

import React from 'react';
import { expect } from 'chai';
import { shallow } from 'enzyme';

import MyComponent from './MyComponent';

describe('<MyComponent />', () => {
  it('renders children when passed in', () => {
    const wrapper = shallow((
      <MyComponent>
        <div className="unique" />
      </MyComponent>
    ));
    expect(wrapper.contains(<div className="unique" />)).to.equal(true);
  });
});

Fair enough!

It passes, which means it has a <div> element and the unique class.

But wait...
--What if in the same Commit, I completely destroyed the styling of the unique class? This test will still pass 🀨

There is another way of testing React Components - Jest Snapshot testing. That's also another unit testing mechanism which is not enough on its own. You can read my another post here Jest Snapshot Testing for React Components is useless? Is it slowly dying? πŸ§πŸ€”πŸ˜

Also, we may have hundreds of React Components in the project. (If your project is big enough, you could even have thousands). If we unit-test all the components separately, that doesn't necessarily mean all the components will work together well.

Even if we ignore the remote API for a second, it still doesn't guarantee that.

Oh, another fun thing:
--Our application is a web app and it will run in a browser (and even in multiple browsersβ€Š--β€Šas we don't know which browsers people will use).

But we aren't testing components in a single browser yet. How can we guarantee that it will work properly in different browsers?

Long, long ago in 2015 (long ago, because in the web or JS world, 2015 is considered ancient times), Google posted an article on the Testing Pyramid:

Alt Text

They suggested to do mostly unit testing and not much end-to-end (E2E) testing: "Just Say No to More End-to-End Tests".

Martin Fowler gave his thoughts on the TestPyramid even before in 2012:

Its essential point is that you should have many more low-level UnitTests than high level BroadStackTests running through a GUI.

Alt Text

This thought from Martin seems realistic, so you can actually take your decision based on your financial resources and team strength.

Kent C. Dodds merged these both and came up with this-

Alt Text

Awesome!

Also, he proposed what he calls the "Testing Trophy", which focuses mostly on the integration testing.

Alt Text

With today's web, I think we cannot rely on unit testing alone. It's better to use a lot of integration testing to make the application solid.

As Guillermo Rauch said -

Alt Text

The "application" concept on the modern web is changing rapidly. The browser can do a lot more than before. JavaScript is evolving with ESNext. In fact, time has also changed the testing tools. Nowadays, we have more tools and frameworks to do integration and E2E testing in a much better way.

I plan to show how we can do integration and E2E testing in a better way for modern web applications in future articles.

Till then,
Cheers!
πŸ‘‹

Discussion

pic
Editor guide
Collapse
elmuerte profile image
Michiel Hendriks

100% coverage is useless. 80%-90% coverage by unit tests is generally good enough, above that you are starting to waste time. Time better spend on other ways of testing, like this article mentions.

Besides these forms of tests

  • Unit
  • Integration
  • End-to-End

There are also other important tests which are often forgotten:

  1. Load/stress tests: To find out how well the system behaves under high load. It builds upon integration or e2e tests, but with 2-3 times the peak volume you would normally expect. It is a test of scalability.
  2. Endurance tests: To find out if your system keeps running for days, weeks, months. Similar to a load test, but instead of stressing the system you want to find out if the system remains stable when it processes a peak load for a day, or week. This helps you to find resource leaks, or performance degradation. You may also want to build is some load trends, like the usual sine of peak load to none, back to peak again. And maybe add a large pause in the middle of every 3rd peak load. To test 100% to 0% and back to 100%, this quite often has side effects.
Collapse
destro_mas profile image
Shahjada Talukdar Author

Thanks for your views! πŸ‘‹
Yeah, fully agree with Load tests and etc.

Definitely those are important and forgotten stuff.
This Article was mostly written from the initial ground as a first step.
I will try to continue writing more on other detailed ways.

What kind of tools you normally use for Load or Endurance testing?

Collapse
elmuerte profile image
Michiel Hendriks

Which tools are available really depends on the software under test. Tools like JMeter or Gatling are good if you can produce load with rather simple sequence of HTTP calls.

Taurus is an interesting "wrapper" around a bunch of test frameworks to turn them into load tests.

The software I work on has rather complex asynchronous "conversations" which mostly happen via message queues. I was able to create an integration test in JMeter, but it was unsuitable for performance load testing as the test framework was the bottleneck. So I need to develop our own tooling. For this I am planning to use the Apache Camel, and maybe re-use parts of the Citrus Framework (we are a Java shop).

So maybe there is no tool available for your software which can produce significant load, you can often use parts of other integration testing frameworks. Most integration frameworks are not geared towards running the same tests over and over again in multiple threads. But there is a chance you can create a wrapper around this framework which will do exactly that (like Taurus has done).

Note, with load and endurance testing you do not have to fully validate everything. Shallow verification are often sufficient enough. You do not want the verification tests to slow how the test tool. Detailed validation should have been covered in the integration tests. If the tooling support is, you could do samples of detail validation (e.g. every 1000th run is verified to greater detail.)

Thread Thread
destro_mas profile image
Shahjada Talukdar Author

As you said, you are a Java shop, you mostly do API implementation and do load testing for the REST APIs only? or you also do the testing for the whole web app end-to-end?

Thread Thread
elmuerte profile image
Michiel Hendriks

It's message oriented enterprise software for business process automation. So the load tests we do are via message queues for specific business processes. We don't really have REST APIs. It's complex data structures with somewhat long complex transactions, often various sequential asynchronous transactions which eventually produce an output in the form of a message on a different queue.

So the load testing is in the form of pumping messages on a queue, with a different process reading messages from a different queue, which is needed to send a follow message. This a few times, to complete a whole "operation". Not the easiest to load test.

Collapse
Sloan, the sloth mascot
Comment deleted
Collapse
nickholmesde profile image
Nick Holmes

Unit Tests (100% or otherwise) are not an appropriate tool for assessing the design of your code.

Could be 100% Code Coverage, properly modularized & SOLID, and still have a blatant bug in it;

Here's some code:

public string foo(bool a, bool b)
{
    string x = "Hello World";

    if(a)
        x = null;

    if(b)
        x = x.ToUpper();

    return x;
}

and here are the tests for 100% code coverage:

Assert.AreSame("Hello World", target.foo(false, false));
Assert.AreSame(null, target.foo(true, false));
Assert.AreSame("HELLO WORLD", target.foo(true, true));
Collapse
mburszley profile image
Maximilian Burszley

"Blatant bug" or intentionally sabotaged?

Collapse
destro_mas profile image
Shahjada Talukdar Author

That's a good point you mentioned.
If you can really do "properly modularized & follows SOLID", then Unit testing can help of course, for good. I am not against Unit Testing. I just wanted to make sure - this is not enough, not the only thing!
But do you think it's really easy to do "properly modularized & SOLID" and find out all the cases for Unit testing?
Maybe you and your team do it properly.
Actually, in reality, it does not happen always :(

Collapse
armw4 profile image
Antwan R. Wimberly

Thank you!!! This solidified the points I’ve been trying to exercise to less experienced developers and less technical managers who pretend to be aware of these nuances, quirks, and idiosyncrasies. πŸ’₯!!!

Collapse
destro_mas profile image
Shahjada Talukdar Author

You are welcome!
If you like, feel free to share πŸ‘‹

Collapse
armw4 profile image
Antwan R. Wimberly

Haha for sure I’ve already posted a link to your article as a comment within my Refactoring article on GitHub - my guy 🀝 πŸ―πŸ¦πŸ‘

Thread Thread
destro_mas profile image
Shahjada Talukdar Author

πŸ˜€πŸ€

Collapse
destro_mas profile image
Shahjada Talukdar Author

I agree with your comment, Andy!
From my point of view, Having 20%-50% Unit test coverage can be enough just to make sure the app is not broken in compile/build time. I am saying 20-50 because it can vary person-to-person, company-to-company or product-to-product.

Other than that the app should have lots of Integration and E2E tests!

destro_mas profile image
Shahjada Talukdar Author

Yeah, the concept is the point here!