React/Redux application with Azure DevOps: Part 9 Integration Test and End-to-End test

In the previous post, I updated existing CI/CD pipeline to support the new application.

In this article, I look into integration test and e2e testing.

Unit Test, Integration Test and End to End test

So far, I keep adding unit test. Isn't it enough to test the application? The answer is, NO. Not enough.

When Unit Test goes wrong

The principal of unit testing is to remove all the dependency so that I can purely test the logic of the function.

For example, if I want to test funcA which looks like below code,

const myFunc = {

    funcA(x) {
        return this.funcB(x);
    },

    funcB(x) {
        if (x % 2 === 0) {
            return true;
        }
        else {
            return false;
        }
    }
}

module.exports = myFunc;

then my test should look like this.

const myFunc = require('./myFunc');

it('Should return true', () => {
    jest.spyOn(myFunc, 'funcB').mockReturnValueOnce(true);
    const x = 2; 
    expect(myFunc.funcA(x)).toBe(true);
});

it('Should return false', () => {
    jest.spyOn(myFunc, 'funcB').mockReturnValueOnce(false);
    const x = 3; 
    expect(myFunc.funcA(x)).toBe(false);
});

I mock the behavior of funcB so that the test won't be affected by funcB result, and I can focus on logic inside funcA. This looks right, until I change implementation of funcB.

Even though I changed funcB like below, all the unit tests still pass.

const myFunc = {

    funcA(x) {
        try {
            return this.funcB(x);
        }
        catch {
            return false;
        }
    },

    funcB(x) {
        if (x % 2 === 0) {
            return true;
        }
        else {
            throw new Error("something went wrong");
        }
    }
}

module.exports = myFunc;

Yes, it's unit test bug, but it happens a lot as I forget to update dependent test when I change implementation. This is an example of two functions in a module, but image, if the dependency comes from different module, then it's even harder to track the changes.

Integration Test

Integration test runs some amount of dependent code as well. In the example above, I can find the issue if I won't mock the funcB.

const myFunc = require('./myFunc');

it('Should return true', () => {
    jest.spyOn(myFunc, 'funcB').mockReturnValueOnce(true);
    const x = 2; 
    expect(myFunc.funcA(x)).toBe(true);
});

it('Should return false', () => {
    jest.spyOn(myFunc, 'funcB').mockReturnValueOnce(false);
    const x = 3; 
    expect(myFunc.funcA(x)).toBe(false);
});

it('Should return true without mock', () => {
    const x = 4; 
    expect(myFunc.funcA(x)).toBe(true);
});

it('Should return false without mock', () => {
    const x = 5; 
    expect(myFunc.funcA(x)).toBe(false);
});

When I run the test, I can see the results like below.

End to End (E2E) test

Even though integration test works good, I don't test entire call stack. For example, I still mock backend service when I write integration test code.

But, from user point of view, it's important to test including backend. To do so, I need to run the application and use browser to test in case of web application. This is called E2E test where I don't mock anything.

Of course everyone or every projects may have different definition, but this is my definition.

Integration Test for React

I use Shallow rendering to test component which won't fully render child components. This is perfect for unit test. However, I need to render child components when it comes to integration test.

I also need to decide which part I should mock.

This time, I decided to mock axios module, and write function test from App level.

I can use @testing-library/react to render the component into DOM, and trigger actions such as bottom click or enter input value.

Initial page

When I open the application, it looks like this.

Let's implement integration test for this.

1. Install types.

npm install --save-dev @types/testing-library__dom @types/testing-library__react

2. Add App.Integration.test.tsx in src folder and add following code.

• I don't mock redux store, so just create store and wrap with Provider
• Mock axios get function to return dummy vote
• Look for component as "cat:5" and "dog:5"
• use debug() to display the result

/// App.Integration.test.tsx

import React from 'react';
import App from './App';
import { render, waitForDomChange, cleanup } from  '@testing-library/react';
import { Provider } from 'react-redux';
import store from './redux/store';
import axios from 'axios';
import { Vote, VoteData } from './api/voteAPI';

afterEach(() => {
  jest.clearAllMocks();
  cleanup();
});

it('should render dummyVote result', async () => {
  const dummyVote = new Vote('1', [5, 5], ['cat', 'dog']);
  jest.spyOn(axios, 'get').mockResolvedValueOnce({ data: new VoteData(dummyVote)});
  const { debug, getAllByText } = await render(<Provider store={store}><App /></Provider>);
  await waitForDomChange();
  debug();
  expect(getAllByText(/cat:5/).length).toBe(1);
  expect(getAllByText(/dog:5/).length).toBe(1);
});

3. Run the test. I can see debug result as well as passed result. It looks like I need to re-think about what object to pass to my action though.

render API renders component including child components. Then I use waitForDomChange to wait until dom is refreshed as a result of useEffect.

Click event

Now, I could test the initial page. Let test when I click buttons such as "+" for cat or "add candidate". One thing I need to consider here is which component I should test.

As I mock axios, it returns desired result data whatever the input is. But it is called by voteAPI and I want to make sure expected function has been called with expected input.

Let's do it.

1. Replace import section to import additional element.

import React from 'react';
import App from './App';
import { render, fireEvent, waitForDomChange, cleanup } from  '@testing-library/react';
import { Provider } from 'react-redux';
import store from './redux/store';
import axios from 'axios';
import voteAPI, { Vote, VoteData } from './api/voteAPI';

2. Add test. This time, I wait for "cat:5" first, then click the first found button which has "+" as text by using getAllByText.

Another thing I test here is to see if "updateAsync" on voteAPI is called with expected parameter which proves that increment logic works as expected.

it('should increment cat', async () => {
    const dummyVote = new Vote('1', [5, 5], ['cat', 'dog']);
    jest.spyOn(axios, 'get').mockResolvedValue({ data: new VoteData(dummyVote) });
    const dummyCatIncrementVote = new Vote('1', [6, 5], ['cat', 'dog']);
    jest.spyOn(axios, 'put').mockResolvedValue({ data: new VoteData(dummyCatIncrementVote) });
    const updateAsyncFn = jest.spyOn(voteAPI.prototype, 'updateAsync')

    const { getAllByText } = await render(<Provider store={store}><App /></Provider>);
    await waitForElement(() => getAllByText(/cat:5/));

    fireEvent.click(getAllByText(/\+/)[0]);
    await waitForDomChange();
    expect(updateAsyncFn).toBeCalledTimes(1);
    expect(updateAsyncFn).toBeCalledWith(dummyCatIncrementVote);
    expect(getAllByText(/cat:6/).length).toBe(1);
    expect(getAllByText(/dog:5/).length).toBe(1);
});

3. Add another test for decrement scenario.

it('should decrement cat', async () => {
    const dummyVote = new Vote('1', [5, 5], ['cat', 'dog']);
    jest.spyOn(axios, 'get').mockResolvedValue({ data: new VoteData(dummyVote) });
    const dummyCatDecrementVote = new Vote('1', [4, 5], ['cat', 'dog']);
    jest.spyOn(axios, 'put').mockResolvedValue({ data: new VoteData(dummyCatDecrementVote) });
    const updateAsyncFn = jest.spyOn(voteAPI.prototype, 'updateAsync')

    const { getAllByText } = await render(<Provider store={store}><App /></Provider>);
    await waitForElement(() => getAllByText(/cat:5/));

    fireEvent.click(getAllByText(/-/)[0]);

    await waitForDomChange();
    expect(updateAsyncFn).toBeCalledTimes(1);
    expect(updateAsyncFn).toBeCalledWith(dummyCatDecrementVote);
    expect(getAllByText(/cat:4/).length).toBe(1);
    expect(getAllByText(/dog:5/).length).toBe(1);
});

4. Finally, add candidate scenario.

it('should add rabbit', async () => {
    const dummyVote = new Vote('1', [5, 5], ['cat', 'dog']);
    jest.spyOn(axios, 'get').mockResolvedValue({ data: new VoteData(dummyVote) });
    const dummyRabbitVote = new Vote('1', [5, 5, 0], ['cat', 'dog', 'rabbit']);
    jest.spyOn(axios, 'put').mockResolvedValue({ data: new VoteData(dummyRabbitVote) });
    const updateAsyncFn = jest.spyOn(voteAPI.prototype, 'updateAsync')

    const { getAllByText, getByTestId, getByText } = await render(<Provider store={store}><App /></Provider>);
    await waitForElement(() => getAllByText(/cat:5/));

    fireEvent.change(getByTestId('input'), { target: { value: 'rabbit' } });
    fireEvent.click(getByText(/Add candidate/));

    await waitForDomChange();
    expect(updateAsyncFn).toBeCalledTimes(1);
    expect(updateAsyncFn).toBeCalledWith(dummyRabbitVote);
    expect(getAllByText(/cat:5/).length).toBe(1);
    expect(getAllByText(/dog:5/).length).toBe(1);
    expect(getAllByText(/rabbit:0/).length).toBe(1);
});

5. Run the test and confirm the results.

End to End test

Amongst many choices, I found puppeteer interesting, which is "headless" chrome to interact with web application. It is not design purely for testing, but there are so many examples which explain how to use jest and puppeteer for e2e test.

There are also several useful libraries I utilize this time.

jest-puppeteer: This makes setup easier.
jest-junit: This writes test results with junit format.

To run e2e test, I have to run the server first. There are several choices, but jest-puppeteer start the server before testing, and shutdown when all the tests completed. Isn't it great?? I love the feature.

I wonder where I should setup e2e testing, as server lives in react-backend folder, but I decided to create separate folder this time.

Setup

Let's add npm project inside the application.

1. Add e2e folder and initialize npm project. Run the command in my-react-redux-app folder.

mkdir e2e
cd e2e
npm init -y

2. Install modules.

npm install --save-dev axios jest jest-junit jest-puppeteer puppeteer ts-jest typescript @types/axios @types/expect-puppeteer @types/jest @types jest-environment-puppeteer @types/puppeteer

3. Add jest.config.js. I usually set ts-jest for preset, but as I use puppeteer this time, I move ts-jest to transform.

/// jest.config.js

module.exports = {
  preset: 'jest-puppeteer',
  transform: {
        "^.+\\.ts?$": "ts-jest"
  },
  reporters: [
    "default", "jest-junit"
  ]
};

4. Add jest-puppeteer.config.js. This is where I can specify how to start server for test. I explicitly increased the launchTimeout (default:5000)

/// jest-puppeteer.config.js

module.exports = {
  server: {
    command: 'npm start',
    port: 8081,
    launchTimeout: 50000
  },
}

5. Update package.json script section. If I setup this test inside react-backend, I didn't have to write start like this, but at least it works.

• You can find all parameters in Jest CLI option page.

"scripts": {
  "test": "jest --runInBand",
  "start": "cd ../react-backend && node -r module-alias/register ./dist"
},

Add test

I focus on three scenario only. I want to test increment, decrement and add candidate scenario.

One thing I knew was that I should have created dev database, but I didn't. So I reluctantly overwrite production data whenever I do test. It never happens in real-world, but this is just my learning how to use technologies, so I just did it.

1. Add tests folder and app.spec.ts in the folder.

• Reset the test data "beforeAll" test
• Reset the test data "afterEach" test

import * as puppeteer from 'puppeteer';
import axios from 'axios';
import { Vote , VoteData } from '../../src/api/voteAPI';

var browser: puppeteer.Browser = null;
const baseUrl: string = process.env.baseUrl || "http://localhost:8081";
const testVote = new VoteData(new Vote('1', [1, 0], ['cat', 'dog']));

beforeAll(async () => {
   await axios.put(`${baseUrl}/api/votes`, testVote);
});

afterEach(async () => {
   await browser.close();
   await axios.put(`${baseUrl}/api/votes`, testVote);
});

2. Add increament test.

• After 'goto' the page, I wait a second as I didn't find a good way to wait until 'useEffect' completed
• Do the same after click
• Compare vote count before and after the click

it('should increment', async () => {
   browser = await puppeteer.launch();
   var page = await browser.newPage();
   await page.goto(baseUrl, {waitUntil:"networkidle0"});
   const initial = await page.evaluate(() => { return document.querySelector('.voteBox div').textContent });
   const initialValue = initial.split(':')[1];
   await page.click('.voteBox button');
   await page.waitFor(1000);
   const incremented = await page.evaluate(() => { return document.querySelector('.voteBox div').textContent });
   const incrementedValue = incremented.split(':')[1];
   expect(Number.parseInt(initialValue) + 1).toBe(Number.parseInt(incrementedValue));
})

3. Add decrement, too.

• Almost identical to increment test but the difference is how I find "-" button, which maybe better to give id or class so that it is easier to access

it('should decrement', async () => {
   browser = await puppeteer.launch();
   var page = await browser.newPage();
   await page.goto(baseUrl, {waitUntil:"networkidle0"});
   const initial = await page.evaluate(() => { return document.querySelector('.voteBox div').textContent });
   const initialValue = initial.split(':')[1];
   await page.click('.voteBox :nth-child(3)');
   await page.waitFor(1000);
   const decremented = await page.evaluate(() => { return document.querySelector('.voteBox div').textContent });
   const decrementedValue = decremented.split(':')[1];

   expect(Number.parseInt(initialValue) - 1).toBe(Number.parseInt(decrementedValue));
})

4. Finally, add candidate.

it('should add rabbit', async () => {
   browser = await puppeteer.launch();
   var page = await browser.newPage();
   await page.goto(baseUrl, {waitUntil:"networkidle2"});
   await page.type(".candidateBox input", "rabbit");
   await page.click('.candidateBox button');
   await page.waitFor(1000);
   const voteBoxCounts = await page.evaluate(() => { return document.querySelectorAll('.voteBox').length });
   expect(voteBoxCounts).toBe(3);
})

5. Run the test and check the result.

npm test

CI/CD

I completed implementing both integration and e2e test. So let's CI/CD.

1. Update azure-pipeline.yml first. I just added e2e test part, as integration test runs as with unit test.

# Node.js React Web App to Linux on Azure
# Build a Node.js React app and deploy it to Azure as a Linux web app.
# Add steps that analyze code, save build artifacts, deploy, and more:
# https://docs.microsoft.com/azure/devops/pipelines/languages/javascript

trigger:
  branches:
    include:
    - master
  paths:
    exclude:
    - azure-pipelines.yml

variables:

  # Azure Resource Manager connection created during pipeline creation
  azureSubscription: '2e4ad0a4-f9aa-4469-be0d-8c8f03f5eb85'

  # Web app name
  devWebAppName: 'mycatdogvoting-dev'
  prodWebAppName: 'mycatdogvoting'

  # Environment name
  devEnvironmentName: 'Dev'
  prodEnvironmentName: 'Prod'

  # Agent VM image name
  vmImageName: 'ubuntu-latest'

stages:
- stage: Build
  displayName: Build stage
  jobs:  
  - job: Build
    displayName: Build
    pool:
      vmImage: $(vmImageName)

    steps:
    - task: DownloadSecureFile@1
      name: productionEnv
      inputs:
        secureFile: 'production.env'

    - task: DownloadSecureFile@1
      name: developmentEnv
      inputs:
        secureFile: 'development.env'

    - task: DownloadSecureFile@1
      name: testEnv
      inputs:
        secureFile: 'test.env'

    - script: |
        mkdir $(System.DefaultWorkingDirectory)/react-backend/env
        mv $(productionEnv.secureFilePath) $(System.DefaultWorkingDirectory)/react-backend/env
        mv $(developmentEnv.secureFilePath) $(System.DefaultWorkingDirectory)/react-backend/env
        mv $(testEnv.secureFilePath) $(System.DefaultWorkingDirectory)/react-backend/env
      displayName: 'copy env file'
    - task: NodeAndNpmTool@1
      inputs:
        versionSpec: '12.x'

    - script: |
        npm install
        CI=true npm test -- --reporters=jest-junit --reporters=default
        npm run build
      displayName: 'test and build frontend'

    - script: |
        cd react-backend
        npm install
        npm run test
        npm run build
      displayName: 'test and build backend'

    - script: |
        cd e2e
        npm install
        npm run test
      displayName: 'e2e test'

    - task: PublishTestResults@2
      inputs:
        testResultsFormat: 'JUnit'
        testResultsFiles: |
          junit.xml
          **/*junit*.xml
        failTaskOnFailedTests: true

    - task: ArchiveFiles@2
      displayName: 'Archive files'
      inputs:
        rootFolderOrFile: '$(Build.SourcesDirectory)/react-backend'
        includeRootFolder: false
        archiveType: zip
        archiveFile: $(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip
        replaceExistingArchive: true

    - upload: $(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip
      artifact: drop

- stage: DeployToDev
  displayName: Deploy to Dev stage
  dependsOn: Build
  condition: succeeded()
  jobs:
  - deployment: Deploy
    displayName: Deploy to Dev
    environment: $(devEnvironmentName)
    pool: 
      vmImage: $(vmImageName)
    strategy:
      runOnce:
        deploy:
          steps:            
          - task: AzureRmWebAppDeployment@4
            displayName: 'Azure App Service Deploy: $(devWebAppName)'
            inputs:
              azureSubscription: $(azureSubscription)
              appType: webAppLinux
              WebAppName: $(devWebAppName)
              packageForLinux: '$(Pipeline.Workspace)/drop/$(Build.BuildId).zip'
              RuntimeStack: 'NODE|12-lts'
              StartupCommand: 'npm run start -- --env=development'

- stage: DeployToProd
  displayName: Deploy to Prod stage
  dependsOn: DeployToDev
  condition: succeeded()
  jobs:
  - deployment: Deploy
    displayName: Deploy to Prod
    environment: $(prodEnvironmentName)
    pool: 
      vmImage: $(vmImageName)
    strategy:
      runOnce:
        deploy:
          steps:            
          - task: AzureRmWebAppDeployment@4
            displayName: 'Azure App Service Deploy: $(prodWebAppName)'
            inputs:
              ConnectionType: 'AzureRM'
              azureSubscription: '$(azureSubscription)'
              appType: 'webAppLinux'
              WebAppName: '$(prodWebAppName)'
              packageForLinux: '$(Pipeline.Workspace)/drop/$(Build.BuildId).zip'
              RuntimeStack: 'NODE|12-lts'
              StartupCommand: 'npm run start'

2. Update .gitignore to add e2e part.

# e2e
/e2e/node_modules
/e2/junit*.xml
...

3. Commit the change.

git add .
git commit -m "add integration and e2e test"
git pull
git push

4. Confirm the pipeline runs successfully. I can see the e2e test results, too. The reason why duration exceeds 30 minutes is that I forget to "approve" to move to Prod stage :P

Summary

In this article, I implemented integration test and e2e test. There so many other way to achieve the same or even better, but at least I could do what I wanted at first place.

I will play with new libraries I used this time to see if I can find better way to tests.