[Puppeteer][Mocha] Upgrade your implementation code with coverage.

Kirill Zhdanov — Wed, 21 Aug 2019 07:49:23 +0000

Test pyramid

Since Puppeteer releases, end-2-end testing becomes a fast and reliable way for testing features. Most things that you can do manually in the browser can be done using Puppeteer. Even more, headless Chrome reduces the performance overhead and native access to the DevTools Protocol makes Puppeteer awesome. Imagine, each time we develop front-end we just check the final view in the browser, and without TDD we faced with Test Pyramid Ice cream Anti-pattern But we like Ice cream, so why we need a trade-off? I show you how to upgrade your sources with tests no matter what engine your use with confidence that your application works as expected because Puppeteer will check Features instead of you.

Setup

I have complete step-by-step instruction README.md based on a simple project I forked from and had provided it with feature-rich test upgrade to show off. So, if you have one other please:
1) Install dependencies in your root
npm i puppeteer mocha puppeteer-to-istanbul nyc -D
2) Expose your instance on endpoint (my light solution for *.html http-server)
3) Create test directory and fill {yourFeature}_test.js with next suitable template (note before and after hooks), try to extend it with your project-specific selectors and behaviors:

const puppeteer = require('puppeteer');
const pti = require('puppeteer-to-istanbul');
const assert = require('assert');

/**
 * ./test/script_test.js
 * @name Feature testing
 * @desc Create Chrome instance and interact with page.
 */

let browser;
let page;

describe('Feature one...', () => {
    before(async () => {
        // Create browser instance
        browser = await puppeteer.launch()
        page = await browser.newPage()
        await page.setViewport({ width: 1280, height: 800 });
        // Enable both JavaScript and CSS coverage
        await Promise.all([
            page.coverage.startJSCoverage(),
            page.coverage.startCSSCoverage()
          ]);
        // Endpoint to emulate feature-isolated environment
        await page.goto('http://localhost:8080', { waitUntil: 'networkidle2' });
    });
    // First Test-suit
    describe('Visual regress', () => {
        it('title contain `Some Title`', async () => {
            // Setup
            let expected = 'Some Title';
            // Execute
            let title = await page.title();
            // Verify
            assert.equal(title, expected);
        }).timeout(50000);

    });
    // Second Test-suit
    describe('E2E testing', () => {
        it('Some button clickable', async () => {
            // Setup
            let expected = true;
            let expectedCssLocator = '#someIdSelector';
            let actual;
            // Execute
            let actualPromise = await page.waitForSelector(expectedCssLocator);
            if (actualPromise != null) {
                await page.click(expectedCssLocator);
                actual = true;
            }
            else
                actual = false;
            // Verify
            assert.equal(actual, expected);
        }).timeout(50000);
    // Save coverage and close browser context
    after(async () => {
        // Disable both JavaScript and CSS coverage
        const jsCoverage = await page.coverage.stopJSCoverage();
        await page.coverage.stopCSSCoverage();

        let totalBytes = 0;
        let usedBytes = 0;
        const coverage = [...jsCoverage];
        for (const entry of coverage) {
            totalBytes += entry.text.length;
            console.log(`js fileName covered: ${entry.url}`);
            for (const range of entry.ranges)
                usedBytes += range.end - range.start - 1;
        }
        // log original byte-based coverage
        console.log(`Bytes used: ${usedBytes / totalBytes * 100}%`);
        pti.write(jsCoverage);
        // Close browser instance
        await browser.close();
    });
});

Execute

Run your test described above against scripts on endpoint with mocha command
Get coverage collected while test run with nyc report.
I suggest you instrument your package.json with the next scripts makes it super easy to run tasks like npm test or npm run coverage

  "scripts": {
    "pretest": "rm -rf coverage && rm -rf .nyc_output",
    "test": "mocha --timeout 5000 test/**/*_test.js",
    "server": "http-server ./public",
    "coverage": "nyc report --reporter=html"
  },

Coverage

In my project I have coverage about 62%
We can report it as html and look closer
You can see that Branches and Functions both 100% covered. While I tested Puppeteer coverage feature (like Coverage devTool) I filed that bug

[Bug] Incorrect branches hits coverage statistic #22

storenth posted on Dec 27, 2018

When nyc report --reporter=html done, I try to look at the ./coverage/index.html and found a great defect in Branch coverage number, it always 100%. To validate this issue I suggest clone this simple repo for local reproduce.

View on GitHub

Unit vs E2E testing wars

Someone who has enough passion to research without getting bored I tell what, we need more attention to unit testing frameworks like Mocha used to write tests from Unit to Acceptance but not to unit or end tests themselves. No matter what test you write if your codebase covered I think. Times have changed. Now, with coverage available feature other tools like the Traceability matrix as a measure of quality look ugly, because stakeholders still need to believe in tester's word. But pay attention to the forth best practices in the domain of code coverage from Google to use coverage information pragmatically.

Contribute

I strongly recommend take some time and review my github-working-draft project before you will stuck.
I will appreciate any collaboration and feedback. Feel free to reach me any questions.

Testing. Back to basics.

Kirill Zhdanov — Fri, 16 Aug 2019 06:00:02 +0000

I’ve been asked several times about the difference between Quality Assurance Engineer (QA) and Tester (QC) and I figured out that even people who develop software confused such simple things, so, what about the others? But you know what, they don't care, and it is their right!

People just need quality products! Managers want bug [1] free software of higher quality, users want quality apps, and this can be achieved through testing [2] no matter what. But what does that mean? What is Quality? It is all about satisfaction, that's it! The measure of quality is the happiness of the end-users! How to reach that abstraction becomes a true test for team [3].

While developing any product, especially software without testing, we face risks [4] and this is likely related to funds loss [5]. So, if you don't care about testing but you need quality think twice and let quality assurance processes get to work. This applies to all team members, but mostly to Engineers who master test design [6] - Quality Assurance Engineers.

So, the intention that man should be happy with quality apps is solved with best practice approach [7]. Once you get involved in testing you will learn about types of testing [8] and test documentation like test plan. Then you can handle test cases [9] based on test design techniques [10]. After all, you will discover automation and by then will know the difference between QA and QC [11].

Glossary, simple way:

mismatch between actual and expected behavior
complete product examination through verification, validation based on specs for issue detection https://smartbear.com/blog/test-and-monitor/verification-and-validation-the-difference/
people who develop applications like designers, programmers, testers
(a) the possibility of something bad happening (by Cambridge dictionary); (b) action in the hope of a good outcome (by google dictionary)
(a) 125 Million Mars Probe Lost Due to Simple Math Error http://articles.latimes.com/1999/oct/01/news/mn-17288 (b) The many human errors that brought down the Boeing 737 Max https://www.theverge.com/2019/5/2/18518176/boeing-737-max-crash-problems-human-error-mcas-faa
testing pyramid, a framework for designing a test suite that optimizes for speed of tests execution with confidence that your application works as expected https://www.codecademy.com/articles/tdd-testing-pyramid
locate Seven Testing Principles at International Software Testing Qualifications Board https://www.istqb.org/downloads/send/51-ctfl2018/208-ctfl-2018-syllabus.html
the different types of software testing by Atlassian https://www.atlassian.com/continuous-delivery/different-types-of-software-testing
steps to reproduce with expected results based on requirements on the application
https://medium.com/@jeffereydunn90/common-test-design-techniques-you-must-know-59df3d47409
find the answer here https://www.portnov.com/session-1-questions

Books:

Testing Dot Com, or allowance for the abuse of bugs in Internet startups, by Savin R. 2007
A Practical Guide to Continuous Delivery by Eberhard W. 2017
Software Testing Automation Tips, by Alpaev G. 2017

DEV Community: Kirill Zhdanov