DEV Community: Scott White

Using Google's OCR API with Puppeteer for Visual Testing

Scott White — Mon, 08 Feb 2021 23:11:53 +0000

The Puppeteer framework is a Node.js library that can be used for automated browser testing. This library provides high-level access to chromium-based browsers through the dev tools protocol. In automated testing, you can use assertion libraries like chai.js or should.js to make assertions about elements or objects that should appear in an application.

One of the most common scenarios in automated testing is to assert if a word or phrase is displayed in a web application. There are few methods available for a developer to accomplish this task.

One of them is to iterate through all the elements in a web page to look for the matching word. However, this method is inefficient and error-prone. Besides, querying every element is time-consuming and will result in incorrect assertions as you have not scoped any part of the page, meaning the word could appear in unexpected locations.

   // Iterate through all elements
          const contents = await page.$$('*');

          for (let i = 0; i < contents.length; i++) {
              const text_value = await (await contents[i].getProperty('innerText')).jsonValue();
              if (text_value.match(/Hello/g)) {
                  console.log(text_value);
              }
          }

A better method is selecting elements using the XPath to find the element which contains the specified word. This is far simpler and more efficient than iterating through all the elements. However, without a defined scope, this method will also provide results from the whole web page.

          // Using XPath
          const title = await page.$x("//h1[contains(text(), 'Domain')]");
          let text = await page.evaluate(h1 => h1.textContent, title[0]);
          console.log(text)

The most effective method would be to use optical character recognition (OCR) to specify a part of the page and carry out searching within that scope to find the location of the word. Using OCR allows you to utilize a powerful character recognition algorithm without having to go through HTML elements.

What is OCR?

Optical Character Recognition, or OCR, is a technology that enables you to convert images of text content into machine-readable, digital data. OCR can convert any type of text content, whether it’s handwritten or printed, into editable and searchable data using an image of that text content. This is done by processing the image through a machine-learning algorithm to clean and recognize each character of a specific image.

There are multiple open-source OCR tools like pytesseract or EasyOCR, which can be used to integrate OCR functionality into a program. However, these tools require significant configurations to get up and running to provide results with an acceptable accuracy level.

Google provides a ready-made solution to integrate OCR functionality to an application using its Vision API. It will be used in this tutorial since it abstracts most of the model fine-tuning. The Vision API provides two distinct detection types to extract text from images, as mentioned below.

TEXT_DETECTION - This will detect and extract text from any provided image. The resulting JSON will include the extracted string and individual words with their bounding boxes.

DOCUMENT_TEXT_DETECTION - This will also detect and extract text from any image but is optimized to be used for dense text and documents. Its JSON output will feature all the details, including extracted strings plus page, block, paragraph, word, and break information.

Let’s take a look at the following image, which contains a dense text block.

The following code can be used to implement both detection types to extract the text in that image but there will be some differences in the output. If you want to use “DOCUMENT_TEXT_DETECION”, uncomment it and comment the line type: 'TEXT_DETECTION'.

const [result] = await client.annotateImage({
        image: {
            content: imageContent,
        },
        features: [{
            type: 'TEXT_DETECTION',
            // type: 'DOCUMENT_TEXT_DETECTION',
        }]
    });

    const textAnno = result.textAnnotations;
    textAnno.forEach(text => console.log(text));

TEXT_DETECTION Output

DOCUMENT_TEXT_DETECTION Output

As you can see, while TEXT_DETECTION was able to simply extract the text from the image, the DOCUMENT_TEXT_DETECTION was also able to identify the period (.) punctuation mark as the separating character. This is due to the DOCUMENT_TEXT_DETECTION being geared towards denser text blocks like documents and can extract more intricate details, while TEXT_DETECTION is more suited for large text objects like street signs, billboards, etc.

Google Vision API with Puppeteer

In this section, you will be using DOCUMENT_TEXT_DETECTION as you are working with web content. Now, let’s start incorporating Vision API (OCR) within a Puppeteer program.

First, capture a screenshot of the text area you need to extract text from the Puppeteer's page.screenshot() function.

 const browser = await puppeteer.launch({headless:true});
        const page = await browser.newPage();
        await page.goto('https://www.example.com');  

        const imageFilePath = 'ocr-text-block.png'

        // Take screenshot
        await page.screenshot({
            path: imageFilePath,
            clip: { x: 0, y: 0, width: 800, height: 400 },
            omitBackground: true,
        });

Then you have to encode the image to base64 format in order to pass the resulting data to the Vision API.

        // Read the image contents
        const imageContent = Buffer.from(fs.readFileSync(imageFilePath)).toString('base64');

Next, pass the data to the Vision API and you will get the following result.

        // Pass the content
        const client = new vision.ImageAnnotatorClient();
        const [documentAnnotateResult] = await client.annotateImage({
            image: {
                content: imageContent,
            },
            features: [{
                type: 'DOCUMENT_TEXT_DETECTION',

            }]
        });

        // Get the annotated result
        console.log(documentAnnotateResult.fullTextAnnotation)

As you have obtained the OCR result without any errors, compare the image you have captured with the corresponding output.

Captured image (ocr-text-block.png)

OCR Result

That’s it! Congratulations, now you have successfully integrated the OCR functionality. The next step would be to iterate over the text annotations and find a match against the string you are searching for. In this instance, you will be searching for a string called “permission”.

        var point_x = 0
        var point_y = 0
        //Iterate the elements
        const textAnno = documentAnnotateResult.textAnnotations;
        textAnno.forEach(text => {
            if(text.description === 'permission') {

                point_x = text.boundingPoly.vertices[0].x;
                point_y = text.boundingPoly.vertices[0].y;            
            }
        }
        );

Iterate through the results using a forEach loop and if a matching text is found, obtain its coordinates from boundingPoly.vertices and store them in point_x and point_y variables.

Then, you have to invoke the document.getElementFromPoint function to get the DOM element that contains the matched text.

        // Find the element using elementFromPoint
        const foundElement = await page.evaluateHandle((x,y) => document.elementFromPoint(x,y), point_x,point_y);
        console.log(foundElement._remoteObject.description)

Since the word “permissions” is inside a paragraph tag (<p>), the console output will show a DOM object description attribute like the following.

Limitations of OCR

OCR is a powerful tool when it comes to identifying web content. However, it is not foolproof, and its major issue will be the accuracy of character recognition.

Missing some characters entirely.
Mixing up similar characters (6 to b, o to 0, I to l, etc.)
Messing up spaces in documents.

The points mentioned above are some facts that lead to the inaccuracy of the result. Refer to the following image and its OCR result.

Image

OCR Result

You can assert that the word in the image is “Los Altos”, but the OCR function mistakenly detects the letter “t” as the plus (+) sign. This will cause incorrect character recognition and even breaks the word Altos into three separate segments. (AL, + and OS). These kinds of issues are very common when dealing with handwritten text in images or documents.

To mitigate these issues and keep the Puppeteer test resilient, you need to have some room for variance in OCR results. The simplest method would be to handle the commonly mistaken characters. For example, suppose you encounter a 6 that doesn't match your string, substitute 6 with b, and check again. However, This method will quickly become cumbersome and not a very flexible solution to handle multiple inconsistencies. \

Another option would be to implement a word distance algorithm based on a string metric like Levenshtein distance, which measures the difference between two sequences. You could specify a threshold that allows for some variance in the results while still getting the confidence that the desired text was found in the specified area.

Going beyond visual testing

Visual testing is an essential piece of the testing toolkit. It's a great way to ensure you aren't introducing visual anomalies as you make changes to your app, and it can be helpful in making simple assertions like the presence of text or images.

However, image-based-testing is not one-size-fits-all. To successfully end-to-end test your product, you'll likely need to support a much wider range of assertions and actions (including uploading files, verifying text messages or emails are received, etc.).

To support the automation of testing these harder assertions and experiences, you'll need to expand your toolkit to go beyond visual testing.

Looking to go beyond visual testing?

walrus.ai provides visual assertions and handles everything mentioned in this article out of the box. Beyond that, walrus.ai can be used to test your hardest user experiences end-to-end. Drop us a line if you'd like to see it in action!

5 Tips for Writing Better Unit Tests

Scott White — Thu, 04 Feb 2021 17:07:00 +0000

What are Unit Tests?

Unit testing is the testing of individual parts or components of a program. The purpose of unit testing is to test the functionality of a single component and verify its behavior independently from other components. A unit in unit testing includes the smallest piece of functionality that can be tested. The main objective of a unit test is to isolate the smallest parts of a program and test the functionality of each part. The basic structure of a unit test is designed to check if the output of a defined code block matches the given condition. For example, when testing a function, we specify a set of input arguments and test if the function returns the expected output, as shown below.

func.py

# ==== Function ==== #
class Calculations:

  # Function to Test
    def calculate_total(value1, value2):
        return int(value1 + value2)

test.py

# ==== Unit Test ==== #
import unittest
from func import Calculations

# Test Class
class TestingFunctions(unittest.TestCase):

    # Test Case
    def test_calculation(self):
        input_one = 4
        input_two = 6
        total = Calculations.calculate_total(input_one, input_two)
        # Identify if output is equal to zero
        self.assertEqual(total, 10)

if __name__ == '__main__': 
    unittest.main()

Avoid Complex Setups

Since we are testing the functionality of the simplest testable unit during the unit testing, it is better to keep the tests also as simple as possible. One of the major difficulties developers face during unit testing is the test cases becoming hard to understand and change due to making them unnecessarily complex.

The AAA Rule provides a good guideline to keep a unit test simple as much as possible. In unit testing, AAA stands for Arrange, Act, and Assert. The first step is to arrange the test case with the input variables, properties, and conditions to get the expected output. The next step is to act upon your arrangements by calling the methods or functions. In the final step, which is the assertion, you can assert to verify the expected output using a test framework. This method makes the codebase of test cases much easier to read and understand.

import unittest

# AAA Methodology
class TestingStringFunctions(unittest.TestCase):

    def test_string_manipulation(self):
        # Arrange
        text = "Hello World"

        # Act
        new_text = ''.join(reversed(text))

        # Assert
        self.assertEqual(new_text, 'dlroW olleH')

Another important thing to consider when performing unit tests is the test dependencies. If a test case depends on another test case, changing a single test case could affect other test cases, which will, in turn, defeat the purpose of individual unit tests.

Exhaustiveness is not the Goal

It is always a good idea to perform comprehensive unit testing for your program. However, it does not mean that you should write test cases for each and every scenario. Considering every edge case beforehand is a waste of valuable development time.

Tests should be straight forward covering the common flow of your code block. Once simple tests are passing, you can increase the coverage of the test to include edge cases and cover multiple boundaries. When a bug is discovered in the development cycle, you can expand the test coverage to include that bug. This will keep your test cases up to date without having to create separate test cases to tackle the bugs.

Write Tests First When Fixing Bugs

The first intention of a developer when discovering a bug is to try and fix it somehow. Although she would be able to fix the bug at that time, her understanding of the bug may be limited and it might cause new issues down the pipeline in the future.

Therefore, it is always a good practice to create a test for the discovered bug encompassing the components that cause the bug. This way, you can verify if the bug relates to the indicated components or is caused by an external factor. Having a simple and straightforward test will lead to an easier bug fix by quickly exposing the problems.

When you write your test, make sure to run it before fixing the bug and verifying that the test fails. Otherwise, you may write a test that doesn’t actually encapsulate the bug, therefore defeating the entire purpose.

Once a bug is fixed, you should not discard that unit test. These tests can then also be used in regression testing to test the overall functionality of the program. It will be an extra benefit as we know both the failing and passing conditions of that test in a future event when a similar bug is discovered. That will also make it easy to know where to apply the fix.

Mock with Caution

Mocking is a way of simulating the behavior of a real object in a controlled manner by creating an artificial object (mock object) similar to it. Mock objects enable developers to mock the behavior of complex functions without having to invoke the function. This is useful when dealing with external services where you require those services only to execute the underlying code block but they are not needed for the test case.

Mocks are also useful in increasing the speed and efficiency of a test case. Without mocks, if a test calls for an external library or a function, you must wait for that function to be completed to continue the test. This will lead to slow test execution times and can even introduce unnecessary errors or behavior inside the test case. For example, when dealing with the file systems and network functions, you could use a mock object to simulate those behaviors, effectively speeding up the test while focusing on the core functionality of the test.

import unittest
from unittest import mock
from func import delete_file

class TestFunctions(unittest.TestCase):

    # Mock the os module
    @mock.patch('func.os')
    def test_file_delete(self, mockobj):
        file_name = "test.txt"
        delete_file(str(file_name))
        # Verify the parameters were passed correctly to the function
        mockobj.remove.assert_called_with(str(file_name))

The most important thing to remember when using mocks is not to mock the functionality of the test case. Mocks should not be used if you are testing for database queries, API calls, or any functionality that has a direct impact on your overall test.

Keep Tests Independent

The tests should be orthogonal to each other and be used to test a single functionality at a time. The test classes should be tested independently and should not depend on anything other than the testing framework.

When an order is needed to carry out unit testing, you can use the inbuilt setup and teardown methods of the test framework to isolate the test cases. A good test case should not be affected by any implementation changes. When dealing with API changes, a program with a solid code base with correctly implemented individual unit tests is vital for identifying errors and making the program up and running again.

Besides, creating independent tests enables us to easily modify each test case while increasing the test spectrum without causing any side effects to other tests. In test automation environments, independent tests are the preferred way to integrate with CI/CD pipelines.

The next step

Unit tests are a great way to discover and resolve bugs in units, ensuring that every single part of the program works well. However, if you want to test different parts of the program working together, in their actual environment, you need to go for integration testing or end-to-end. You can learn more about the testing hierarchy here.

Integration tests abstracted up a level from unit tests, executing and asserting on code paths that traverse between multiple units or modules. Now we're starting to ensure the different pieces of our application interact with each other as expected.

Even farther are end-to-end (E2E) tests. These are tests aimed to cover the entire surface area of your application from top to bottom. These should generally follow the code paths expected from your end-users to ensure they're as close to reality as possible.

Looking for an easy way to take the next step beyond unit testing? Check out walrus.ai

Tests can be written in plain english — no difficult setup required
Test your most challenging user experiences – emails, multi-user flows, third-party integrations
Zero maintenance — walrus.ai handles test maintenance for you, so you never experience flakes

Simple Steps for Testing a Chrome Extension in Puppeteer

Scott White — Wed, 03 Feb 2021 22:03:28 +0000

What is Puppeteer?

Puppeteer is a Node library that can be used for the automated testing of chrome extensions. It provides a high-level API control over Chrome or Chromium via the DevTools protocol. One great advantage of this library is that it can be used to do most of the things you manually do in the browser.

When testing chrome extensions, there are two major test scenarios, which are testing the compatibility of the extension and testing functionality of the extension. You can test the compatibility of a chromium-based browser extension by loading that extension on each browser variant. When it comes to testing the functionality, the puppeteer library provides various methods to imitate user interactions. This allows developers to create automated test cases to inspect and analyze the behavior of an extension.

Testing Extension Compatibility

Chrome extensions are inherently cross-browser compatible. Most chrome extensions can also be used in other chromium-based browsers such as MS Edge (Chromium), Vivaldi, Brave, etc. Puppeteer provides users the ability to specify the browser in the puppeteer class. In the following section, you will see how to load a browser extension in different browsers.

When using the normal puppeteer library, it will install the latest Chromium build, and therefore it is not required to provide the location of the browser and get called directly.

const puppeteer = require('puppeteer');

// Calling the default chromium installation
const browser = await puppeteer.launch();

However, if you are using the lightweight puppeteer-core or defining a different browser, you need to provide the executablePath option in the puppeteer.launch method. The below code demonstrates how to test the installation of an extension on MS Edge (Chromium)

browser_test.js

const puppeteer = require('puppeteer');

(async () => {
    // Path to extension folder
    const paths = 'F:\\puppeteer\\ext\\cplklnmnlbnpmjogncfgfijoopmnlemp\\';
    try {
        console.log('==>Open Browser');
        const browser = await puppeteer.launch({
            // Define the browser location
            executablePath: 'C:\\Program Files (x86)\\Microsoft\\Edge Beta\\Application\\msedge.exe',
            // Disable headless mode
            headless: false,
            // Pass the options to install the extension
            args: [
                `--disable-extensions-except=${paths}`,
                `--load-extension=${paths}`,
                `--window-size=800,600`
                ]
        });

        console.log('==>Navigate to Extension');
        const page = await browser.newPage();
        // Navigate to extension page
        await page.goto('chrome-extension://fgbekoibaojaiamoheiklljjiihibdcb/panel.html');
        // Take a screenshot of the extension page
        console.log('==>Take Screenshot');
        await page.screenshot({path: 'msedge-extension.png'});

        console.log('==>Close Browser');
        await browser.close();
    }
    catch (err) {
        console.error(err);
    }
})();

The following code-block will open a new MS Edge (Chromium) browser and install the chrome extension using the provided optional arguments.


--disable-extensions-except=<Extension Folder Path>
--load-extension=<Extention Folder Path>

The load-extension option will load the plugin while the --disable-extensions-except option will disable other extensions that might interfere with the testing.

A new browser page (tab) will then be created, and you will navigate to the extension URL and take a screenshot. The extension URL can be defined in the below format.


<Extension Protocol>://<Extension ID>/<Extension Page>

\
Additionally, the code is encapsulated within an async function to enforce an asynchronous operation in each stage of the puppeteer process.

You can make sure that the test is successful by checking the terminal output and the captured screenshot.

Terminal

Captured Screenshot (msedge-extension.png)

You can define the location for any other chromium-based browser in the same way that is used to define the location for MS Edge (Chromium).

Testing the Extension Functionality

Now you know how to load an extension to a browser, and next comes the functionality testing of the extension. But before that, you have to tackle a major obstacle in defining the extension URL. Whenever a new browser instance is loaded, it will create a new extension ID, making it difficult to define a static path for the extension.

However, Puppeteer provides a way to query for the extension ID by utilizing a background script. There, the extension ID can be extracted using the targets method in the Browser class, as shown below.

find-extensionid.js

const puppeteer = require('puppeteer');

(async () => {
  // Define the extension path
    const paths = 'F:\\puppeteer\\ext\\cplklnmnlbnpmjogncfgfijoopmnlemp\\';
    try {
        console.log('==>Open Browser');
        // Configure the browser (Default Chromium Installation)
        const browser = await puppeteer.launch({
            headless: false,
            // Chrome options
            args: [
            `--disable-extensions-except=${paths}`,
            `--load-extension=${paths}`,
            `--window-size=800,600`
            ]
        });

        // Name of the extension
        const extensionName = 'iMacros for Chrome';

        // Find the extension
        const targets = await browser.targets();
        const extensionTarget = targets.find(({ _targetInfo }) => {
            return _targetInfo.title === extensionName && _targetInfo.type === 'background_page';
        });

        // Extract the URL
        const extensionURL = extensionTarget._targetInfo.url;
        console.log("\nExtracted URL ==>" + extensionURL);
        const urlSplit = extensionURL.split('/');
        console.log("Split URL ==>");
        console.log(urlSplit);
        const extensionID = urlSplit[2];
        console.log("Extension ID ==>" + extensionID +"\n");

        // Define the extension page
        const extensionEndURL = 'panel.html';

        //Navigate to the page
        console.log('==>Navigate to Extension');
        const page = await browser.newPage();
        await page.goto(`chrome-extension://${extensionID}/${extensionEndURL}`);

        console.log('==>Take Screenshot');
        await page.screenshot({path: 'chromium-extension.png'});

        console.log('==>Close Browser');
        await browser.close();
    }
    catch (err) {
        console.error(err);
    }
})();

In the above code block, you are querying for the chrome extension using the targets method. The targets method can be used to access the target of any page managed by the browser. In this instance, the target is specified using the extension name (This refers to the extension title defined in the manifest.json file of the extension).

Once the extension is located, you can extract the complete URL and then split it to retrieve the extension ID. You can observe this process using the console.log() outputs as shown in the following screenshot.

The last step is to create a variable to pass the necessary page with the extension. You have created a new variable called extensionEndURL to indicate the page and finally pass both the extension ID and the necessary page to the page.goto() method to direct the browser to the extension location.

User Interactions

Now, you have launched the browser and obtained the extension programmatically. The only remaining task is to interact with the extension. Puppeteer provides multiple ways to find elements within a webpage and interact with them. Let us go through the following code sample to see them in action.

google-search.js

const puppeteer = require('puppeteer');

(async () => {
    try {
        // Calling the default chromium installation
        const browser = await puppeteer.launch({headless:false, defaultViewport: null});
        const page = await browser.newPage();
        await page.goto('https://www.google.com');

        // Setup Viewport
        await page.setViewport({width: 1280, height: 800})

        // Interact with webpage
        // Search Google
        await page.type('#tsf > div:nth-child(2) > div.A8SBwf > div.RNNXgb > div > div.a4bIc > input', 'Puppeteer');

        await page.$eval('input[name=btnK]', el => el.click());

        // Wait for results
        await page.waitForNavigation();

        // Take screenshot
        await page.screenshot({path: 'google-search.png'});

        await browser.close();
    }
    catch (err) {
        console.error(err);
    }
})();

Above, you have defined a simple script to navigate to Google and perform a search. You are providing input to the search field using the page.type() method and CSS selector to indicate the element and enter the search term. Then, using the page.$eval() method, you are querying for the search button and clicking on that button. This method will use the document.querySelector function to search for the specific element. Then you have to wait for the page to load and take the screenshot.

All these interactions are based on DOM object querying. The Document Object Model is queried by using the $ (querySelector) and $$ (querySelectorAll) APIs provided by Puppeteer.

When it comes to testing a Chrome extension, you can use the same methods to interact with the extension. The following code-block illustrates how to interact with the “iMacros for Chrome” extension using the methods mentioned above. When the extension loads, you will click on the RECORD tab and navigate to the macro recording section.

//Navigate to the page
        console.log('==>Navigate to Extension');
        const page = await browser.newPage();
        await page.goto(`chrome-extension://${extensionID}/${extensionEndURL}`);

        console.log('==>Click on RECORD tab')
        await page.$eval('#record-tab-label', el => el.click());

        console.log('==>Take Screenshot');
        await page.screenshot({path: 'chromium-extension.png'});

RESULT

Bonus Script

The following script includes the things you have learned up to now and creates a script that will execute in multiple browsers like Chromium, MS Edge (Chromium), Vivaldi, and Google Chrome. It will also test the extension loading and functionality and then end the program by taking a screenshot. This script will run in all the browsers simultaneously while executing each action within the browsers in an asynchronous manner.

ext_test.js

const puppeteer = require('puppeteer');
const path = require('path');

// Required global variables
const ext_install_path = path.join(__dirname, '\\ext\\cplklnmnlbnpmjogncfgfijoopmnlemp\\');
const image_path = path.join(__dirname, '\\img\\');
const extensionEndURL = 'panel.html'

// Function to obtain extension ID
async function getID(browserObj) {
    // Name of the extension
    const extensionName = 'iMacros for Chrome';

    // Find the extension
    const targets = await browserObj.targets();
    const extensionTarget = targets.find(({ _targetInfo }) => {
        return _targetInfo.title === extensionName && _targetInfo.type === 'background_page';
    });

    // Extract the URL
    const extensionURL = extensionTarget._targetInfo.url;
    const urlSplit = extensionURL.split('/');
    const extID = urlSplit[2];

    return extID;
}

// Testing Built-in Chromium
(async () => {
    console.log("===> Testing Chromium")
    const browser = await puppeteer.launch({
        headless: false,
            // Chrome options
            args: [
            `--disable-extensions-except=${ext_install_path}`,
            `--load-extension=${ext_install_path}`,
            ]
    });

    const extensionID = await getID(browser);
    const page = await browser.newPage();

    await page.goto(`chrome-extension://${extensionID}/${extensionEndURL}`);
    await page.$eval('#record-tab-label', el => el.click());

    await page.screenshot({path: path.join(image_path, 'default-chromium.png')});

    await browser.close();
  })();

// Testing MS Edge (Chromium)
(async () => {
    console.log("===> Testing MS Edge (Chromium)")
    const browser_location = 'C:\\Program Files (x86)\\Microsoft\\Edge Beta\\Application\\msedge.exe'

    const browser = await puppeteer.launch({
        executablePath: browser_location,
        headless: false,
        args: [
            `--disable-extensions-except=${ext_install_path}`,
            `--load-extension=${ext_install_path}`,
            `--window-size=800,600`
            ]
    });

    const extensionID = await getID(browser);
    const page = await browser.newPage();

    await page.goto(`chrome-extension://${extensionID}/${extensionEndURL}`);
    await page.$eval('#record-tab-label', el => el.click());

    await page.screenshot({path: path.join(image_path, 'msedge-extension.png')});

    await browser.close();
})();

// Testing Vivaldi
(async () => {
    console.log("===> Testing Vivaldi")
    const browser_location = 'C:\\Program Files\\Vivaldi\\Application\\vivaldi.exe';

    const browser = await puppeteer.launch({
        executablePath: browser_location,
        headless : false,
        args: [
            `--disable-extensions-except=${ext_install_path}`,
            `--load-extension=${ext_install_path}`,
            ]
    });

    const extensionID = await getID(browser);
    const page = await browser.newPage();

    await page.goto(`chrome-extension://${extensionID}/${extensionEndURL}`);
    await page.$eval('#record-tab-label', el => el.click());

    await page.screenshot({path: path.join(image_path, 'vivaldi.png')});

    await browser.close();
})();

// Testing Google Chrome
(async () => {
    console.log("===> Testing Google Chrome")
    const browser_location = 'C:\\Program Files\\Google\\Chrome\\Application\\chrome.exe'; 

    const browser = await puppeteer.launch({
        executablePath: browser_location,
        headless : false,
        args: [
            `--disable-extensions-except=${ext_install_path}`,
            `--load-extension=${ext_install_path}`,
            ]
    });

    const extensionID = await getID(browser);
    const page = await browser.newPage();
    await page.setViewport({width: 1280, height: 800})
    await page.goto(`chrome-extension://${extensionID}/${extensionEndURL}`);
    await page.$eval('#record-tab-label', el => el.click());

    await page.screenshot({path: path.join(image_path, 'google-chrome.png')});

    await browser.close();
})();

Conclusion

In this article, we’ve learned about Puppeteer and how it can be used to test Chrome extensions. Puppeteer is a powerful library that can be customized according to the needs of the user. Puppeteer can also be integrated with other Javascript testing frameworks like Jest, Mocha, or Jasmine in order to create a complete test suite.

Writing a test for your Chrome extension? There's an easier way.

While the above is a good starting point for building Chrome Extension testing in-house, it's still quite a challenging process. If you're looking for an easier way to test your Chrome extension, check out walrus.ai!

Tests can be written in plain english — no difficult setup required
Chrome extensions can be tested in an actual browser, not just in a new window/tab
Zero maintenance — walrus.ai handles test maintenance for you, so you never experience flakes

Can I get an example?

Imagine you wanted to test Zoom's Google calendar Chrome extension. Writing a test for that is as easy as writing 4 lines of code.

$ walrus -u https://zoom.us -i \
  'Click on the Zoom Chrome Extension' \
  'In the extension popup, click "Schedule a meeting"' \
  'In the browser, click Save' \
  'In the browser, verify a calendar event loads'

Want to learn more? Let us know.

Why end-to-end tests are flaky, and how to fix them

Scott White — Mon, 27 Apr 2020 15:04:01 +0000

End-to-end tests are an essential tool to verify that the true customer experience is working every time you make changes to your application.

They're also not without their problems.

End-to-end tests are notoriously difficult to maintain, for a couple of reasons:

When making changes to your app, you need to refactor your old tests to be able to handle the new changes.
Even when nothing changes about your app, sometimes your tests fail anyway, forcing you to investigate the failure. These are called test flakes.

What is a flaky test?

A test that is flaky will pass sometimes and fail other times, even when nothing changes about the test case itself. In other words, flaky tests are non-deterministic.

Why do flaky tests matter?

Flaky tests hurt reliability, and carry large explicit and implicit costs:

❌ Flaky tests hinder productivity

Every time a test flakes, a developer needs to investigate whether the failure is true or false. If the "flake rate" of your tests is constant, every new test you add represents more time the engineering team will have to spend in the future to maintain that test.

Furthermore, test failures have a cascading effect on overall productivity. In a CI/CD environment, test failures lead to failed pipelines, which lead to delayed deploys.

When the whole point of CI/CD is to speed up development cycles, flaky tests can defeat the whole purpose.

💸 Organizational costs

The hidden, and largest cost of test flakiness is the damage it has on the overall trust of the testing process.

If engineers cannot trust the results of their tests (because they don't know if the failures are true or false), they will not want to write tests to begin with. Test flakiness is the catalyst in a negative feedback loop:

If the tests are flaky, engineers won't trust the tests
If engineers don't trust the tests, they won't write them in the first place
If the codebase doesn't have tests, no one trusts the codebase
If no one trusts the codebase, then they need to write tests (which starts the cycle over again)

😡 Emotional costs

The least obvious cost to the organization, but the most obvious to the engineer who writes tests is the emotional toll they take. No one likes to write flaky tests knowing that they'll be stuck maintaining them in perpetuity.

Common causes of test flakiness, and how to deal with them

While end-to-end tests are notoriously flaky, they're at least predictably flaky. Being cognizant of these root causes goes a long way to authoring that are resilient to test flakes.

⏰Asynchronous waiting

Waiting for a fixed amount of time between steps in a test drastically increases the chances that your test will falsely fail between any given steps.

To avoid "waiting" between steps for fixed periods of time, instead write your tests to poll your application continuously to ensure it is in the appropriate state before asserting on the step criteria.

💡With walrus.ai, no wait steps are necessary between instructions. Simply relay the user story in plain English, and all transitions are handled automatically.

🎬Inconsistent starting state

If the initial conditions for your testing environment differ between runs, your tests will likely fail to even kick off because the starting assertions won't pass.

To avoid this, make sure that your application is in a consistent state every time before the test starts. To ensure the environment is the same every time, you should tear down the environment after every test run, whether it's a pass or a fail. That way if a flake occurs, it won't affect future test runs as well.

💡With walrus.ai, you can specify setup or teardown instructions directly in the body of the test, so you never have to worry about an inconsistent testing environment.

↩️Dependencies

If your tests depend on each other, or even worse, if they depend on themselves, they won't be able to run concurrently (which you will want to do if you are running tests in a CI/CD environment).

Concurrency and dependency doesn't have to be explicit either. If you use the same account or environment to run two separate tests that require different initial conditions (some user permission flipped on vs. off, for example), those tests are implicitly dependent.

To avoid this, consider splitting up the accounts you run tests with to be distinct. That way, you can mitigate implicit dependencies, and more easily execute setup and teardown within the test.

If tests are truly dependent, then they shouldn't run concurrently. Instead, trigger a dependent test based on the pass of the upstream test.

💡With walrus.ai, you can specify separate account credentials for different tests by using environment variables, thereby eliminating implicit dependency.

Is there a one-size-fits-all solution to test flakiness?

Walrus.ai lets engineers ship flake-free end-to-end tests in minutes:

😎Easy – tests can be written in plain English

🛠No maintenance – flakes and refactors are handled entirely by walrus.ai

💯Coverage for your whole app – test your hardest user experiences with ease, including APIs, third-party integrations, emails, etc.

Testing Database Interactions with Jest

Scott White — Fri, 24 Apr 2020 18:15:07 +0000

Testing Database Interactions with Jest

Jest has quickly become one of the most popular Javascript testing libraries. While Jest may be mostly used in the context of frontend applications, at walrus.ai we use Jest for testing our backend Node.js services as well.

Jest aims to make testing 'delightful', and a large component of that delight comes from speed. Jest by default runs concurrently with worker processes, a pattern that encourages and even requires test isolation. While this is relatively simple to accomplish for frontend code, there's a shared mutable state elephant in the room for backend code: the database.

Why test database interactions?

For unit tests, we generally follow best practice of mocking any interactions that are outside the unit. Consider the following function:

async function changeUserName(db, userId, username) {
  const userRepository = db.getRepository(User);
  const updated = await userRepository.updateUserName(userId, username);

  return updated;
}

This function takes a handle to database connection, the userId, and a new username, and updates the username in the database. We abstract away the underlying SQL necessary to make the database update with the Repository pattern. This allows us to test this function rather easily.

describe('changeUserName', () => {
  it('should update username in db', async () => {
    const db = { getRepository: jest.fn() };
    const repository = { updateUserName: jest.fn() };

    db.getRepository.mockReturnValue(repository);
    repository.updateUserName.mockReturnValue(Promise.resolve('updated'));

    const result = await changeUserName(db, '1', 'username');

    expect(result).toEqual('updated');
    expect(repository.updateUserName).toHaveBeenCalledTimes(1);
    expect(repository.updateUserName).toHaveBeenCalledWith('1', 'username');
  });
});

However, what if we want to test our actual Repository? The code for the repository probably looks something like this:

class UserRepository {
  ...
  public async update(id, username) {
      await this.db.sql(`
        UPDATE users SET username = :username WHERE id = :id
    `, { id, username });
  }
}

If we wanted to test method, we could obviously mock the db connection and assert that .sql is called with the expected parameters. But what if this SQL is invalid, or more probably, what if the SQL is valid but is doing the wrong thing?

Presumably, at some point, we'll want to test the actual interaction with the database. We won't get into what we actually call these tests (there's probably 1000s of internet discussions on whether we've crossed the line from unit tests to integration tests by involving a real database), we'll simply cover how to do it safely and concurrently with Jest.

Setting up the database for Jest

As we've discussed, Jest by default runs tests concurrently — this makes sharing a database problematic. Tests that are running in parallel will clobber each other's database state, causing spurious failures and flakes.

The simplest option to overcome this limitation is to run Jest with the --runInBand option. This forces Jest to only use one process to run all your tests. However, this probably will make your test suite far slower. At walrus.ai this took our test suite from 10s of seconds to a few minutes, and simply wasn't tenable for our CI/CD processes of constant deployments.

Luckily, parallel testing with a database is a pretty solved problem. We can simply spin up a database for each worker process we're using. If all tests running against a particular database are run serially, then we don't have to worry about parallel processes mutating database state.

The easiest solution for this would be something like the following. We could spin up a database before each Jest worker, and shut it down after. Since all tests within a worker run serially, each worker could operate on its individual database safely.

beforeWorker(async () => {
  db = await createDatabase(`db_${process.env.JEST_WORKER_ID}`);
});

// All tests run serially here.

afterWorker(async () => {
  await destroyDatabase(db);
});

Unfortunately, while Jest exposes the JEST_WORKER_ID environment variable to distinguish between workers, it doesn't expose any simple way of hooking in per-worker setup and teardown methods.

This means that we can't dynamically spin up and tear down databases for each worker. We can, however, do the next best thing, using a static number of Jest workers.

First, we'll need a test setup script that prepares our multiple databases.

Note: The following code examples use Typeorm, but the code could easily be extended for any other database interaction library such as Sequelize, Massive.js, Knex etc.

(async () => {
  const connection = await createConnection({
    type: 'postgres',
    username: process.env.DATABASE_USER,
    password: process.env.DATABASE_PASSWORD,
    database: process.env.DATABASE_MASTER',
    host: process.env.DATABASE_HOST,
    port: 5432,
  });
  const databaseName = `walrus_test_template`;
  const workers = parseInt(process.env.JEST_WORKERS || '1');

  await connection.query(`DROP DATABASE IF EXISTS ${databaseName}`);
  await connection.query(`CREATE DATABASE ${databaseName}`);

  const templateDBConnection = await createConnection({
    name: 'templateConnection',
    type: 'postgres',
    username: process.env.DATABASE_USER,
    password: process.env.DATABASE_PASSWORD,
    database: 'walrus_test_template',
    host: process.env.DATABASE_HOST,
    migrations: ['src/migrations/*.ts'],
    port: 5432,
  });

  await templateDBConnection.runMigrations();
  await templateDBConnection.close();

  for (let i = 1; i <= workers; i++) {
    const workerDatabaseName = `walrus_test_${i}`;

    await connection.query(`DROP DATABASE IF EXISTS ${workerDatabaseName};`);
    await connection.query(`CREATE DATABASE ${workerDatabaseName} TEMPLATE ${databaseName};`);
  }

  await connection.close();
})();

With this script, we create a connection to our database, Postgres in this instance. Then, for each worker that we are using (set statically in the JEST_WORKERS environment variable, we initialize a new database.

Because we're using Postgres, we can use a handy feature called Template Databases. This makes new database creation cheap, and allows us to only run our migrations once. At a high level, we create one template database, run our migrations once against the template database, and then quickly copy over the database for each Jest worker.

Now, we simply have to connect to the correct database in all of our tests. With the JEST_WORKER_ID environment variable, this is trivial:

beforeAll(async () => {
  connection = await createConnection({
    type: 'postgres',
    host: process.env.DATABASE_HOST,
    port: 5432,
    username: process.env.DATABASE_USER,
    password: process.env.DATABASE_PASSWORD,
    database: `walrus_test_${process.env.JEST_WORKER_ID}`,
    logging: false,
    entities,
    namingStrategy: new SnakeNamingStrategy(),
  });
});

afterAll(async () => {
  connection.close();
});

Now all of our workers will use individual databases, allowing us to run tests in parallel.

Cleaning up between tests

While parallel tests are no longer problematic, we still have a problem with serial tests. Consider the following example, again of our contrived UserRepository:

describe('UserRepository', () => {
    it('should create user', () => {
    await repository.createUser('username');

    expect(await repository.countUsers()).toEqual(1); 
  });

    it('should delete user', () => {
    const userId = await repository.createUser('username');
    await repository.deleteUser(userId);

    expect(await repository.countUsers()).toEqual(0); 
  });
});

Notice anything wrong here? The second test will fail. While we're setting up different databases for each of our parallel workers, tests within the same file are run serially. This means the user created in the first test is still present in the second test, causing the test to fail.

We considered two approaches to solve this problem. The first approach is to wrap each test in a database transaction:

beforeEach(() => {
  db.startTransaction();
});

afterEach(() => {
  db.rollbackTransaction();
});

With this approach, any database updates made within the test are wrapped into the initialized transaction. When the test is finished, we simply rollback the transaction, discarding any of those updates. While this approach is fast and generally supported by all databases, it's not always the best for certain classes of integration-tests.

Sometimes, the behavior under test may actually be the transaction itself. For example, we may want to test that when an update fails, certain components of the update are preserved (committed), and others are rolled back. This logic would require us to manually start and stop transactions within our code, and wrapping the code in a parent transaction with this method would keep us from effectively testing rollbacks.

Another, simpler but slower approach, is to just clear out the database after before every test. While this may be slower, it's less likely to bite us later. We can do this in a simple beforeEach block.

beforeEach(async () => {
    const queryRunner = getConnection().createQueryRunner();

    await queryRunner.query(`
      DO
      $func$
      BEGIN
        EXECUTE (
          SELECT 'TRUNCATE TABLE ' || string_agg(oid::regclass::text, ', ') || ' CASCADE'
            FROM pg_class
            WHERE relkind = 'r'
            AND relnamespace = 'public'::regnamespace
        );
      END
      $func$;
    `);
    await queryRunner.release();
  });

The above code iterates through all our tables and clears them using the SQL TRUNCATE command. In the walrus.ai test suite, this occurs in the order of milliseconds, and is a worthwhile performance tradeoff for keeping our tests simple.

Conclusion

By clearing the database between tests, and using one test per worker, we can continue getting the delightful Jest experience for testing database connected backend applications.

While testing database interactions with Jest helps increase unit and integration test coverage without sacrificing test stability, running true end-to-end tests with browser automation tools like Selenium or Cypress can still be flaky and unstable.

Jest's goal is to make unit and integration testing 'delightful' — our goal at walrus.ai is to do the same for end-to-end testing. Engineering teams can write tests in plain english, and we take care of automating the tests, resolving flakes, and maintaining tests as their applications change. Our contrived example above showed how to test the database end of a username update, here's how simple the corresponding end-to-end test with walrus.ai could be:

walrus -u your-application.com -i \
  'Login' \
  'Change your username' \
  'Verify you receive a confirmation email at the new email address' \
  'Verify your username is changed'

Why engineers suck at writing integration tests

Scott White — Tue, 21 Apr 2020 18:57:58 +0000

In a perfect world, we can all agree that having automated tests are a good thing. They improve reliability, prevent bugs from making it to production, and reduce the load on manual QA, in addition to a whole host of other less obvious, but crucial benefits.

But we don't live in a perfect world.

The real downside to automated tests is writing them in the first place. While important for any CI/CD development environment, writing and maintaining tests is the least favorite part of a developer's job. As we all know, if we don't like to do something, we probably won't do it well.

In short, engineers suck at writing integration tests because writing integration tests sucks

What's so bad about writing integration tests?

✏ Writing integration tests is too hard

Writing a test is fundamentally expressing a user story. You want to make sure that your customer can take a set of actions in a particular order, and your application responds the way you would expect based on those actions. For example, if you're Amazon, you want to make sure that your customer can add an item to the cart, and the item actually loads in the cart when it's opened.

While easy to express in English, writing a test for that story in something like Selenium requires making complicated assertions on every step of the story (make sure a button loads containing some text, make sure it's clickable, make sure a page loads, that an image of the item loads, etc.).

Furthermore, whatever framework you use to test is probably not the framework you use to do most of your job, which means every time you go to write tests, you have to brush off the test-writing skills before you can really get productive.

🛠 Maintaining them is too costly

Test maintenance comes in many forms:

The cost of updating tests every time you make changes to your application

Writing tests is important when you're making changes to your application. At the same time, when you make changes to your application, you likely need to update your tests if any of the changes conflict with the assertions you're making inside of those tests. This circularity means that every time you're making big changes to your site, not only do you need to write new tests for that functionality, you probably also need to update all of the tests you've already written so they don't break.

The cost of investigating and fixing flaky tests

A flaky test fails sometimes and passes other times, despite no changes to the user story it is testing. Every time one of these flakes occurs, you need to investigate whether it is a true failure of the user story, or a false failure of the user story. The more tests you add, the more investigation you need to do when your test suite flakes.

The cost of held-up deploys

When tests fail, even falsely in the event of flakes, CI/CD pipelines will fail, which means your important changes won't make it to production. The whole purpose of CI/CD is to speed up the process of making continuous improvements, but flaky or brittle tests defeat that purpose entirely.

Every test that you write doesn't only carry the cost of authoring it; it also carries the cost of all future maintenance you'll be responsible for related to that test. As such, developers tend to view tests as a burden, and who wants to voluntarily take on a burden?

❓ Results aren't trustworthy

The whole point of writing tests is to verify that any changes you make to your application don't break the user experience. If you have to investigate every failure due to the flaky nature of most testing frameworks, it defeats the whole purpose of testing (to improve reliability, prevent bugs, and move faster). This untrustworthiness makes writing tests feel pointless, and nobody wants to do pointless work.

✅ Full coverage is difficult to achieve

Browser testing technologies have not kept up with the increasing complexity of web applications. Tools like Selenium or Cypress don't have easy ways to handle user experiences like third-party integrations, file uploads or downloads, interacting with maps, or sending/receiving emails.

If those types of flows are important to your product, then writing tests for the functionality around them will feel pointless, because it only represents a partial solution, and won't be testing the true user experience.

Nobody wants to build a partial solution. When faced with a partial solution, it's easier to build nothing at all.

What needs to change?

Getting engineers to write better tests means solving the problems that make them bad at writing tests in the first place. So to get engineers to write better tests:

We need to make tests easier to write
We need to make tests easier to maintain by eliminating flakiness
We need to eliminate false negatives and false positives from our test results
We need to make it possible to test the full functionality of our applications

Luckily, walrus.ai is here to help!

End-to-end tests can be written in plain English in minutes, without handling any mapping on your end
Tests are maintained entirely by walrus.ai, so you don't need to update tests when you change your application
Results are verified by walrus.ai, so you only get true failures or true passes
Walrus.ai can test the most complicated flows, including third-party integrations, file upload or download, 2fa, or email-based flows.

Tips for working remotely as a developer?

Scott White — Thu, 12 Mar 2020 22:48:22 +0000

In light of recent events, lots of folks are probably working remotely who typically wouldn't.

I'd love to hear from this community what works help you be productive at home, and also what helps you stay sane when you're in the house all day!

Three Common Pitfalls of CI/CD, and How to Avoid Them

Scott White — Fri, 22 Nov 2019 22:52:30 +0000

CI/CD is ultimately about getting your ideas to market faster by continuously shipping code in smaller, more testable increments. However, to successfully adopt this process, it requires the coordinated buy-in of the entire engineering and testing teams, the adoption of tools to facilitate the process, and major process changes.

Moving to CI/CD is risky if you don't take the time to think through how to handle the transition.

So what are the common mistake teams make when switching to CI/CD?

Making it too easy to bypass pipelines

CI/CD is kind of like working out — it requires a specific set of habits, and a lot of discipline to stick to them. If you skip too many days at the gym, you fall out of the habit and stop going entirely.

In a similar vein, if you give your team a way to bypass part of your CI/CD pipeline, the whole process won't stick. When your (hopefully automated) testing catches bugs, it'll be too tempting to bypass your checks, convincing yourself you'll fix the issues later.

Only in the most extraordinary circumstances should you give anyone a way to bypass pipelines.

Picking the wrong metrics to track, or not tracking metrics at all

Ideally, CI/CD should help you get your ideas to market faster, reduce the risk of any given deployment, ship fewer bugs to production, and ultimately make your team happier because they're shipping more code.

If those are your goals, the metrics you track should align with those objectives. Aggregate lines of code doesn't tell you anything about bugs, the customer experience, or velocity.

Your metrics should measure velocity and highlight chokepoints in your processes

Velocity requires every step of the continuous delivery process to happen quickly, with minimal transition friction. Track how long the transition takes between each step in the process, and the main causes for slow transitions.

How long does it take for your team to discover and review pull requests
Is your unit and integration test coverage high?
How often are your tests failing?

Your metrics should track the customer experience

CI/CD requires moving quickly and confidently. Automated tests help in terms of providing a layer of protection against a broken customer experience, but rarely can they catch everything.

Identify key business metrics that reflect the true state of the customer experience.
Make dashboards of key metrics publicly viewable, so if a deploy goes out that breaks the customer experience, you can catch it quickly.

Building CD on top of unstable CI

Automated tests are essential to any successful CI/CD process. To continuously deploy, requiring human intervention to test for bugs is untenable.

Lack of test coverage

Moving to CI/CD without test coverage defeats the entire purpose of the process. It doesn't matter how often you push to production — if your product has bugs, you are delivering a poor customer experience.

Tests help you spot bugs before your customers do, and culturally prevent you from shipping buggy code.

By no means should you always have 100% test coverage, but you should always ensure that the most important flows for the customer experience, and any experiences governed by business logic are thoroughly tested.

Flaky tests based on poor testing frameworks or standards

If you write tests that need to be changed any time you refactor code, you may end up worse-off than if you never wrote them in the first place. Tests should be abstract enough to track user stories (how will these user progress through the onboarding experience), and not so specific that something as small as change in button text would cause a failure.

Furthermore, if you do not have a strong process for training engineers to implement tests, the process of writing them will be untenably slow, and defeat the purpose of CI/CD.

Lack of alignment between development, testing, and other stakeholders in the customer experience

In a strong culture of CI/CD, the engineers who write the tests, the testers who bug-bash flows, and the product teams who manage the customer experience should all have a say in which tests are written, and their ongoing maintenance.

Up front, product managers should enumerate the user stories for testing, engineers should implement tests per those stories, and testers should ensure the customer experience behaves as expected, as well as monitor production metrics to ensure no business logic fundamentally fails.

How to build a better culture of testing

An effective implementation of CI/CD requires organizational buy-in, access to right tools, and continuous measurement.

A strong culture should care about implementing tests, and monitoring that the customer experience behaves as expected after every deploy.

Tests should be easy to write, and flexible enough to not break when code is refactored
Tests should be flexible enough to not break when code is refactored
Product teams should be invested in the testing process — enumerating user stories that are important to validate during CI pipelines

And your tools should make all of that easy. At walrus.ai, we're trying to do just that.

Five Ways to Shorten your Continuous Delivery Cycle

Scott White — Fri, 08 Nov 2019 15:25:50 +0000

The age of multi-month release cycles is coming to a close, and the movement of Continuous Integration (CI) and Continuous Delivery (CD) is taking its place.

CI/CD is ultimately about adopting a set of practices and tools to accelerate the deployment of software, from the moment its written, to the time it is live on a site.

The benefits of CI/CD are numerous:

You can get to market with your initial idea or new feature faster
The risk of any given deployment is smaller
Because your team will be able to reliably ship code without manual intervention, they will feel more productive, professionally fulfilled, and ultimately happier

The touch-points in the CI/CD release cycle are numerous, and should be optimized individually to cut down cycle time as much as possible, so your team can deliver software faster, more reliably, and with fewer headaches.

Understanding continuous deployment

At its core, CI/CD is about increasing the speed in which software can be delivered to customers by automating the deployment of code while maintaining confidence in its stability.

Continuous Integration (CI) is the practice of continually integrating code into one shared repository, and ensuring that the code delivers the desired customer experience. Continuous Delivery (CD) governs how that continually integrated code is actually deployed in an automated fashion.

There are four essential ingredients for any successful CI/CD process:

Code is checked in as frequently as possible
Test coverage stays high
Tests are fully automated
Deployment happens without the need for human intervention

Maintaining a culture of CI/CD requires significant investment in process, an organizational commitment to velocity, and tools to facilitate each step in the code review review, CI, and CD stages of delivery.

The five ways to speed up your continuous deployment cycles

To increase the velocity at which you launch software, you should optimize every stage of the delivery cycle, from code review, to building and testing your code, to actually deploying the changeset.

Here are our five tips to shorten your cycles, starting with how you write code, and ending with how you deploy it.

Identify the minimum viable changeset

Keeping pull requests small both increases the speed at which they are reviewed, and reduces the risk associated with downstream failures.

If your pull requests are massive, no one will want to review them. As such, writing smaller PRs decrease the "time-to-first-review" for each PR.
Smaller PRs are easier to review, and so the "time-stuck-in-review" goes down the smaller the changeset.
Large PRs that touch many parts of the codebase carry a larger risk of subsequent merge conflicts, as well as potential test failure.

Reduce time-to-discovery of pull requests

Code sitting in limbo carries a great cost. It slows down velocity, and discourages the writer of the code.

To ensure PRs are reviewed promptly, adopt tools to surface accountability to the team. Pull reminders is a great way to surface PRs that need review, and notify team members in a public channel.
Eliminate the bystander effect by adding fewer reviewers to each PR, so each reviewer feels more accountability for that review.

Automate your testing

Test automation is an essential ingredient to a well-running CI/CD process. Dependencies on manual QA or any sort of human intervention hold up deploys and defeat the purpose of continuous delivery.

However, if not implemented correctly, tests can hold up your deploys just as much as humans. Flakey tests caused by frequent changes to the site can lead to failures.

Ultimately, to maintain continuous deployment, you need high coverage, with flexible automated tests. You can't just have one.

Want better test coverage with one line of code?

Try walrus.ai – integration tests in one line of code, in plain English. We handle the rest.

Adopt a feature flag system

Feature flags are a great mechanism to test changes in production without rolling them out to all of your users at once.

Create dynamic flags that deliver the new experience to people in your specific domain.
Ramp up exposure of your updated experience over time in order to reduce the risk of your changeset even further.

Focus on production monitoring

CI/CD requires moving quickly and confidently. Automated tests help in terms of providing a layer of protection against a broken customer experience, but rarely can they catch everything.

Identify key business metrics that reflect the true state of the customer experience.
Make dashboards of key metrics publicly viewable, so if a deploy goes out that breaks the customer experience, you can catch it quickly.

How to Work with Designers as an Engineer

Scott White — Tue, 05 Nov 2019 17:33:37 +0000

A product's design isn't just how it looks. It's how it works, and how it makes you feel.

As the point of discovery for new products transitions from real-world sales to web and mobile interfaces, design is increasingly a critical differentiator for companies trying to acquire and retain customers.

At the same time, advancements in frontend architecture, and the movement to more frequent site updates through continuous deployment, means that design and engineering teams have to work more closely than ever before.

The relationship between design and engineering rests on a few core principles:

Engineers need to empathize with designers — what they care about, what they look for in a design, what their ideal process is.

Design needs to understand engineering constraints — what can be built quickly, what's possible given team staffing.

Both need to develop an intuition for the user — at the end of the day, both want to deliver something that solves an important customer need.

Developing design intuition

As with most collaborative relationships, the key to developing a strong partnership relies on understanding your colleague's core objectives and their process for accomplishing those.

At first, understanding a design is not as intuitive as reading a product spec or reviewing code, because the intention behind the design is not explicitly spelled out.

Seek to understand intention

Simply put, design is the process of identifying and solving problems. When reviewing designs, ask the designer why they made the choices they did. What problem are they trying to solve? What are the principles that govern this design? Why is this solution ideal?

Ask about process

The fidelity of a design is a function of where it currently lives in the design process. Some designers are comfortable showing designs before they've been critiqued. Some are comfortable sharing only after they've been tested with users. When you receive designs, ask where they are in the designer's process. Are they ready for implementation? Waiting on user feedback?

Understand design systems

Most companies have established design patterns that are used to make iterating on new work faster. Like an alphabet for design at the company. When you understand both the intention behind a given design, as well as the patterns or tools that are already available in bringing that intention to fruition, you're in a better position to critique a design. Is this custom element worth building when we already have a component that sufficiently solves the problem we are trying to address, and doesn't fly in the face of the intention behind the overall design?

Providing engineering context

There are many avenues to solve a particular problem. There are fewer avenues to solve that problem quickly. If you want to deliver software that solves a customer problem quickly, it's critical to identify the lowest-scope solution that is still viable. To identify the "minimum" in "minimum viable product", you need to understand what constraints you're operating under.

Communicate constraints early

In the early stages of a design, the designer is typically thinking more about the problem they are trying to solve, the customer profile of the person who suffers that problem, and what the ideal solution looks like.

As an engineer, providing context about the constraints of the system are helpful in shaping the studio-space for the designer, and preventing him/her from exploring paths that would ultimately be infeasible.

Give your designer guard-rails — what data is available in this part of the user experience, what kinds of elements take a lot longer to build than others, how much more time building a custom component would take than using an existing one from the design system.

Make yourself available

Go to design reviews. Make yourself a resource during the exploration phase of a designer's process. Sit with your designer. Teach your designer why certain approaches are more feasible than others. Help your designer develop her/his own intuition about the constraints of your system.

Developing empathy for the user

Ultimately, the goals of the engineer and the designer are the same — to solve a problem for the customer. The customer and his/her problem have to be the north star, both for the design and the implementation.

Find a common vocabulary

When discussing a design or the final implementation of that design, it helps to have a well-understood vocabulary with which to provide feedback. Write down the problem you're trying to solve, and use it as a reference when providing feedback. Come up with the principles that govern the solution, and validate that what you're implementing fulfills those principles.

Many disagreements stem from simple miscommunication. Coming up with a common vocabulary with which to discuss designs ensures that disagreements are about what is actually important — the customer experience, and not the words we use to describe it.

Fill in the gaps

Even the most thorough designs rarely cover all of the possible edge-cases that one could experience in the real customer experience. As the engineer implementing the design, you'll probably encounter these edge-cases before anyone else does.

Understanding the problem you're trying to solve, and the intention behind the overall design will enable you to make important judgement calls when fixing these edge-cases. You can always work out the kinks later, but coming up with a strong first-pass based on the principles governing the overall solution will always be much faster than going back to design for further iteration on edge-cases.

Meet your customers

At the end of the day, there's no better way to understand how your customer users your product than to actually watch them. Developing an intuition for your customer's pain points, what they care about, and how they actually use your product will sharpen the judgment you use to make any change to the customer experience.

Are you an engineer who cares about the user?

The first step to writing any piece of software should be to identify the user story for what you're building — what problem is it solving, and exactly how will the user interact with it?

Then you have to write tests to ensure your user can actually complete the story you want them to. Wouldn't it be great if you could just write the user story, and the tests would write themselves?

With walrus.ai, you can write integration tests in one line of code, in plain english.

Four Unexpected Benefits of Writing Tests

Scott White — Mon, 04 Nov 2019 17:07:36 +0000

No one likes writing tests.

If you're setting them up for the first time, it's a pain (getting your servers going, training your team on a framework, etc.). Even if you're used to writing them, it can feel like overkill when you're just trying to develop new features that customers will get value from.

Despite being a pain to write, integration tests are a great example of delayed gratification — there's an up-front cost, but they pay for themselves in the long run.

In fact, a test's role in preventing a bug makes it some of the highest-value code an engineer can write.

Tests save time — and not just for the developer

Bugs do not exist in a vacuum.

Most bugs are discovered by the users themselves, where they then make the long-and-windy journey back to the developer with the right context. Along the way, a bug might be handled by a support person, who then passes it along to their boss, who might get in touch with a product manager, who will know an engineer who is familiar with that part of the codebase.

At each step in this journey, the bug gets more expensive as time is invested from each stakeholder in its investigation and resolution.

All the while, the customer is waiting for a resolution, every minute more sure that they want to cancel.

Integration tests eliminate that whole journey. The moment a bug is deployed related to someone's code, that person gets notified. No more telephone, no more lengthy resolution process.

Tests improve decision-making

Many companies use A/B testing to measure the impact of new features, or user-facing changes of any kind to their site.

When you introduce more variants (test groups) to your site, you increase the probability of bugs, as you are supporting multiple different versions of the site which in turn increases the overall surface area you need to support.

Bugs in any of these variants can have the effect of skewing the results of your experiment. Furthermore, without test coverage, you may not even be aware that these bugs exist and are influencing the outcome of your experiment.

As such, implementing tests in your experimental code ensures that you don't make important decisions based on flawed data.

Tests help your brand

Social media, customer reviews, and word-of mouth are increasingly the primary ways that millennials make purchasing decisions.

Nothing kills a potential customer's intent to purchase like seeing an angry tweet about a bad bug.

A bug-free experience is a prerequisite for customer acquisition and retention, and integration tests are the best lever you have to ensure that your customer experience is bug-free.

Tests print money

Tests have high ROI on two dimensions: they prevent money-losing bugs, and they help developers spend more of their time building features, not fixing bugs.

For any critical business process (such as customer checkout, coupon code redemption, etc.), tests can make all the difference. Without tests, you increase the probability that a fraudulent actor can exploit your coupon system, or a legitimate customer can't make it through checkout.

Furthermore, while tests have an up-front cost (you have to write them), they save time in the long-run, particularly when you take into account the many stakeholders that are involved in solving bugs.

But writing tests is hard...right?

Historically, writing tests has been a pain. You have to get servers running, adopt a framework, and write them for every new feature.

Writing tests does not have to be hard

With walrus.ai, you can write integration tests in one line of code.

⚡️ Fast: Get results in 5 minutes
🏝 Easy: Write tests in plain English, and we'll handle the rest
👩🏻‍💻 Developer-first: One API call, no sales calls

Try if free!

How to Work Remotely Without Going Insane

Scott White — Fri, 01 Nov 2019 17:28:15 +0000

The future of work is remote.

In recent years, fully-remote companies like GitLab, Zapier, and InVision have proven that a remote culture is not just a perk, but a clear business advantage. Technology is accelerating this transition. We can chat instantaneously in Slack, coordinate a global videoconference through Zoom, and track our entire roadmap in Asana.

For employees, remote work provides more flexibility (hours, family time, no need to move when switching jobs), fewer distractions (no sales calls happening next to your desk), and shorter commutes (or no commute). For companies, a remote culture makes it easier to hire, reduces office costs, and naturally attracts self-motivated workers.

Despite its advantages, remote work can have drawbacks. Between loneliness and isolation, organizational ambiguity, unclear work/life boundaries, and complex coordination, remote work offers several challenges.

None of these challenges are insurmountable. At walrus.ai, we've thought a lot about how we can improve our remote-first culture. By introspecting on the disadvantages of remote work, we can structure our days thoughtfully to mitigate or eliminate these disadvantages at the root-cause.

Remote work is not without challenges

Working remotely can be lonely

Humans are social animals. Research suggests that when unplanned, spontaneous social interactions prevent loneliness. When you work remotely, you miss such interactions that are endemic to the office — chatting at the desks, passing people in the hallways, sitting down for lunch. If you're not careful, it's easy to feel isolated, particularly if you're transitioning into a remote job from a bustling office.

It's easy to feel out of the loop

If documentation at your particular company is poor, it's easy to feel lost as a remote worker. If your goals are ambiguous, remote policies ill-defined, and spontaneous meetings occurring behind closed doors, it's easy to feel like you're left out as a remote employee.

Coordination is challenging

When a team works across multiple time-zones, coordinating real-time meetings is challenging. A bright-and-early meeting in San Francisco might be squarely when you want to tuck in your child in Amsterdam.

Work/Life boundaries can become blurred

If you live, work, and sleep in the same physical space, the lines between these your work time and your personal time can easily fade. Furthermore, the flexibility of remote work can cause your work hours to be different than your social community, increasing feelings of isolation.

How to structure your days to live your best remote life

✅ Define Done

When you are in a traditional office, it's clear that the day is ending when people start to filter out. When you work remotely, you don't have such obvious cues, and it's easy to lose track of when you should stop working.

To counteract this, give yourself a target. When you start working for the day, choose a rough timeframe when you want to finish your work for that day. If something about your day changes significantly that causes you to step away from working (exercise, last-minute appointment, etc.), re-evaluate and adjust your target end-time, and try to stick to that time.

▶️ Identify triggers

When you sit down at your desk in an office, your body knows it's time to work. When you wake up in the same place you'll work for the day, there's no clear start to the work day.

Teach your brain to start working by associating certain actions (or "triggers") with the commencement of work. For example, walk around the block, and when you return, start your work. Or simply stay in your PJs up until you're ready to start working, then change your clothes and immediately kick off your work day.

Choosing a routine or process to start the day will help sharpen the lines between life and work, and will get you into deep focus faster.

📄 Document everything

Successful remote cultures rely on strong documentation. When in doubt, you should document what you are working on, even when it feels like too much. Create publicly-trackable tasks for everything you plan on working on. Block out time on your public calendar so everyone knows what you're up to during the day. Write down your approach for whatever task you're working on in a publicly-viewable document.

Thoughtful and publicly available documentation helps others gain visibility on what you're working on, reduces the need for meetings, and gives you more structure in what you're going to work on in a given day. Furthermore, the more available documentation, the less likely anyone will feel out of the loop with respect to decision-making thought processes or what's being worked on.

👩🏻‍💻 Choose a work area

If behavioral psychology has taught us anything, it is that animals behave predictably to repeated stimuli. Just like a dog might start salivating when you walk to the cabinet in which its food is held, you'll want to take a nap if you climb into bed.

To get into the work mindset, choose an area of the house (or a different place, like a co-working space or a coffee shop) to start your day that you only associate with work.

And for the love of god, don't work in your bed, or you'll want to sleep while you work, and you'll think about work when you're trying to sleep.

❌ Eliminate unhealthy micro-temptations

When we finish a small task, it's in our nature to want to take a break. These breaks can be healthy – they provide us the opportunity to take stock of where we are, recharge, and focus on the next task at hand.

However, there's a limit past which these breaks take us into unproductive territory.

If too many micro-temptations (easily available snacks in the kitchen, the TV playing in the background) are present while you're trying to work, you may find yourself constantly getting distracted whenever you finish a task, no matter how small.

To stay in the zone, keep the snack food farther out of reach, keep the TV off, and you'll get more done, and feel a greater reward when you take a well-deserved break.

👋 Find time for informal communication

Spontaneous interactions, and unstructured time with friends and colleagues make us happier. To avoid feeling lonely while working remotely, proactively set aside time with your team to get to know eachother outside of a work context — coffee chats with folks in your area, social calls, or team offsites are all good ways to bond.

Furthermore, and ironically, working remotely may also force you to be more proactive about scheduling time to spend time with your friends, particularly if you work non-traditional hours.

Lastly, play with your work environment. Go to a coffee shop or a library where other people like you will be working.

📈 Measure results, not hours

When you're in an office, it's natural to measure productivity in terms of "days of work". Remote work requires more self-discipline than simply clocking in and out of an office. At a company level, a culture of measuring results gives every employee more clarity about what he/she should be working on.

At an individual level, setting measurable goals and tracking results is a great way to stay on track with the global objectives of the company without constantly checking in with other people, as well as a good way to determine when to stop working (when you accomplish the specific goal you set out for a day).

Wrapping it up

As more companies adopt tools to enable better synchronous collaboration and asynchronous task management, the shift to remote work will continue to grow.

To ensure this transition is successful, both for the company, and for the individual, specific measures should be taken to address the inherent disadvantages of remote work.

At the end of the day, you can conquer these disadvantages by structuring your days to draw clear lines between work and life, triggering your mind to focus on work, and building in social time to your schedule.

Have any other tips for living your best remote life? Let us know!

Are you a remote engineer?

Remote teams use walrus.ai to automate their integration tests. Try it out free!