DEV Community: walrus.ai

Prompting Users to Reload Your Next.js App After an Update

Jake Marsh — Mon, 05 Apr 2021 16:37:25 +0000

The pages of a Next.js application are served in one of two ways: server-side rendering, or client-side rendering. It's important to understand the distinction, and when each scenario occurs. (There is also static generation, but we will disregard that for this walkthrough.)

Server-side rendering is when the underlying Node.js server is handling the request, loading the corresponding page component (and any data dependencies), and returning the populated HTML that results. A page will be server-side rendered if it's the initial request to load the page, and the page implements either getInitialProps or getServerSideProps.

Client-side rendering is when the Javascript in the browser has taken over the handling of the request, and React will handle rendering the new page component and reconciling any differences. Client-side rendering occurs when the user has already loaded your application, and is navigating via the Next.js router (either directly) or via the <Link /> component.

The important caveat with client-side rendering is that once the user has loaded the application and each of the pages, requests are no longer being made to the server to render any of them -- the client is handling it all. This means that if you deploy a new version of your application while someone is using it, they could continue seeing and using the previous version of your application until they happen to reload.

This can cause issues if you're making breaking changes, or fixing bugs, or making any other changes you'd prefer your users see ASAP. This risk is multiplied by the number of people using your application. So how can you handle new deploys on the client to ensure our users get the latest version?

Next.js allows customizing the Webpack configuration used at build time via a next.config.js file. It will automatically pass in various relevant arguments; the one that we're interested in is buildId. By default, this is a random string unique to each build.

Combined with Webpack's DefinePlugin, you can expose this buildId to our application by replacing any checks for process.env.BUILD_ID with the real buildId:



// next.config.js

module.exports = {
  ...
  webpack(config, { buildId }) {
    config.plugins.push(
      new webpack.DefinePlugin({
        'process.env': {
          BUILD_ID: JSON.stringify(buildId),
        },
      }),
    );

    return config;
  },
};

This means that the resulting bundles served on the client will have the real buildId available to them when checking process.env.BUILD_ID. Since these bundles stay loaded as client-side navigation occurs, this will remain a static reference to the buildId loaded on the client.

Next, you'll want to also expose this process.env.BUILD_ID variable in our server-side environment. This is because when you deploy a new version of your application, anything being handled by the server will immediately be operating on the newest version. You can do this via Next.js's API routes:



// pages/api/build-id.ts

import { NextApiRequest, NextApiResponse } from 'next';

export default (_req: NextApiRequest, res: NextApiResponse): void => {
  res.status(200).json({
    buildId: process.env.BUILD_ID,
  });
};

With this new endpoint exposing process.env.BUILD_ID from the server, you have a route we can hit at any time to get the newest deployed buildId: /api/build-id.

Since the client will now have a static reference to its own buildId, and the server now has the endpoint always returning the newest buildId, we can implement our own polling and diffing to determine if the user needs to reload. Below is a component that encapsulates this logic, polling for the latest buildId every 30 seconds via a useInterval hook. This can then be rendered anywhere in your application.



import { useEffect, useRef } from 'react';
import request from 'superagent';

function useInterval<P extends Function>(
  callback: P,
  { interval, lead }: { interval: number; lead?: boolean },
): void {
  const savedCallback = useRef<P>(null);

  useEffect(() => {
    savedCallback.current = callback;
  }, [callback]);

  useEffect(() => {
    const tick = (): void => savedCallback.current();

    lead && tick();

    if (interval !== null) {
      const id = setInterval(tick, interval);

      return () => clearInterval(id);
    }
  }, [interval]);
}

export function DeployRefreshManager(): JSX.Element {
  useInterval(
    async () => {
      const { buildId } = (await request.get('/api/build-id')).body;

      if (buildId && process.env.BUILD_ID && buildId !== process.env.BUILD_ID) {
        // There's a new version deployed that we need to load
      }
    },
    { interval: 30000 },
  );

  return null;
}

At walrus.ai, we display a non-closable modal to the user with just one possible action: reloading the page. Clicking this button simply calls window.location.reload().

This is achieved by simply setting a boolean state value in the above if statement, and conditionally returning your modal element from the component instead of always returning null.

You'll now be able to rely on the fact that your users are always using the latest version of your application, or at least being prevented from taking actions until they've reloaded.

Using Google's OCR API with Puppeteer for Visual Testing

Scott White — Mon, 08 Feb 2021 23:11:53 +0000

The Puppeteer framework is a Node.js library that can be used for automated browser testing. This library provides high-level access to chromium-based browsers through the dev tools protocol. In automated testing, you can use assertion libraries like chai.js or should.js to make assertions about elements or objects that should appear in an application.

One of the most common scenarios in automated testing is to assert if a word or phrase is displayed in a web application. There are few methods available for a developer to accomplish this task.

One of them is to iterate through all the elements in a web page to look for the matching word. However, this method is inefficient and error-prone. Besides, querying every element is time-consuming and will result in incorrect assertions as you have not scoped any part of the page, meaning the word could appear in unexpected locations.

   // Iterate through all elements
          const contents = await page.$$('*');

          for (let i = 0; i < contents.length; i++) {
              const text_value = await (await contents[i].getProperty('innerText')).jsonValue();
              if (text_value.match(/Hello/g)) {
                  console.log(text_value);
              }
          }

A better method is selecting elements using the XPath to find the element which contains the specified word. This is far simpler and more efficient than iterating through all the elements. However, without a defined scope, this method will also provide results from the whole web page.

          // Using XPath
          const title = await page.$x("//h1[contains(text(), 'Domain')]");
          let text = await page.evaluate(h1 => h1.textContent, title[0]);
          console.log(text)

The most effective method would be to use optical character recognition (OCR) to specify a part of the page and carry out searching within that scope to find the location of the word. Using OCR allows you to utilize a powerful character recognition algorithm without having to go through HTML elements.

What is OCR?

Optical Character Recognition, or OCR, is a technology that enables you to convert images of text content into machine-readable, digital data. OCR can convert any type of text content, whether it’s handwritten or printed, into editable and searchable data using an image of that text content. This is done by processing the image through a machine-learning algorithm to clean and recognize each character of a specific image.

There are multiple open-source OCR tools like pytesseract or EasyOCR, which can be used to integrate OCR functionality into a program. However, these tools require significant configurations to get up and running to provide results with an acceptable accuracy level.

Google provides a ready-made solution to integrate OCR functionality to an application using its Vision API. It will be used in this tutorial since it abstracts most of the model fine-tuning. The Vision API provides two distinct detection types to extract text from images, as mentioned below.

TEXT_DETECTION - This will detect and extract text from any provided image. The resulting JSON will include the extracted string and individual words with their bounding boxes.

DOCUMENT_TEXT_DETECTION - This will also detect and extract text from any image but is optimized to be used for dense text and documents. Its JSON output will feature all the details, including extracted strings plus page, block, paragraph, word, and break information.

Let’s take a look at the following image, which contains a dense text block.

The following code can be used to implement both detection types to extract the text in that image but there will be some differences in the output. If you want to use “DOCUMENT_TEXT_DETECION”, uncomment it and comment the line type: 'TEXT_DETECTION'.

const [result] = await client.annotateImage({
        image: {
            content: imageContent,
        },
        features: [{
            type: 'TEXT_DETECTION',
            // type: 'DOCUMENT_TEXT_DETECTION',
        }]
    });

    const textAnno = result.textAnnotations;
    textAnno.forEach(text => console.log(text));

TEXT_DETECTION Output

DOCUMENT_TEXT_DETECTION Output

As you can see, while TEXT_DETECTION was able to simply extract the text from the image, the DOCUMENT_TEXT_DETECTION was also able to identify the period (.) punctuation mark as the separating character. This is due to the DOCUMENT_TEXT_DETECTION being geared towards denser text blocks like documents and can extract more intricate details, while TEXT_DETECTION is more suited for large text objects like street signs, billboards, etc.

Google Vision API with Puppeteer

In this section, you will be using DOCUMENT_TEXT_DETECTION as you are working with web content. Now, let’s start incorporating Vision API (OCR) within a Puppeteer program.

First, capture a screenshot of the text area you need to extract text from the Puppeteer's page.screenshot() function.

 const browser = await puppeteer.launch({headless:true});
        const page = await browser.newPage();
        await page.goto('https://www.example.com');  

        const imageFilePath = 'ocr-text-block.png'

        // Take screenshot
        await page.screenshot({
            path: imageFilePath,
            clip: { x: 0, y: 0, width: 800, height: 400 },
            omitBackground: true,
        });

Then you have to encode the image to base64 format in order to pass the resulting data to the Vision API.

        // Read the image contents
        const imageContent = Buffer.from(fs.readFileSync(imageFilePath)).toString('base64');

Next, pass the data to the Vision API and you will get the following result.

        // Pass the content
        const client = new vision.ImageAnnotatorClient();
        const [documentAnnotateResult] = await client.annotateImage({
            image: {
                content: imageContent,
            },
            features: [{
                type: 'DOCUMENT_TEXT_DETECTION',

            }]
        });

        // Get the annotated result
        console.log(documentAnnotateResult.fullTextAnnotation)

As you have obtained the OCR result without any errors, compare the image you have captured with the corresponding output.

Captured image (ocr-text-block.png)

OCR Result

That’s it! Congratulations, now you have successfully integrated the OCR functionality. The next step would be to iterate over the text annotations and find a match against the string you are searching for. In this instance, you will be searching for a string called “permission”.

        var point_x = 0
        var point_y = 0
        //Iterate the elements
        const textAnno = documentAnnotateResult.textAnnotations;
        textAnno.forEach(text => {
            if(text.description === 'permission') {

                point_x = text.boundingPoly.vertices[0].x;
                point_y = text.boundingPoly.vertices[0].y;            
            }
        }
        );

Iterate through the results using a forEach loop and if a matching text is found, obtain its coordinates from boundingPoly.vertices and store them in point_x and point_y variables.

Then, you have to invoke the document.getElementFromPoint function to get the DOM element that contains the matched text.

        // Find the element using elementFromPoint
        const foundElement = await page.evaluateHandle((x,y) => document.elementFromPoint(x,y), point_x,point_y);
        console.log(foundElement._remoteObject.description)

Since the word “permissions” is inside a paragraph tag (<p>), the console output will show a DOM object description attribute like the following.

Limitations of OCR

OCR is a powerful tool when it comes to identifying web content. However, it is not foolproof, and its major issue will be the accuracy of character recognition.

Missing some characters entirely.
Mixing up similar characters (6 to b, o to 0, I to l, etc.)
Messing up spaces in documents.

The points mentioned above are some facts that lead to the inaccuracy of the result. Refer to the following image and its OCR result.

Image

OCR Result

You can assert that the word in the image is “Los Altos”, but the OCR function mistakenly detects the letter “t” as the plus (+) sign. This will cause incorrect character recognition and even breaks the word Altos into three separate segments. (AL, + and OS). These kinds of issues are very common when dealing with handwritten text in images or documents.

To mitigate these issues and keep the Puppeteer test resilient, you need to have some room for variance in OCR results. The simplest method would be to handle the commonly mistaken characters. For example, suppose you encounter a 6 that doesn't match your string, substitute 6 with b, and check again. However, This method will quickly become cumbersome and not a very flexible solution to handle multiple inconsistencies. \

Another option would be to implement a word distance algorithm based on a string metric like Levenshtein distance, which measures the difference between two sequences. You could specify a threshold that allows for some variance in the results while still getting the confidence that the desired text was found in the specified area.

Going beyond visual testing

Visual testing is an essential piece of the testing toolkit. It's a great way to ensure you aren't introducing visual anomalies as you make changes to your app, and it can be helpful in making simple assertions like the presence of text or images.

However, image-based-testing is not one-size-fits-all. To successfully end-to-end test your product, you'll likely need to support a much wider range of assertions and actions (including uploading files, verifying text messages or emails are received, etc.).

To support the automation of testing these harder assertions and experiences, you'll need to expand your toolkit to go beyond visual testing.

Looking to go beyond visual testing?

walrus.ai provides visual assertions and handles everything mentioned in this article out of the box. Beyond that, walrus.ai can be used to test your hardest user experiences end-to-end. Drop us a line if you'd like to see it in action!

5 Tips for Writing Better Unit Tests

Scott White — Thu, 04 Feb 2021 17:07:00 +0000

What are Unit Tests?

Unit testing is the testing of individual parts or components of a program. The purpose of unit testing is to test the functionality of a single component and verify its behavior independently from other components. A unit in unit testing includes the smallest piece of functionality that can be tested. The main objective of a unit test is to isolate the smallest parts of a program and test the functionality of each part. The basic structure of a unit test is designed to check if the output of a defined code block matches the given condition. For example, when testing a function, we specify a set of input arguments and test if the function returns the expected output, as shown below.

func.py

# ==== Function ==== #
class Calculations:

  # Function to Test
    def calculate_total(value1, value2):
        return int(value1 + value2)

test.py

# ==== Unit Test ==== #
import unittest
from func import Calculations

# Test Class
class TestingFunctions(unittest.TestCase):

    # Test Case
    def test_calculation(self):
        input_one = 4
        input_two = 6
        total = Calculations.calculate_total(input_one, input_two)
        # Identify if output is equal to zero
        self.assertEqual(total, 10)

if __name__ == '__main__': 
    unittest.main()

Avoid Complex Setups

Since we are testing the functionality of the simplest testable unit during the unit testing, it is better to keep the tests also as simple as possible. One of the major difficulties developers face during unit testing is the test cases becoming hard to understand and change due to making them unnecessarily complex.

The AAA Rule provides a good guideline to keep a unit test simple as much as possible. In unit testing, AAA stands for Arrange, Act, and Assert. The first step is to arrange the test case with the input variables, properties, and conditions to get the expected output. The next step is to act upon your arrangements by calling the methods or functions. In the final step, which is the assertion, you can assert to verify the expected output using a test framework. This method makes the codebase of test cases much easier to read and understand.

import unittest

# AAA Methodology
class TestingStringFunctions(unittest.TestCase):

    def test_string_manipulation(self):
        # Arrange
        text = "Hello World"

        # Act
        new_text = ''.join(reversed(text))

        # Assert
        self.assertEqual(new_text, 'dlroW olleH')

Another important thing to consider when performing unit tests is the test dependencies. If a test case depends on another test case, changing a single test case could affect other test cases, which will, in turn, defeat the purpose of individual unit tests.

Exhaustiveness is not the Goal

It is always a good idea to perform comprehensive unit testing for your program. However, it does not mean that you should write test cases for each and every scenario. Considering every edge case beforehand is a waste of valuable development time.

Tests should be straight forward covering the common flow of your code block. Once simple tests are passing, you can increase the coverage of the test to include edge cases and cover multiple boundaries. When a bug is discovered in the development cycle, you can expand the test coverage to include that bug. This will keep your test cases up to date without having to create separate test cases to tackle the bugs.

Write Tests First When Fixing Bugs

The first intention of a developer when discovering a bug is to try and fix it somehow. Although she would be able to fix the bug at that time, her understanding of the bug may be limited and it might cause new issues down the pipeline in the future.

Therefore, it is always a good practice to create a test for the discovered bug encompassing the components that cause the bug. This way, you can verify if the bug relates to the indicated components or is caused by an external factor. Having a simple and straightforward test will lead to an easier bug fix by quickly exposing the problems.

When you write your test, make sure to run it before fixing the bug and verifying that the test fails. Otherwise, you may write a test that doesn’t actually encapsulate the bug, therefore defeating the entire purpose.

Once a bug is fixed, you should not discard that unit test. These tests can then also be used in regression testing to test the overall functionality of the program. It will be an extra benefit as we know both the failing and passing conditions of that test in a future event when a similar bug is discovered. That will also make it easy to know where to apply the fix.

Mock with Caution

Mocking is a way of simulating the behavior of a real object in a controlled manner by creating an artificial object (mock object) similar to it. Mock objects enable developers to mock the behavior of complex functions without having to invoke the function. This is useful when dealing with external services where you require those services only to execute the underlying code block but they are not needed for the test case.

Mocks are also useful in increasing the speed and efficiency of a test case. Without mocks, if a test calls for an external library or a function, you must wait for that function to be completed to continue the test. This will lead to slow test execution times and can even introduce unnecessary errors or behavior inside the test case. For example, when dealing with the file systems and network functions, you could use a mock object to simulate those behaviors, effectively speeding up the test while focusing on the core functionality of the test.

import unittest
from unittest import mock
from func import delete_file

class TestFunctions(unittest.TestCase):

    # Mock the os module
    @mock.patch('func.os')
    def test_file_delete(self, mockobj):
        file_name = "test.txt"
        delete_file(str(file_name))
        # Verify the parameters were passed correctly to the function
        mockobj.remove.assert_called_with(str(file_name))

The most important thing to remember when using mocks is not to mock the functionality of the test case. Mocks should not be used if you are testing for database queries, API calls, or any functionality that has a direct impact on your overall test.

Keep Tests Independent

The tests should be orthogonal to each other and be used to test a single functionality at a time. The test classes should be tested independently and should not depend on anything other than the testing framework.

When an order is needed to carry out unit testing, you can use the inbuilt setup and teardown methods of the test framework to isolate the test cases. A good test case should not be affected by any implementation changes. When dealing with API changes, a program with a solid code base with correctly implemented individual unit tests is vital for identifying errors and making the program up and running again.

Besides, creating independent tests enables us to easily modify each test case while increasing the test spectrum without causing any side effects to other tests. In test automation environments, independent tests are the preferred way to integrate with CI/CD pipelines.

The next step

Unit tests are a great way to discover and resolve bugs in units, ensuring that every single part of the program works well. However, if you want to test different parts of the program working together, in their actual environment, you need to go for integration testing or end-to-end. You can learn more about the testing hierarchy here.

Integration tests abstracted up a level from unit tests, executing and asserting on code paths that traverse between multiple units or modules. Now we're starting to ensure the different pieces of our application interact with each other as expected.

Even farther are end-to-end (E2E) tests. These are tests aimed to cover the entire surface area of your application from top to bottom. These should generally follow the code paths expected from your end-users to ensure they're as close to reality as possible.

Looking for an easy way to take the next step beyond unit testing? Check out walrus.ai

Tests can be written in plain english — no difficult setup required
Test your most challenging user experiences – emails, multi-user flows, third-party integrations
Zero maintenance — walrus.ai handles test maintenance for you, so you never experience flakes

Testing Database Interactions with Jest

Scott White — Fri, 24 Apr 2020 18:15:07 +0000

Testing Database Interactions with Jest

Jest has quickly become one of the most popular Javascript testing libraries. While Jest may be mostly used in the context of frontend applications, at walrus.ai we use Jest for testing our backend Node.js services as well.

Jest aims to make testing 'delightful', and a large component of that delight comes from speed. Jest by default runs concurrently with worker processes, a pattern that encourages and even requires test isolation. While this is relatively simple to accomplish for frontend code, there's a shared mutable state elephant in the room for backend code: the database.

Why test database interactions?

For unit tests, we generally follow best practice of mocking any interactions that are outside the unit. Consider the following function:

async function changeUserName(db, userId, username) {
  const userRepository = db.getRepository(User);
  const updated = await userRepository.updateUserName(userId, username);

  return updated;
}

This function takes a handle to database connection, the userId, and a new username, and updates the username in the database. We abstract away the underlying SQL necessary to make the database update with the Repository pattern. This allows us to test this function rather easily.

describe('changeUserName', () => {
  it('should update username in db', async () => {
    const db = { getRepository: jest.fn() };
    const repository = { updateUserName: jest.fn() };

    db.getRepository.mockReturnValue(repository);
    repository.updateUserName.mockReturnValue(Promise.resolve('updated'));

    const result = await changeUserName(db, '1', 'username');

    expect(result).toEqual('updated');
    expect(repository.updateUserName).toHaveBeenCalledTimes(1);
    expect(repository.updateUserName).toHaveBeenCalledWith('1', 'username');
  });
});

However, what if we want to test our actual Repository? The code for the repository probably looks something like this:

class UserRepository {
  ...
  public async update(id, username) {
      await this.db.sql(`
        UPDATE users SET username = :username WHERE id = :id
    `, { id, username });
  }
}

If we wanted to test method, we could obviously mock the db connection and assert that .sql is called with the expected parameters. But what if this SQL is invalid, or more probably, what if the SQL is valid but is doing the wrong thing?

Presumably, at some point, we'll want to test the actual interaction with the database. We won't get into what we actually call these tests (there's probably 1000s of internet discussions on whether we've crossed the line from unit tests to integration tests by involving a real database), we'll simply cover how to do it safely and concurrently with Jest.

Setting up the database for Jest

As we've discussed, Jest by default runs tests concurrently — this makes sharing a database problematic. Tests that are running in parallel will clobber each other's database state, causing spurious failures and flakes.

The simplest option to overcome this limitation is to run Jest with the --runInBand option. This forces Jest to only use one process to run all your tests. However, this probably will make your test suite far slower. At walrus.ai this took our test suite from 10s of seconds to a few minutes, and simply wasn't tenable for our CI/CD processes of constant deployments.

Luckily, parallel testing with a database is a pretty solved problem. We can simply spin up a database for each worker process we're using. If all tests running against a particular database are run serially, then we don't have to worry about parallel processes mutating database state.

The easiest solution for this would be something like the following. We could spin up a database before each Jest worker, and shut it down after. Since all tests within a worker run serially, each worker could operate on its individual database safely.

beforeWorker(async () => {
  db = await createDatabase(`db_${process.env.JEST_WORKER_ID}`);
});

// All tests run serially here.

afterWorker(async () => {
  await destroyDatabase(db);
});

Unfortunately, while Jest exposes the JEST_WORKER_ID environment variable to distinguish between workers, it doesn't expose any simple way of hooking in per-worker setup and teardown methods.

This means that we can't dynamically spin up and tear down databases for each worker. We can, however, do the next best thing, using a static number of Jest workers.

First, we'll need a test setup script that prepares our multiple databases.

Note: The following code examples use Typeorm, but the code could easily be extended for any other database interaction library such as Sequelize, Massive.js, Knex etc.

(async () => {
  const connection = await createConnection({
    type: 'postgres',
    username: process.env.DATABASE_USER,
    password: process.env.DATABASE_PASSWORD,
    database: process.env.DATABASE_MASTER',
    host: process.env.DATABASE_HOST,
    port: 5432,
  });
  const databaseName = `walrus_test_template`;
  const workers = parseInt(process.env.JEST_WORKERS || '1');

  await connection.query(`DROP DATABASE IF EXISTS ${databaseName}`);
  await connection.query(`CREATE DATABASE ${databaseName}`);

  const templateDBConnection = await createConnection({
    name: 'templateConnection',
    type: 'postgres',
    username: process.env.DATABASE_USER,
    password: process.env.DATABASE_PASSWORD,
    database: 'walrus_test_template',
    host: process.env.DATABASE_HOST,
    migrations: ['src/migrations/*.ts'],
    port: 5432,
  });

  await templateDBConnection.runMigrations();
  await templateDBConnection.close();

  for (let i = 1; i <= workers; i++) {
    const workerDatabaseName = `walrus_test_${i}`;

    await connection.query(`DROP DATABASE IF EXISTS ${workerDatabaseName};`);
    await connection.query(`CREATE DATABASE ${workerDatabaseName} TEMPLATE ${databaseName};`);
  }

  await connection.close();
})();

With this script, we create a connection to our database, Postgres in this instance. Then, for each worker that we are using (set statically in the JEST_WORKERS environment variable, we initialize a new database.

Because we're using Postgres, we can use a handy feature called Template Databases. This makes new database creation cheap, and allows us to only run our migrations once. At a high level, we create one template database, run our migrations once against the template database, and then quickly copy over the database for each Jest worker.

Now, we simply have to connect to the correct database in all of our tests. With the JEST_WORKER_ID environment variable, this is trivial:

beforeAll(async () => {
  connection = await createConnection({
    type: 'postgres',
    host: process.env.DATABASE_HOST,
    port: 5432,
    username: process.env.DATABASE_USER,
    password: process.env.DATABASE_PASSWORD,
    database: `walrus_test_${process.env.JEST_WORKER_ID}`,
    logging: false,
    entities,
    namingStrategy: new SnakeNamingStrategy(),
  });
});

afterAll(async () => {
  connection.close();
});

Now all of our workers will use individual databases, allowing us to run tests in parallel.

Cleaning up between tests

While parallel tests are no longer problematic, we still have a problem with serial tests. Consider the following example, again of our contrived UserRepository:

describe('UserRepository', () => {
    it('should create user', () => {
    await repository.createUser('username');

    expect(await repository.countUsers()).toEqual(1); 
  });

    it('should delete user', () => {
    const userId = await repository.createUser('username');
    await repository.deleteUser(userId);

    expect(await repository.countUsers()).toEqual(0); 
  });
});

Notice anything wrong here? The second test will fail. While we're setting up different databases for each of our parallel workers, tests within the same file are run serially. This means the user created in the first test is still present in the second test, causing the test to fail.

We considered two approaches to solve this problem. The first approach is to wrap each test in a database transaction:

beforeEach(() => {
  db.startTransaction();
});

afterEach(() => {
  db.rollbackTransaction();
});

With this approach, any database updates made within the test are wrapped into the initialized transaction. When the test is finished, we simply rollback the transaction, discarding any of those updates. While this approach is fast and generally supported by all databases, it's not always the best for certain classes of integration-tests.

Sometimes, the behavior under test may actually be the transaction itself. For example, we may want to test that when an update fails, certain components of the update are preserved (committed), and others are rolled back. This logic would require us to manually start and stop transactions within our code, and wrapping the code in a parent transaction with this method would keep us from effectively testing rollbacks.

Another, simpler but slower approach, is to just clear out the database after before every test. While this may be slower, it's less likely to bite us later. We can do this in a simple beforeEach block.

beforeEach(async () => {
    const queryRunner = getConnection().createQueryRunner();

    await queryRunner.query(`
      DO
      $func$
      BEGIN
        EXECUTE (
          SELECT 'TRUNCATE TABLE ' || string_agg(oid::regclass::text, ', ') || ' CASCADE'
            FROM pg_class
            WHERE relkind = 'r'
            AND relnamespace = 'public'::regnamespace
        );
      END
      $func$;
    `);
    await queryRunner.release();
  });

The above code iterates through all our tables and clears them using the SQL TRUNCATE command. In the walrus.ai test suite, this occurs in the order of milliseconds, and is a worthwhile performance tradeoff for keeping our tests simple.

Conclusion

By clearing the database between tests, and using one test per worker, we can continue getting the delightful Jest experience for testing database connected backend applications.

While testing database interactions with Jest helps increase unit and integration test coverage without sacrificing test stability, running true end-to-end tests with browser automation tools like Selenium or Cypress can still be flaky and unstable.

Jest's goal is to make unit and integration testing 'delightful' — our goal at walrus.ai is to do the same for end-to-end testing. Engineering teams can write tests in plain english, and we take care of automating the tests, resolving flakes, and maintaining tests as their applications change. Our contrived example above showed how to test the database end of a username update, here's how simple the corresponding end-to-end test with walrus.ai could be:

walrus -u your-application.com -i \
  'Login' \
  'Change your username' \
  'Verify you receive a confirmation email at the new email address' \
  'Verify your username is changed'

Why engineers suck at writing integration tests

Scott White — Tue, 21 Apr 2020 18:57:58 +0000

In a perfect world, we can all agree that having automated tests are a good thing. They improve reliability, prevent bugs from making it to production, and reduce the load on manual QA, in addition to a whole host of other less obvious, but crucial benefits.

But we don't live in a perfect world.

The real downside to automated tests is writing them in the first place. While important for any CI/CD development environment, writing and maintaining tests is the least favorite part of a developer's job. As we all know, if we don't like to do something, we probably won't do it well.

In short, engineers suck at writing integration tests because writing integration tests sucks

What's so bad about writing integration tests?

✏ Writing integration tests is too hard

Writing a test is fundamentally expressing a user story. You want to make sure that your customer can take a set of actions in a particular order, and your application responds the way you would expect based on those actions. For example, if you're Amazon, you want to make sure that your customer can add an item to the cart, and the item actually loads in the cart when it's opened.

While easy to express in English, writing a test for that story in something like Selenium requires making complicated assertions on every step of the story (make sure a button loads containing some text, make sure it's clickable, make sure a page loads, that an image of the item loads, etc.).

Furthermore, whatever framework you use to test is probably not the framework you use to do most of your job, which means every time you go to write tests, you have to brush off the test-writing skills before you can really get productive.

🛠 Maintaining them is too costly

Test maintenance comes in many forms:

The cost of updating tests every time you make changes to your application

Writing tests is important when you're making changes to your application. At the same time, when you make changes to your application, you likely need to update your tests if any of the changes conflict with the assertions you're making inside of those tests. This circularity means that every time you're making big changes to your site, not only do you need to write new tests for that functionality, you probably also need to update all of the tests you've already written so they don't break.

The cost of investigating and fixing flaky tests

A flaky test fails sometimes and passes other times, despite no changes to the user story it is testing. Every time one of these flakes occurs, you need to investigate whether it is a true failure of the user story, or a false failure of the user story. The more tests you add, the more investigation you need to do when your test suite flakes.

The cost of held-up deploys

When tests fail, even falsely in the event of flakes, CI/CD pipelines will fail, which means your important changes won't make it to production. The whole purpose of CI/CD is to speed up the process of making continuous improvements, but flaky or brittle tests defeat that purpose entirely.

Every test that you write doesn't only carry the cost of authoring it; it also carries the cost of all future maintenance you'll be responsible for related to that test. As such, developers tend to view tests as a burden, and who wants to voluntarily take on a burden?

❓ Results aren't trustworthy

The whole point of writing tests is to verify that any changes you make to your application don't break the user experience. If you have to investigate every failure due to the flaky nature of most testing frameworks, it defeats the whole purpose of testing (to improve reliability, prevent bugs, and move faster). This untrustworthiness makes writing tests feel pointless, and nobody wants to do pointless work.

✅ Full coverage is difficult to achieve

Browser testing technologies have not kept up with the increasing complexity of web applications. Tools like Selenium or Cypress don't have easy ways to handle user experiences like third-party integrations, file uploads or downloads, interacting with maps, or sending/receiving emails.

If those types of flows are important to your product, then writing tests for the functionality around them will feel pointless, because it only represents a partial solution, and won't be testing the true user experience.

Nobody wants to build a partial solution. When faced with a partial solution, it's easier to build nothing at all.

What needs to change?

Getting engineers to write better tests means solving the problems that make them bad at writing tests in the first place. So to get engineers to write better tests:

We need to make tests easier to write
We need to make tests easier to maintain by eliminating flakiness
We need to eliminate false negatives and false positives from our test results
We need to make it possible to test the full functionality of our applications

Luckily, walrus.ai is here to help!

End-to-end tests can be written in plain English in minutes, without handling any mapping on your end
Tests are maintained entirely by walrus.ai, so you don't need to update tests when you change your application
Results are verified by walrus.ai, so you only get true failures or true passes
Walrus.ai can test the most complicated flows, including third-party integrations, file upload or download, 2fa, or email-based flows.

Writing Resilient End-to-End Tests

Jake Marsh — Mon, 20 Apr 2020 18:38:03 +0000

What is end-to-end testing?

End-to-end testing is the process of testing your application through replicating the expected full flows and behaviors of an end user. For example, if you wanted to test that an application's signup flow was working, you would use code to automatically navigate and complete the signup flow in the browser just like a user.

By testing the full functionality of your application in this manner, you're ensuring everything is working as expected from the client, to the API, to the database. It eliminates the seams that occur when unit testing each parts of your application in isolation.

If it's possible to gain more confidence with end-to-end tests, why don't people do more of it? Because it's hard. One of the biggest issues is resiliency: what happens if you change the text of a button that your test was looking for? What happens if you hit a network spike and something takes longer than expected? Without handling cases like these, your end-to-end tests are likely to start failing.

Why is resiliency important?

In a perfect world, end-to-end tests should be run any time a change is made to your application. This is because the smallest (perceived) change is always capable of causing unforeseen issues throughout the rest of your application. To achieve this, teams will typically execute their end-to-end tests within their CI/CD pipeline.

If your end-to-end tests are running inside your CI/CD pipeline, that means they're directly intertwined with your deployment process. Any delays or failures will impede (or prevent) the deployment from completing. Change the button text in your signup flow from "Next" to "Continue"? If your tests weren't updated for that change, you're about to hit a deployment failure and lose up to 30 minutes of precious time.

What is a flake, and what causes them?

A "flake" is a (usually) non-deterministic failure caused by something that is either intermittent or unexpected. Due to the nature of end-to-end tests and the many layers involved, a flake can be caused by any number of things. Some of the more common examples are:

Asynchronous delays. If submitting a form on your page suddenly takes 5 seconds instead of 2, the page will not be behaving as your tests may expect. Spinners may stick around, buttons may not re-enable.
Identifier changes. As we've mentioned, changing something like the text of a button could cause failures. That's also true of non-user-facing identifiers such as HTML classes or IDs, which can change at any time due to unrelated engineering updates.
Actual application/UX changes. In the case that your team has actually carried out a large redesign or refactor of your application, your existing tests are very likely to start failing.

Flakes and How to Avoid Them

Asynchronous Delays

Asynchronous delays, often resulting from network requests, are the largest and broadest category of test flakes. This is because asynchronous requests are happening quite often in an average web application, affecting many other parts of the app.

Submitting the login form? Waiting for a response from the API. Trying to view a list of data? Waiting for data to be fetched. Just clicked a button? Waiting for the request to finish before updating the UI. You may even have to wait on something like an email being sent and arriving in your test inbox.

These are all examples of things that most end-to-end tests will be doing often: navigating, filling out forms, clicking elements, waiting for things to happen. And they're all unpredictable!

What can you do?

There are a few things you can do in your end-to-end tests to better handle any form of asynchronous delay. None of these are foolproof, but they should get most of the way to resiliency around these issues.

1. Explicitly wait on network requests when possible. Many testing libraries provide APIs to hook into the loading status of the browser page. This can be done when you're submitting a form that you know will result in a redirect, or when you're clicking a link. In Puppeteer, for example, they provide page.waitForNavigation(). Clicking a button to submit a form, then, might look something like this:

await Promise.all([
   page.waitForNavigation(),
   page.click('button[type="submit"]'),
]);

2. Wait often to allow for arbitrary load times. Even when using a helper like waitForNavigation, there may be additional delay involved post-request. For example, the application may take a second or two to enable the button you'll be clicking next. In instances like these, it can be helpful to have a simple wait helper to wait a small amount of hardcoded time before proceeding:

// Helper
function wait(time) {
  return new Promise((resolve) => setTimeout(resolve, time));
}  

// Usage
await Promise.all([
    page.waitForNavigation(),
    page.click('button[type="submit"]'),
]);

await wait(2000);
await page.click('.some-later-rendered-element');

3. Use polling. Many testing libraries will provide a method (or its easy to write your own), that "polls" for an element on the screen. This makes your test more resilient to those millisecond differences when finding an element to operate on.

// Bad
await page.click('button');

// Good
await (await page.waitForSelector('button')).click();

If you're testing some process that can be particularly long and/or unpredictable such as querying a large set of data for a new entry, or receiving an email, it can be helpful to carry out your logic and assertions in a custom polling method. This means you're checking on some defined interval for the assertion (i.e. "my data entry did appear"), with a hardcoded timeout that determines a "failure".

await pollUntilTimeout(async () => {
   return !!(await page.$('.some-new-element'));
}, {
   intervalMs: 1000,
   timeoutMs: 30000,
   timeoutMessage: 'New element not found.',
});

Identifier Changes

Most identifiers or attributes used in testing exist for other purposes. Classes, for example, are primarily used for styling. Element tags are used for DOM layout and specific to element purpose. Any time one of these identifiers is changed for their real purpose (updating a button's styling, changing a button to a link), your tests can break.

One option is to define and enforce strict rules around naming behaviors and update processes for element identifiers. For example, you could add a new class to every element you'll be using in your test, prefixed with test-, and ensure those are always kept in sync. However, this can be a painful process to remember and enforce.

Ideally, you can write your tests to be better prepared for changes to identifiers.

What can you do?

1. Avoid relying on "internals". Things like class and id should generally be avoided and assumed to be unstable, as they may change at any time due to internal engineering efforts. Of course there may still be scenarios in which you have no other option, but be prepared for changes to affect your tests.

// Bad
await page.$('input.form--input.some-styling.text');

// Good
await page.$("input[type='text'");

2. Use selectors based on actual functionality to the end-user. Since the goal of your end-to-end test is to ensure that an end-user is still able to complete some flow, it makes sense that it should be written in the same way a user may look for something. A user of your application isn't going to be looking for the selector button.btn.btn--green when they want to create a new post, they'll look for a button that contains the words "Create Post".

// Bad
await page.click('button.btn.btn--green');

// Good
await (await page.waitForXPath("//button[contains(., 'Create Post')]"));

3. Be (reasonably) lenient with your matchers. In the example above, we're searching for a button with the text "Create Post". But what if a change is made and it now says "Create New Post"? Barely anything has changed, but your test is going to break. If there's no other conflicts on the page, it can be helpful to loosen your matcher. What if we just search for "Create"?

// Bad
await (await page.waitForXPath("//button[contains(., 'Create Post')]"));

// Good
await (await page.waitForXPath("//button[contains(., 'Create')]"));

Real Application/UX Changes

The last major cause of end-to-end test failures are larger instances of what we've already discussed: intentional changes to your application and/or UX. If your signup flow, for example, is completely redesigned from the ground up, your test logic and matchers will likely no longer be correct.

Although this class of changes is least common, they are by far the most expensive in terms of time to update the applicable end-to-end tests. There's not much you can do in your tests to prepare for large sweeping changes, due to the need to balance non-colliding and specific matchers with the possibility of an entirely different page.

What can you do?

walrus.ai is a new solution for end-to-end testing with ease. All you provide is a user story written in plain English instructions, and the test is then interpreted and executed by a human and automated for all future runs. Any changes or flakes are handled, and so your tests will only fail when there's a true failure.

Three Common Pitfalls of CI/CD, and How to Avoid Them

Scott White — Fri, 22 Nov 2019 22:52:30 +0000

CI/CD is ultimately about getting your ideas to market faster by continuously shipping code in smaller, more testable increments. However, to successfully adopt this process, it requires the coordinated buy-in of the entire engineering and testing teams, the adoption of tools to facilitate the process, and major process changes.

Moving to CI/CD is risky if you don't take the time to think through how to handle the transition.

So what are the common mistake teams make when switching to CI/CD?

Making it too easy to bypass pipelines

CI/CD is kind of like working out — it requires a specific set of habits, and a lot of discipline to stick to them. If you skip too many days at the gym, you fall out of the habit and stop going entirely.

In a similar vein, if you give your team a way to bypass part of your CI/CD pipeline, the whole process won't stick. When your (hopefully automated) testing catches bugs, it'll be too tempting to bypass your checks, convincing yourself you'll fix the issues later.

Only in the most extraordinary circumstances should you give anyone a way to bypass pipelines.

Picking the wrong metrics to track, or not tracking metrics at all

Ideally, CI/CD should help you get your ideas to market faster, reduce the risk of any given deployment, ship fewer bugs to production, and ultimately make your team happier because they're shipping more code.

If those are your goals, the metrics you track should align with those objectives. Aggregate lines of code doesn't tell you anything about bugs, the customer experience, or velocity.

Your metrics should measure velocity and highlight chokepoints in your processes

Velocity requires every step of the continuous delivery process to happen quickly, with minimal transition friction. Track how long the transition takes between each step in the process, and the main causes for slow transitions.

How long does it take for your team to discover and review pull requests
Is your unit and integration test coverage high?
How often are your tests failing?

Your metrics should track the customer experience

CI/CD requires moving quickly and confidently. Automated tests help in terms of providing a layer of protection against a broken customer experience, but rarely can they catch everything.

Identify key business metrics that reflect the true state of the customer experience.
Make dashboards of key metrics publicly viewable, so if a deploy goes out that breaks the customer experience, you can catch it quickly.

Building CD on top of unstable CI

Automated tests are essential to any successful CI/CD process. To continuously deploy, requiring human intervention to test for bugs is untenable.

Lack of test coverage

Moving to CI/CD without test coverage defeats the entire purpose of the process. It doesn't matter how often you push to production — if your product has bugs, you are delivering a poor customer experience.

Tests help you spot bugs before your customers do, and culturally prevent you from shipping buggy code.

By no means should you always have 100% test coverage, but you should always ensure that the most important flows for the customer experience, and any experiences governed by business logic are thoroughly tested.

Flaky tests based on poor testing frameworks or standards

If you write tests that need to be changed any time you refactor code, you may end up worse-off than if you never wrote them in the first place. Tests should be abstract enough to track user stories (how will these user progress through the onboarding experience), and not so specific that something as small as change in button text would cause a failure.

Furthermore, if you do not have a strong process for training engineers to implement tests, the process of writing them will be untenably slow, and defeat the purpose of CI/CD.

Lack of alignment between development, testing, and other stakeholders in the customer experience

In a strong culture of CI/CD, the engineers who write the tests, the testers who bug-bash flows, and the product teams who manage the customer experience should all have a say in which tests are written, and their ongoing maintenance.

Up front, product managers should enumerate the user stories for testing, engineers should implement tests per those stories, and testers should ensure the customer experience behaves as expected, as well as monitor production metrics to ensure no business logic fundamentally fails.

How to build a better culture of testing

An effective implementation of CI/CD requires organizational buy-in, access to right tools, and continuous measurement.

A strong culture should care about implementing tests, and monitoring that the customer experience behaves as expected after every deploy.

Tests should be easy to write, and flexible enough to not break when code is refactored
Tests should be flexible enough to not break when code is refactored
Product teams should be invested in the testing process — enumerating user stories that are important to validate during CI pipelines

And your tools should make all of that easy. At walrus.ai, we're trying to do just that.

Five Ways to Shorten your Continuous Delivery Cycle

Scott White — Fri, 08 Nov 2019 15:25:50 +0000

The age of multi-month release cycles is coming to a close, and the movement of Continuous Integration (CI) and Continuous Delivery (CD) is taking its place.

CI/CD is ultimately about adopting a set of practices and tools to accelerate the deployment of software, from the moment its written, to the time it is live on a site.

The benefits of CI/CD are numerous:

You can get to market with your initial idea or new feature faster
The risk of any given deployment is smaller
Because your team will be able to reliably ship code without manual intervention, they will feel more productive, professionally fulfilled, and ultimately happier

The touch-points in the CI/CD release cycle are numerous, and should be optimized individually to cut down cycle time as much as possible, so your team can deliver software faster, more reliably, and with fewer headaches.

Understanding continuous deployment

At its core, CI/CD is about increasing the speed in which software can be delivered to customers by automating the deployment of code while maintaining confidence in its stability.

Continuous Integration (CI) is the practice of continually integrating code into one shared repository, and ensuring that the code delivers the desired customer experience. Continuous Delivery (CD) governs how that continually integrated code is actually deployed in an automated fashion.

There are four essential ingredients for any successful CI/CD process:

Code is checked in as frequently as possible
Test coverage stays high
Tests are fully automated
Deployment happens without the need for human intervention

Maintaining a culture of CI/CD requires significant investment in process, an organizational commitment to velocity, and tools to facilitate each step in the code review review, CI, and CD stages of delivery.

The five ways to speed up your continuous deployment cycles

To increase the velocity at which you launch software, you should optimize every stage of the delivery cycle, from code review, to building and testing your code, to actually deploying the changeset.

Here are our five tips to shorten your cycles, starting with how you write code, and ending with how you deploy it.

Identify the minimum viable changeset

Keeping pull requests small both increases the speed at which they are reviewed, and reduces the risk associated with downstream failures.

If your pull requests are massive, no one will want to review them. As such, writing smaller PRs decrease the "time-to-first-review" for each PR.
Smaller PRs are easier to review, and so the "time-stuck-in-review" goes down the smaller the changeset.
Large PRs that touch many parts of the codebase carry a larger risk of subsequent merge conflicts, as well as potential test failure.

Reduce time-to-discovery of pull requests

Code sitting in limbo carries a great cost. It slows down velocity, and discourages the writer of the code.

To ensure PRs are reviewed promptly, adopt tools to surface accountability to the team. Pull reminders is a great way to surface PRs that need review, and notify team members in a public channel.
Eliminate the bystander effect by adding fewer reviewers to each PR, so each reviewer feels more accountability for that review.

Automate your testing

Test automation is an essential ingredient to a well-running CI/CD process. Dependencies on manual QA or any sort of human intervention hold up deploys and defeat the purpose of continuous delivery.

However, if not implemented correctly, tests can hold up your deploys just as much as humans. Flakey tests caused by frequent changes to the site can lead to failures.

Ultimately, to maintain continuous deployment, you need high coverage, with flexible automated tests. You can't just have one.

Want better test coverage with one line of code?

Try walrus.ai – integration tests in one line of code, in plain English. We handle the rest.

Adopt a feature flag system

Feature flags are a great mechanism to test changes in production without rolling them out to all of your users at once.

Create dynamic flags that deliver the new experience to people in your specific domain.
Ramp up exposure of your updated experience over time in order to reduce the risk of your changeset even further.

Focus on production monitoring

CI/CD requires moving quickly and confidently. Automated tests help in terms of providing a layer of protection against a broken customer experience, but rarely can they catch everything.

Identify key business metrics that reflect the true state of the customer experience.
Make dashboards of key metrics publicly viewable, so if a deploy goes out that breaks the customer experience, you can catch it quickly.

How to Work Remotely Without Going Insane

Scott White — Fri, 01 Nov 2019 17:28:15 +0000

The future of work is remote.

In recent years, fully-remote companies like GitLab, Zapier, and InVision have proven that a remote culture is not just a perk, but a clear business advantage. Technology is accelerating this transition. We can chat instantaneously in Slack, coordinate a global videoconference through Zoom, and track our entire roadmap in Asana.

For employees, remote work provides more flexibility (hours, family time, no need to move when switching jobs), fewer distractions (no sales calls happening next to your desk), and shorter commutes (or no commute). For companies, a remote culture makes it easier to hire, reduces office costs, and naturally attracts self-motivated workers.

Despite its advantages, remote work can have drawbacks. Between loneliness and isolation, organizational ambiguity, unclear work/life boundaries, and complex coordination, remote work offers several challenges.

None of these challenges are insurmountable. At walrus.ai, we've thought a lot about how we can improve our remote-first culture. By introspecting on the disadvantages of remote work, we can structure our days thoughtfully to mitigate or eliminate these disadvantages at the root-cause.

Remote work is not without challenges

Working remotely can be lonely

Humans are social animals. Research suggests that when unplanned, spontaneous social interactions prevent loneliness. When you work remotely, you miss such interactions that are endemic to the office — chatting at the desks, passing people in the hallways, sitting down for lunch. If you're not careful, it's easy to feel isolated, particularly if you're transitioning into a remote job from a bustling office.

It's easy to feel out of the loop

If documentation at your particular company is poor, it's easy to feel lost as a remote worker. If your goals are ambiguous, remote policies ill-defined, and spontaneous meetings occurring behind closed doors, it's easy to feel like you're left out as a remote employee.

Coordination is challenging

When a team works across multiple time-zones, coordinating real-time meetings is challenging. A bright-and-early meeting in San Francisco might be squarely when you want to tuck in your child in Amsterdam.

Work/Life boundaries can become blurred

If you live, work, and sleep in the same physical space, the lines between these your work time and your personal time can easily fade. Furthermore, the flexibility of remote work can cause your work hours to be different than your social community, increasing feelings of isolation.

How to structure your days to live your best remote life

✅ Define Done

When you are in a traditional office, it's clear that the day is ending when people start to filter out. When you work remotely, you don't have such obvious cues, and it's easy to lose track of when you should stop working.

To counteract this, give yourself a target. When you start working for the day, choose a rough timeframe when you want to finish your work for that day. If something about your day changes significantly that causes you to step away from working (exercise, last-minute appointment, etc.), re-evaluate and adjust your target end-time, and try to stick to that time.

▶️ Identify triggers

When you sit down at your desk in an office, your body knows it's time to work. When you wake up in the same place you'll work for the day, there's no clear start to the work day.

Teach your brain to start working by associating certain actions (or "triggers") with the commencement of work. For example, walk around the block, and when you return, start your work. Or simply stay in your PJs up until you're ready to start working, then change your clothes and immediately kick off your work day.

Choosing a routine or process to start the day will help sharpen the lines between life and work, and will get you into deep focus faster.

📄 Document everything

Successful remote cultures rely on strong documentation. When in doubt, you should document what you are working on, even when it feels like too much. Create publicly-trackable tasks for everything you plan on working on. Block out time on your public calendar so everyone knows what you're up to during the day. Write down your approach for whatever task you're working on in a publicly-viewable document.

Thoughtful and publicly available documentation helps others gain visibility on what you're working on, reduces the need for meetings, and gives you more structure in what you're going to work on in a given day. Furthermore, the more available documentation, the less likely anyone will feel out of the loop with respect to decision-making thought processes or what's being worked on.

👩🏻‍💻 Choose a work area

If behavioral psychology has taught us anything, it is that animals behave predictably to repeated stimuli. Just like a dog might start salivating when you walk to the cabinet in which its food is held, you'll want to take a nap if you climb into bed.

To get into the work mindset, choose an area of the house (or a different place, like a co-working space or a coffee shop) to start your day that you only associate with work.

And for the love of god, don't work in your bed, or you'll want to sleep while you work, and you'll think about work when you're trying to sleep.

❌ Eliminate unhealthy micro-temptations

When we finish a small task, it's in our nature to want to take a break. These breaks can be healthy – they provide us the opportunity to take stock of where we are, recharge, and focus on the next task at hand.

However, there's a limit past which these breaks take us into unproductive territory.

If too many micro-temptations (easily available snacks in the kitchen, the TV playing in the background) are present while you're trying to work, you may find yourself constantly getting distracted whenever you finish a task, no matter how small.

To stay in the zone, keep the snack food farther out of reach, keep the TV off, and you'll get more done, and feel a greater reward when you take a well-deserved break.

👋 Find time for informal communication

Spontaneous interactions, and unstructured time with friends and colleagues make us happier. To avoid feeling lonely while working remotely, proactively set aside time with your team to get to know eachother outside of a work context — coffee chats with folks in your area, social calls, or team offsites are all good ways to bond.

Furthermore, and ironically, working remotely may also force you to be more proactive about scheduling time to spend time with your friends, particularly if you work non-traditional hours.

Lastly, play with your work environment. Go to a coffee shop or a library where other people like you will be working.

📈 Measure results, not hours

When you're in an office, it's natural to measure productivity in terms of "days of work". Remote work requires more self-discipline than simply clocking in and out of an office. At a company level, a culture of measuring results gives every employee more clarity about what he/she should be working on.

At an individual level, setting measurable goals and tracking results is a great way to stay on track with the global objectives of the company without constantly checking in with other people, as well as a good way to determine when to stop working (when you accomplish the specific goal you set out for a day).

Wrapping it up

As more companies adopt tools to enable better synchronous collaboration and asynchronous task management, the shift to remote work will continue to grow.

To ensure this transition is successful, both for the company, and for the individual, specific measures should be taken to address the inherent disadvantages of remote work.

At the end of the day, you can conquer these disadvantages by structuring your days to draw clear lines between work and life, triggering your mind to focus on work, and building in social time to your schedule.

Have any other tips for living your best remote life? Let us know!

Are you a remote engineer?

Remote teams use walrus.ai to automate their integration tests. Try it out free!

The Five Most Common Bugs you Should be Writing Tests for

Scott White — Thu, 31 Oct 2019 17:09:01 +0000

Users have higher expectations than ever before. At the same time, the customer experience is increasingly the top priority because of its far-reaching implications with respect to customer activation and retention.

The easiest way to kill a good customer experience is with a bad bug. That's where tests come in.

Tests are instrumental to delivering a good customer experience – they spot bugs before the customer ever has a chance to.

Some technologies like static typing and linting can help get you part of the way there, but ultimately, you should be writing tests to ensure your business logic and the corresponding customer experience work as you'd expect them to.

At walrus.ai, we help teams automate their most annoying integration tests. We've outlined some of the most frequent bugs that our customers have experienced that you can write simple tests to prevent.

What are the most common bugs you should be writing tests for?

Unresponsive interactive elements

Unresponsive elements are deadly. Often they manifest themselves in onboarding experiences (continue buttons, input forms, etc.), and can prevent the customer from taking a valuable action, such as signing up, or in the instance below, getting a car loan!

Form validation

Forms are where you collect the most important business data from your customers. Whether it's their credit card information, or information that will be used to engage the customer (such as email and phone number, the quality of this data is certainly important to the business, and can have a meaningful impact on the customer experience (for example, when you are scheduling a demo and enter an incorrect phone number!).

Asynchronous data

User actions that have asynchronous dependencies are particularly prone to bugs, and can have significant implications for the business. For example, if you don't evaluate whether someone has previously claimed a promotion code fast enough, your system provides a window for coupon fraud.

Empty data

Empty data errors occur when users expect certain results (like search results, or data represented in a report), but the frontend doesn't render any real results, despite the data existing. These bugs are particularly frustrating for the user, because it isn't even clear that an error is occurring.

Multiple popups or modals

Modals are a great way to engage users — whether it's introducing them to a new feature in your product, or merchandising a discount. But not both at the same time.

Not only are overlapping modals problematic with respect to ADA compliance, they also dilute the call-to-action for the end-user, and generally make your product look untrustworthy.

Writing tests sucks. How can I make it painless?

It's hard to write tests because computers don't think like humans do.

A human thinks about what the user is trying to accomplish in a particular experience (say, checkout), and wants to make sure that the user can successfully accomplish that task. But computers require much more specific instructions — make sure this button is green and contains specific text, for example. Writing tests requires humans to think like computers.

But there are new ways of writing tests!

With walrus.ai, you can write complex integration tests with one line of code, in plain english.

Try it for free!

Why you suck at estimating timelines, and what you can do about it

Scott White — Wed, 30 Oct 2019 17:08:01 +0000

When it was announced, the Sydney opera house had a target completion year of 1963, and a budget of $7 million.

Exactly a decade later (1973), and ~$100 million over budget (14x the original estimate!), the Sydney Opera house opened its doors.

Humans are predictably terrible at estimating timelines. We're optimistic, we disregard our past experiences, and we consistently fail to grasp the intricacies of the tasks we need to accomplish.

This phenomenon has a name –– the planning fallacy.

The reach of the planning fallacy goes well beyond construction projects. It affects the projects we do at work, how we are evaluated, and ultimately whether we successfully accomplish our goals.

The predictability of the planning fallacy is its biggest weakness. It's root causes are well-understood, and by taking a structured approach to timeline planning, you can overcome the planning fallacy.

Why does the planning fallacy happen?

The Planning Fallacy was originally proposed by famed behavioral economics Daniel Kahnmenan and Amos Tversky in 1979, and has been closely studied ever since. Researchers have coalesced on a consistent set of findings.

🌏 Humans think broadly

A simple task like cleaning the kitchen is comprised of multiple smaller tasks: washing the pots and pans, unloading and filling the dishwasher, cleaning the stove, taking out the trash — the list goes on. If you simply think about the one large task, it's easy to assume it will be easy to tackle. Once you enumerate exactly what that task entails, you begin to see how it would take longer than you expect.

🔀 Humans are bad at managing dependencies

Determining the interdependency of complex projects does not come naturally to us. We gravitate to what is easiest for us, not to what will unblock or accelerate the larger project.

🌟 Humans are optimistic

Humans love the happy path. To help us complete the task at hand, we think about the desired end state. The natural consequence of this optimism is two-fold: we think tasks will be easier than they actually are, and we don't plan for failure.

Given that projects contain a great deal of complexity and uncertainty, involve multiple people, and carry a large cost in the case of failure, it's completely understandable that we struggle with estimating how long they will take.

Luckily, there's a way to conquer the planning fallacy.

How to come up with a timeline that conquers the planning fallacy

Once we truly understand what leads to the Planning Fallacy, we can take steps to avoid those root-causes in our planning process.

Below are the steps you can take to ensure your timeline is realistic, and doesn't become a victim of our flawed intuition.

Get granular

As we've discussed, a seemingly simple task can be comprised of several harder-than-expected subtasks.

The smaller the task, the easier it is to estimate.

When determining a project timeline, break up the initiative to its smallest constituent elements, and estimate those instead. Once you have all of them, add them together to get the larger timeline.

Start broad. What are the biggest separable parts of of the project.
Within each of those, identify the user-specific interactions. In other words, put yourself in the shoes of a user to determine what they're seeing that you need to build.
Within each user-story, break down exactly what needs to be done to deliver the experience. In the end, these should ideally be tasks that will be assigned to specific individuals.

Once you have enumerated the subtasks that comprise the entire project, estimate how long each of those subtasks will take.

Then just add them up, and you have the overall number of days a hypothetical individual would take to complete the project.

Determine your time budget

Once you have the amount of time your project will take, it's time to figure out how much time you actually have with your current level of staffing.

To determine your time budget, the first step is identifying who is working on the project, and how much time he/she has allocated to it specifically.

In an ideal world, he/she is working on the project 8 hours a day, 5 days a week. But between onboarding new employees, vacation time, holidays, and other time out of the office, the realistic time budget looks much different than the real time budget.

Once you have your time budget, you can determine who should work on which specific tasks.

But before you can do that, you need to determine the order of the tasks you need to accomplish.

Demystify dependencies

Understanding how the tasks that comprise your project are dependent on each other is a prerequisite to determining the order in which you'll complete those tasks, and therefore who is working on them.

To determine the order of the tasks that you'll need to accomplish, go through the entire list and identify which tasks need to be completed to start work on the other tasks. Go task-by-task and identify if it depends on another task or is a dependency on another task.

Once you understand those relationships, make a graph to determine if there are any choke-points or risky tasks. Once you understand these relationships, you can determine how to tackle them in sequence with the staffing you have available.

HINT: A good tool to graph dependencies of tasks is DiagremmeR, which can be run in R.

Plan with failure in mind

As we've discussed, humans are optimistic, and would prefer to visualize the happy path. Unfortunately, the happy path is often the path less trodden, so it's incumbent upon you to plan with failure in mind.

In your time budget, account for lost time

Few people spend their entire work week on whatever project they are working on. Between ramp-up, vacation time, bugs, and holidays, most engineers are working at less than 75% capacity on a given project. Explicitly account for this lost time in your timeline to provide a more realistic estimate.

Add testing to the project plan

In a perfect world, everything we build would behave as expected.

Unfortunately, we don't live in a perfect world. Building a resilient infrastructure to identify and resolve bugs is a costly but worthwhile investment, and ensures that your product offers a good user experience.

As part of your project, carve out specific time for your developers to write the tests that will ensure your project behaves as you'd expect.

Want to spend less time writing these tests?

Try walrus.ai – an API that lets you write integration tests in plain english, with one line of code. Your team can spend more time writing the code that your customers will see, and less time working on tests. Want to learn more? Get your first test results in five minutes.

Putting it all together

Humans are bad at creating timelines, but at least we're predictably bad. By adopting a process that addresses the root-causes of the Planning Fallacy, we can begin to estimate projects with more accuracy.

In the end, creating a better timeline comes down to a few simple steps:

Identify the tasks that comprise your project to the granularity of what an individual would work on in a given day, and estimate how long each of those tasks will take. Be prepared for the unhappy path, and add tasks like writing integration tests into the plan.
Determine the budget of time you have available per person who is working on your project
Outline the dependencies of the tasks that comprise your project to determine the order you need to complete them
Based on the availability in your time budget, identify who should work on each task

Than you just have to follow the plan.

Planning for failure, but want to move fast?

Spend less time writing tests to ensure your product works as expected.

Walrus.ai lets you write integration tests in plain english, in one line of code.

Try it for free, and get your test results within 5 minutes.