DEV Community: Dennis Martinez

Dead-Simple API Tests With SuperTest, Mocha, and Chai

Dennis Martinez — Tue, 22 Sep 2020 11:00:00 +0000

If you have to create automated tests for an API, you will most likely use or explore using Postman. Postman is possibly the most well-known API development and testing tool out there and with good reason. It's an excellent tool for both developers and testers to create documentation and demonstrate how your application APIs should work.

Using Postman gives you an excellent starting point for building a test suite to check your API works as expected. However, depending on your test cases and API endpoints, you'll likely run into limitations with Postman:

Because Postman does so much, its interface can feel confusing and somewhat bloated if all you want to do is API testing.
Once you start testing more than a handful of API endpoints, it can feel a bit messy to organize your different scenarios for each one.
If you want to use Postman on a continuous integration environment, you'll have to use Newman, the command-line companion to Postman. While both tools should technically work the same, they're still separate tools, and you might stumble with issues where your test results differ.
If you have multiple team members collaborating on API testing and documentation, Postman's pricing can get a bit steep for small organizations, since it's a monthly fee per user.

I recently worked on a project that's using Postman for API documentation and testing. The team began experiencing most of these pain points directly, so we set out to look for a better solution to get the team building automated tests as they continued building the API. The team had no dedicated testers, so the development team was responsible for test automation.

Since the current developers are comfortable with JavaScript, we began looking for JavaScript tools to help with these efforts. After some experimenting, we landed on a lovely combination of tools that made our API testing effortless to build and easy to maintain. After implementing these testing tools, our automation coverage skyrocketed.

The JavaScript tools to run your API tests

The application under test was a Node.js application, so we wanted to find testing tools that worked well in that environment. Thankfully, the Node.js ecosystem has no shortage of excellent tools for all your testing needs. You'll find a library or framework to run everything from basic unit tests to end-to-end tests and everything in between.

With so many choices at our disposal, our focus was to find simple-to-use, battle-tested libraries that have been around for some time. One of the team's desires was to find stable tools that any JavaScript developer could easily pick up. After tinkering around with a few well-known libraries, we found some great libraries that fit the bill.

SuperTest

SuperTest provides a high-level abstraction for testing HTTP requests - perfect for APIs. If you have a Node.js application that runs an HTTP server (like an Express application), you can make requests using SuperTest directly without needing a running server. One of the nice things about SuperTest is that while it can run tests without any additional tools, it can integrate nicely with other testing frameworks, as you'll see next.

Mocha

One of the better-known JavaScript testing frameworks, Mocha runs on both Node.js and the browser, making it useful for testing asynchronous functionality. One of the cool things about Mocha is it allows you to write your tests in different styles like BDD (it, describe, etc.) and TDD (suite, test, etc.). Mocha fits in nicely with SuperTest, helping you organize your tests in your team's preferred way.

Chai

Chai is an assertion library that you can pair with other testing frameworks like Mocha. While not strictly necessary for writing a test suite, it provides a more expressive and readable style for your tests. Like Mocha, Chai allows you to choose BDD-style (expect) or TDD-style (assert) assertions so that you can combine the library with most frameworks without any clashes.

Using these three tools, you can create a fast, stable, and maintainable automated test suite for your APIs with little effort.

Putting these tools into play

To demonstrate how SuperTest, Mocha, and Chai work together, we'll use these tools to automate a few tests for an application called Airport Gap. The Airport Gap application provides a RESTful API to help others use it to improve their API automation testing skills.

Keep in mind that the Airport Gap application is not a Node.js application, so this article won't show how you can use these testing tools to integrate directly with Node.js. However, you can still use them to build tests for any accessible API. This article will create the tests in a separate code repository, but if you have a Node.js application, these tools will work best with your test code alongside the app.

First, create a new project inside an empty directory and initialize it by running npm init -y to create a default package.json file. For now, you don't have to edit this file. With the project initialized, you can set up the latest versions of SuperTest, Mocha, and Chai libraries with the following command:

npm install --save supertest mocha chai

That's all you need to get started with creating automated tests for your API. Let's start by creating your first API test for the Airport Gap application.

The Airport Gap documentation shows all available endpoints you can use for your tests. Let's start with the endpoint that returns all available airports, GET /airports. This endpoint returns a paginated list of 30 airports at a time, so a quick way to verify that this works is to create a test that calls the endpoint and returns a list of 30 results.

Create a new file inside the project directory called airports.test.js, which you'll use to write your test code. You can name this test file anything you prefer, but including .test.js as part of the filename makes it easier to execute the tests as the test suite expands. In the new file, let's write our first API test. Here's the code, and we'll explain what's going on after:

const request = require("supertest")("https://airportgap.dev-tester.com/api");
const expect = require("chai").expect;

describe("GET /airports", function () {
  it("returns all airports, limited to 30 per page", async function () {
    const response = await request.get("/airports");

    expect(response.status).to.eql(200);
    expect(response.body.data.length).to.eql(30);
  });
});

If you're familiar with JavaScript, this test will be readable even if you've never used any of the libraries in the project.

First, the test loads the SuperTest library and assigns it to the variable request. The SuperTest library returns a function which you can use to initialize an HTTP server. You can also pass a string with the URL of the host you want to use if you're not working directly with a Node.js application, which is what we're doing in this article.

Notice that the specified host is the API's base URL, including the /api subdirectory. Using the base URL allows you to make requests to your API endpoints without needing to write the entire URL every time, as you'll see later when we use SuperTest inside our test scenario.

The next library loaded comes from Chai. Since Chai allows you to use both TDD and BDD assertion styles, you need to specify which one you want to use. For these examples, we're going with the BDD style, using the expect interface. If you prefer the should BDD interface or assert with the TDD style, you can easily switch using Chai. It's one reason why we chose the library since it accommodates different tastes for any team.

After loading the required libraries, now you'll get into the heart of your test scenarios. Following the BDD style, the test uses Mocha's describe interface to group your test scenarios. The describe function accepts a string as a description of the tests and a function to define your test cases. Like Chai, you can use the TDD interface instead if that's your preference. You don't have to load any Mocha libraries, since we'll use Mocha's runner to execute the tests.

The it function from Mocha is the place to define a single test scenario. Like the describe function, the first argument is a string to describe the test case, and the second argument is a function to write the code for your test steps. Notice that we're using an asynchronous function as the second argument for it. Inside the test, you'll make requests using SuperTest, which returns a promise. Using async and await allows you to resolve the promise to get the API response more cleanly instead of resolving the promise through chaining.

The test scenario has two steps. First, you use SuperTest's request function to call the API using the get function. This function requires at least one parameter - the URL for your request. Since we initialized the request function with our base URL for the Airport Gap API, it's unnecessary to write the entire URL when making requests. All you need is the endpoint, and SuperTest automatically appends it to your base URL.

As mentioned, the get function returns a promise, so to resolve it cleanly, you can use the await keyword. SuperTest makes a request to your host and endpoint, and saves the response in the response variable, which you'll use to run the test's assertions. SuperTest fetches lots of information from the API request, like the body, headers, status codes, and much more.

With the API response in hand, you can finally make your assertions to ensure that the API works as expected. The test uses Chai with the expect function and the to and eql chained methods to construct your assertion. Chai has tons of methods for building assertions, and it's worthwhile to read which ones are available to help you create your tests as needed.

This test contains two assertions. First, the test verifies if the API request's status code was 200 - meaning that the request was successful - using response.status. The next assertion looks at the response body (response.body) and checks if the data key contains 30 items. SuperTest is smart enough to check the content type from the response and appropriately parses the information into a JavaScript object. It makes verifying your JSON APIs much easier to do since you don't have to worry about parsing the response.

The test is all set up and ready to execute. To run your tests using Mocha, you can use the mocha executable included when installing the package. The easiest way to use it is with the npx command, which will find the executable inside your project. Open your terminal and run the following command:

npx mocha airports.test.js

If everything works as expected, Mocha will execute your tests and show your test results. The default reporter shows the description of your tests, grouped by the describe method, and displays the results and the execution time for each test.

You've written and executed your first API test using SuperTest, Mocha, and Chai! In less than ten lines of code (not counting blank lines), you already have an automated test to verify an API request that you can re-run at any time. It can't get any simpler than that.

Running POST request tests

Let's write another test, this time checking how a POST request to the API works. The POST /airports/distance endpointallows you to send two airport codes, and it returns the distance between them in different units of length. Let's see how SuperTest handles the request. Under the existing test in airports.test.js, create a new test case:

describe("POST /airports/distance", function () {
  it("calculates the distance between two airports", async function () {
    const response = await request
      .post("/airports/distance")
      .send({ from: "KIX", to: "SFO" });

    expect(response.status).to.eql(200);

    const attributes = response.body.data.attributes;
    expect(attributes).to.include.keys("kilometers", "miles", "nautical_miles");
    expect(attributes.kilometers).to.eql(8692.066508240026);
    expect(attributes.miles).to.eql(5397.239853492001);
    expect(attributes.nautical_miles).to.eql(4690.070954910584);
  });
});

This test is a bit longer than the first example, but most of the way we use the SuperTest, Mocha, and Chai libraries are similar, so we won't go into detail for this test. Let's focus on the parts that did change.

The first difference is the way you need to make the request to the API. Since this endpoint is a POST request, you'll use the post function. The function works the same as get, and you only need to specify the endpoint for the API. However, you can chain the send function to your request to submit any required parameters. Since we're testing a JSON API, you can use a regular JavaScript object with your parameters, and SuperTest sends the correct request body.

Another difference is one of the assertions made to verify that the API response contains specific keys. Here, we're using Chai's include and keys methods to confirm that the response includes the keys with the calculated distances. You can check the entirety of the API response body, but we're just going to do some spot-checks for purposes of this article. We also perform validations on this test's actual values, but these are also for demonstration purposes. You might not want to run these kinds of assertions if your API data can easily change.

Now that you've seen the changes in these tests, it's time to execute them to make sure everything's working as expected. You can run the tests the same way as before:

npx mocha airports.test.js

Now you have two test scenarios, and if everything is correct, you'll have two successful test results for different API requests and endpoints.

Testing authenticated endpoints

The examples shown so far run tests against public API endpoints. What if you have an API that requires authentication? The Airport Gap API has some endpoints that require authentication. These protected endpoints require an API token passed as a header. For instance, one endpoint that requires authentication is the POST /favorites API endpoint. This endpoint allows an Airport Gap user to save their favorite airports to their account to look up later.

Let's begin creating a few tests to validate this behavior. First, we'll cover the test case to verify that the POST /favorites endpoint doesn't allow access without a token. After verifying that the Airport Gap API won't allow access, we'll write a test that accesses the same endpoint, this time with an authentication token.

To keep the test suite organized, create a new file in the project directory called favorites.test.js. Inside this new file, let's first write the test scenario to ensure that an unauthenticated user can't access this endpoint:

const request = require("supertest")("https://airportgap.dev-tester.com/api");
const expect = require("chai").expect;

describe("POST /favorites", function () {
  it("requires authentication", async function () {
    const response = await request.post("/favorites").send({
      airport_id: "JFK",
      note: "My usual layover when visiting family",
    });

    expect(response.status).to.eql(401);
  });
});

By now, the test structure should be familiar to you. We're loading up the SuperTest and Chai libraries, create a new test group and set up a test scenario to validate that the endpoint requires authentication. The POST /favorites endpoint requires the airport_id parameter and also accepts an optional note parameter, both of which we'll use in our request. When making a request to a protected endpoint in the Airport Gap API without a valid token, the API returns a 401 response, which is what we're checking here.

Run this new test scenario to make sure it's working as expected:

npx mocha favorites.test.js

You'll see the now-familiar results for this test case:

Now that you verified how the API works without authentication, let's go through a longer flow to make similar requests with an authentication token. If you have an Airport Gap account, you can find your API token in your account page and use it directly in your tests either by setting it in the code or through an environment variable. We'll use an environment variable to keep sensitive keys out of the codebase.

The next example follows an end-to-end flow that uses multiple authenticated API endpoints. The test starts by creating a new favorite airport in the user's account. Then, it updates the newly-created record through an API request and validates the data returned. Finally, the test will delete the record, and we'll validate that it's not found anymore.

In the favorites.test.js file, add your new test case under the existing scenario:

it("allows an user to save and delete their favorite airports", async function () {
  // Check that a user can create a favorite.
  const postResponse = await request
    .post("/favorites")
    .set("Authorization", `Bearer token=${process.env.AIRPORT_GAP_TOKEN}`)
    .send({
      airport_id: "JFK",
      note: "My usual layover when visiting family",
    });

  expect(postResponse.status).to.eql(201);
  expect(postResponse.body.data.attributes.airport.name).to.eql("John F Kennedy International Airport");
  expect(postResponse.body.data.attributes.note).to.eql("My usual layover when visiting family");

  const favoriteId = postResponse.body.data.id;

  // Check that a user can update the note of the created favorite.
  const putResponse = await request
    .put(`/favorites/${favoriteId}`)
    .set("Authorization", `Bearer token=${process.env.AIRPORT_GAP_TOKEN}`)
    .send({
      note: "My usual layover when visiting family and friends",
    });

  expect(putResponse.status).to.eql(200);
  expect(putResponse.body.data.attributes.note).to.eql("My usual layover when visiting family and friends");

  // Check that a user can delete the created favorite.
  const deleteResponse = await request
    .delete(`/favorites/${favoriteId}`)
    .set("Authorization", `Bearer token=${process.env.AIRPORT_GAP_TOKEN}`);

  expect(deleteResponse.status).to.eql(204);

  // Verify that the record was deleted.
  const getResponse = await request
    .get(`/favorites/${favoriteId}`)
    .set("Authorization", `Bearer token=${process.env.AIRPORT_GAP_TOKEN}`);

  expect(getResponse.status).to.eql(404);
});

The API requests made using the request function all look the same, except for a new portion we haven't used previously. To send the authentication token as part of the request as a header, you can chain the set function to your request. This function uses two parameters. The first parameter is the name of the request header, and the second parameter is the value you want to send to the server for that header. For the Airport Gap API, it expects to find the Authorization header with the value of Bearer token=<token>.

After setting up this end-to-end test, let's execute it and see how it goes.

This test can give you a clearer idea of how potent SuperTest is for making HTTP requests of all kinds. Here, you see the different HTTP methods you can use, and how chaining different methods like send and set allows you to pass along all the data your API requests need. This test can be improved in a few ways, like cleaning up the account favorites if there's an error in the middle of the execution, but we'll leave it as an exercise to the reader.

Cleaning things up

Although the current tests are relatively short and straightforward, you can begin to take some steps in organizing and cleaning up the test suite now. It's a good practice to try to get some organization going in your codebase before it spirals out of control.

The first thing you might have noticed is that we have some duplication creeping in. We have two separate files with the same setup to load the libraries. For these basic examples, it's not a big deal. But imagine you continue expanding this test suite and have a few more files. If you have to change the setup, like using a different base URL for the API, you'll have to go into each one and adjust it manually. It'll be nice to have it in one place.

You can begin organizing your test setup with a configuration file that you can place in the root of your project directory. The configuration file can export some of the common functionality used throughout your test suite, which you can include where needed. That way, you can keep some of your setup and configuration in a single place.

To do this, start by creating a new file called config.js inside of your project directory. Inside this file, you can move the common setup used in each test suite and export these functions:

const request = require("supertest")("https://airportgap.dev-tester.com/api");
const expect = require("chai").expect;

module.exports = {
  request,
  expect,
};

Now you can replace the setup at the beginning of both test files with this configuration:

const { request, expect } = require("./config");

Your tests should work the same with this change, and your test suite becomes more maintainable by having the basic setup consolidated in a single place. If you need to set up additional libraries or configure the existing functions differently, you only need to do them once in the configuration file.

One last thing to do is to make it easier to execute your tests from the command line. Typing npx mocha *.test.js is simple to do, but we can make it easier by adding a quick command to execute your tests. Open the package.json file and find the scripts key. By default, it includes a test command, which doesn't do anything. Replace the value of the test key with your Mocha command (the npx command is no longer necessary):

{
  "scripts": {
    "test": "mocha *.test.js"
  },
  // The rest of your configuration remains the same.
}

With this change, all you need to execute your tests is to run the npm test command. While this change doesn't save a ton of time now, it helps in other ways. Most JavaScript projects use the npm test command as a standard way to execute tests regardless of the testing tools used, so anyone joining your team can get up and running quickly. Another benefit is that it keeps your test command the same if you have to include additional command-line flags in the future.

If you want to check out the source code for the project shown in this article, it's available on GitHub: https://github.com/dennmart/dead_simple_api_testing.

Summary

With lots of web applications relying on APIs to work, you need to make sure these systems are always working as expected. A fast and stable automated test suite will ensure that the APIs that power the essential parts of your business continue to work as they grow and expand.

If your team uses JavaScript - which is likely since it's one of the most-used programming languages nowadays - you won't have to step away from your existing toolset and environment to create your tests. You can find plenty of testing frameworks and libraries to build your test automation for your APIs.

In this article, you saw the combination of three tools to allow you to build a robust automated test suite for APIs quickly. SuperTest enables you to make any HTTP requests with ease. The Mocha testing framework organizes and runs your tests in the way your team prefers, whether its TDD or BDD style. Chai's assertions fit in nicely with Mocha to validate your API responses. All three together combine to create a maintainable and speedy test suite.

These aren't the only tools you can use, though. As mentioned in this article, you have plenty of options to choose from if you want to build your test automation around JavaScript. If you don't like Mocha, you have similar frameworks like Jest or Jasmine. If Chai isn't your cup of tea (pun intended), other assertion libraries like should.js or unexpected work equally well.

API testing doesn't have to be complicated. After all, the only thing APIs do is receive a request and send back a response. With a few tools in place, you can create a simple yet powerful test suite to make sure your APIs are as reliable as possible to keep your applications running smoothly.

How do you and your team handle API testing? What issues or pain points have you ran into? Let me know by leaving your comments below!

How Can You Tell If Your Automated Tests Are Any Good?

Dennis Martinez — Tue, 15 Sep 2020 11:00:00 +0000

While browsing the Ministry of Testing forums the other day, I stumbled upon a thread that caught my attention. The thread's title was, "How do you tell how good an Automation implementation is?" As someone interested in checking out how others handle their test automation, I was interested in seeing what others had to say about this topic.

The thread did not disappoint in terms of the quantity and quality of replies. Lots of experienced testers from all over chimed in and gave their thoughts about this question. Many talked about their personal experiences with other organizations when working on automation. Some offered hands-on practical advice, while others provided a more theoretical point of view. The thread contained a nice mix of useful feedback, and we can learn a few guiding principles for our implementations.

Most of the responses were great, and I encourage everyone to read through the thread. It'll likely get you thinking. As I read through every answer, I began noticing that many of the replies shared some common themes between the different testers who took the time to share their thoughts. It feels like there's a shared agreement in the test automation world on what makes an automation implementation effective.

The forum thread reminded me a lot about my personal experiences, and I saw a lot of my own thoughts scattered throughout the words written by other testers. Here are some of the main takeaways I got out of the entire discussion, and where many ideas and feelings overlapped.

It's hard to tell what good implementation is, but it's easy to spot a bad one

As evidenced by the question from the forum thread's original poster, it's tough to know when your automated test implementation is any good. You might have some thoughts about what makes any testing "good", but it's not something you can quantify with hard evidence.

Although it's difficult to recognize a good test implementation, it's super-easy to spot a bad one. You can see it coming from a mile away. We all know the signs of a bad setup. It might be that the tests are so flaky you never know when they'll pass or fail, or the tests leave considerable gaps in coverage. Slow tests that take forever to execute are also a sign.

You can come up with plenty of reasons why a test suite isn't all that great. However, many people use these reasons to gauge their implementation incorrectly. You might think that if your automation efforts don't have any "bad" signs, then it must be good. Not so fast.

Even so-called "good" signs of test automation are unreliable indicators for the health of your implementation. For instance, if you have a stable test suite that doesn't fail, it doesn't mean you're testing the right things. You might have tests that don't fail, but they're not catching regressions either. Another example is when you have tons of coverage for the application under test. Lots of coverage might mean that the team cared more about metrics instead of writing an efficient and effective test suite.

It's simple to fix the issues that slow down your progress; it's also simple to get misled by what looks good on the surface. Handle the apparent signs of bad automation implementation, but don't ignore the good parts either.

Code matters just as much as what you're testing

The vast majority of testing teams often care only about the end result of their work. Typically, the attention lands on whether the tests pass, [and if the tests help the rest of the team catch or prevent issues that break the application.

Of course, we need to show that our efforts pay off for everyone involved in the product. If your tests don't help with the product's quality, they're practically useless. Unfortunately, paying attention only to what's visible can lead to you neglecting what's driving these results - your actual test code.

No matter how diligent you are in caring for your codebase, every team eventually reaches a point where things don't perform as well as they used to. Everyone has to tend to their past work for refactoring or even deleting code that has outlived its usefulness.

Having a fast and stable test suite is excellent, but you also need to ensure that you can keep those tests running in optimal condition for the long haul. The time you spend maintaining tests is an essential factor for a solid test automation implementation. If you have to spend hours or days wrestling with your codebase to modify anything, your implementation will stagnate and eventually stop being useful.

Every organization and team has different time and budget constraints for what they can do with their testing efforts. However, making sure your codebase allows the team to build and grow the test suite rapidly and with few issues will pay off tenfold in increased quality.

"Good" is a team effort

The thread on the Ministry of Testing forum has plenty of excellent comments and suggestions about distinguishing a good test automation implementation from a bad one. It has lots of different strategies and points of view, which are great to learn from and use for your work.

After reading through the entire thread, I noticed a common thread uniting every response. Although most of the answers offered something specific from the person responding to the thread, my main takeaway from the discussion is that everyone has their particular version of what a good test automation implementation is.

Every person and every team will have their unique definition of what's considered good and what isn't, and it will vary greatly depending on who you ask. If you ask this question to ten different testers, you'll get at least eight different responses. It wouldn't surprise me if you had ten entirely different answers.

Everyone's circumstance is unique, so it's not uncommon to have different priorities based on the information we have in our hands at any given time. What I consider a good implementation of a test suite (like clean and maintainable code) might register a low-priority item for you and your team. It might not even be on the "good list" for my own team or a particular project we're working on.

If you're spending too much time pondering if your test automation is any good, you shouldn't make this decision on your own. Bring this question up to the rest of your team and see what discussion brings to the table. One of the posters of the forum thread put it best:

"The only real indication that I have is if the team is satisfied with the automation, then it's probably good. It's at least good enough."

Summary

If you work on implementing test automation for your company, chances are you're wondering if what you're doing is right. As shown by the question posed in the Ministry of Testing forum, you're not alone. It's not always a negative thing, either - It's great to think of ways to improve your work.

Everyone has their thoughts and opinions about this question, but you can pull out a few themes from the responses to help guide your decisions.

One of the first things to realize is that it's almost impossible to know what a good test automation implementation looks like. It's easy to see a lousy test suite: slow tests that often fail, tests that don't cover anything useful, etc. But don't let that fool you - what looks good on the surface might cover potential issues that don't serve the team.

Something you can check to determine the quality of your implementation is if you built the underlying codebase for the long haul. Code maintainability and simplicity go a long way in a good test suite. It can be the difference between a long-lasting stable test suite and one that disappears because no one wants to touch the code.

Finally, remember that figuring out this process isn't an individual exercise. Everyone has their definition of "good", and it can differ by team, person, or project. It's best to take the opinions of those around you and your current circumstances and mix them into your definition of "good" for where you are at any given time.

It's okay to check what other testers are doing, and use their experiences to mold yours. But in the end, it all lies with you and your team. If it's good enough for you, that's all that matters.

What ways do you and your team use to determine if your test automation is good? Let me know in the comments section below!

Avoid These 3 Mistakes With Your End-To-End Tests

Dennis Martinez — Tue, 08 Sep 2020 11:00:00 +0000

I've recently been thinking a lot about all of the projects I've worked on throughout my career as a developer focused on testing. I've been working on web applications for over 15 years, and while technology has moved forward since the days when I began, a lot of the underlying architecture for most of the work I've done has been the same.

When you work on the same tech stack in different projects, you'll tend to find similar libraries and frameworks in use. For example, you'll often find RSpec in use for Ruby on Rails applications, or Jest for JavaScript applications. Some teams follow standard practices with these tools, and anyone familiar with them can jump in quickly.

This behavior also extends to teams that are either new to test automation or don't dedicate too many resources to maintaining their existing test suite. These teams often make mistakes with their automated testing that show up again and again throughout different organizations.

This article goes through three of the most common mistakes I've seen across multiple teams when building and maintaining an end-to-end test suite for their application.

1) Testers write too many tests

I don't know about you, but the first time I discovered automating end-to-end tests, I felt like I unlocked a magical skill that would save me an incredible amount of time throughout the workweek. "You mean I can write some code and it'll do my testing work for me? Sign me up!"

Whenever someone learns the wonders of test automation, their first inclination often is to test absolutely everything that they can. It's a noble goal, but it's also one that leads to wasted time and tons of frustration down the road. If you add long-running end-to-end tests to the mix, that problem can quickly spiral out of control before you know it.

It's great to write end-to-end tests to verify that a particular workflow behaves as expected from start to finish. The problem happens when you want to verify all of your workflows. These tests are often slow, and if you write too many of them, it'll slow the entire team down to a crawl. Eventually, you'll end up with a test suite that no one wants to run because it takes forever to complete.

Reliability and maintainability are also essential factors to a healthy codebase. The more end-to-end tests you have, the higher the risk of your test code becoming brittle and unmaintainable. Since these tests cover lots of ground, it's not unusual for them to break due to any minor change somewhere in the system. That means you'll have to update your tests more often. In the end, you'll have to spend more time making sure your test suite is working and easy to update.

The solution to minimizing these risks is eliminating tests that provide little benefit while keeping your test suite lean. Most teams want to automate as much as possible, but end-to-end testing isn't meant for complete coverage of your application. Focus on writing test cases for what matters the most. Using end-to-end tests to automate your app's most critical sections allows you to reap the most benefit out of your time by preventing slow, unstable, and messy test suites.

2) Testers won't ask other team members for help

Failure to communicate between teams is one of the leading causes that stop projects dead in their tracks. You can't expect a team to put in quality work when no one talks to each other. Unfortunately, it seems to happen quite frequently with testers. This problem isn't one-sided, though. Either the testing team doesn't talk with others outside of the group, or testers don't receive any information from the rest of the organization.

Any good testing team knows that quality is a whole team effort. While a QA team can have the sole responsibility of writing automated end-to-end tests, it doesn't mean they can or should do everything on their own. Testers can't live in a silo. They need outside assistance to perform at their best. Testers have plenty of opportunities to bring in other team members to help with quality across the organization.

The first group you can call in for support are the developers on your team. They'll have all the technical knowledge behind what you're testing and can help make your life easier by making the app more testable. For instance, they can help improve how you can identify page elements on the website for your tests or do reviews to improve your code.

Another group you can tap for help are the folks responsible for DevOps. With end-to-end tests, you'll most likely need separate environments and services for testing purposes. These folks can help set up these systems to make it easier for you to execute your tests without disrupting the work of others. DevOps can also improve the efficiency of your tests by observing how the system under test performs during your test runs and improving any bottlenecks along the way.

It doesn't stop at with the technical support either. Non-technical members of the team have invaluable information that can guide your testing efforts in different ways. Product managers have better insight into the application's current usage, which can help you shape your test plan. Customer support can shed insight on areas where real-world users have often stumbled upon bugs. These are just a few examples you can find throughout any company.

You can't do everything by yourself, and you shouldn't be expected to, either. Don't isolate yourself or be shy, and ask around for help. It'll ensure your end-to-end tests are both useful and exactly what you need to increase the quality of everyone's work.

3) Testers don't fully understand how the application under test works

As testers, it's easy to place our focus on testing alone. After all, it's the work that you're responsible for completing. No one expects you to start doing software development or answer customer support requests along with your work. However, having a one-track mind when it comes to quality often leads to subpar results because you're missing the details that help build a stable test foundation.

This issue often manifests itself in the choice of tools you and your team make at the beginning of your work on the test suite. Without having a firm grasp on how the application under test does its magic, you're at risk of choosing the wrong tools for the job. You can probably get your tests working with an inadequate tool, but chances are it will hurt the test suite's stability and long-term maintainability.

For instance, using a heavy, full-featured end-to-end testing framework might be excessive for an app with few interconnecting pieces, leaving you with a slow and bloated test suite. Likewise, using a simple integration testing tool for an app with a complex backend architecture won't provide the coverage you need, missing significant gaps that an end-to-end framework can handle with ease.

I recently covered this topic in more detail, which will help you avoid the pitfalls of choosing the wrong tools when starting with test automation.

Not understanding how the application under test works also ties into the previous point about talking with other team members. You can waste tons of time trying to automate some tests because you're unaware of different services or potential technical limitations that can hinder your testing efforts.

For example, an application can have an asynchronous job processing system running in the background to process certain information. If you didn't talk with the development team, you might have been unaware of how the system processes the data. You may have spent countless hours wondering why your automated tests for that section didn't work properly without knowing this.

You don't need to be an expert on the complete technology stack that runs the applications you're testing. But having some knowledge around how the system works will improve your testing efforts. You'll know which tools are more appropriate, and you'll learn how to tackle different areas according to how the system works.

Summary

When you work on different end-to-end test automation projects, you might spot a couple of mistakes that happen more than once. These issues most often pop up with teams that are just starting with test automation or treat it as an afterthought.

The first issue is when testers want to automate all the things and write too many tests. Knowing the power of automation can become addictive. It's easy to go overboard with this power and try to cover every nook and cranny in your application. While it often comes from good intentions, it leads to buggy and sluggish test suites.

Another issue that pops up is testers isolating themselves from the rest of the team. They don't take time to get insight from others, missing out on beneficial details that can shape their test plans. Everyone working on the same project, from developers to system administrators to project managers, has insight that you won't get in your day-to-day assignments.

One of the most prevalent problems happens when testers focus too much on automation and too little on the system that they're testing. They know how the application works on a functional level, but they're unaware of what lies beneath the surface. Because of this lack of knowledge, testers waste time using inefficient tools and running into roadblocks that others could have cleared.

Thankfully, solving these issues doesn't take a lot of effort if you catch them early. Keep your end-to-end tests lean and focused on handling what's essential and leave the rest to other forms of testing. Ask for help from the rest of the team and use their feedback to improve what you test and how you're testing. Finally, take the time to understand the applications you're working with - a little time spent up-front will save you lots of time down the road.

Even when using the same libraries and frameworks, every team works in their way. You'll encounter different problems from time to time. As long as you become aware of the obstacles ahead and can work towards resolving the issues, you'll continue boosting your skills and continue on your path of being the best tester you can be.

Have you encountered one or more of these common mistakes in your testing efforts? Are there other mistakes you've seen more than once in your testing work? Leave a comment below and share your experiences!

Continuous Testing with Travis CI and LambdaTest

Dennis Martinez — Tue, 01 Sep 2020 09:10:00 +0000

When you build any software program, one of your goals is to have as many people using it without any issues. If your app has too many problems that impede the vital functionality you want for your purposes, no one's going to stick around. That's why testing and test automation are essential parts of a product's lifecycle.

With web applications, testing can get complicated quickly. Often, it starts with a few manual tests in a specific environment. It allows your team to get a head-start when you decide to move to automated tests. Once you have some automation going on, your testing efforts expand to cover the most-used platforms. Test automation allows you to validate your functionality across different environments with ease.

However, even if you cover the main browsers and devices in use these days, this amount of testing isn't enough. Once your app is out there in the real world, you'll find a seemingly infinite number of combinations of platforms. You not only have to manage different versions of the web browsers you test your app against, but you'll also have to deal with them in varying combinations of devices and operating systems.

Most organizations use virtualization to spin up the environments that they need for testing purposes. These days it's dead-simple to run virtualized systems. However, you'll need to maintain these systems, and it's often a pain to manage when you scale up. Also, virtualization is not a substitute for real hardware. You might run into scenarios where you'll need the real thing to replicate a problem and fix it.

Instead of wasting time and money handling different environments to test your web application, you can use LambdaTest to address your cross-browser test automation needs.

LambdaTest - The most powerful cross-browser testing tool online

LambdaTest is a cloud-based testing platform that allows teams to test websites or mobile applications in different environments, using both desktop browsers and browsers on mobile devices. With just a few clicks and some minor configuration, you can spin up any operating system and browser combination to have a usable system to run your tests.

You can access any operating system from Windows XP to Windows 10 and all Mac OS X editions since 2011. You can also use any version of most-used web browsers from the past 15 years. Whether you need to ensure that your application runs in older environments or on new systems, LambdaTest can handle it for you.

Here are a few of the things you can do with LambdaTest to boost your test automation efforts:

Perform manual and automated tests on your web application, using both virtualized environments and real hardware devices.
Automatically generate screenshots of your application across multiple systems to perform visual regression testing.
Validate the responsiveness of your web app across multiple iOS and Android devices.
Run your tests on a scalable Selenium Grid system with access to over 2000 desktop and mobile browsers and operating systems.
Test from more than 27 countries to ensure everyone across the globe can reach your application.
Integrate with many third-party applications like Jira, TestRail, and Slack to keep your testing efforts in sync across your organization.

LambdaTest provides lots of other excellent tools to help with your testing organization. The LT Browser aids with responsive testing during development. There's an issue tracker to keep track of what needs fixing inside the service. If you use Google Chrome, a Chrome extension quickly sends screenshots of your app during manual testing.

I've used similar cloud-testing services in the past, but most of them only offer a fraction of the tools that LambdaTest provides. It's a complete package that covers almost all of your testing needs.

LambdaTest in action

In this article, I want to demonstrate how LambdaTest works by running a continuous testing workflow for an existing web application. For this demonstration, I'll use an automated end-to-end test suite built with the TestCafe testing framework. Although LambdaTest's infrastructure is built on Selenium Grid, we can also use other testing frameworks on the platform for test automation, even if they don't use Selenium under the hood.

The rest of the article assumes you have an active LambdaTest account set up. If you don't have an account and want to follow along, you can register for free. LambdaTest has a free tier with limited access that allows you to use all of their services. It's perfect for helping you get started.

Setting up TestCafe and LambdaTest

The test suite we'll use in this article comes from my book, End-to-End Testing with TestCafe. It contains 11 end-to-end tests covering different functionality for the book's companion web application, TeamYap.

Setting up an existing TestCafe test suite to use LambdaTest's automation services is straightforward. The LambdaTest team maintains a TestCafe plugin that allows you to run your TestCafe tests on the LambdaTest platform. To install the plugin, run the following command inside your test project directory:

npm install testcafe-browser-provider-lambdatest

All the configuration for the plugin is handled by setting up environment variables. Using environment variables allows you the flexibility to set up your automated test runs differently, depending on the environment, without the need to modify any files. Most of LambdaTest's settings have sensible defaults, and you probably won't need to change them. You do need to set up two environment variables that are required for the plugin to work:

LT_USERNAME - Your LambdaTest username, automatically generated for you after signing up.
LT_ACCESS_KEY - Your LambdaTest account's secret access key.

Your LambdaTest username and access are available in your account profile settings. Once you have this information, you can set the required environment variables with the following commands in your preferred terminal on Mac OS X and Linux:

export LT_USERNAME=<LambdaTest Username>
export LT_ACCESS_KEY=<LambdaTest Password>

For Windows systems, use the following commands through the Command Prompt or PowerShell:

set LT_USERNAME=<LambdaTest Username>
set LT_ACCESS_KEY=<LambdaTest Password>

To verify that your LambdaTest credentials are set up correctly, you can check the list of available browsers and operating systems provided by the LambdaTest service through TestCafe using the following command:

testcafe --list-browsers lambdatest

If your credentials are correct, this command prints a long list of browser aliases that you can use to run your TestCafe tests on LambdaTest. The browser aliases are namespaced with lambdatest, and contain the combination of the browser and operating system for both desktop and mobile devices. For instance, the browser alias "lambdatest:Firefox@78.0:Windows 10" is for a Windows 10 system using Firefox 78, and "lambdatest:Safari@13.0:MacOS Catalina" is for a Mac OS X Catalina (10.15) system using Safari 13. If you don't see a list of browser aliases, double-check your credentials and set them appropriately.

Running tests on LambdaTest

Now that you have LambdaTest and TestCafe set up, all that's left to do is execute your tests on the service. Thanks to TestCafe's plugin system, the command to perform this action is identical to running your tests locally:

testcafe <LambdaTest browser alias> <test files>

For example, if you want to run all of the tests on the project on Mac OS X Catalina using an instance of Microsoft Edge version 84, you can use the following command:

testcafe "lambdatest:MicrosoftEdge@84.0:MacOS Catalina" *_test.js

This command automatically sets up a tunnel between your system and LambdaTest, and triggers your test using the specified environment on the cloud. TestCafe receives the results for each test as they complete. The reporter plugin also shows a link to the LambdaTest session so that you can keep track of the test run on the service.

While the tests run, you can check the LambdaTest dashboard to see your test runs and current sessions' stats.

When the test run finishes, LambdaTest closes the session and keeps a record of your test run. In the Automation section of your LambdaTest account, you'll see a timeline of your builds and automated test runs. Here, you can access all of the test run information, including a video recording of the test run on LambdaTest's system.

When you use the TestCafe plugin to run your tests without modifying any additional settings, it will use the following:

The build name used is "Untitled".
The test run name used is automatically generated by TestCafe.
The screen resolution is 1024x768 pixels.

As mentioned earlier, you can modify LambdaTest's default settings through environment variables:

LT_BUILD - The name of the build to use for this automated test run.
LT_TEST_NAME - The name you want to use to identify your test run.
LT_RESOLUTION - The screen resolution for the LambdaTest environment, defined by width and height in pixels (like "1600x1200", for instance).

You can also modify other settings, including logging JavaScript console messages, record network packets, and disable video recording through other environment variables. See LambdaTest's TestCafe plugin documentation for more details, and LambdaTest's Selenium documentation for information on the platform's capabilities.

Continuous testing with LambdaTest and Travis CI

It's pretty nice having your TestCafe tests running on the cloud with LambdaTest. The LambdaTest plugin makes the entire process effortless. However, we can take it a step further by setting up continuous integration to automatically run your tests. If your project receives an update, your CI service can run your tests and alert you immediately if the changes cause any regressions.

To show how continuous testing works, we'll use one of the most popular continuous integration services: Travis CI. Travis CI is a hosted continuous integration service that syncs with your code repository to get your test automation up and running in no time. This article will show you how to connect your TestCafe code repository to TravisCI and set it up to run your tests on LambdaTest automatically.

TravisCI is free for open-source projects, and signing up is as simple as logging in with your preferred source code management provider. TravisCI supports GitHub, GitLab, Bitbucket, and Assembla, so if your project's code is on one of these platforms, you can automatically connect it to TravisCI.

After signing up with your source code management provider, TravisCI syncs up with your account to fetch your available code repositories. You can visit your account settings to search for the repo you want to connect with TravisCI and click on the toggle switch next to the repo name. Turning the toggle switch to "on" sets up your code repo with a webhook that notifies TravisCI of any changes.

Before you can use TravisCI to build your project, you need a bit of configuration. First, you need to set up the environment variables that LambdaTest needs, as discussed earlier in this article. TravisCI makes it easy to set up environment variables securely.

When you turn on the project inside TravisCI, click on the "Settings" button next to the toggle switch. You can set up the environment variables your project needs to run the build properly among the project settings. Here, you can set up the LambdaTest environment variables. Also, the TestCafe test suite uses a few environment variables for the account credentials throughout the test scenarios, so we need to set them here.

The final step in the process is to configure your project to tell TravisCI what it needs to do for every build. You can manage this configuration through a build configuration file called .travis.yml file, placed inside your code repository. In this file, you need to specify the language support for your project, at a minimum. Depending on the language, TravisCI will run a few commands by default, but you have control over the entire process if needed.

For our project, TestCafe is a testing tool that uses Node.js under the hood, so we can set our build configuration to use a Node.js platform when running builds on TravisCI. It's also a good practice to specify at least one Node.js version to ensure that your tests run on a known working version. Inside of the TestCafe test suite project, you can create the .travis.yml file in the root of the project with this configuration:

language: node_js
node_js:
  - 12

By defining node_js as the language for this project, TravisCI automatically runs the following commands:

npm install to install all of the project's dependencies.
npm test to run your tests automatically.

The project already has a valid package.json file containing the dependencies needed to run the TestCafe test suite on LambdaTest. The file also has the test script with the same command used earlier, so no further configuration is necessary with TravisCI. Here's the completed package.json file for reference:

{
  "name": "end_to_end_testing_with_testcafe",
  "version": "1.0.0",
  "description": "Demo repo showing how to run TestCafe tests using LambdaTest and TravisCI",
  "main": "login_test.js",
  "scripts": {
    "test": "testcafe 'lambdatest:MicrosoftEdge@84.0:MacOS Catalina' *_test.js"
  },
  "repository": {
    "type": "git",
    "url": "git+https://github.com/dennmart/lambdatest_travisci_testcafe.git"
  },
  "author": "Dennis Martinez",
  "license": "MIT",
  "bugs": {
    "url": "https://github.com/dennmart/lambdatest_travisci_testcafe/issues"
  },
  "homepage": "https://github.com/dennmart/lambdatest_travisci_testcafe#readme",
  "dependencies": {
    "testcafe": "^1.8.7",
    "testcafe-browser-provider-lambdatest": "^2.0.2"
  }
}

With the build configuration file in the project and TravisCI connected to the code repository, your tests will now run after every repo change. To test this out, push the .travis.yml file to the remote repository. If everything is set up correctly, TravisCI will automatically trigger a build after a couple of seconds.

The build starts by setting up the environment variables, installing all project dependencies, and runs the test script. With the environment variables in place, your tests will run as they did before. The LambdaTest plugin creates a tunnel to its service and runs the test in the specified operating system and browser.

With some minor setup, you can leverage the power of LambdaTest to execute your TestCafe tests in any environment, using TravisCI to run them for you automatically.

Summary

Testing your web application on a combination of operating systems, web browsers, and devices can be a hassle. It's costly and time-consuming to maintain different environments to ensure that your app works. With LambdaTest, it doesn't have to be a pain.

LambdaTest provides an easy yet powerful way to test any web application in a variety of setups. It provides all the tools to ensure your app works as intended, from spinning up different operating systems and browsers on-demand to performing other testing duties like responsive and visual testing.

This article shows how simple it is to set up a continuous testing workflow with just about any modern environment, thanks to LambdaTest and TravisCI. The examples shown here are for running a TestCafe suite, and LambdaTest provides a dead-simple way to run your tests on the cloud. If you don't use TestCafe, LambdaTest also supports lots of other languages and frameworks, so chances are you'll have support for your existing test suite.

These days, you have plenty of tools at your disposal to make your test automation journey easier. LambdaTest is one of the most robust and powerful services available to help you tackle any potential issues before your users encounter them. It'll improve the quality of your application with little extra effort.

Have you used LambdaTest with your projects or in your organization? Share your experiences in the comments below!

This post is sponsored by LambdaTest, a cloud-based cross-browser testing tool to perform manual or automated testing on over 2000 browsers online. All reviews and opinions expressed in this article are my own and based on my personal experience with the service.

Don't Start Your Automation Strategy With Tools

Dennis Martinez — Tue, 25 Aug 2020 11:00:00 +0000

"I started learning about Selenium but don't know what I'm doing."

"Someone recommended I use TestCafe, but it's not easy to use for testing my API."

"My team jumped into using Cypress for our UI tests, but we're a bit stuck after realizing our app needs testing some functionality on multiple browser tabs."

Every week when I scan around different websites and forums around the Internet, I see automation testers making comments like these. Often, these comments come from people who are new to test automation. Most of this discussion often centers around one thing and one thing only: test automation tools.

Whenever somebody drops a post in a forum asking what test automation tool they should use for their work, tons of opinions start flooding in. If you've ever searched for "best automation tools" in any search engine, you'll find a seemingly infinite amount of lists and articles containing hundreds of options at your disposal.

It's overwhelming and confusing for someone new to automation, having to wade through so many comments and opinions, trying to make sense with how to proceed on their journey. Most of you have gone through this experience when starting. I know I did. I still do whenever I have to expand my skills to automate testing on different platforms.

Most of us tend to reach for tools first when doing something new. It happens not just in test automation, but almost every facet in our lives. When someone wants to create a podcast, they often research microphones before recording something with their laptop's built-in mic or even their smartphone. People wishing to learn to speak a new language buy tons of books and other materials before saying their first word. And automation testers usually ask which programming language or software to use before figuring out what they need to test.

Of course, you need tools to accomplish whatever you set out to do. You can't record a podcast without a mic, say something in a different language without reading or hearing it somewhere, or write an automated test without knowing how to code or use a codeless automation tool. However, starting your test automation efforts with tooling is a big mistake I've seen repeatedly.

Why jumping in with tools is a huge mistake

There's nothing wrong with sampling and experimenting with different test automation tools for your goals. As mentioned, you'll eventually need to use something to get your automation strategy moving forward. The main problem lies with how quickly some people jump into using any tool for their work.

A typical mistake I've seen by newer automation testers is when they start using a test automation tool. The tool works for a few of their basic scenarios, and they instantly go all in for the long haul. By itself, it isn't a problem if the tool covers all of the defined test cases. But usually, they'll realize the tool they chose won't work for everything they need.

When this happens, one of two things happens: the tester begins to try to make the tool fit their workflow to avoid sunk costs, or they have to backtrack and find a different tool. If they stick with their chosen tool, it'll lead to a brittle test suite and challenging to maintain. Otherwise, they'll have to go back to the drawing board with something different. In either scenario, they'll waste lots of time and effort.

The key here is to find the right tool for the job with your automation strategy before committing. The best way to do that isn't to focus on tools - it's learning about the application under test first, before anything else.

Learn what you're testing before reaching for tools

When you get distracted by the potential of shiny new test automation tools, you'll often lose sight of what you should spend more time thinking about - making sure you know how the application you're testing works. You'd be surprised at how much you can discover by paying attention to the details that make your app unique.

To go even further, you should also spend time learning a bit about the underlying tech running the application and how everything works. You don't have to do a deep dive and become an expert on every detail, but knowing what makes the application run will help with your testing efforts. For example, what kind of database does the app use? Does it connect to external services? Does it perform asynchronous tasks in the background? Knowing these details can help shape a better test plan that can consider specific scenarios based on what makes the app work as it does.

As you understand more about how the application works, you'll begin to think about the higher risk areas that can affect different parts of the app. What areas are most fragile and require more robust test coverage? For instance, if you discover that the app relies on third-party APIs to work correctly, you can make sure to include enough test coverage around this functionality.

You don't need to handle all of this on your own. The rest of your team is a treasure trove of information to help with your automation efforts. Developers can tell you which areas are most likely to break with new changes or new features that will modify existing functionality. Product managers can provide metrics that indicate which parts of the app most of your customers use daily. Having additional details from other departments can help you prepare better tests and avoid testing areas that don't matter in the bigger picture.

The more you know, the less time you'll waste

With some up-front work, you'll have a clear idea about how the app works both on the surface and internally, and you'll have "inside information" from people with in-depth knowledge of the app's current and future state. With all this information in hand, you can begin introducing test automation and tooling into the mix. By this point, you'll know which tools are the right ones for the job, and it'll save you and your team lots of time and effort with your automation work.

For example, imagine you're testing a web application with a sophisticated user interface or many moving pieces under the hood. You can choose a testing framework like TestCafe or Cypress and plan to write longer end-to-end tests to validate what you need. In another example, maybe you have an app with a simple UI, but the underlying APIs need more attention. An end-to-end testing framework may be overkill. Instead, you can use tools more suitable for API testing like Postman or SoapUI.

If you had immediately jumped in to test automation without knowing how your application works, you would have likely chosen the wrong tools for the task at hand. Focusing on API testing when it's best to ensure a complex user interface works well won't help your efforts. Only having unit tests when an application runs with different interconnecting systems like message queues, asynchronous background task runners, or separate microservices is also a waste of time.

Taking the approach of learning about what you're testing will take more time for you to get started. In today's fast-paced work environment, it may create disagreements in teams that want to get things done right now. But the time you spend learning now will save a lot of effort and frustration in the long run by focusing on testing the right things while avoiding spending too much time on what doesn't matter. Quality is a trait that builds and compounds over time, not something that's rushed for a quick result.

Summary

One of the first things that automation testers tend to do, especially those new to test automation, is to find tools to start creating a test suite. However, it's better to learn about what you're testing before you learn what you'll use to test.

Before reaching for the tools, start by learning all you can about how your application under test works. You have to learn how the average end-user interacts with the app to spot which areas are prime candidates for testing. This information will give you a starting point for your test plan.

Beyond the user interface, you should also take the time to learn the underlying components that make the app run smoothly. You don't have to know how everything works, but being aware of what's happening beneath the surface will give you more information on what to test.

Finally, take the time to talk with others outside of the testing team. Others who have more hands-on experience with the application under test can provide details no one else can give. If you stay inside your bubble, you'll miss out on this invaluable insight that can make your testing efforts easier.

You'll need tools eventually, but jumping in with tools first can lead you to pick something that you'll end up fighting against the whole way. Once you know what to test and which areas to automate, it becomes a lot easier to figure out which tools to use and how to implement them into your workflow.

What's your strategy when starting your automation testing efforts in a new project? Share your experiences in the comments below!

How to Send Your TestCafe Test Results to TestRail

Dennis Martinez — Tue, 18 Aug 2020 11:00:00 +0000

If you're working among a group of testers at a company, your team is most likely using a test case management tool. These tools are indispensable for testing teams. A useful test case management tool provides a single source for the organization's test plans, allows the team to collaborate with creating and maintaining test cases, and see reports of your test runs.

In my experience, these tools work great with manual testing scenarios, as they provide testers with all they need to do their work. However, it's a bit tricky to get these systems in sync with automated testing. Sometimes the test case management tool doesn't provide a simple way to integrate external tools. Other times, your test automation framework makes it difficult to use its results elsewhere.

Thankfully, most modern test case management tools provide many ways to hook up third-party tools, such as test automation frameworks. For instance, a web-based test case management tool can have an API to create a new test run, aggregate your test results, and close the run. This way, any system can manage this information without the need to do it manually.

One such tool with a robust API is TestRail. It's one of the most popular test case management tools out there. Given its popularity, chances are your preferred automated testing framework has a library or plugin that sends your test run details to TestRail.

In this article, I'll show you how to use TestCafe to log your test runs in TestRail automatically.

The need to track automated tests in TestRail

In my previous job, the testing team used TestRail to keep track of their testing work. Their test plans and cases were detailed in the system, giving the rest of the team visibility into how they ran tests for our projects. It also helped non-QA team members like developers and project managers identify areas of risk, helping

Initially, QA performed most of their tests manually, and they would use TestRail as their guide. As the company grew and we had more projects to work on, this approach didn't scale very well. Some projects began to suffer, quality-wise, because there wasn't much time for the team to do what they needed. That's where we started to introduce more automated tests into the mix.

When the team began to write end-to-end tests with TestCafe, we wanted to acknowledge those automated tests alongside the team's manual testing efforts. At first, the team went through the results for the automated test suite and manually marked the results for each test case in TestRail. Obviously, doing this isn't the best use of a tester's time.

As more test automation got introduced to our projects, the testing team wanted to find a way to keep TestRail synchronized with our automated test runs since it took too much time to keep this information up to date on all relevant systems. We noticed that someone created a TestCafe reporter called testcafe-reporter-html-testrail that handles this for you.

After getting it set up, it worked exactly for what we needed. The reporter automatically opened a new test run on TestRail and sent over the results for our test cases. It worked great for the testers, who needed to focus on manual testing or verifying reported failures.

Making sure TestRail is ready to receive your TestCafe test results

Before showing how the integration between TestRail and TestCafe works, let's make sure TestRail is set up correctly. In the examples for this article, I'll use a TestRail Cloud account, which I set up for the TeamYap application test plan. The TestRail account has a single repository for TeamYap, and I added different sections for each test case similar to the structure for the automated tests.

Keep in mind that the sections and test case names don't matter with the integration between TestRail and TestCafe. The main thing you'll need is the Test Case ID, which you'll see in use later in this article. The important thing is that you have enough test cases written in TestRail to match with the automated tests written with TestCafe.

Additionally, you need to enable TestRail's API to allow TestCafe to send its results to your account. This setting is not enabled by default. To enable the TestRail API, log in to your account as an administrator, and go to Administration (on the upper-right corner on your dashboard), click on Site Settings, and select the API section (last icon on the navigation). Check the box labeled "Enable API" and save the settings. You don't need to enable session authentication for the API, so you can leave it unchecked.

After setting up a project with your test cases and enabling the API, your TestRail account is ready to accept your automated test results. The setup process detailed here are for TestRail Cloud. TestRail also allows you to install an instance of the application to your servers instead of a cloud-based account. I haven't used TestRail Server, but if your organization uses it instead of TestRail Cloud, I assume that the setup process is the same.

Setting up the TestCafe reporter for publishing test runs to TestRail

This article's tests come from a TestCafe test suite repository using the same tests used in my book, End-to-End Testing with TestCafe. This repo contains 11 end-to-end tests covering functionality for the TeamYap application explicitly built to give readers hands-on learning opportunities with TestCafe.

Installing the TestRail reporter for a TestCafe test suite is simple. As with most TestCafe reporters, it's available as a plugin that you can install using npm, just like any other TestCafe plugin. However, there is a caveat: The latest version of the reporter plugin doesn't work.

I reached out to the plugin owner to ask why the plugin doesn't work. He told me that the current version is meant for his organization's internal use and is currently not working with TestRail anymore, with no plans to fix it. The source code for the latest version of the plugin is private, so I couldn't lend him a hand to correct the problem. I don't understand his reasoning for not forking the project into a private repo or how he gained ownership of the npm repository, but that's a separate topic.

In the meantime, the workaround is to install an older version of the plugin. If you attempt to use the current version, it prints out a warning message asking you to use version 2.0.6 of the plugin for TestRail functionality. However, any 2.0 version will work. For this project, I'll use version 2.0.8 of the plugin, which can be installed with the following command:

npm install --save testcafe-reporter-html-testrail@2.0.8

Once the reporter plugin is installed for your TestCafe project, you can use the reporter when running your tests. Before you can do that, you still need some additional setup.

The way the reporter plugin works is by using environment variables to configure the integration with TestRail, and your TestCafe test names need to be formatted in a specific way to link them with a TestRail test case. Below, you'll find the different settings you need to set up the reporter.

Using environment variables to configure your TestRail credentials

The reporter plugin uses certain environment variables for the TestRail API credentials. You need to set the following environment variables if you want to send your TestCafe test run results to TestRail.

TESTRAIL_ENABLE: This variable accepts a boolean value (true or false) to tell the reporter plugin to send your test run results to TestRail. By default, it's set to false, which is useful to avoid publishing your test results when running tests locally. Usually, this variable is set to true in a continuous integration environment.
TESTRAIL_HOST: In this variable, you need to set up the host of your TestRail instance. For example, the TestRail account used in this article is hosted at https://devtester1.testrail.io/. Keep in mind that it's important to include the full URL, including the protocol (https://, in this example).
TESTRAIL_USER: Here, you'll specify the username of a TestRail account with access to your project. When the reporter plugin sends the test run results to TestRail, they'll get created under the account specified here.
TESTRAIL_PASS: You can use either the TestRail account password for the user specified in the previous environment variable, or use an API key created for the account. It's recommended to use an API key since you can limit access to your TestRail account and easily revoke it.
PROJECT_NAME: This variable indicates the TestRail project name containing the test cases for your automated test suite.
PLAN_NAME: TestRail uses the concept of "Test Plans" to manage multiple test runs if you need to run your tests on different configurations. This variable allows you to specify a plan name. If you don't set a plan name, it will publish your test results under the TestAutomation_1 test plan by default.

If you're having issues publishing your TestCafe results to TestRail, verify that you set all of these environment variables correctly.

Formatting your TestCafe test descriptions

To link a specific TestCafe test case to a TestRail test case, the reporter plugin requires a specific formatting of the test name in TestCafe:

test("<Test Type> | <Test Name> | <TestRail Test Case ID>", async t => {
  // Your test code goes here as usual.
});

Replace the following segments of the example above with your test case details:

<Test Type>: The type of test (like "smoke" or "regression").
<Test Name>: The name of your test, a description of what it does.
<TestRail Test Case ID>: The test case ID from TestRail that will link with your TestCafe test.

Of those three segments, the most important one is the TestRail Test Case ID - it must match a test case for the project in your TestRail account. The other two segments (test type and test name) don't affect the integration between TestCafe and TestRail. However, you should still ensure this information accurately describes your test.

Here's an example from the test repo used in this article. This test is a smoke test for verifying that the login functionality works correctly for the TeamYap application, where the TestRail test case ID is "C1":

test("Smoke | User with valid account can log in | C1", async t => {
  // Test code...
});

Running tests and sending results to TestRail

With the necessary environment variables set and your test names adequately formatted, all that's left is to run your tests and have the results sent automatically to TestRail.

The only thing you need to do is to specify the reporter when executing the TestCafe test suite. You can easily do this from the command line:

testcafe chrome:headless *_test.js --reporter html-testrail

This command will run your tests as you would expect. The output will be different from the default TestCafe reporter, but it will display the results of each test similarly.

After the test suite completes its execution and the TESTRAIL_ENABLE environment variable is set to true for your system, the reporter publishes the results to your TestRail account. If your TestRail account credentials are correct, you'll see the details for the newly created TestRail test run in the output.

The project and test plan names used are the ones you set up in the environment variables (as explained above). The name of the test run on TestRail will be a string that includes a timestamp of the test run, and the browser details provided by TestCafe.

If you go to your TestRail account, you'll see your newly-created test run under your test plans and the results for each test. Each test gets automatically linked to the test case, so over time, you can generate reports, view the history of test results, and export the data for your needs. You can then perform your usual TestRail duties like closing the test plan or re-assigning failing tests to other team members.

One thing to keep in mind is that you don't need to match the exact number of test cases between TestRail and TestCafe. For example, if you don't format the test name in TestCafe to include a TestRail test case ID, you can still execute it. It'll run, but the result won't show up in the test run on TestRail.

However, you will need to ensure that any test case IDs in the TestCafe test name match a test case ID on TestRail. For example, if you format a TestCafe test name to use a test case ID of "C1234", you must have a TestRail test case with that ID. Otherwise, the reporter will raise an error and send incomplete test results to TestRail.

This automatic integration isn't for all teams

When the organization I worked for used this approach, it worked well for their needs since they were at the beginning stages of creating the test suite. All they wanted was for the TestCafe test runs to get included in TestRail and avoid having to go into TestRail to manually add the scenarios covered by the project's end-to-end tests.

However, this automatic integration between TestCafe and TestRail might not work for your team in a few scenarios. Here are a few situations where it's probably best to run your automated tests and update TestRail separately:

You have lots of test cases in TestRail, but few automated tests covering them in TestCafe. Automatically creating test runs in TestRail for just a handful of automated tests isn't worth the effort unless you plan to boost your test automation efforts shortly.
You need more fine-grained organization around test cases in TestRail, such as setting time estimates and priorities. You can't set any of this information from your TestCafe tests using the reporter plugin, so your team will have to set these by hand.
You need to run the same tests in different environments that aren't supported or available in TestCafe. Each time you execute your TestCafe tests, it creates a test run under the specified test plan name. If your project needs to run your test cases in an environment TestCafe doesn't support or can't access, you'll need to create these test runs in TestRail manually.

The way the plugin uses the formatting of the test names in TestCafe also isn't ideal. The formatting can look messy since you have to include the type of test and ID along with the description. It's also easy to accidentally format the description incorrectly. Using metadata would be ideal for specifying TestRail related data like the test case ID, assignee, and more. Unfortunately, the reporter plugin only checks the TestCafe test name for its integration.

It all boils down to how much effort your team needs to maintain your TestRail account and TestCafe test suite. Sometimes these integrations cost more time than you think it could save. Pay attention to whether these approaches make your work easier or if it becomes a burden on your team.

Summary

Test case management tools are great to keep you and your team up to date with test plans for your projects. However, getting your test automation integrated with these systems often requires additional work to connect your automated test runs with your defined test cases.

If you use TestRail to manage your test cases and have a TestCafe test suite, you can easily keep your test case management tool synchronized. Thanks to TestCafe's easy-to-use plugin system, you can add a reporter called testcafe-reporter-html-testrail to handle this integration for you. All you need to do is configure a few environment variables and format your TestCafe test names to include the TestRail test case ID.

When correctly set up, the reporter plugin automatically submits your TestCafe test run to the configured TestRail account, linking each test to its result. Now you can manage your automated test results in your test case management tool with just some up-front setup.

While this approach might not work for all testing teams, it does help eliminate the busywork of entering your automated test results into TestRail by hand. If that's all your organization needs, this approach will save you lots of time while keeping your entire team on the same page with your test plans.

How have you managed your test automation with your test case management tools? Share your experiences in the comments section!

Running End-To-End Tests Without Blocking Your Team

Dennis Martinez — Tue, 11 Aug 2020 11:00:00 +0000

When you spend some time working on test automation, chances are you'll reach a point where your test suite is more of a hassle than a help. It takes longer and longer to execute your tests. The feedback loop between developers committing code and receiving the automated test results grows larger every day. Eventually, your team will begin to ignore your tests because they won't want to wait around.

If you reach this point, your first inclination might be to take the "scorched earth" approach and start again from scratch. I know I've had plenty of days with that thought. However, you can take steps to salvage the remnants of your test suite and get it back to a state where it doesn't block your team.

This article details my recent experience with an end-to-end test suite that slowed down the entire team and how we took a different road for executing those tests to keep the team's workflow moving along.

The Initial Honeymoon Phase

At the beginning of 2019, the previous company I worked at started to get serious about automating end-to-end tests for our projects. Previous attempts were made to implement automated UI testing, but those efforts never went far. Most projects in our organization relied on manual testing from the in-house QA team.

The teams for some of those projects had issues finishing their allocated work for the sprint on time. The main problem they faced was a slow regression testing phase. It took the QA team too much time to perform a full regression test, and every time they found a bug, the delays would increase until the team missed the delivery deadline.

The organization wanted to automate the repetitive work the QA team did for each sprint, giving testers more time to perform higher-value tasks like exploratory testing. We also wanted to get our projects to a point where we could implement continuous delivery. End-to-end testing would help us build confidence in our applications to release new changes to production automatically.

We got to work to begin the initial implementation for one project. We decided to use the TestCafe testing framework, since the project made extensive use of JavaScript, and we wanted part of the development team to pitch in since they had the product knowledge. The organization also wanted developers to pair with QA team members who wanted to learn more about automated testing.

After a few weeks, the development and testing teams managed to automate a good chunk of the regression tests, and others helped integrate it into the existing workflow. Whenever the team pushed new changes to the code repository, it would execute the end-to-end tests after running the existing automated test suite, including unit and functional tests.

From the start, the team started seeing how including these end-to-end tests would help the project. Instead of wasting time on the mind-numbing, repetitive work that would often get pushed to the end of the development cycle, the team had part of it taken care for them. It began to free the testing team to perform other tasks, while developers had more feedback after their changes.

However, not everything was rosy. We started to feel lots of bumps in this automated testing road.

The Hostility Phase

As the team continued to expand the automated test suite and increase the coverage for the application, an all-too-common issue reared its ugly head - the tests were slowing down the entire team.

End-to-end tests tend to be slow and flaky, and our initial attempts at writing these tests were no exception. Since this was the first time many on the team were doing any test automation, speed and reliability were missing in the test suite. Builds were running at least five times slower than before and would continually fail for no reason.

One of the mistakes we made as a team was attempting to automate too much, too quickly. In our quest to automate as much of the regression test cases as possible, we also built many extensive tests. These tests performed too many steps to cover as much functionality as possible, which led to slow performance and high flakiness.

Because of these extensive, unstable tests, the feedback loop between the time developers pushed out a code change, and the notification of the test results increased every day. If you throw in the increase in build failures, that creates an unhappy team - and rightfully so.

The development team didn't want the end-to-end tests to run after every code commit they pushed to the repository. We changed the workflow to run these tests only when specific branches were updated, like the release candidate branch or the main branch that we used to deploy to production.

This move helped minimize the build times during development. However, it was merely a placebo because running the tests infrequently created additional problems. Regressions were caught much later in the development cycle - often just before the project's release date. Eventually, it got to a point where the entire team ran into similar delays as they had before implementing the automated test suite.

Some on the team wanted to cut our losses, scrap the end-to-end tests, and get back to manual testing with additional resources. However, we didn't give up and put our heads together to find a way through.

The Adjustment Phase

As mentioned earlier, one of the issues we had with the test suite was that most test cases performed too many test cases. We also noticed that some tests executed almost the same steps every time, changing the data slightly or performing different assertions. These tests felt like duplicate work, so we did our best to trim unnecessary test cases.

You can only go so far with this approach, depending on your application. In our case, it was a rather complex application with many different scenarios. The QA team had lots of various tests that they felt necessary to run because they had experienced problems in the past when skipping those areas even though it felt repetitive.

With development wanting the builds to run faster while testing wanted to be thorough, we eventually reached a compromise. Instead of deleting tests, the QA team sat down to classify the end-to-end tests by type and priority. The team tagged each test with a label, like "smoke" or "sanity", and either high, medium, or low priority.

With this information in our test code, we could set our continuous integration system only to run higher-priority smoke tests after code changes. These tests took only 25% of the time to run compared to running the entire test suite, which was acceptable to avoid blocking the development team for too long.

For the remainder of the tests, we configured the continuous integration system to trigger a build for running the entire automated test suite at night for most of the team. If the test suite failed, the system sent a notification to the project's Slack channel so the team could catch it when they returned the next business day.

I found this split to be the best of both worlds. The development team didn't get stuck waiting for the test suite's results after pushing out new code, while the testing team was able to keep the tests they built without sacrificing thoroughness.

Much better, but there's still room for improvement

While this worked well, these changes still weren't perfect. We ran into our fair share of issues throughout the project while building the automated test suite.

Despite splitting up the end-to-end tests and only running a subset throughout the day, the team still had to wait a bit too long sometimes for the tests triggered by their changes to run. In the days leading up to a release, the team's activity tended to spike, and the continuous integration service had to queue up multiple builds at a time. The solution to this problem is often to throw money at the issue, meaning pay to get more build capacity.

Another issue that surfaced leading up to a deadline was an increase in the frequency of regressions occurring. Of course, it's great that the test suite caught these problems before they shipped to production. However, since many of the regressions were found during the nightly builds, it would disrupt the team's day since they had to deal with it. This problem can get solved by running the full test suite more often, although we struggled to find a good way to balance build times and acceptable feedback loops.

I also noticed that we needed to be extra-vigilant about how to classify any new tests. As the team built new functionality, they also created new automated test cases. However, many of these new tests got classified as high-priority smoke tests, and it wasn't long before the build times after each commit crept up to unacceptable levels. It's good to occasionally review your existing test suite to either reclassify tests or cull them from if they're no longer necessary or useful.

Still, even with these occasional troubles, the automated end-to-end test suite massively improved the testing team's efficiency. After a few months since we began implementing test automation to the project, the time used to perform regression testing in each sprint was cut nearly in half, and fewer bugs slipped through the cracks into production.

Summary

Automated testing is an excellent way to speed up your team by freeing them from the repetitive nature of regression testing. Instead of taking their time to go through the same test cases repeatedly, automating these steps can let them do other kinds of work to help boost the project's quality.

However, automation is not a silver bullet, and if you're not careful, you may run into plenty of issues. When starting with test automation, teams tend to want to automate everything through the UI and build lots of end-to-end tests. This tactic isn't sustainable. Eventually, you'll end up with a slow and unreliable test suite that no one on the team wants to use.

If you reach this point, you don't have to scrap everything and start again. You can take a few steps to change how your automated test suite behaves and avoid slowing down your project and your team.

A quick thing you can do with your existing test suite is to determine which tests should run frequently and which you can defer at a later time. Not every test should be a high-priority scenario. If you can extract a subset of tests that give you a high degree of confidence that the application is working well, you can set up your workflow to execute them first.

With the remainder of the tests, you can take advantage of test automation tools like continuous integration systems to run them when it doesn't interrupt the team's workday. Any long-running tests or tasks that you can execute at a time that doesn't block anyone will help avoid any bottlenecks during the development and testing cycles.

The key to running end-to-end tests is to automate as much as you can to give you and your team the freedom to worry about other important issues for your project. Even if it's not perfect, you'll increase the quality of your applications with automation giving you the time to do your best work.

How do you deal with long-running builds or test suites that the rest of the team doesn't like to execute? Let me know in the comments section below!

Getting Drone CI Up and Running With TestCafe Quickly

Dennis Martinez — Tue, 21 Jul 2020 11:00:00 +0000

A few weeks ago, I found myself looking for a continuous integration platform to self-host on my own servers. I have a few smaller projects (which software developer doesn't?) that I don't update frequently but want to ensure they still work whenever they get updated.

Typically, my continuous integration service of choice is CircleCI. I'm very familiar with how to use it effectively with all sorts of projects. However, I wanted to look for a self-hosted solution for these smaller projects because I had some spare computing power to spin up a CI solution. And while CircleCI has a free plan, I didn't want to rely on having enough build time whenever I focused on these projects in the future.

The most popular self-hosted CI server out there is Jenkins, and with good reason. It's been around for a long time - I've used its previous incarnation, Hudson, back in 2007 - and it generally works. But I didn't want to use Jenkins for a couple of reasons. The system feels exceptionally dated, as they haven't had any visual changes for a long time. Also, it relies on a plugin system that leads to issues whenever they're updated. I remember Jenkins not being the most comfortable system to maintain.

Last year I stumbled upon a relatively new continuous integration system called Drone CI. It looked interesting, but I didn't have any use for it, so I never got a chance to try it. Now that I wanted to spin up my own CI instance, I gave Drone CI another shot. The service was straightforward to set up and works excellent. I'm pretty happy with the results.

This week, I wanted to write about the process of setting up an instance of Drone CI using Docker. To show the CI service in use and how easy it is to run your automated tests with it, I set it up to run a TestCafe test suite every time the code repository - hosted on GitHub - gets updated.

Overview of Drone CI's architecture

You can configure Drone CI in many different ways, depending on your code environment and preferred architecture. Regardless of your setup, the typical Drone CI setup consists of two main components:

The Drone CI server: The server is responsible for configuring your code repositories, handling communication with the repo, and dealing with users for logging into the CI service.
The Drone CI runner: The runner polls the Drone CI server frequently to check if there are any new jobs to process. Once the server has a job, the runner handles the execution of your pipeline.

You'll need to run an instance of both the server and the runner for your builds to execute. You'll only need to spin up one instance of the Drone CI server and connect it to your preferred source code management provider. The server will need at least one runner to process the jobs as it receives them. You can also configure multiple runners if you need to run your pipeline in different ways, but this article won't cover that scenario.

Both the server and runner use Docker images to start their respective instances. As far as I can tell, there's no alternative form of installation for Drone CI. I consider this a benefit since it makes setting up and updating both the server and the runners a breeze.

Setting up the Drone CI server with GitHub

For this article, I'll set up Drone CI to connect to GitHub, since it's where most of my code repositories live. Drone CI also supports most major source code management providers like GitLab and Bitbucket, as well as some smaller self-hosted Git solutions like Gitea and Gogs. Check out the Drone CI documentation to learn how to use these other providers.

Before setting up the server, you need to first create an oAuth application on the GitHub account containing the repositories you want Drone CI to access. The oAuth application will allow you to sign in using your GitHub account and access your code for your Drone CI instance. Follow the instructions on GitHub's Developer documentation to create a new oAuth application.

Setting up the oAuth application on GitHub is simple. However, you need to set up the proper URLs to point at your Drone CI server:

Homepage URL: This URL lets you access the main Drone CI interface after setting up the server and the runner. You can set it to any fully qualified domain name you'll use for your instance of Drone CI. In the example image below, the Drone CI instance is set up https://ci.dev-tester.com/.
Authorization Callback URL: This URL is where GitHub redirects you after authorizing your account to share data with your Drone CI instance. This URL must be the same domain as the Homepage URL with the /login endpoint appended. In the example below, you can see the Authorization Callback URL is https://ci.dev-tester.com/login.

After your oAuth application gets created, you'll land on the app's settings page on GitHub. Here, you'll see two special keys: Client ID and Client Secret. These keys are used when running the Drone CI server, so make sure you copy them since you'll use them soon.

There's one more step needed before running the Drone CI server. For any Drone CI runner to communicate with your server, you need to generate a secret string to serve as authentication between the server and runners. This string essentially tells the Drone CI server that any runner attempting to poll it must have the same secret set up to avoid rogue runners in other networks trying to gain access to your repositories.

The secret can be any string you want to use, as long as it's randomized and not easy to guess. A quick way to generate a secure string on the command line is with the following command:

openssl rand -hex 16

Finally, it's time to get your instance of the Drone CI server up and running in your system. As mentioned earlier, Drone CI uses Docker, so all you need is to run the Drone CI server image with a volume to store the server data locally and a few environment variables for configuration.

The environment variables used to configure your instance of Drone CI vary, depending on your needs. An excellent place to begin is with this example command I used to start my Drone CI server configured with GitHub:

docker run -d \
  -v /var/lib/drone:/data \
  -e DRONE_GITHUB_CLIENT_ID={{GITHUB OAUTH CLIENT ID}} \
  -e DRONE_GITHUB_CLIENT_SECRET={{GITHUB OAUTH CLIENT SECRET}} \
  -e DRONE_RPC_SECRET={{GENERATED SECRET}} \
  -e DRONE_SERVER_HOST=ci.dev-tester.com\
  -e DRONE_SERVER_PROTO=https \
  -e DRONE_TLS_AUTOCERT=true \
  -e DRONE_USER_FILTER=dennmart \
  -p 80:80 \
  -p 443:443 \
  --restart always \
  --name drone \
  drone/drone:1

This Docker command uses the latest 1.x version (1.9.0, as if this writing) of the official Drone CI server images. The command sets up a volume using the /var/lib/drone directory on your system and maps it to the /data directory inside of the Drone CI container. It also publishes both port 80 and 443 for accessing the interface through your web browser. The environment variables in the command are used for configuring Drone. I'll go through the ones used in this example command:

DRONE_GITHUB_CLIENT_ID: The value of this variable is the Client ID from the GitHub oAuth application you set up earlier.
DRONE_GITHUB_CLIENT_SECRET: The value of this variable is the Client Secret from the GitHub oAuth application.
DRONE_RPC_SECRET: The value of this variable is the generated secret string mentioned earlier in this article, to allow communication between the server and the runner you'll set later.
DRONE_SERVER_HOST: Here, you'll set up the domain of your Drone CI server instance, as established in the GitHub oAuth application. Note that you don't need to specify the protocol; you only need the domain.
DRONE_SERVER_PROTO: In this environment variable, you can set whether you want your public-facing instance of Drone CI to use standard HTTP or secure HTTPS connections. Here, I'm using HTTPS to keep the communication secure.
DRONE_TLS_AUTOCERT: One useful feature provided by Drone CI is its ability to generate an SSL certificate using Lets Encrypt automatically. Setting this environment variable to true handles this step for you and configures the Drone CI server to accept secure requests. The certificate generation doesn't occur by default, so it won't do anything unless you use this setting.
DRONE_USER_FILTER: By default, Drone CI sets up a public-facing interface that anyone with a GitHub account can access. While a logged-in user can only see their GitHub repos, you might not want others running builds on your systems. This environment variable limits the GitHub users or organizations that can log in to your Drone CI instance. In this example, I'm only granting access to my personal GitHub account.

That's all you need to get a running instance of the Drone CI server. The first time you visit the URL for the server (as set up with the GitHub oAuth application), you'll be redirected to GitHub to give your instance of Drone CI permission for accessing your repos. If you don't get redirected or Drone CI doesn't load in the browser, check the Docker logs in your system by running the command docker logs drone and verify if any errors occurred during the setup process.

Once GitHub has permission, you'll get redirected back to your Drone CI server, and you'll get logged in to the main interface. The first time you arrive at this interface, Drone CI spends a couple of minutes synchronizing with your GitHub account to pull the information for your code repositories.

Setting up the Drone CI runner with Docker

After syncing your GitHub information, you'll see all of your repos and configure the ones you want to run your builds. However, the Drone CI server won't do anything by itself. You'll need to run a Drone CI runner instance. As mentioned earlier in this article, the runner is what does the heavy lifting of executing your builds. Without it, the Drone CI server won't have a place to send jobs for execution.

Drone CI has different kinds of runners for executing your builds. Each runner has its pros and cons, depending on the workload you need to use. The Drone CI documentation contains advice for when to use or avoid each runner for your projects, so it's well worth exploring which one suits you the best.

For this article, the simplest runner to use is the Docker runner, since it'll get set up on the same server. Like setting up the Drone CI server, getting the runner set up consists of a Docker command and environment variables for configuration. Here's the command I used to set up my runner:

docker run -d \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -e DRONE_RPC_PROTO=https \
  -e DRONE_RPC_HOST=ci.dev-tester.com \
  -e DRONE_RPC_SECRET={{GENERATED SECRET}} \
  -e DRONE_RUNNER_CAPACITY=2 \
  -e DRONE_RUNNER_NAME=${HOSTNAME} \
  -p 3000:3000 \
  --restart always \
  --name runner \
  drone/drone-runner-docker:1

The command to set up the Docker runner uses the latest 1.x version (1.4.0, as if this writing) of the official Drone CI Docker Runner image. The command mounts the Docker daemon socket from your system to the container. This volume mount is essential to allow the runner to spin up Docker instances from within the container when running your builds. It also publishes port 3000 for communication between the server and the runner. The environment variables used in this command are the following:

DRONE_RPC_PROTO: This environment variable sets up the protocol used between the Drone CI server and the runner, either secure (https) or insecure (http). It's recommended to use https, especially if setting up the runner on a different server.
DRONE_RPC_HOST: The value of this variable is the hostname of the Drone CI server to allow the runner to poll and receive jobs to execute.
DRONE_RPC_SECRET: This value is the secret string you generated earlier. It needs to be the same value as the string used when setting up the Drone CI server. Otherwise, the runner can't connect to the server to receive jobs.
DRONE_RUNNER_CAPACITY: With this value, you can specify how many concurrent jobs you want the runner to process. This value depends on your server's capacity and the resources used to run your jobs, so you may need to experiment to find a number that works well for you.
DRONE_RUNNER_NAME: This variable is an optional setting to give the runner a unique name. The Drone CI server stores this information when it sends a job for processing to know which runner executed the build. Here, it's using the hostname from the system's HOSTNAME environment variable.

With that command, you have the Drone CI runner set up, ready to process any jobs it fetches from the server. To make sure the runner is configured correctly, check the logs using the docker logs runner command. If you set everything correctly, you'll see the runner start and ping the Drone CI server successfully:

time="2020-07-10T06:33:08Z" level=info msg="starting the server" addr=":3000"
time="2020-07-10T06:33:08Z" level=info msg="successfully pinged the remote server"

If the runner can't connect to the server, make sure the server is running, and the DRONE_RPC_SECRET environment variable used to start up both the server and runner are the same. Those issues were the main problems I encountered when first setting up Drone CI.

With the Drone CI server and runner set up, the last step is to connect a code repository and set it up to run your builds after every commit.

Setting up your repository for Drone CI

For this article, I created a repository containing a TestCafe test suite based on the code samples used in my book End-to-End Testing with TestCafe. This test suite includes 11 end-to-end tests covering an application called TeamYap, built to complement the book. I'll use this repository to demonstrate how to connect Drone CI to a GitHub repo and configure the build process to run the tests.

First, log in to the Drone CI server interface. Once Drone CI pulls in your GitHub repositories, find the repo you want to use and click the "Activate" link. This link takes you to the settings page for the repo where you can connect it with GitHub.

Activating the repo sets up a webhook in the GitHub repo that will send information to Drone CI every time new code gets committed. When Drone CI receives this data, it creates a job for a build and waits for a runner to pick it up.

After activating the repo, you can spend some time configuring how it will interact with Drone CI. You can set up cron jobs to automatically run builds at predetermined times, or prevent pull requests or forked repos to trigger new builds, among other settings.

For the TestCafe test suite used in this article, I had to set up a few secrets. Some of the tests use environment variables to avoid putting plain-text passwords in the repository. You can set these secrets through the Drone CI interface, which I can then configure as environment variables when running the build. In this example, I set up two secrets, called admin_password and regular_password. You'll see these secrets in use later in this article.

The last step needed to run your builds on Drone CI is to set up a configuration file inside the code repository to set up a pipeline. A pipeline is the series of steps you want the runner to execute, like running tests or deploying your code to a staging environment. These steps are defined in a YAML file placed in the root of your code repository. By default, these steps should be in a file called .drone.yml, but you can change this in the settings page on Drone CI.

Inside the .drone.yml file, you can begin to set up the steps needed to trigger a build successfully. A high-level overview of the steps used to run TestCafe tests for this example are:

Pull in the TestCafe Docker image to use as the base for running the tests.
Set up environment variables from the configured secrets inside Drone CI.
Execute the tests.

Here's the .drone.yml file used in this repository for running these steps:

kind: pipeline
type: docker
name: default

steps:
- name: testcafe
  image: testcafe/testcafe
  commands:
  - /opt/testcafe/docker/testcafe-docker.sh chromium *_test.js
  environment:
    ADMIN_PASSWORD:
      from_secret: admin_password
    REGULAR_PASSWORD:
      from_secret: regular_password

Here's a brief explanation of each key and value in this file:

kind: Defines the kind of process Drone CI uses to process the remainder of the YAML in this file. In this case, it tells Drone CI that these instructions are to execute a pipeline of instructions using the pipeline value.
type: Defines what pipeline to use for running the steps defined later in the file. Since we're using a Docker runner, it uses the docker value.
name: Defines the name of the pipeline for identifying when running builds. This example uses default, but it can be anything.
steps: Defines a series of steps for execution in the runner. Inside of this key, you'll have an array of pipeline steps that run serially. This configuration file only has a series of pipeline steps with their configuration settings.

Within the steps key, you'll find additional configuration settings:

name: Defines the name of the pipeline step, which is useful for seeing each step of the process in the Drone CI interface. You can set any identifiable name in this setting.
image: Since this is a Docker pipeline, this defines the Docker image you want to use to execute the commands for the step. Drone CI pulls this image automatically and clones your code inside of the container.
commands: Defines an array of shell commands that get executed inside of the Docker container. If any command returns a non-zero status after execution, the pipeline step will fail.
environment: Allows you to set up any environment variables inside of the Docker container. In this example, the file sets up an environment variable by using the from_secret key. This key tells Drone CI to fetch the value from the Secrets section in the project's settings.

This YAML file is all that's needed to run the TestCafe tests in Drone CI. Commit this file to the root of your code repository and push it to GitHub. Your Drone CI instance will receive a webhook from GitHub and kick off the build.

Before wrapping up, I wanted to explain a few issues I had to deal with before getting the test suite to execute successfully.

Pitfalls encountered when running TestCafe tests on Drone CI using Docker

For these tests, I opted to use the official TestCafe Docker image, which I have written about before. This Docker image has everything needed to execute a TestCafe test suite, so you don't have to spend time setting up dependencies using a different image.

When you use the Docker image outside of Drone CI, you simply need to specify a browser (either Chromium or Firefox, which are already set up in the image) and the test files. However, you can't run your tests like this in Drone CI. That's because any commands specified in the commands section of the .drone.yml file will override the ENTRYPOINT command specified in the Docker image.

The TestCafe image contains an entry point that's important for getting tests to run. The entry point executes a script that sets up a display server to allow the browsers to work inside the container and runs the TestCafe executable with a few arguments. Initially, I attempted to execute the test suite using the command testcafe chromium *_test.js in .drone.yml, but this overrode the entry point, and the tests couldn't run.

To get around this, I'm specifying the same script used as the entry point defined in the image's Dockerfile, along with the commands to execute the tests. Using the script sets up the Docker container correctly, allowing the tests to run successfully.

Using the script for the entry point isn't ideal since its purpose is not immediately apparent. Any developer or tester looking at this file would have to find the script in TestCafe's repository to figure out what it does. Also, if the maintainers of the TestCafe Docker image change this entry point, your builds will begin to fail, and you'll have to update the Drone CI configuration. But for now, this keeps our builds working.

Summary

This article barely scratched the surface of what Drone CI can do. You can run multiple pipelines in your project, like setting up different steps for unit and integration tests in the same build. You can also configure it to automatically deploy your code upon a successful build. The Drone CI documentation goes into more detail about all of its functionality.

In this article, I showed how simple it is to get an instance of Drone CI set up. You learned how to connect your GitHub account to the Drone CI server, and set up a runner to poll and fetch new jobs for execution. It also has a simple example of running end-to-end tests using TestCafe and some tricky areas you may need to navigate to get your builds working.

Besides how well Drone CI worked for my use case, I was surprised at how quickly I managed to get this system working. Between setting up the server and runner and getting my tests to run, the entire process took me about an hour starting from scratch.

If you're looking for a self-hosted continuous integration solution, I recommend taking Drone CI for a spin. It covers almost all continuous integration and continuous delivery needs your organization may have, regardless of which source code management provider or server architecture you use.

Stop the QA Gatekeeping Now

Dennis Martinez — Tue, 14 Jul 2020 11:00:00 +0000

Recently on Twitter, I ran across a tweet where someone asked "As a tester, when do you give the approval for release?" Few questions make me react in a less-than-positive way upon first reading them. This question did precisely that. I felt a bit agitated with this topic because it digs up some unpleasant memories from previous workplaces.

The most frustrating projects I've worked on as a developer were those with this kind of "QA gatekeeping", where QA decided whether to approve a release or hold it back. Whenever I've been on teams that placed the responsibility for a project's deployment to production solely on the testing team, disputes inevitably happened. Sometimes it happened as the development cycle ended, while other times, it was a silent ticking time bomb that blew up weeks or months down the road.

Online, I've seen many testers talk about their teams handling the responsibility of project releases, and every time it causes me endless frustration. I don't understand why requiring QA or testers to approve a release is still something that organizations do these days. I'm not diminishing the importance of QA in the release cycle, nor blaming testers for creating friction between different team members. When placed in these situations, everyone does their job the best they could and with good intentions.

However, putting testers as the group responsible for a release to occur puts the unfair pressure on them. It also creates silos between product, development, and testing - whether intentional or not. In the end, you have constant finger-pointing, erosion of trust, and someone becoming the scapegoat if things go wrong. These issues serve no one in the organization.

Anti-patterns of QA gatekeeping

In my experience, I've noticed two typical scenarios emerging in these kinds of projects that caused long-term issues to the teams and the organization as a whole.

Scenario #1: QA holds up releases due to stringent bug classification

Some testing teams are more lenient when it comes to classifying defects during the development cycle. The definition of quality seems to vary by team - what's a small issue for a QA group is sometimes a blocker for others. It's a tricky issue to balance since there's no "one size fits all" approach to determining the severity of a bug.

The issue with this comes when a QA team uses strict bug classification to hold up a release, no questions asked. If the team logged bugs with a specific label, the development team was obligated to fix every single one of them before the code could be deployed. I understand the reason why. After all, we shouldn't deploy code with a visible defect that hurts the product. But this system breaks down when bugs are classified incorrectly.

I have one frustrating example that comes to mind immediately. I once had a pull request blocked from being merged and deployed because the QA team found what they labeled a high-priority defect in the feature. The defect? The spacing between an image and the text underneath it was off by one pixel - and only in Internet Explorer 11. It held up the release of the project until the issue got resolved.

Of course, the QA team should log the bug as something they found during their testing. But to this day, I don't understand why they placed this issue as a blocker. The problem was so subtle that most people on the team could barely notice it even pointed out. The spacing wasn't breaking any functionality. And according to our analytics, less than 1% of those using our product used Internet Explorer 11.

This tiny issue caused a lot of back and forth between myself, the tester, and other stakeholders in the project. In the end, the bug was marked as "won't fix". Since the team was distributed across the world and working in different time zones, this back and forth ended up delaying the release. Admittedly, it also took plenty of time to repair the trust between everyone involved.

This example shows that keeping a system where QA can hold up releases because of their bug classification can create situations that harm the overall project. Bugs should be labeled accordingly, but there needs to be some space for re-evaluating these classifications since not everyone sees things the same way.

Scenario #2: Bugs still slip to production, and QA gets chewed out for letting it happen

Another scenario I've seen with organizations that require QA to give the green light before deploying is with the inevitable bug getting through the cracks. In the typical release cycle, development spends some portion of their time doing bug fixes and clearing out the QA backlog. When QA doesn't find any blockers, they'll give a thumbs up, and the project goes out to the customers.

Still, nobody's perfect. Bugs will slip by no matter how much testing is done before a release or how much time developers spend in bug-fixing mode. In organizations with QA gatekeeping, guess who gets the blame when the product team receives a bug report from a user. It'll fail mainly on the testers, every single time.

These issues happen the most when there's a massive time crunch to develop and release a product. Often, the team is overworked, scope creep gets out of control, and no one has a firm grasp on building quality work. When there's little time to develop a product, quality is almost always the first thing to fly out the window.

I have been part of teams that have the mindset of building stuff fast and not worry about writing tests because "QA will handle that". I've never believed in that philosophy. As a developer, I've been scolded by team leads because I refused to push a feature I had been developing until I finished writing some automated tests for it. They wanted me to submit my untested code and let QA test it for me.

On those types of teams, there's a cycle of pushing a ton of work to QA for testing. With a lack of time or resources to fully dedicate to the project, the testers do the best job they can to prevent any deadlines from slipping away. The risk of bugs getting deployed to production increases. Since the project passed last through the testing team, they'll always be the scapegoat, while the development team gets by relatively unscathed. It inevitably kills the team's morale.

How to stop the QA gatekeeping now

If you're in a team with one (or both) of these issues, you should try to help your organization as soon as you can, before your project continues sinking further and further into a hole.

The project teams that I've seen with the most overall success when it comes to delivering quality products on time had a few common traits.

They made testing a whole team effort

The most productive teams running projects with the fewest number of bugs or defects that I've been a part of have made everyone responsible for testing. The responsibility of testing still went to QA, because that's their expertise. But the rest of the stakeholders for the project still did their fair share in their own way:

Developers took the time to write automated tests for their code. At the very least, they covered their work with unit tests, but I've also seen a few development teams creating end-to-end tests for new features.
Product managers and designers frequently checked on staging environments to do acceptance testing for new features and make sure the product looked and functioned as expected.
The people responsible for DevOps made sure to set proper monitoring and alerts, like setting up continuous integration systems that performed various types of testing when new code gets committed.

These teams didn't rely solely on QA to determine whether the project was fit for deployment or if it still needed more work. If any of the teams thought there was something necessary to address before releasing to production, it was discussed between each group. For instance, if QA found an issue, they would talk with product and development to see if it's something to fix now or defer to a later time.

This approach worked exceptionally well because it fostered discussion across different disciplines. Instead of everyone working in their bubble, they would come together to discuss potential problems before they created further delays. It provided additional context to the issues at hand and cleared up any ambiguity about the severity of a defect (like my off-by-one-pixel issue).

If your organization feels like separate silos, it's best to bring it to the organization's attention and foster more unity across job functions. Testing is a cornerstone of excellent products, and everyone needs to work together to get there.

They tested early and often

Besides having everyone involved in the project to do their part with testing, these productive teams also tested as much as they could, as early as possible. They didn't wait until a pre-determined spot in the project's schedule to begin testing - it was part of their regular routine.

Making testing a part of everyone's work is easier said than done, but these teams took steps to make it dead-simple to bake in quality from the start by making testing part of the workflow. For example:

DevOps and the developers set up systems that automatically generated testing environments when new code or a pull request was committed. This workflow allowed them to create staging servers for new features and help non-technical folks quickly perform acceptance testing.
Different tests were run at various points throughout the day automatically. When developers committed new code to a branch, it kicked off a process to run a few quick tests. When merging code into the main branch, it ran more thorough tests. At night, a full battery of tests ran and generated reports of the project's current state.
The QA teams had free reign to run manual and exploratory testing alongside with other responsibilities. This type of testing wasn't a single explicit activity tacked on the timeline. The entire team understood that QA was working alongside everyone else at all times.

With the systems in place to run automated tests and create environments for new features before deploying, testing never had to wait. None of these team members had to overthink about quality during their workday. Whenever they did have to dig in to do testing, they had what they needed to dive in quickly. It helped a lot with minimizing risk, especially at the end of the cycle or sprint.

Summary

Some organizations still do "QA gatekeeping", where the QA team is the group responsible for the release of a project. This practice is harmful in many ways, putting unfair pressure on testers, and it promotes a culture of dissent between different teams working on the same project.

I've seen two anti-patterns occur when organizations have this practice in place. One issue is that projects get delayed due to QA blocking small and sometimes insignificant issues. Another problem is when bugs inevitably slip through to production, QA becomes the scapegoat because they're the ones giving the go-ahead.

The most productive teams I've observed have a few similarities that have helped them avoid these pitfalls. These teams have everyone working on the project do their part with testing instead of solely placing the responsibility with QA. They also test frequently as early as possible, setting systems in place to make it easy for everyone to do their part in ensuring a quality product.

Quality is something that works best when it's baked into the team's workflow, involving everyone who's a part of the project. If your organization practices some of the anti-patterns mentioned in this article, bring it to the attention of those who can help you make a change. QA needs to stop being the gatekeeper for your projects to thrive.

Are you involved in a testing team that serves as a gatekeeper? Has it helped your organization, or has it created problems? Share your story by leaving a comment!

Now Is the Time for You to Super-Charge Your Skills

Dennis Martinez — Tue, 07 Jul 2020 11:00:00 +0000

Every morning when I check my social media feed, I've seen more and more messages like this:

Due to the current pandemic affecting everyone globally, developers and testers have seen their jobs disappear overnight when companies have to lay off workers for their businesses to survive.

Another trend of messages I've noticed come from those who still have their engineering and QA jobs, but have received salary reductions and are starting to look elsewhere. They feel stressed and overworked because they're doing the work of those who got let go while making less money. A developer friend who works in a startup that reduced half of the engineering department told me she's doing the work of at least three of the people who the organization let go.

The job market is slowly recovering in many places across the globe, but it still seems like people are still in the market for work, unable to find something that fits their talents. With more and more people getting laid off from all types of jobs, it's becoming increasingly difficult to find a suitable position elsewhere, regardless of whether it's your choice or your former employer made the choice for you.

Everybody's in a different spot in life right now, but in the bigger picture, we're all in this together. We're all on the same boat.

Still, that doesn't mean you have to stay there.

Now is the time for you

Right now, life is distracting. Every day we're reminded of what's happening and how much everything changed in just a few months. Our attention is pulled in a thousand directions - often away from what would benefit us the most. I continuously see messages from people saying that their productivity has tanked since this entire situation erupted.

Regardless of what's going on in your world, there's one thing that I believe could propel you to come out better than ever out of this current situation - improving yourself and getting prepared for what's next for you.

At this exact moment, it's a perfect time to level up your skills. Yes, life is distracting at this moment. But it's also the best time to get things done because everyone else is distracted too. If you place those distractions to the side and focus on adding something new to your skill set, you'll have little to no interruptions from others. Now is the time for you to do that thing you've thought of doing for so long.

Why upgrade your skills during this time?

Taking the time to upgrade your existing skills and add new ones for your career helps you stand out from a crowded field of developers and testers seeking the same positions as you. These days, it's especially true with a highly competitive IT and testing industry and more job seekers for fewer vacancies.

It's tough for both new testers and developers trying to break in for their first jobs in tech and engineers with years of professional experience under their belts. Many recruiters these days have to sift through hundreds of applications for a single role. In some cases, adding that one extra skill in your resume can be the difference between getting hired or having your resume shoved aside.

It might feel like your career will never be the same. But there will come a time where things will return to a sense of normalcy. When that time comes, you need to be ready to jump in head-first. It's up to you to demonstrate that whatever development or testing job comes up, you can fill in those shoes with no problems.

Even if you do have a job at this time, it's still beneficial to improve your skillset, especially if you're seeking a salary increase or a promotion. The common consensus between economists indicates that wages have stagnated all over the world. Staying stuck in place limits your earning potential. You can't expect your organization to give you a substantial raise if you don't offer more than you do now, so it's also a great time to boost your abilities for your benefit.

So what can you do?

Everyone's path for gaining new skills is different, but I've found that you can start with three steps to get you on the right track.

1. Focus on you - only you - and figure out what you want

It may seem obvious that knowing what to do is the first step, but it's surprising how many people falter in the beginning. It's not that you don't know what to do. Usually, it's that you have too many ideas on what to do. The choices become overwhelming, and the fear of choosing the wrong thing keeps you from moving forward. Choosing one goal early in the process helps guide you where you want to go. Here are a couple of ideas for developers and testers:

If you're a manual tester and want to learn automation, focus on learning the basics of programming or how to use an automation framework.
If you're looking for a new job, find a gap in your abilities and fill it to expand your employment opportunities.
If you have a job and want to improve your earning potential, find ways to increase your career value, like becoming ISTQB certified.

Remember that whatever you choose here isn't set in stone. If you feel like the choice you made won't benefit you, you can always choose something else. Just make sure you're not running away at the first sign of struggle because that will keep you in the same place you are now - which I'm guessing you don't want to be in.

The other crucial element of this step is to focus on what you want. It's easy to fall into the trap of paying attention only to what others expect from you, or what others would think of you. That line of thinking is one of the most dangerous traps you can fall into, and I'd bet it's most responsible for killing people's dreams, more than anything else.

That's not to say that you need to neglect everyone else around you. You still need input from people who might be affected by your decisions, like your spouse or close family members. But if there's something you feel strongly about doing, you need to focus on yourself first so that you can focus on everyone else better later.

2. Take one step right now

Whatever the choice you make, think about the one thing you can do right now that would have the most significant and immediate impact on your goal. The path to every goal has a series of steps, and some of those steps you can take action on at this very moment.

Acting quickly serves you in many ways:

It'll get you on your way towards your goal now, even if you're still worried about being able to do this thing.
It'll motivate you to take the next step toward your target because the road ahead will seem less difficult to traverse.
In some cases, it might give you a sign that you shouldn't do this now, so you can find something else without losing much.

Your first step should be something relatively small that you can do in the next couple of minutes. For example, it could be buying a book or course that gets you started on learning that new skill you want. For job seekers, it could be sending your resume to a few companies you've had your eye on, or doing more research on what you need to find a better position.

The key is to do something as soon as you can so you can get moving. If you don't start moving, it's really easy to become paralyzed due to inactivity. By taking one step forward, you break that pattern, and it makes the next one a little more attainable.

3. Work on your goal for 30 days

Taking one step is excellent to start gaining some traction towards your goals. But one step alone won't take you far. You'll need to do more than a single action if you want to acquire new skills. Ideally, it'll be something every day, no matter how big or small.

An excellent way to get moving and stay moving is to work on your goal every day for the next couple of weeks. As developers and testers, many of us are familiar with sprints at work. Set a personal sprint for yourself. In the next 30 days, carve out some time for leveling up your skills.

Here are some of the things you can do to improve your development and testing skills every day:

Read a portion of a book or view part of a video course and take notes on what you learn.
Build an application or set up a test automation framework to showcase your abilities to potential employers.
Find an open-source project that you can contribute to with bug fixes or writing automated tests.

At this point, be aware of your constraints. If you don't have much time to improve existing skills or obtain new ones, don't set out a plan to work a few hours on it every day. It'll end up with you throwing your hands in the air and giving up quickly. A great strategy that has honestly changed my life is what's known as mini habits.

Instead of thinking you have to spend hours working towards your goals daily, you should think of the least you can do every day to move forward. If you can do more than the absolute minimum, that's fine, but do something every day. The secret sauce of this strategy is that it helps you avoid the frustration of feeling that you didn't do enough, and instead focuses on celebrating even the tiniest of wins that keep you coming back every single day.

If this strategy sounds useful for you, I highly recommend reading the books Mini Habits: Smaller Habits, Bigger Results, and its follow-up, Elastic Habits: How to Create Smarter Habits That Adapt to Your Day, both by Stephen Guise. They transformed the way I approach my work and learning.

My story and how this article came about

The reason why I'm writing this is that 2020 has been pretty rough when it comes to my career and my goals. Everything has changed quite a bit for me in the past few months due to the current pandemic and its impact across the world.

In January 2020, I decided to leave the organization I had worked at for almost five years. During the months leading to that decision, I felt like the role I was in wasn't going to serve my career moving forward. I felt stuck and, despite looking for additional ways to provide value to the company, I didn't feel like any change would happen within the existing structure. It was time for a change.

I didn't want to jump immediately into another full-time job for another organization. I planned to return to freelancing and consulting, which I had done successfully in 2014 and 2015. In early February, I let my network know that I would become available to help on a contract basis. Almost immediately, I lined up four projects for the upcoming months. I had a plan of action, and the outlook looked positive for my new path.

In March, the COVID-19 pandemic exploded around the world, and all of those projects I had in my pipeline vanished just as quickly as they came.

"No problem," I thought. "I'll just start looking for other projects." But that was much easier said than done. It turns out that finding any kind of remote contract work became almost impossible.

As more people got laid off or furloughed from their jobs, more people began seeking the same work I was hunting. Not only were they looking for those jobs, but they were willing to do it for much less than before they lost their previous jobs. I know the value and experience I can provide to those who hire me, and I'm not willing to slash my rates by over 75% just to land any job, as some responses indicated. I didn't want to be involved in a race to the bottom.

Still, it was incredibly stressful to find myself out of work for as long as I did. At this point, I fell into a deep funk, quite honestly bordering on depression. I couldn't find any suitable work, and I felt paralyzed about what to do next. I had a ton of ideas floating in my head, and I didn't know where to begin.

At around this point is when I started taking my own advice. The first step was to focus on myself and what I wanted to do. Did I really want to spend my entire days looking for scraps of work in an increasingly crowded field because I felt like that's what was expected of me?

Deep down, I knew I didn't want to do that. I didn't want to spend so much time and energy trying to compete with others who undervalue their work significantly. I felt it would take a while before things got to normal with organizations looking to hire developers and testers. That's when I decided to focus entirely on me and do something I wanted to do for a long time - write a book.

I had the idea of writing a book about TestCafe for a while, but never took the first step towards making it happen. When I decided to focus on me and what I wanted to do, I started to learn what it took to write a book. I never wrote a book before, and it's not as simple as writing a few pages and publishing it somewhere. There was a lot of research on what I would write about and how to go about it.

I spent some time every day working on something to make the book project come to life. At first, it was researching TestCafe deeper and deciding what to write. I needed to write the book in a way that made sense for both beginners and people familiar with the framework. Next, it was figuring out what tools to use to write and publish the book. Then I began reading a lot about marketing to learn how to spread the word effectively.

Every single day, I took some kind of action to move forward. It was exhausting, trying to balance the acquisition of new skills, making progress towards the book, keeping Dev Tester going with fresh new articles every week, and keeping my personal life in balance. However, keeping track of what I could do, even if it was a tiny win, steered me toward my goal.

I'm proud of the results of the work I put in. I finished writing my book and will release it soon, which I hope helps many people who want to learn about TestCafe. I also learned a few new skills that I'll put into use not only for future work but for myself and new products that I'm building soon. This period taught me that I have something to offer and that I can do it independently.

Three months ago, I was in a rather dark place with my career and didn't know what would come out of it. But deciding on what I wanted to focus on for myself and not for others, taking a few initial steps, and working on the goal every day - it honestly transformed my life.

I'm not writing this story to boast or impress anyone. I want everyone reading this to know that now is the perfect time to work on yourself. When this crazy, turbulent time passes by, and the world gets back to normal - and it will get back to normal, despite how things may seem - you'll be farther ahead than everyone else around you. The reason is that you took advantage of this time of crisis to do something about it.

Start now.

End-to-End Testing with TestCafe Book Excerpt: Intercepting HTTP Requests

Dennis Martinez — Tue, 30 Jun 2020 11:00:00 +0000

This article is an excerpt from one of the chapters of my upcoming book, End-to-End Testing with TestCafe. If you're interested in learning more about the book, visit https://testingwithtestcafe.com.

As more web applications shift to using more interactive user experiences, testing will get more complicated with all the moving parts in place. Since end-to-end testing verifies how a system works as a whole, testers also need to ensure that the communication between external services works as expected.

Usually, these requests happen asynchronously in the background through JavaScript. The client-side portion of the application won't see what's happening until the external service returns a response unless there's some progress indicator programmed in the interface. Even then, there's no guarantee on what's getting returned to the client, making it difficult to perform any assertions on the page.

Intercepting an HTTP request and its response gives you the ability to take control over what happens during asynchronous calls to other servers. Some of the primary uses for intercepting HTTP requests are:

Recording all requests and responses from a remote service

You can open up your preferred browser's developer tools during manual testing and see the different network requests made in the application. This information allows you to see everything related to the request and the response, like the headers sent to the server, and the data returned to the client.

However, during automated testing, you won't have a chance to check the developer tools to see what's going on during the testing process. TestCafe allows you to record any HTTP requests made during the testing process and verify that the data returned from the server is what you expect.

Reliably manage responses from third-party services to ensure repeatable tests

As mentioned, responses coming from an HTTP request are usually out of our control as testers. Some systems provide testing environments, such as a sandbox environment that gives you control over its responses. In most cases, your tests won't have the luxury of specialized testing setups, and you'll have to use a real interface to the service.

Depending on the service where you make a request, you may not always receive the same data every time, making it tricky to create robust validations in your tests. With TestCafe, you can manipulate a response from an HTTP request to create repeatable tests without worrying about the data returned from a service you don't manage.

Creating tests for services that are under development or difficult to replicate

Testing should take place early in the development process to bake in quality from the start. The problem is that when an application is under active development, you'll have rapid changes occurring every day. It's impossible to build a test suite if the service continually changes the way it responds.

Another issue teams may face, particularly small bootstrapped teams, is not having infrastructure available to set up complex systems for testing purposes. Sometimes an organization doesn't have the resources to set up an environment that's not easy to duplicate on a smaller scale.

Fortunately, you don't have to wait for the team to finish developing or setting up a system before you can write tests with TestCafe. By creating a mock that simulates the interfaces you need to verify, you can build tests for environments that aren't readily available for use.

Recording and manipulating external requests have many real-world uses that can save time and help build a reliable and stable test suite. Some examples are:

It's not uncommon for developers to accidentally break an application because of a small change in the response for an HTTP request. You can include assertions to check that those requests and responses work as expected, and raise alerts early during testing if there's a change that can potentially break the application.
If your application performs intensive calculations that take time to execute and complete, it can slow down your test suite or cause your tests to fail due to timeouts. You can bypass these demanding functions with a mock to keep your tests speedy.
Testing doesn't always cover the happy path. You'll also need to test scenarios that don't occur under regular use, such as a network or database error. Typically, you shouldn't be able to trigger one of these errors on your own. However, you can mock an error response from an HTTP request to cover those scenarios during your test run.
If you practice test-driven development (TDD), you don't need to wait for systems to get entirely built before creating new tests. Mocking these systems accelerates the process by allowing you to test complex interfaces before they exist.
Sometimes you won't have access to some services because they contain sensitive information that you don't want to expose during testing. You can bypass any HTTP request made to these servers with a mock to keep your info safe inside of your tests.

Now that you understand the reasons you'd want to intercept HTTP requests in your tests, let's take a look at how you can do this in TestCafe. TestCafe has different hooks to grab HTTP requests, depending on whether you need to record or manipulate the responses.

Logging HTTP requests

First, let's see how you can log any HTTP request that occurs in the application during testing. TestCafe has a class called RequestLogger that checks any HTTP request that occurs while the test runs and records both the request the browser makes and the response it receives from the server.

To log your HTTP requests, you need to import the RequestLogger constructor to your tests and use it to create an instance of the class.

import { RequestLogger } from "testcafe";

const logger = RequestLogger();

The RequestLogger constructor accepts two optional parameters. The first optional parameter is a filter to tell the request logger which HTTP requests it should track. If a filter is not specified, the request logger tracks every HTTP request made during the test. Usually, this behavior isn't beneficial since it logs everything that loads on the page, like images and scripts. You'll want to be more specific about which requests to capture during your test run.

You can filter which HTTP requests you want to record in different ways:

By exact URL

You can pass a string to log any requests sent to a specific URL.

// Only logs requests made to this specific URL.
const logger = RequestLogger("https://teamyap.app/api/posts");

Multiple URLs

If you need to track requests to multiple URLs, you can use an array to specify more than one URL.

// Only logs requests made to the specific URLs
// defined in the array.
const logger = RequestLogger([
    "https://teamyap.app/api/posts",
    "https://teamyap.app/api/conversations"
]);

Using regular expressions

You can use regular expressions to match a URL by a pattern. Using regular expressions is useful when the request URL changes depending on the situation, like an ID associated with a resource in the app.

// Logs requests made to URLs matching this regular expression
// like "https://teamyap.app/api/posts/12345/comments"
const logger = RequestLogger(/\/api\/posts\/(\d+)\/comments/);

Filtering AJAX requests or by request method

If you use a string, an array of strings, or a regular expression, the request logger will record all responses made to any matching URL, regardless of whether it's an asynchronous (AJAX) request or the request method (like GET or POST). If you want to get more specific, you can use an object with the following properties:

url - The URL you want to log requests from.
method - The HTTP method you want to record (GET, POST, PUT, PATCH, or DELETE).
isAjax - A boolean flag to only record asynchronous HTTP requests.

// Only logs asynchronous POST requests made to
// the specific URL.
const logger = RequestLogger({
    url: "https://teamyap.app/api/posts",
    method: "post",
    isAjax: true
});

More control using a predicate function

If you need to fine-tune the request logger even further, you can use a predicate function. The function takes a request parameter that contains different properties that you can use to match the exact HTTP requests you want to filter:

url - The URL you want to log requests from.
method - The HTTP method you want to record.
isAjax - A boolean flag to only record asynchronous HTTP requests.
body - A string containing the body of the request.
headers - An object containing the request headers in key-value form.
userAgent - A string identifying the user agent making the request.

You can run conditional statements on one or more of these properties inside of the predicate function. The request logger will only log requests where all the conditions are true.

// Only logs asynchronous PATCH requests made to
// the specific URL that have a header to indicate
// this is a JSON request.
const logger = RequestLogger(request => {
    return request.url === "https://teamyap.app/api/posts" &&
        request.method === "patch" &&
        request.isAjax &&
        request.headers['Content-Type'] === 'application/json';
});

The second optional parameter for a request logger is an object that lets you configure which information you want the logger to capture. By default, the request logger only returns basic details like the request and response timestamps, the request URL and method, and the response status code. If you need more information like the headers and body, you can set it in this optional object with the following boolean properties:

logRequestHeaders - Lets you specify if you want the request logger to log the headers made by the HTTP request.
logRequestBody - Lets you specify if you want the request logger to log the body of the HTTP request.
stringifyRequestBody - If you set logRequestBody to true, the request body of the request gets recorded as a Node.js Buffer object by default. This option converts the request body to a string. If you set this option to true without setting logRequestBody to true, TestCafe will throw an error.
logResponseHeaders - Lets you specify if you want the request logger to log the headers of the server response.
logResponseBody - Lets you specify if you want the request logger to log the body of the server response.
stringifyResponseBody - If you set logResponseBody to true, the response body of the request gets recorded as a Node.js Buffer object by default. This option converts the response body to a string. If you set this option to true without setting logResponseBody to true, TestCafe will throw an error.

// Logs requests made to the specific URL, including
// the response header and body.
const logger = RequestLogger("https://teamyap.app/api/posts", {
    logResponseHeaders: true,
    logResponseBody: true,
    stringifyResponseBody: true
});

After creating a request logger instance, you can access additional properties and methods that let you manage any captured requests in your tests. These allow you to look at all requests and responses, run assertions, and clear the logger instance.

If you want to view all the requests and responses the logger captures in a test, the requests property returns an array of objects containing details about every intercepted request.

// Somewhere inside of your tests after running a few actions.
logger.requests;

The objects inside of the requests array contain information about the request and the response from the server. Each object will also include the headers and body for the request and response if you set additional options like logRequestHeaders or logResponseBody. Here's an example of a few logged requests:

[
  {
    id: 'yqw_dSpIl',
    testRunId: 'p0P1LuJtg',
    userAgent: 'Chrome 83.0.4103.116 / macOS 10.15.5',
    request: {
      timestamp: 1592980791729,
      url: 'https://teamyap.app/api/posts',
      method: 'get'
    },
    response: {
      statusCode: 200,
      timestamp: 1592980791879,
      headers: [Object],
      body: ''
    }
  },
  {
    id: 'K79qg9fRi',
    testRunId: 'p0P1LuJtg',
    userAgent: 'Chrome 83.0.4103.116 / macOS 10.15.5',
    request: {
      timestamp: 1592980793340,
      url: 'https://teamyap.app/api/posts',
      method: 'post'
    },
    response: {
      statusCode: 201,
      timestamp: 1592980793505,
      headers: [Object],
      body: '{"id":1,"body":"Can someone leave a comment?"}}'
    }
  }
]

You can use the requests property to run assertions, like verifying the number of requests by checking the length of the array. However, the request logger has additional methods - contains and count - to help you validate specific requests, like checking if a request happened or if an exact number of requests occurred during testing.

The contains method lets you run an assertion by specifying a predicate function, similar to the second optional parameter when creating an instance of a request logger. If the conditions of the function match a logged request, it will return true. Otherwise, it returns false.

As an example, let's say you have a logger that intercepted the requests shown when we talked about the requests property. If you want to assert that the logger captured one POST request and received a status code of 201, you can run the following assertion:

await t.expect(logger.contains(request => {
  return request.request.method === "post" &&
      request.response.statusCode === 201;
})).ok()

The count method works similarly to contains. You can set a predicate function to set conditions for matching captured requests in the logger. The function then returns the number of requests that match those conditions.

You can use the count method to perform assertions to verify that the logger captured a specific amount of requests. For instance, using the same example logger requests as above, if you want to confirm that the test intercepted one GET request from the specified URL, you can use the following assertion:

await t.expect(logger.count(request => {
  return request.request.url === "https://teamyap.app/api/posts" &&
        request.request.method === "get"
})).eql(1)

Note

The contains and count methods both return a Promise object. If you use either of these methods in an assertion, TestCafe uses the Smart Assertion Query Mechanism, as discussed in Chapter 10.

Finally, if you need to clear the intercepted requests at any point in your tests, you can use the clear method.

// Somewhere inside of your tests after running a few actions.
logger.clear();

You usually won't need to clear requests captured by the request logger. You can fine-tune which requests to catch when instantiating the request logger object, and methods like contains and count help you refine your assertions. However, some applications make unnecessary and redundant requests that make it challenging to run assertions, so the clear method is useful in those scenarios.

Mocking HTTP requests

Besides logging HTTP requests and verifying what it captures, TestCafe also lets you alter the responses for these requests. The RequestMock class sets up a request mocker object, which you can then use to manipulate requests as needed for your tests. To set up a request mocker, import the RequestMock constructor method to your tests. Once imported, you can use it to create an instance of the class.

import { RequestMock } from "testcafe";

const mock = RequestMock();

However, the object won't do much by itself. Instances of RequestMock have two required methods that you must chain together to create a mock: onRequestTo and respond. Together, these methods form a request mocker that lets you manage how you want any of the application's HTTP requests to respond during testing.

The onRequestTo method allows you to specify which HTTP request you want to intercept to manage its response during the test run. The argument required by the method is the same as the optional filtering argument used when creating a RequestLogger instance:

A string containing the exact URL.
An array of URLs.
A regular expression.
An object that lets you specify the URL, method, and if it's an AJAX request.
A predicate function.

The respond method allows you to specify what you want to use as the response during testing. With this method, you can manage what the browser receives for your testing needs. You can set the body of the response, its HTTP status code, and set the response headers. This method allows you to set three optional arguments.

The first optional argument is the mocked body of the response. You can return a string to simulate an HTML response, an object or array for a JSON response, or a function to customize the response body even further. Most applications use the response body to update the page, so setting the body lets you control how it reacts to a request. If not specified, the mock will return an empty HTML response.

The second optional argument is the numeric HTTP status code of the response. For instance, you can have the request return "200" for a successful response, "404" for a "Page Not Found" response, or "500" to simulate internal server errors. Many asynchronous requests check the status code to indicate if a request was successful or not, so this option may be necessary for your tests. By default, the request mocker will return a status code of "200".

The third optional argument is an object to set custom headers to the response. Some applications use specific headers for it to work correctly. You can use this argument to set those headers as needed. When this option is empty, TestCafe sets a content-type header according to the first argument:

If the mocked body of the response is an array or object, the value of the content-type header will be application/json.
If the mocked body of the response is a string, the value of the content-type header will be text/html; charset=utf-8.

Here are a few different examples of setting up a request mocker:

// Mock any call to the specified URL and return an
// object to simulate a JSON response.
RequestMock()
    .onRequestTo("https://teamyap.app/api/posts")
    .respond([{ id: 1, body: "Can someone leave a comment?" }]);

// Mock any call to URLs that match the regular expression
// pattern, and return an HTML response with a 404 status code.
RequestMock()
    .onRequestTo(/\/api\/posts\/(\d+)\/comments/)
    .respond("<div>Not found</div>", 404);

// Mock any call to URLs that satisfy the conditions of the
// predicate function, and return an empty HTML response with
// a 201 status code and a custom header.
RequestMock()
    .onRequestTo(request => {
        return request.url === "https://teamyap.app/api/posts" &&
            request.method === "post" &&
            request.isAjax;
    })
    .respond(, {
        "Content-Length": "100"
    });

You can also chain multiple onRequestTo and respond methods to a single RequestMock instance to mock more than one request, each with a different response:

// Mocks different URLs and returns a different response
// for each chained method.
RequestMock()
    .onRequestTo({
        url: "https://teamyap.app/api/posts",
        method: "post",
        isAjax: true
    })
    .respond(null, 201)
    .onRequestTo(/\/api\/posts\/(\d+)\/comments/)
    .respond("<div>Not found</div>", 404);

Note

One of the most common uses of setting custom headers when mocking an asynchronous request to an external service is to deal with the Cross-Origin Resource Sharing mechanism, also known as CORS. CORS is a security mechanism handled by your web browser that allows one application in a domain to specify who can access its resources.

On a basic level, the way CORS works is by performing an initial request to the API before our desired API request, known as a preflight request. The preflight request verifies if the originating server where the request is coming from is allowed to make a cross-domain request. If it is, then it returns a successful response with a few headers that tell the browser it's okay to make further API requests.

When you want to mock an API request that goes to a different domain, the browser still goes through its standard CORS check, even with a request mocker object. That means you need to set the appropriate headers in your mocks to make the browser think that the CORS check passed.

The headers that the browser needs to perform a successful CORS check vary, depending on the configuration of the application under test. If you need to set these headers, ask the application developers which headers are required to make cross-origin HTTP requests during testing.

Attaching to tests and fixtures

Creating a request logger or mocker instance won't work on its own. You need to tell TestCafe to use these hooks in a specific test or fixture to log and mock HTTP requests using the objects you create in your test suite.

The fixture and test functions have a method called requestHooks where you can attach either an instance of RequestLogger or RequestMock. Setting up a request logger or mocker instance in this method will automatically set them up for intercepting HTTP requests.

const logger = RequestLogger("https://teamyap.app/api/posts");
const mock = RequestMock()
    .onRequestTo("https://teamyap.app/api/posts")
    .respond(null, 201);

// Attaches the request logger to the fixture and logs
// HTTP requests for all tests under the fixture.
fixture("My test fixture")
    .requestHooks(logger);

// Attaches the request mocker and intercepts and
// mocks HTTP requests for this test only.
test
    .requestHooks(mock)
    ("Test to mock HTTP requests", async t => {
        // Your test code goes here
    });

If you need to attach more than one request hook in a fixture or test, you can pass them in an array or define them as multiple parameters.

// Both of these examples attach the request logger and
// mocker to the fixture.
fixture("My test fixture")
    .requestHooks([logger, mock]);

fixture("My test fixture")
    .requestHooks(logger, mock);

Once a hook is attached to a fixture or test, TestCafe will intercept HTTP requests as defined in the request logger or mocker objects as soon as you run your tests. You can use a request logger object inside the tests to validate any captured requests, and request mockers will automatically simulate any responses matching the hook.

In some cases, you may not want to have a request hook intercepting HTTP requests from the start of your test run. For instance, there might be scenarios where it's okay to allow a request to occur without any interference from a request mocker, and after a few actions, you want to begin catching those requests and mock the response.

You aren't limited to setting a request logger or mocker from the start of your tests. The test controller object has two methods to give you more control over request hooks so you can attach and detach hooks in the middle of a test as needed. The t.addRequestHooks method attaches a request hook at any point in your test, and the t.removeRequestHooks method detaches the hook when you no longer want to intercept the requests.

test("Test to mock HTTP requests", async t => {
    // Run a few actions without intercepting HTTP requests.

    // Attach a request mocker to begin intercepting requests.
    await t.addRequestHooks(mock);

    // Run more actions

    // Detach the mocker to stop intercepting requests.
    await t.removeRequestHooks(mock);
});

If you found this article useful, you can pre-order the End-to-End Testing with TestCafe book at https://testingwithtestcafe.com and receive $10 off the book's original price when pre-ordering before the expected release date (on or before July 15, 2020).

If you sign up to the mailing list on the book's website, you'll receive the first three chapters of the book for free. In addition to the book sample, you'll get an exclusive discount to pre-order the book.

End-to-End Testing with TestCafe Book Excerpt: Reporters

Dennis Martinez — Tue, 23 Jun 2020 11:00:00 +0000

Earlier in this book, way back in Chapter 2, we briefly touched upon reporters in TestCafe. Reporters are used for displaying the results of your test run using different kinds of formats. So far, you've only seen the default reporter when running your tests, but TestCafe has other reporters available to use if you need to show your test results in a different format.

TestCafe ships with five different reporters you can begin to use immediately. Each reporter displays the outcome of your test execution in various ways, giving you some options on how you need to manage the results. You can use them for showing the results of each test as they run, including detailed error information if an assertion fails. You can also use the output of a reporter for an external service to process.

TestCafe's built-in reporters

Below are the built-in reporters included with TestCafe, created and maintained by the TestCafe team.

spec

The spec reporter is the default used by TestCafe. This reporter groups your tests by fixture and displays each test scenario's full name under its fixture. At the end of the test run, it shows the number of tests executed, the result of each test (passed, skipped, or failed), and the total time the test execution took. It does not display how long each test took to execute.

Configuration examples

The following examples use the spec reporter when running your tests using the configured browser and test files.

Command line setting:

testcafe chrome *_test.js -r spec
testcafe chrome *_test.js --reporter spec

Configuration file setting:

{
  "reporter": "spec"
}

list

The list reporter displays both the fixture name and test scenario name as a list, one per line. The output is similar to the spec reporter, with the main difference being that it doesn't group the tests by its fixture. At the end of the test run, it shows the number of tests executed, the result of each test (passed, skipped, or failed), and the total time the test execution took. It does not display how long each test took to execute.

Configuration examples

The following examples use the list reporter when running your tests using the configured browser and test files.

Command line setting:

testcafe chrome *_test.js -r list
testcafe chrome *_test.js --reporter list

Configuration file setting:

{
  "reporter": "list"
}

minimal

The minimal reporter only uses a few symbols to display the result of your tests, without showing any fixture or test scenario names. Passing tests are shown as a dot, failing tests with an exclamation point, and skipped tests with a dash. At the end of the test run, it only shows the number of tests executed and the result of each test (passed, skipped, or failed). It does not display how long each test took to run, or the total time the test execution took.

Configuration examples

The following examples use the minimal reporter when running your tests using the configured browser and test files.

Command line setting:

testcafe chrome *_test.js -r minimal
testcafe chrome *_test.js --reporter minimal

Configuration file setting:

{
  "reporter": "minimal"
}

xunit

The xunit reporter returns the results of your test run in XML, using the standard xUnit format (more widely known as the JUnit format). This reporter is useful when executing your tests in a continuous integration system such as Jenkins or CircleCI. Usually, continuous integration systems can collect, process, and format these XML files into a readable document as part of your test run history.

<?xml version="1.0" encoding="UTF-8" ?>
<testsuite name="TestCafe Tests: Chrome 83.0.4103.106 / macOS 10.15.5" tests="9" failures="0" skipped="0" errors="0" time="34.554" timestamp="Thu, 18 Jun 2020 05:44:13 GMT" >
  <testcase classname="TeamYap Administrator sections" name="TeamYap admin can see admin sections on the sidebar" time="5.082">
  </testcase>
  <testcase classname="TeamYap Administrator sections" name="TeamYap user can&#39;t see admin sections on the sidebar" time="4.138">
  </testcase>
  <testcase classname="TeamYap Administrator sections" name="TeamYap admin can access organization settings" time="1.813">
  </testcase>
  <testcase classname="TeamYap Administrator sections" name="TeamYap admin can add and sort profile questions" time="6.337">
  </testcase>
  <testcase classname="TeamYap Feed" name="Logged-in user can create new feed post" time="3.091">
  </testcase>
  <testcase classname="TeamYap Feed" name="Logged-in user can comment on feed post" time="4.045">
  </testcase>
  <testcase classname="TeamYap Login" name="User with valid account can log in" time="2.899">
  </testcase>
  <testcase classname="TeamYap Responsive Test" name="The application hides the sidebar when resizing viewport" time="2.919">
  </testcase>
  <testcase classname="TeamYap Settings" name="User can update and delete their profile picture" time="4.158">
  </testcase>
</testsuite>

Configuration examples

The following examples use the xunit reporter when running your tests using the configured browser and test files.

Command line setting:

testcafe chrome *_test.js -r xunit
testcafe chrome *_test.js --reporter xunit

Configuration file setting:

{
  "reporter": "xunit"
}

json

The json reporter returns the results of your test run in a simple JSON format. Like the xunit reporter, this reporter works well when you want to process your test results in another service. Since most build systems accept the xUnit / JUnit XML format as a standard, you may not need these reporters for those kinds of environments. However, for custom services, JSON is a flexible alternative. It also contains additional information about your test run, like metadata and screenshot paths.

{
  "startTime": "2020-06-18T05:47:40.385Z",
  "endTime": "2020-06-18T05:48:15.520Z",
  "userAgents": [
    "Chrome 83.0.4103.106 / macOS 10.15.5"
  ],
  "passed": 9,
  "total": 9,
  "skipped": 0,
  "fixtures": [
    {
      "name": "TeamYap Login",
      "path": "/home/dennmart/src/end_to_end_testing_with_testcafe/login_test.js",
      "meta": {},
      "tests": [
        {
          "name": "User with valid account can log in",
          "meta": {},
          "errs": [],
          "durationMs": 3331,
          "screenshotPath": null,
          "skipped": false
        }
      ]
    },
    {
      "name": "TeamYap Feed",
      "path": "/home/testcafe/end_to_end_testing_with_testcafe/feed_test.js",
      "meta": {},
      "tests": [
        {
          "name": "Logged-in user can create new feed post",
          "meta": {},
          "errs": [],
          "durationMs": 3096,
          "screenshotPath": null,
          "skipped": false
        },
        {
          "name": "Logged-in user can comment on feed post",
          "meta": {},
          "errs": [],
          "durationMs": 4504,
          "screenshotPath": null,
          "skipped": false
        }
      ]
    },

    // Shortened for brevity...

  ],
  "warnings": []
}

Configuration examples

The following examples use the json reporter when running your tests using the configured browser and test files.

Command line setting:

testcafe chrome *_test.js -r json
testcafe chrome *_test.js --reporter json

Configuration file setting:

{
  "reporter": "json"
}

Other ways of using reporters

When running your tests with any reporter, it will output its results in the terminal as standard output (stdout). During your test suite development, you'll want to see the results as your tests execute, so using the standard output in your terminal works well. But if you need to process the results in a separate service, you'll usually need to store them in a file. TestCafe allows you to save the test results from any reporter to a file by specifying a file path along with the configured reporter.

In the command line, you can add the file path after the reporter type, separated by a colon. The following command line example shows how to save the results of a test run in a JSON file under the reports sub-directory in your current project.

testcafe chrome *_test.js -r json:reports/test_results.json

If you want to configure the same setting in your .testcaferc.json configuration file, the reporter key needs to be an object with a name and output property for setting the reporter type and file path, respectively.

{
  "reporter": {
    "name": "json",
    "output": "reports/test_results.json"
  }
}

Also, you're not limited to using a single reporter when executing your tests. You can combine multiple reporters for a single test run. The only issue to keep in mind when using more than one reporter is that only one can output the results in the terminal. The rest of the reporters used in the test run need to save the results as a file. If you attempt to output the results of more than one reporter to stdout, TestCafe will throw an error.

In the command line, you can specify multiple reporters by separating each reporter type with a comma, keeping in mind that only one reporter can display its results in the terminal. The following command line example shows how to use the minimal and xunit reporters in a single test run.

testcafe chrome *_test.js -r minimal,xunit:reports/results.xml

If you use the configuration file to set up multiple reporters, the reporter key needs to be an array of objects, each using the name and output properties as required.

{
  "reporter": [
    {
      "name": "minimal"
    },
    {
      "name": "xunit",
      "output": "reports/results.xml"
    }
  ]
}

If these reporters don't serve your needs or you want to extend TestCafe's functionality even further, you can find reporters created by the TestCafe community by searching for the term testcafe-reporter at https://www.npmjs.com/. You'll discover reporters that can change the way the test results look or submit the results to external test management tools like TestRail and Jira for managing your test cases automatically.

If you found this article useful, visit https://testingwithtestcafe.com. If you sign up to the mailing list, you'll receive the first three chapters of the book for free. You'll also receive exclusive updates and be among the first to know when the book is available for purchase for a discount.