DEV Community: Laura Cressman

3 End-to-End Testing Pitfalls to Avoid

Laura Cressman — Wed, 04 Aug 2021 16:19:55 +0000

After being in the quality assurance (QA) space for several years, we've had a front row seat to how companies big and small do QA. The main area of struggle we see is with setting up automated end-to-end tests. End-to-end tests go through your application like a user would, to check that critical workflows like sign up aren't broken. These tests can be difficult to create and integrate into your process.

If you don't have automated tests yet, getting started can feel daunting. In this guide we explore three common pitfalls companies face, and provide solutions for how to avoid them.

Pitfall #1: Trying to run before you walk

We've had this conversation more than a few times now:

Company: "We need to run hundreds of tests on every browser and device size on every commit."

Us: "Cool! How many tests do you have now?"

Company: "...zero."

While it's great to aspire to running hundreds of tests on every commit, these lofty goals can paralyze companies from getting started in the first place.

It's easy to get hung up on a long list of "must haves" like testing different versions of Chrome or creating preview environments for every commit. When starting from scratch, the only "must have" should be running a single test however you can. Otherwise, you risk constantly delaying your testing efforts while searching for the perfect solution.

Avoid putting undue pressure on your team to build the perfect QA process upfront. Instead, start with the smallest attainable goal and go from there.

If you don't have any tests yet, create one basic test and run it only on Chrome. If you don't have a way to test every commit, run tests once a day on staging. You can always improve your process later, without sacrificing coverage today.

In short, don't beat yourself up for not having your dream QA process in place. Every company has to start somewhere. The most important thing is to start at all.

Pitfall #2: Using the wrong (or no) success metrics

There are many ways to do QA, such as having developers write tests or using an outsourced team. Regardless of how it's implemented, you should measure the success of your QA efforts.

Ultimately, QA is about providing your customers with a quality product. Automated tests check that user flows (like adding an item to the cart or logging in) aren't riddled with bugs. Therefore, a "north star" QA metric should tie into your user experience.

In practice, many companies don't know whether they're doing a good job with QA. The result is treating QA like a box to be checked, rather than a way to earn customer love and gain a competitive advantage.

We think the single best QA metric is user workflow coverage. This means the share of common user actions (like creating a comment or changing a password) that are tested.

While it's impossible to get to 100% coverage (there are infinite ways to use a site), prioritizing test cases based on what your users are actually doing keeps QA aligned with your larger business goals.

Today, few companies have the tooling in place to understand the most common and most buggy user journeys. One place to start is by reviewing user data from services like Hotjar and FullStory. You can also use QA Wolf for this, which converts your analytics into test cases.

Example of user interactions recorded with QA Wolf

Pitfall #3: Not treating automated testing as an engineering problem

Automated QA can be as simple as running one test on staging. Still, it's critical to keep your team's long-term success in mind from day one.

It can be tempting to do something - anything - to get QA off your plate. A common trap is to delegate automating QA to non-technical team members like interns or customer success. While these team members can contribute to an existing process, they typically don't have the skills to create a good process themselves.

There are two basic parts to standing up automated QA: creating stable tests and integrating them into your process. Both involve specific engineering knowledge. Just as you wouldn't rely on a product manager to shape best practices for your React components, you should make sure to give QA the engineering attention it deserves.

One manifestation of this is the rise of "no code" testing tools. Though the message is appealing (I don't have to think about QA - my intern can do it!), we have yet to see these tools work well in practice. Every site has its own quirks that only the flexibility of code can automate. No code has its place elsewhere, but when it comes to testing: just say no to no code.

If you don't have the time or skills to set up a process yourself, find someone (internal or external) with the engineering chops to build that solid foundation. Then you can invite more of your team to take QA to the next level.

Conclusion

We hope that this post has made getting started with testing seem less scary. All companies, including yours, are capable of taking that first step. And when you do, your customers will thank you. :)

At QA Wolf, we manage QA for you by converting user analytics into automated tests. If you'd like to learn more, or just want to talk testing, you can reach me at laura@qawolf.com. 🐺

Why We Open Sourced Our Entire Product

Laura Cressman — Wed, 13 Jan 2021 23:29:59 +0000

A little over one year ago, we open sourced the first version of QA Wolf, a tool to help developers create browser tests.

Our goal with QA Wolf was to help developers ship confidently and improve the quality of web-based software. Since then, we've shipped over a hundred releases based on feedback from our users. ❤️

Despite these improvements, we learned that QA Wolf was still too difficult to use for many developers (including ourselves). They still needed to install a Node.js package, set up CI, and become a detective to understand why their test failed. 🕵️

These learnings inspired us to build QA Wolf 2.0: an easy way to create, run, and debug tests directly from the browser. And we had an important decision to make...

Should QA Wolf 2.0 be open source?

We kicked around various ideas, including open sourcing parts of the codebase while keeping others closed source. Then we asked ourselves, "what if we just open source all of it?"

It sounded terrifying at first. Our minds dwelled on the possibility of someone ripping off our entire product.

After the initial wave of anxiety subsided, the arguments for open sourcing came clearer into focus. We talked it over with our investors, and they agreed: the pros vastly outweighed the cons.

So today we are open sourcing QA Wolf 2.0. We are excited for you to check it out, and look forward to your feedback!

Our code reviewer Finn

Here is why we open sourced everything:

It helps build a community

QA Wolf is built for developers who want an easy way to set up browser tests. Open sourcing allows us to meet our audience where they are, and invites the feedback we need to build a great experience.

Open sourcing helps gain developers' trust. They can use QA Wolf without worrying about vendor lock-in. They can peek at the source code, and edit it for their own use case.

Being open source also makes product improvements public, building confidence about the future of the project.

We will move faster and build a better product

Building in the open is scary. Someone might discover a security hole, or make fun of that gross code we wrote.

We've found this to be a powerful forcing function. Having open issues motivates us to fix them faster, and we feel embarrassed if we don't show consistent progress. We also write better code knowing that anyone can see it.

Finally, we will be grateful for any contributions we get from being open source. 🙏

Open core is a headache

We considered adopting an "open core" model, where only some parts of the codebase would be publicly available. As we evaluated this option, we realized that it created more problems than it solved.

Open core software is harder to contribute to and maintain. We would have to manage separate packages for various parts of the codebase, and no one else could contribute to the closed source code.

On top of deployment headaches, we would need to constantly evaluate whether new code should be open or closed source. Having a single open source project relieves us and future developers from this burden. It also allows our community to have maximum visibility into our codebase.

It just feels right

"Be open" is one of our values, and open sourcing feels like the right thing to do.

When we reflect on the good things that have happened to us this year, all of them came from being an open source company. We got invaluable feedback from the very first iteration of QA Wolf. We were also able to raise a pre-seed round (more to come soon in an upcoming post!).

After committing to making our source code public, we had to answer one more question.

What license do we use?

We want to clarify one thing upfront: we're still not sure we made the right decision. We did what we thought was best at the time to keep moving forward. In the future, our license may change based on your feedback.

After studying open source licenses, we narrowed down our options:

Use a permissive license like Apache-2.0. This license allows people to use your code however they want, including to directly compete with your service.
Use the Business Source License (BSL). This license allows people to use your code however they want, except to directly compete with your service without buying a license. Over the past few years, companies like Sentry and CockroachDB have adopted this license.

We decided to pursue both options.

We've licensed most folders (CLI and helpers, code generation and test runner) under Apache-2.0. QA Wolf 2.0 would not be possible without similarly licensed open source projects like the Playwright test framework. It's important that we give back to the community that has given so much to us.

The remainder of our product is licensed under BSL. When evaluating this option, we found that Sentry's goals for licensing resonated with us:

"Anyone should be able to run [the service] for themselves or their business"
"No difference between our cloud service and our open source product (no open core model)"
"Minimal limitations on usage of code, as free as possible"
"Protection from other companies selling our work"

Our goal is to build a sustainable business around our product, and we need to have the resources to do that. BSL allows us to share our source code while giving us some protection over our work.

In the future, we may license the entirety of QA Wolf under Apache-2.0. We're starting here to ensure that we get more rather than less permissive over time.

Conclusion

Thanks for making it this far. We hope you'll try QA Wolf 2.0 and let us know how it goes. 🐺

If you'd like to get in touch, we'd love to hear from you at hello@qawolf.com.