DEV Community: Lauren Christianson

Impact-Driven Automation Testing For Ecommerce Websites

Lauren Christianson — Tue, 06 Dec 2022 16:09:08 +0000

Every piece of software is unique: different tech stacks, varying coding styles, wide arrays of use cases, and distinct ways in which the software itself will be exercised. Due to this, testing methodologies, be it manual or automated, are also unique.

While this certainly keeps our jobs fresh and exciting, it presents a problem that is left to the expertise of the developers and QA engineers. A plethora of questions need to be answered before we begin designing an automation approach; including:

What approach should we follow in selecting an automation framework?
Which CI/CD tools should be used for accelerating automation testing?
What should be the frequency at which automated tests need to run?

Automation engineers must maintain a robust tool belt in order to execute the respective automation tasks effectively. However, given the myriad of CI/CD tools and frameworks to choose from, it can be challenging to keep our skills up-to-date on top of our day-to-day activities at work.

After working on a variety of applications in completely different industries (online gaming, financial services, advertising, travel, video streaming, eCommerce, and currently commercial insurance), I’ve identified working patterns and necessary components to succeed in QA and test automation in these domains.

If you work in the eCommerce industry, I suggest bookmarking this article as the tips mentioned below can help you in your day-to-day work.

There are intricacies in every domain that, if known, can drastically improve an automation engineer’s efficacy on a team. Foundational knowledge of QA and test automation best practices paired with a specialized, industry-specific toolset will have a significant positive impact on a team.

US eCommerce sales grew more than 30% in 2020, which is projecting a strong shift in buying patterns of consumers. As the world shifts from in-person shopping to online shopping, eCommerce giants like Target, Amazon, Walmart, and Best Buy are increasing their engineering efforts to enhance their platforms further.

With this meteoric shift comes an increased need for strong QA practices since these businesses are now seeing a larger share of their revenue come from online fulfillment.

Here are QA & test automation best practices that are derived from my QA experience with companies like Best Buy.

Utilize data-driven testing

When you hear the term data-driven in the context of testing, it commonly means setting up test scenarios so that the same scenario is run ‘X’ number of times, where X is the amount of data permutations that exist for the given scenario. For example: Verify that a customer can add X products to the shopping cart.

As an automation engineer, I may opt to implement automation testing using BDD. TDD vs BDD is a quest for some developers. The choice of an appropriate BDD framework can depend on the team’s prior experience with BDD tools. I personally prefer the Cucumber test framework since it is a popular, easy to use framework. For the test scenario mentioned above, I could do something like this (simplified example):

Scenario Outline: Add to Cart
  Given a logged in user
  When I add <items> to my cart
  Then I should see <items> in my cart

  Examples:
    | items |
    |     1 |
    |     5 | 
    |    25 |

This allows me to easily add thorough test coverage for common workflows in the application. For example, I would likely use an API endpoint to add items to my cart, then call another endpoint to get all items in my cart once they’ve been added. These tests could also be run through the UI, resulting in lengthy runtimes and more test flakiness.

Let’s look at my second definition of data-driven testing: obtaining real, production user data and statistics to drive test scenario design. In the example above, I arbitrarily chose three different test scenarios: adding a single item to the shopping cart, adding 5 items, and adding 25 items. Test scenarios should, ideally, never be arbitrarily chosen!

When I join a team, I always ask for production reports — this, in my opinion, is some of the most valuable data you can leverage for your test suite. Based on this information, you can set up test scenarios and have greater confidence that your most common use cases are working as expected.

This data is a goldmine that can be extensively used for setting up the baseline test cases. Once the data is available to the entire team, they have visibility into the most critical pieces of the application’s functionality. The type of production data you use depends on the system under test, but here are a few examples of the most prominent data used in the testing of e-commerce websites:

Common types of order fulfillment — delivery, in-store pickup, curbside pickup, locker pickup
Common size of orders — single item, 2–10 items, 10+ items, bulk orders
Insights into the most commonly ordered items

Depending on the user metrics derived from the above data, we can create our baseline test scenarios, accounting for a large amount of our application’s real use cases.

*Read *—12 Mistakes In An E-commerce Website That Affect Online Sales

Leverage Cloud-based testing solutions

A majority of eCommerce sites can be broken into the following core functionalities:

Product search
Addition of product to shopping cart
Order summary and modification
Check out and payments

It is common for separate development teams to own these features, which unfortunately creates silos within the organization. Intertwining different pieces of the system with each other is one of the major uphill tasks when it comes to eCommerce development and testing.

If you are working on product pages, you’re reliant on the search development teams and the inventory development teams. In distributed eCommerce platforms, visibility into the state of applications in the test system will save teams a significant amount of time during testing activities.

With cloud-based automated cross browser testing solutions like LambdaTest, teams have increased visibility into the state of upstream and downstream apps in the test system. It enables testing of the different site functionalities across browsers and platforms at a much faster pace! It leverages the cloud to its maximum potential.

LambdaTest Automation Dashboard

Another great benefit of cloud-based tools is the transparency into other teams’ test suites. eCommerce organizations are large and require cross-team collaboration to ensure company-wide success. Tools like LambdaTest enable teams to have transparent testing efforts, enabling teams to understand the rules and requirements of applications outside of their own.

You can also refer to the below video tutorial on how to perform real-time browser testing on LambdaTest.

Check this out: How To Debug Websites Using Developer Tools for safari

Think like a user, but test like An Engineer

It’s likely that you are working with highly distributed and complex systems when working in the eCommerce industry. As a result, if you were to automate your tests to mimic a real user’s path through the system, your tests would be huge, slow, and prone to failures caused by upstream dependencies.

If your system allows it, utilize writing directly to a database or posting to a web service endpoint to eliminate time-consuming test data setup.

An example of where this can be useful:

Testing Cart functionality: My high-level suggestion here would be to do thorough API automation testing. Here are some of the relevant endpoints:

POST: /addToCart/{sku} → {sku} is associated to an item and this call adds the item to the user’s cart
GET: /cart → All items currently in the user’s cart are returned
DELETE: /removeFromCart/{sku} → {sku} is removed from the cart

Your test cases can quickly add items to the cart with the POST call, retrieve all items in the cart by calling the /cart endpoint, then use the DELETE call to clean up the test. Using the earlier example, a test like this would probably run in a matter of seconds. However, the test could take several minutes and be vulnerable to upstream dependencies in the Search application and the Item Details application. This is especially true if you handled the data setup and test execution at the UI level.

This approach to test data setup can be used within your UI tests and functional areas in an eCommerce platform.

Check this out: Cross Browser Testing Cloud- Browser & app testing cloud to perform both exploratory and automated testing web across 3000+ different browsers, real devices and operating systems.

Leverage feature flags and isolate new features in production

eCommerce platforms require 24/7 uptime and availability. In Agile software development, frequent releases are recommended to reduce the overall risk introduced by not allowing changes to pile up over long periods of time. I would recommend embracing shift left testing for eCommerce website testing so that development and testing can move hand-in-hand.

// Detect dark theme var iframe = document.getElementById('tweet-1040496791161131008-379'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1040496791161131008&theme=dark" }

While it’s important to have either a roll-back or fall-forward strategy in case an issue is introduced in production, there are ways to isolate features so that a new piece of functionality can also be tested in production before rolling it out to the entire system of users.

Here are some of the ways in which you can leverage flags in testing:

Enabled/disabled feature flags: Akin to turning a lightswitch on and off, these flags are useful for enabling and disabling features for short periods of time. This can be used for production testing during a low-traffic time period.
User-based flags: You can really narrow down the visibility of a feature in production using these flags.

In general, don’t be afraid to test in production (safely!), if it is possible to do so. Features can be easily isolated to your own production user account (and/or others on your team) to give you higher confidence that the application is functioning as expected in a true production environment. This is particularly useful in eCommerce, as it is hard to end-to-end test the full user flow in test environments considering the expansive size of the e-commerce platform.

Read — Top Automation Testing Trends To Look Out In 2021

Run Selenium, Cypress, and Appium tests on LambdaTest to scale with the demand of your website and web apps.

Check this out: How To Debug Websites Using Dev tools in Safari

Wrapping It Up

While major eCommerce platforms — Target, Walmart, Amazon, Best Buy, and more — are competitors trying to differentiate from one another, remember that their systems are generally the same at a high level. A customer searches for a product, adds it to their cart, checks out, and can view/modify their order.

Of course, underlying systems that power this simple-sounding user flow are large and significantly complex inventory systems, supply chain systems, integrations with retail POS systems, and integrations with marketplaces like eBay. Thankfully, the testing we can do at the database, web service, and UI application layers cover the user-facing functionality that customers interact with daily.

Enabling Agile Software Testing With Automation

Lauren Christianson — Tue, 06 Dec 2022 15:20:46 +0000

Over the years, I’ve worked with teams at varying levels of Agile maturity. Some, completely new to Agile, others ‘doing’ Agile (talking the talk), and very few being Agile (walking the walk). Regardless of a team’s Agile maturity, there is almost always one commonality: a struggle for QA to keep up with the lightning-fast pace of Agile software development.

When teams are learning how to operate with agility, they see how fast things move. This is a great benefit of Agile, but it comes with some assumptions that, if ignored, can cause a team to struggle with frequently delivered iterations of their product.

I consider automated testing to be an unspoken necessity for a team to truly experience the benefits of agility. A team can only go as fast as the tools, pipelines, and processes that enable it.

“Automated testing decreases regression time and increases confidence in each release.”

If you’re new to Selenium and wondering what it is then we recommend checking out our guide — What is Selenium?

Read: Why Selenium Automation Testing In Production Is Pivotal For Your Next Release?

Let’s talk about approaches that teams can take to level up their automated testing.

The realm of test automation tools and frameworks can feel overwhelming. Every team has different needs, so it can be a daunting task to build up an automation solution from scratch.

Even teams with an established automation suite will constantly find new frameworks and automation trends to follow in the space (all-in-one automation solutions, visual UI regression tools, and Selenium alternatives — to name a few!).

That said, there are generally similar approaches to leveling up a team’s automation, depending on the team’s current state.

In true Agile fashion, we’ll identify some team personas that necessitate uniquely tailored solutions to becoming automation superstars.

Persona 1: Team has no automated testing in place; all test activities are performed with a manual approach.
Persona 2: Team has automated testing in place, but the automation is not benefiting the team.
Persona 3: Team has automated testing in place; the automation provides value and acts as an enabler for the team. This looks different for everyone!

Now, let’s deep dive into each of these types of teams and identify how to help them succeed in their automation journey:

Persona 1: How To Get Started If Your Team Has No Automation In Place

Identify the skill set of the team:

QA teams have different make-ups. Some are 100% manual testers. Some are 100% developers, maybe with zero testing experience. Others have a combination of manual and technical testing experience.

When working with an existing team, it’s important to know your team’s background and skill sets so that their strengths & skills can be leveraged in the best possible manner.

If an existing team’s QA members are solely manual testers with no technical background, they’ll need a different introduction to automation than developers or SDETs (Software Development Engineers In Test).

They might be interested in coding and should therefore be given proper programming training before delving into automation. This will ensure their long-term success.

On the other hand, a manual tester may not want to code but can still contribute to automated testing activities. In this case, the team may want to consider an automation implementation using BDD (Behavior Driven Development) frameworks like Cucumber so that non-technical team members can chip in and help write test scenarios.

Run Selenium, Cypress, and Appium tests on LambdaTest to scale with the demand of your website and web apps.

Check this out: Getting Started With Playwright automation Framework

Automation should be used to boost efficiency in manual testing:

Examples include using Postman’s test scripts while manually testing APIs, scalable cross browser testing tools like LambdaTest, creating JavaScript bookmarklets for easier form-filling, or custom bashing scripts to generate test data. Automation engineers and developers can utilize automation in creative ways to speed up manual testing!

For a team with a mix of manual and technical testers, I highly recommend using the philosophy of paired programming and partnering up with people on the team that have complementary skills.

If you have someone with a lot of domain knowledge who doesn’t have a coding background and another person with strong technical skills but less subject matter expertise, these two could be an extremely powerful pair!

In any case, the team must take inventory of their skills when laying the foundation for their automation solution. Once this is established, there are other factors to consider before jumping into automation.

There are too many to list (and might vary from team to team or project to project), but here are a few common ones:

Tech stack: If the technical team members solely program in JavaScript, you likely wouldn’t choose a Python-specific automation framework.
Technical skill sets: If non-technical team members want to contribute to automated testing, consider a Behavior-Driven Development (BDD) automation solution.
Application functionality: Mobile app? Web app? API? There are frameworks for everything!
Understand what you’re automating: Lay out test scenarios, identify critical functionality of the application, user flows — this could help you choose the best-suited test automation tools. For example, if your app has graphs, charts, complex diagrams, etc., that are critical pieces of functionality, you might consider a solution that supports visual UI testing.

The key here is to dedicate time to planning the solution before jumping in. Involving developers and technical leads is beneficial.

Once the team has laid out the groundwork, you’re ready to start automating. I highly recommend becoming familiar with the Test Automation Pyramid concept before you start. I’ll go over some anti-patterns and things not to do below.

Check this out: Getting Started With Playwright automated testing Framework

Persona 2: Course-correcting To Get Your Automated Testing On The Right Path

Unfortunately, it’s very common for teams to throw together an automated test solution without taking a planned approach to it from the start. As a result, testers end up with unreliable tests in an unmaintainable suite.

In my own experience, I have joined teams that have two or more years invested into an automation framework, only to discover it is not scalable for their needs and scrap the whole thing for a new one.

While I am a proponent of switching out tool sets for new, shiny ones that enable you to work more efficiently, no one wants to discard years of hard work.

If any of these pain points sound familiar, it might be time to look at how you can improve your automation.

Tests take a long time to execute (several hours or longer).
Your regression tests are being run on testers’ local machines.
Tests fail a lot for unknown reasons.
There is no easy way to run a subset of the full test suite.
Testers are spending hours upon hours updating tests that break when there are UI changes in the application.
Your regression suite has hundreds of UI, end-to-end, and user story-based tests.
Test runs are unreliable and have false positives/false failures.

I’ve experienced each of these at least once! While all of these can cause a lot of pain for a tester, there are solutions. For Agile teams, everything moves fast. The most common problem that I see teams face, however, is time. How do you balance automation work with all the other day-to-day activities going on?

Prove the potential of automation:

If QA is having a hard time getting Product Owners/BAs/leadership on board allocating time to automate, show them a proof-of-concept or demo of how much time can be saved with automating.

Automated testing creates more time for QAs to test how only humans can provide faster feedback to developers and save time during regression. All of this equates to delivering features faster.

Ensure there is visibility into the automated tests:

The entire development team (QA, Devs, Product Owners) should feel some level of ownership in the quality of the product — including automated testing.

I have found it extremely valuable to get the Product Owner’s inputs on critical user scenarios, which I then turned into automated smoke testing run against every pull request, giving the whole team higher confidence that the most important pieces of functionality were always working.

// Detect dark theme var iframe = document.getElementById('tweet-1408124555646803977-78'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1408124555646803977&theme=dark" }

When the team knows what tests are running, they’ll chime in during new feature development and make sure there’s time to automate new tests! Automation and testing effort should also be included in the ‘definition of done’ for a story. Another great way to keep an eye on the automation is to demo framework enhancements or test runs in Sprint Reviews!

Make it useful:

I have seen teams that dedicate hundreds of hours to a huge automation suite, only to run it once a month. Automation is a time investment, and it only pays off if it’s utilized fruitfully.

Ideally, you have automated tests that run against every pull request — even if that is a small subset of smoke tests. Using CI/CD tools enables you to schedule nightly regression runs, trigger executions based on code merge, and more.

Persona 3: Your Automation Is Awesome — How You Can Make It Even Better

Even the most impressive test suites can always be improved! On teams with a mature automation setup, here are a few things I like to tackle with them:

Cleaning up spec files and focusing on improving the maintainability of the entire automation framework.
Identifying any areas for flakiness and making them more reliable — every test suite has some weak spots!
Looking for opportunities to move tests closer to the application code by utilizing API calls or database interactions to take steps out of brittle UI tests.
If the QA team comprises dedicated automation engineers and dedicated manual testers, blend these roles so that there are no bottlenecks in the testing process.
Running tests in parallel (be cautious with shared test environments, test accounts, or test data) using a tool like LambdaTest to speed up regression execution exponentially.

Continuously dedicate time to automation:

Do not let automation become ‘set and forget.’ Automation can quickly become outdated and obsolete if the team is not actively maintaining it. When developing new features, look closely at how to ensure test coverage while keeping the maintainability of the test suite intact.

Shift the testing left:

Run automated tests against every PR, after every merge request, and against every release build. The earlier in the development cycle that an issue is caught, the less costly it is to fix.

Read How “Shift-Left” Testing Can Help Your Product Quality?

Always explore new things:

Have you heard of Contract testing? What about Visual UI testing? Automation tools and frameworks are evolving and bringing us innovative ways to speed up testing as we’ve never seen before. It’s up to us, as engineers, to use them to solve problems for the betterment of our teams.

Check this out: Getting Started With Playwright browser testing Framework

Wrapping it Up!

On Agile teams, it is crucial to have some form of automated testing in place. Without it, testing inevitably becomes a bottleneck to delivering features faster. Manual testing will always have its place in ensuring that software functions well for humans. No tool or software will ever perfectly replace or replicate the human eye and the critical thinking that is such a valuable component of testers’ mentalities. I challenge you to boundary test the current limits of your testing and take your software’s quality assurance to the next level. Your team will thank you for it!