DEV Community: Mike Ralduxin

How to Cut Regression Testing from Weeks to Hours Without Automating Everything

Mike Ralduxin — Mon, 31 Aug 2026 15:23:25 +0000

A healthcare platform I worked with needed two and a half weeks to complete a regression pass. Smoke testing alone took seven days. The obvious recommendation — the one everyone reaches for — was "automate everything." It would also have been the wrong place to start.

Here's the trap. If you point automation at a bloated, outdated, UI-heavy suite, you don't get fast regression. You get an oversized, expensive, automated version of the same slow process, plus a maintenance bill that grows every sprint. A faster test runner does not create a faster regression process — it just runs the wrong tests more quickly.

The teams that actually go from weeks to hours don't start with the runner. They start by asking where the time is really going — and most of it is not in test execution. On that healthcare platform, we eventually got regression down from 2.5 weeks to a single day while raising coverage from 50% to 90%. Automation was part of it. It was nowhere near the whole story.

This is a guide to the whole story: what to cut, where to test, when to run, and how to investigate failures — so that automation removes specific bottlenecks instead of freezing an inefficient manual suite into code.

Why regression testing takes weeks

The first thing to fix is a measurement mistake. Most teams track test execution time and quietly assume it's the same thing as regression lead time. It isn't, and the gap between them is where your weeks disappear.

Regression lead time = preparation + environment setup + queue time + execution + failure investigation + reruns + reporting

Execution is one term in that sum, and often not the biggest. You can halve your runtime and still ship on the same day if the other six terms are untouched. Before you optimize anything, break your lead time into these buckets and see which one actually hurts. In my experience it's rarely the one people complain about.

Here's where the time usually leaks.

The suite grows but never shrinks

Every team is good at adding tests. Almost none are disciplined about removing them. So the suite accretes scenarios that are duplicated across specs, outdated relative to the current product, disconnected from any real risk, already covered at a lower level, or so vaguely written that a pass or fail tells you nothing. That's not coverage. It's ballast — and you re-run it on every release, forever, until someone decides to look.

Every test runs for every change

A one-line copy edit on a marketing page should not trigger the same regression scope as a change to the payments flow, the authentication layer, or a database migration. When it does, you're paying the worst-case validation cost on every release regardless of actual risk. Teams need more than one suite, sized to different levels of risk — not a single monolith that runs identically no matter what changed.

Too many checks live at the UI level

UI end-to-end tests are the slowest, most fragile, and most expensive-to-maintain layer you have. They break when a selector shifts, they need a running environment and real data, and they retry slowly. That doesn't make UI tests useless — but the UI is the wrong place to verify a business rule you could assert against an API or a service in a fraction of the time, with far less flake. The classic test pyramid exists for exactly this reason, and most slow suites are shaped like an inverted one.

Environments and test data are prepared by hand

This is the quiet killer. Automation can't compensate for an environment that behaves differently every run. If your team is manually configuring environments, sharing mutable accounts, hand-seeding databases, depending on third-party services you can't control, and relying on tests running in a specific order, then a good chunk of your "regression time" is really setup and firefighting. The tests are just where the instability finally surfaces.

Failures take longer to investigate than tests take to run

A two-hour run can still block a release for two days. Once it goes red, someone has to work out whether the failure is a real product defect, a broken test, an environment hiccup, a bad data state, or a flaky third-party call. If that triage is slow and manual, investigation — not execution — is your bottleneck. I've seen teams shave twenty minutes off runtime and celebrate, while the two days of failure analysis after each run sat completely untouched.

Cut the regression scope before accelerating execution

The cheapest speedup is the test you stop running. Before touching the runner, shrink and reshape what regression even means.

Audit the existing suite. Go through it — ideally the slowest and most-changed specs first — and put five questions to every test:

What product risk does this test actually cover?
Has it caught a meaningful defect in recent memory?
Is the same behavior already verified somewhere else?
Does it still reflect how the product works today?
Does it genuinely need to run before every release?

A test that can't be tied to a current risk gets updated to match reality, demoted to a less frequent suite, or deleted. Deleting tests makes people nervous, so be concrete about the trade-off: a test that verifies nothing anyone can name is not protecting you — it's taxing every release for coverage theater.

Split regression by risk and purpose. Once the suite is honest, stop treating it as one thing. A practical structure has four tiers:

PR checks — fast API, component, and critical smoke tests, on every pull request.
Critical regression — core revenue, security, authentication, and data flows, on merge to main.
Extended regression — secondary workflows and less common edge cases, nightly.
Compatibility regression — browsers, devices, operating systems, and localizations, before release or on demand.

The point isn't the exact names. It's that these suites don't all need to run at the same moment in the delivery cycle, so you stop paying compatibility-matrix costs to validate a backend bug fix.

Use change-based test selection. You don't need an AI platform to run fewer tests intelligently. Start with plain mapping: for a given change, what product area moved, which services and integrations it touches, which critical user journeys depend on it, where defects have clustered historically, and which platforms or locales are at risk. Run the tests that give the strongest signal for this change. The goal is never "run fewer tests at random" — it's "run the tests that matter for what actually changed," and let the full extended and compatibility passes happen on their own cadence.

Automate the bottleneck — not the entire suite

Now, and only now, automation. The discipline here is subtractive: automate the specific thing that's slow and repetitive, not the entire inventory.

A test is a strong automation candidate when it's run often, tied to a high-impact risk, stable enough to give deterministic results, expensive or slow to do by hand, backed by data you can create programmatically, and needed across several environments or configurations. Those tests pay back the automation cost quickly and keep paying.

Just as important: decide what stays manual, on purpose. Manual testing is not a legacy practice you're phasing out. Keep humans on exploratory testing, brand-new or fast-changing features, usability and subjective UX judgment, rare scenarios where automation maintenance would cost more than it saves, visual or physical interactions that are hard to model reliably, and any failure that needs real investigative judgment. The aim is to remove repetitive manual regression, not human testing — those are different things, and confusing them is how teams end up with brittle automation and no one who actually understands the product.

Move checks below the UI. This is where the title's promise is kept. Validate business rules at the service or API level, test integration boundaries on their own, keep UI automation focused on a small set of critical end-to-end journeys, and stop asserting the same fact through three layers.

ChargeAfter, a payments platform, shows what this buys you. Its regression cycle dropped from two weeks to six hours — and the lever was structural, not the raw script count. We built 1,000+ automated API tests and around 4,000 combined UI and API scripts (Cypress and Playwright), reached roughly 90% coverage, then arranged them by when they run: a 10-minute API suite after every pull request, smoke tests every two hours (each 15–20 minutes), all on a dedicated environment with multiple pipelines and 12 parallel threads. The six-hour number came from API-level coverage, CI triggers, environments, and parallelism working together. If we'd simply recorded 4,000 UI scripts and hit "run," we'd have built a slower, flakier two-week suite.

Reduce execution and waiting time

With scope cut and the right things automated, attack the remaining lead-time buckets: setup, queue time, and reruns.

Stabilize environments and data first — before adding a single worker. Automation on top of an unpredictable environment just produces fast, confident nonsense. What "stable" looks like in practice: dedicated or reproducible test environments, data factories and API-based data creation instead of hand-seeded databases, isolated accounts per parallel worker, a health check that fails fast if the environment is sick before you run 3,000 tests against it, controlled stubs or mocks for external dependencies you can't govern, and automatic cleanup so one run doesn't poison the next.

Then parallelize — carefully. Parallel execution is not "turn the worker count up." It's a property your tests have to earn. Before scaling out, the suite must not depend on execution order, must not have two tests mutating the same records, must be able to create isolated data per worker, must fit inside the environment's capacity, and should distribute evenly so you're not waiting on one 40-minute shard while eleven others sit idle.

The payoff is real when the groundwork is there. Across projects I've seen 10 parallel threads on Abbott, 12 on ChargeAfter, 15 on GoodShape, and up to 20 on CipherHealth — a healthcare product running 1,400+ E2E and 250+ API tests to a five-hour regression pass. But the trade-off has to be said plainly: parallel execution amplifies your existing data and environment problems — it does not fix them. Point twenty workers at shared mutable data and you don't get twenty times the speed. You get twenty times the flake.

Run the right suite at the right time. Tie each suite to a pipeline stage:

Pull request: API checks and critical smoke tests
Main branch: critical regression
Nightly: extended regression
Before release: compatibility and high-risk end-to-end scenarios
On demand: localization, migration, performance, specialized suites

Handle fail-fast with judgment, not as a global switch. Aborting a run after a major infrastructure failure saves CI time and is the right call. Aborting after the first ordinary test failure often hides the three other real defects that run would have surfaced, and you learn about them one painful cycle at a time. The value can also be in the rerun and failure-handling logic itself: on Xola, a booking platform doing 5–10 releases a working week, we cut the main automation build from about an hour to roughly 20 minutes largely by fixing how the pipeline handled internal reruns and early failure detection — not by rewriting the tests. That same pipeline work is what surfaces about 70% of blocker, critical, and major bugs early.

Make the regression suite trustworthy

A fast suite that engineers don't believe is worthless. The moment people start re-running "just to be sure" or waving through red builds, you've lost the entire point of regression, however quick it is.

Treat flakiness as a defect, not weather. Flaky tests are more common than teams admit — Google's widely cited analysis found that almost 16% of their tests had some level of flakiness and about 1.5% of all test runs reported a flaky result, at a company with serious testing infrastructure. So track flaky tests separately from product failures, give every unstable test a named owner, quarantine only temporarily, record why something was quarantined, and put a deadline on repair-or-removal. Retries have a legitimate use — diagnosing an intermittent failure — but unlimited retries to manufacture a green build is just hiding a defect behind a loop.

Classify failures automatically. When a run goes red, the expensive part is figuring out what kind of red. Bucket failures into assertion/product failures, locator/automation failures, test-data failures, environment failures, and network/third-party failures — and surface that classification in the report. Getting an engineer to "this is an environment failure, not your code" in thirty seconds is often worth more than shaving five minutes off raw execution. Triage time is lead time.

Keep maintenance under control. Review tests when the related functionality changes, delete obsolete ones instead of muting them, keep selectors and test data separate from business logic so a UI tweak doesn't cascade, watch your slowest tests, and actually measure the engineering hours going into suite upkeep. And stop treating test count as a success metric. "We have 6,000 automated tests" tells me nothing about whether you can ship on Friday.

What the results can look like

Numbers help, as long as they're read correctly: these are outcomes from specific engagements, not benchmarks you'll hit by copying one tactic.

The healthcare platform from the opening is the fullest example. When we started, Abbott ran regression in 2.5 weeks and smoke testing in 7 days, on 50% coverage, with outdated test cases, no automation in place, and just 5 test devices. After the rebuild: regression in 1 day, smoke in 1 day, coverage at 90%, the relevant regression scope up 60%, 1,500 automated scenarios (part of 1,600+ automated test cases), 10 parallel threads, 20+ devices, and 27 localizations with about 70% of translations covered by autotests — run across 4 browsers by a team of 8 automation and 11 manual QA engineers.

Read those numbers carefully, because the tempting misreading is "automation did this." It didn't. The transformation came from cleaning up outdated test cases, widening relevant coverage by 60% rather than just encoding what existed, a real device and localization strategy, CI integration, and parallel execution — with the 1,500 automated scenarios as one component among several.

Other engagements point the same direction, with numbers worth reading closely:

CipherHealth: a five-hour regression pass and a ten-minute smoke suite on 20 parallel threads (1,400+ E2E and 250+ API scripts, 90% of features covered). Production bugs fell 35% in the first seven months, and the product passed 240+ successful releases — up from a baseline where 20% of releases shipped with major bugs.
Planetly: regression in under an hour and smoke in under ten minutes, from 400+ end-to-end tests at roughly 90% coverage — about a 3x cut in regression time versus the earlier manual process.
GoodShape: 90%+ of test cases automated (~3,800 scripts, ~95% overall coverage, ~99% of smoke automated) and a roughly 95% drop in post-release regression bugs, with regression running about 2x faster.
Xola: regression held to one to two hours across 3,200+ scripts and 8,300+ checks (90% coverage, 160+ devices) while shipping 5–10 releases a working week.

Different stacks, different tactics — none got there by automating their existing suite as-is.

A practical implementation plan

Treat this as a shape, not a schedule. The durations depend entirely on your starting mess.

Phase 1 — Establish the baseline. Measure full regression lead time and split it into preparation, waiting, triage, and reruns. Identify the slowest and least reliable tests. Write down your current release risks.

Phase 2 — Reduce and restructure. Remove obsolete and duplicate tests. Create the critical, extended, and compatibility suites. Map tests to risks and product areas. Decide which suite runs at which pipeline stage.

Phase 3 — Automate high-value bottlenecks. Start with the critical, repetitive scenarios. Add API-level coverage before expanding UI automation. Automate test-data creation. Add fast PR and smoke checks.

Phase 4 — Improve execution. Introduce isolated parallel workers, optimize the slowest tests, make environments reproducible, and add automatic reporting and failure classification.

Phase 5 — Stabilize and maintain. Track flakiness, assign test ownership, review the suite continuously, and keep removing tests that no longer cover real risk. The cleanup in Phase 2 is not a one-time event.

How to measure whether the strategy worked

Do not use the raw count of automated tests as your headline metric. It rewards exactly the bloat you're trying to remove. Track instead:

Total regression lead time (the real number)
Execution time per suite
Time to first useful failure
Smoke-suite duration
Failure investigation time
Flaky-test rate and rerun rate
Percentage of critical risks actually covered
Escaped regression defects and overall post-release defect rate
Automation maintenance effort
Release frequency

And hold one uncomfortable idea alongside these: higher coverage is not automatically better. Ninety percent coverage that's slow, flaky, or aimed at low-risk corners of the product is worse than seventy percent that's fast, trusted, and pointed at the flows that generate revenue and lose customers.

The shift that actually matters

You don't cut a multi-week regression cycle by automating every test. You cut it by refusing to treat regression as one enormous suite that runs the same way for every release.

Three things to take away:

Reduce the scope before you accelerate it. The fastest test is the one you correctly stopped running.
Automate the bottleneck, not the inventory. Encode the slow, repetitive, high-risk checks — and push them below the UI wherever you can.
Optimize the whole feedback loop, not just execution. Preparation, environments, queueing, triage, and reruns are usually where your weeks are hiding.

These are the same principles we lean on when rebuilding regression testing and test automation strategy for products that can no longer afford multi-day release validation. Start by measuring where your lead time actually goes. It's almost never where you think.

Complete Guide on How to Build a Maintainable Selenium Framework

Mike Ralduxin — Thu, 28 Jan 2021 10:38:55 +0000

Selenium is second to none when it comes to automation testing tools. Thousands of automation QA engineers efficiently use it every day to facilitate the execution of testing. If you are a newbie and want to get the philosophy of Selenium, there are some tips on how to get the most out of it and create a maintainable and robust Selenium automation framework.

A brief overview of the Selenium framework

Selenium framework can be defined as a set of testing tools, patterns, and practices that provide opportunities for efficient test automation, code readability, and easy code maintenance.

A plenty of automation framework types are differentiated. Still, if you prefer to work with Selenium, you can build three of them:

Data-driven framework splits test script logic off from test data. It gives an opportunity to store test data in external storage. For this purpose, a QA engineer can use MS Excel Sheets, SQL Database, CSV files, text files, XML files, ODBC sources, etc.
Keyword-driven framework speeds up the process of automated testing. Each set of functions and instructions also stored in an external file has a particular keyword. As a result, QA engineers can easily pick up functions for testing.
As its name implies, a hybrid framework is a combination of two previous frameworks. Either test data or keywords are externalized. While keywords are placed in a segregated Java class file, test data is located in an Excel file or Properties file.

When a great volume of data should undergo testing, a data-driven framework is definitely the best choice. If you are going to deal mostly with functionalities, it is better to apply a keyword-driven framework. In situations when a large volume of either data or functions should be tested, it makes sense to leverage a hybrid framework.

Why Do We Need A Selenium Framework?

Selenium framework considerably simplifies the work of automation QA engineers. Code and data would be kept together without a framework, and it would heavily affect reusability and readability. When it is properly developed and implemented, the Selenium framework brings multiple advantages such as code reuse and readability, better portability, easier code maintenance, minimum human intervention during test running, maximum test coverage, easy reporting, and many others.

Merits of the Selenium automation framework

Numerous merits make the Selenium automation framework so popular among QA experts from different corners of the world. Let's have a look at its main advantages to understand why it is an unbeatable automation testing tool:

Availability

Selenium is freely available, and everyone can use it without paying a penny whereas most other solutions are commercial.

Most user actions can be reproduced

Selenium WebDriver can copy the most various user actions. Therefore, QA engineers can automate various inputs like dragging and dropping, mouse-clicking, button pressing, clicking and holding, and so on.

User-friendliness

Selenium automation framework is easily implemented and easy-to-use. Its interface is pretty simple as well. Besides, QA experts can enrich their frameworks with different extensions according to their needs and project requirements.

The most various scenarios can be covered

Selenium provides various options; that is why every QA expert most likely finds something that suits their scenario.

Multilingual capability

It is indeed a helpful feature of Selenium. QA experts enjoy freedom while choosing a programming language. Building the Selenium automation framework they can use Java, C#, JavaScript, Python, Ruby, Perl, or PHP.

Cross-browser compatibility

Selenium supports all commonly used browsers — Safari, Chrome, Opera, Firefox, Edge, Internet Explorer.

Support of various operating systems

Selenium supports such OSs as Windows, Linux, Mac, UNIX, etc.

Integration with numerous tools

To significantly simplify automated testing, Selenium smoothly integrates with numerous tools. For instance, integration with TestNG & JUnit enables test case management and report generation. Integration with Maven, Jenkins, and Docker gives an opportunity for deployment process automation.

Reusability

Created scripts are compatible with different browsers. As a result, QA engineers can run numerous testing scenarios utilizing the same base.

Maintainability

QA experts can quickly modify code and easily maintain scripts due to an opportunity to refactor and group their test cases.

Community Support

The Selenium community is indeed wide and active. As a result, a ton of information can be found, and valuable pieces of advice can be received when QA experts need them.

Unique options

With Selenium WebDriver, QA experts can reproduce clicking on the browser “back” and “forward” buttons. This feature can be pretty useful in some situations and most other solutions do not offer it.

How to build maintainable Selenium framework

A maintainable and reliable test automation framework is 50% of test automation success as it enables QA experts to create effective test scripts. Below you can find some aspects you should take into consideration to build a robust Selenium automation framework.

To create a framework that would satisfy all your needs, it is vital to identify processes that should be automated. That is why you should study the internal structure and workflows of your application to understand which processes can be and must be automated and define requirements for your future test automation framework.

Programming language

As we have already mentioned, Selenium supports plenty of programming languages and you should decide which one to use for your framework design. In this case, we advise you consider a language that has been used for your application development or a language at which most of your colleagues are good.

Framework architecture

Selenium framework architecture consists of the Selenium Core component and the Selenium Test component. It is advised to begin with the Selenium Core component which controls browser instances and element interactions. As soon as your Selenium Core component is ready, you can proceed to the Selenium Test component. You can use a page object model (POM) to design this component. It is indeed very useful due to its ability to eliminate code redundancy and cut down testing expenses.

Unit framework

Unit framework is utilized for test verification and test case grouping and prioritization. Working on your automation framework development you should choose a unit framework that will be integrated. In these terms, TestNG is considered as one of the best unit frameworks for Selenium as it offers multiple benefits, among which are such valuable features as parameterizing, sequencing, annotations, and grouping.

CI/CD implementation

To get the most out of your automation framework you should also consider CI/CD implementation. In these terms, a range of helpful tools should be integrated with your framework:

Dependency managers like Maven, NuGet, Gradle, NPM, and Ant give an opportunity to efficiently manage dependencies and librarie. Build tools also come in handy when it comes to source code writing, dependent library creation, and test running.
Version control systems as GitHub, Bitbucket, or TFS significantly simplify work on the source code of your tests. Still, if your source code should be kept private, it is highly advised to implement your own version control system using Git.
To speed up product delivery and get a competitive advantage, it is essential to integrate automated tests into CI/CD pipelines. That is why, integration with one of the CI tools such as Jenkins, TFS, or Bamboo is a must.
- Report generation

A reliable test automation framework should also provide test results generating accurate testing reports. These reports should be clear, readable, and available for either technical or non-technical staff who need them. As Selenium does not provide a built-in report option, you should integrate an external tool. For instance, you can utilize TestNG or JUnit. However, reports generated by these tools are not readable. To create human-readable reports, external libraries like ExtentReport or Allure should be used.

Conclusion

Test automation can significantly simplify and optimize testing processes. However, to achieve excellent results, it is essential to prepare a strong foundation, i.e., create an effective test automation framework. In these terms, Selenium is the best choice. It provides almost everything you need, while options that are not available still can be used due to integration with multiple tools. While building your Selenium automation framework, you should take into consideration numerous aspects as project requirements, processes that should be automated, programming language, framework architecture, unit framework, report generation, CI/CD implementation, etc. Still, at the end of the day, all efforts will be justified because a well-thought framework enables QA team members to create effective automated scripts and maintain them easily.

Manual Testing Guide

Mike Ralduxin — Tue, 29 Sep 2020 08:40:32 +0000

Testing is one of the most important stages of the software development life cycle. These days, there are two main approaches to software testing: manual and automated. Most QA teams skillfully combine both approaches as they are not entirely interchangeable and have either advantages or disadvantages. We are going to consider manual testing in detail. So let's start from scratch and consider the manual testing basics.

Definition of manual testing

Manual testing is a software examination and utilization from the end-user perspective to assess its quality, detect existing defects, and uncover possible weaknesses. In contrast to automated testing that utilizes a software program to conduct tests, manual testing reckons upon human's actions.

Manual testing process

As a rule, to provide professional and comprehensive testing, most manual testers organize their workflow in the following way:

Requirement studying

Manual testers must get acquainted with the requirements to ensure a successful QA process. Thus, they can understand the software's objective, operation principles, main functions, and possible vulnerabilities. Testers can make up excellent test cases and identify more defects using this information. However, project specifications and user stories are not always available. In this case, manual testers should be more inventive and check other information resources to realize how an application should work.

Drawing up of test cases

As soon as manual testers have gotten acquainted with project requirements, they start to draw up test cases using various test design techniques. Test cases are used as instructions for manual testers, as they contain the consecutive steps for a particular function or scenario testing. They should be rather detailed. Therefore, other testers will also be able to carry out your tests, not asking clarification questions.
Some manual testers still use an Excel document to file their test cases. However, various case management tools are available; they can significantly simplify test case creation, storage, and management.

Useful articles:
How to write test cases
Manual Test Cases Writing Hacks

Test conduction and bug reporting

After all test cases have been made up, and the test environment has been prepared, manual testers go ahead with test conduction. If manual testers detect bugs during testing, they must log bug reports using a bug tracking system. The bug reports must also be detailed and contain reproduction steps, an expected result, an actual result, and relevant attachments (screenshots, screen recording, etc.). Therefore, a developer can get bug essence right and quickly reproduce it.

Analysis of testing results

As a rule, after testing completion, manual testers undertake an analysis to receive useful metrics that can help make the right consequences and plan future activities. It can take just a few clicks if you use one of the various test case management systems.

Types of Manual Testing

In our manual testing guide, we cannot leave out the manual testing types. Frankly speaking, there are countless manual testing types, but we will consider the most widely used.

Exploratory Testing

Generally, manual testers turn to exploratory testing when they want to understand the basic idea of the software or get acquainted with its new features. It has no guidelines, and testers explore systems following their own initiative and curiosity. Exploratory testing is carried out whenever a tester thinks it is required.

Usability Testing

There is no chance to effectively carry out usability testing without engaging people because the user-friendliness of software is evaluated on the ground of users behavior and emotions. During usability testing, manual testers make sure that an application is simple, convenient, and intuitive to avoid end-users confusion and frustration.

GUI testing

In the process of GUI testing manual testers check fonts, color palette, windows, buttons, and all other GUI elements and characteristics to guarantee that there are no errors or confusions and provide the best user experience.

Localization

Manual testers perform localization testing to ensure that an application meets various language and cultural requirements. Localization testing is particularly important if an application is targeted at a global user community.

Compatibility testing

Compatibility testing is conducted to ensure that an application works properly with other software, different hardware, network, etc. Compatibility testing is rather time-intensive. For this reason, most QA teams tend to automate this process. Still, if there is a need to carry it out manually, it is advised to define the top-priority browsers and platforms and concentrate on them.

Smoke Testing

Smoke testing verifies only the key functionality of the application to define whether it corresponds to its primary objective and does not contain critical bugs. Smoke testing is a kind of preliminary checking that is carried out before more critical, in-depth testing.

Regression Testing

Regression testing is executed before every release to check whether updates have not affected existing functionalities. Regression testing of large, sophisticated applications can be extremely time-consuming. For this reason, if there are opportunities, many companies try to automate it.

Main Manual Testing Techniques

Three manual testing techniques are differentiated. So, let's consider each of them one by one.

Black box testing

Black box testing is conducted in a way that resembles end-users interaction with a system. It doesn't require knowledge of the internal structure and coding skills. Conducting black-box testing, QA engineers interact with UI, test functionality, and verify whether outputs coincide with expected results.

White box testing

White box testing technique is usually utilized by developers to review code and dataflows and detect possible hidden bugs. This technique requires good coding skills.

Grey Box testing

This manual testing technique is a balanced combination of either black box testing or white box testing. Grey box testing covers all aspects and deals with everything but in a more superficial manner.

Common Myths of Manual Testing

A manual tester profession appeared quite a long time ago, and a considerable number of myths have grown around it. Right now, we will debunk the most common myths.

Manual testing is easy

It is probably the most widely spread myth. According to mainstream thinking, everybody can do testing. Let me disagree. Manual testing is a sort of art. A special mindset and skills are required to conduct testing as it is not just a random clicking, as some people may think. A manual tester must be extremely detail-oriented, obsessed with product quality, think extraordinary, know different software development aspects, and many others. Not everyone can be a professional manual tester.

Manual testing is out-of-date

This misbelief is becoming very popular these days. Some people think that manual testing has become out-of-date, and automated testing will completely replace it very soon. We must reassure you that manual testing is the base of the QA process. Automated testing would never displace it. Manual testing helps make a stable product, and only after that, automated testing gets into a game. Automated testing is impossible without previous manual testing.

Manque programmers choose the testing profession

It's worth saying that not every programmer can become a good tester while many manual testers successfully master programming to understand all aspects of software operation and move to a new level.

Testers get down to work only on the post-development stage

There is an opinion that manual testers idle most of the time and get to work in the end. It is a blunder. If some companies indeed use this approach, the project has severe problems. The involvement of a QA team at a final stage is a big risk for product quality and schedule. Testers must get to work as early as possible to have enough time to study documentation, analyze risks, draw up and conduct test cases.

Importance of Manual Testing

As was mentioned earlier, some people have doubts about the importance of manual testing. Nevertheless, any professional QA engineer can easily refute this statement. There are many reasons why manual testing is indispensable, so let's consider some of them.

Certain tests must be conducted only manually

User experience plays an important role in quality assurance. In some situations, just a human can properly conduct certain tasks and provide an appropriate assessment. For instance, it is impossible to imagine the conduction of usability testing or localization testing without human involvement. Compatibility and interactions with mobile devices can't be checked with automated testing as well. Application performance during wi-fi turning off and turning on, simultaneous running with other apps, incoming calls or messages, etc. can be checked only by a human too.

Automated testing only supports human testers

Automated testing cannot substitute manual testing but can be and even must be effectively used in combination with it. While automated tests save time with repetitive tasks, manual testing efforts can concentrate on more creative work.

Most bugs are hidden in unexpected places

Even keeping up with the previously prepared test cases, manual testers can still uncover bugs in places they have not expected to find. That is very important. Testers detect many bugs while they are checking something else. In its turn, automated testing can't spot defects that it hasn't been programmed to search.

Automation is not budget-friendly for small-scale projects

Automation requires not only considerable initial investments but also high maintenance expenses. For large scale projects, automation can be indeed beneficial in a long-term perspective. However, for small projects, automation is still a waste of either time or money.

Automation often falls behind sprints

Due to continuous script updating, it is difficult for automated testing to keep up with sprints. Only perfectly managed automation can manage to do this.

Manual testers assess product from the end-users perspective

Manual testers usually act as end-users, so they not only ensure correct work of application and help define new ways of product improvement with their pieces of advice and suggestions.

Repetition and variability are of great importance

Repetition and variability are two characteristics of effective testing. Automated testing is appropriate for the repetitive testing process, but it's not enough. Manual testing, in its turn, can provide needed variability.

Top Manual Testing Tools List

Manual testers have at their disposal various useful tools that considerably simplify the testing process. Here you can find the most widely used.

Test Case Management Tools (TestRail, Zephyr, Practitest, TestLink)

These tools are handy and used by manual testers to easily create, manage, update, and store test cases.

Related article: Best Test Management Tools

Bug Tracking Tools (Jira, Trello, Bugzilla, Mantis)

Bug tracking solutions are indispensable helpers of manual testers. With their help, QA engineers create, modify, store and manage bug reports as well as get valuable statistics.

Related Article: Best Bug Tracking Tools

Cross Browser Testing Tools (BrowserStack, Sauce Labs, Browser Shots)

These tools help manual testers quickly verify whether the UI elements are rendered correctly across multiple browsers and how the code responds in each browser.

Related Article: Best Cross Browser Testing Tools

Screen Capture Tools (CloudApp, Loom, Skitch)

Manual testers often have to attach images and videos to bug reports to describe an issue more precisely. These tools help to create needed images and screen recordings quickly.

Conclusion

Manual testing involves much effort and can also take much time. Nevertheless, it is essential to ensure a high level of software quality. Automated testing provides fantastic opportunities, but despite its advantages, automated scripts can test just what they have been written to test, while manual testing is more adaptable and creative. A human tester can always find things that an automated test cannot. To gain the best results and guarantee the top-notch quality, QA teams must use a combination of either manual or automated testing, as they are two sides of the same coin.

A Guide to automated testing for microservices

Mike Ralduxin — Wed, 22 Apr 2020 08:47:01 +0000

Today a lot of applications use microservice architecture, but what should we know and how we should test this type of software? In this article, we will consider basic theoretical concepts about CDC testing and testing tools that will be helpful.

Microservices architecture - is a collection of small services and each service responsible for certain functionality and can be written by using different programming languages, approaches, and solutions. One of the main goals of microservices-based application it's define strong contracts between the various microservices.

Consumer-driven contract testing (CDC) - is a technique that allows us to test each application in isolation. After executing consumer tests we get generated pact files that will be used for getting verification results during executing provider tests. As a result, we will have artifacts that will be placed in Pact-broker.

Let’s familiarize ourselves with the main concepts from the official sources.
Pact - is a code-first consumer-driven contract testing tool, and is generally used by developers and testers who code. In general, a contract is between a consumer (for example, a client that wants to receive some data) and a provider (for example, an API on a server that provides the data the client needs).
Pact Broker - is an application for sharing for consumer driven contracts and verification results. It is optimized for use with "pacts" (contracts created by the Pact framework), but can be used for any type of contract that can be serialized to JSON. The Pact Broker is an open source tool that requires you to deploy, administer and host it yourself but also they provide plug-and-play option at pactflow.io.
Mocha - is a feature-rich JavaScript test framework running on Node.js and in the browser, making asynchronous testing simple and fun. Mocha allows you to use any assertion library.
Chai - is a BDD / TDD assertion library for node and the browser that can be delightfully paired with any javascript testing framework.
Axios - promise based HTTP client for the browser and node.js.

Simple example of CDC testing.

Here is a code snippet from package.json where you can see all packages that will be used.

{
"@pact-foundation/pact": "^9.9.3",
"axios": "^0.19.0",
"chai": "^3.5.0",
"mocha": "^5.1.1"
}

Step 1. Consumer tests.

There are two different specs that will be describing TODO API - “JSONPlaceholder” and USER API - “REQ | RES” . The first example you can see below. Second spec the same except the request and response objects.

todoClient.js


const getUserById = (id, baseURL) => {
   return axios.request({
       method: "GET",
       baseURL,
       url: `/users/${id}`,
       headers: { Accept: "application/json" },
   })
};

todo.spec.js


describe("The Todo API", () => {
 const mockServerURL = 'http://localhost:1234';

 // Create the Pact object to represent your provider
 const provider = new Pact({
   port: 1234,
   log: path.resolve(process.cwd(), "logs", "mockserver-integration.log"),
   dir: path.resolve(process.cwd(), "pacts"),
   consumer: "Consumer",
   provider: "Todo Provider",
   spec: 2,
   logLevel: "error",
   pactfileWriteMode: "merge"
 });

 // this is the response you expect from your Provider
 const EXPECTED_BODY = {
   userId: 1,
   id: 1,
   title: 'delectus aut autem',
   completed: false
 };

 // Start the mock server
 before(() => provider.setup());

 // Write Pact when all tests done
 after(() => provider.finalize());

 // verify with Pact, and reset expectations
 afterEach(() => provider.verify());

 describe("Get todo by id", () => {
   before(async () => {
     const interaction = {
       state: "Get todo",
       uponReceiving: "A request for one todo",
       withRequest: {
         method: "GET",
         path: "/todos/1",
         headers: {
           Accept: "application/json",
         },
       },
       willRespondWith: {
         status: 200,
         headers: {
           "Content-Type": "application/json; charset=utf-8",
         },
         body: EXPECTED_BODY,
       },
     };
     // add interactions to the Mock Server
     await provider.addInteraction(interaction)
   });

   it("GET /todos/1", async () => {
     let { data } = await getTodoById(1, mockServerURL);
     expect(data).to.eql(EXPECTED_BODY);
   });
 })
});

After running tests we will receive the following pact files:
consumer-user_provider.json
consumer-todo_provider.json:

{
 "consumer": {
   "name": "Consumer"
 },
 "provider": {
   "name": "Todo Provider"
 },
 "interactions": [
   {
     "description": "A request for one todo",
     "providerState": "Get todo",
     "request": {
       "method": "GET",
       "path": "/todos/1",
       "headers": {
         "Accept": "application/json"
       }
     },
     "response": {
       "status": 200,
       "headers": {
         "Content-Type": "application/json; charset=utf-8"
       },
       "body": {
         "userId": 1,
         "id": 1,
         "title": "delectus aut autem",
         "completed": false
       }
     }
   }
 ],
 "metadata": {
   "pactSpecification": {
     "version": "2.0.0"
   }
 }
}

Step 2. Publishing pact files. In this example we will use pactflow.io (Developer FREE plan).

publish.pacts.js


const gitHash = require('child_process')
    .execSync('git rev-parse --short HEAD')
    .toString().trim();

const opts = {
   pactFilesOrDirs: ['./pacts/'],
   pactBroker: 'https://yourCompanyName.pact.dius.com.au',
   pactBrokerToken: 'yourPactBrokerToken',
   tags: ['prod', 'test'],
   consumerVersion: gitHash
};

pact
   .publishPacts(opts)
   .then(() => {
       console.log('Pact contract publishing complete!');
    })
   .catch(e => {
       console.log('Pact contract publishing failed: ', e)
   });

For getting pactBrokerToken you need to proceed to /settings/api-tokens and copy Read/write token (CI)

After running this script we will receive the following results:

Step 3. Pacts Verification


provider.spec.js
const opts = {
   logLevel: "INFO",
   providerVersion: "1.0.0",
   pactBrokerUrl: 'https://yourCompanyName.pact.dius.com.au',
   pactBrokerToken: 'yourPactBrokerToken',
   publishVerificationResult: true
};

describe("Pacts Verification", () => {
   it("Validates the expectations of Todo provider", () => {
       return new Verifier({
           ...opts,
           provider: 'Todo Provider',
           providerBaseUrl: 'https://jsonplaceholder.typicode.com',
       }).verifyProvider();
   });

   it('Validates the expectations of User provider ', () => {
       return new Verifier({
           ...opts,
           provider: 'User Provider',
           providerBaseUrl: 'https://reqres.in/api',
       }).verifyProvider();
   });
});

After running tests we will receive the following results:

VIEW PACT:

Let’s change the title in the expected result from step 1 for simulating an error.

Previous object:


 const EXPECTED_BODY = {
   userId: 1,
   id: 1,
   title: 'delectus aut autem',
   completed: false
 };

New object:


 const EXPECTED_BODY = {
   userId: 1,
   id: 1,
   title: 'Our new title that will trigger the error',
   completed: false
 };

Result after running tests:

Console output:

Conclusions.

Testing microservices it's a very difficult process that requires a good knowledge of different testing tools for achieving needed goals. In this article, we considered basic concepts and simple example of writing CDC tests using Pact. You can deeply familiarize yourself with this powerful testing tool on the official site and use all provided features for testing your microservices.