Continuous Testing
The goal of continuous testing is to provide fast and continuous feedback on the level of business risk, allowing for a rapid go/no go decision on the feature delivery.
Motivation:
• Risks are exposed as soon as they are introduced
• Prevent those problems from progressing to the next stage of SDLC
• Reduce time and effort for finding and fixing defects
Challenges:
One of the major impediments achieving an efficient continuous testing outcome is Test Design.
Business/PO’s make go/no go decision based on the user journey. The disconnect happens when:
- tests are developed without an understanding of the business context.
- tests designed either based on requirements or to test the unit of code.
Testing should be a collaborative effort of Dev, QA, and Business, aligned with priorities of the line of business. Remember: Quality is everybody’s responsibility!
What is BDD?
BDD enables software teams to work in collaboration deriving business value. This allows us to build a shared understanding of the work being done. Teams use domain-specific language and produce system documentation based on this shared understanding.
Example:
Feature: Consumer Credit Card Application
Scenario: Reject credit card application if the applicant’s credit score is less than 450
Given An individual applied for a credit card from an affiliate corporate site
When the individual has a credit score of less than 450
Then reject the credit card application with an error message
And show suggestions for secured-card options.
https://cucumber.io/docs/bdd/?sbsearch=What%20is%20BDD describes in detail the essential three-step activity as an iterative process to achieve an executable specification deriving business value. Note the Discovery and Formulation phases. At the end of the formulation phase, there should be multiple readable acceptance criteria as part of the Definition of Ready for the user story.
Anti-Pattern:
Feature: CreditCardApplication API
Scenario: Validate submitCreditCardApplication returns ‘LOW_CREDIT_SCORE’ code for credit score < 450
Given the client sends submitCreditCardApplication a message
And getCreditScore api returns applicant.score.value less than 450
Then return status code 404 with error code “LOW_CREDIT_SCORE”
And the response has section applicant.misc.suggestions
The above is an anti-pattern because the scenario does not describe the business value. A product owner or a business unit owner may find difficulty at deriving the value it provides to the product even though it is ultimately testing the obvious rules (per dev/qa technical team members).
Most often the Dev/SDET/QE ends up writing the anti-pattern as the intention is to test the underlying code or business rule.
How does Continuous Testing benefit from BDD?
Ideally, all tests should be executed across all non-production test environments. All tests need to be run at some point in the build pipeline, but not all tests need to be run all the time.
Testing must begin early; it must be logically componentized and must be executed continuously.
BDD feature files allow the business to pick on tests that prioritize the business value and those can serve as the minimum tests which should be run on a per commit basis. If the tests are too technical to read and do not throw light on the underlying business value then you end up with the temptation of running all the tests or do not understand what the failure of the test means to the product, hence makes it difficult to make an informed and quick go/no go decision. You still run all your Unit Tests, but the user journey tests can be prioritized in a local/dev environment. A similar Test selection needs to be done for each upstream environment the build is promoted.
BDD way allows the team to compartmentalize the application and derive logical tests focusing user journey.
Artifacts produced by Continuous Testing process:
• List of actionable tasks
• Metrics on the quality of the process leading to continuous improvement
Continuous testing should be able to achieve predictability. It should help understand the impact of a change "early". Predictability, in turn, will pave the way to be faster in repetitions.
Gotcha's: (For another day)
• Heavy reliance on manual and/or automated “GUI” tests. (Emphasize on API Tests)
• Test execution process (Lack of Dynamic Test Automation)
• Test environment and Test data constraints (Service Virtualization);
• Test infrastructure
Oldest comments (0)