DEV Community: kapil

Test Environments- What & Why

kapil — Tue, 12 Jan 2021 12:28:16 +0000

Why do we have so many/less test environments, why is volume of test data so small on dev environment, where am I going to test critical bug in production without disturbing BAU development, How about testing db cluster setup; where should I test those? All valid questions? Without a further ado let’s get on to answer few of them.

First thing first: What is test environment

Test environment is a set of hardware on which application under test(AUT) and supporting software are deployed in order to assist in testing.

The good part of working in my current organisation is that you get a chance to work on different projects in a very short span of time. Different project means different requirements, different setup, different personalities and different arguments for the same topic. Lately I have been talking to different people about, what all environments do we need in a given project?

Short answer to the question is: “it depends” on the project and current setup of the organisation infrastructure. While this is true but for simplicity we can categorise project in below 3 categories. There are difference Devops maturity model your project can be in. For further reading, refer state of Devops over here:

Following waterfall SDLC
Projects transitioning to Agile & CI/CD
Project already running in CI/CD

As of now I have worked in 2 different types of projects, yet to work on a project following Continuous delivery.

Ok coming back to test environments; with Devops it feels very tempting to have to too many environments and at the same time without Devops we try to manage in whatever test environment is available to us. Both of the approaches comes with pros & cons. Too many test environment comes with overhead of maintaining, deploying application and infrastructure cost. Less test environment might increase testing time and impact quality of scenarios one wants to perform in a project.

Another way to look at the test environment requirement is the number of testing/quality gates and where they will run for successful and timely delivery of software.

Some of the testing activities in typical project will be; Dev testing, Infra testing, Automation testing, Qa Exploratory testing, Integration testing, Business user testing, Compatibility testing, Security testing, Performance testing, Hotfix bug testing/Signoff etc

The bare minimum environment for a typical enterprise project which I have seen so far are Dev, QA, UAT, Perf environments apart from Production but again it varies from project to project(yes, it depends). There are other optional environments also like CI(Automation), Infra, SIT.

One question that comes to mind after seeing all such environments is, do we need so many environments? Lemme know your thoughts in the comments section

Let's get down to what & why of all these major test environments.

Dev environment

Primary owner: Developers

Purpose & Activities:

Used for dev testing and assist in development activities, can choose to run automated E2E testing
Automated security scans[Static code does not require any environment though]
Authentication/Authorisation are relaxed to assist in testing

Environment Setup:

Trimmed down version of production, if there are multiple cluster in production this will typically have 1 cluster setup

Test Data:

Small size with good enough variation of test data
Can be mocked for better stability and different scenarios creations

Deployed artefact version: Always latest

QA environment

Primary owner: QAs/ QA Lead

Purpose & Activities:

Acceptance testing & Story Sign off
Design/UI testing
Exploratory testing
Can be used to run automated regression suite
Bug bash

Environment Setup:

Trimmed down version of production, if there are multiple cluster in production this will typically have 1 cluster setup

Test Data:

Should be always integrated with stable backend
Different shape & size of test data from production with masked PII
Test data pipeline to update test data with a certain frequency [like once in a month etc]

Deployed artefact version: Runs behind or equivalent to Dev environment

SIT Environment

This environment is usually optional as all of these activities can also be performed in QA environment but typically in enterprise we do have separate SIT environment for smooth working of different teams.

Primary owner: Release manager/ QA Lead

Purpose & Activities:

Integration testing of releasable artefacts
All other environments usually connects with dependent artefacts of this environment.
Manual backward compatibility testing e.g. new backend and older front end [mobile apps]
Manual regression suite [if applicable]
Manual Security/Penetration testing
Roll back strategy testing
Verbiage verification if any

Environment Setup:

Trimmed down version of production, if there are 5 node setup . This environment will have 3 nodes.

Test Data:

Should be always integrated with stable backend
Different shape & size of test data
No Mocking
Test data pipeline to update test data with a certain frequency [like once in a month etc]

Deployed artefact version: Typically runs behind QA environment

UAT Environment

Primary owner: Release Manager/ QA Lead

Purpose & Activities:

Stands for user acceptance testing or SME/Business owner testing: mimicking actual business scenarios.
Usually requirement miss etc are feedbacks from this environment
Business owner bug bashes etc
Hotfix bug testing

Test Data:

Replica of production with PII masked.

Deployed artefact version: Typically runs behind SIT by a version

Perf Environment

As the name suggests, this environment is used to run various performance & benchmarking tests.

Primary owner: QA lead, Release Manager or Performance team if separate

Purpose & Activities:

Benchmarking with simulating different load scenarios for certifying speed, scalability & stability of application

Environment Setup: Same as production or very close to production

Test Data: Similar to production in shape & size.

Deployed artefact version:

Run behind by SIT by a version or equivalent to SIT.

Above list is my experience of working in enterprise accounts, do let me know your setup in comments section. I have seen few variation e.g. having separate automation environment or separate Infra environment for testing roll back strategy or infra scripts but typically above is good enough list. You can find different names of testing environment in different organisation but once we know the purpose we can always reason about the same. Also there are numerous hacks we can put in order to assist in test environment. But general rule of thumb is; all access and setup of test environment should be as close as possible to production; be it opening to internet to test data setup and should assist in verification of Application under test.

Happy Testing!

One Failing Unit Test

kapil — Sun, 10 Jun 2018 12:59:42 +0000

unit

Currently I am working in a project as QA where we are doing re-model of Sql database to new shiny Graph db and writing a new rest service on top of it.

Greenfield “yay” . I have been hearing a lot about Test Pyramid for quite sometime and now is the time when will be witnessing how it is built with all ups and downs.

In test pyramid most of the automated test should be covered at unit level, then some at integration and few for end to end journeys.

Unit testing for most of my career was black box. Also as a gate keeper of App(this statement is debatable, lets park this for future), trusting test cases which are majority in number, which you pretend to understand and also have not written, is not very comfortable position to be in.

Hence I wanted to keep my eyes ear open for anything which is not caught by unit test. And my wish was answered very soon.

It all started with a small ad hock change request from a BA in our team.

Requirement

BA: Hey can we please change the “id” to “key” in the response.

Dev: Sure.

QA: will be tested in a jiffy.

Requirement

Lets look at some pseudo code

Implementation

Lets assume we are building simple App which returns Employee details. Pseudo code will look something like below.

Scala case class as model
Serialization and deserialization
Neo4j connection
Query to get Employee

**case class** Employee(id: String, firstName:String, lastName:String)
**implicit val** employeeFormat = jsonFormat3(Employee)


def getEmployee(id:String): Employee {
//read neo4j data and return case class 
query = "match (e:Employee {id:id})

//some code to parse query result and make Employee model
return Employee(e)
}

Minimal test case & seed test data for above code will be.

**def** testEmployee():Unit={
//spin in-memory neo4j
//add create (e1:Employee {id:'1', firstName:'Joe', lastName:'Doe')
//val employee = getEmployee("1")
//assert employee model with "key"

Everything is good so far, developer actually failed test cases by changing “id” to “key” in assertion of testEmployeModel test case, fixed code and “Yay” TDD.

The Problem

While doing e2e testing we loaded data in test db.

create (e1:Employee {key:’1', firstName:’Joe’, lastName:’Doe’)

Hit the service and “internal server error”.

Though the fix for this bug was pretty straightforward, but it kept me thinking what could have gone wrong here. We have written unit test case for this requirement, even development was done TDD way, now how come this bug?

If you know the answer then continue else go through the implementation one more time and see if you can figure out the problem.

The Fix

Change query in getEmployee method to

query = "match (e:Employee {key:id}) return Employee(e)

The Miss

There are 2 fold miss in this.

Requirement understanding
Test Fixture

Requirement from BA was never understood/explained clearly as implementation was easy and never discussed the “why” behind it. Consumer of the service wants “key” instead of “id” but also it was the database value which was going to change in the data-ingestion part.

If some more effort was spent in understanding the test case, and test data in particular for a unit test case before making any change to code. This problem must have been caught and fixed then and there itself.

But now imagine once the project matures, these test fixtures would be present in thousands of test how would someone know what to fix/update.

Its better to have no test case then a test case giving false positive results.

Have you ever experienced this problem of updating or forgetting to update hundreds of test fixture for any change in functionality, if yes any solution/strategy to handle. Do share in comment section.

API Mocks and why should we care

kapil — Tue, 13 Feb 2018 06:33:51 +0000

Service virtualization, Fakes, Stubs, Mocks, dummies etc are all different categories of Test Doubles which on broader level return a response for a given request in a controlled manner.

A request can be a method call, Object creation , database interaction or an webAPI interaction.

In below article we will be talking about API mocking and try to discuss some of what, why, how, when of API Mocks and also some challenges with them. If you are more concerned about the difference amongst all different types of Test Doubles then please head over here and here.

Enough said lets get started

What is Mocking

Mocks is nothing but an imitation(of what?). It simulates behaviour of real API but in a more a controlled manner.

A simple mock server consist of a server, which on matching certain request returns pre-defined response and other parameters associated such as response code, headers etc.

Why API Mocking

Getting quick and consistent feedback for a test are some of the key principles of continuous integration and API Mocks really help in facilitating this.

Quick feedback is very obvious, but lets understand.

Quicker the feedback of testing(either manual /automation) earlier bugs are caught and fixed. Have you been in a situation where a single test scenario take 15 min to complete with flaky payment service (roughly around 12 API calls), If yes embrace API Mocks.

What do we mean by “consistent/deterministic results”?.

Consider an airline booking application where we have to check whether user is able to

Search for flight from PUN-DEL
Select Seats for the passenger
Add trip extras like “in-flight wifi” etc
Complete Purchase

Over here search results, availability of seats and Trip Extras will vary depending on the current state of the flight, hence non deterministic tests.

But if we have mocked API responses for seat availability, then seat 25A will always be available.

Other benefits of API mocks are

If we are doing visual comparison of screen, then data painted on each screen also needs to be consistent in order for visuals comparison to work.
Validating manually scenarios when test environment is not available by pointing to Mocks.
Validating issues(prod/new issue) with different version of apps by pointing to same mock environment.
If backend and UI development are happening in parallel then UI needs a contract/dummy response to work with.

What to Mock?

As this article is pretty much about mocking backend, so yes backend should be mocked, but what if your team is owning some bit of backend(1 or 2 service) and UI.

Question to ask is

Should we mock backend APIs that we own or all the backend APIs to UI should be mocked.

Backend Mocked

If you are looking for a straight answers then “We should mock all the backend APIs of components which are owned by Team”. agree?

Let say team somehow agreed to approach where everything backend to UI will be mocked. Refer below

UI Backend Mocked

One fine day if SHIM layer owned by team is changed without making changes to UI then your UI test case will NOT raise any flags as they are always working with the mocked response and test will be returning false positives.

False positive is the last thing which you want in environment where confidence in automation test case is really crucial.

Food for thought : Is this problem with every API mocking(Let me know your thoughts in comments section, if yes how we can keep a check?)

How to API mock

First thing First : For mocking API, we need configuration in our app which can selectively point to mocks environment or live services based on the switch. It can be a javascript function which can change host & port or setting page in test builds for mobile app where one can select environment.

For writing mocks there are many open source tools available, personally I never felt the need to go for the paid solution for mocks or service virtualization.

Some of the popular open source solutions are wiremock, MounteBank. There is good documentation and “Getting Started” help available on respective tools website, but just to give glimpse how easy is to write mock for GET request, here is a snippet.

A simple mock GET request for “greet” in wiremock looks as follows

public class Test {

public static void main(String[] args){
        stubFor(get(urlEqualTo("/greet"))
                .willReturn(aResponse().withBody("Hello World")));
    }
}

If responses are big then storing them in file and returning file content is something to consider.

Also mocked response can be templatized to include date sensitive responses like greet users with “todays date”.

public static void main(String[] args){
    stubFor(get(urlEqualTo("/greet"))
            .willReturn(aResponse().withBody("Hello World, today is:"+ DateTime.now())));
}

When to API mocks

It again depends upon the project, if we have agreed on why part of it, based on the automation need, it should be early in the project also if any of the backend service is flaky or slow then those should be mocked.

Challenges

As we have seen above API mocks really comes handy for UI automation and other manual scenarios but there are some challenges associated with it.

States in Mocks

Mocking gets more complex if we have states in our app for example : /greeting api will return “Hello World” and then on subsequent /greeting we need to return “Hello Universe”.

Great powers comes with great responsibilities

Also, this prevents parallelization of test cases as one fine day states will interfere and test will fail and debugging will take ages.

Stale mocks

As we discussed in what to mocks section, there are chances mocks will become stale and there is no clear way to know when they have become stale(unless clubbed with Contract tests) .

Templates

Templatization issues for Date time needs to be sorted very carefully, if UI tests and mocks are running in different timezones.

Broken Request

Consider a request with all mandatory objects

{
  "action": "greet",
  "name": "kapil",
  "date": "2018-02-01"
}

In order to make mocks simple, quite many times mocks mapping for the above request would be something like

stubFor(post(urlEqualTo("/greet"))
        .withRequestBody(containing("greet"))
        .withRequestBody(containing("kapil"))
        .willReturn(aResponse().withBody("Hello Kapil!")));

Now consider a scenario where there is a bug and mandatory “date” object is not coming in request, our UI automation test case will always pass again giving false positives.

Effort for mock updates

Last but not the least, effort required to update mocks(or automated way to do that) when backend changes is something to consider upfront and this is coming from an experience of working in project that has ~3000 mock responses.

And thats pretty much it, I hope this article has given insights in API mocking. Will be discussing about some challenges in follow up post. Do let me know your feedback in comment section, and if you like the article do share.