Continuous Testing and ATDD

Code on GitHub.

Software and code quality requires more than just engineering discipline, it requires testing and measurement. In this article we’ll take a look at how Acceptance Test Driven Development (ATDD) and Continuous Testing go hand-in-hand to improve code quality, reduce time to market and reduce overall risk.

ATDD has been around for awhile now but you’d be surprised by the number of people who still don’t know it. At its heart it means that requirements are executable tests. The business can define requirements, which can be executed by a test automation test framework to demonstrably prove that the delivered software meets the functional requirements. That sounds pretty abstract but don’t worry, we’ll end this post with working code and you’ll see for yourself what how this ends up as code.

The amazing thing about ATDD is that everyone speaks the same language, Gherkin, and that we can automate the testing of the delivered software against those requirements. We’ll cover what Gherkin is too and how is gets translated into code.

But making sure the software does what the client has asked is not the end of the story. Code quality is something that the client often does not see and is difficult to measure (see this resource by SeaLights about measuring code quality). Sometimes it surfaces as a bug or instability, but often it simply shows itself by longer delivery times. It’s hard to measure and therefore hard to prioritize. If you can’t measure it or even define it then it can end up being ignored leading to greater and greater problems. That is where Continuous Testing comes in.

Continuous Testing is the next evolution of automated testing. It means we know the current state of the quality of the code being delivered. It combines test executions, test coverage and a bunch of other metrics into a holistic view of your software. It also means that those automated tests of yours are integrated into your CI/CD pipeline and are run continuously.

Used together you know that your software delivers the functionality requested by the client and that it meets the code quality standard necessary to make the software maintainable going forward.

The Case For ATDD

Your colleagues working in the day-to-day business side of your company do not speak the same language as you the developer. When your company asks for a new application or a new feature, they do so in their own language with business terminology mixed in. They have no idea that what they request is ambiguous or infeasible. So requirements end up ambiguous and lacking in the detail required to turn them into software. Not to mention the forgotten requirements.

ATDD addresses this by writing down all requirements in a syntax called Gherkin. Gherkin is a language that is understandable by both the business and the developer. It is normal language but in a structure that allows it to be mapped to code.

Because we can map it to code, we can automate it and create repeatable, automated tests that prove that the software conforms to a functional requirement. Non-functional requirements are another matter and out of scope. So ATDD brings all parties together in a common vision of the project and tools automate the conformance of this vision.

I have worked with ATDD in two different teams and in my experience:

• your developers are happy because they have rigorous requirements that have no room for misinterpretation or vagueness.
• your QA is happy because the boring and repetitive manual testing work has been automated, allowing them to work on exploratory and load testing.
• your client is happy because the software developed will more closely match what they expect and the overall quality is higher.

Also time to market is quicker because:

• There are less back and forths between developers and QA
• Test automation eliminates a lot of slow, manual testing

In this article we are going to build some automated acceptance tests using Visual Studio 2017, MS Test and SpecFlow. The end result will be integration tests that can be executed like any other automated test in a testing pipeline. The differences are that the tests will be the requirements and written in plain English.

SpecFlow and Gherkin

Gherkin is the syntax we’ll use to describe our requirements. It allows us to use normal language that the business can understand but that can be hooked up to a test automation framework for execution. While we cover SpecFlow in this article, the important takeaway is how we use Gherkin. Gherkin is the syntax that all the ATDD frameworks use.

If you have ever used the Arrange, Act, Assert (AAA) pattern for clean unit tests then Gherkin will come easily to you. Gherkin uses Given, When, Then which map onto to AAA directly. The Given describes the current state of the system, When describes the action to be taken and Then describes the expected result.

Challenge: We have been tasked with creating the logic for our site login. Users login with a username and a password. In addition to testing that this basic login functionality works, we want to rate limit login attempts.

Step 1 - Add the SpecFlow extension

As we are using Visual Studio 2017, we need to open the Extensions and Updates window and search for “SpecFlow for Visual Studio 2017” and add it.

Step 2 - Add a test project

Add a new Unit Test Project then add the NuGet package SpecFlow to the project.

Finally ensure that in the app.config we use the Mstest test framework.

<unitTestProvider name="Mstest" />

You could use xUnit or nUnit also.

Step 3 - Create our first feature

SpecFlow stores all the Gherkin tests in feature files. Go ahead click Add New Item and add a feature file to the project called BasicLogin.feature.

Replace the existing text with a new feature description and a single gherkin scenario:

Feature: LoginSuccess
Login usernames and passwords are evaluated against stored usernames and passwords and a login result is returned.

@loginSuccess
Scenario: Correct username and password produces a success response
    Given that the user 'john' exists and his password is 'monkey'
    When the username 'john' with password 'monkey' is supplied
    Then the result should be 'Success'

As you can see the scenario clearly describes the functional behaviour expected. This is easy for a business analyst and a programmer to understand.

Step 4 - Generate C# steps class

Now we need to turn that Gherkin into executable code. Right click on some text in the feature file and click Generate Step Definitions. In the new window click Generate.

You will now have a C# steps class without any logic, but with methods and their arguments bound to the gherkin Given, When, Then lines.

[Binding]
public class BasicLoginSteps
{
    [Given(@"that the user '(.*)' exists and his password is '(.*)'")]
    public void GivenThatTheUserExistsAndHisPasswordIs(string p0, string p1)
    {
        ScenarioContext.Current.Pending();
    }

    [When(@"the username '(.*)' with password '(.*)' is supplied")]
    public void WhenTheUsernameWithPasswordIsSupplied(string p0, string p1)
    {
        ScenarioContext.Current.Pending();
    }

    [Then(@"the result should be '(.*)'")]
    public void ThenTheResultShouldBe(string p0)
    {
        ScenarioContext.Current.Pending();
    }
}

We can run our Gherkin scenario and SpecFlow has hooked it all up and we get string arguments with the values of our Gherkin lines!

Now we need to do something with it. In this toy example, credentials are stored in an in memory dictionary.

[Binding]
public class BasicLoginSteps
{
    [Given(@"that the user '(.*)' exists and his password is '(.*)'")]
    public void GivenThatTheUserExistsAndHisPasswordIs(string p0, string p1)
    {
        CredentialRepository.Credentials.Clear();
        CredentialRepository.Credentials.Add(p0, p1);
    }

    [When(@"the username '(.*)' with password '(.*)' is supplied")]
    public void WhenTheUsernameWithPasswordIsSupplied(string p0, string p1)
    {
        var sut = SutFactory.CreateLoginService();
        var result = sut.Login(p0, p1);
        ScenarioContext.Current.Add("result", result);
    }

    [Then(@"the result should be '(.*)'")]
    public void ThenTheResultShouldBe(string p0)
    {
        Assert.AreEqual(p0, ScenarioContext.Current["result"].ToString());
    }
}

In the Given method we store those credentials in our credential store. If this were real, here we would add those credentials to our real credential store, or mock them.

In the When method we execute the login code with the username and password passed from the Gherkin scenario. We then store the result in the ScenarioContext. Because our test runs across multiple methods, and even multiple classes, we can store state in the ScenarioContext.

In the Then method we compare what we stored in the ScenarioContext with what the Gherkin scenario expects.

What is really nice about this is that I can now add a few more Gherkin scenarios and not change this C# code at all. Which is great for productivity and clean code.

Step 5 - Add more scenarios without the need for more C

We now add three more login scenarios without changing our steps class.

@loginSuccess
Scenario: Correct username with different case and password produces a success response
    Given that the user 'john' exists and his password is 'monkey'
    When the username 'John' with password 'monkey' is supplied
    Then the result should be 'Success'

@loginFailure
Scenario: User does not exist
    Given that the user 'john' exists and his password is 'monkey'
    When the username 'jane' with password '12345' is supplied
    Then the result should be 'UserDoesNotExist'

@loginFailure
Scenario: Password is wrong
    Given that the user 'john' exists and his password is 'monkey'
    When the username 'john' with password '12345' is supplied
    Then the result should be 'PasswordWrong'

Step 6 - More Gherkin Syntax - Backgrounds

So far we have seen the feature description and scenarios. Each scenario has a @tag for organising the tests, and a Given, When, Then structure.

Now we are going to look at Backgrounds. Sometimes you end up with the same Given over and over again. In these cases you can reduce duplication by adding a Background to your feature.

Feature: RateLimiting
    Login attempts of a given user must be limited to a certain amount X within a time period Y. 
    This is important to avoid brute force attacks on our users.

Background:
    Given that the user 'john' exists and his password is 'monkey'
    And that the user 'mick' exists and his password is '123456'

@rateLimiting
Scenario: Number of attempts less than limit
    Given the limit period is 10 seconds
    And the limit in that period is 3
    When 2 logins with 0 seconds delay are attempted with user 'john' and password '123'
    Then the results should be 'WrongPassword,WrongPassword'

@rateLimiting
Scenario: Attempts inside rate limit
    Given the limit period is 10 seconds
    And the limit in that period is 3
    When 4 logins with 4 seconds delay are attempted with user 'john' and password '123'
    Then the results should be 'WrongPassword,WrongPassword,WrongPassword,WrongPassword'

@rateLimiting
Scenario: Attempts exceeds limit
    Given the limit period is 10 seconds
    And the limit in that period is 3
    When 4 logins with 0 seconds delay are attempted with user 'john' and password '123'
    Then the results should be 'WrongPassword,WrongPassword,ReachedRateLimit,ReachedRateLimit'

As we need the user John for each scenario we put it in the feature Background and avoid the repetition. When we generate the steps class you’ll notice that the Background givens are not included as they already exist in the BasicLoginSteps.cs.

Also new in this feature is the use of comma separated list values. But it doesn’t get parsed into a List by default. The C# for the When with the CSV value are bound to a string argument.

[Then(@"the results should be '(.*)']")]
public void ThenTheResultsShouldBe(string p0)
{
    ScenarioContext.Current.Pending();
}

This can be modified to:

[Then(@"the results should be '(.*)']")]
public void ThenTheResultsShouldBe(List<string> p0)
{
    ScenarioContext.Current.Pending();
}

[StepArgumentTransformation]
public List<String> TransformToListOfString(string commaSeparatedList)
{
    return commaSeparatedList.Split(',').ToList();
}

You want that TransformToListOfString method in a separate helper class as it will be need by multiple steps classes. Just remember to add the [Binding] attribute to the class so that SpecFlow can find it.

Step 7 - More Gherkin Syntax - Tables

When we have very data driven acceptance tests, a table is more readable and generally easier to work with.

We can use tables in two ways:

• Make table arguments for our bound methods
• Use tables for executing a scenario multiple times with different variables

Let’s make a table based version of our BasicLogin feature.

Feature: BasicLoginWithTables
    When correct usernames and passwords are supplied a login success
    response is returned

Background:
    Given the following users exist:
    | User | Password |
    | john | monkey   |
    | mick | 123456   |
    | joe  | 654321   |

@TableBased
Scenario: Usernames and passwords produce different responses
    When the username <username> with password <password> is supplied
    Then the result should be <result>

Examples:
    | username | password | result           |
    | john     | monkey   | Success          |
    | john     | 12345    | WrongPassword    |
    | jane     | 123      | UserDoesNotExist |

The background creates the three users and their passwords. The scenario gets executed three times, with the values specified in the Examples table. This makes some tests even easier to read and faster to write.

The C# behind the scenario is the same as our original steps class because it still works with individual variables. Whereas the Given in the Background accepts a Table as a method argument.

[Given(@"the following users exist:")]
public void GivenTheFollowingUsersExist(Table table)
{
    ScenarioContext.Current.Pending();
}

If we create a UserRecord class, we can cast this table to a List. You need to add the line using TechTalk.SpecFlow.Assist; to your class.

[Given(@"the following users exist:")]
public void GivenTheFollowingUsersExist(Table table)
{
    CredentialRepository.Credentials.Clear();
    var users = table.CreateSet<UserRecord>();
    foreach(var user in users)
        CredentialRepository.Credentials.Add(user.Username, user.Password);
}

Step 8 - Add your tests to your testing pipeline.

Because the SpecFlow tests are treated like any other unit or integration tests they integrate into your existing testing pipeline like any other test. You can now get immediate feedback every time there is a code check-in.

Conclusions

Continuous Testing is a must in modern software development. With the use of Acceptance Test Driven Development you see major improvements in code quality and reduced time to market. Frameworks like SpecFlow make it so easy to do! Combine that with other code quality indicators, code coverage and other testing metrics and you reduce the risk involved with each software project. Developers are happier, QA is happier and your client is happier.

Code on GitHub.

Comments

[James Richmond]

Great article!
I have never heard of ATDD before.
Which unit testing/mocking framework do you use?

[Jack Vanlightly]

In my team we're using xUnit as the test runner and Moq as the mocking framework. But in other older codebases other teams in my org are using MsTest and NMock. I have worked in both and I haven't found a huge difference between them. xUnit has a nice way of doing data driven tests but I prefer to use SpecFlow for al my tests.

I find that Gherkin is good for describing my unit tests as well. The unit tests are just for me and my team, but it makes it easier to understand the nuiances between tests.