DEV Community: danieltallentire

Load testing with Atata and Selenoid (Part 1)

danieltallentire — Thu, 15 Apr 2021 22:12:04 +0000

I'm Technical Director at Parker Software, a company with a long history of making market-leading live chat software and automation components.

I'm writing this article as part of CTO West Midlands writing group.

Atata is a great open source project created by @yevgeniyshunevych

What are we doing
Why Atata?
The Solution

What is load testing normally like for WhosOn?

Normally, we run our load tests in WhosOn using our APIs. This allows us to simulate the volume of requests coming in to the WhosOn server, but with a cheap and easy setup.
These are normally load tests like "1,000 concurrent chats" or "500 connected operators".

What is wrong with doing this?

We've recently come across some instances where load has caused a problem, but not to do with the overall capacity. The problem has been caused by behaviours of customers and operators that cause spikes. These spikes may cause an overload in a single component of the system, or a cascade through the system.
The spike behaviour might not be simple - it may have multiple factors.

If your load test is too simple it doesn't capture the real world behaviour of the operators.

What behaviour do we want to replicate?

We would like to create a load test that can simulate a shift change using our operator web client. During shift change, a backlog of queuing chats will build up - this could reach 100 chats in the queue, and as the new shift comes on board, logging in 50-100 users within a minute, those chats are then allocated out to the new operators. During this period, supervisors are also executing a variety of real time reports and pulling metrics against operators who are finishing up their shift, and monitoring the queue lengths to make sure that the active customers get serviced promptly.

Why Atata and not one of the regular load test systems like JMeter?

We already use Atata within our testing pipelines. Our devs are familiar with it.
It allows us to easily vary our load tests, and select tests for different scenarios by altering our coded tests.
It lets us execute tests in a variety of different browsers.
We can record screenshots or video the results if we want to see what the experience is like.

The solution

As part of our Azure DevOps process, we will create a test suite that can be executed for every release.
This will run a two stage process.

test the overall chat system with pick up by a bot.
test the client login system, with pick up by a UI automated chat agent.

We will use Selenoid to run a cluster of selenium instances in docker containers. This could be swapped out for any selenium cluster.

An XUnit test will drive the overall process, and a threshold will be set on various metrics to show whether that test was successful or not.

The test will also output a test file reporting on key metrics from the cluster, including how long the chats took to be handled, end-to-end message time and similar.

The code

I already started an earlier attempt at creating an Atata load test tool a few weeks ago. This was just using a basic set of browser on my box, but I ran into a lot of problems.

Firstly, it killed my computer, trying to pop open new browser windows, and second, it wouldn't be easy to transfer this into the DevOps pipeline.

Setting up Selenoid

I'm running on windows, so a few hoops to jump through to get Selenoid up and running:

update docker
change docker to use linux mode
force docker to use the correct linux context and permissions & 'C:\Program Files\Docker\Docker\DockerCli.exe' -SwitchDaemon
download the correct version of the selenoid configuration manager from https://github.com/aerokube/cm/releases
run ./cm.exe selenoid start to start up selenoid
run ./cm.exe selenoid-ui start to start up the UI so I can see what's happening

Now we have selenoid running, we can open up the selenoid UI on localhost:8080 and see what the connection details are for selenoid:

Building a global context

Based on this, and the Atata documentation, I created a global context driver implementation:

var capabilities = new DesiredCapabilities();
capabilities.SetCapability(CapabilityType.BrowserName, "chrome");
capabilities.SetCapability(CapabilityType.BrowserVersion, "89.0");
var driver = new RemoteWebDriver(new Uri("http://localhost:4444/wd/hub"), capabilities);

AtataContext.GlobalConfiguration
    .UseDriver(driver);

Creating a runner class

Since I need to run multiple Atata instances, I don't want to launch Atata as normal from a test - I need to create some tasks that can run Atata within them and maintain a separate context.

AtataContext.ModeOfCurrent = AtataContextModeOfCurrent.AsyncLocal;

This line of code instructs Atata to ensure that each asynchronous code block has its own instance of the Atata context.

Building the context

private void CreateContext()
{
    _context = null;
    var contextBuilder = AtataContext.Configure();
    _context = contextBuilder.Build();
}

public async Task SetupContext()
{
    await Task.Run(() =>
    {
        CreateContext();
        finishTime = DateTime.Now.AddMinutes(2);
    });
}

This creates the individual context for each LoadTestRunner

Running the test

public async Task Run(CancellationToken cancellationToken)
{
    AtataContext.Current = _context;
    try
    {
        var chatPage = Go.To<Survey>(url: "http://192.168.1.70/newchat/chat.aspx?domain=www.test.com")
            .VisitorName.SetRandom()
            .StartChat.ClickAndGo<Chatting>();

        while (!cancellationToken.IsCancellationRequested)
        {
            if (finishTime <= DateTime.Now)
            {
                chatPage.CloseWindowButton.Click();

                return;
            }

            await Task.Delay(10000);

            chatPage.ChatReply.SetRandom()
                .SendButton.Click();
        }
    }
    catch (Exception ex)
    {
        Console.WriteLine(ex.ToString());
    }
}

This does a really simple start chat function, with only filling in the name field, sending a message every ten seconds, and closing after two minutes.

Time to try it out

I gave it a run, with a single session executing.
Success! It created a session in selenoid, and completed the test successfully - I got a chat popping up.

OK - time to up it to five sessions.

FAIL it only launched one session, and I got a ton of errors.

What went wrong?

Digging in to the debugging, I found that it is only launching one instance of the driver, not multiple as expected.

I changed the global code to:

var capabilities = new DesiredCapabilities();
capabilities.SetCapability(CapabilityType.BrowserName, "chrome");
capabilities.SetCapability(CapabilityType.BrowserVersion, "89.0");

AtataContext.GlobalConfiguration
    .UseDriver(() =>
    {
        return new RemoteWebDriver(new Uri("http://localhost:4444/wd/hub"), capabilities);
    });

This now lets me launch 5 simultaneous sessions.

Can it go further?

I played around with parameters to reduce the memory and CPU usage, but the most I got working was 10 sessions - this partly seems to be to do with my IIS as well - something internally is preventing the sessions from connecting. Indeed, during the test my box prevented a manual chat session request from loading. My cpu was maxed out, and the memory usage was very high. This should work better in the cloud.

This is the selenoid setting I used to create a new instance with limited CPU and memory usage:
./cm.exe selenoid start --vnc --tmpfs 248m -g "-limit 20 -mem 128 -cpu 1.0"

What is next?

For Part 2, I'm going to explore moving this into our Azure DevOps Pipeline.

I need to optimise the memory usage further as well to ensure we can use more of these instances for a quick burst of testing.

Using Atata to create UI tests for user defined web forms

danieltallentire — Fri, 26 Mar 2021 08:58:25 +0000

The Problem

At Parker Software, our chat software allows for customers to create whatever inputs they want to show on the pre-chat survey, which is asked before a user enters the chat:

These fields can be any html field type, including simple text inputs, password fields, radio buttons and drop down lists.

When we setup a new theme for a customer, we want to ensure that the layout and behaviour remains the same when any updates and modifications to the core chat window take place. At the moment this is a manual process whereby the upgrade is performed, and a QA team member will run through the chat window, inspecting it from errors. This manual process is somewhat error prone.

I want to create an automated solution that can work regardless of the fields that have been set in the drop down

We use Atata to do some of our other automated UI testing, and I wanted to have a go at using it to solve this problem.

What is Atata?

Atata is an automated testing framework based on Selenium. You can use C# code to define pages as class files, then really simply set up a wide variety of tests without needing to worry about what the client side looks like.

Thanks to Yevgeniy Shunevych for making Atata so awesome and putting out continuous updates and improvements.

What does it look like?

What are the challenges?

Atata doesn't have a "dynamic" control built in - it usually requires all elements that you want to interact with to be predefined, so I will need to spend some time working on resolving that.

The setup

I'm using Atata with XUnit - we use XUnit for all our unit testing, so Atata has to run through that - it is a little more difficult as Atata doesn't have a native XUnit adapter, but only a few extra changes are needed to integrate with XUnit's logs.

The code

You can get the code at https://github.com/danieltallentire/AtataDynamicForms

Attempt 1

First, I attempted to use the ControlList class with the built in Input control.

using Atata; 

namespace AtataDynamicFormTester 
{ 
    using _ = ChatStartPage; 

    [WaitForLoadingIndicator] 
    class ChatStartPage : Page<_> 
    { 
        [ControlDefinition(ContainingClass = "input-group")] 
        public ControlList<Input<string, _>, _> SurveyFields { get; private set; } 

        public Button<Chatting, _> StartChat { get; private set; } 

        public _ SetRandom() 
        { 
            foreach (var field in SurveyFields) 
            { 
                field.SetRandom(); 
            } 

            return this; 
        } 
    } 
}

Each control on the form is wrapped in a div with the input-group class. This means we can use it with the list locator to find each of the inputs inside.

This came unstuck pretty quickly. It was able to detect and find the elements in the control list, but unable to set a value on them, even with a simple text field.

Attempt 2

My next attempt was to use a simple custom control to wrap the input control. In Atata a custom control is a class derived from the parent Control.

The containing class now moves to part of the control definition.
We can rely on this element existing, so we can iterate the fields.

    [ControlDefinition(ContainingClass = "input-group")] 
    class DynamicControl<TOwner> : Control<TOwner> where TOwner : PageObject<TOwner> 
    { 
        [FindByClass("form-control")] 
        public Input<string, TOwner> TextInput { get; set; } 

        public void SetRandom() 
        { 
            TextInput.SetRandom(); 
        } 
    }

The list is then changed to reference it:

  [ControlDefinition(ContainingClass = "input-group")] 
        public ControlList<DynamicControl<_>, _> SurveyFields { get; private set; }

Success!

Well... some success. This worked fine for simple fields like Name and Company. It got unstuck when it reached an email address, trying to fill it as a random field, and completely ignored the other controls.

My next thing to change was to try to switch based on the input value:

public void SetRandom() 
{ 
    switch (Input.Attributes.Type) 
    { 
        case "text": 
            Input.SetRandom(); 
            break; 
        case "email": 
            Input.Set(Atata.Randomizer.GetString("{0}@{0}.com")); 
            break; 
        case "date": 
            Input.Set("03/03/2021"); 
            break; 
        case "checkbox": 
            Input.Set("true"); 
            break; 
        default: 
            // ignore 
            break; 
    } 
}

OK. Getting closer - this worked nicely for the email address and the date - although its a bit clunky setting the date as a string.
Checkbox didn't work at all, and the select wasn't even detecting.

Attempt 3

Looking again at the classes, I'd forgotten that select elements aren't input elements... d'oh!

I had a play around, and found that I could add multiple elements in to the class:

        [FindByClass("form-control")] 
        public Input<string, TOwner> Input { get; set; } 

        [FindById("fld")] 
        [TermFindSettings(TargetAttributeType = typeof(FindByIdAttribute), Match = TermMatch.StartsWith)] 
        public CheckBox<TOwner> Checkbox { get; set; } 

        [FindByClass("form-control")] 
        public Select<TOwner> Select { get; set; }

And it wouldn't error out for the missing ones in the cases where the control didn't contain those elements, however if I tried to access the control Input on a select element it would throw an exception (rightly, because the Input doesn't exist).

I found that I could create a check to see if the element was found by checking for the Exists property with the IsSafely flag set:

 if (Checkbox.Exists(new SearchOptions() { IsSafely = true }))

This caused a slowdown, so this lead me to the final code with a predefined timeout:

 [ControlDefinition(ContainingClass = "form-group")]
    class DynamicControl<TOwner> : Control<TOwner> where TOwner : PageObject<TOwner>
    {
        [FindByClass("form-control", Timeout = 0.01)]
        public Input<string, TOwner> Input { get; set; }

        // checkboxes aren't wrapped by a nice form-control wrapper, so have to use a different search method
        [FindById("fld", Timeout = 0.01)]
        [TermFindSettings(TargetAttributeType = typeof(FindByIdAttribute), Match = TermMatch.StartsWith)]
        public CheckBox<TOwner> Checkbox { get; set; }

        [FindByXPath(".//*[contains(concat(' ', normalize-space(@class), ' '), ' form-control ')]/descendant-or-self::input[@type='date']", Timeout = 0.01)]
        public DateInput<TOwner> DateInput { get; set; }

        [FindByClass("form-control", Timeout = 0.01)]
        public Select<TOwner> Select { get; set; }

        public void SetRandom()
        {
            if (Checkbox.Exists(new SearchOptions() { IsSafely = true }))
            {
                Checkbox.Set(true);
            }
            else if(Select.Exists(new SearchOptions() { IsSafely = true }))
            {
                Select.Set(Select.Options[Atata.Randomizer.GetInt(0, Select.Options.Count - 1)].Value);
            }
            else if (DateInput.Exists(new SearchOptions() { IsSafely = true}))
            {
                DateInput.SetRandom();
            }
            else if (Input.Exists(new SearchOptions() { IsSafely = true }))
            {
                switch (Input.Attributes.Type)
                {
                    case "text":
                        Input.SetRandom();
                        break;
                    case "email":
                        Input.Set(Atata.Randomizer.GetString("{0}@{0}.com"));
                        break;
                    default:
                        // ignore
                        break;

                }
            }           
        }
    }

I needed to use a manual XPath search for the date input - this was because Atata by default allows for date inputs to be a normal text input with a mask, so was also picking up the default text boxes as date fields too.

I had to add a new randomizer to make the DateInput SetRandom work.
Now I've got the chat window starting nicely.

The containing class was actually incorrect in my earlier examples, and I changed it to form-group - this was caused by the checkboxes not being inside an input-group div.

Just enabling screenshotting in the places I want them:

        public void FillPrechatSurvey()
        {
            Go.To<ChatStartPage>(url: "http://localhost/newchat/chat.aspx?domain=www.parkersoftware.com")
                .Report.Screenshot("Start Chat")
                .SetRandom()
                .Report.Screenshot("Filled Out")
                .StartChat.ClickAndGo()
                .Report.Screenshot("Chatting")
                .Wait(5)
                .CloseWindowButton.Click()
                .Report.Screenshot("Finished");
        }

And the results:

What next?

In order to make this good for production I've got to do some little tweaks - making the resolution and the URL something that can be passed in as an environment variable so it can be run inside DevOps. This is reasonably straight forward.

In the longer term, I want to be able to configure some built in routes and selections to validate that the data sent in the set random is what is received by the server. The WhosOn Echo Bot can send the data back, but I'll have to save the output somehow with SetRandom.

Update

Yevgeniy suggested some refactorings to make my rough code a little neater:

 [ControlDefinition(ContainingClass = "form-group")]
    [FindFirst(TargetAllChildren = true)]
    class DynamicControl<TOwner> : Control<TOwner> where TOwner : PageObject<TOwner>
    {
        // checkboxes aren't wrapped by a nice form-control wrapper, so have to use a different search method
        [FindById(TermMatch.StartsWith, "fld")]
        public CheckBox<TOwner> Checkbox { get; set; }

        public DateInput<TOwner> DateInput { get; set; }

        public EmailInput<TOwner> EmailInput { get; set; }

        public Select<TOwner> Select { get; set; }

        public TextInput<TOwner> TextInput { get; set; }

        public void SetRandom()
        {
            if (Checkbox.IsPresent)
            {
                Checkbox.Set(true);
            }
            else if(Select.IsPresent)
            {
                Select.Set(Select.Options[Atata.Randomizer.GetInt(0, Select.Options.Count - 1)].Value);
            }
            else if (EmailInput.IsPresent)
            {
                EmailInput.Set(Randomizer.GetString("{0}@{0}.com"));
            }
            else if (TextInput.IsPresent)
            {
                TextInput.SetRandom();
            }
            else if (DateInput.IsPresent)
            {
                DateInput.SetRandom();
            }

        }
    }

Changing the inputs to use IsPresent is much more succinct.
Adding the top level attribute for TargetAllChildren means that the class name isn't required.

Using TextInput instead of date input means that the order can be changed to allow a better ordering, and get the correct text or date input.

For this to work I had to add an EmailInput as well to catch type='email'

DEV Community: danieltallentire

Load testing with Atata and Selenoid (Part 1)

Contents

What is load testing normally like for WhosOn?

What is wrong with doing this?

What behaviour do we want to replicate?

Why Atata and not one of the regular load test systems like JMeter?

The solution

The code

Setting up Selenoid

Building a global context

Creating a runner class

Building the context

Running the test

Time to try it out

What went wrong?

Can it go further?

What is next?

Using Atata to create UI tests for user defined web forms

The Problem

What is Atata?

What does it look like?

What are the challenges?

The setup

The code

Attempt 1

Attempt 2

Attempt 3

What next?

Update