Introducing TestRail in your K6 tests

Introducing TestRail in your K6 tests

Working smarter than harder or being lazy is the reason why we automate things. Let me try to show you my attempt to optimise TestRail integration with K6...

Use case

In my day-to-day work I am using Azure Pipeline. I will describe the process of Continuous Integration. Some parts might be project-specific, but the general idea and approach won't change much if you are using different technologies.

TestRail

There is a couple of prerequisites that you have to do before we get started.

1. Go to you project and add a Test Suit:
2. After that we want to feed the Suit with the tests. Add some tests to that Suit. Remember that everyone will be using that, so even though you know what TestX is, it is better to give it a more descriptive name:
3. We will need a user that is able to create Test Runs and update the results of the tests. You can use your credentials, but I advise to use (create) the user that whole company, project or team is using during automation tests.
4. If you have a user we should create an API token (that is not mandatory, we can use our password instead, but the password might change...). In upper right corner click on user and go to "My Settings". From there create a token:

TestRail API

Whole documentation can be found here, but we will focus only on Runs and Results.

Creating a Test Run:

1. You will need Project ID (69, in our case), which you can find in the URL when you view the project: company.testrail.io/index.php?/projects/overview/69
2. And Test Suit ID (2137 in our case) we just created: company.testrail.io/index.php?/suites/view/2137

Playing around:

I prefer to have my user:token (or user:password) encoded. So it looks "nice" in the header. Let us encode the user:token here. Now we can send the cUrl:

curl -X POST \
  'https://company.testrail.io/index.php?/api/v2/add_run/69' \
  -H 'Authorization: Basic myTokenHere' \
  -H 'Content-Type: application/json' \
  -d '{
    "suite_id": 2137,
    "name": "Performance Test Run",
    "include_all": true
}'

You just created a new test run! If the above is working for you, we can proceed to the next step. Keep in mind what was the response, we will need that later.

Creating Test Run in our framework

We want to create a Test Run each time we run the scenarios from our Test Suit. Meaning we only want to create it once and then change the results of the tests in that Test Run when our K6 tests are done.
I am using Azure Pipeline so I have a step in my K6 job that looks like this:

- task: PowerShell@2
  name: CreateTestRun
  inputs:
    targetType: 'filePath'
    filePath: $(System.DefaultWorkingDirectory)/scripts/CreateTestRun.ps1
    arguments: -token $(trtoken)

and the content of the file CreateTestRun.ps1 is:

param ($token)

$Uri = "https://company.testrail.io/index.php?/api/v2/add_run/69"
$Body = @{
        suite_id = 2137
        name = "Performance Test Run: $(Get-Date -Format "dd/MM/yyyy, HH:mm")"
        include_all = true
} | ConvertTo-Json
$Headers = @{
        "Authorization" = "Basic $token"
        "Content-Type" = "application/json"
}
$Result = Invoke-RestMethod -Uri $Uri -Method 'Post' -Headers $Headers -Body $Body
$runId = $Result.id

Write-Host "##vso[task.setvariable variable=runId;isOutput=true]$runId"

The script is grabbing and saving the ID of the created Test Run so I can pass it later to K6:

-e RUNID=$(CreateTestRun.runId)
-e TRTOKEN=$(trtoken)

Updating test results in K6

After we created a Test Run and we have its ID we can start our K6 tests. Each time we finish one of our scenarios, we should update the results in that run.
K6 scenario is passed if it passes all of our thresholds. So we have to work around that. After the test a report is created. In each of my scenarios I have a handleSummary function:

import tools from '../tools/tools.js'
import { textSummary } from 'https://jslib.k6.io/k6-summary/0.0.1/index.js';

export function handleSummary(data) {
    const success = tools.CheckThresholds(data, 'iteration_duration')
    tools.UpdateTestrail(success, 142927)
    return {
        'logs/DataGridTest2.json': JSON.stringify(data),
        'stdout': textSummary(data, { indent: ' ', enableColors: true}),
    }
}

Now, there is a couple of things I have to explain. We won't go deep into my framework so I will just explain what we are seeing here.

tools.CheckThresholds()

Is the function that checks the status of my thresholds. It is a custom function that I created and stored in tools.js file:

function CheckThresholds(data, metric) {
    const res = data['metrics'][metric]['thresholds'] ? data['metrics'][metric]['thresholds'] : true
    return Object.values(res).every(obj => obj['ok'])
}

It checks the chosen metric, either built-in or custom.

OK, so we now know if the test is passed or failed. Time to update it! Let us take a look at

tools.UpdateTestrail()

and unfold the mistery what 142927 number means...

function UpdateTestrail(success, caseId) {
    const host = 'https://company.testrail.io'
    let status = 3 //untested

    if (success) {
        status = 1
    } else {
        status = 5
    }

    let body = {
        "results": [
            {
                "case_id": caseId,
                "status_id": status,
            }
        ]
    };
    let params = { 
        headers: {
            Authorization: `Basic ${__ENV.TRTOKEN}`,
            'content-type': 'application/json'
        }
    };

    http.post(`${host}/index.php?/api/v2/add_results_for_cases/${__ENV.RUNID}`,JSON.stringify(body),params);
}

Let us go back to our original Test Suit, which is a base for our Test Run:
Number 142927 is simply one of our Tests!

What happen?

We created Test Run and grabbed its ID.
Then we passed that ID to K6 and run a scenario.
We have thresholds set up so we know when our scenario fails.
After the test is done we updated the results in TestRail.
Now we can proceed to the next scenario, where the only thing that is different is the TestId (142927). And our updated Test Run will look something like that:

Comments

[KsaweryP]

Looks pretty simple and well explained. Good job and thanks :)

[Oliver Brancov]

Hi, @kwidera thank you for this article. I'm consistently getting

error":"Authentication failed: invalid or missing user\/password or session cookie.

When attempting your approach. I'm able to create successful curl requests with authorization like:

'[e-mail]:{mytoken}

But when I try it from my code it doesn't seem to work. Any suggestions? Do I need to do some specific authentication before I run this request? I've tried base64 encoding my Token as well and that didn't work.

export const updateTestrail = (success, caseId) => {
    const host = 'https://mycompany.testrail.io'
    let status = 3

    if (success) {
        status = 1
    } else {
        status = 5
    }

    let body = {
        "results": [
            {
                "case_id": caseId,
                "status_id": status,
            }
        ]
    };
    let params = { 
        headers: {
            'Authorization': `Basic ${__ENV.TRTOKEN}`,
            'Content-Type': 'application/json'
        }
    };

    const testRailResult = http.post(`${host}/index.php?/api/v2/add_results_for_cases/3`, JSON.stringify(body),params);
    console.log(`TestRail APIL ${testRailResult.body} ${testRailResult.status} using token ${__ENV.TRTOKEN}`)