My impressions of using the Drill performance testing tool

Drill is a relatively new performance testing tool. The tool is written in the Rust language, which affects its performance and optimization. In today's article, we will look at its capabilities and see how it performs in a commercial project against other tools.

Idea of the tool

The main idea was to create a lightweight tool and simplicity in writing test scenarios. Therefore, the syntax known from the Ansible tool in yaml format was relied on. This is ultimately supposed to make working on test scenarios more efficient and effective. The advantage of the whole thing is that it is very easy to plug scenarios into CI CD processes.

Writing scenarios

As I mentioned earlier, scenarios are written in yaml format. Let's look at an example of a test scenario. I wanted the scenario to represent as reliably as possible the most frequently performed HTTP operations.

File: benchmark.yml.

---

concurrency: 1
base: 'https://test-api.k6.io'
iterations: 1

plan:
  - name: Fetch public crocs
      request:
      url: /public/crocodiles/1/
    assign: response

  - name: Assert status
    assert:
      key: response.status
      value: 200

  - name: Create a test user
    request:
      url: /user/register/
      method: POST
      body: '{{ item.txn }}'
      headers:
        Content-Type: 'application/json'
    with_items_from_csv:
      file_name: ./fixtures/register.csv
      quote_char: "\'"
    assign: response

  - name: Assert status
    assert:
      key: response.status
      value: 201

  - name: Authenticate the new user
    request:
      url: /auth/token/login/
      method: POST
      body: '{{ item.txn }}'
      headers:
        Content-Type: 'application/json'
    with_items_from_csv:
      file_name: ./fixtures/login.csv
      quote_char: "\'"
    assign: response_auth

  - name: Assert status
    assert:
      key: response_auth.status
      value: 200

  - name: Create a new crocodile
    request:
      url: /my/crocodiles/
      method: POST
      body: '{"name":"Croc Name","sex":"M","date_of_birth":"2019-01-01"}'
      headers:
        Authorization: 'Bearer {{ response_auth.body.access }}'
        Content-Type: 'application/json'
    assign: response_new_crocodile

  - name: Assert status
    assert:
      key: response_new_crocodile.status
      value: 201

  - name: Fetch private crocs
    request:
      url: /my/crocodiles/
      headers:
        Authorization: 'Bearer {{ response_auth.body.access }}'
    assign: response

  - name: Assert status
    assert:
      key: response.status
      value: 200

  - name: Update the croc
    request:
      url: /my/crocodiles/{{ response_new_crocodile.body.id }}
      method: PATCH
      body: '{"name":"New Name"}'
      headers:
        Authorization: 'Bearer {{ response_auth.body.access }}'
        Content-Type: 'application/json'
    assign: response

  - name: Assert status
    assert:
      key: response.status
      value: 200

  - name: Get updated croc
    request:
      url: /my/crocodiles/{{ response_new_crocodile.body.id }}
      headers:
        Authorization: 'Bearer {{ response_auth.body.access }}'
        Content-Type: 'application/json'
    assign: response

  - name: Assert status
    assert:
      key: response.status
      value: 200

  - name: Delete the croc
    request:
      url: /my/crocodiles/{{ response_new_crocodile.body.id }}
        headers:
        Authorization: 'Bearer {{ response_auth.body.access }}'
        Content-Type: 'application/json'
    assign: response

  - name: Assert status
    assert:
      key: response.status
      value: 200

File login.csv:

txn
'{"username":"[e-mail]","password":"superCroc2019"}'

Register.csv file:

txn
'{"username":"[e-mail]","first_name":"Crocodile","last_name":"Owner","password":"superCroc2019"}'

At this stage I also encountered the first problem - scripting is very tricky, error information in the console is unreadable and debugging requests is inefficient.

Another problem is the parametrization of requests. It is impossible to generate data from the test scenario level (as in the Gherkin language known from BDD), which turns out to be a huge problem. We also don't have the possibility of complex data manipulation. All this adds another layer of abstraction in the form of generators written in other languages.

Another problem is the performance of the tool itself. The Rust language was chosen for its performance reasons. In the case of Drill, however, it works inefficiently. This can be seen in the comparison of the tool at https://k6.io. I myself prepare a similar comparison based on resource consumption, discrepancies in the number of requests made and server response times. However, all this deserves a separate article.

Running the scenario

The startup is simple and does not require much work. Let's check what happens after running the script in the console.

drill --benchmark benchmark.yml --stats

Console output:

figaro@pop-os ~/D/p/drill [101]> drill --benchmark benchmark.yml --stats
Concurrency 1
Iterations 1
Rampup 0
Base URL https://test-api.k6.io

Fetch public crocs        https://test-api.k6.io/public/crocodiles/1/ 200 OK 644ms
Assert status             response.status=200?
Create a test user        https://test-api.k6.io/user/register/ 201 Created 3062ms
Assert status             response.status=201?
Authenticate the new user https://test-api.k6.io/auth/token/login/ 200 OK 3113ms
Assert status             response_auth.status=200?
Create a new crocodile    https://test-api.k6.io/my/crocodiles/ 201 Created 204ms
Assert status             response_new_crocodile.status=201?
Fetch private crocs       https://test-api.k6.io/my/crocodiles/ 200 OK 189ms
Assert status             response.status=200?
Error connecting 'https://test-api.k6.io/my/crocodiles/12374994': reqwest::Error { kind: Request, url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("test-api.k6.io")), port: None, path: "/my/crocodiles/12374994", query: None, fragment: None }, source: TimedOut }
Assert status             response.status=200?
Get updated croc          https://test-api.k6.io/my/crocodiles/12374994 200 OK 658ms
Assert status             response.status=200?
Error connecting 'https://test-api.k6.io/my/crocodiles/12374994': reqwest::Error { kind: Request, url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("test-api.k6.io")), port: None, path: "/my/crocodiles/12374994", query: None, fragment: None }, source: TimedOut }
Assert status             response.status=200?

Fetch public crocs        Total requests            1
Fetch public crocs        Successful requests       1
Fetch public crocs        Failed requests           0
Fetch public crocs        Median time per request   647ms
Fetch public crocs        Average time per request  645ms
Fetch public crocs        Sample standard deviation 0ms
Fetch public crocs        99.0'th percentile        647ms
Fetch public crocs        99.5'th percentile        647ms
Fetch public crocs        99.9'th percentile        647ms

Create a test user        Total requests            1
Create a test user        Successful requests       1
Create a test user        Failed requests           0
Create a test user        Median time per request   3064ms
Create a test user        Average time per request  3056ms
Create a test user        Sample standard deviation 0ms
Create a test user        99.0'th percentile        3064ms
Create a test user        99.5'th percentile        3064ms
Create a test user        99.9'th percentile        3064ms

Authenticate the new user Total requests            1
Authenticate the new user Successful requests       1
Authenticate the new user Failed requests           0
Authenticate the new user Median time per request   3129ms
Authenticate the new user Average time per request  3121ms
Authenticate the new user Sample standard deviation 0ms
Authenticate the new user 99.0'th percentile        3129ms
Authenticate the new user 99.5'th percentile        3129ms
Authenticate the new user 99.9'th percentile        3129ms

Create a new crocodile    Total requests            1
Create a new crocodile    Successful requests       1
Create a new crocodile    Failed requests           0
Create a new crocodile    Median time per request   205ms
Create a new crocodile    Average time per request  204ms
Create a new crocodile    Sample standard deviation 0ms
Create a new crocodile    99.0'th percentile        205ms
Create a new crocodile    99.5'th percentile        205ms
Create a new crocodile    99.9'th percentile        205ms

Fetch private crocs       Total requests            1
Fetch private crocs       Successful requests       1
Fetch private crocs       Failed requests           0
Fetch private crocs       Median time per request   189ms
Fetch private crocs       Average time per request  189ms
Fetch private crocs       Sample standard deviation 0ms
Fetch private crocs       99.0'th percentile        189ms
Fetch private crocs       99.5'th percentile        189ms
Fetch private crocs       99.9'th percentile        189ms

Interestingly, the tool at this stage is very unstable. I'm not sure if this is due to the tool's settings, while with 5 other tools I tested, this error did not occur. While writing this article, the thought occurred to me that perhaps the Connection: Keep-Alive header is missing, causing the connection to be closed. However, it doesn't change the fact that on average, it doesn't pass the first step until the 4th time the script is run.

Conclusion

The Drill tool has a beautiful idea behind it. However, it continues to be a relatively fairly new tool, which in its current form is unstable. I will keep an eye on its development, because with more stability in the tool, it may prove to be a good alternative for quick projects.

Sources

https://github.com/fcsonline/drill