Drill is a relatively new performance testing tool. The tool is written in the Rust language, which affects its performance and optimization. In today's article, we will look at its capabilities and see how it performs in a commercial project against other tools.
Idea of the tool
The main idea was to create a lightweight tool and simplicity in writing test scenarios. Therefore, the syntax known from the Ansible tool in yaml format was relied on. This is ultimately supposed to make working on test scenarios more efficient and effective. The advantage of the whole thing is that it is very easy to plug scenarios into CI CD processes.
Writing scenarios
As I mentioned earlier, scenarios are written in yaml format. Let's look at an example of a test scenario. I wanted the scenario to represent as reliably as possible the most frequently performed HTTP operations.
File: benchmark.yml.
---
concurrency: 1
base: 'https://test-api.k6.io'
iterations: 1
plan:
- name: Fetch public crocs
request:
url: /public/crocodiles/1/
assign: response
- name: Assert status
assert:
key: response.status
value: 200
- name: Create a test user
request:
url: /user/register/
method: POST
body: '{{ item.txn }}'
headers:
Content-Type: 'application/json'
with_items_from_csv:
file_name: ./fixtures/register.csv
quote_char: "\'"
assign: response
- name: Assert status
assert:
key: response.status
value: 201
- name: Authenticate the new user
request:
url: /auth/token/login/
method: POST
body: '{{ item.txn }}'
headers:
Content-Type: 'application/json'
with_items_from_csv:
file_name: ./fixtures/login.csv
quote_char: "\'"
assign: response_auth
- name: Assert status
assert:
key: response_auth.status
value: 200
- name: Create a new crocodile
request:
url: /my/crocodiles/
method: POST
body: '{"name":"Croc Name","sex":"M","date_of_birth":"2019-01-01"}'
headers:
Authorization: 'Bearer {{ response_auth.body.access }}'
Content-Type: 'application/json'
assign: response_new_crocodile
- name: Assert status
assert:
key: response_new_crocodile.status
value: 201
- name: Fetch private crocs
request:
url: /my/crocodiles/
headers:
Authorization: 'Bearer {{ response_auth.body.access }}'
assign: response
- name: Assert status
assert:
key: response.status
value: 200
- name: Update the croc
request:
url: /my/crocodiles/{{ response_new_crocodile.body.id }}
method: PATCH
body: '{"name":"New Name"}'
headers:
Authorization: 'Bearer {{ response_auth.body.access }}'
Content-Type: 'application/json'
assign: response
- name: Assert status
assert:
key: response.status
value: 200
- name: Get updated croc
request:
url: /my/crocodiles/{{ response_new_crocodile.body.id }}
headers:
Authorization: 'Bearer {{ response_auth.body.access }}'
Content-Type: 'application/json'
assign: response
- name: Assert status
assert:
key: response.status
value: 200
- name: Delete the croc
request:
url: /my/crocodiles/{{ response_new_crocodile.body.id }}
headers:
Authorization: 'Bearer {{ response_auth.body.access }}'
Content-Type: 'application/json'
assign: response
- name: Assert status
assert:
key: response.status
value: 200
File login.csv:
txn
'{"username":"ue4xxx3vv@backbugs.com","password":"superCroc2019"}'
Register.csv file:
txn
'{"username":"ue4xxx3vv@backbugs.com","first_name":"Crocodile","last_name":"Owner","password":"superCroc2019"}'
At this stage I also encountered the first problem - scripting is very tricky, error information in the console is unreadable and debugging requests is inefficient.
Another problem is the parametrization of requests. It is impossible to generate data from the test scenario level (as in the Gherkin language known from BDD), which turns out to be a huge problem. We also don't have the possibility of complex data manipulation. All this adds another layer of abstraction in the form of generators written in other languages.
Another problem is the performance of the tool itself. The Rust language was chosen for its performance reasons. In the case of Drill, however, it works inefficiently. This can be seen in the comparison of the tool at https://k6.io. I myself prepare a similar comparison based on resource consumption, discrepancies in the number of requests made and server response times. However, all this deserves a separate article.
Running the scenario
The startup is simple and does not require much work. Let's check what happens after running the script in the console.
drill --benchmark benchmark.yml --stats
Console output:
figaro@pop-os ~/D/p/drill [101]> drill --benchmark benchmark.yml --stats
Concurrency 1
Iterations 1
Rampup 0
Base URL https://test-api.k6.io
Fetch public crocs https://test-api.k6.io/public/crocodiles/1/ 200 OK 644ms
Assert status response.status=200?
Create a test user https://test-api.k6.io/user/register/ 201 Created 3062ms
Assert status response.status=201?
Authenticate the new user https://test-api.k6.io/auth/token/login/ 200 OK 3113ms
Assert status response_auth.status=200?
Create a new crocodile https://test-api.k6.io/my/crocodiles/ 201 Created 204ms
Assert status response_new_crocodile.status=201?
Fetch private crocs https://test-api.k6.io/my/crocodiles/ 200 OK 189ms
Assert status response.status=200?
Error connecting 'https://test-api.k6.io/my/crocodiles/12374994': reqwest::Error { kind: Request, url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("test-api.k6.io")), port: None, path: "/my/crocodiles/12374994", query: None, fragment: None }, source: TimedOut }
Assert status response.status=200?
Get updated croc https://test-api.k6.io/my/crocodiles/12374994 200 OK 658ms
Assert status response.status=200?
Error connecting 'https://test-api.k6.io/my/crocodiles/12374994': reqwest::Error { kind: Request, url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("test-api.k6.io")), port: None, path: "/my/crocodiles/12374994", query: None, fragment: None }, source: TimedOut }
Assert status response.status=200?
Fetch public crocs Total requests 1
Fetch public crocs Successful requests 1
Fetch public crocs Failed requests 0
Fetch public crocs Median time per request 647ms
Fetch public crocs Average time per request 645ms
Fetch public crocs Sample standard deviation 0ms
Fetch public crocs 99.0'th percentile 647ms
Fetch public crocs 99.5'th percentile 647ms
Fetch public crocs 99.9'th percentile 647ms
Create a test user Total requests 1
Create a test user Successful requests 1
Create a test user Failed requests 0
Create a test user Median time per request 3064ms
Create a test user Average time per request 3056ms
Create a test user Sample standard deviation 0ms
Create a test user 99.0'th percentile 3064ms
Create a test user 99.5'th percentile 3064ms
Create a test user 99.9'th percentile 3064ms
Authenticate the new user Total requests 1
Authenticate the new user Successful requests 1
Authenticate the new user Failed requests 0
Authenticate the new user Median time per request 3129ms
Authenticate the new user Average time per request 3121ms
Authenticate the new user Sample standard deviation 0ms
Authenticate the new user 99.0'th percentile 3129ms
Authenticate the new user 99.5'th percentile 3129ms
Authenticate the new user 99.9'th percentile 3129ms
Create a new crocodile Total requests 1
Create a new crocodile Successful requests 1
Create a new crocodile Failed requests 0
Create a new crocodile Median time per request 205ms
Create a new crocodile Average time per request 204ms
Create a new crocodile Sample standard deviation 0ms
Create a new crocodile 99.0'th percentile 205ms
Create a new crocodile 99.5'th percentile 205ms
Create a new crocodile 99.9'th percentile 205ms
Fetch private crocs Total requests 1
Fetch private crocs Successful requests 1
Fetch private crocs Failed requests 0
Fetch private crocs Median time per request 189ms
Fetch private crocs Average time per request 189ms
Fetch private crocs Sample standard deviation 0ms
Fetch private crocs 99.0'th percentile 189ms
Fetch private crocs 99.5'th percentile 189ms
Fetch private crocs 99.9'th percentile 189ms
Interestingly, the tool at this stage is very unstable. I'm not sure if this is due to the tool's settings, while with 5 other tools I tested, this error did not occur. While writing this article, the thought occurred to me that perhaps the Connection: Keep-Alive header is missing, causing the connection to be closed. However, it doesn't change the fact that on average, it doesn't pass the first step until the 4th time the script is run.
Conclusion
The Drill tool has a beautiful idea behind it. However, it continues to be a relatively fairly new tool, which in its current form is unstable. I will keep an eye on its development, because with more stability in the tool, it may prove to be a good alternative for quick projects.
Top comments (0)