DEV Community: Simme

Keyoxide Proof

Simme — Sun, 21 Jan 2024 12:29:17 +0000

My Keyoxide Proof is aspe:keyoxide.org:LHBS3ZA7NH55JKYJFMMKZTQ3DI.

With this you should be able to verify the validity of my identity on the Keyoxide platform.

Observability is becoming mission critical, but who watches the watchmen?

Simme — Wed, 14 Sep 2022 22:18:24 +0000

Before we get started, I just want to get this out of the way: I work at Canonical, and more specifically, I run the observability product team there, currently doing lots of cool stuff around observability in Juju on both K8s and machines. In this piece, I'm actively trying to stay neutral, but it is nonetheless information worth disclosing. I'm also hiring, so if you're also super excited about building world-class observability solutions, don't be shy - apply!

The last couple of years, there has been quite a lot of development in the area of lowering the barrier of entry for observability. There are now quite a few, reasonably mature options out there that lets you set up a good monitoring stack either through a few clicks or by a few one-liners in the terminal.

In the managed open-source space, the most successful one so far probably is Grafana Cloud, but there definitely is no shortage of closed-source vendors providing APM solutions where everything you need to get started is to drop either a single or multiple agents into your cluster or your machine. Even in the case of self-hosted open-source, there are quite a few options available. Observatorium, OpsTrace and COS all provide different degrees of out-of-the-box turn-key experiences, even if the most popular option here remains to roll it yourself, picking the tools you think are the best for the job.

With the increasing interest in observability as a practice, and decreasing barrier of entry, a lot of organisations will, if they haven't already, find out that observability will become more and more critical as their practices improve, to the point where I would argue that it no longer is icing on the cake, making the work of the SREs easier, but mission-critical for their entire business.

Who watches the watchmen?

As this transition in value happens, a new question is starting to gain in importance: who watches the watchmen? Or to put it in words that speak less of my geeky obsession with comics, and more of the topic at hand: what observes the observability stack? How will we be made aware if it is starting to have issues?

A former colleague of mine used to say that "you only get one chance at making a good impression, but for an observability solution, this is especially true". And I truly think they were (are) right! I mean, let's be honest: if you've been burnt by your production monitoring even once, that solution will have a REALLY hard, if not even impossible, time convincing you to trust it again.

Never getting an expected alert might very well mean your critical business services might end up broken without you knowing. To really twist the knife, also imagine that you've not been alerted that the stacks alerting capabilities are broken, or that no telemetry is being collected anymore. While the value of an observability stack is known, most of us don't really put that much effort into making sure our stack itself stays healthy. What happens, for instance, if our log ingesters suddenly start to choke due to a spike in error logs? If our alerting tool starts to crash loop? Would we even notice? I'll go out on a limb and make an educated guess that for many of us, maybe even the vast majority, the answer would be no.

What I'm trying to say here is that while the stability of course is important, the solution does not need to be fault free - it can't be really, just as no software can. What it does need to be, however, is capable of letting you know when its starting to misbehave, so you can take proper action early.

Understanding the failure modes

Before we go into analysing likely failure modes, I just want to make one thing clear: the visualisation tool, like Grafana, is not something I consider to be a critical part of the stack. While useful, as long as the rest of our tooling works, we'd always be able to spin up a new visualisation tool somewhere else and connect our datasources to it.

To keep it short: as long as our alerting continues to work, and the telemetry signals get collected - we're good. We should of course monitor our visualisation tool as well, but from a comparative point of view, it's by far the least important one. Instead, let's focus on two really critical, fairly common failure modes:

Alerts not firing

Let's say we're running a stack where Alertmanager is responsible of alerting. If this (or these) Alertmanager stops working, there will no longer be anything in place to alert us about the fact that it's down (duh). Some would probably argue that this is why you have something like Grafana in place, with dedicated monitoring screens on the wall displaying the state of your solution in realtime. I don't know about you, but I personally forget to look at those screens as soon as I get caught up in something. I also want to be able to get lunch, go to the bathroom every now and then or even refill my cup of coffee.

The solution to this actually is just as simple as it is elegant! We need to set something up, somewhere else, preferably as far away as possible from the stack itself, that continuously receives an always-triggering alert from Alertmanager. In the absence of such an alert, this something will notify you that Alertmanager isn't checking in as expected.

As for any specific tool or service to help you with this, it's totally up to you. Popular solutions include Dead Man's Snitch , Cronitor and Healthchecks.io with the last one being available as open-source in addition to their managed offering. But in reality, you could very well hack something together yourself that would do the job just fine. The important part here is that it needs to serve as a dead man's switch, immediately firing and alerting if your alerting tool fails to check in.

Telemetry missing

We can of course monitor the CPU and memory consumption of our ingesters to give us an early warning of when things are about to go south. We may also monitor and alert on the ingestion rate, using for instance the prometheus_remote_storage_succeeded_samples_total metric. This metrics however, is leaving us a bit vulnerable, as the ingesters being overloaded naturally also will prevent the very same ingesters to ingest metrics about themselves and their own performance.

Just as for the previous failure mode, this one will also require us to alert in the absence of something expected. In this case, rather than the absence of an alert, we want to alert on the absence of telemetry being ingested. PromQL and LogQL both have facilities for this, using absent and absent_over_time. This will allow us to set up an alert rule that tracks the absence of a metric for a certain time range, and when there no longer is any new data points within that range, the alert will trigger. As for the alerting expression, we could use the ingestion rate metric above, or something even simpler like the up metric, wrapping it in an absent function. Anything will do really, as long as it is being ingested regularly.

What's next?

This is by no means an exhaustive list of failure modes for an observability stack. It pinpoints two fairly common and fairly critical scenarios that are easily guarded against.

As your understanding of your observability stack deepens, you'll be able to identify more possible failure modes, and using the telemetry provided by each component of your stack; guard against them. The very same telemetry is not only useful for observing the behaviour of your stack, but also for observing the behaviour of your incident response team. But that will be a topic for some other time.

Error Economics - How to avoid breaking the budget

Simme — Mon, 23 Aug 2021 00:00:00 +0000

At SLOConf 2021 I talked about how we may use error budgets to add pass/fail criterias to reliability tests we run as part of our CI pipelines.

As Site Reliability Engineers, one of our primary goals is to reduce manual labor, or toil, to a minimum while at the same time keeping the systems we manage as reliable and available as possible. To be able to do this in a safe way, it's really important that we're able to easily inspect the state of the system.

To measure whether we're successful in this endeavour, we establish service level agreements (SLA), service level indicators (SLI) and service level objectives (SLO). Traditional monitoring is really helpful in doing this, but it won't allow you to take action until the issue is already present, likely already affecting your users, in prod.

To be able to take action proactively, we may use something like a load generator or reliability testing tool to simulate load in our system, measuring how it behaves even before we've released anything in production. While we'll never will be able to compensate fully for the fact that it won't be running in production, we can still simulate production-like load, possibly even while injecting real-world turbulence into the system, giving us a pretty good picture of what we can expect in production as well.

How do we measure success for site reliability?

When we start out creating these service level artifacts, it's usually tempting to over-engineer them, trying to take every edge case into account. My recommendation is that you try to avoid this to the extent possible.

Instead, aim for a simple set of indicators and objectives, that are generic enough so that you may use them for multiple systems. You may then expand on them and make them more specific as your understanding of the systems you manage increase over time. Doing this is likely to save you a lot of time, as we otherwise tend to come up with unrealistic or irrelevant measurements or requirements, mainly due to our lack of experience.

Service Level Indicators

Service level indicators are quantitative measures of a system's health. To make it easy to remember, we may think of this as what we are measuring. If we, for instance, try to come up with some SLIs for a typical web application, we are likely to end up with things like request duration, uptime, and error rates.

Creating SLIs and SLOs

What should be included? In most cases, we only want to include valid requests. A good formula to follow when crafting SLIs is available in the Google SRE docs. Those state that an SLI equals the amount of good events, divided by the amount of valid events, times a hundred, expressed as percentages.

As an example: if a user decides to send us a request that is not within the defined constraints of the service, we should of course handle it gracefully, letting the user know the request is unsupported. However, we shouldn't be held responsible for the request not being processed properly.

Service Level Objectives

Service level objectives on the other hand, are the targets we set for our SLIs. Think of it as what the measures should show to be OK. For instance, if our SLI is based on request duration, and shows how many percent of all requests are below 500ms, our SLO would express how big a percentage we expect to be below 500ms for our service to be considered to be within the limits.

What is an error budget?

An error budget is the remainder of the SLI once the SLO has been applied. For instance, if our SLO is 99.9%, that would mean our error budget is the remaining .1% up to a 100%. To not breach our SLOs, we then need to be able to fit all events that would not adhere to the criteria we set up into that .1%. This includes outages, service degradations and even planned maintenance windows.

What I'm trying to say is that while it might feel tempting to go for four nines, or even three as your SLO (99.99%, 99,9%), this has astronomic impact on the engineering effort needed. For a downtime/unavailability SLI, a three nine SLO basically means that you can afford as little as:

Daily: 1m 26s
Weekly: 10m 4s
Monthly: 43m 49s
Quarterly: 2h 11m 29s
Yearly: 8h 45m 56s

For comparing "nines", navigate to uptime.is.

In my experience, very few systems are critical enough to motivate this level of availability. With an SLO like this, even with rolling restarts and zero downtime deploys, we can't really afford to make any mistakes at all.

Burning the budget

When would it be acceptable to burn the budget on purpose? I like to use the following sentence as a rule of thumb:

It is only acceptable to burn error budget on purpose if the goal of the activity causing the burn is to reduce the burn-rate going forward.

Setting expectations

Picking our SLIs

In this demo, we'll be testing a made-up online food ordering service called Hipster Pizza. As service level indicators, we'll be using the response time of requests and the HTTP response status success rate.

Picking our SLOs

What would be reasonable SLOs for these SLIs? First we got to ask ourselves if we already have commitments to our customers or users in the form of SLAs. If we do, we at the very least need to stay within that.

However, it's also good to agree on our internal ambitions. And usually, these ambitions turn out to be far less forgiving than whatever we dare to promise our users.

In this example, we'll use the following SLOs:

95% of all valid requests will have a response time below 300ms
99.9% of all valid requests will reply with a successful HTTP Response status.

This means that the error budget for response time is 5%, while the error budget for HTTP success is 0.1%.

Measuring

To know whether we are able to stay within budget, we need to measure this in production. And we also need to assign a time window to our SLOs. For instance, that the SLO is measured on a month-to-month basis, or a sliding 7-day window.

We also need to test this somehow continuously to make sure whether a certain change introduces regression, preventing us from hitting our target. This is where k6, or load generators in general, come in.

Most of the time we only speak about monitoring our SLOs. I would like to propose that we take this one step further. With the traditional approach of monitoring, we're not really equipped to react prior to consuming the budget, especially with the extremely tight budgets we had a look at earlier. Instead we're only going to be alerted once we're already approaching SLO game over.

Don't get me wrong here, I still believe we need, and should, monitor our production SLOs, but we should also complement this with some kind of indicative testing, allowing us to take action before the budget breach has occured. Possibly even stopping the release altogether until the issue has been resolved.

By running a test that simulates the traffic and behavior of users in production, we're able to extrapolate the effect a change would have over time and use that as an indicator of how the change would affect production SLOs.

Before we get into that, however, we also need to talk a bit about scheduled downtime, or maintenance windows.

Accounting for scheduled downtime

In a real-world production system, these likely occur all the time. In some cases, this is possible without requiring any downtime whatsoever, but for the vast majority, some downtime every now and then is unavoidable, even with rolling restarts, canaries, feature flags and red-green deployments in place.

We should put some time into identifying what activities we do that actually require downtime, and account for that in our test. If our SLOs are measured on a month to month basis, and we usually have 10 minutes of downtime every workday, we also need to deduct a corresponding amount from our error budget.

For a month with 31 one days, 22 of them being workdays, 10 minutes of downtime every workday would mean we have a planned downtime of 220 minutes per 744 hours, or 0.0049%.

220/(744*60) = 0,0049%

We'll now adjust the SLOs we use in our test accordingly, prior to calculating the error budget. Heavily simplified, not taking usage volume spread and such into account, this would in our case mean the actual error budgets for our test would be 0,0951% and 4,9951%.

Demo

By using these calculated error budgets, we may then express them as thresholds in our tests, and use them as pass/fail criteria for whether our build was successful or not. And once we have those in our CI workflow, we'll also be able to increase our confidence in product iterations not breaking the error budget.

Let's have a look at how this could look in k6. k6 is available for free and as open-source. Hooking it up with your pre-existing CI pipelines is usually done without any additional cost or significant time investment.

If you're using some other load testing tool that also support setting runtime thresholds, this will likely work just as well there. For this demo, we're gonna use this small test script.

import http from 'k6/http'

export const options = {
  vus: 60,
  duration: '30s'
}

export default function() {
  const res = http.get('https://test-api.k6.io')
}

So what does this script actually do? For a duration of 30 seconds, it will run 50 virtual users in parallel, all visiting the page https://test-api.k6.io as many times as possible. In a real world scenario, this test would most likely be a lot more extensive, and try to mimic a user's interaction with the service we're defining our SLO for.

Let's run our test and have a look at the stats it returns:

http_req_duration..............: avg=132.05ms min=101.44ms med=127.55ms max=284.2ms p(90)=156.19ms p(95)=165.75ms
http_req_failed................: 0.00% ✓ 0 ✗ 6576

As you can see, we already get all the information we need to be able to make out whether we fulfil our SLOs. Let's also define some thresholds to automatically detect whether our test busted our error budget or not.

Let's set those as our thresholds in our k6 script.

  export const options = {
    thresholds: {
      http_req_duration: ['p(95.0049)<300'] // 95% below 300ms, accounting for planned downtime
      http_req_failed: ['rate<0.00951'] // 99,99049% successful, accounting for planned downtime
    }
  }

That's it! By using your SLOs and SLIs as pass/fail thresholds in your CI workflow you'll be able to increase your confidence in product iterations not breaking the error budget.

Running distributed k6 tests on Kubernetes

Simme — Thu, 11 Feb 2021 00:00:00 +0000

📖What you will learn

What the operator pattern is and when it is useful

Deploying the k6 operator in your kubernetes cluster

Running a distributed k6 test in your own cluster

⚠️ Experimental

The project used in this article is experimental and changes a lot between commits. Use at your own discretion .

Introduction

One of the questions we often get in the forum is how to run distributed k6 tests on your own infrastructure. While we believe that running large load tests is possible even when running on a single node, we do appreciate that this is something some of our users might want to do.

There are at least a couple of reasons why you would want to do this:

You run everything else in Kubernetes and would like k6 to be executed in the same fashion as all your other infrastructure components.
You have access to a couple of high-end nodes and want to pool their resources into a large-scale stress test.
You have access to multiple low-end or highly utilized nodes and need to pool their resources to be able to reach your target VU count or Requests per Second (RPS).

Prerequisites

To be able to follow along in this guide, you’ll need access to a Kubernetes cluster, with enough privileges to apply objects.

You’ll also need:

The Kubernetes Operator pattern

The operator pattern is a way of extending Kubernetes so that you may use custom resources to manage applications running in the cluster. The pattern aims to automate the tasks that a human operator would usually do, like provisioning new application components, changing the configuration, or resolving problems that occur.

This is accomplished using custom resources which, for the scope of this article, could be compared to the traditional service requests that you would file to your system operator to get changes applied to the environment.

The operator will listen for changes to, or creation of, K6 custom resource objects. Once a change is detected, it will react by modifying the cluster state, spinning up k6 test jobs as needed. It will then use the parallelism argument to figure out how to split the workload between the jobs using execution segments.

Using the k6 operator to run a distributed load test in your Kubernetes cluster

We'll now go through the steps required to deploy, run, and clean up after the k6 operator.

Cloning the repository

Before we get started, we need to clone the operator repository from GitHub and navigate to the repository root:



$ git clone https://github.com/k6io/operator && cd operator

Deploying the operator

Deploying the operator is done by running the command below, with kubectl configured to use the context of the cluster that you want to deploy it to.

First, make sure you are using the right context:



$ kubectl config get-contexts

CURRENT NAME CLUSTER AUTHINFO NAMESPACE
* harley harley harley
          jean jean jean
          ripley ripley ripley

Then deploy the operator bundle using make. This will also apply the roles, namespaces, bindings and services needed to run the operator.



$ make deploy

/Users/simme/.go/bin/controller-gen "crd:trivialVersions=true" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
cd config/manager && /Users/simme/.go/bin/kustomize edit set image controller=ghcr.io/k6io/operator:latest
/Users/simme/.go/bin/kustomize build config/default | kubectl apply -f -
namespace/k6-operator-system created
customresourcedefinition.apiextensions.k8s.io/k6s.k6.io created
role.rbac.authorization.k8s.io/k6-operator-leader-election-role created
clusterrole.rbac.authorization.k8s.io/k6-operator-manager-role created
clusterrole.rbac.authorization.k8s.io/k6-operator-proxy-role created
clusterrole.rbac.authorization.k8s.io/k6-operator-metrics-reader created
rolebinding.rbac.authorization.k8s.io/k6-operator-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/k6-operator-manager-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/k6-operator-proxy-rolebinding created
service/k6-operator-controller-manager-metrics-service created
deployment.apps/k6-operator-controller-manager created

Writing our test script

Once that is done, we need to create a config map containing the test script. For the operator to pick up our script, we need to name the file test.js. For this article, we’ll be using the test script below:



import http from 'k6/http';
import { check } from 'k6';

export let options = {
  stages: [
    { target: 200, duration: '30s' },
    { target: 0, duration: '30s' },
  ],
};

export default function () {
  const result = http.get('https://test-api.k6.io/public/crocodiles/');
  check(result, {
    'http response status code is 200': result.status === 200,
  });
}

Before we continue, we'll run the script once locally to make sure it works:



$ k6 run test.js

If you’ve never written a k6 test before, we recommend that you start by reading this getting started article from the documentation, just to get a feel for how it works.

Let’s walk through this script and make sure we understand what is happening: We’ve set up two stages that will run for 30 seconds each. The first one will ramp up to linearly to 200 VUs over 30 seconds. The second one will ramp down to 0 again over 30 seconds.

In this case the operator will tell each test runner to run only a portion of the total VUs. For instance, if the script calls for 40 VUs, and parallelism is set to 4, the test runners would have 10 VUs each.

Each VU will then loop over the default function as many times as possible during the execution. It will execute an HTTP GET request against the URL we’ve configured, and make sure that the responds with HTTP Status 200. In a real test, we'd probably throw in a sleep here to emulate the think time of the user, but as the purpose of this article is to run a distributed test with as much throughput as possible, I've deliberately skipped it.

Deploying our test script

Once the test script is done, we have to deploy it to the kubernetes cluster. We’ll use a ConfigMap to accomplish this. The name of the map can be whatever you like, but for this demo we'll go with crocodile-stress-test.

If you want more than one test script available in your cluster, you just repeat this process for each one, giving the maps different names.



$ kubectl create configmap crocodile-stress-test --from-file /path/to/our/test.js

configmap/crocodile-stress-test created

⚠️ Namespaces

For this to work, the k6 custom resource and the config map needs to be deployed in the same namespace.

Let’s have a look at the result:



$ kubectl describe configmap crocodile-stress-test

Name: crocodile-stress-test
Namespace: default
Labels: <none>
Annotations: <none>

Data
====
test.js:
----
import http from 'k6/http';
import { check } from 'k6';

export let options = {
  stages: [
    { target: 200, duration: '30s' },
    { target: 0, duration: '30s' },
  ],
};

export default function () {
  const result = http.get('https://test-api.k6.io/public/crocodiles/');
  check(result, {
    'http response status code is 200': result.status === 200,
  });
}

Events: <none>

The config map contains the content of our test file, labelled as test.js. The operator will later search through our config map for this key, and use its content as the test script.

Creating our custom resource (CR)

To communicate with the operator, we’ll use a custom resource called K6. Custom resources behave just as native kubernetes objects, while being fully customizable. In this case, the data of the custom resource contains all the information necessary for k6 operator to be able to start a distributed load test:



apiVersion: k6.io/v1alpha1
kind: K6
metadata:
  name: k6-sample
spec:
  parallelism: 4
  script: crocodile-stress-test

For Kubernetes to know what to do with this custom resource, we first need to specify what API Version we want to use to interpret its content, in this case k6.io/v1alpha1. We’ll then set the kind to K6, and give our resource a name.

As the specification for our custom resource, we now have the option to use a couple of different properties:

Parallelism

Configures how many k6 test runner jobs the operator should spawn.

Script

The name of the config map containing our script.js file.

Separate

Whether the operator should allow multiple k6 jobs to run concurrently at the same node. The default value for this property is false, allowing each node to run multiple jobs.

Arguments

Allowing you to pass arguments to each k6 job, just as you would from the CLI. For instance --tag testId=crocodile-stress-test-1,--out cloud, or —no-connection-reuse.

Deploying our Custom Resource

We will now deploy our custom resource using kubectl, and by that, start the test:



$ kubectl apply -f /path/to/our/k6/custom-resource.yml

k6.k6.io/k6-sample created

Once we do this, the k6 operator will pick up the changes and start the execution of the test. This looks somewhat along the lines of what is shown in this diagram:

Let’s make sure everything went as expected:



$ kubectl get k6 

NAME AGE
k6-sample 2s

$ kubectl get jobs

NAME COMPLETIONS DURATION AGE
k6-sample-1 0/1 12s 12s
k6-sample-2 0/1 12s 12s
k6-sample-3 0/1 12s 12s
k6-sample-4 0/1 12s 12s

$ kubectl get pods
NAME READY STATUS RESTARTS AGE
k6-sample-3-s7hdk 1/1 Running 0 20s
k6-sample-4-thnpw 1/1 Running 0 20s
k6-sample-2-f9bbj 1/1 Running 0 20s
k6-sample-1-f7ktl 1/1 Running 0 20s

The pods have now been created and put in a paused state until the operator has made sure they’re all ready to execute the test. Once that’s the case, the operator deploys another job, k6-sample-starter which is responsible for making sure all our runners start execution at the same time.

Let’s wait a couple of seconds and then list our pods again:



$ kubectl get pods

NAME READY STATUS RESTARTS AGE
k6-sample-3-s7hdk 1/1 Running 0 76s
k6-sample-4-thnpw 1/1 Running 0 76s
k6-sample-2-f9bbj 1/1 Running 0 76s
k6-sample-1-f7ktl 1/1 Running 0 76s
k6-sample-starter-scw59 0/1 Completed 0 56s

All right! The starter has completed and our tests are hopefully running. To make sure, we may check the logs of one of the jobs:



$ kubectl logs k6-sample-1-f7ktl

[...]

Run [100%] paused
default [0%]

Run [100%] paused
default [0%]

running (0m00.7s), 02/50 VUs, 0 complete and 0 interrupted iterations
default [1%] 02/50 VUs 0m00.7s/1m00.0s

running (0m01.7s), 03/50 VUs, 13 complete and 0 interrupted iterations
default [3%] 03/50 VUs 0m01.7s/1m00.0s

running (0m02.7s), 05/50 VUs, 41 complete and 0 interrupted iterations
default [4%] 05/50 VUs 0m02.7s/1m00.0s

[...]

And with that, our test is running! 🎉 After a couple of minutes, we’re now able to list the jobs again to verify they’ve all completed:



$ kubectl get jobs

NAME COMPLETIONS DURATION AGE
k6-sample-starter 1/1 8s 6m2s
k6-sample-3 1/1 96s 6m22s
k6-sample-2 1/1 96s 6m22s
k6-sample-1 1/1 97s 6m22s
k6-sample-4 1/1 97s 6m22s

Cleaning up

To clean up after a test run, we delete all resources using the same yaml file we used to deploy it:



$ kubectl delete -f /path/to/our/k6/custom-resource.yml

k6.k6.io "k6-sample" deleted

Which deletes all the resources created by the operator as well, as shown below:



$ kubectl get jobs
No resources found in default namespace.

$ kubectl get pods
No resources found in default namespace.

⚠️ Deleting the operator

If you for some reason would like to delete the operator altogether, just run make delete from the root of the project..

The idea behind the operator however, is that you let it remain in your cluster between test executions, only applying and deleting the actual K6 custom resources used to run the tests.

Things to consider

While the operator makes running distributed load tests a lot easier, it still comes with a couple of drawbacks or gotchas that you need to be aware of and plan for. For instance, the lack of metric aggregation.

We’ll go through in detail how to set up the monitoring and visualisation of these test runs in a future article, but for now, here’s a list of things you might want to consider:

Metrics will not be automatically aggregated

Metrics generated by running distributed k6 tests using the operator won’t be aggregated, which means that each test runner will produce its own results and end-of-test summary.

To be able to aggregate your metrics and analyse them together, you’ll either need to:

1) Set up some kind of monitoring or visualisation software and configure your K6 custom resource to make your jobs output there.

2) Use logstash, fluentd, splunk, or similar to parse and aggregate the logs yourself.

Thresholds are not evaluated across jobs at runtime

As the metrics are not aggregated at runtime, your thresholds won’t be evaluated using aggregation either. Currently, the best way to solve this is by setting up alarms for passed thresholds in your monitoring or visualisation software instead.

Overpopulated nodes might create bottlenecks

You want to make sure your k6 jobs have enough cpu and memory resources to actually perform your test. Using parallelism alone might not be sufficient. If you run into this issue, experiment with using the separate property.

Experimental

As mentioned in the beginning of the article, the operator is experimental, and as such it might change a lot from commit to commit.

Total cost of ownership

The k6 operator significantly simplifies the process of running distributed load tests in your own cluster. However, there still is a maintenance burden associated with self-hosting. If you'd rather skip that, as well as the other drawbacks listed above, and instead get straight to load testing, you might want to have a look at the k6 cloud offering.

Performance testing gRPC services

Simme — Thu, 12 Nov 2020 00:00:00 +0000

🎉 New in v0.29.0

v0.29.0 contained a lot of interesting features. Have a look at the release notes for details!

📖What you will learn

What gRPC is.

How gRPC differs from JSON-based REST.

Creating and executing your first gRPC performance test using k6.

Outline

What is gRPC
API Types
The proto definition
Getting started
Summary
See also

What is gRPC

gRPC is a light-weight open-source RPC framework. It was originally developed by Google, with 1.0 being released in August 2016. Since then, it's gained a lot of attention as well as a wide adoption.

In comparison to JSON, which is transmitted as human-readable text, gRPC is binary, making it both faster to transmit and more compact. In the benchmarks we've seen between gRPC and JSON-based REST, gRPC has proved to be a lot faster than its more traditional counterpart.

A benchmark by Auth0 reported up to 6 times higher performance, while other benchmarks, likethis one by Alex Pliutau orthis one by Ruwan Fernando, suggests improvements of up to 10 times.

For chatty, distributed systems, these improvements accumulate quickly, making the difference not only noticeable in benchmarks, but also by the end-user.

API types

gRPC supports four different types of RPCs, unary, server streaming, client streaming, and bi-directional streaming. In reality, the messages are multiplexed using the same connection, but in the spirit of keeping things simple and approachable, this is not illustrated in the gRPC service model diagrams below.

Unary

Unary calls work the same way as a regular function call: a single request is sent to the server which in turn replies with a single response.

Server Streaming

In server streaming mode, the client sends a single request to the server, which in turn replies with multiple responses.

Client Streaming

The client streaming mode is the opposite of the server streaming mode. The client sends multiple requests to the server, which in turn replies with a single response.

Bi-directional streaming

In bi-directional streaming mode, both the client and the server may send multiple messages.

The `.proto` definition

The messages and services used for gRPC are described in .proto files, containingProtocol buffers, or protobuf, definitions.

The definition file is then used to generate code which can be used by both senders and receivers as a contract for communicating through these messages and services. As the binary format used by gRPC lacks any self-describing properties, this is the only way for senders and receivers to know how to interpret the messages.

Throughout this article, we'll use the hello.proto definition available for download on thek6 grpcbin website. For details on how to build your own grpc proto definition, see this excellent article from the official gRPC docs.

// ./definitions/hello.proto

// based on https://grpc.io/docs/guides/concepts.html

syntax = "proto2";

package hello;

service HelloService {
  rpc SayHello(HelloRequest) returns (HelloResponse);
  rpc LotsOfReplies(HelloRequest) returns (stream HelloResponse);
  rpc LotsOfGreetings(stream HelloRequest) returns (HelloResponse);
  rpc BidiHello(stream HelloRequest) returns (stream HelloResponse);
}

message HelloRequest {
  optional string greeting = 1;
}

message HelloResponse {
  required string reply = 1;
}

Getting started

With k6 v0.29.0, we're happy to introduce a native client for gRPC communication. In this early release, we've settled for providing a solid experience for unary calls. If any of the other modes would be particularly useful for you, we'd love to hear about your use case so we can move it up our backlog.

The current API for working with gRPC in k6 using the native client is as follows:

Method	Description
Client.load(importPaths, ...protoFiles)	Loads and parses the given protocol buffer definitions to be made available for RPC requests.
Client.connect(address [,params])	Opens a connection to the given gRPC server.
Client.invoke(url, request [,params])	Makes an unary RPC for the given service/method and returns a Response.
Client.close()	Close the connection to the gRPC service.

Creating the test

The gRPC module is a separate package, available from your test script as k6/net/grpc. Before we can use it, we first have to create an instance of the client. Instantiating the client, as well as the .load operation, is only available during test initialization, ie. directly in the global scope.

import grpc from 'k6/net/grpc';

const client = new grpc.Client();

Next, we'll load a .proto definition applicable for the system under test. For the purpose of this article, we'll use k6 grpcbin. Feel free to change this to whatever you please but keep in mind that you will also need an appropriate .proto definition for the server you're testing. The .load() function takes two arguments, the first one being an array of paths to search for proto files, and the second being the name of the file to load.

import grpc from 'k6/net/grpc';

const client = new grpc.Client();
client.load(['definitions'], 'hello.proto');

Once that is done, we'll go ahead and write our actual test.

import grpc from 'k6/net/grpc';

const client = new grpc.Client();
client.load(['definitions'], 'hello.proto');

export default () => {
  client.connect('grpcbin.test.k6.io:9001', {
    // plaintext: false
  });

  const data = { greeting: 'Bert' };
  const response = client.invoke('hello.HelloService/SayHello', data);

  check(response, {
    'status is OK': (r) => r && r.status === grpc.StatusOK,
  });

  console.log(JSON.stringify(response.message));

  client.close();
  sleep(1);
};

So let's walk through this script to make sure we understand what's happening. First, we use the.connect() function to connect to our system under test. By default, the client will set plaintext to false, only allowing you to use encrypted connections. If you, for any reason, need to connect to a server that lacks SSL/TLS, just flip this setting to true.

We then continue by creating the object we want to send to the remote procedure we're invoking. In the case of SayHello, it allows us to specify who the greeting should address using the greeting parameter.

Next, we invoke the remote procedure, using the syntax <package>.<service>/<procedure>, as described in our proto file. This call is made synchronously, with a default timeout of 60000 ms (60 seconds). To change the timeout, add the key timeout to the config object of .connect() with the duration as the value, for instance '2s' for 2 seconds.

Once we've received a response from the server, we'll then make sure the procedure executed successfully. The grpc module includes constants for this comparison which are listed here

Comparing the response status with grpc.StatusOK, which is 200 OK just like for HTTP/1.1 communication, ensures the call was completed successfully.

We'll then log the message in the response, close the client connection, and sleep for a second.

Executing the test

The test can be executed just like any other test, although you need to make sure you're on at least version v0.29.0 to have access to the gRPC module. To check this, run the following command:

$ k6 version
k6 v0.29.0 ((devel), go1.15.3, darwin/amd64)

Anything less than v0.29.0 here, will require you to first update k6. Instructions on how to do that can be found here.

Once that's out of the way, let's run our test:

$ k6 run test.js

          /\      |‾‾| /‾‾/   /‾‾/
     /\  /  \     |  |/  /   /  /
    /  \/    \    |     (   /   ‾‾\
   /          \   |  |\  \ |  (‾)  |
  / __________ \  |__| \__\ \_____/ .io

  execution: local
     script: /Users/simme/code/grpc/test.js
     output: -

  scenarios: (100.00%) 1 scenario, 1 max VUs, 10m30s max duration (incl. graceful stop):
           * default: 1 iterations for each of 1 VUs (maxDuration: 10m0s, gracefulStop: 30s)

INFO[0000] {"reply":"hello Bert"} source=console

running (00m01.4s), 0/1 VUs, 1 complete and 0 interrupted iterations
default ✓ [======================================] 1 VUs 00m01.4s/10m0s 1/1 iters, 1 per VU

    ✓ status is OK

    checks...............: 100.00% ✓ 1 ✗ 0
    data_received........: 3.0 kB 2.1 kB/s
    data_sent............: 731 B 522 B/s
    grpc_req_duration....: avg=48.44ms min=48.44ms med=48.44ms max=48.44ms p(90)=48.44ms p(95)=48.44ms
    iteration_duration...: avg=1.37s min=1.37s med=1.37s max=1.37s p(90)=1.37s p(95)=1.37s
    iterations...........: 1 0.714536/s
    vus..................: 1 min=1 max=1
    vus_max..............: 1 min=1 max=1

From the output, we can now tell that our script is working and that the server indeed responds with a greeting addressed to who, or what, we supplied in our request body. We can also see that our check was successful, meaning the server responded with 200 OK.

Summary

In this article, we've gone through some of the fundamentals of gRPC and how it works. We've also had a look at the gRPC client introduced in k6 v0.29.0. Last, but not least, we've created a working test script demonstrating this functionality.

And that concludes this gRPC load testing tutorial. Thank you for reading!

Setting up a Kubernetes lab cluster

Simme — Thu, 17 Sep 2020 20:07:28 +0000

This article assumes some experience with containers in general, using for instance docker, as well as a basic understanding of what Kubernetes is and what kind of problems it aims to solve.

Introduction

k3s is a lightweight Kubernetes distribution. It’s been validated by the CNCF as a true Kubernetes distribution, making it a viable option to running the full, original Kubernetes project, especially for labs or dev environments.

What would then be the perks of running k3s over k8s? Well, k8s is designed with really high availability and scalability in mind.

After all, it’s built by Google to support workloads at their scale of operation. If you read some of the best practices on how to design your clusters, you’ll soon realize that it takes a lot of machines just to get started.

Some common best practices include

separating the nodes and the controller plane leaders
scaling out etcd to multiple nodes - or even its own cluster, as well as
separating ingress nodes from regular work nodes to make sure they stay snappy for incoming requests.

While all of this is sound advice, setting up such an architecture could easily have you end up with 2-3 leader instances, 2-3 etcd servers, 1-2 ingress servers. That’s 5 servers in the best case, but more likely 6 or 7. This might feel like overkill for a lab or dev environment, and certainly not feasible for most local labs.

This is where k3s really shines!

It consists of one binary of less than 50MB!
As it is a CNCF certified distribution, it is fully compliant to the more complete, full k8s.
Instead of using etcd, it uses an embedded SQLite database, which is fully sufficient for most local or lab use cases, although not suitable for HA environments.

Creating our virtual machines

As I’m running on macOS, the first thing we need to do is to create a Linux VM capable of running k3s. This can be done easily using multipass, a command-line tool for fast and simple orchestration of Ubuntu VMs.

Let’s create a leader and two nodes:

$ multipass launch \
    --name k3s \
    --cpus 4 \
    --mem 4g \
    --disk 20g

Launched: k3s

$ multipass launch \
    --name k3s-node1 \
    --cpus 1 \
    --mem 1024M \
    --disk 3G

Launched: k3s-node1

$ multipass launch \
    --name k3s-node2 \
    --cpus 1 \
    --mem 1024M \
    --disk 3G

Launched: k3s-node2

Installing k3s

Installation is simple, and if we where installing k3s directly on our machine, we’d get away with just piping the install script to sh like this:

$ curl -sfL https://get.k3s.io | sh -

However, as we’re going to be running k3s in multiple virtual machines using multipass, we need to take a somewhat different approach:

$ multipass exec k3s -- bash -c \
    "curl -sfL https://get.k3s.io | K3S_KUBECONFIG_MODE="644" sh -"

[INFO] Finding release for channel stable
[INFO] Using v1.18.8+k3s1 as release
[INFO] Downloading hash https://github.com/rancher/k3s/releases/download/v1.18.8+k3s1/sha256sum-amd64.txt
[INFO] Downloading binary https://github.com/rancher/k3s/releases/download/v1.18.8+k3s1/k3s
[INFO] Verifying binary download
[INFO] Installing k3s to /usr/local/bin/k3s
[INFO] Creating /usr/local/bin/kubectl symlink to k3s
[INFO] Creating /usr/local/bin/crictl symlink to k3s
[INFO] Creating /usr/local/bin/ctr symlink to k3s
[INFO] Creating killall script /usr/local/bin/k3s-killall.sh
[INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO] env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO] systemd: Creating service file /etc/systemd/system/k3s.service
[INFO] systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO] systemd: Starting k3s

$ export K3S_LEAD_URL="https://$(multipass info k3s | grep "IPv4" | awk -F' ' '{print $2}'):6443"
$ export \
    K3S_TOKEN="$(multipass exec k3s -- /bin/bash -c "sudo cat /var/lib/rancher/k3s/server/node-token")"

With that, our leader is created and the leader URL as well as the node token has been exported to variables. Let’s set up our nodes and add them to the cluster.

$ multipass exec \
    k3s-node1 \
    -- /bin/bash -c \
    "curl -sfL https://get.k3s.io | K3S_TOKEN=${K3S_TOKEN} K3S_URL=${K3S_LEAD_URL} sh -"

[INFO] Finding release for channel stable
[INFO] Using v1.18.8+k3s1 as release
[INFO] Downloading hash https://github.com/rancher/k3s/releases/download/v1.18.8+k3s1/sha256sum-amd64.txt
[INFO] Downloading binary https://github.com/rancher/k3s/releases/download/v1.18.8+k3s1/k3s
[INFO] Verifying binary download
[INFO] Installing k3s to /usr/local/bin/k3s
[INFO] Creating /usr/local/bin/kubectl symlink to k3s
[INFO] Creating /usr/local/bin/crictl symlink to k3s
[INFO] Creating /usr/local/bin/ctr symlink to k3s
[INFO] Creating killall script /usr/local/bin/k3s-killall.sh
[INFO] Creating uninstall script /usr/local/bin/k3s-agent-uninstall.sh
[INFO] env: Creating environment file /etc/systemd/system/k3s-agent.service.env
[INFO] systemd: Creating service file /etc/systemd/system/k3s-agent.service
[INFO] systemd: Enabling k3s-agent unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s-agent.service
  → /etc/systemd/system/k3s-agent.service.
[INFO] systemd: Starting k3s-agent


$ multipass exec \
    k3s-node2 \
    -- /bin/bash -c \
    "curl -sfL https://get.k3s.io | K3S_TOKEN=${K3S_TOKEN} K3S_URL=${K3S_LEAD_URL} sh -"

[INFO] Finding release for channel stable
[INFO] Using v1.18.8+k3s1 as release
[INFO] Downloading hash https://github.com/rancher/k3s/releases/download/v1.18.8+k3s1/sha256sum-amd64.txt
[INFO] Downloading binary https://github.com/rancher/k3s/releases/download/v1.18.8+k3s1/k3s
[INFO] Verifying binary download
[INFO] Installing k3s to /usr/local/bin/k3s
[INFO] Creating /usr/local/bin/kubectl symlink to k3s
[INFO] Creating /usr/local/bin/crictl symlink to k3s
[INFO] Creating /usr/local/bin/ctr symlink to k3s
[INFO] Creating killall script /usr/local/bin/k3s-killall.sh
[INFO] Creating uninstall script /usr/local/bin/k3s-agent-uninstall.sh
[INFO] env: Creating environment file /etc/systemd/system/k3s-agent.service.env
[INFO] systemd: Creating service file /etc/systemd/system/k3s-agent.service
[INFO] systemd: Enabling k3s-agent unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s-agent.service
  → /etc/systemd/system/k3s-agent.service.
[INFO] systemd: Starting k3s-agent

As the last step, we need to copy the kubeconfig file created during setup to our local machine so that we’ll be able to access the cluster from our local machine.

$ multipass exec k3s \
    -- /bin/bash -c \
    "sudo cat /etc/rancher/k3s/k3s.yaml" \
    > k3s.yaml

The kubeconfig file will contain a lot of references to localhost, so we also need to switch those to our leader IP.

$ sed -i '' "s/127.0.0.1/$(multipass info k3s | grep IPv4 | awk '{print $2}')/" k3s.yaml

Let’s install kubectl locally and move our config file to the right location. If you’re not using macOS, you will need to switch this to the appropriate command for your platform.

$ brew install kubectl
$ cp k3s.yaml ~/.kube/config

Let’s try a couple of commands to make sure everything works.

$ kubectl get nodes

NAME STATUS ROLES AGE VERSION
k3s-node2 Ready <none> 18m v1.18.8+k3s1
k3s Ready master 28m v1.18.8+k3s1
k3s-node1 Ready <none> 26m v1.18.8+k3s1

$ kubectl get pods --all-namespaces

NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system metrics-server-7566d596c8-2n65k 1/1 Running 0 29m
kube-system local-path-provisioner-6d59f47c7-z4625 1/1 Running 0 29m
kube-system helm-install-traefik-j26f7 0/1 Completed 0 29m
kube-system coredns-7944c66d8d-gc2r4 1/1 Running 0 29m
kube-system svclb-traefik-f7lrp 2/2 Running 0 28m
kube-system traefik-758cd5fc85-9zmpq 1/1 Running 0 28m
kube-system svclb-traefik-rrpz4 2/2 Running 0 27m
kube-system svclb-traefik-cg6s8 2/2 Running 0 18m

If your output looks somewhat along the lines of this, then awesome! Our lab environment is now ready and we should be able to use it just as we would with any Kubernetes cluster.

Conclusion

In this article, we’ve gone through how to deploy a local Kubernetes lab cluster on our local computer using the incredibly lightweight k3s. We’ve also added two nodes to our new cluster.

Next time, we’ll explore how to do a deployment, how to set up metallb for external IP provisioning, as well as how to use load balancing services to allow access to our deployed pods.

Thank you for reading! 🙏🏼

If you liked this article and would like to read more like it, make sure to press Follow as well as the 💜 button.

In response to COVID-19, the k6 team now offers free access to k6 Cloud to non-profits fighting the pandemic

Simme — Tue, 31 Mar 2020 15:30:00 +0000

The novel Coronavirus is continuing to spread around the world and has now reached most corners of the globe with devastating health, social and economical effects as a consequence, so we hope that you and your loved ones are safe and in good health.

The pandemic will come to an end, and when it does all of us will have been affected. Many will lose a loved one. Many will lose a job or see a business go under. It is our collective actions now that will determine how many. It is a time to show solidarity, as we fight together against Covid-19.

As a globally distributed team, we see the effects this pandemic is having on all of our countries, so as passionate technologists we want to offer a free subscription to our k6 Cloud load testing service to all non-profit organizations and projects working on apps or websites to help their communities, countries and the world to get through this pandemic.

If you are working for an organization or project around Covid-19, or know someone who is, reach out to our support team (or support@k6.io) with details of the project and we’ll help get a subscription setup equivalent to our Pro plan.

As a business, we’re fortunate to be in an industry that can continue to operate (mostly) as usual. Most of our team is normally working remotely, and all of us have been remote for the past couple of weeks, to be on the safe side.

Stay safe.

The k6 Team

Performance Testing with Generated Data using k6 and Faker

Simme — Tue, 17 Mar 2020 15:14:17 +0000

Introduction

A lot of the time, while performance testing, it might not be a huge issue if the data you submit as part of your tests only vary slightly. In some cases, however, you might find yourself in a position where you'd like to keep not only the user interactions but also the data, as realistic as possible. How do we accomplish this without having to maintain long data tables? In this article, we'll explore how we may utilize fakerjs and k6 to perform load tests using realistic generated data.

What is k6?

k6 is an open-source performance testing tool written and maintained by the team at k6. One of the main goals of the project is to provide users with a developer-centered, code-first approach to performance testing.

🤓 Completely new to k6?

Then it might be a good idea to start out with this Beginners guide to k6, written by Mostafa Moradian.

What is Faker?

Faker is a tool used for generating realistic data. It's available for a lot of different languages - python, ruby, php and java to name a few.

In this particular case, we'll use the javascript implementation, fakerjs, as it allows us to use it from within our test script, rather than generating the data before execution.

Goals

Historically performance testing, to a large extent, has been performed by running your test and then manually analyzing the result to spot performance degradation or deviations. k6 uses a different approach, utilizing goal-oriented performance thresholds to create pass/fail tollgates. Let's formulate a scenario (or use case if you prefer) for this test and what it tries to measure.

The Acme Corp Scenario

Acme Corp is about to release a submission form, allowing users to sign up for their newsletter. As they plan to release this form during Black Friday, they want to make sure that it can withstand the pressure of a lot of simultaneous registrations. After all, they are a company in the business of making everything, so they expect a surge of traffic Friday morning.

Our test goals

While we could very well set up complex custom thresholds, it's usually more than enough to stick with the basics. In this case, we'll measure the number of requests where we don't receive an HTTP OK (200) status code in the response, as well as the total duration of each request.

We'll also perform the test with 300 virtual users, which will all perform these requests simultaneously.

Configuration

In k6, we express this as:




const formFailRate = new Rate('failed form fetches');
const submitFailRate = new Rate('failed form submits');

export const options = {
    // ...
    vus: 300,
    thresholds: {
      'failed form submits': ['rate<0.1'],
      'failed form fetches': ['rate<0.1'],
      'http_req_duration': ['p(95)<400'],
    },
};

What does this mean?

So, let's go through what we've done here. With 300 virtual users trying to fetch and submit the subscription form every second, we've set up the following performance goals:

Less than 10% are allowed to fail in retrieving the form
Less than 10% are allowed to fail in submitting the form data
Only 5% or less are permitted to have a request duration longer than 400ms

The actual test

Now, let's get on to the actual test code. The test code, which is executed by each VU once for each iteration, is put inside an anonymous function. We then expose this function as a default export.

The sleep test 😴

To make sure our environment is working, I usually start by setting up a test that does nothing except sleeping for a second and execute it once.



import { sleep } from 'k6';

export default function() {
    sleep(1);
}

Which, when run, produces output similar to this:

Adding our thresholds



import { sleep } from 'k6';
import { Rate } from 'k6/metrics';

const formFailRate = new Rate('failed form fetches');
const submitFailRate = new Rate('failed form submits');

export const options = {
  // ...
  vus: 300,
  duration: '10s',
  thresholds: {
    'failed form submits': ['rate<0.1'],
    'failed form fetches': ['rate<0.1'],
    'http_req_duration': ['p(95)<400'],
  },
};

export default function() {
  formFailRate.add(0);
  submitFailRate.add(0);
  sleep(1);
}

Notice the two new lines in the default function? For each iteration, we're now adding data points to our threshold metrics, telling it that our requests did not fail. We'll hook these up to do something meaningful as we proceed. We also added a duration to make the script run for more than one iteration.

For now, running the script should give you the following output:

Yay, it passes! Two green checks!

Adding requests

To be able to measure anything useful, we also need to add some actual requests. In this example, we'll use https://httpbin.test.loadimpact.com/ as our API, which is our mirror of the popular tool HTTPBin. Feel free to use whatever HTTP Request sink you prefer!



import { sleep } from 'k6';
import { Rate } from 'k6/metrics';
import http from 'k6/http';

const baseUrl = 'https://httpbin.test.loadimpact.com/anything';
const urls = {
  form: `${baseUrl}/form`,
  submit: `${baseUrl}/form/subscribe`,
};


const formFailRate = new Rate('failed form fetches');
const submitFailRate = new Rate('failed form submits');

export const options = {
    vus: 300,
    duration: '10s',
    thresholds: {
      'failed form submits': ['rate<0.1'],
      'failed form fetches': ['rate<0.1'],
      'http_req_duration': ['p(95)<400'],
    },
};

const getForm = () => {
  const formResult = http.get(urls.form);
  formFailRate.add(formResult.status !== 200);
}

const submitForm = () => {
  const submitResult = http.post(urls.submit, {});
  submitFailRate.add(submitResult.status !== 200);
}

export default function() {
  getForm();
  submitForm();
  sleep(1);
}

And once again:

The output now also includes metrics around our HTTP requests, as well as a little green check next to the duration.

Adding Bundling and Transpiling

Now that we've got our script to work, it's almost time to add faker. Before we do that, we need to make sure that k6 can use the faker library.

As k6 does not run in a NodeJS environment, but rather in a goja VM, it needs a little help. Thankfully, it's not that complex. We'll use webpack and babel to achieve this, but any bundler compatible with babel would likely work.

Let's start by initializing an npm package and add all the dependencies we'll need:



$ yarn init -y && yarn add \
    @babel/core \
    @babel/preset-env \
    babel-loader \
    core-js \
    webpack \
    webpack-cli

We'll then create our webpack config. The details of webpack and babel are outside the scope of this article, but there are plenty of great resources out there on how it works.



// webpack.config.js

module.exports = {
    mode: 'production',
    entry: './src/index.js',
    output: {
        path: __dirname + '/dist',
        filename: 'test.[name].js',
        libraryTarget: 'commonjs'
    },
    module: {
        rules: [
            { test: /\.js$/, use: 'babel-loader' },
        ]
    },
    stats: {
        colors: true
    },
    target: "web",
    externals: /k6(\/.*)?/,
    devtool: 'source-map',
}

and the .babelrc file:



{
    "presets": [
      [
        "@babel/preset-env",
        {
          "useBuiltIns": "usage",
          "corejs": 3
        }
      ]
    ]
  }

We'll also modify our package.json so that we can launch our tests using yarn:



{
  "name": "k6-faker",
  "scripts": {
+   "pretest": "webpack",
+   "test": "k6 run ./dist/test.main.js"
  },
  ...
}

🧠 Did you know?

Using pre or post at the beginning of a script name, results in that script running before/after the script you're invoking. In this case, the pretest script ensures that every time we run our test, webpack first creates a new, fresh bundle from the source code. - Sweet, huh? 👍🏻

Enter Faker!

Let's get right into it then! The first step is to add faker to our dependencies:



$ yarn add faker

Faker has a quite extensive library of data that it's able to generate, ranging from company details to catchphrases and profile pictures. While these are all handy to have, we'll only use a tiny subset of what faker has to offer. Our object follows this structure:



{
  name: 'jane doe',
  title: 'intergalactic empress',
  company: 'Worldeaters Inc',
  email: 'jane@doe.example',
  country: 'N/A'
}

We'll now go ahead and create a service that we may use to generate said persons:



// subscriber.js

import * as faker from 'faker/locale/en_US'; 

export const generateSubscriber = () => ({
    name: `SUBSCRIPTION_TEST - ${faker.name.firstName()} ${faker.name.lastName()}`,
    title: faker.name.jobTitle(),
    company: faker.company.companyName(),
    email: faker.internet.email(),
    country: faker.address.country()
});

👿 Possible performance issues ahead!

All dependencies added tend to balloon the memory consumption to some extent, especially when they scale up to 300 concurrent instances. Because of this, it's crucial that we only import the locale(s) we are using in our test case.

While putting together the example repository for this article, I noticed that using faker adds about 2.3MB of memory per VU, which for 300 VUs resulted in a total memory footprint of around 1.5GB.

You can read more about the javascript performance in k6 and how to tune it here.

You might have noticed that we prepend the name of the generated user with SUBSCRIPTION_TEST. Adding a unique identifier for your test data is just something I find convenient to be able to quickly filter out all dummy data I've created as part of a test. While optional, this is usually a good idea - especially if you test against an environment that you can't easily prune.

Final assembly

Now, let's put it all together!



// index.js

import { sleep } from 'k6';
import http from 'k6/http';
import { Rate } from 'k6/metrics';

import { generateSubscriber } from './subscriber';

const baseUrl = 'https://httpbin.test.loadimpact.com/anything';
const urls = {
  form: `${baseUrl}/form`,
  submit: `${baseUrl}/form/subscribe`,
};

const formFailRate = new Rate('failed form fetches');
const submitFailRate = new Rate('failed form submits');

export const options = {
  vus: 300,
  duration: '10s',
  thresholds: {
    'failed form submits': ['rate<0.1'],
    'failed form fetches': ['rate<0.1'],
    'http_req_duration': ['p(95)<400']
  }
};

const getForm = () => {
    const formResult = http.get(urls.form);
    formFailRate.add(formResult.status !== 200);
}

const submitForm = () => {
    const person = generateSubscriber();    
    const payload = JSON.stringify(person);

    const submitResult = http.post(urls.submit, payload);
    submitFailRate.add(submitResult.status !== 200);
}

export default function() {
    getForm();
    submitForm();
    sleep(1);
}



// subscriber.js

import * as faker from 'faker/locale/en_US'; 

export const generateSubscriber = () => ({
    name: `SUBSCRIPTION_TEST - ${faker.name.firstName()} ${faker.name.lastName()}`,
    title: faker.name.jobTitle(),
    company: faker.company.companyName(),
    email: faker.internet.email(),
    country: faker.address.country()
});

And with that, we're ready to go:

Closing thoughts

While the flexibility you get by combining the javascript engine used in k6 with webpack and babel is near endless, it's essential to keep track of the memory consumption and performance of the actual test. After all, getting false positives due to our load generator being out of resources is not particularly helpful.

All the code from this article is available as an example repository on
GitHub, which I try to keep up to date with new versions of k6 and faker.

I'd love to hear your thoughts, so please hit me up with questions and comments in the field below. 👇🏼

What is the favorite thing you keep on your desk?

Simme — Sat, 14 Mar 2020 14:24:44 +0000

Keeping figurines, rubber ducks, alarm clocks and other kinds of decoration on ones desk seems to be quite common these days.

It's not hard to get that you'd like to pimp out the place where you spend roughly eight hours every day.

I'm curious as to what you keep on your desks, and more specifically; which one your favorite is, and why?

My favorite? A little vault boy bobblehead that I bought at a carnival. Why? He's happy, always seems to dig the music I play (he's a bobblehead after all) and always prepared to take on whatever life throws at him.

Open source load testing tool review 2020

Simme — Wed, 04 Mar 2020 00:00:00 +0000

This article is a guest post, written by Ragnar Lönn.

Comparing the best open source load testing tools since 2017!

It has been almost three years since we first published our first comparison & benchmark articles that have become very popular, and we thought an update seemed overdue as some tools have changed a lot in the past couple of years. For this update, we decided to put everything into one huge article - making it more of a guide for those trying to choose a tool.

First, a disclaimer: I, the author, have tried to be impartial, but given that I helped create one of the tools in the review (k6), I am bound to have some bias towards that tool. Feel free to read between the lines and be suspicious of any positive things I write about k6 ;)

About the review

The list of tools we look at hasn’t changed much. We have left out The Grinder from the review because despite being a competent tool that we like, it doesn’t seem to be actively developed anymore, making it more troublesome to install (it requires old Java versions) and it also doesn’t seem to have many users out there. A colleague working with k6 suggested we’d add a tool built with Rust and thought Drill seemed a good choice, so we added that to the review. Here is the full list of tools tested, and what versions we have tested:

Apachebench 2.3
Artillery 1.6.0
Drill 0.5.0 (new)
Gatling 3.3.1
Hey 0.1.2
Jmeter 5.2.1
k6 0.26.0
Locust 0.13.5
Siege 4.0.4
Tsung 1.7.0
Vegeta 12.7.0
Wrk 4.1.0

So what did we test then?

Basically, this review centers around two things:

1. Tool performance
How efficient is the tool at generating traffic and how accurate are its measurements?

2. Developer UX
How easy and convenient is the tool to use, for a developer like myself?

Automating load tests is becoming more and more of a focus for developers who do load testing, and while there wasn’t time to properly integrate each tool into a CI test suite, the author tried to figure out how well suited a tool is to automated testing by downloading, installing and running each tool from the command line and via scripted execution.

The review contains both hard numbers for e.g. tool performance, but also a lot of very subjective opinions from the author on various aspects, or behaviour, of the tools.

All clear? Let’s do it! The rest of the article is written in first-person format to make it hopefully more engaging (or at least you’ll know who to blame when you disagree with something).

Chapters

Chapter 1: History and status
Where did the tools come from and which ones are actively developed/maintained?

Chapter 2: Usability review
What functionality do they have and how easy are they to use for a developer?

The top non-scriptable tools

The top scriptable tools

The rest of the tools

Chapter 3: Performance review and benchmarks
How efficient are the tools at generating traffic and how accurate are their measurements?

Max traffic generation capability

Memory usage

Measurement accuracy

End summary

Chapter 1: History and status

Tool overview

Here is a table with some basic information about the tools in the review.

Tool	Apachebench	Artillery	Drill	Gatling
Created by	Apache foundation	Shoreditch Ops LTD	Ferran Basora	Gatling Corp
License	Apache 2.0	MPL2	GPL3	Apache 2.0
Written in	C	NodeJS	Rust	Scala
Scriptable	No	Yes: JS	No	Yes: Scala
Multithreaded	No	No	Yes	Yes
Distributed load generation	No	No (Premium)	No	No (Premium)
Website	httpd.apache.org	artillery.io	github.com/fcsonline	gatling.io
Source code	svn.apache.org	github.com/artilleryio	github.com/fcsonline	github.com/gatling

Tool	Hey	JMeter	k6	Locust
Created by	Jaana B Dogan	Apache foundation	Load Impact	Jonathan Heyman
License	Apache 2.0	Apache 2.0	AGPL3	MIT
Written in	Go	Java	Go	Python
Scriptable	No	Limited (XML)	Yes: JS	Yes: Python
Multithreaded	Yes	Yes	Yes	No
Distributed load generation	No	Yes	No (Premium)	Yes
Website	github.com/rakyll	jmeter.apache.org	k6.io	locust.io
Source code	github.com/rakyll	github.com/apache	loadimpact@github	locustio@github

Tool	Siege	Tsung	Vegeta	Wrk
Created by	Jeff Fulmer	Nicolas Niclausse	Tomás Senart	Will Glozer
License	GPL3	GPL2	MIT	Apache 2.0 modified
Written in	C	Erlang	Go	C
Scriptable	No	Limited (XML)	No	Yes: Lua
Multithreaded	Yes	Yes	Yes	Yes
Distributed load generation	No	Yes	Limited	No
Website	joedog.org	erland-projects.org	tsenart@github	wg@github
Source code	JoeDog@github	processone@github	tsenart@github	wg@github

Development status

OK, so which tools are being actively developed today, early 2020?

I looked at the software repositories of the different tools and counted commits and releases since late 2017 when I did the last tool review. Apachebench doesn't have its own repo but is a part of Apache httpd so I skipped it here as Apachebench is fairly dead, development-wise anyway.

It's positive to see that several of the projects seem to be moving fast! Jmeter could do with more frequent releases perhaps? Locust seems to have picked up speed the past year, as it had only 100 commits and one release in 2018, but in 2019 it had 300 commits and 10 releases. And looking at the sheer number of commits, Gatling, Jmeter and k6 seem to be moving very fast.

Looking at Artillery gives me the feeling that the open source version get a lot less attention than the premium version. Reading the Artillery Pro Changelog (there seems to be no changelog for Artillery open source) it looks as if Artillery Pro has gotten a lot of new features the past two years, but when checking commit messages in the Github repo of the open source Artillery, I see what looks mostly like occasional bug fixes.

Apachebench

This old-timer was created as part of the tool suite for the Apache httpd webserver. It's been around since the late 90's and was apparently an offshoot of a similar tool created by Zeus Technology, to test the Zeus web server (an old competitor to Apache's and Microsofts web servers). Not much is happening with Apachebench these days, development-wise, but due to it being available to all who install the tool suite for Apache httpd, it is very accessible and most likely used by many, many people to run quick-and-dirty performance tests against e.g. a newly installed HTTP server. It might also be used in quite a few automated test suites.

Artillery

Shoreditch Ops LTD in London created Artillery. These guys are a bit anonymous, but I seem to remember them being some kind of startup that pivoted into load testing either before or after Artillery became popular out there. Of course, I also remember other things that never happened, so who knows. Anyway, the project seems to have started sometime 2015 and was named "Minigun" before it got its current name.

Artillery is a written in Javascript, and using NodeJS as its engine.

Drill

Drill is the very newest newcomer of the bunch. It appeared in 2018 and is the only tool written in Rust. Apparently, the author - Ferran Basora - wrote it as a side project in order to learn Rust.

Gatling

Gatling was first released in 2012 by a bunch of former consultants in Paris, France, who wanted to build a load testing tool that was better for test automation. In 2015 Gatling Corp was founded and the next year the premium SaaS product "Gatling Frontline" was released by Gatling Corp. On their web site they say they have seen over 3 million downloads to date - I'm assuming this is downloads of the OSS version.

Gatling is written in Scala, which is a bit weird of course, but it seems to work quite well anyway.

Hey

Hey used to be named Boom, after a Python load testing tool of that name, but the author apparently got tired of the confusion that caused, so she changed it. The new name keeps making me think "horse food" when I hear it, so I'm still confused, but the tool is quite ok. It's written in the fantastic Go language, and is fairly close to Apachebench in terms of functionality. The author stated that one aim when she wrote the tool was to replace Apachebench.

Jmeter

This is the old giant of the bunch. It also comes from the Apache software foundation, is a big, old Java app and has a ton of functionality, plus it is still being actively developed. The last two years it has seen more commits to its codebase than any other tool in the review. I suspect that Jmeter is slowly losing market share to newer tools, like Gatling, but given how long it's been around and how much momentum it still has, it's a sure bet that it'll be here a long time yet. There are so many integrations, add-ons etc for Jmeter, and whole SaaS services built on top of it (like Blazemeter), plus people have spent so much time learning how to use it, that it will be going strong for many more years.

A super-awesome tool! Uh, well, like I wrote earlier I am somewhat biased here. But objective facts are these: k6 was released in 2017, so is quite new. It is written in Go, and a fun thing I just realized is that we then have a tie between Go and C - three tools in the review are written in C, and three in Go. My two favourite languages - is it coincidence or a pattern?!

k6 was originally built, and is maintained by, Load Impact - a SaaS load testing service. Load Impact has several people working full time on k6 and that, together with community contributions, means development is very active. Less known is why this tool is called "k6" but I'm happy to leak that information here: after a lengthy internal name battle that ended in a standoff, we had a 7-letter name starting with "k" that most people hated, so we shortened it to "k6" and that seemed to resolve the issue. You gotta love first-world problems!

Locust

Locust is a very popular load testing tool that has been around since at least 2011, looking at the release history. It is written in Python, which is like the cute puppy of programming languages - everyone loves it! This love has made Python huge, and Locust has also become very popular as there aren't really any other competent load testing tools that are Python-based (and Locust is scriptable in Python too!)

Locust was created by a bunch of swedes who needed the tool themselves. It is still maintained by the main author, Jonathan Heyman, but now has many external contributors also. Unlike e.g. Artillery, Gatling and k6, there is no commercial business steering the development of Locust - it is (as far as I know) a true community effort. Development of Locust has been alternating between very active and not-so-active - I'm guessing it depends on Jonathan's level of engagement mainly. After a lull in 2018, the project has seen quite a few commits and releases the past 18 months or so.

Siege

Siege has also been around quite a while - since the early 2000's sometime. I'm not sure how much it is used but it is referenced in many places online. It was written by Jeff Fulmer and is still maintained by him. Development is ongoing, but a long time can pass between new releases.

Tsung

Our only Erlang contender! Tsung was written by Nicolas Niclausse and is based on an older tool called IDX-Tsunami. It is also old - i.e. from the early 2000's and like e.g. Siege still developed but in a snail-like manner.

Vegeta

Vegeta is apparently some kind of Manga superhero, or something. Damn it, now people will understand how old I am. Really, though, aren't all these aggressive-sounding names and word choices used for load testing software pretty silly? Like, you do vegeta attack ... to start a load test. And don't get me started on "Artillery", "Siege", "Gatling" and the rest. Are we trying to impress an audience of five-year olds? "Locust" is at least a little better (though the "hatching" and "swarming" it keeps doing is pretty cheesy)

See? Now I went off on a tangent here. Mental slap! OK, back again. Vegeta seems to have been around since 2014, it's also written in Go and seems very popular (almost 14k stars on Github! The very popular Locust, for reference, has about 12k stars). The author of Vegeta is Tomás Senart and development seems quite active.

Wrk

Wrk is written in C, by Will Glozer. It's been around since 2012 so isn't exactly new, but I have been using it as kind of a performance reference point because it is ridiculously fast/efficient and seems like a very solid piece of software in general. It actually has over 23k stars on Github also, so it probably has a user base that is quite large even though it is less accessible than many other tools (you'll need to compile it). Unfortunately, Wrk isn't so actively developed. New releases are rare.

I think someone should design a logotype for Wrk. It deserves one.

Chapter 2: Usability review

I'm a developer, and I generally dislike point-and-click applications. I want to use the command line. I also like to automate things through scripting. I'm impatient and want to get things done. I'm kind of old, which in my case means I'm often a bit distrustful of new tech and prefer battle-proven stuff. You're probably different, so try to figure out what you can accept that I can't, and vice versa. Then you might get something out of reading my thoughts on the tools.

What I've done is to run all the tools manually, on the command line, and interpreted results either printed to stdout, or saved to a file. I have then created shellscripts to automatically extract and collate results.

Working with the tools has given me some insight into each tool and what its strengths and weaknesses are, for my particular use case. I imagine that the things I'm looking for are similar to what you're looking for when setting up automated load tests, but I might not consider all aspects, as I haven't truly integrated each tool into some CI test suite (that may be the next article to write). Just a disclaimer.

Also, note that the performance of the tools has coloured the usability review - if I feel that it's hard for me to generate the traffic I want to generate, or that I can't trust measurements from the tool, then the usability review will reflect that. If you want details on performance you'll have to scroll down to the performance benchmarks, however.

RPS

You will see the term RPS used liberally throughout this blog article. That acronym stands for "Requests Per Second", a measurement of how much traffic a load testing tool is generating.

VU

This is another term used quite a lot. It is a (load) testing acronym that is short for "Virtual User". A Virtual User is a simulated human/browser. In a load test, a VU usually means a concurrent execution thread/context that sends out HTTP requests independently, allowing you to simulate many simultaneous users in a load test.

Scriptable tools vs non-scriptable ones

I've decided to make a top list of my favourites both for tools that support scripting, and for those that don't. The reason for this is that whether you need scripting or not depends a lot on your use case, and there are a couple of very good tools that do not support scripting, that deserve to be mentioned here.

What's the difference between a scriptable and a non-scriptable tool?

A scriptable tool supports a real scripting language that you use to write your test cases in - e.g. Python, Javascript, Scala or Lua. That means you get maximum flexibility and power when designing your tests - you can use advanced logic to determine what happens in your test, you can pull in libraries for extra functionality, you can often split your code into multiple files, etc. It is, really, the "developer way" of doing things.

Non-scriptable tools, on the other hand, are often simpler to get started with as they don't require you to learn any specific scripting API. They often also tend to be a bit less resource-consuming than the scriptable tools, as they don't have to have a scripting language runtime and execution contexts for script threads. So they are faster and consume less memory (generally, not true in all cases). The negative side is they're more limited in what they can do.

OK, let's get into the subjective tool review!

The top non-scriptable tools

Here are my favourite non-scriptable tools, in alphabetical order.

Hey

Hey is a simple tool, written in Go, with good performance and the most common features you'll need to run simple static URL tests. It lacks any kind of scripting, but can be a good alternative to tools like Apachebench or Wrk, for simple load tests. Hey supports HTTP/2, which neither Wrk nor Apachebench does, and while I didn't think HTTP/2 support was a big deal in 2017, today we see that HTTP/2 penetration is a lot higher than back then, so today it's more of an advantage for Hey, I'd say.

Another potential reason to use Hey instead of Apachebench is that Hey is multi-threaded while Apachebench isn't. Apachebench is very fast, so often you will not need more than one CPU core to generate enough traffic, but if you do, then you'll be happier using Hey as its load generation capacity will scale pretty much linearly with the number of CPU cores on your machine.

Hey has rate limiting, which can be used to run fixed-rate tests.

Hey help output

Hey summary

Hey is simple, but it does what it does very well. It's stable, among the more performant tools in the review, and has very nice output with response time histograms, percentiles and stuff. It also has rate limiting, which is something many tools lack.

Vegeta

Vegeta has a lot of cool features like the fact that its default mode is to send requests at a constant rate and it adapts concurrency to try and achieve this rate. This is very useful for regression/automated testing, where you often want to run tests that are as identical to eachother as possible, as that will make it more likely that any deviating results are the result of a regression in newly committed code.

Vegeta is written in Go (yay), performs very well, supports HTTP/2, has several output formats, flexible reporting and can generate graphical response time plots.

If you look at the runtime screenshot above, you'll see that it is quite obvious that Vegeta was designed to be run on the command line; it reads from stdin a list of HTTP transactions to generate, and sends results in binary format to stdout, where you're supposed to redirect to a file or pipe them directly to another Vegeta process that then generates a report from the data.

This design provides a lot of flexibility and supports new use cases like e.g. basic load distribution through remote shell-execution of Vegeta on different hosts and then copying the binary output from each Vegeta "slave" and piping it all into one Vegeta process that generates a report. You also "feed" (over stdin) Vegeta its list of URLs to hit, which means you could have a piece of software executing complex logic that generates this list of URLs (though that program would not have access to the results of transactions, so it is doubtful how useful such a setup would be I guess).

The slightly negative side is that the command line UX is not what you might be used to, if you've used other load testing tools, and neither is it the simplest possible, if you just want to run a quick command-line test hitting a single URL with some traffic.

Vegeta help output

Vegeta summary

Overall, Vegeta is a really strong tool that caters to people who want a tool to test simple, static URLs (perhaps API end points) but also want a bit more functionality. Or people who want to assemble their own load testing solution and need a flexible load generator component that they can use in different ways. Vegeta can even be used as a Golang library/package if you want to create your own load testing tool.

The biggest flaw (when I'm the user) is the lack of programmability/scripting, which makes it a little less developer-centric.

I would definitely use Vegeta for simple, automated testing.

Wrk

Wrk may be a bit dated, and doesn't get a lot of new features these days, but it is such a !#&%€ solid piece of code. It always behaves like you expect it to, and it is running circles around all other tools in terms of speed/efficiency. If you use Wrk you will be able to generate 5 times as much traffic as you will with k6, on the same hardware. If you think that makes k6 sound bad, think again because it is not that k6 is slow. It's just that Wrk is so damn fast. Comparing it to other tools, Wrk is 10 times faster than Gatling. 15-20 times faster than Locust and over 100 times faster than Artillery.

The comparison is a bit unfair as several of the tools let their VU threads run much more sophisticated script code than what Wrk allows, but still. You'd think Wrk offered no scripting at all, but it actually allows you to execute Lua code in the VU threads and theoretically, you can create test code that quite complex. In practise, however, the Wrk scripting API is callback-based and not very suitable at all for writing complicated test logic. But it is also very fast. I did not execute Lua code when testing Wrk this time - I used the single-URL test mode instead, but previous tests have shown Wrk performance to be only minimally impacted when executing Lua code.

However, being fast and measuring correctly is about all that Wrk does. It has no HTTP/2 support, no fixed request rate mode, no output options, no simple way to generate pass/fail results in a CI setting, etc. In short, it is quite feature-sparse.

Wrk help output

Wrk summary

Wrk is included among the top non-scriptable tools because if your only goal is to generate a truckload of simple traffic against a site, there is no tool that does it more efficiently. It will also give you accurate measurements of transaction response times, which is something many other tools fail at when they're being forced to generate a lot of traffic.

The top scriptable tools

To me, this is the most interesting category because here you'll find the tools that can be programmed to behave in whatever strange ways you desire!

Or, to put it in a more boring way, here are the tools that allow you to write test cases as pure code, like you're used to if you're a developer.

Note that I list the top tools in alphabetical order - I won't rank them because lists are silly. Read the information and then use that lump that sits on top of your neck, to figure out which tool YOU should use.

Gatling

Gatling isn't actually a favourite of mine, because it is a Java app and I don't like Java apps. Java apps are probably easy to use for people who spend their whole day working in a Java environment, but for others they are definitely not user-friendly. Whenever something fails in an app written in mostly any other language you'll get an error message that often helps you figure out what the problem is. If a Java app fails, you'll get 1,000 lines of stack trace and repeated, generical error messages that is of absolutely zero help whatsoever. Also, running Java apps often require manual tweaking of JVM runtime parameters. Perhaps Java is well suited for large enterprise backend software, but not for command-line apps like a load testing tool, so being a Java app is a clear minus in my book.

If you look at the screenshot above, you'll note that you have to add parameters to your test inside a "JAVA_OPTS" environment variable, that is then read from your Gatling Scala script. There are no parameters you can give Gatling to affect concurrency/VUs, duration or similar, but this has to come from the Scala code itself. This way of doing things is nice when you're only running something in an automated fashion, but kind of painful if you want to run a couple of manual tests on the command line.

Despite the Java-centricity (or is it "Jave-centrism"?), I have to say that Gatling is a quite nice load testing tool. Its performance is not great, but probably adequate for most people. It has a decent scripting environment based on Scala. Again, Scala is not my thing but if you're into it, or Java, it should be quite convenient for you to script test cases with Gatling.

Here is what a very simple Gatling script may look like:

import io.gatling.core.Predef._
import io.gatling.http.Predef._
import scala.concurrent.duration._

class GatlingSimulation extends Simulation {

  val vus = Integer.getInteger("vus", 20)
  val duration = Integer.getInteger("duration", 10)

  val scn = scenario("Scenario Name") // A scenario is a chain of requests and pauses
    .during (duration) {
      exec(http("request_1").get("http://192.168.0.121:8080/"))
        }
        setUp(scn.inject(atOnceUsers(vus)))
}

The scripting API seems capable and it can generate pass/fail results based on user-definable conditions.

I don't like the text based menu system you get by default when starting Gatling. Luckily, that can be skipped by using the right command-line parameters. If you dig into it just a little bit, Gatling is quite simple to run from the command line.

The documentation for Gatling is very good, which is a big plus for any tool.

Gatling has a recording tool that looks competent, though I haven't tried it myself as I'm more interested in scripting scenarios to test individual API end points, not record "user journeys" on a web site. But I imagine many people who run complex load test scenarios simulating end user behaviour will be happy the recorder exists.

Gatling will by default report results to stdout and generate nice HTML reports (using my favourite charting library - Highcharts) after the test has finished. It's nice to see that it has lately also gotten support for results output to Graphite/InfluxDB and visualization using Grafana.

Gatling help output

Gatling summary

Overall, Gatling is a very competent tool that is actively maintained and developed. If you're using Jmeter today, you should definitely take a look at Gatling, just to see what you're missing (hint: usability!).

As I was involved in the creation of k6, it's not strange that I like the choices that project has made. The idea behind k6 was to create a high-quality load testing tool for the modern developer, which allowed you to write tests as pure code, had a simple and consistent command-line UX, had useful results output options, and had good enough performance. I think all these goals have been pretty much fulfilled, and that this makes k6 a very compelling choice for a load testing tool. Especially for a developer like myself.

k6 is scriptable in plain Javascript and has what I think is the nicest scripting API of all tools I've tested. This API that makes it easy to perform common operations, test that things behave as expected, and control pass/fail behaviour for automated testing. Here is what a very simple k6 script might look like:

import http from 'k6/http';
import { check } from 'k6';

export default function() {
  var res = http.get('http://192.168.0.121:8080/');
  check(res, {
    'is status 200': r => r.status === 200
  });
}

The above script will make each VU generate a HTTP transaction and then check that the response code was 200. The status of a check like this is printed on stdout, and you can set up thresholds to fail the test if a big enough percentage of your checks are failing. The k6 scripting API makes writing automated performance tests a very nice experience, IMO.

The k6 command line interface is simple, intuitive and consistent - It feels modern. k6 is among the faster tools in this review, it supports all the basic protocols (HTTP 1/2/Websocket), has multiple output options (text, JSON, InfluxDB, StatsD, Datadog, Kafka). Recording traffic from a browser is pretty easy as k6 can convert HAR files to k6 script, and the major browsers can record sessions and save them as HAR files. Also, there are options to convert e.g. Postman collections to k6 script code. Oh yeah, and the documentation is stellar overall (though I just spoke to a guy working on the docs and he was dissatisfied with the state they're in today, which I think is great. When a product developer is satisfied, the product stagnates)

What does k6 lack then? Well, load generation distribution is not included, so if you want to run really large-scale tests you'll have to buy the premium SaaS version (that has distributed load generation). On the other hand, its performance means you're not very likely to run out of load generation capacity on a single physical machine anyway. It doesn't come with any kind of web UI, if you're into such things. I'm not.

One thing people may expect, but which k6 doesn't have, is NodeJS-compatibility. Many (perhaps even most?) NodeJS libraries can not be used in k6 scripts. If you need to use NodeJS libs, Artillery may be your only safe choice (oh nooo!).

Otherwise, the only thing I don't like about k6 is the fact that I have to script my tests in Javascript! JS is not my favourite language, and personally, I would have preferred using Python or Lua - the latter being a scripting language Load Impact has been using for years to script load tests and which is very resource-efficient. But in terms of market penetration, Lua is a fruit fly whereas JS is an elephant, so choosing JS over Lua was wise. And to be honest, as long as the scripting is not done in XML (or Java), I'm happy.

Like mentioned earlier, the open source version of k6 is being very actively developed, with new features added all the time. Do check out the Release notes/Changelog which, btw, are some of the best written that I've ever seen (thanks to the maintainer @na-- who is an ace at writing these things).

k6 help output

k6 also deserves a shoutout for its built-in help, which is way nicer than that of any other tool in this review. It has a docker-style, multi-level k6 help command where you can give arguments to display help for specific commands. E.g. k6 help run will give you an extensive help text showing how to use the runcommand.

k6 summary

Not that I'm biased or anything, but I think k6 is way ahead of the other tools when you look at the whole experience for a developer. There are faster tools, but none faster that also supports sophisticated scripting. There are tools that support more protocols, but k6 supports the most important ones. There are tools with more output options, but k6 has more than most. In practically every category, k6 is average or better. In some categories (documentation, scripting API, command line UX) it is outstanding.

Locust

The scripting experience with Locust is very nice. The Locust scripting API is pretty good though somewhat basic and lacks some useful things other APIs have, such as custom metrics or built-in functions to generate pass/fail results when you want to run load tests in a CI environment.

The big thing with Locust scripting though is this - you get to script in Python! Your mileage may vary, but if I could choose any scripting language to use for my load tests I would probably choose Python. Here's what a Locust script can look like:

from locust import TaskSet, task, constant
from locust.contrib.fasthttp import FastHttpLocust

class UserBehavior(TaskSet):
  @task
  def bench_task(self):
    while True:
      self.client.get("/")

class WebsiteUser(FastHttpLocust):
  task_set = UserBehavior
  wait_time = constant(0)

Nice, huh? Arguably even nicer than the look of a k6 script, but the API does lack some things like built-in support for pass/fail results (Blazemeter has an article about how you can implement your own assertions for Locust, which involves generating a Python exception and getting a stack trace - sounds a bit like rough terrain to me). Also, the new FastHttpLocust class (read more about it below) seems a bit limited in functionality (e.g. not sure there is HTTP/2 support?)

Locust has a nice command-and-control web UI that shows you live status updates for your tests and where you can stop the test or reset statistics. Plus it has easy-to-use load generation distribution built in. Starting a distributed load test with Locust is as simple as starting one Locust process with the --master switch, then multiple processes with the --slave switch and point them at the machine where the master is located.

Here is a screenshot from the UI when running a distributed test. As you can see I experienced some kind of UI issue (using Chrome 79.0.3945.130) that caused the live status data to get printed on top of the navigation menu bar (perhaps the host string was too long?) but otherwise this web UI is neat and functional. Even if I wrote somewhere else that I'm not into web UI's, they can be quite nice sometimes when you're trying to control a number of slave load generators and stay on top of what's happening.

Python is actually both the biggest upside and the biggest downside with Locust. The downside part of it stems from the fact that Locust is written in Python. Python code is slow, and that affects Locust's ability to generate traffic and provide reliable measurements.

The first time I benchmarked Locust, back in 2017, the performance was horrible. Locust used more CPU time to generate one HTTP request than any other tool I tested. What made things even worse was that Locust was single-threaded, so if you did not run multiple Locust processes, Locust could only use one CPU core and would not be able to generate much traffic at all. Luckily, Locust had support for distributed load generation even then, and that made it go from the worst performer to the second worst, in terms of how much traffic it could generate from a single physical machine. Another negative thing about Locust back then was that it tended to add huge amounts of delay to response time measurements, making them very unreliable.

The cool thing is that since then, the Locust developers have made some changes and really speeded up Locust. This is unique as all other tools have stayed still or regressed in performance the past two years. Locust introduced a new Python class/lib called FastHttpLocust, which is a lot faster than the old HttpLocust class (that was built on the Requests library). In my tests now, I see a 4-5x speedup in terms of raw request generation capability, and that is in line with what the Locust authors describe in the docs also. This means that a typical, modern server with 4-8 CPU cores should be able to generate 5-10,000 RPS running Locust in distributed mode. Note that distributed execution will often still be necessary as Locust is still single-threaded. In the benchmark tests I also note that Locust measurement accuracy degrades more gracefully with increased workload when you run it in distributed mode.

The nice thing with these improvements, however, is that now, chances are a lot of people will find that a single physical server provides enough power for their load testing needs when they run Locust. They'll be able to saturate most internal staging systems, or perhaps even the production system (or a replica of it). Locust is still among the lower-performing tools in the review, but now it feels like performance is not making it unusable anymore.

Personally, I'm a bit schizophrenic about Locust. I love that you can script in Python (and use a million Python libraries!). That is the by far biggest selling point for me. I like the built-in load generation distribution, but wouldn't trust that it scales for truly large-scale tests (I suspect the single --master process will become a bottleneck pretty fast - would be interesting to test). I like the scripting API, although it wouldn't hurt if it had better support for pass/fail results and the HTTP support with FastHttpLocust seems basic. I like the built-in web UI. I don't like the overall low performance that may force me to run Locust in distributed mode even when on a single host - having to provision multiple Locust instances is an extra complication I don't really want, especially for automated tests. I don't like the command line UX so much.

About distributed execution on a single host - I don't know how hard it would be to make Locust launch in --master mode by default and then have it automatically fire off multiple --slave daughter processes, one per detected CPU core? (everything of course configurable if the user wants to control it) Make it work more like e.g. Nginx. That would result in a nicer user experience, IMO, with a less complex provisioning process - at least when you're running it on a single host

Locust help output

Locust summary

If it wasn't for k6, Locust would be my top choice. If you're really into Python you should absolutely take a look at Locust first and see if it works for you. I'd just make sure the scripting API allows you to do what you want to do in a simple manner and that performance is good enough, before going all in.

The rest of the tools

Here are my comments on the rest of the tools.

Apachebench

Apachebench isn't very actively developed and getting kind of old. I'm including it mainly because it is so common out there, being part of the bundled utilities for Apache httpd. Apachebench is fast, but single-threaded. It doesn't support HTTP/2 and there is no scripting capability.

Apachebench summary

Apachebench is good for simple hammering of a single URL. Its only competitor for that use case would be Hey (which is multi-threaded and supports HTTP/2).

Artillery

Artillery is a seriously slow, very resource-hungry and possibly not very actively developed open source load testing tool. Not a very flattering summary I guess, but read on.

It is written in Javascript, using NodeJS as its engine. The nice thing about building on top of NodeJS is NodeJS-compatibility: Artillery is scriptable in Javascript and can use regular NodeJS libraries, which is something e.g. k6 can't do, despite k6 also being scriptable in regular Javascript. The bad thing about being a NodeJS app, however, is performance: In the 2017 benchmark tests, Artillery proved to be the second-worst performer, after Locust. It was using a ton of CPU and memory to generate pretty unimpressive RPS numbers and response time measurements that were not very accurate at all.

I'm sad to say that things have not changed much here since 2017. If anything, Artillery seems a bit slower today.

In 2017, Artillery could generate twice as much traffic as Locust, running on a single CPU core. Today, Artillery can only generate 1/3 of the traffic Locust can produce, when both tools are similarly limited to using a single CPU core. Partly this is because Locust has improved in performance, but the change is bigger than expected so I'm pretty sure Artillery performance has dropped also. Another data point that supports that theory is Artillery vs Tsung. In 2017, Tsung was 10 times faster than Artillery. Today, Tsung is 30 times faster. I believe Tsung hasn't changed in performance at all, which then means Artillery is much slower than it used to be (and it wasn't exactly fast back then either).

The performance of Artillery is definitely an issue, and an aggravating factor is that open-source Artillery still doesn't have any kind of distributed load generation support so you're stuck with a very low-performing solution unless you buy the premium SaaS product.

The artillery.io site is not very clear on what differences there are between Artillery open source and Artillery Pro, but there appears to be a Changelog only for Artillery Pro, and looking at the Github repo, the version number for Artillery open source is 1.6.0 while Pro is at 2.2.0 according to the Changelog. Scanning the commit messages of the open source Artillery, it seems there are mostly bug fixes there, and not too many commits over the course of 2+ years.

The Artillery team should make a better effort at documenting the differences between Artillery open source and the premium product Artillery Pro, and also write something about their intentions with the open source product. Is it being slowly discontinued? It sure looks that way.

Another thing to note related to performance is that nowadays Artillery will print "high-cpu" warnings whenever CPU usage goes above 80% (of a single core) and it is recommended to never exceed that amount so as not to "lower performance". I find that if I stay at about 80% CPU usage so as to avoid these warnings, Artillery will produce a lot less traffic - about 1/8 the number of requests per second that Locust can do. If I ignore the warning messages and let Artillery use 100% of one core, it will increase RPS to 1/3 of what Locust can do. But at the cost of a pretty huge measurement error.

All the performance issues aside, Artillery has some good sides also. It is quite suitable for CI/automation as it is easy to use on the command line, has a simple and concise YAML-based config format, plugins to generate pass/fail results, outputs results in JSON format, etc. And like previously mentioned, it can use regular NodeJS libraries, which offer a huge amount of functionality that is simple to import. But all this is irrelevant to me when a tool performs the way Artillery does. The only situation where I'd even consider using Artillery would be if my test cases had to rely on some NodeJS libraries that k6 can't use, but Artillery can.

Artillery summary

Only ever use it if you've already sold your soul to NodeJS (i.e. if you have to use NodeJS libraries).

Drill

Drill is written in Rust. I've avoided Rust because I'm scared I may like it and I don't want anything to come between me and Golang. But that is probably not within the scope of this article. What I meant to write was that Rust is supposed to be fast, so my assumption is that a load testing tool written in Rust would be fast too.

Running some benchmarks, however, it quickly becomes apparent that this particular tool is incredibly slow! Maybe I shouldn't have been so quick to include Drill in the review, seeing as it is both quite new and not yet widely used. Maybe it was not meant to be a serious effort at creating a new tool? (given that the author claims that Drill was created because he wanted to learn Rust). On the other hand it does have a lot of useful features, like a pretty powerful YAML-based config file format, thresholds for pass/fail results, etc. so it does look like a semi-serious effort to me.

So - the tool seems fairly solid, if simple (no scripting). But when I run it in my test setup it maxes out four CPU cores to produce a mind-bogglingly low ~180 requests/second. I ran many tests, with many different parameters, and that was the best number I could squeeze out of Drill. The CPU's are spending cycles like there is no tomorrow, but there are so few HTTP transactions coming out of this tool that I could probably respond to them using pen and paper. It's like it is mining a Bitcoin between each HTTP request! Compare this to Wrk (written in C) that does over 50,000 RPS in the same environment and you see what I mean. Drill is not exactly a poster child for the claim "Rust is faster than C".

What is the point of using a compiled language like Rust if you get no performance out of your app?? You might as well use Python then. Or no, Python-based Locust is much faster than this. If the aim is ~200 RPS on my particular test setup I could probably use Perl! Or, hell, maybe even a shell script??

I just had to try it. Behold the curl-basher

So this Bash script actually gives Drill a run for its money, by executing curl on the command line multiple times, concurrently. It even counts errors. With curl-basher.sh I manage to eke out 147 RPS in my test setup (a very stable 147 RPS I have to say) and Drill does 175-176 RPS so it is only 20% faster. This makes me wonder what the Drill code is actually doing to manage to consume so much CPU time. It has for sure set a new bottom record for inefficiency in generating HTTP requests - If you're concerned about global warming, don't use Drill!

Drill summary

For tiny, short-duration load tests it could be worth considering Drill, or if the room is a bit chilly.

Jmeter

Here's the 800-pound gorilla. Jmeter is a huge beast compared to most other tools. It is old and has acquired a larger feature set, more integrations, add-ons, plugins, etc than any other tool in this review. It has been the "king" of open source load testing tools for a long time, and probably still is.

In the old days, people could choose between paying obscene amounts of money for an HP Loadrunner license, paying substantial amounts of money for a license of some Loadrunner-wannabe proprietary tool, or pay nothing at all to use Jmeter. Well, there was also the option of using Apachebench or maybe OpenSTA or some other best-forgotten free solution, but if you wanted to do serious load testing, Jmeter was really the only usable alternative that didn't cost money.

So the Jmeter user base grew and grew, and development of Jmeter also grew. Now, 15 or so years later, Jmeter has been actively developed by a large community for longer than any other load testing tool, so it isn't strange that it also has more features than any other tool.

As it was originally built as an alternative to old, proprietary load testing software from 15-20 years ago, it was designed to cater to the same audience as those applications. I.e. it was designed to be used by load testing experts running complex, large-scale integration load tests that took forever to plan, a long time to execute and a longer time to analyse the results from. Tests that required a lot of manual work and very specific load testing domain knowledge. This means that Jmeter was not, from the start, built for automated testing and developer use, and this can clearly be felt when using it today. It is a tool for professional testers, not for developers. It is not great for automated testing as its command line use is awkward, default results output options are limited, it uses a lot of resources and it has no real scripting capability, only some support for inserting logic inside the XML configuration.

This is probably why Jmeter is losing market share to newer tools like Gatling, which has a lot in common with Jmeter so it offers an attractive upgrade path for organisations that want to use a more modern tool, with better support for scripting and automation, but want to keep their tooling Java-based. Anyway, Jmeter does have some advantages over e.g. Gatling. Primarily it comes down to the size of the ecosystem - all those integrations, plugins etc I mentioned. Performance-wise, they are fairly similar. Jmeter used to be one of the very best performing tools in this review, but has seen its performance drop so now it's about average and pretty close to (perhaps slightly faster than) that of Gatling.

Jmeter summary

The biggest reasons to choose Jmeter today, if you're just starting out with load testing, would be if you:

had a need to test lots of different protocols/apps that only Jmeter had support for, or
you're a Java-centric organisation and want to use the most common Java-based load testing tool out there, or
you want a GUI load testing tool where you point and click to do things

If none of those are true, I think you're better served by Gatling (which is fairly close to Jmeter in many ways), k6 or Locust. Or, if you don't care so much about programmability/scripting (writing tests as code) you can take a look at Vegeta.

Siege

Siege is a simple tool, similar to e.g. Apachebench in that it has no scripting and is primarily used when you want to hit a single, static URL repeatedly. The biggest feature it has that Apachebench lacks is its ability to read a list of URLs and hit them all during the test. If you don't need this feature, however, just use Apachebench (or perhaps better - Hey) would be my advice. Because Siege is bound to give you a headache if you try doing anything even slightly advanced with it - like figuring out how fast the target site is responding when you hit it with traffic, or generating enough traffic to slow down the target system, or something like that. Actually, just running it with the correct config or command line options, though they're not too many, can feel like some kind of mystery puzzle game.

Siege is unreliable, in more than one way. Firstly, it crashes fairly often. Secondly, it freezes even more often (mainly at exit, can't tell you how many times I've had to kill -9 it). If you try enabling HTTP keep-alive it crashes or freezes 25% of the time. I don't get how HTTP keep-alive can be experimental in such an old tool! HTTP keep-alive itself is very old and part of HTTP/1.1, that was standardized 20 years ago! It is also very, very commonly used in the wild today, and it has a huge performance impact. Most every HTTP library have support for it. HTTP keep-alive keeps connections open between requests, so the connections can be reused. If you're not able to keep connections open it means that every HTTP request results in a new TCP handshake and a new connection. This may give you misleading response time results (because there is a TCP handshake involved in every single request, and TCP handshakes are slow) and it may also result in TCP port starvation on the target system, which means the test will stop working after a little while because all available TCP ports are in a CLOSE_WAIT state and can't be reused for new connections.

But hey, you don't have to enable weird, exotic, experimental, bleeding-edge stuff like HTTP keep-alive to make Siege crash. You just have to make it start a thread or two too many and it will crash or hang very quickly. And it is using smoke and mirrors to avoid mentioning that fact - it has a new limit config directive that sets a cap on the max number you can give to the -c (concurrency) command-line parameter - the one determining how many threads Siege will start. The value is set to 255 by default, with the motivation that Apache httpd by default can only handle 255 concurrent connections, so using more than that will "make a mess". What a load of suspicious-looking brown stuff in a cattle pasture. It so happens that during my testing, Siege seems to become unstable when you set the concurrency level to somewhere in the range 3-400. Over 500 and it crashes or hangs a lot. More honest would be to write in the docs that "Sorry, we can't seem to create more than X threads or Siege will crash. Working on it"

Siege's options/parameters make up an inconsistent, unintuitive patchwork and the help sometimes lies to you. I still haven't been able to use the -l option (supposedly usable to specify a log file location) although the long form --log=x seems to work as advertised (and do what -l won't).

Siege performs on par with Locust now (when Locust is running in distributed mode), which isn't fantastic for a C application. Wrk is 25 times faster than Siege, offers pretty much the same feature set, provides much better measurement accuracy and doesn't crash. Apachebench is also a lot faster, as is Hey. I see very few reasons for using Siege these days.

The only truly positive thing I can write is that Siege has implemented something quite clever that most tools lack - a command line switch (-C) that just reads all config data (plus command-line params) and then prints out the full config it would be using when running a load test. This is a very nice feature that more tools should have. Especially when there are multiple ways of configuring things - i.e. command-line options, config files, environment variables - it can be tricky to know exactly what config you're actually using.

Siege summary

Run fast in any other direction.

Tsung

Tsung is our only Erlang-based tool and it's been around for a while. It seems very stable, with good documentation, is reasonably fast and has a nice feature set that includes support for distributed load generation and being able to test several different protocols.

It's a very competent tool whose main drawback, in my opinion, is the XML-based config similar to what Jmeter has, and its lack of scriptability. Just like Jmeter, you can actually define loops and use conditionals and stuff inside the XML config, so in practise you can script tests, but the user experience is horrible compared to using a real language like you can with e.g. k6 or Locust.

Tsung is still being developed, but very slowly. All in all I'd say that Tsung is a useful option if you need to test one of the extra protocols it supports (like LDAP, PostgreSQL, MySQL, XMPP/Jabber), where you might only have the choice between Jmeter or Tsung (and of those two, I much prefer Tsung).

Tsung summary

Try it if you're an Erlang fan.

curl-basher

Yeah, well.

Chapter 3: Performance review and benchmarks

Load testing can be tricky because it is quite common that you run into some performance issue on the load generation side that means you're measuring that systems' ability to generate traffic, not the target systems' ability to handle it. Even very seasoned load testing professionals regularly fall into this trap.

If you don't have enough load generation power, you may either see that your load test becomes unable to go above a certain number of requests per second, or you may see that response time measurements become completely unreliable. Usually you'll see both things happening, but you might not know why and mistakenly blame the poor target system for the bad and/or erratic performance you're seeing. Then you might either try to optimize your already-optimized code (because your code is fast, of course) or you'll yell at some poor coworker who have zero lines of code in the hot paths but it was the only low-performing code you could find in the whole repo. Then the coworker gets resentful and steals your mouse pad to get even, which starts a war in the office and before you know it, the whole company is out of business and you have to go look for a new job at Oracle. What a waste, when all you had to do was make sure your load generation system was up to its task!

This is why I think it is very interesting to understand how load testing tools perform. I think everyone who use a load testing tool should have some basic knowledge of its strengths and weaknesses when it comes to performance, and also occasionally make sure that their load testing setup is able to generate the amount of traffic required to properly load the target system. Plus a healthy margin.

Testing the testing tools

A good way of testing the testing tools is to not test them on your code, but on some third-party thing that is sure to be very high-performing. I usually fire up an Nginx server and then I load test by fetching the default "Welcome to Nginx" page. It's important, though, to use a tool like e.g. top to keep track of Nginx CPU usage while testing. If you see just one process, and see it using close to 100% CPU, it means you could be CPU-bound on the target side. Then you need to reconfigure Nginx to use more worker threads.

If you see multiple Nginx processes but only one is using a lot of CPU, it means your load testing tool is only talking to that particular worker process. Then you need to figure out how to make the tool open multiple TCP connections and issue requests in parallell over them.

Network delay is also important to take into account as it sets an upper limit on the number of requests per second you can push through. If the network roundtrip time is 1 ms between server A (where you run your load testing tool) and server B (where the Nginx server is) and you only use one TCP connection to send requests, the theoretical max you will be able to achieve is 1/0.001 = 1,000 requests per second. In most cases this means that you'll want your load testing tool to use many TCP connections.

Whenever you're hitting a limit, and can't seem to push through any more requests/second, you try to find out what resource you've run out of. Monitor CPU and memory usage on both load generation and target sides with some tool like top.

If CPU is fine on both sides, experiment with the number of concurrent network connections and see if more will help you increase RPS throughput.

Another thing that is easily missed is network bandwidth. If, say, the Nginx default page requires a transfer of 250 bytes to load, it means that if the servers are connected via a 100 Mbit/s link, the theoretical max RPS rate would be around 100,000,000 divided by 8 (bits per byte) divided by 250 => 100M/2000 = 50,000 RPS. Though that is a very optimistic calculation - protocol overhead will make the actual number a lot lower so in the case above I would start to get worried bandwidth was an issue if I saw I could push through max 30,000 RPS, or something like that. And of course, if you happen to be loading some bigger resource, like e.g. an image file, this theoretical max RPS number can be a lot lower. If you're suspicious, try making changes to the size of the file you're loading and see if it changes the result. If you double the size and get half the RPS you know you're bandwidth limited.

Finally, server memory can be an issue also. Usually, when you run out of memory it will be very noticeable because most things will just stop working while the OS frantically tries to destroy the secondary storage by using it as RAM (i.e. swapping or thrashing).

After some experimentation you'll know exactly what to do to get the highest RPS number out of your load testing tool, and you'll know what its max traffic generation capacity is on the current hardware. When you know these things you can start testing the real system that you'd like to test, and be confident that whenever you see e.g. an API end point that can't do more than X requests/second you'll immediately know that it is due to something on the target side of things, not the load generator side.

These benchmarks

The above procedure is more or less what I have gone through when testing these tools. I used a small, fanless, 4-core Celeron server running Ubuntu 18.04 with 8GB RAM as the load generator machine. I wanted something that was multi-core but not too powerful. It was important that the target/sink system could handle more traffic than the load generator was able to generate (or I wouldn't be benchmarking the load generation side - the tools).

For target, I used a 4Ghz i7 iMac with 16G RAM. I did use the same machine as work machine, running some terminal windows on it and having a Google spreadsheet open in a browser, but made sure nothing demanding was happening while tests were running. As this machine has 4 very fast cores with hyperthreading (able to run 8 things in parallell) there should be capacity to spare, but to be on the safe side I have repeated all tests multiple times at different points in time, just to verify that the results are somewhat stable.

The machines were connected to the same physical LAN switch, via gigabit Ethernet.

Practical tests showed that the target was powerful enough to test all tools but perhaps one. Wrk managed to push through over 50,000 RPS and that made 8 Nginx workers on the target system consume about 600% CPU. It may be that Nginx couldn't get much more CPU than that (given that 800% usage should be the absolute theoretical max on the 4-core i7 with hyperthreading) but I think it doesn't matter because Wrk is in a class of its own when it comes to traffic generation. We don't really have to find out whether Wrk is 200 times faster than Artillery, or only 150 times faster. The important thing is to show that the target system can handle some very high RPS number that most tools can't achieve, because then we know we actually are testing the load generation side and not the target system.

Raw data

There is a spreadsheet with the raw data, plus text comments, from all the tests run. More tests than these in the spreadsheet were run though - for instance, I ran a lot of tests to find out the ideal parameters to use for the "Max RPS" tests. The goal was to cram out as many RPS as was inhumanly possible, from each tool, and for that some exploratory testing was required. Also, whenever I felt a need to ensure results seemed stable I'd run a set of tests again and compare to what I had recorded. I'm happy to say there was usually very little fluctuation in the results. Once I did have an issue with all tests suddenly producing performance numbers that were notably lower than they were before. This happened regardless of which tool was being used, and eventually led me to reboot the load generator machine, which resolved the issue.

The raw data from the tests can be found here

What I've tried to find out

Max traffic generation capability

How many requests per second could each tool generate in this lab setup? Here I tried working with most parameters available, but primarily concurrence (how many threads the tool used, and how many TCP connections) and things like enabling HTTP keep-alive, disabling things the tool did that required lots of CPU (HTML parsing maybe), etc. The goal was to cram out as many requests per second out of each tool as possible, AT ANY COST!!

The idea is to get some kind of baseline for each tool that shows how efficient the tool is when it comes to raw traffic generation.

Memory usage per VU

Several of the tools are quite memory-hungry and sometimes memory usage is also dependent on the size of the test, in terms of virtual users (VUs). High memory usage per VU can prevent people from running large-scale tests using the tool, so I think it is an interesting performance metric to measure.

Memory usage per request

Some tools collect lots of statistics throughout the load test. Primarily when HTTP requests are being made it is common to store various transaction time metrics. Depending on exactly what is stored, and how, this can consume large amounts of memory and be a problem for intensive and/or long-running tests.

Measurement accuracy

All tools measure and report transaction response times during a load test. There will always be a certain degree of inaccuracy in these measurements - for several reasons - but especially when the load generator itself is doing a bit of work it is common to see quite large amounts of extra delay being added to response time measurements. It is useful to know when you can trust the response time measurements reported by your load testing tool, and when you can't, and I have tried to figure this out for each of the different tools.

Max traffic generation capability

Here is a chart showing the max RPS numbers I could get out of each tool when I really pulled out all the stops, and their memory usage:

Pretty obvious is that Wrk has no real competition here. It is a beast when it comes to generating traffic, so if that is all you want - large amounts of HTTP requests - download (and compile) Wrk. You won't be displeased!

But while being a terrific request generator, Wrk is definitely not perfect for all uses (see review), so it is interesting to see what's up with the other tools. Let's remove Wrk from the chart to get a better scale:

Before discussing these results, I'd like to mention that three tools were run in non-default modes in order to generate the highest possible RPS numbers:

Artillery

Artillery was run with a concurrency setting high enough to cause it to use up a full CPU core, which is not recommended by the Artillery developers and results in high-cpu warnings from Artillery. I found that using up a full CPU core increased the request rate substantially, from just over 100 RPS when running the CPU at ~80% to 300 RPS when at 100% CPU usage. The RPS number is still abysmally low, of course, and like the Artillery FAQ says and like we also see in the response time accuracy tests, response time measurements are likely to be pretty much unusable when Artillery is made to use all of one CPU core.

k6 was run with the --compatibility-mode=base command line option that disables newer Javascript features, stranding you with old ES5 for your scripting. It results in a ~50% reduction in memory usage and a ~10% general speedup which means that the max RPS rate goes up from ~10k to ~11k. Not a huge difference though, and I'd say that unless you have a memory problem it's not worth using this mode when running k6.

Locust

Locust was run in distributed mode, which means that 5 Locust instances were started: one master instance and four slave instances (one slave for each CPU core). Locust is single-threaded so can't use more than one CPU core, which means that you have to distribute load generation over multiple processes to fully use all the CPU on a multi-CPU server (they should really integrate the master/slave mode into the app itself so it auto-detects when a machine has multiple CPUs and starts multiple processes by default). If I had run Locust in just one instance it would only have been able to generate ~900 RPS.

We also used the new FastHttpLocust library for the Locust tests. This library is 3-5 times faster than the old HttpLocust library. However, using it means you lose some functionality that HttpLocust has but which FastHttpLocust doesn't.

What has changed since 2017?

I have to say these results made me a bit confused at first, because I tested most of these tools in 2017, and expected performance to be pretty much the same now. The absolute RPS numbers aren't comparable to my previous tests of course, because I used another test setup then, but I expected the relationships between the tools to stay roughly the same: e.g. I thought Jmeter would still be one of the fastest tools, and I thought Artillery would still be faster than Locust when run on a single CPU core.

Jmeter is slower!

Well, as you can see, Jmeter performance seems pretty average now. From my testing it seems Jmeter has dropped in performance by about 50% between versions 2.3 and the one I tested now - 5.2.1. It could be a JVM issue maybe. I tested with OpenJDK 11.0.5 and Oracle Java 13.0.1 and both performed pretty much the same, so it seems unlikely it is due to a slower JVM. I also tried upping the -Xms and -Xmx parameters that determine how much memory the JVM can allocate, but that didn't affect performance either.

Artillery is now glacially slow, and Locust is almost decent!

As for Artillery, it also seems to be about 50% slower now than two years ago, which means it is now as slow as Locust was two years ago when I whined endlessly about how slow that tool was. And Locust? It is the single tool that has substantially improved performance since 2017. It is now about 3 times faster than it was back then, thanks to its new FastHttpLocust HTTP library. It does mean losing a little functionality offered by the old HttpLocust library (which is based on the very user-friendly Python Requests library), but the performance gain was really good for Locust I think.

Siege is slower!

Siege wasn't a very fast tool two years ago, though written in C, but somehow its performance seems to have dropped further between version 4.0.3 and 4.0.4 so that now it is slower than Python-based Locust when the latter is run in distributed mode and can use all CPU cores on a single machine.

Drill is very, very slow

Drill is written in Rust, so should be pretty fast, and it makes good use of all CPU cores, which are kept very busy during the test. Noone knows what those cores are doing, however, because Drill only manages to produce an incredibly measly 176 RPS! That is about on par with Artillery, but Artillery only uses one CPU core while Drill uses four! I wanted to see if a shellscript could generate as much traffic as Drill. The answer was "yeah, pretty much". You can try it yourself: curl-basher

Vegeta can finally be benchmarked, and it isn't bad!

Vegeta used to offer no way of controlling concurrency, which made it hard to compare against other tools so in 2017 I did not include it in the benchmark tests. Now, though, it has gotten a -max-workers switch that can be used to limit concurrency and which, together with -rate=0 (unlimited rate) allows me to test it with the same concurrency levels as used for other tools. We can see that Vegeta is quite performant - it both generates lots of traffic and uses little memory.

Summarizing traffic generation capability

The rest of the tools offer roughly the same performance as they did in 2017.

I'd say that if you need to generate huge amounts of traffic you might be better served by one of the tools on the left side of the chart, as they are more efficient, but most of the time it is probably more than enough to be able to generate a couple of thousand requests/second and that is something Gatling or Siege can do, or a distributed Locust setup. However, I'd recommend against Artillery or Drill unless you're a masochist or want an extra challenge. It will be tricky to generate enough traffic with those, and also tricky to interpret results (at least from Artillery) when measurements get skewed because you have to use up every ounce of CPU on your load generator(s).

Memory usage

What about memory usage then? Let's pull up that chart again:

The big memory hogs are Tsung, Jmeter, Gatling and Locust. Especially our dear Java apps - Jmeter and Gatling - really enjoy their memory and want lots of it. Locust wouldn't be so bad if it didn't have to run in multiple processes (because it is single-threaded), which consumes more memory. A multithreaded app can share memory between threads, but multiple processes are forced to keep identical sets of a lot process data.

These numbers give an indication about how memory-hungry the tools are, but they don't show the whole truth. After all, what is 500 MB today? Hardly any servers come without a couple of GB of RAM, so 500 MB should never be much of an issue. The problem is, however, if memory usage grows when you scale up your tests. Two things tend to make memory usage grow:

long running tests that collect a lot of results data
ramping up the number of VUs / execution threads

To investigate these things I ran two suites of tests to measure "Memory usage per VU" and "Memory usage per request". I didn't actually try to calculate the exact memory use per VU or request, but ran tests with increasing amounts of requests and VUs, and recorded memory usage.

Memory usage per VU level

Here we can see what happens as you scale up the number of virtual users (VUs). Note that the numbers shown are average memory use throughout a very short (10 second) test. Samples have been taken every second during the test, so 9-10 samples typically. This test should really be done with more VUs, maybe going from 1VU to 200 VU or something, and have the VUs not do so much so you don't get too much results data. Then you'd really see how the tools "scale" when you're trying to simulate more users.

But we can see some things here. A couple of tools seem unaffected when we change the VU number, which indicates that either they're not using a lot of extra memory per VU, or they're allocating memory in chunks and we haven't configured enough VUs in this test to force them to allocate more memory than they started out with. You can also see that with a tool like e.g. Jmeter it's not unlikely that memory could become a problem as you try to scale up your tests. Tsung and Artillery also look like they may end up using a ton of memory if you try to scale up the VU level substantially from these very low levels.

Memory usage per request volume

For this test, I ran all the tools with the same concurrency parameters but different test durations. The idea was to make the tools collect lots of results data and see how much memory usage grew over time. The plot shows how much the memory usage of each tool changes when it goes from storing 20k transaction results to 1 million results.

As we can see, Wrk doesn't really use any memory to speak of. Then again, it doesn't store much results data either of course. Siege also seems quite frugal with memory, but we failed to test with 1 million transactions because Siege aborted the test before we could reach 1 million. Not totally unexpected as Siege only sends one request per TCP socket - then it closes the socket and opens a new one for next request. This starves the system of available local TCP ports. You can probably expect any larger or longer test to fail if you're using Siege.

Tsung and Artillery seems to grow their memory usage, but not terribly fast, as the test runs on. k6 and Hey have much steeper curves and there you could eventually run into trouble, for very long running tests.

Again, the huge memory hogs are the Java apps: Jmeter and Gatling. Jmeter goes from 160MB to 660MB when it has executed 1 million requests. And note that this is average memory usage throughout the whole test. The actual memory usage at the end of the test might be twice that. Of course, it may be that the JVM is just not garbage collecting at all until it feels it is necessary - not sure how that works. If that's the case, however, it would be interesting to see what happens to performance if the JVM actually has to do some pretty big garbage collection at some point during the test. Something for someone to investigate further.

Oh and Drill got excluded from these tests. It just took way too much time to generate 1 million transactions using Drill. My kids would grow up while the test was running.

Measurement accuracy

Sometimes, when you run a load test and expose the target system to lots of traffic, the target system will start to generate errors. Transactions will fail, and the service the target system was supposed to provide will not be available anymore, to some (or all) users.

However, this is usually not what happens first. The first bad thing that tends to happen when a system is put under heavy load, is that it slows down. Or, perhaps more accurately, things get queued and service to the users gets slowed down. The transactions will not complete as fast as before. This generally results in a worse user experience, even if the service is still operational. In cases when this performance degradation is small, users will be slightly less happy with the service, which means more users bounce, churn or just don't use the services offered. In cases where performance degradation is severe, the effects can be a more or less total loss of revenue for e.g. an e-commerce site.

This means that it is very interesting to measure transaction response times. You want to make sure they're within acceptable limits at the expected traffic levels, andkeep track of them so they don't regress as new code is added to your system.

All load testing tools try to measure transaction response times during a load test, and provide you with statistics about them. However, there will always be a measurement error. Usually in the form of an addition to the actual response time a real client would experience. Or, put another way, the load testing tool will generally report worse response times than what a real client would see.

Exactly how large this error is, varies. It varies depending on resource utilisation on the load generator side - e.g. if your load generator machine is using 100% of its CPU you can bet that the response time measurements will be pretty wonky. But it also varies quite a lot between tools - one tool may exhibit much lower measurement errors overall, than another tool.

As a user I'd like the error to be as small as possible because if it is big it may mask the response time regressions that I'm looking for, making them harder to find. Also, it may mislead me into thinking my system isn't responding fast enough to satisfy my users.

Here is a chart showing reported response times at various VU/concurrency levels, for the different tools:

As we can see, one tool with a military-sounding name increases the scale of the chart by so much that it gets hard to compare the rest of the tools. So I'll remove the offender, having already slammed it thoroughly elsewhere in this article. Now we get:

OK, that's a bit better. So first maybe some info about what this test does. We run each tool at a set concurrency level, generating requests as fast as possible. I.e. no delay in between requests. The request rate varies - from 150 RPS to 45,000 RPS depending on which tool and which concurrency level.

If we start by looking at the most boring tool first - Wrk - we see that its MEDIAN (all these response times are medians, or 50th percentile) response time goes from ~0.25ms to 1.79ms as we increase the VU level from 10 to 100. This means that at a concurrency level of 100 (100 concurrent connections making requests) and 45,000 RPS (which was what Wrk achieved in this test) the real server response time is below 1.79 ms. So anything a tool reports, at this level, that is above 1.79 ms is pretty sure to be delay added by the load testing tool itself, not the target system.

*Why median response times?, you may ask. Why not a higher percentile, which is often more interesting. It's simply because it's the only metric (apart from "max response time") that I can get out of all the tools. One tool may report 90th and 95th percentiles, while another report 75th and 99th. Not even the mean (average) response time is reported by all tools (I know it's an awful metric, but it is a very common one).

The tools in the middle of the field here report 7-8 ms median response times at the 100 VU level, which is ~5-6 ms above the 1.79 ms reported by Wrk. This makes it reasonable to assume that the average tool adds about 5 ms to the reported response time, at this concurrency level. Of course, some tools (e.g. Apachebench or Hey) manage to generate a truckload of HTTP requests while still not adding so much to the response time. Others - like Artillery - only manages to generate very small amounts of HTTP requests but still add very large measurement errors while doing so. Let's look at a chart showing the RPS number vs median response time measurement. And remember that the server side here is likely more or less always able to give a median response time of less than 1.79 ms.

Let's look at response times vs requests/second (RPS) each tool generates, as this gives an idea about how much work the tool can perform and still provide reliable measurements.

Again, Artillery is way, way behind the rest, showing a huge measurement error of roughly +150 ms while only being able to put out less than 300 requests per second. Compare that with Wrk, which outputs 150 times as much traffic while producing 1/100th of the measurement error and you'll see how big the performance difference really is between the best and the worst performing tool.

I know Artillery people will say "But this is just because he used up all the CPU, despite Artillery printing high-CPU warnings". Well, I also ran a test where I slowed down Artillery so those warnings never appeared. I still used 100 concurrent visitors/users, but they each ran scripts with built-in sleeps that meant CPU usage was kept at around 80% and no warnings were printed. The RPS rate ended up being a lot worse, of course - it was 63 RPS. The response time measurement? 43.4 ms. More than +40 ms error.

So even when Artillery is being run "correctly" and producing an astonishing 63 RPS it still adds a measurement error that is 20 times bigger than that which Wrk adds, when Wrk is producing close to 1,000 times as much traffic. I haven't tested it, but I wouldn't be surprised if curl-basher did better than Artillery in this category.

Let's remove Artillery from the chart again:

It's interesting to see the four tools that have the highest measurement errors (excluding Artillery) perform quite similarly here: Siege, Gatling, Jmeter and Locust. They all do just under 3,000 RPS on my setup when simulating 100 VU, and they all seem to add similar amounts of measurement error: between 20 and 30 ms.

Jmeter used to be one of the more performant tools in these benchmarks, but it seems it has gotten a lot less so over the last 2-3 years. Siege has also sunk quite a bit, and its performance now doesn't really give a hint that it's a tool written in C. Instead, Python-based Locust has sailed up and placed itself next to these other tools, being equally good at generating traffic, if not quite as good at measuring correctly.

Tsung impresses again. While being an old and not so actively maintained tool, its load generation capabilities are quite decent and the measurements are second to none but Wrk.

Drill is just weird.

Vegeta, Apachebench, k6 and Hey all seem to be quite good at generating traffic while keeping the measurement error reasonably low. Bias warning here again, but it makes me happy to see k6 end up smack in the middle in all these benchmarks, given that it is executing sophisticated script logic while the tools that outperform it don't.

End summary

k6 rulez!

Or, uh, well it does, but most of these tools have something going for them. They are simply good in different situations.

I don't really like Gatling , but understand why others like it in the "I need a more modern Jmeter" use case.
I like Hey in the "I need a simple command-line tool to hit a single URL with some traffic" use case.
I like Vegeta in the "I need a more advanced command-line tool to hit some URLs with traffic" use case.
I don't like Jmeter much at all, but guess non-developers may like it in the "We really want a Java-based tool/GUI tool that can do everything" use case.
I like k6 (obviously) in the "automated testing for developers" use case.
I like Locust in the "I'd really like to write my test cases in Python" use case.
I like Wrk in the "just swamp the server with tons of requests already!" use case.

Then we have a couple of tools that seem best avoided:

Siege is just old, strange and unstable and the project seems almost dead.
Artillery is super-slow, measures incorrectly and the open source version doesn't seem to be moving forward much.
Drill is accelerating global warming.

Good luck!

Practical DevOps #3: Shifting Left

Simme — Tue, 14 Jan 2020 22:01:00 +0000

One topic I always keep coming back to when it comes to DevOps is shift-left testing.

What is shift left testing?

The term was initially coined by Larry Smith in 2001 in an article in Dr. Dobbs Journal and refers to how we, by testing as early as possible, may deliver both faster and with higher quality.

What if we instead of postponing testing until the sprint is over and we’ve delivered a new increment to QA were to test all the time; together?

Tests are expensive

Many a manager has expressed their dissatisfaction with engineers spending too much time working on test automation. The usual reasoning is that test automation is expensive, and that Devs shouldn’t be doing the job of the QA team. The development team themselves might also reject test automation as too time-consuming, arguing that their time is better spent developing new features.

And they’re not entirely wrong. Test automation IS expensive. However, it’s not nearly as expensive as manual testing. There are a couple of reasons as to why this usually holds:

Lost context

The developer(s) who implemented the feature has moved on to other tickets, which eliminates the possible benefit of still being “in that context” mentally as the testing is performed.

This, in turn, means that they’ll have to stop whatever they’re doing and brush up on the details any time the one performing the manual testing requires assistance.

Limiting flow

Building a feature, just to put it in the ready for testing stack until it gets tested is no way to deliver features. What if testing is postponed? Creating a fix might very well be a week or two from now, as sprint scopes seldom change once the sprint has started. The same goes for actually retesting the fix.

You probably wouldn’t wait for a burger joint that waits for the cashier to tell the kitchen that the burger’s still raw on one side. They could just flip it an additional time as they cook it to make sure it’s done before they pass it on. Your features are no different.

Assurance decay

The only thing we’ll be able to guarantee is that it worked at the exact time of testing. After that, all bets are off. One could argue that the assurance given by manual tests decays, or rots, with time. Every time we want to increase the assurance again, we’ll have to execute the tests again. Automated tests on the other hand, while sometimes requiring some additional development effort, may be executed on every commit or push, without any additional effort.

When to tests

So, when should we test? Only during the development phase of each feature or ticket?

I’d like to argue that we should test all the time. From doing whiteboard simulations together with stakeholders or while designing our feature, all the way to chaos engineering experiments in production.

With that said, it’s still important that we aim at testing something as early as possible and reasonable. Otherwise, we’ll risk passing the defects on to the next step in our development process, making the process of fixing them a lot more expensive.

Every tool has its purpose

It might sound like I’m proposing that we get rid of manual testing completely. That’s not the case. There’s still an extremely valid use case for manual testing: exploratory testing.

While test automation is great for making sure that what we know, or assume, is indeed still valid, it’s very ill-suited for exploring new features, finding unintended ways in which they might break.

So, if you ever find yourself writing a test script (as in instruction, not shell- or javascript), immediately stop what you’re doing, take a step back and get started on automating it instead!

Dealing with rejection as a Speaker

Simme — Wed, 25 Dec 2019 19:43:34 +0000

During the last 12 months, I’ve been doing a lot of public speaking, both as an instructor and as a conference/meetup speaker. If I count, I end up at:

6 2-day classes classes
7 conference or meetup talks
1 podcast episode (in Swedish, available here)

While being truly grateful for this opportunity, this also means a lot of really hard work and a whole bunch of rejections. Probably about 9 or 10 times the amount of accepts. According to others, the typical acceptance rate might even be as low as 10%. I’ve been rejected for more than 20 conferences this year.

This, of course, takes a toll on your feeling of self-worth. As humans, we often tend to focus on these “negative” experiences over the positive ones.

So, how do we deal with that? Is there any way to make sure that these experiences don’t weigh us down to the point of giving up?

Your worth is not defined in comparison to others

When applying to conferences, your CFP submission is usually compared to a bunch of other submissions. Sometimes hundreds of them! It’s easy to get the feeling that the program committee is comparing you to these other speakers.

That is not the case! A lot of the time, the program committee won’t even know who submitted what proposal until the final pick, sometimes not even then.

There are always more reasons than one

Sure, your proposal might very well be poorly written or uninteresting. Most of the time, that’s likely not the case. Consider aspects like conference focus topics, bad fit, lack of additional speaker slots, overlapping CFP submissions or budget constraints. Those are just as likely reasons as your proposal being bad!

Every rejection is a learning opportunity

What could we have done differently? Consider following up on the actual speaker lineup and see what you can learn from their titles and abstracts to improve your chances next time.

Ask the committee about constructive feedback

Depending on what vibe I get from the organizers, I usually email them back and thank them for their response and ask whether they’de be willing to give some additional feedback as to why my proposal was rejected and what I could have done differently.

This might not always be feasible depending on the availability of committee spokespersons and whether they’ve enough time on their hands to actually formulate some feedback. In case you don’t get a reply: remember that most organizers are working pro bono and out of an interest in contributing to their community.

Focus on your accomplishments

I write lists. Actually, I write a lot of lists. At least once a year, usually quarterly, I try to sit down and write a list of what I’ve accomplished during that year. These lists contain a whole lot of different things, like:

Books I’ve read.
Conferences or meetups I’ve attended or spoken at.
Articles I’ve written.
Contacts I’ve established.
Classes I’ve taught and people I’ve helped in some way.
Non-techy accomplishments or decisions I’m especially proud of.

Conclusion

You are not defined or valued by the amount of CFP acceptances you receive, nor the rejections.
See each rejection as a learning opportunity and as such, take some time to reflect over what you could improve next time.
The program committee did not reject you, only your proposal.

DEV Community: Simme

Keyoxide Proof

Observability is becoming mission critical, but who watches the watchmen?

Who watches the watchmen?

Understanding the failure modes

Alerts not firing

Telemetry missing

What's next?

Error Economics - How to avoid breaking the budget

How do we measure success for site reliability?

Service Level Indicators

Creating SLIs and SLOs

Service Level Objectives

What is an error budget?

Burning the budget

Setting expectations

Picking our SLIs

Picking our SLOs

Measuring

Accounting for scheduled downtime

Demo

Running distributed k6 tests on Kubernetes

📖What you will learn

⚠️ Experimental

Introduction

Prerequisites

The Kubernetes Operator pattern

Using the k6 operator to run a distributed load test in your Kubernetes cluster

Cloning the repository

Deploying the operator

Writing our test script

Deploying our test script

⚠️ Namespaces

Creating our custom resource (CR)

Parallelism

Script

Separate

Arguments

Deploying our Custom Resource

Cleaning up

⚠️ Deleting the operator

Things to consider

Metrics will not be automatically aggregated

Thresholds are not evaluated across jobs at runtime

Overpopulated nodes might create bottlenecks

Experimental

Total cost of ownership

See also

🙏🏼 Thank you for reading!

Performance testing gRPC services

🎉 New in v0.29.0

📖What you will learn

Outline

What is gRPC

API types

Unary

Server Streaming

Client Streaming

Bi-directional streaming

The .proto definition

Getting started

Creating the test

Executing the test

Summary

See Also

Setting up a Kubernetes lab cluster

Introduction

Creating our virtual machines

Installing k3s

Conclusion

Thank you for reading! 🙏🏼

In response to COVID-19, the k6 team now offers free access to k6 Cloud to non-profits fighting the pandemic

Performance Testing with Generated Data using k6 and Faker

Introduction

What is k6?

🤓 Completely new to k6?

What is Faker?

Goals

The Acme Corp Scenario

Our test goals

The `.proto` definition