Loadtest Websocket Server

TLDR;

Apache Jmeter is the best tool for creating complex and completely custom test plan for different scenarios. In this post, we dig deeper into it and explain how to set it up and how to create our test plan for a websocket server from absolute beginning to getting desired output.
We create 1000 websocket connections in a 20 seconds time span and send 20 messages per connection to our echo server.

Background

Whenever we develop a server for clients to connect to, we cannot foresee the details of the performance and over-optimization at the time of development is not a good practice. To be honest, never think of optimization and never code for performance in development stage (to a certain extent though).
Recently, I had to develop a websocket server, which is an interface for client/backend asynchronous communication. After finishing the first version of the server, it was time to benchmark it and see the throughput for connection and read/write operations.
There are some projects out there which provide a mechanism to load-test your websocket server, but my first search and exploration of them was not promising. I came to conclusion that developing a simple Go tool for my use-case would be better than using these barely maintained and vague tools. There is Thor project which has not maintained since 2017 and Apache Jmeter which seemed overly complicated and don't have any built-in support for websocket load testing.
However, after more research and before starting developing my own tool, I decided to give Jmeter a chance and see if I can achieve what I want, and mate was it appealing?
I enjoyed that much that decided to write this blog post and introduce you to Jmeter and its setup process if you haven't used it already, so we can enjoy together.

Set-up the Environment

This section explains the whole process of installing Jmeter and preparing everything for Websocket testing to the point where we can start creating our test plan.

Install Java

Make sure you have Java Development Kit and Runtime Environment installed.

~ java --version
openjdk 18.0.1.1 2022-04-22
OpenJDK Runtime Environment Homebrew (build 18.0.1.1+0)

If you are on Mac, use Homebrew to install the latest JDK or simply skip to the next step as JDK will be installed as a dependency by Homebrew.

~ brew install openjdk

Install Apache Jmeter

You can download the binary files or if you're on Mac, simply use Homebrew to install it:

~ brew install jmeter

Make sure jmeter -v prints out the version or otherwise try to add jmeter bin directory to your path.

Websocket Sampler Plugin

As discussed before, Jmeter does not support Websocket load testing internally, but luckily, there are some sampler developed by community for websocket. In this tutorial, we are using the sampler developed by Peter Doornbosch, which also provide a pretty good walkthrough in its repository's readme file.
To install the plugin, we take the easy approach which is using Jmeter's plugin manager. Usually, you can run Jmeter GUI by executing jmeter in the terminal and the GUI will be up.
From the top menu, navigate to Options->Plugins Manager which is the last item in the list.

If Plugins Manager is not available, use jmeter plugins website to install it. Download the .jar file and copy it to jmeter's /lib/ext directory. If you have installed Jmeter using Homebrew, that would be /usr/local/Cellar/jmeter/5.5/libexec/lib/ext.
After Opening Plugins Manager, navigate to available plugins tab, search for websocket and mark the checkbox next to WebSocket Samplers by Peter Doornbosch. Click on Apply Changes and Restart Jmeter. Jmeter will be restarted, and everything is ready to configure our test plan.

Configure WebSocket Test Plan

Jmeter is amazingly feature rich, which makes it complicated to configure. But, once you are a bit familiar with it, everything starts to make sense.
In this section we review how to set up a simple test plan to load test our websocket server.
Every session has a Test Plan in the sidebar where you can rename it by clicking on it.

Thread Group

Every test plan can have one or more thread group. Each thread group declare a workflow to perform some steps to load test your server. So, creating a thread group is necessary.
You can create a new thread group by right-click on you test plan and then add->Threads (Users)->Thread Group.
Rename the thread group to whatever makes sense in your case. I call it WebsocketCycle.

Action to be taken after a Sample error: Here you can configure what happens when a sample faces an error. Options are pretty self-descriptive and need no more explanation. I will leave it as Continue which will proceed to the next step of thread group itself.

Number of Threads (users): Every thread is a new user. In other words, it says that repeats the steps defined in this thread group X amount of time in different threads and possibly concurrently with other threads. I will set this to 500.

Ramp-up period: allows you to configure the amount of time that Jmeter should reach the number of threads you defined in previous step. For instance, if number of threads is 1000, and ramp-up period is 10 seconds, it takes 10 seconds for Jmeter to start all those 1000 threads. In other words, if threads are taking more than 10 seconds to complete, at second 10, you will have 1000 threads live that are executing your defined steps. I set this to 20seconds.

Loop Count: Each thread can repeat your defined steps for multiple times. It's not a separate thread and every loop cycle within a thread will be executed consecutively one after another and not concurrently. I set this to 2.

With this simple setup, I am telling my test plan that it should repeats for 1000 times (500 * 2), where I will have at least 500 requests in the span of 20 seconds.

Authorization Header

My Websocket server is not openly accessible and it needs proper authorization before upgrading HTTP connection. In this step, we set an authorization header to our connection requests.
To do this, we add the first step of our thread group by right-clicking on our thread group (WebsocketCycle) and then add->Config Element->HTTP Header Management. I rename it to AuthorizationHeaderRandomUser.
Click on the Add button at the bottom of the page to add a Header key/value pair. I add Authorization as name and Bearer <my-token>.
To expand our knowledge of Jmeter capabilities, I decided to by pass token verification by providing a JSON payload as {"id": "<user_id>"} as a token for testing purpose. Therefore I don't verify the token to extract user ID anymore.
However, I have another restriction on my server that every user can have at most 3 active connections. But, if I use the previous token with a fix number for user ID, that restriction is hit, and no more than 3 connections cannot be established and therefore, load-testing would be non-sense.
But is there any way to generate a random number for that authorization header? Of course there is :)
Jmeter has hundreds of auxiliary functions that can be used for different purpose, but the function we need here is called random and Jmeter uses a syntax like ${__<func_name>(inp1, inp2) for calling functions. In our case, it would be ${__random(1, 99999, USERID)} which will generate a random number between 1 and 99999. It prints in the place of usage in addition to storing it in USERID variable.
So, my authorization header value looks like this:

{"id": "${__random(1, 99999, USERID)}"}

You can explore other functions of Jmeter on its official website.

Websocket Connection

In the next step, I want my thread loop to create a websocket connection to my server. Now is the time to use our already installed websocket sampler plugin.
Right click on the thread group (WebsocketCycle) and then click on Add->Sampler->Websocket Open Connection.
In this approach, I create a connection in every loop of every thread and use that connection for all other steps that I want to define. In other words, the test plan is being configured to open 1000 connections in its life cycle.
Choose the scheme and your server host and port and path to websocket handler if it is not listening on the root of your host. If you choose wss do not forget to provide the TLS port (e.g. 443).

Websocket Read/Write

I have enabled an echo feature on my websocket server that write every message it reads back to the connection. In this way we can test the latency of read/write and message loss if there is any.
Right click on WebsocketCycle and click on Add->Sampler->Websocket request-response Sampler.

If you don't have an echo server, you can just add a Write Sampler and don't wait for server's response.

As discussed before, in Connection section, I use use existing connection to continue with the connection we already created in previous step.
My Websocket server expects messages in textual format. If yours is expecting Binary, you can choose it in the Data section. Then provide a sample text payload you want to send to server.
In my use-case, user ID should be part of the payload. Now we can use ${USERID} whenever we want to use our randomly generated user ID in authorization step.

If there is no response from the server within the time frame provided by Response (read) timeout section, this step will be counted as an error in the final result.

Multiple Read/Write Per Connection

I think to myself that we have started connection, we are sending a request and expecting a response, but isn't it better to send more than one request per opened connection? and my answer is, Yes.
Now, it's time to have fun with another feature of Jmeter, where we can create loop.
Right click on WebsocketCycle and then click on Add->Logic Controller->Loop Controller. Set Loop Count as 20, which means that every step that is wrapped by this step will be repeated 20 times.
Drag the previously created step (request-read sampler) and drop it on Loop Controller to visually show that it is nested.
Now, our write/read process happens 20 times per connection instead of one. Sweeeet...!

Close Connection

Since write/read process happens asynchronously, I do not close the connection from client side since it results in closing the connection while we are waiting for server response.
In my case, server closes the connection automatically after a timeout. And in almost all cases, there is a ping/pong mechanism on the server and client where server automatically closes the connection if it receives no pong for its ping request.
If this not true in your case, I strongly suggest to come-up with one. However, you can either use Websocket Write Sampler without waiting for server response, or use a timer to delay the connection closure.

Report and Result

Our test plan is set-up and completely ready to be executed. However, before running it, let add some reporting steps to our test plan.
You can add these reporting elements to your thread group or to the test plan itself. If we add it to thread group, they show the result for that single thread group. But, if you have multiple thread group and want to have an aggregated result, you have to create these reporting elements under the test plan.
For this use-case, we have one thread group, but I will create the reporting elements under the test plan any way.
Right click on the Test Plan and then we add the following reporting elements:

• Add->Listener->View Results Tree: Shows every steps and the details about the request and the response.
• Add->Listener->Summary Report: See the next section for screenshot
• Add->Listener->Aggregate Report: See the next section for screenshot

Initiate The Load Testing

Now that everything is set and ready, simply click on the green run button to start the test plan. Test plan creates 1000 connections and send 20 messages per connection and wait expect 20 messages back from server.
And below are the screenshots from the reporting elements we added earlier.

Conclusion

This was just a tiny picture of how powerful and flexible Apache Jmeter is. Check all the available testing elements in Documentation and expand your load testing journey to other servers other than WebSocket. Don't forget to check the list of functions for creating a completely dynamic test plan as well.

Until the next time...