Quick and Visual Guide to API Performance Testing with Apache JMeter

In this post, we will get our hands dirty with Apache JMeter. If you aren't familiar with load testing concepts, I suggest you check out my previous blog post explaining load test concepts.

[0] Installation & Setup

Step 0.1

To install Jmeter, you need to install the latest Java Runtime Environment (JRE). Then, you can install Jmeter from here.

Open JMeter by typing jmeter on your CLI or by clicking the JMeter logo.

Step 0.2

To load test, we need an application to load test with. For this purpose, I have prepared a simple loyalty application written in Python/Flask with 3 endpoints: sign in, create a loyalty card, and get details about a card. Now, let's clone the repo and install dependencies:

# clone the app
git clone [e-mail]:jamby1100/simple-loyalty-flask-app.git

# install dependencies
cd simple-loyalty-flask-app
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

The app saves a card to a DynamoDB database. You will need to create a DynamoDB database in your AWS account with a partition key of "card_number". You can also opt to use the offline version of DynamoDB here. If you are a bit uncomfortable using DynamoDB, you can modify the code repository to just return a hardcoded response.

In the snippet below, let's run the Flask web application we made so it can be ready to accept requests:

# run the server
export FLASK_CONFIG=development
export FLASK_APP=main.py
export FLASK_DEBUG=1
export SPECIALMESSAGE="This is a special message from the ship"
export DYNAMODB_TABLE_NAME="loyalty_cards"
export DYNAMODB_REGION_NAME="ap-southeast-1"

flask run

Let's get load testing!

Shift to the JMeter screen, and let's add the components of our JMeter Test Plan.

[1] Test Plan & Thread Groups

Once you have JMeter open, you will see the blank Jmeter console. On the left-hand section, you will see the components of your JMeter test plan. Follow the instructions on the screenshot to create your first thread group:

Thread groups define how the test plan runs. Each thread group is composed of a sequence of user behaviors you are trying to simulate. For example, buying a product can be composed of (1) going to the homepage, (2) clicking the product page, and (3) heading to Checkout. You can run multiple thread groups in a single test plan (i.e., simulating buying a product and creating a review simultaneously), but we don't advise this for people who are just starting in JMeter.

At the thread group level, we should also define configurations on how the load test should run.

• Number of Threads (Users): indicates how many "virtual users" or VUs are we simulating (i.e., 500 users browsing the site simultaneously)
  • STEP 1.1.1: Set to 10
• Ramp-up Period (Seconds): indicates how many seconds we will need to get from zero VUs to the virtual users set in the option above.
  • STEP 1.1.2: Set to 1 second
• Same user on each iteration: If you are using cookies to save the auth token received from logging in, then ticking this allows your next iterations to keep using that cookie. More information here
  • STEP 1.1.3: Tick this checkbox

Types of Load Test

The next configuration items in the Thread Group define what type of load test we are creating.

• Time-based: Inside each thread group, we define a series of HTTP requests that simulate user activity. Imagine those steps being done over and over again by 20 VUs non-stop for 20 minutes.
  • HOW: Tick the "Infinite" and the "Specify Thread Lifetime" checkboxes. Then, set the Duration (seconds) field to define how long you want the testing to happen
• Loop Based: Per user, we define how many times they will do the series of steps specified in the thread group. If we have a loop count of 10 and 20 VUs, our 20 users will repeat the sequence of steps 10 times each, resulting in 200 iterations.
  • HOW: Untick "Infinite" and specify the loop count

STEP 1.1.4 Tick the "Infinite", "Same User on each iteration", and "Specify Thread Lifetime" checkboxes. Let's set the duration field to 30 for a 30-second load testing.

The settings we chose for this load test simulates 10 users that will repeatedly do each step we will later specify. They will keep doing so nonstop until the 30-second thread lifetime expires.

This is not the setting we will choose when doing the real load testing. We have chosen these settings for now because they return feedback quickly - thus, allowing us to build our load test in a fast way. The next blog post in the series will cover making our load test "production-ready".

[2] HTTP Request Defaults and HTTP Header Manager

In this step, let's create an HTTP Request Defaults and HTTP Header Manager Config Element by right-clicking the Thread Group we created in Step 1.1. Then, under Add, click Config Element and add the 2 elements mentioned above, as shown in the screencap below:

STEP 2.1 Click the HTTP Header Manager element. On the screen that appears, click "Add" to add the headers that you would like to have at every request. For our load testing, let's have the Content-Type header have the value "application/json". We also added an Auth header whose value will be equal to whatever we will assign to the "Auth" variable (you will see this in action later).

STEP 2.2 Now, let's move on to the HTTP Request Defaults. This element allows us to set default values for:

• Protocol: HTTP or HTTPS
• Server Name or IP: for the base URL of our application
• Port Number

For our testing, set the server name's value to the IP address of the Flask web application we started in step 0. Set the port number to 5000 (default port of Flask).

[3] HTTP Request

The HTTP request allows us to specify the individual HTTP requests that comprise our test plan. Add an HTTP Request based on the instructions below:

In the HTTP request screen, let's add details about our first HTTP request. Our first endpoint is POST /users/signin, and we reflect that on the HTTP request screen below. We don’t write the base URL anymore because we already did that in our HTTP Request Default Config element.

Also, let's add the login credentials to our request body:

{
    "email": "[e-mail]",
    "password": "jambyiscool"
}

[4] View Results Tree & Summary Report Listener

As it is, we can already run the load test. But it won't be of any use at this point because we can't see the results. With listeners, we can examine the results to see if we need to make changes to our load test.

Add a View Results Tree and a Summary Report Listener based on the instructions below:

STEP 4.1 Run the load test

With 2 listeners already in place, we can now run the load test. We do this by clicking the Play button (green triangle) on the toolbar. I have annotated the toolbar for easier reference. You can do other actions like stoping the test (stop sign) and clearing the previous load test results (gear and broom logo).

STEP 4.2 View Results Tree

With this listener, you can examine each request done during your load test and look at the request and response body of each. It's essential to do this as some APIs return HTTP 200, yet their response body contains an error message indicating that the request failed (we call this "soft errors").

Soft errors deflate our error rate because JMeter thinks the request succeeded since it returned HTTP 200 (even though in the response, there was an error message). In the next post, we will discuss how to configure JMeter to check the response to see if there really was an error.

STEP 4.3 Summary Report

In the summary report, we can see high-level statistics about each endpoint, such as the response times and error rates.

[5] Let's add 2 more HTTP Requests

STEP 5.1

Now, it's time to create our second endpoint. If you recall, our second endpoint creates the loyalty card with the path /loyalty-cards and a method of POST.

Create an HTTP request as you did in Step 3. Left-click on the HTTP Request you just created and drag it below the HTTP Request created previously.

Then, select POST, add the path and add the body.

{
    "first_name": "Raphael",
    "last_name": "Jambalos"
}

STEP 5.2

At this point, our load test plan is getting messy because there are 2 HTTP requests and no proper label. Let's rectify that by adding names to our HTTP Requests, as seen below. Once you change the name, it is automatically reflected on the left-hand side.

Also, it's best to drag the 2nd HTTP request right below the 1st HTTP Request. This makes it easier to read and ensures that the login endpoint will be executed before the create card endpoint.

STEP 5.3

Now, let's try running the load test. Even though each "virtual user" signed in as a first step, all requests to create a loyalty card fail. This is because we are not extracting and using the token returned by the /users/signin request.

[6] JSON Extractor

Let's rectify that by adding a JSON Extractor based on the instructions below:

Once created, drag the JSON Extractor post-processor to be under the /users/signin request. Then, add the JMeter variable and JSON path expressions to extract the token from the response and place it to the Auth variable.

Recall in Step 2.1, we created the HTTP Header Manager with the Auth header getting its value from the variable Auth using the special syntax ${Auth}. With the JSON extractor, we extracted the token from the response of the /users/signin request and placed it in the Auth variable. Now, after the login operation, each HTTP request should already have the login token.

Re-running the test, we should have all requests working already.

[7] Homework

After creating a loyalty card, we have the option to see more details about the card we just created. We do this by using the card_number returned in the card creation API (POST /loyalty-cards) and using it as a parameter to the 3rd endpoint we have:

• Path: /loyalty-cards/{card-number}
  • example: /loyalty-cards/4444375498978382
• Method: GET

What's next?

With the 7 steps above, we now have a working load test plan to test our application's performance. But we aren't ready just yet.

Next week, I will publish a post to help you refine this test plan further and apply the best practice principles in this blog post to create realistic load tests.

Special Thanks

Special thanks to Allen for making my posts more coherent.

The authoritative JMeter book of Rodrigues, Mouawad, and Milamber entitled Master Apache JMeter From load testing to DevOps really helped me understand JMeter in more clarity. I highly suggest you purchase this book if you want to learn more about JMeter.

Photo by Theme Inn on Unsplash