DEV Community: humlix

Testing REST APIs

humlix — Thu, 14 Jan 2021 09:22:00 +0000

For many companies, it is crucial to build and maintain APIs for the Web. Building robust APIs that will be maintained and improved over time comes with a lot of challenges. This article will go through things to remember when testing REST APIs and show you what makes Humlix different from other testing tools available. For a more basic introduction to creating a REST API, you can check out the article Create and test RESTful APIs.

If you haven’t heard about Humlix before, it is a tool to help out when building and testing RESTful APIs. It can automatically generate tests to discover errors in API endpoints.

REST

REST (Representational State Transfer) is a software architecture style relying on stateless communication, often using the HTTP protocol. REST APIs (also referred to as RESTful APIs) make it possible for systems implemented with different technologies to collaborate and understand each other.

Specifying a contract

Using, e.g., OpenAPI, it is possible to specify the API before writing code and let teams develop in parallel by implementing code according to the contract. It also makes it easier for tools to understand how the API works and allows them to display the API documentation or generate code based on the specification.

An OpenAPI specification states, e.g., HTTP status code, payload, headers, and makes it possible to use tools like Humlix to understand the API and use that as a base for generating tests.

Ways of testing

Different ways of testing a REST API is to cover:

Contract testing or schema validation
- Use tooling to verify that the API corresponds with a given OpenAPI specification.
Basic positive testing
- E.g., use code or tooling to test happy paths
Testing with valid and invalid input
- Verify that the API handles valid and invalid input both for query and path parameters and for the payload correctly

Traditionally REST APIs are often tested using positive tests. Developers write tests for the API's happy paths and possibly extending it with some edge cases that they think of quickly.

Tooling

Today's tooling for testing REST APIs often lets the user create test cases by manually writing some test code, e.g., using Javascript or doing calls given example data. Using tooling like that makes it easy to reach a basic level of testing quickly.

Humlix test data generation logic makes it possible to discover edge cases that would be hard to think of for a developer writing all the tests manually. Make sure to look at this article describing how Humlix was used to discover bugs in GitLab CE. Humlix unique way of understanding an OpenAPI specification combined with Property-based testing makes it easy to find otherwise hard to find bugs.

Developer experience (DX)

Remember to walk through the experience for developers that will use the API. How is the API documented? Is it easy to figure out how to use the API? Provide code examples showing how to authenticate and start using the API.

DX is, generally speaking, how developers feel while using an API. With good DX, developers have a better chance of making quality products than using an API with horrible DX. If your API is public, DX will be one selling point compared to your competitors.

Thank you for reading

Visit Humlix to learn more about how you can generate tests for your Web APIs to improve your products' quality.

Testing of a RESTful API

humlix — Fri, 09 Oct 2020 07:59:57 +0000

This is a follow-up to the article RESTful API quick tutorial that went trough basic REST-topics.

There is a lot of tutorials online that describe how to manually set up tests for your API. With Humlix, you can either import, e.g., an OpenAPI specification or use the request builder to set up your API. Humlix will then automatically generate and execute tests for you.

We will use Humlix to test the GitLab Groups API. We will use a Docker-container from Gitlab: gitlab/gitlab-ce:13.1.0-ce.0.

Set up the API within Humlix

You need to download Humlix. Remember to sign up for the Humlix mailing list to stay up to date with the latest news. You can follow the Humlix getting started guide if you don't know how to run it.

Select the HOME menu and click on ADD SUITE and create a test suite called GitLab Suite.

Press the Add request link and create a new test request called Get a group list by page.

Go to the ENVIRONMENT VARIABLES tab and change the environment name to GitLab Environment. Add two variables base_url and gitlab_token. Set the values according to your setup. Make sure to use your token for GitLab.

Go to the REQUEST tab so that you can see the Humlix test request builder. Select GET verb and Http protocol and add base_url/api/v4/groups. Add a new query parameter page of type integer according to the picture below.

You can run a test with the example data you entered by pressing the Run single test button. This will verify that you can connect to the GitLab service trough the API endpoint that you built.

If everything works as expected, you should receive something like this in the log area.

The result above shows that our setup works and that your Gitlab API token is correct.

Disclaimer: The next step will generate many requests targeting the selected environment. Make sure that it is a test- or dev-environment that you can easily re-create if needed. Don't run this against a production environment.
Press the Generate tests button to see some real magic. This will trigger Humlix test generation, and during that time, you will see a progress indicator at the top of the user interface. When it is done, you should be able to see something similar to the image below.

Experiment with the test request settings, if you don't get the same result. E.g., you can change the Max number of generated test calls to 1000 and rerun the test generation.

Looking for the cause

We can see the cause of the problem by going into the production.log file for GitLab and see that during the execution of some SQL statement, the error was:

2020-10-02_19:43:52.85686 ERROR: bigint out of range
2020-10-02_19:43:52.85689 STATEMENT: SELECT "namespaces".* FROM "namespaces" WHERE "namespaces"."type" = 'Group' ORDER BY "namespaces"."name" ASC, "namespaces"."id" ASC LIMIT 20 OFFSET 9223372036854775820

We have found a bug in the production code!
We can also verify the bug using this cURL command:

curl -I localhost/api/v4/groups?page=461168601842738792
HTTP/1.1 500 Internal Server Error

Wrap up

Now you have seen how easy it is to let Humlix create tests for you. Download and try it out if you haven't already! Make sure to subscribe to our newsletter, so you don't miss coming articles and updates around testing Web APIs.

Create and test RESTful APIs

humlix — Mon, 05 Oct 2020 18:52:48 +0000

Using Humlix for test generation

Welcome to the first article in a series on how to develop and test RESTful APIs. In this series, we will utilize Humlix when trying out different REST endpoints.

If you haven’t heard about Humlix before, it is a tool to help out when building and testing RESTful APIs. It can automatically generate tests to discover errors in API endpoints.

The series contains the following articles:

RESTful API quick tutorial (in this article)
Testing of a RESTful API (coming soon)

For many companies, it is crucial to build and maintain APIs for the Web. Building robust APIs that will be maintained and improved over time comes with a lot of challenges. In this guide, we will go from the basics of creating a RESTful API and continue by showing how to test it using Humlix to generate tests for us.

RESTful API quick tutorial

REST APIs are implemented daily by developers all over the world to enable communication between systems. They make it possible for systems implemented with different technologies to collaborate and understand each other.

We will go through some of the principles when building an API for the Web:

Using the request methods (GET, POST, etc.) correctly
Naming API endpoints
Using parameters for filtering
Using correct HTTP response codes

Use HTTP request methods correctly

HTTP defines request methods used to show the action to perform on a resource. HTTP verbs are another common way to refer to these request methods. Some of the most commonly used verbs are:

GET — Get one or more resources
POST — Create one or more resources
PUT — Update one or more resources
DELETE — Delete one or more resources

Naming API endpoints

When developing software, methods are often named using verbs to describe their intent, e.g., GetPerson or SetLimit. Because we have HTTP verbs used to indicate the action, we can name our endpoints using nouns instead, i.e., /persons instead of /getPersons.

Let’s look at a real-life example from GitLab and their Groups API. That API provides a GET /groups endpoint to get a list of groups for the logged-on user. They didn’t call it /getGroups because they used the GET HTTP verb to state the intention.

GET /api/v4/groups

Use parameters for filtering

RESTful APIs can provide means to sort or filter responses on the server-side so that clients don’t have to massage the response data. The technique for this is to use query parameters.

Let’s go back to the GitLab example above where the GET /groups endpoint can handle sorting if clients add a query string ?sort=asc.

GET /api/v4/groups?sort=asc

The image below shows how to use Humlix to make test requests to the GET /groups endpoint.

Use correct HTTP response status codes

HTTP response status codes are a way for an endpoint to inform the client of the request’s status. Was the call a success, or did it fail?

There are five categories of status codes, where the first digit defines the class of the response.

1XX — used for information, e.g., backend received a request
2XX — successfully handled request
3XX — redirection occurred
4xx — client request cannot be fulfilled, e.g., due to the wrong syntax
5xx — server failed to fulfill a valid request

Be sure to utilize more than just 200 OK responses for your API. Check this list of HTTP response status codes.

Thank you for reading

We will soon publish the follow-up article, Testing of a RESTful API, to show you how to generate RESTful APIs tests.

How to create 1040 tests targeting Gitlab in 5 minutes with Humlix

humlix — Sun, 04 Oct 2020 13:10:21 +0000

Humlix is a brand new web API request builder and quality assurance tool that automatically generates tests. All you need to do is provide examples of how to call your API methods, and Humlix will use it to start doing its magic.

To put Humlix to a real test, we decided to try it on a real industry giant, and GitLab seemed to be a suitable candidate because of its size and that we can run it locally. Therefore, in this post, we will let Humlix create tests targeting GitLab Community Edition (gitlab-ce) and specifically its GitLab Groups API. We will use gitlab/gitlab-ce:11.2.0-ce.0 docker image and later try to repeat the same scenario on the currently latest version gitlab/gitlab-ce:13.1.0-ce.0

Let’s start with the end

By showing Humlix how to call the /api/v4/groups Gitlab Group API method, we were able to discover a bug that would have been hard to detect using regular unit tests. You can see that Humlix generated 1040 tests towards gitlab-ce in the screenshot below and that calling:

/api/v4/groups?page=461168601842738792

resulted in a status 500 Internal Server Error.

We can see the cause of the problem by going into the production.log file for GitLab and see that during the execution of some SQL statement the error was:

2020-10-02_19:43:52.85686 ERROR: bigint out of range
2020-10-02_19:43:52.85689 STATEMENT: SELECT "namespaces".* FROM "namespaces" WHERE "namespaces"."type" = 'Group' ORDER BY "namespaces"."name" ASC, "namespaces"."id" ASC LIMIT 20 OFFSET 9223372036854775820

So in just a couple of minutes, we were able to detect bugs in production code.

A user has already reported this issue in the GitLab issue tracker system. To check if GitLab has fixed the bug, we can repeat the same scenario on the currently latest version gitlab/gitlab-ce:13.1.0-ce.0. Humlix once again located the same problem.

We can also verify it using this cURL command:

curl -I localhost/api/v4/groups?page=461168601842738792 HTTP/1.1 500 Internal Server Error

As you can see, this problem has existed for a long time, even between different versions, but Humlix was able to find the bug within minutes.

How to generate tests with Humlix

You need to download Humlix. Remember to sign up for the Humlix mailing list to stay up to date with the latest news. You can follow the Humlix getting started guide if you don’t know how to run it.

Select the HOME menu and click on ADD SUITE and create a test suite called GitLab Suite.

Press the Add request link and create a new test request called Get a group list by page.

Go to the ENVIRONMENT VARIABLES tab and change the environment name to GitLab Environment. Add two variables base_url and gitlab_token. Set the values according to your setup.

Go to the REQUEST tab so that you can see the Humlix test request builder. Select GET verb and Http protocol and add base_url/api/v4/groups. Add a new query parameter page of type integer according to the picture below.

Now we are ready to perform some tests. To verify that your setup is working, you can run a test with precisely the data you entered in the request builder by pressing the Run single test button.

If everything works as expected, you should receive something like this in the log area.

The result above shows that our setup works and that your Gitlab API token is correct.

Disclaimer: Don’t perform the next step unless you are sure you are targeting a test environment running GitLab that you can re-create without losing anything important. Humlix will generate many requests and could cause a heavy load on production servers.

Now you can press the Generate tests button to see some real magic.

If you don’t get the same result, you can experiment with the test request settings. E.g., you can change the Max number of generated test calls to 1000 and rerun the test generation.

Now you have seen what Humlix is capable of in just a few minutes and how easy it is to let Humlix create tests for you. Download and try it out if you haven’t already!