DEV Community

Cover image for Easy Integration Testing with Venom!
Aurélie Vache
Aurélie Vache

Posted on

Easy Integration Testing with Venom!

#testing #integration

When working in the development of applications / and websites, the diction “testing is life” is true and we will see in this article that it is not only unit tests and that creating and automating integration tests can be child's play ;-).

Integration tests, what for?

You have participated in the development of a site or an application, you have, of course, coded the unit tests (UT) and now you are “bored” with the integration tests, but what is that again?

Integration tests make it possible to test the functionalities of a system on all the components built via user scenarios.

With unit tests, we check the proper functioning of a specific part of an application/software, of a unit, while integration tests, as its name suggests, make it possible to test all the components in functional scenarios:

In the testing phase, integration testing falls between unit testing and validation testing.

The objective of integration tests is to detect errors that could not be detected (because they were not tested) during the unit testing phase.

Venom

To write and run our integration tests, we'll use Venom.
Venom is a tool created and made open-source by OVHcloud: https://github.com/ovh/venom

Venom

The tool, written in Go (and yes another one ;-) ), dates from 2017, it is open-sourced and I learned about it following a presentation made during an edition of Devoxx France conference.
We found it very promising at the time and decided to use it for our integration tests. A few years later of use in one of my previous professional experiences, I am still happy to talk about this great little tool and this will be an opportunity to test the new features of version 1.O+.

Concretely, it is necessary to write test suites (testsuite) in a YAML file. Venom runs executors (scripts, HTTP request, web, IMAP, etc.) and enforces assertions. It can also generate xUnit result files.

The tool has a list of existing executors:

For my part, I mainly use two of them: exec and http, which already cover a large majority of the cases to be tested.

But if this list is not enough for you, you can create your own executors ;-).

The tool has a list of assertions/statements, here is a non-exhaustive list:

  • ShouldEqual
  • ShouldNotEqual
  • ShouldAlmostEqual
  • ShouldBeNil
  • ShouldNotBeNil
  • ShouldBeTrue
  • ShouldBeFalse
  • ShouldBeZeroValue
  • ShouldBeGreaterThan
  • ShouldBeGreaterThanOrEqualTo
  • ShouldBeLessThan
  • ShouldBeLessThanOrEqualTo
  • ShouldBeBetween
  • ShouldNotBeBetween
  • ShouldContain
  • ShouldNotContain
  • ShouldContainKey
  • ShouldNotContainKey
  • ShouldBeIn
  • ShouldNotBeIn
  • ShouldBeEmpty
  • ShouldNotBeEmpty
  • ShouldHaveLength
  • ShouldStartWith
  • ShouldEndWith
  • ShouldBeBlank
  • ShouldContainSubstring
  • ShouldNotContainSubstring
  • ShouldBeChronological
  • ShouldNotExist
  • ...

Full list: https://github.com/ovh/venom

Venom runs integration tests which must be listed in a YAML format file.

In these test scenarios, Venom provides common variables:

  • {{.venom.testsuite}}
  • {{.venom.testsuite.filename}}
  • {{.venom.testcase}}
  • {{.venom.teststep.number}}
  • {{.venom.datetime}}
  • {{.venom.timestamp}}

One of the handy features of Venom is that we can use the output of a test case as a parameter or input to a next test case.

For example, we can query a web service that returns a JSON containing two elements: an ID and a user name for example, then reuse the user ID as a query parameter for a second web service.

We will see this case in more detail and as an example later in the article.

Installation

It has been pointed out to me that I very often put a certain expression in (all?) my articles, so I will not break the rule and use it again:
"Let's start with the beginning", let's start by installing Venom on our machine.

Go, go, go!

We can now install the Venom tool on our machine. Here are the commands to run:

  • For linux: 
$ curl https://github.com/ovh/venom/releases/download/v1.0.1/venom.linux-amd64 -L -o /usr/local/bin/venom && chmod +x /usr/local/bin/venom
Enter fullscreen mode Exit fullscreen mode
  • For MacOS: 
$ curl https://github.com/ovh/venom/releases/download/v1.0.1/venom.darwin-amd64 -L -o bin/venom
Enter fullscreen mode Exit fullscreen mode

And that's all … ;-). The tool is installed and ready to use!

To test it, we will execute the help command of the tool:

$ venom -h
Venom - RUN Integration Tests

Usage:
  venom [command]

Available Commands:
  help        Help about any command
  run         Run Tests
  update      Update venom to the latest release version: venom update
  version     Display Version of venom: venom version

Flags:
  -h, --help   help for venom

Use "venom [command] --help" for more information about a command.
Enter fullscreen mode Exit fullscreen mode

Let's understand the tool with an example

To understand how Venom works, let's create and run a first test suite together. The first assertions we are going to make, in this test suite, consist in checking whether the site we want to test (a public REST API, OVHcloud API for example):

  • is reachable (respond with a 200 status code)
  • responds in less than 5 seconds
  • returns a valid response (in JSON format)

Start by creating your test suite in a file named testsuite.yml for example.
Open it in your favorite editor or IDE and fill it with this content:

name: APIIntegrationTest

vars:
  url: https://eu.api.ovh.com/

testcases:
- name: GET http testcase, with 5 seconds timeout
  steps:
  - type: http
    method: GET
    url: {{.url}}/1.0/
    timeout: 5
    assertions:
    - result.statuscode ShouldEqual 200
    - result.timeseconds ShouldBeLessThan 1
    - result.bodyjson ShouldContainKey apis
    - result.body ShouldContainSubstring /dedicated/server
    - result.body ShouldContainSubstring /ipLoadbalancing
Enter fullscreen mode Exit fullscreen mode

And to run our test suite, just call the run command and pass it the path to your yml file as a parameter, if in your directory you have several yml files containing test suites. Or you just need to execute only the venom run command (without argument), if there is only one testsuite in your folder:

$ venom run

 • APIIntegrationTest (testsuite.yml)
     • GET-http-testcase-with-5-seconds-timeout SUCCESS
Enter fullscreen mode Exit fullscreen mode

You have written and run your first test suite with the HTTP Runner, congratulations! :)

Now, let's dissect this test suite:

name: APIIntegrationTest (1)

vars: (2)
  url: https://eu.api.ovh.com/

testcases: (3)
- name: GET http testcase, with 5 seconds timeout (4)
  steps:
  - type: http (5)
    method: GET
    url: {{.url}}/1.0/
    timeout: 5
    assertions: (6)
    - result.statuscode ShouldEqual 200
    - result.timeseconds ShouldBeLessThan 1
    - result.bodyjson ShouldContainKey apis
    - result.body ShouldContainSubstring /dedicated/server
    - result.body ShouldContainSubstring /ipLoadbalancing
Enter fullscreen mode Exit fullscreen mode

(1) A test suite (testsuite) must have a name
(2) You can define variables that you can use in test cases
(3) It is composed of one or more test cases (testcase)
(4) A test case is composed of a name and a sequence of steps
(5) A step is composed of a type (exec by default), a script (if type = exec) as well as assertions (tests to be performed)
(6) The assertions (affirmations) are the tests that we will do to validate that our application/site/script works as expected

Let's write our integration tests

We are now going to write a suite of integration tests that will allow us to get to the heart of the matter, to see several Venom executors and to realize that it is good to know some tips when doing Venom to avoid wasting several hours on a problem.

Let's start by creating a yaml file named “testsuite.yaml”:

name: IntegrityTest

vars:
  url: https://dog.ceo/api

testcases:
Enter fullscreen mode Exit fullscreen mode

The first test we are going to do, in this test suite, is to test whether the site we want to test (a public REST API):

  • is accessible
  • if its answer is not empty
  • if the expected elements are in its response (which is in JSON format) 
name: IntegrityTest

vars:
  url: https://dog.ceo/api

testcases:
- name: GetDogs
  steps:
  - type: http
    method: GET
    headers:
       Accept: application/json
       Content-Type: application/json
    url: "{{.url}}/breeds/list/all"
    retry: 1
    delay: 2
    assertions:
    - result.statuscode ShouldEqual 200
    - result.bodyjson ShouldContainKey message
Enter fullscreen mode Exit fullscreen mode

In this small example we use the notion of variable which is to be defined in the vars block at the top of the yaml file. We define a url variable.
This variable can be called and reused throughout the file thanks to the formalism {{.myvarname}}.

Once we've made the API call, we're testing two things:

  • that the result of the HTTP code is 200 (everything is fine)
  • that the JSON we retrieve contains the message key (it does because the JSON starts like this): {"message":{"affenpinscher":[],"african":[],"airedale" :[],"akita":[],"appenzeller":[],"basenji":[],"beagle":[],"bluetick":[],"borzoi":[],"bouvier": [],"boxer":[],"brabancon":[],"briard":[],"buhund":["norwegian"],"bulldog":["boston","english","french" ...]...}}

We do not change a winning team, we will test our test case:

$ venom run testsuite.yaml
 • IntegrityTest (testsuite.yaml)
     • GetDogs SUCCESS
Enter fullscreen mode Exit fullscreen mode

Great, the test works ;-).

Concretely, our JSON (once formatted in a readable way) is presented in this form:

{
    "message": {
        "affenpinscher": [],
        "african": [],
        "airedale": [],
        "akita": [],
        "appenzeller": [],
        "basenji": [],
        "beagle": [],
        "bluetick": [],
        "borzoi": [],
        "bouvier": [],
        "boxer": [],
        "brabancon": [],
        "briard": [],
        "buhund": [
            "norwegian"
        ],
        "bulldog": [
            "boston",
            "english",
            "french",
          ...
        ],
        ...
      }
}
Enter fullscreen mode Exit fullscreen mode

To go even further, we are now going to explore our JSON in order to test if in our JSON we have a bulldog attribute that contains an array and that its first element is equal to “bulldog”.

We therefore add an assertion that tests the existence of a “boston” element as the first element of the bulldog array:

    assertions:
    - result.statuscode ShouldEqual 200
    - result.bodyjson ShouldContainKey message
    - result.bodyjson.message.bulldog.bulldog0 ShouldEqual "boston"
Enter fullscreen mode Exit fullscreen mode

Let's run the test suite again:

$ venom run testsuite.yaml
 • IntegrityTest (testsuite.yaml)
     • GetDogs SUCCESS
Enter fullscreen mode Exit fullscreen mode

Cool, our test case still works.

Note that a nice feature of Venom is that you can reuse a value retrieved from a result of a test case, in another test case.
So if we want to retrieve the first sub-breed of bulldog retrieved above, it will suffice to call it like this:

{{.GetDogs.result.bodyjson.message.bulldog.bulldog0}}
Enter fullscreen mode Exit fullscreen mode

We are going to use it to make an API call that lists the images available for boston type bulldogs:

- name: GetBostonBulldog
  steps:
  - type: http
    method: GET
    url: "https://dog.ceo/api/breed/bulldog/{{.GetDogs.result.bodyjson.message.bulldog.bulldog0}}/images"
    retry: 1
    delay: 2
    assertions:
    - result.statuscode ShouldEqual 200
Enter fullscreen mode Exit fullscreen mode

Another cool feature is writing test cases that run a CLI or a script, for example:

- name: GetDogName
  steps:
  - script: echo 'bulldog'
    assertions:
    - result.code ShouldEqual 0
Enter fullscreen mode Exit fullscreen mode

This test case executes echo ‘bulldog’ and tests if the result of the script is successful or not (exit code equal to 0).

If we run a venom run on our three test cases, we can see that the test suite still works:

$ venom run testsuite.yaml
 • IntegrityTest (testsuite.yaml)
     • GetDogs SUCCESS
     • GetBostonBulldog SUCCESS
     • GetDogName SUCCESS
Enter fullscreen mode Exit fullscreen mode

One thing to know is that if you need to run a script on several lines, there is a trick to know for it to work. The “multi line script” works with this formalism for the script executor:

- name: Says hi to readers
  steps:
  - script: |
           #!/bin/bash
           echo “salut les lecteurs” | base64
    assertions:
       - result.code ShouldEqual 0
       - result.systemout ShouldNotEqual ""
       - result.systemout ShouldEndWith "=="
       - result.systemerr ShouldEqual ""
Enter fullscreen mode Exit fullscreen mode

You must imperatively put a pipe “|”.

Bonus tracks

Running integration tests automatically

When you write your integration tests, very quickly comes the need for automatic execution of these, on several different environments.

Is it possible ? I have good news for you, the answer is yes.

You can, create a Docker image containing Venom, use it in your CI/CD chain, and depending on the environment on which you want to run your tests, you just have to pass your parameters from your CI tool: Jenkins/Gitlab /CircleCI…, until running the tests through Venom.

How is it possible ? Thanks to the ability to pass variables/parameters to our testsuite when running the latter, thanks to setting a variable in the testsuite and using the --var parameter during the venom run:

mytestsuite.yml:

name: IntegrityTest

vars:
  url: https://myurl.dev/myendpoint

testcases:
- name: CreateUser
  steps:
  - type: http
    method: POST
    headers:
       Accept: application/json
       Content-Type: application/json
    url: "{{url}}/v1/user"
    retry: 1
    delay: 2
    assertions:
    - result.statuscode ShouldEqual 201
    - result.bodyjson ShouldContainKey id
    extracts:
      result.bodyjson.id: '{{license=.+}}'
Enter fullscreen mode Exit fullscreen mode 
$ venom run mytestsuite.yml --var url=http://myurl.dev/myendpoint
Enter fullscreen mode Exit fullscreen mode

The example above takes a URL as a parameter, with a default value, and tests an HTTP call with the POST method to the url http://myurl.dev/myendpoint/v1/user, passing in the headers an Accept and a Content-Type and validating if the HTTP response code is a 201 code AND that in the response, in JSON format, there is an element named “id”.

Export of the report in xUnit format

Venom allows an export of the result as a report in xUnit format:

$ venom run testsuite.yaml  --format=xml --output-dir="test-results"
 • IntegrityTest (testsuite.yaml)
     • GetDogs SUCCESS
     • GetBostonBulldog SUCCESS
     • GetDogName SUCCESS
Writing file test-results/test_results.xml
Enter fullscreen mode Exit fullscreen mode

Let's look at the contents of this output XML file:

$ cat test-results/test_results.xml
<?xml version="1.0" encoding="utf-8"?><testsuites>
  <testsuite name="IntegrityTest" package="testsuite.yaml" tests="3">
    <testcase classname="testsuite.yaml" name="GetDogs" time="0.415826625">
      <system-out></system-out>
      <system-err></system-err>
    </testcase>
    <testcase classname="testsuite.yaml" name="GetBostonBulldog" time="0.1746975">
      <system-out></system-out>
      <system-err></system-err>
    </testcase>
    <testcase classname="testsuite.yaml" name="GetDogName" time="0.047147083">
      <system-out><![CDATA[bulldog]]></system-out>
      <system-err></system-err>
    </testcase>
  </testsuite>
</testsuites>
Enter fullscreen mode Exit fullscreen mode

I find this export super practical as a step to add to its pipeline of its CI/CD chain.

Running multiple test suites

If you want to run several test suites, without having to launch several venom run it is possible:

$ venom run *
 • APIIntegrationTest (tests/testsuite.yml)
     • GET-http-testcase-with-5-seconds-timeout SUCCESS
 • IntegrityTest (testsuite.yaml)
     • GetDogs SUCCESS
     • GetBostonBulldog SUCCESS
     • GetDogName SUCCESS
Enter fullscreen mode Exit fullscreen mode

Or if your YAML are in a “tests” directory:

$ venom run tests/*
 • Title of TestSuite (tests/test.yaml)
     • TestCase-with-default-value-exec-cmd-Check-if-exit-code-1 SUCCESS
     • Title-of-First-TestCase SUCCESS
 • APIIntegrationTest (tests/testsuite.yml)
     • GET-http-testcase-with-5-seconds-timeout FAILURE
Enter fullscreen mode Exit fullscreen mode

Conclusion

Venom is a small tool very practical for writing and running integration tests. The only downside is that it's not well known. A few years ago I regretted the fact that its documentation was not thorough enough and that there was slowness concerning the processing of issues and PRs and I am happy to see that this problem is corrected and that the tool is actively maintained :).

Top comments (0)

Read next

bhleb profile image

Exploring ts-mockito: An Alternative Mocking Library for Node.js Unit Testing

Hleb Bandarenka -

mikestopcontinues profile image

How to Run Snapshot Tests in CI/CD with Playwright

Mike Stop Continues -

rainerhahnekamp profile image

How do I test and mock Standalone Components?

Rainer Hahnekamp -

mikestopcontinues profile image

Configuring Snapshot Tests in Playwright

Mike Stop Continues -