Command line application tests made easy with Outthentic

Writing tests is an essential part of software development. There are a plenty of frameworks to do that. In this post I'd like to introduce Outthentic - an universal script engine with embedded testing facilities, allow one to create BDD/Integration tests for command line applications easy.

Install

We will need the the cutting edge of Outthentic to see some recent features created especially for test development.

$ cpanm https://github.com/melezhik/outthentic.git

Application to test

Say, we have an application script - application.bash - which does all useful job.

Our goal to test this script properly:

$ nano /usr/local/bin/application.bash

#!bash

echo "Hello world"

Let's outline our simple testing plan:

• Ensure that script exit code is 0
• Ensure that script output certain string to STDOUT
• Skip test for a certain environment
• Abort test if script is not installed in the system

Ensure that script exit code is 0

This exit code is checked by default when external script gets through Outthentic, we just need to add a thin layer to enable this test:

$ mkdir check-exit-code

$ nano check-exit-code/story.bash

#!/bash

application.bash

Now let's run out first test and see the output:

$ strun --story check-exit-code


2018-09-26 16:35:47 :  [path] /check-exit-code/
/root/projects/outthentic-dev.to//check-exit-code/story.bash: line 1: /usr/local/bin/application.bash: Permission denied
not ok  scenario succeeded
STATUS  FAILED (2)

Well, the test has failed. And we see the reason. We forgot to set execution bit, it's easy to fix and re-run test again. This is what we have write our tests for - to see things that do not work, rather then see things that are fine.

$ chmod a+x /usr/local/bin/application.bash

2018-09-26 20:07:27 :  [path] /check-exit-code/
Hello world
ok      scenario succeeded
STATUS  SUCCEED

A few words about status code emitted by Outthentic. As you might have noticed the first failed test produces status code 2, which generally means test failures, there are 3 exit codes provided by Outthentic:

• 0 tests are passed, everything is ok
• 2 tests are failed, something definitely went wrong
• 1 some tests are passed, some are not, there is something wrong or there are some warnings

The last case makes it possible to use Outthentic tests as Consul check scripts

Ensure that script outputs some string to STDOUT

Before going into details, let's think why we would need it?

A few reasons ( I wonder if a reader would provide more ) :

• Some external scripts do not provide a sane exit code, the only reasonable thing we can test is an output

• None zero exit code might not means that program works unexpectedly. Again we can check output rather then exit code

• Program emits zero exit code ( finishes successfully ) but makes different output messages when being called with different parameters. So to check various test cases we need to run the same program differently and check different output

As our application is just an example, the check for "Hello world" string is trivial:

$ nano check-exit-code/story.check

Hello World

Now run:

$ strun --story check-exit-code

2018-09-26 17:23:29 :  [path] /check-exit-code/
Hello world
ok      scenario succeeded
ok      text has 'Hello world'
STATUS  SUCCEED

There are more things you can check with Outthentic. Imagine we want to check the program's output consists of sequential numbers from 1 to 10:

$ nano check-exit-code/story.check

begin:
generator: <<CODE
[ map { "regexp: ^\\d+$_" } (1..10) ]
CODE
end:

But let's go further.

Splitting tests for different cases

In real projects we can eventually have many test cases mapped to many test scenarios. Outthentic is flexible enough to adopt for such a scheme, running multiples tests as a suite.

Let's reorgonize our test structure a bit:

$ mkdir check-exit-code

$ nano check-exit-code/story.bash

#!/bash

application.bash


$ mkdir check-stdout 

$ nano check-exit-code/story.bash

#!/bash

application.bash


$ nano check-stdout/story.check

Hello World

Now we have two test scenarios. To check exit code and to check returned output.

These tests overlaps by the fact they both runs application.bash however for our purposes it is not critical, we just create different tests to emphasize what we what to test.

$ tree 

.
├── check-exit-code
│   └── story.bash
└── check-stdout
    ├── story.bash
    └── story.check

Finally Outthentic allows us to run all our tests recursively:

$ strun --recurse

2018-09-26 17:34:52 :  [path] /check-exit-code/
Hello world
ok      scenario succeeded
STATUS  SUCCEED
2018-09-26 17:34:52 :  [path] /check-stdout/
Hello world
ok      scenario succeeded
ok      text has 'Hello world'
STATUS  SUCCEED
STATUS  SUCCEED

As we have and more and more tests, we might need to narrow down the output, it is achievable by --format option of Outthentic test runner:

$ strun --recurse --format=production
2018-09-26 17:36:14 :  [path] /check-exit-code/
2018-09-26 17:36:14 :  [path] /check-stdout/

Now let's go to the two last points of our testing plan.

Sometimes we need to skip our tests for some reasons or even raise exception if some preliminary conditions are not met, so we don't waste our time and run tests polluting console with unnecessary messages, as we already know that we might not run test suite.

Quit test for some environment

Say, we don't want run tests for production environment which is defined by passing environment variable:

$ export environment=production

Outthentic allows to *immediately * quit test execution phase by using quit function, let's see an example of it:

$ nano check-exit-code/hook.bash

#!bash
if test "$environment" = "production"; then
  quite "production tests are disabled, please use dev environment"
fi

$ strun --story check-exit-code

2018-09-26 18:10:54 :  [path] /check-exit-code/
? forcefully exit:  production tests are disabled, please use dev environment
STATUS  SUCCEED

The opposite idea is to let your tests fail immediately upon a certain condition. You should choose outthentic_die function for this:

$ nano check-exit-code/hook.bash

#!bash

which application.bash 2>/dev/null || \
  outthentic_die "application.bash is not installed. You should install it to run tests"

$ unlink /usr/local/bin/application.bash
$ strun --story check-exit-code

2018-09-26 18:16:49 :  [path] /check-exit-code/
!! forcefully die:  application.bash is not installed. You should install it to run tests
STATUS  FAILED (2)

Now our project structure looks like this:

.
├── check-exit-code
│   ├── hook.bash
│   └── story.bash
└── check-stdout
    ├── story.bash
    └── story.check

Hook.bash is an example Outthentic hooks - small script gets run before main test run, you can read more about hooks on Outthentic documentation pages.

Decentralized or centralized testing model

As you could have noticed, we would have to add hook.bash to every test where we want to ensure preliminary conditions are met. In this scheme every test is treated as independent unit, and this follows the pattern we can see in many testing frameworks. While it seems easier to implement, it also results in duplication of code. Now we have hook.bash scripts doing the same job for every story.

Alternatively we might check those preliminary conditions ( like environment and application being installed ) once, in the very beginning, before we run any test.

This approach leads us to the opposite scheme, where all test become deepened on some "main" entry point where we can:

• perform initialization steps ( preliminary conditions check )

• call tests as functions in order

In Outthentic this type of design could be easily implemented through so called "story modules". Laterally when you call tests as a functions.

Let's slightly refactor our project to implement the idea:

$ nano hook.bash

#!bash

# this a main entrypoint 

if test "$environment" = "production"; then
  quite "production tests are disabled, please use dev environment"
fi

which application.bash 2>/dev/null || \
  outthentic_die "application.bash is not installed. You should install it to run tests"

run_story "check-exit-code"
run_story "check-stdout"


$ # we don't need hook.bash per story anymore

$ rm check-exit-code/hook.bash

# we make our tests - modules or functions, just when we copy those ones to modules/ folder

$ mkdir modules

$ mv check-exit-code check-stdout modules/

So we end up with this structure with one main entrypoint ( hook.bash ) and two dependable tests ( check-exit-code/story.bash , check-stdout/story.bash ), also notice that we run preliminary conditions check inside main entrypoint and control how and what tests to run.

.
├── hook.bash
└── modules
    ├── check-exit-code
    │   └── story.bash
    └── check-stdout
        ├── story.bash
        └── story.check

Now we are ready to run tests, note that we don't need --recurse option anymore, because all the tests sequence is defined through the hook.bash file.

$ strun

2018-09-26 20:24:20 :  [path] /modules/check-exit-code/
Hello world
ok      scenario succeeded
2018-09-26 20:24:20 :  [path] /modules/check-stdout/
Hello world
ok      scenario succeeded
ok      text has 'Hello world'
STATUS  SUCCEED

The end

This was just a brief introduction into testing capabilities provided by Outthentic framework. If you like it - go to GH pages to see all the details.

Feel free to share you opinions here in comments.