My first open source project - super simple cli testing tool

Introduction

In this article I will cover my first open source project commander. It tests cli apps based on tests defined in a simple yaml file.

Check it out: https://github.com/SimonBaeumer/commander

The problem?

For another project I needed a tool which had the following abilities:

• Use the same tool for different operating systems (mainly osx, linux, windows)
• Simple and easy writing of tests
• Simple installation (no heavy lib or language installation)
• Language independent
• pls not another DSL

Especially the heavy installation and language independence were important for me because I wanted to test an app inside different docker containers.

Sadly there wasn't a single tool for it. bats just works for bash, go's icmd and python's expect require you to install a language and be familiar with it.

Why using commander?

• Easy installation
• Generate tests automatically
• Fast execution
• Platform independent
• Just yaml

Installation

Download the binary and give it execution permissions. If you want to call it directly from your terminal add it your path environment variable.

Linux & osx

# linux amd64
curl -L https://github.com/SimonBaeumer/commander/releases/download/v1.0.1/commander-linux-amd64 -o commander
chmod +x commander 

# osx amd64
curl -L https://github.com/SimonBaeumer/commander/releases/download/v1.0.1/commander-darwin-amd64 -o commander
chmod +x commander 

windows amd64

curl -L https://github.com/SimonBaeumer/commander/releases/download/v1.0.1/commander-windows-amd64.exe -o commander.exe

... or download it manually and add it to your path.

Quick start!

# Let commander create your test!
$ .commander add echo hello dev.to
written to /current/working/dir/commander.yaml

# ... written to the default commander.yaml
$ cat commander.yaml
tests:
  echo hello dev.to:
    exit-code: 0
    stdout: hello dev.to

# Executes the generated commander config
$ commander test
Starting test file commander.yaml...

✓ echo hello dev.to

Duration: 0.003s
Count: 1, Failed: 0

A full commander.yaml example

As you can see the test suite is easy to understand and write.

tests:
  echo hello world:
    stdout:
      lines:
        1: hello world # assert a specifc line
      contains:
        - hello world # this is the default
      exactly: hello # Match stdout exactly
    # The same can be done for stderr
    exit-code: 0
    config:
      timeout: 5ms # set timeout for test
      dir: /tmp # set working dir
      env:
        SOME_ENV: value

  it should have a title: # You can also define a title to be more clear...
    command: echo title # ... and define your command under test here
    exit-code: 0

Define default test configs

Further you can define a default configuration for all tests. This can be overwritten by the local test configs.

config: # Define the config globally
  timeout: 5ms
  dir: /tmp
  env:
    KEY: VALUE

tests:
  print env and dir:
    command: "echo $KEY $(pwd)"
    stdout: VALUE /tmp
    exit-code: 0

Debbuging tests

Sometimes you need to debug your commands. This can be done with --verbose flag.

# Generate the test
$ commander add echo hello dev.to

# Execute it with logging output
$ commander test --verbose
Starting test file commander.yaml...

2019/03/28 16:56:48 title: 'echo hello dev.to'  ExitCode:  0
2019/03/28 16:56:48 title: 'echo hello dev.to'  Stdout:  hello dev.to
2019/03/28 16:56:48 title: 'echo hello dev.to'  Stderr:  
✓ echo hello dev.to

Duration: 0.002s
Count: 1, Failed: 0

Executing

# Execute another file instead of the default
$ commander test another.yaml

# Filter tests
$ commander test commander.yaml "echo hello"

Generating tests

# Add a test to the default commander.yaml
$ ./commander add echo hello
written to /tmp/commander.yaml

# Write to a given file
$ ./commander add --file=test.yaml echo hello
written to test.yaml

# Write to stdout and file
$ ./commander add --stdout echo hello
tests:
  echo hello:
    exit-code: 0
    stdout: hello

written to /tmp/commander.yaml

# Only to stdout
$ ./commander add --stdout --no-file echo hello
tests:
  echo hello:
    exit-code: 0
    stdout: hello

Conclusion

With commander you can easy generate your test cases and execute them on different platforms without learning a new language or installing annoying libs and tools.

Contributions are always warmly welcomed!

I hope some of you can use it in their projects :)

Link: https://github.com/SimonBaeumer/commander

Notice: Please give me feedback for my first blog post!