Test command line outputs with Github Actions

Context

As an open source project maintainer on Ritchie CLI, I recently had the necessity to tests if some command line outputs were working as expected.

What happened

I had to test the CLI commands on various operating system (windows, linux and macos), and I couldn't find an action to help us achieving what we wanted.

The project was developed in Golang, so I could easily create a binary for each OS and execute it. But I had to use some command like <command-line> &> output.txt to get an output file, and then compare it with a fix .txt file on the repository to check if the command output was the expected one.

Some of the issues I faced:

☞ I couldn't test any error behaviour (because the workflow would failed) without adding a || true at the end of the command above (to ignore the failure).

☞ Commands line could be different depending on the OS (DOS/Unix).

☞ It was difficult to check dynamic outputs (that would change for each execution, for example a data or version value that evolve all the time).

I ended up creating this action to resolve all those specific issues.

What does this action?

This action allows the user to compare a command line output (success or error) with a file content located on the repository, or to check if it contains a specific expression.

Therefore, it can be used to check almost everything a command line can return on the screen or create:
☞ if a specific line appears in a file using a cat command.
☞ a whole file content,
☞ if a file is located inside a specific directory through a ls command after a file creation,
☞ ...

Moreover, it's compatible with all os-supported runner. I had an issue to test it with windows at first due to the different between DOS/Unix, but as I could use a composite action performing operations using bash (as well a strip-ansi npm cli lib) to by-pass this problem.

How the action works

That way, I could check a command line output using 3 different ways:

☞ if the output is equal to an assert file
☞ if the output contains a specific line from an assert file
☞ if the output contains an expression

A successful display on the Actions tab on Github UI would look like this:

If the output is not the expected one, the action will also inform the difference (using a diff command) between both files / expressions.

My Workflows

The action is called Assert Command Line Output and you will find below different examples using it (with success and error tests for each one).

The action.yml file can be found here.

1️⃣ Assert file content

Expecting command output to be EQUAL to assert.txt file content

    steps:
      - uses: actions/checkout@v2.3.4
      - uses: GuillaumeFalourd/assert-command-line-output@v2
        with:
          command_line: ls -lha
          assert_file_path: path/to/assert.txt
          expected_result: PASSED

Expecting command output to be DIFFERENT than the assert.txt file content

    steps:
      - uses: actions/checkout@v2.3.4
      - uses: GuillaumeFalourd/assert-command-line-output@v2
        with:
          command_line: ls -lha
          assert_file_path: path/to/assert.txt
          expected_result: FAILED

2️⃣ Assert specific file line

Expecting command output line 3 to be EQUAL than the assert.txt file content in line 3

    steps:
      - uses: actions/checkout@v2.3.4
      - uses: GuillaumeFalourd/assert-command-line-output@v2
        with:
          command_line: ls -lha
          assert_file_path: path/to/assert.txt
          expected_result: PASSED
          specific_line: 3

Expecting command output line 3 to be DIFFERENT than the assert.txt file content in line 3

    steps:
      - uses: actions/checkout@v2.3.4
      - uses: GuillaumeFalourd/assert-command-line-output@v2
        with:
          command_line: ls -lha
          assert_file_path: path/to/assert.txt
          expected_result: FAILED
          specific_line: 3

3️⃣ Assert specific expression

Expecting command output to contain specific expression

    steps:
      - uses: actions/checkout@v2.3.4
      - uses: GuillaumeFalourd/assert-command-line-output@v2
        with:
          command_line: ls -lha
          contains: runner
          expected_result: PASSED

Expecting command output NOT to contain specific expression

    steps:
      - uses: actions/checkout@v2.3.4
      - uses: GuillaumeFalourd/assert-command-line-output@v2
        with:
          command_line: ls -lha
          contains: error
          expected_result: FAILED

Submission Category

This action could be useful for Maintainers (Maintainer Must-Haves) as well as Deployments (DIY Deployments) to check some specific outputs from commands used inside a CI/CD pipelines on Github Actions, or any periodic checks.

Yaml File or Link to Code

• The action can be found on the Github Marketplace

• You can find a full implementation example for each OS on the action repository, using cat or ls -lha commands as demonstrations:

☞ UBUNTU
☞ MACOS
☞ WINDOWS

Additional Resources / Info

☞ You can contribute to the repository following the Contributing.md file

☞ This action uses the Apache License 2.0

Conclusion

I believe everything is not perfect yet, this action evolved a lot since the first version, and I hope some of you will find it useful and will want to contribute to make it better. Thank you for reading 🚀