Yet Another DIY Testing Framework (part 1 of 3) - The idea

Disclaimer

All listed in this article (if this text can be considered as an article) is related to the world of backend Java, other modern enterprise technologies may have much better situation with a testing and testing frameworks.

Also an author is not a best in the world expert in Java testing frameworks, so there is a possibility that somewhere in the wild already exists some framework that solves all the issues and better option to use instead of inventing described below bicycle.
(If you know such, please let me know in comments)

Introduction (What problem we are solving?)

There are a lot of more or less good testing frameworks in the wild. Some of existing testing frameworks doing their job very well and are de facto a standard, like a "cucumber" for BDD testing.

And also important about all these testing frameworks that is more popular framework have more features and therefore have higher learning curve, less fast as in development new tests, as in tests execution, and may have (not necessary, for sure, but ...) some overcomplicated processes that is leading to unclear tests.

The Idea

Technical side

Java hava a build in scripting engine that can be used for multiple purposes.
This embedded script ion engine have a number of advantages for our task:

1. It is really easy to use. Minimal usage example is about 3 lines of code
2. One of the scripting languages implemented in that engine, and also build in is a JavaScript. Well know by almost everybody and extremely popular nowadays.
3. This engine have a bi-directional bridge between Java and JavaScript (and actually every other scripting language, but here we are interested in JavaScript)

And some disadvantages where most important is a performance. JavaScript code executed by this scripting engine is about 500 times slower than plain Java. Not a NodeJS performance for sure, but this performance is not an issue taking into account our area of interest.

The Framework Idea

Idea is quite simple:

1. Write an “executor” for “test scripts” enriched with useful functions for your application testing.
2. Write a “test runner” that will: 2.1. Find all “test scripts” from particular folder 2.2. Execute scripts with “executor” and collect the result 2.3. Assemble a testing report from collected results.

Framework in details

Test Script

To be honest “test script” is not a part of the framework, but anyway must be explained.

Test Script is a JavaScript file that tests some application function.
Test script is:

• use additional build in into “executor” functions to communicate with Application
• write logs to stdout (to simplify “test runner” task of gathering results
• fail with exceptions in case of some issues detected in tested application

Executor

Executor is a Java application that can execute JavaScript file passed there as a parameter.
Additionally executor should have a application configuration, to be able to connect to it in a right way.

In case of failed script execution, Executor must exit with non zero exit code.

Test Runner

Is an application implemented in any language you like (To be consistent better to implement it in Java or JavaScript to minimize amount of different technologies need to know by developers).

Test Runner scan a folder with tests and execute files from there with Executor.

Tests may be organized to sub folders by some Application aspect to test and test runner may process those sub folders in parallel (for improving performance).

Runner for each executed script collect:

1. Output from stdout (logs and errors)
2. Script exit code
3. Execution time

After executing all tests (scripts) all collected information may be assembled into one report (or number of reports in case of parallel tests execution).

Repot is a document in your preferred format (HTML, PDF, DOC, TXT, etc) with list of tests and their status.

At the end test runner must exit with non-zero exit code in case at least one test failed.
(This will simplify integration with external CI/CD systems)

How it expected to work (example)

For example if our testing framework consists of two java parts:

1. Executor - test-executor.jar
2. Test Runner - test-runner.jar

And we have following list of tests:

tests/
  Test_endpoint1_get.is
  Test_endpoint1_post.js
  Test_endpoint1_delete.js
  Test_endpoint1_auth.js

1. Tests run will be following

$ java -jar test-runner.jar tests/

1. Execution of one test script will be following

$ java -jar test-executor.jar tests/Test_endpoint1_post.js

1. All test report would be like this

Tests report:
tests/Test_endpoint1_get.is - ok (0.05sec)
tests/Test_endpoint1_post.js - ok (0.10sec)
tests/Test_endpoint1_delete.js - ERROR (exit code - 1)
  … Detailed log of error …
tests/Test_endpoint1_auth.js - ok (0.03sec)

PS: Please let me know if you like the idea and will be interested to see this framework implementation.