DEV Community

loading...

Simple blackbox testing with Raku and Sparrow6

melezhik profile image Alexey Melezhik Updated on ・6 min read

Sparrow6 is a Raku automation framework with embedded tool that allows to create blackbox testing scenarios.

The feature is called Task Checks DSL.

Blackbox testing is based on the idea of agnosticism of internal structure of an object being tested. Sparrow6 takes a very simple approach allows one to create regexp based rules to analyze external scripts output.

Following are some examples.


Plain string search

Just create a task check file that includes a searched string

task.bash

!bash
echo "Hello World"

task.check

Hello

Test run:

perl6 -MSparrow6::DSL -e task-run

Output:

20:01:06 12/23/2019 [/home/melezhik/projects/blackbox-testing/plain-string] Hello World
[task check] stdout match <Hello> True

Regexp search

Sparrow6 allows to use Raku regular expressions to search in script output:

task.check

regexp: ^^ Hello \s+ World

Test run:

perl6 -MSparrow6::DSL -e task-run

Output:

20:03:20 12/23/2019 [/home/melezhik/projects/blackbox-testing/regexp] Hello World
[task check] stdout match <^^ Hello \s+ World> True

Sequential search

Sometime one need to test if a sequence of lines appears in a script output. begin:, end: notation is used to check sequences:

task.bash

!bash
echo "ON"
echo 1
echo 2
echo 3
echo "OFF"

task.check

begin:
  ON
  1
  2
  3
  OFF
end

Test run:

perl6 -MSparrow6::DSL -e task-run

Output:

20:34:06 12/23/2019 [/home/melezhik/projects/blackbox-testing/sequential] ON
20:34:06 12/23/2019 [/home/melezhik/projects/blackbox-testing/sequential] 1
20:34:06 12/23/2019 [/home/melezhik/projects/blackbox-testing/sequential] 2
20:34:06 12/23/2019 [/home/melezhik/projects/blackbox-testing/sequential] 3
20:34:06 12/23/2019 [/home/melezhik/projects/blackbox-testing/sequential] OFF
[task check] stdout match (s) <ON> True
[task check] stdout match (s) <1> True
[task check] stdout match (s) <2> True
[task check] stdout match (s) <3> True
[task check] stdout match (s) <OFF> True

If switch first two digits, the test will fail:

task.bash

!bash
echo "ON"
echo 2
echo 1
echo 3
echo "OFF"

Test run:

perl6 -MSparrow6::DSL -e task-run

Output:

20:37:22 12/23/2019 [/home/melezhik/projects/blackbox-testing/sequential] ON
20:37:22 12/23/2019 [/home/melezhik/projects/blackbox-testing/sequential] 2
20:37:22 12/23/2019 [/home/melezhik/projects/blackbox-testing/sequential] 1
20:37:22 12/23/2019 [/home/melezhik/projects/blackbox-testing/sequential] 3
20:37:23 12/23/2019 [/home/melezhik/projects/blackbox-testing/sequential] OFF
[task check] stdout match (s) <ON> True
[task check] stdout match (s) <1> False
[task check] stdout match (s) <2> False
[task check] stdout match (s) <3> False
[task check] stdout match (s) <OFF> False
=================
TASK CHECK FAIL

Range search

If one doesn't care about lines order and need to check
that lines are included between certain lines, it's possible to reshape the previous example using range notation: between {'RexExp1'} {'RegExp2'}, where RexExp1 sets a regexp pattern for the very first string in the range, and the second one does the same for the very last string in the range, all other strings should appear within the range in no particular order.

task.check

between: {'ON'} {'OFF'}
  1
  2
  3
end:

Test run:

perl6 -MSparrow6::DSL -e task-run

Output:

20:49:56 12/23/2019 [/home/melezhik/projects/blackbox-testing/ranges] ON
20:49:56 12/23/2019 [/home/melezhik/projects/blackbox-testing/ranges] 2
20:49:56 12/23/2019 [/home/melezhik/projects/blackbox-testing/ranges] 1
20:49:56 12/23/2019 [/home/melezhik/projects/blackbox-testing/ranges] 3
20:49:56 12/23/2019 [/home/melezhik/projects/blackbox-testing/ranges] OFF
[task check] stdout match (r) <1> True
[task check] stdout match (r) <2> True
[task check] stdout match (r) <3> True

Dynamic search

Sparrow6 equips one with an ability to generate search criteria in run time, using a variety of programming languages ( Bash,Perl,Python,Ruby,Raku,Powershell)

Let's reshape the last example using generator: notation:

task.check

generator: <<CODE
!perl6

  say "between: \{'ON'\} \{'OFF'\}\n",(1 ... 3).join("\n"), "\nend:";

CODE

Changing shebang we choose different language to implement dynamic search.

Let's do implementation on Bash.

task.check

generator: <<CODE
!bash

  echo "between: {'ON'} {'OFF'}"
  for i in {1..3}; do echo $i; done
  echo 'end:'

CODE

Captures

Captures allow to extract certain chunks from searched strings and process them.

Let's count a total sum for all numbers included between "ON" and "OFF" strings:

task.check

between: {'ON'} {'OFF'}
 regexp: (\d+)
end:

code: <<CODE
!perl6

my $total;
for captures()<> -> $c {
 $total+=$c[0]
}

update_state(%( cnt => $total ))

CODE

Test run:

perl6 -MSparrow6::DSL -e 'my %state = task-run; say %state.perl'

Output:

23:13:23 12/23/2019 [/home/melezhik/projects/blackbox-testing/captures] ON
23:13:23 12/23/2019 [/home/melezhik/projects/blackbox-testing/captures] 2
23:13:23 12/23/2019 [/home/melezhik/projects/blackbox-testing/captures] 1
23:13:23 12/23/2019 [/home/melezhik/projects/blackbox-testing/captures] 3
23:13:23 12/23/2019 [/home/melezhik/projects/blackbox-testing/captures]
OFF
[task check] stdout match (r) <(\d+)> True
{:cnt(6)}

Asserts

Asserts allow to set predicates - statements that return true of false and create related checks.

In previous example let's check if total sum of numbers is equal 6:

task.check

between: {'ON'} {'OFF'}
 regexp: (\d+)
end:

generator: <<CODE
!perl6

my $total;
for captures()<> -> $c {
 $total+=$c[0]
}

say "assert: { $total == 6  }", " total sum is 6";

CODE

Test run:

perl6 -MSparrow6::DSL -e task-run

Output:

19:28:12 12/24/2019 [/home/melezhik/projects/blackbox-testing/asserts] ON
19:28:12 12/24/2019 [/home/melezhik/projects/blackbox-testing/asserts] 1
19:28:12 12/24/2019 [/home/melezhik/projects/blackbox-testing/asserts] 2
19:28:12 12/24/2019 [/home/melezhik/projects/blackbox-testing/asserts] 3
19:28:12 12/24/2019 [/home/melezhik/projects/blackbox-testing/asserts] OFF
[task check] stdout match (r) <(\d+)> True
[task check] <total sum is 6> True

Passing parameters

One can pass parameters handled in dynamic search constructions.

task.check

between: {'ON'} {'OFF'}
 regexp: (\d+)
end:

generator: <<CODE
!perl6

my $amount = config()<amount>;

my $total;
for captures()<> -> $c {
 $total+=$c[0]
}

say "assert: { $total == $amount  }", " total sum is amount";
CODE

Output:

perl6 -MSparrow6::DSL -e 'task-run("{$*CWD}",%( amount => 10 ))'
19:34:46 12/24/2019 [/home/melezhik/projects/blackbox-testing/parameters] ON
19:34:46 12/24/2019 [/home/melezhik/projects/blackbox-testing/parameters] 1
19:34:46 12/24/2019 [/home/melezhik/projects/blackbox-testing/parameters] 2
19:34:46 12/24/2019 [/home/melezhik/projects/blackbox-testing/parameters] 3
19:34:46 12/24/2019 [/home/melezhik/projects/blackbox-testing/parameters] OFF
[task check] stdout match (r) <(\d+)> True
[task check] <total sum is 10> False
=================
TASK CHECK FAIL

Streams

The last interesting feature to be mentioned here is called streams.
Steams allow one to iterate over blocks of data found during sequential or range search.

In the last example consider the case when we have more then one block of numbers between 'ON' and 'OFF' delimiters.

task.bash


echo "ON"
echo 1
echo 2
echo 3
echo "OFF"

echo "ON"
echo 3
echo 3
echo 3
echo "OFF"

echo "ON"
echo 2
echo 2
echo 2
echo "OFF"

Let's write an iterator for all the blocks found:

between: {'ON'} {'OFF'}
 regexp: (\d+)
end:

code: <<CODE
!perl6

  my $stream-num = 1;
  for streams_array()<> -> $s {
    for $s<> -> $i {
      say "stream#{$stream-num}: {$i[0]}";
    }
    $stream-num++;
  }

CODE

Test run:

perl6 -MSparrow6::DSL -e 'task-run

Output:

19:50:43 12/24/2019 [/home/melezhik/projects/blackbox-testing/streams] ON
19:50:43 12/24/2019 [/home/melezhik/projects/blackbox-testing/streams] 1
19:50:43 12/24/2019 [/home/melezhik/projects/blackbox-testing/streams] 2
19:50:43 12/24/2019 [/home/melezhik/projects/blackbox-testing/streams] 3
19:50:43 12/24/2019 [/home/melezhik/projects/blackbox-testing/streams] OFF
19:50:43 12/24/2019 [/home/melezhik/projects/blackbox-testing/streams] ON
19:50:43 12/24/2019 [/home/melezhik/projects/blackbox-testing/streams] 3
19:50:43 12/24/2019 [/home/melezhik/projects/blackbox-testing/streams] 3
19:50:43 12/24/2019 [/home/melezhik/projects/blackbox-testing/streams] 3
19:50:43 12/24/2019 [/home/melezhik/projects/blackbox-testing/streams] OFF
19:50:43 12/24/2019 [/home/melezhik/projects/blackbox-testing/streams] ON
19:50:43 12/24/2019 [/home/melezhik/projects/blackbox-testing/streams] 2
19:50:43 12/24/2019 [/home/melezhik/projects/blackbox-testing/streams] 2
19:50:43 12/24/2019 [/home/melezhik/projects/blackbox-testing/streams] 2
19:50:43 12/24/2019 [/home/melezhik/projects/blackbox-testing/streams] OFF
[task check] stdout match (r) <(\d+)> True
[task check] stream#1: 1
[task check] stream#1: 2
[task check] stream#1: 3
[task check] stream#2: 3
[task check] stream#2: 3
[task check] stream#2: 3
[task check] stream#3: 2
[task check] stream#3: 2
[task check] stream#3: 2

Conclusion

Task Checks DSL is a simple, yet powerful tool allow one to write blackbox tests. The feature is a part of Sparrow6 core and provided out of the box.


Thank you for reading and Merry Christmas!

Discussion

pic
Editor guide