Welcome to the Tsonnet series!
If you're just joining, you can check out how it all started in the first post of the series.
In the previous post, we added support for indexing strings and extracted the duplication into an indexable module:
Since I added the Env module, I feel uneasy about the fact that it's only being indirectly tested, so I've decided to cover it thoroughly.
Adding property-based testing for the environment
The environment is supposed to store any type of expr and it is difficult to manually test all possible cases ourselves. I believe property-based testing is perfect for this scenario.
If you don't know what property-based tests are, or want a refresher, I've got you covered with this short article:
Ok, ready?
First things first. Add the dependencies:
diff --git a/dune-project b/dune-project
index 39aaa64..0ca3041 100644
--- a/dune-project
+++ b/dune-project
@@ -30,6 +30,18 @@
(and
:with-test
(>= 1.8.0)))
+ (qcheck-core
+ (and
+ :with-test
+ (>= 0.25)))
+ (qcheck-alcotest
+ (and
+ :with-test
+ (>= 0.25)))
+ (ppx_deriving_qcheck
+ (and
+ :with-test
+ (>= 0.6)))
(bisect_ppx
(and
:with-test
diff --git a/tsonnet.opam b/tsonnet.opam
index 751e24b..1f3ee9d 100644
--- a/tsonnet.opam
+++ b/tsonnet.opam
@@ -14,6 +14,9 @@ depends: [
"dune" {>= "3.16" & >= "3.16.0"}
"menhir" {= "20240715"}
"alcotest" {with-test & >= "1.8.0"}
+ "qcheck-core" {with-test & >= "0.25"}
+ "qcheck-alcotest" {with-test & >= "0.25"}
+ "ppx_deriving_qcheck" {with-test & >= "0.6"}
"bisect_ppx" {with-test & >= "2.8.3"}
"yojson" {>= "2.2.2"}
"odoc" {with-doc}
The qcheck library is a QuickCheck (written in Haskell) inspired property-based testing library for OCaml. It pairs well with other testing libraries, like alcotest which is my preference for unit testing. The ppx_deriving_qcheck is just a submodule from qcheck -- I'll explain why and what it does in a bit.
Now let's configure dune to run the tests using these libraries:
diff --git a/test/dune b/test/dune
index bd42c05..37c860c 100644
--- a/test/dune
+++ b/test/dune
@@ -1,5 +1,3 @@
-(test
- (name test_tsonnet)
- (libraries tsonnet alcotest)
- (deps
- (source_tree ../samples)))
+(tests
+ (names test_env)
+ (libraries tsonnet alcotest qcheck-core qcheck-alcotest))
diff --git a/test/test_tsonnet.ml b/test/test_tsonnet.ml
deleted file mode 100644
index e69de29..0000000
We're going to target the test_env file, and drop test_tsonnet since it's an empty file so far.
And here's the test/test_env.ml file:
open Tsonnet__Ast
module Env = Tsonnet__Env
let succ env expr = Result.ok (env, expr)
let err = Result.error
let test_undefined_variable () =
let result = Result.get_error (Env.find_var "foo" Env.empty ~succ ~err) in
let expected = Result.get_error (Result.error "Undefined variable: foo") in
Alcotest.(check string) "returns error" expected result
let gen_position = QCheck.Gen.return dummy_pos
let rec gen_expr_sized n =
if n <= 0
then
QCheck.Gen.oneofl [Unit]
else
let pos_gen = gen_position in
QCheck.Gen.frequency [
(1, QCheck.Gen.return Unit);
(1, QCheck.Gen.map (fun pos -> Null pos) pos_gen);
(1, QCheck.Gen.map2 (fun pos n -> Number (pos, n)) pos_gen (gen_number));
(1, QCheck.Gen.map2 (fun pos b -> Bool (pos, b)) pos_gen QCheck.Gen.bool);
(1, QCheck.Gen.map2 (fun pos s -> String (pos, s)) pos_gen QCheck.Gen.string);
(1, QCheck.Gen.map2 (fun pos s -> Ident (pos, s)) pos_gen QCheck.Gen.string);
(2, QCheck.Gen.map2
(fun pos exprs -> Array (pos, exprs))
pos_gen
(QCheck.Gen.list_size (QCheck.Gen.int_range 0 3) (gen_expr_sized (n-1)))
);
(2, QCheck.Gen.map2
(fun pos exprs -> Object (pos, exprs))
pos_gen
(QCheck.Gen.list_size
(QCheck.Gen.int_range 0 3)
(QCheck.Gen.pair QCheck.Gen.string (gen_expr_sized (n-1)))
)
);
(1, QCheck.Gen.map4
(fun pos op e1 e2 -> BinOp (pos, op, e1, e2))
pos_gen
gen_bin_op
(gen_expr_sized (n-1)) (gen_expr_sized (n-1))
);
(1, QCheck.Gen.map3
(fun pos op e1 -> UnaryOp (pos, op, e1))
pos_gen
gen_unary_op
(gen_expr_sized (n-1))
);
(2, QCheck.Gen.map2
(fun pos exprs -> Local (pos, exprs))
pos_gen
(QCheck.Gen.list_size (QCheck.Gen.int_range 0 3)
(QCheck.Gen.pair QCheck.Gen.string (gen_expr_sized (n-1)))
)
);
(2, QCheck.Gen.map
(fun exprs -> Seq exprs)
(QCheck.Gen.list_size (QCheck.Gen.int_range 0 3) (gen_expr_sized (n-1)))
);
(1, QCheck.Gen.map3
(fun pos varname e1 -> IndexedExpr (pos, varname, e1))
pos_gen
QCheck.Gen.string
(gen_expr_sized (n-1))
);
]
let gen_expr = gen_expr_sized 5
let arbitrary_expr = QCheck.make gen_expr
let test_lookup =
let local_env = ref Env.empty in
QCheck.Test.make
~count:1000
~name:"Variable lookup successfully retrieves it from the environment"
QCheck.(pair string arbitrary_expr)
(fun (varname, expr) ->
let new_env = Env.Map.add varname expr !local_env in
local_env := new_env;
Env.find_var varname new_env ~succ ~err |> Result.is_ok
)
let () =
let open Alcotest in
run "Env" [
"find_var", [
test_case "Undefined variable" `Quick test_undefined_variable;
QCheck_alcotest.to_alcotest test_lookup;
];
]
Let's break it down:
- The
test_undefined_variablecase is an example-based test to check when variable is not in the environment. - The property-based magic happens in
test_lookup. It will start with an empty environment, generate 1000 random pairs ofstringandarbitrary_expr, update the environment, and look it up usingEnv.find_var. It should successfully retrieve theexprfrom the environment. Pretty cool, isn't it?
Because we want to test the environment as thoroughly as possible, we generate random values of expr, covering all its variants. Since expr is a recursive type, we need to add a limit to avoid generating infinitely nested values with gen_expr_sized 5 -- I'm arbitrarily setting it to 5, as it strikes a good balance between performance and completeness.
The expr type had to be manually generated to exert control, due to its recursive nature. However, the other types are simple enough that we can derive them automatically -- enters ppx_deriving_qcheck:
diff --git a/lib/ast.ml b/lib/ast.ml
index feb40d5..305f79e 100644
--- a/lib/ast.ml
+++ b/lib/ast.ml
@@ -5,23 +5,32 @@ type bin_op =
| Subtract
| Multiply
| Divide
+ [@@deriving qcheck]
type unary_op =
| Plus
| Minus
| Not
| BitwiseNot
+ [@@deriving qcheck]
type number =
| Int of int
| Float of float
+ [@@deriving qcheck]
type position = {
startpos: Lexing.position;
endpos: Lexing.position;
}
The types bin_op, unary_op, and number are derived using ppx_deriving_qcheck. This allows the library to automatically create generators for these types.
Now the only thing missing is configuring the source code to be pre-processed by ppx_deriving_qcheck during the compilation phase:
diff --git a/lib/dune b/lib/dune
index 05c4dc3..75a7340 100644
--- a/lib/dune
+++ b/lib/dune
@@ -2,7 +2,9 @@
(name tsonnet)
(instrumentation
(backend bisect_ppx))
- (libraries yojson))
+ (libraries yojson)
+ (preprocess
+ (pps `ppx_deriving_qcheck`)))
(menhir
(modules parser))
We run and it outputs the results:
$ dune runtest
qcheck random seed: 872135876
Testing `Env'.
This run has ID `SIPXKW3X'.
[OK] find_var 0 Undefined variable.
[OK] find_var 1 Variable lookup successfully retrieves it from the environment.
Full test results in `~/BitMaybeWise/gitlab/tsonnet/_build/default/test/_build/_tests/Env'.
Test Successful in 0.014s. 2 tests run.
It doesn't show in the output, but qcheck performed the lookup 1000 times, with a thousand different values.
Now I'm at ease, as qcheck is battle-testing the find_var function with inputs that I wouldn't be able to generate manually. :)
Conclusion
The combination of qcheck and ppx_deriving_qcheck gives us confidence that our Env module can handle the full spectrum of expressions we throw at it – from simple values to complex nested structures. This investment in testing infrastructure will pay dividends as Tsonnet grows more sophisticated, ensuring that our foundation remains solid as we build more advanced features on top of it.
If you found this post as satisfying as watching 1000 random tests pass, subscribe to Bit Maybe Wise for more Tsonnet adventures. I promise my content generation is more deterministic than our property-based tests!
Photo by Sufyan on Unsplash
Top comments (0)