DEV Community

Hercules Lemke Merscher
Hercules Lemke Merscher

Posted on • Originally published at bitmaybewise.substack.com

Tsonnet #45 - The devil in the details #1

#tsonnet #jsonnet #compiler

Welcome to the Tsonnet series!

If you're not following along, check out how it all started in the first post of the series.

In the previous post, we removed every catch-all pattern match across the compiler, letting OCaml's exhaustiveness checker tell us exactly which variants were being silently ignored:

Tsonnet #44 - The devil in the details #0
bitmaybewise profile

Tsonnet #44 - The devil in the details #0

#jsonnet #tsonnet #compiler #ocaml
1
Comments
4 min read

The scope checker is on the operating table first. Let's go.

Telescope

Scope-checking conditionals

The If variant was the most obvious omission — the scope checker had no arm for it. Before the catch-all removal, it was swallowed by | _ -> ok (). After the first cleanup pass in the previous post it was still sitting in a list of variants that were silently accepted:

(* For any other expression types, no special scope validation needed *)
| Unit | Null _ | Number _ | String _ | Bool _ | EvaluatedObject _
| RuntimeObject _ | ObjectPtr _ | FunctionDef _ | FunctionCall _ | Closure _
| If _ -> ok ()
Enter fullscreen mode Exit fullscreen mode

That's wrong. An if can contain self or $ in its condition, then-branch, or else-branch. If that if is outside an object, the scope checker should catch it.

// samples/errors/conditional_self_out_of_scope.jsonnet
local value = if true then self.one else 1;
value
Enter fullscreen mode Exit fullscreen mode

Before this fix, that passed without error. validate_if recursively walks all three branches:

diff --git a/lib/scope.ml b/lib/scope.ml
index 48dfa83..b4e36a3 100644
--- a/lib/scope.ml
+++ b/lib/scope.ml
@@ -51,10 +50,12 @@ let rec _validate expr context =
     _validate expr context
   | IndexedExpr (_, _, index_expr) ->
     _validate index_expr context
+  | If (_, cond_expr, then_expr, else_expr_opt) ->
+    validate_if cond_expr then_expr else_expr_opt context
   (* For any other expression types, no special scope validation needed *)
   | Unit | Null _ | Number _ | String _ | Bool _ | EvaluatedObject _
   | RuntimeObject _ | ObjectPtr _ | FunctionDef _ | FunctionCall _ | Closure _
-  | If _ -> ok ()
+    -> ok ()
Enter fullscreen mode Exit fullscreen mode 
and validate_if cond_expr then_expr else_expr_opt context =
  _validate cond_expr context >>= fun () ->
  _validate then_expr context >>= fun () ->
  (match else_expr_opt with
  | Some else_expr -> _validate else_expr context
  | None -> ok ()
  )
Enter fullscreen mode Exit fullscreen mode

Now the earlier example correctly errors:

$ dune exec -- tsonnet samples/errors/conditional_self_out_of_scope.jsonnet
ERROR: samples/errors/conditional_self_out_of_scope.jsonnet:1:27 Can't use self outside of an object

1: local value = if true then self.one else 1;
   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[1]
Enter fullscreen mode Exit fullscreen mode

The positive case — self inside an if inside an object — still works as expected:

// samples/conditionals/conditional_object_scope.jsonnet
{
  one: 1,
  two: if true then self.one + 1 else 0,
  three: if false then 0 else $.one + 2,
}
Enter fullscreen mode Exit fullscreen mode

Scope-checking indexed object field access

The ObjectFieldAccess handler was already checking the scope reference — self, $, or ObjVarRef — against the context. But it was ignoring the chain of field expressions.

| ObjectFieldAccess (pos, scope, _) ->
  validate_object_field_access pos scope context
Enter fullscreen mode Exit fullscreen mode

The chain (e.g. [self.one] in obj[self.one].two) was not being validated. A chain expression containing self or $ outside an object went undetected.

// samples/errors/object_field_access_index_self_out_of_scope.jsonnet
local obj = { one: 1 };
obj[self.one].two
Enter fullscreen mode Exit fullscreen mode 
// samples/errors/object_field_access_computed_self_out_of_scope.jsonnet
local obj = { one: 1 };
obj.[self.one].two
Enter fullscreen mode Exit fullscreen mode 
// samples/errors/object_field_access_index_toplevel_out_of_scope.jsonnet
local obj = { one: 1 };
obj[$.one].two
Enter fullscreen mode Exit fullscreen mode

The fix threads the chain through to validate_expression_list:

-  | ObjectFieldAccess (pos, scope, _) ->
-    validate_object_field_access pos scope context
+  | ObjectFieldAccess (pos, scope, chain) ->
+    validate_object_field_access pos scope chain context
Enter fullscreen mode Exit fullscreen mode

Inside validate_object_field_access, after the scope check runs, the chain gets validated in sequence:

-  match scope with
-  | Self | TopLevel ->
-    if not context.in_object then
-      let with_error_msg = match scope with
-        | Self -> Error.Msg.self_out_of_scope
-        | TopLevel -> Error.Msg.no_toplevel_object
-        | ObjVarRef _ -> "" (* unreachable *)
-      in
-      Error.error_at pos with_error_msg
-    else ok ()
-  | ObjVarRef _ ->
-    (* Variable references are allowed anywhere *)
-    ok ()
+  let* () =
+    match scope with
+    | Self | TopLevel ->
+      if not context.in_object then
+        let with_error_msg = match scope with
+          | Self -> Error.Msg.self_out_of_scope
+          | TopLevel -> Error.Msg.no_toplevel_object
+          | ObjVarRef _ -> "" (* unreachable *)
+        in
+        Error.error_at pos with_error_msg
+      else ok ()
+    | ObjVarRef _ ->
+      (* Variable references are allowed anywhere *)
+      ok ()
+  in
+  validate_expression_list chain context
Enter fullscreen mode Exit fullscreen mode

All three variations now produce the correct scope errors.

Scope-checking function definitions

FunctionDef was completely unvalidated. A function defined at the top level could reference self or $ in its body or parameter defaults without triggering any error. validate_function_def walks each parameter's optional default expression, then validates the body:

+  | FunctionDef (_, def) ->
+    validate_function_def def context
   (* For any other expression types, no special scope validation needed *)
   | Unit | Null _ | Number _ | String _ | Bool _ | EvaluatedObject _
-  | RuntimeObject _ | ObjectPtr _ | FunctionDef _ | FunctionCall _ | Closure _
+  | RuntimeObject _ | ObjectPtr _ | FunctionCall _ | Closure _
     -> ok ()
Enter fullscreen mode Exit fullscreen mode 
and validate_function_def def context =
  let* () = List.fold_left
    (fun acc (_, default_expr) ->
      acc >>= fun () ->
      match default_expr with
      | Some expr -> _validate expr context
      | None -> ok ()
    )
    (ok ())
    def.params
  in
  _validate def.body context
Enter fullscreen mode Exit fullscreen mode

The four error samples:

// samples/errors/function_def_self_out_of_scope.jsonnet
local get_value() = self.value;
get_value()
Enter fullscreen mode Exit fullscreen mode 
// samples/errors/function_def_toplevel_out_of_scope.jsonnet
local get_value() = $.value;
get_value()
Enter fullscreen mode Exit fullscreen mode 
// samples/errors/function_def_default_self_out_of_scope.jsonnet
local get_value(value = self.value) = value;
get_value()
Enter fullscreen mode Exit fullscreen mode 
// samples/errors/function_def_default_toplevel_out_of_scope.jsonnet
local get_value(value = $.value) = value;
get_value()
Enter fullscreen mode Exit fullscreen mode

All four now produce the correct scope errors.

Scope-checking function calls

FunctionCall was the next silent passer. A call like get_value(self.value) at the top level should fail:

// samples/errors/function_call_arg_self_out_of_scope.jsonnet
local get_value(value) = value;
get_value(self.value)
Enter fullscreen mode Exit fullscreen mode

validate_function_call validates the callee expression and every argument, positional or named:

+  | FunctionCall (_, call) ->
+    validate_function_call call context
   (* Terminal variants *)
   | Null _ | Number _ | String _ | Bool _ -> ok ()
Enter fullscreen mode Exit fullscreen mode 
and validate_function_call call context =
  let* () = _validate call.callee context in
  List.fold_left
    (fun acc arg ->
      acc >>= fun () ->
      match arg with
      | Positional expr -> _validate expr context
      | Named (_, expr) -> _validate expr context
    )
    (ok ())
    call.args
Enter fullscreen mode Exit fullscreen mode

Six error samples cover callee, positional args, and named args for both self and $:

// samples/errors/function_call_callee_self_out_of_scope.jsonnet
(self.get_value)()
Enter fullscreen mode Exit fullscreen mode 
// samples/errors/function_call_callee_toplevel_out_of_scope.jsonnet
($.get_value)()
Enter fullscreen mode Exit fullscreen mode 
// samples/errors/function_call_arg_self_out_of_scope.jsonnet
local get_value(value) = value;
get_value(self.value)
Enter fullscreen mode Exit fullscreen mode 
// samples/errors/function_call_arg_toplevel_out_of_scope.jsonnet
local get_value(value) = value;
get_value($.value)
Enter fullscreen mode Exit fullscreen mode 
// samples/errors/function_call_named_arg_self_out_of_scope.jsonnet
local get_value(value) = value;
get_value(value = self.value)
Enter fullscreen mode Exit fullscreen mode 
// samples/errors/function_call_named_arg_toplevel_out_of_scope.jsonnet
local get_value(value) = value;
get_value(value = $.value)
Enter fullscreen mode Exit fullscreen mode

Scope-checking closures (unified with function definitions)

Closure was the last variant sitting in the catch-all. It has the same structure as FunctionDef -- both have params with optional defaults and a body. Instead of duplicating the validation, I unified them:

-  | FunctionDef (_, def) ->
-    validate_function_def def context
+  | FunctionDef (_, {name = _; params; body})
+  | Closure (_, {params; body}) ->
+    validate_function_signature_and_body params body context
Enter fullscreen mode Exit fullscreen mode

The old validate_function_def renamed to validate_function_signature_and_body and takes params and body directly:

-and validate_function_def def context =
+and validate_function_signature_and_body params body context =
   let* () = List.fold_left
     (fun acc (_, default_expr) ->
       acc >>= fun () ->
       match default_expr with
       | Some expr -> _validate expr context
       | None -> ok ()
     )
     (ok ())
-    def.params
+    params
   in
-  _validate def.body context
+  _validate body context
Enter fullscreen mode Exit fullscreen mode

Four error samples cover body and defaults for both self and $:

// samples/errors/closure_self_out_of_scope.jsonnet
function() self.value
Enter fullscreen mode Exit fullscreen mode 
// samples/errors/closure_toplevel_out_of_scope.jsonnet
function() $.value
Enter fullscreen mode Exit fullscreen mode 
// samples/errors/closure_default_self_out_of_scope.jsonnet
function(value = self.value) value
Enter fullscreen mode Exit fullscreen mode 
// samples/errors/closure_default_toplevel_out_of_scope.jsonnet
function(value = $.value) value
Enter fullscreen mode Exit fullscreen mode

Cleaning up the non-parser variants

The last diff separates the remaining no-ops into explicit groups with comments explaining why each group doesn't need validation:

-  (* For any other expression types, no special scope validation needed *)
-  | Unit | Null _ | Number _ | String _ | Bool _ | EvaluatedObject _
-  | RuntimeObject _ | ObjectPtr _ | FunctionCall _ | Closure _
-    -> ok ()
+  (* Terminal variants *)
+  | Null _ | Number _ | String _ | Bool _ -> ok ()
+  (* These variants are not produced by parsing source files.
+     Scope validation runs before type checking/interpreting,
+     so they only appear through internal or runtime, and have
+     no source-level scope to check. *)
+  | Unit | EvaluatedObject _ | RuntimeObject _ | ObjectPtr _ -> ok ()
Enter fullscreen mode Exit fullscreen mode

Three groups: terminal literals that have no sub-expressions, internal/runtime-only variants that parsing never produces, and — after the closure unification — nothing left. The last catch-all is gone.

Testing

The cram tests cover both the positive case and every error case. The conditionals.t test confirms that self and $ inside a conditional within an object still works:

diff --git a/test/cram/conditionals.t b/test/cram/conditionals.t
@@ -18,3 +18,7 @@
   2:   [if true then 42]: "value"
      ^^^^^^^^^^^^^^^^^^^^^
   [1]
+
+
+  $ tsonnet ../../samples/conditionals/conditional_object_scope.jsonnet
+  { "one": 1, "three": 3, "two": 2 }
Enter fullscreen mode Exit fullscreen mode

The errors.t file collects all the new scope error tests — conditionals, object field access, function definitions, function calls, and closures:

diff --git a/test/cram/errors.t b/test/cram/errors.t
+  $ tsonnet ../../samples/errors/conditional_self_out_of_scope.jsonnet
+  ERROR: ...:1:27 Can't use self outside of an object
+
+  $ tsonnet ../../samples/errors/conditional_toplevel_out_of_scope.jsonnet
+  ERROR: ...:1:35 No top-level object found
+
+  $ tsonnet ../../samples/errors/object_field_access_index_self_out_of_scope.jsonnet
+  ERROR: ...:2:4 Can't use self outside of an object
+
+  $ tsonnet ../../samples/errors/object_field_access_computed_self_out_of_scope.jsonnet
+  ERROR: ...:2:5 Can't use self outside of an object
+
+  $ tsonnet ../../samples/errors/object_field_access_index_toplevel_out_of_scope.jsonnet
+  ERROR: ...:2:4 No top-level object found
+
+  $ tsonnet ../../samples/errors/function_def_self_out_of_scope.jsonnet
+  ERROR: ...:1:20 Can't use self outside of an object
+
+  $ tsonnet ../../samples/errors/function_def_toplevel_out_of_scope.jsonnet
+  ERROR: ...:1:20 No top-level object found
+
+  $ tsonnet ../../samples/errors/function_def_default_self_out_of_scope.jsonnet
+  ERROR: ...:1:24 Can't use self outside of an object
+
+  $ tsonnet ../../samples/errors/function_def_default_toplevel_out_of_scope.jsonnet
+  ERROR: ...:1:24 No top-level object found
+
+  $ tsonnet ../../samples/errors/function_call_callee_self_out_of_scope.jsonnet
+  ERROR: ...:1:1 Can't use self outside of an object
+
+  $ tsonnet ../../samples/errors/function_call_callee_toplevel_out_of_scope.jsonnet
+  ERROR: ...:1:1 No top-level object found
+
+  $ tsonnet ../../samples/errors/function_call_arg_self_out_of_scope.jsonnet
+  ERROR: ...:2:10 Can't use self outside of an object
+
+  $ tsonnet ../../samples/errors/function_call_arg_toplevel_out_of_scope.jsonnet
+  ERROR: ...:2:10 No top-level object found
+
+  $ tsonnet ../../samples/errors/function_call_named_arg_self_out_of_scope.jsonnet
+  ERROR: ...:2:18 Can't use self outside of an object
+
+  $ tsonnet ../../samples/errors/function_call_named_arg_toplevel_out_of_scope.jsonnet
+  ERROR: ...:2:18 No top-level object found
+
+  $ tsonnet ../../samples/errors/closure_self_out_of_scope.jsonnet
+  ERROR: ...:1:11 Can't use self outside of an object
+
+  $ tsonnet ../../samples/errors/closure_toplevel_out_of_scope.jsonnet
+  ERROR: ...:1:11 No top-level object found
+
+  $ tsonnet ../../samples/errors/closure_default_self_out_of_scope.jsonnet
+  ERROR: ...:1:17 Can't use self outside of an object
+
+  $ tsonnet ../../samples/errors/closure_default_toplevel_out_of_scope.jsonnet
+  ERROR: ...:1:17 No top-level object found
Enter fullscreen mode Exit fullscreen mode

All 19 new tests pass, and every existing cram test still passes unchanged.

Conclusion

The scope checker is clean. Every expr variant that can appear during scope analysis now has explicit handling.

The type checker is next. It has at least as many loose ends as the scope checker did — probably more. Future me gets to sort them out.

The entire diff can be seen here.

Thanks for reading Bit Maybe Wise! Subscribe — and if you don't, the only thing out of scope is your inbox.

Photo by Matthew Ansley on Unsplash

Top comments (0)