loading...

Don’t Interpret Me Wrong: Improvising Tests for an Interpreter

adambrandizzi profile image Adam Brandizzi Originally published at suspensao.blog.br on ・1 min read

I’m in love with the Crafting Interpreters book. In it, Bob Nystrom teach us how to writer an interpreter by implementing a little programming language called Lox. It was a long time since I had so much fun programming! Besides being well-written, the book is funny and teach way more than I would expect. But I have a problem.

The snippets in the bug are written in a way we can copy and paste them. However, the book has challenges at the end of each chapter, these challenges have no source code and sometime they force us to change the interpreter a lot. I do every one of these exercises and as a result my interpreter diverges too much from the source in the book. Consequently, I often break some part of my interpreter.

How to solve that?

Unity tests would be brittle since the code structure changes frequently. End-to-end tests seem more practical in this case. So, for each new feature of the language, I wrote a little program. For example, my interpreter should create closures, and to ensure that I copied the Lox program below to the file counter.lox:

fun makeCounter() {
  var i = 0;
  fun count() {
    i = i + 1;
    print i;
  }

  return count;
}

var counter = makeCounter();
counter(); // "1".
counter(); // "2".

This program result should be the numbers 1 and 2 printed in different lines. So I put these values in a file called counter.lox.out. The program cannot fail either, so I created an empty file called counter.lox.err. (In some cases, it is necessary to ensure the Lox program will fail. In these cases, the file .lox.err should have content.)

Well, I wrote programs and output files for various examples; now I need to compare the programs’ results to the expected outputs. I decided to use the tool that helps me the most in urgent times: shell script. I did a Bash script with a for iterating over all examples:

for l in examples/*.lox
do

done

For each example, I executed the Lox program, redirecting the outputs to temporary files:

for l in examples/*.lox
do
  out=$(mktemp)
  err=$(mktemp)
  java -classpath target/classes/ br.com.brandizzi.adam.myjlox.Lox $l > $out 2> $err
done

Now, we compare the real output with the expected output through diff. When it compares two files, diff returns 0 if there is no difference, 1 if there exists a difference or 2 in case of error. Since in Bash the conditional if considers 0 as true, we just check the negation of diff‘s exit code.

If the program prints something in standard output that is different from what is in its .lox.out file, we have a failure:

for l in examples/*.lox
do
  out=$(mktemp)
  err=$(mktemp)
  java -classpath target/classes/ br.com.brandizzi.adam.myjlox.Lox $l > $out 2> $err

  if ! diff $l.out $out
  then
    FAIL=1
  fi
done

We also check the standard error and the .lox.err file:

for l in examples/*.lox
do
  out=$(mktemp)
  err=$(mktemp)
  java -classpath target/classes/ br.com.brandizzi.adam.myjlox.Lox $l > $out 2> $err

  if ! diff $l.out $out
  then
    FAIL=1
  fi

  if ! diff $l.err $err
  then
    FAIL=1
  fi
done

Finally, I check if there was some failure and report the result:

for l in examples/*.lox
do
  out=$(mktemp)
  err=$(mktemp)
  java -classpath target/classes/ br.com.brandizzi.adam.myjlox.Lox $l > $out 2> $err

  if ! diff $l.out $out
  then
    FAIL=1
  fi

  if ! diff $l.err $err
  then
    FAIL=1
  fi

  if ["$FAIL" = "1"]
  then
    echo "FAIL" $l
  else
    echo "PASS" $l
  fi
done

Not all of my Lox programs can be checked, though. For example, there is a program which times loop executions, it is impossible to anticipate the value it will print. Because of that, I added the possibility to jump some programs: we need just to create a file with the .lox.skip extension:

for l in examples/*.lox
do
  if [-e $l.skip]
  then
    echo SKIP $l
    continue
  fi

  out=$(mktemp)
  err=$(mktemp)
  java -classpath target/classes/ br.com.brandizzi.adam.myjlox.Lox $l > $out 2> $err

  if ! diff $l.out $out
  then
    FAIL=1
  fi

  if ! diff $l.err $err
  then
    FAIL=1
  fi

  if ["$FAIL" = "1"]
  then
    echo "FAIL" $l
  else
    echo "PASS" $l
  fi
done

If, however, I have a Lox example and it does not have expected output files (nor the .lox.skip file) then I have a problem and the entire script fails:

for l in examples/*.lox
do
  if [-e $l.skip]
  then
    echo SKIP $l
    continue
  elif [! -e $l.out ] || [! -e $l.err ]
  then
    echo missing $l.out or $l.err
    exit 1
  fi

  out=$(mktemp)
  err=$(mktemp)
  java -classpath target/classes/ br.com.brandizzi.adam.myjlox.Lox $l > $out 2> $err

  if ! diff $l.out $out
  then
    FAIL=1
  fi

  if ! diff $l.err $err
  then
    FAIL=1
  fi

  if ["$FAIL" = "1"]
  then
    echo "FAIL" $l
  else
    echo "PASS" $l
  fi
done

With that, my test script is done. Let us see how it behaves:

$ ./lcheck.sh
PASS examples/attr.lox
PASS examples/bacon.lox
PASS examples/badfun.lox
PASS examples/badret.lox
PASS examples/bagel.lox
PASS examples/bostoncream.lox
PASS examples/cake.lox
PASS examples/checkuse.lox
PASS examples/circle2.lox
PASS examples/circle.lox
1d0
< 3
1c1
<
---
> [line 1] Error at ',': Expect ')' after expression.
FAIL examples/comma.lox
PASS examples/counter.lox
PASS examples/devonshinecream.lox
PASS examples/eclair.lox
PASS examples/fibonacci2.lox
PASS examples/fibonacci.lox
PASS examples/func.lox
PASS examples/funexprstmt.lox
PASS examples/hello2.lox
PASS examples/hello3.lox
PASS examples/hello.lox
PASS examples/math.lox
PASS examples/notaclass.lox
PASS examples/noteveninaclass.lox
PASS examples/point.lox
PASS examples/retthis.lox
PASS examples/scope1.lox
PASS examples/scope.lox
PASS examples/supersuper.lox
PASS examples/thisout.lox
PASS examples/thrice.lox
SKIP examples/timeit.lox
PASS examples/twovars.lox
PASS examples/usethis.lox
PASS examples/varparam.lox

Oops, apparently I removed the support for the comma operator by accident. Good thing I wrote this script, right?

I hope this post was minimally interesting! Now, I am going to repair my comma operator and keep reading this wonderful book.

(This post is a translation of Não me Interprete Mal: Improvisando Testes para um Interpretador.)

Discussion

pic
Editor guide