Extensions - Lists and for-in loops

Well, here we are. Here’s my interpreter. I want to get philosophical already, but I’ll save it for the musings section. Anyway, I’ve extended Robert’s jlox interpreter by adding support for lists and for-in loops.

What I built: Commits 8409dee, 1f4f09d, 380a16d, 72972f6, 64fdc5f

---

What I understood:

I. Lists

1) The basics:

• We start by adding the tokens/keywords in the TokenType.java file. Because lists use square brackets, we add them (LEFT_BRACKET, RIGHT_BRACKET)
• We then update the scanToken() function in the Scanner to handle the newly added tokens.

case '[': addToken(LEFT_BRACKET); break;
case ']': addToken(RIGHT_BRACKET); break;

• We then create two types of expressions through GenerateAst, one for creating a list (Array) and another for accessing or modifying it (Subscript)

2) Parsing for lists:

• We update the primary() method to catch a LEFT_BRACKET, create a new list, and check for comma-separated elements until the RIGHT_BRACKET is caught.
• We also use chaining to check for sub indices, which if found, are made into a new subscript.

else if (match(LEFT_BRACKET)) {
    Expr index = expression();
    Token bracket = consume(RIGHT_BRACKET, "Expect ']' after index.");
    expr = new Expr.Subscript(expr, bracket, index);
}

3) Visitor methods in the Resolver:

• Because we’ve added new AST nodes (Array, Subscript)

4) Actual List Logic:


• We use a java list to represent a lox list, with every expression/element inside evaluated and stored in the lox list. The SubscriptExpr’s job is to check whether the variable we’re trying to index is actually a list, if the index is a number and within the list size.

5) List functions:

• Finally, we implement global native functions that act as an interface between the user and the Java ArrayList
• push(list, item) calls add() on the java list to append an item to the array.
• pop(list) finds the size() - 1th element and uses java’s remove() method to delete it (the last item) from the list and return it.
• Likewise, len(list) uses java’s size() function to fetch the array’s length.

---

II. For-in loops

1) The basics:

• Add the IN token to the enum in TokenType.java and in to the keywords map in the Scanner.
• checkNext() is a helper function we use to look ahead for an in keyword when we encounter for (var i …. If found, the Parser will build a list-iterator. If in is not found, it is considered a normal for-loop.
• Next, update the forStatement() function in the Parser with a check for in.

2) Desugaring:

• We then desugar for the parser to break down the complex and user-friendly syntax into simpler pieces the interpreter can handle (primitives).
• Side note: Desugaring is also a life-lesson. Breaking down complex problems into smaller and simpler ideas always improves the possibility of solving them. Richard Feynman all the way!
• It creates two hidden variables - _list and _i to store the collection and counter (starting at 0.0) respectively.
• The Parser then creates an AST that resembles a while loop.
• Ex: for (var item in myList) { print item; } effectively becomes:

{
    var _list = myList; 
    var _i = 0;
    while (_i < len(_list)) {
        {
            var item = _list[_i]; // The "item" variable is created here
            print item;           // The original body
        }
    _i = _i + 1;            // The increment
    }
}

• listVar and itemVar in the forStatement() are initialisers, which occur once before the loop begins.
• The forStatement() creates a variable meant for the user (like item) and uses Expr.Subscript to fetch its value from _list at index _i
• The next block increments by adding 1 to _i
• The condition block calls the native len() function to check if the index _i is still less than the list size.
• We use curly braces for scope, to ensure that the hidden variables _list and _i are destroyed as soon as the loop ends. Else, trying to use _list later in the program might confuse the interpreter as it was never previously defined.

---

Musings:
That brings the interpreter to an end. Not “The End,” (because I’m going to keep extending it) but you could say all that I’d intended to do for this project has been accomplished. I took on something that puzzled me, sat and reasoned with it, learnt from it, and finally overcame my hesitation towards it. I now want to get into the “proper” backend of compilation.

The first step is always the hardest one. But we must take it anyway. Mistakes will happen and failing is certain. But rarely will we regret it, if we learn how to learn from our experiences. I’ve taken risks that didn’t yield what I hoped for. It was a little disappointing, but they’ve 100% made me a better person—well informed and sensitive to issues that matter.
This project coincided with such a phase in life. Amidst confusion and uncertainty, it gave me something I could keep coming back to consistently; something I could see tangible progress in; something that gave me the satisfaction of “creating.” We see programming languages all the time, but know so little of all the thought and effort going into them. Every design choice is intended to make programming easier and more intuitive; more accessible too! If I had to sum up in a sentence what I learnt from this—“God is in the details.” I will always be grateful for having done this.

Dear reader, if you’ve followed this project thus far, thank you for your time! Until next time, for the next adventure!