Jess lang (5 Part Series)
Jess is still very much under way, I have been learning a few things about parsing and rust along the way.
A crucial part of any compiler is the lexer, for those who don't know, the lexers sole responsibility is to identify individual tokens and push them into an array, this means that if a token is unknown, the lexer can throw an error. In jess's case, we don't throw an error yet, instead Jess lexer places a PANIC token in the stream so that we know that something was not understood. Jess can currently understand 5 tokens, 2 of which are the panic token and the ambiguous token, the other tokens are left curly and right and the start of an import statement.
Tokens can be split into deterministic and nondeterministic identities, a deterministic is known to the lexer such as a semicolon, nondeterministic tokens include values such as strings, numbers, (for jess) CSS values and more, but crucially unknown tokens are also nondeterministic, they are Ambiguous because they require a bit of probing.
Ambiguous tokens get passed to an identity checker (mini parser) and various mostly regex based tests are run to further extrapolate the identity, afterwards this is returned to the token stream, if all identity checks fail as stated, the panic token is inserted.
Phew we made it to the end of the file and our let while loop, amazing! Because we made it to the end and nothing caught fire 🔥 we get a special token, EOF, end of file, horay... Now what, the stream is serialised to JSON and returned back to the Typescript bindings (the front-end of libjess) as JSON, for now this is just wrote to stdout but it could also be used to potentially get cached, source mapped, used by a vscode language server, dumped for debugging and most importantly passed into the parser which will also be written in Rust.
Need to keep up with a fast-moving software industry?