DEV Community

What is a Lexer, Anyway?

Crystal Durham on October 11, 2019

The first task when implementing any language (that is already specified) is to turn the source code into some sort of Syntax Tree that's meaningfu...
Collapse
 
adam_cyclones profile image
Adam Crockett 🌀

🦄 for me this post is perfectly timed. My first project in rust is alot like this but I can see some potential improvements I can make based on your code.

Collapse
 
cad97 profile image
Crystal Durham

I'm glad it could help! Out of curiosity, what kind of improvements have you spotted?

Collapse
 
adam_cyclones profile image
Adam Crockett 🌀 • Edited

Specifically my tokens are defined in structs Token and stored in a Btree Map where the key is the token and the value is Token struct which I had planned to provide callbacks to handle what is expected after the next token. But I'm struggling to implement (new to rust) I wanted to do something like Marpa parser.

I like how your tokens are just a match statement.