This is an update article of progress made since this one.
Wew! And quite a lot has changed indeed. This is still at the phase where it is still only an exercise in learning about compilers and languages, but the architecture has changed completely in many parts.
The process of taking source text and converting it into code is largely the same as I described in the previous post, but the stack I am using and the methods I used to get there are completely different now.
First off, before the idea is that it was going to be largely based on JS/TS and then compile direct to WebAssembly1. However, at the time of me writing the last post, Wasm was significantly less featureful and so wanting to make a full language in that state was not the best move. On top of that, Wasm's ISA2 is based on a stack machine, which makes the output text fairly different from your typical assembly code.
Nowadays, Slate runs on the desktop but I am still keeping the scripting domain in mind. For now, it uses the LLVM3 compiler backend which takes a custom IR4 text format, which is like a pseudo-assembly, and converts that into the actual machine code and binary format5 for your system. This output you can then run as an actual program.
Also, the program is written in Go6, but the end goal is to eventually get it to a point where it could compile7 itself8.
Lexing
This is done the same as before, and the code involved can be found here: https://github.com/nektro/slate/blob/master/pgk/slate/lexer.go
This process is abstracted away and the actual lexing in done in another supporter package. Languages in this step only have to provide the list of keywords, a list of valid symbols, and characters used for strings. The lex package will then take this info and read the source text for a Slate file and return an array of tokens if it is valid.
This step is where simple errors such as illegal characters or words will be risen.
Parsing
This step is architecturally done completely different than before and the source can be found here: https://github.com/nektro/slate/blob/master/pgk/slate/parser.go
Trying to figure this stage out was a huge source of struggle and stopped work on this project for quite a while. But then, one day I was watching a stream by Andrew Kelley, the creator of the wonderful Zig language.
The a-ha I had whilst watching this stream was that you can take advantage of grammar to make the parser and get a bunch of really cool features for free. For instance, in English if you have a q you can practically guarantee that there is going to be a u after it. You can take advantage of this in programming language parsing as well. You know that if you run into the if keyword, then there is going to be a ( after it, because that's how you defined the language. So in the parser you can codify that and then produce really nice errors if you didn't see it.
Similar to rust errors
error: in if statement, expected `(` saw `pub`
--> src/main.slate:11:5
|
11 | if pub {
| ^^^
Compiling
If the parser has completed its job and no errors have been produced, then we now have a single AST6 object that represents the entirety of our program that we can work with.
Using another helper package, this AST object gets output to LLVM IR text which we can then pass to LLVM to generate our program.
Where We're At Now
The basic program it can compile is located here: https://github.com/nektro/slate/blob/master/tests/basics/01.slate
The LLVM output of which can be found here: https://gist.github.com/nektro/a61777bbcb818d99dddf579b031e9798
Going forward
Given how experimental this is, I really wanna try out some ideas that I've had about languages for a while and see where it goes. If it works out then great! I made something cool. If not, I still learned a ton.
I have some ideas about syntax changes and upgrades to the languages I'm familiar with. I want to try and take a more functional approach than the popular languages as of late and I need to add recursion and more type checking.
The project can be found here https://github.com/nektro/slate and contains all the source code and documentation up to this point, as well as links to my social medias and ways to support me if you so choose.
Oldest comments (1)
Looking good!