DEV Community: Meghan (she/her)

Making Slate [Part 2.1]

Meghan (she/her) — Sun, 13 Sep 2020 01:08:00 +0000

I realized I left a big part out of my last article so today I'm here to fill you all in.

The Making of Slate [Part 2]

Meghan (she/her) ・ Sep 9 '20 ・ 4 min read

One of the big changes I mentioned over part 1 was the switch from using Wasm to using LLVM and I accidentally completely glossed over it. Super sorry about that one.

So,
LLVM is a pseudo-assembly and program collection that takes this "intermediate representation" as they call it and does the actual compilation into machine code. It's very type strong and looks familiar yet foreign.

And once the program is able to produce an AST from the code, it's the Slate compiler's job to then produce LLVM IR.

https://gist.github.com/nektro/a61777bbcb818d99dddf579b031e9798

This gist contains both the code we're working on and the desired LLVM IR output wer're going to produce. Thankfully, the LLVM project provides a Go library we can use. Actually generating IR takes many things to consider, so today is going to cover the first part.

Picking Declaration Order

In our code we have 4 const declarations that equal to functions with some calls in each one. This is no matter, except for if we go to compile these in the order they're written. If this is the case, then we'll create a function object for main, but then when we try to find the function object for print it isn't there (because it's lower down in the file and we haven't compiled it yet). Bummer. Unfortunately this is our problem because we don't want to copy C's behavior of having to put your functions in the order they're used.

The Current Approach

First we set up two variables. The first is a completed map that associates const names to their IR objects. This will contain the declarations that have been completed and can be referenced later at any time.

At the start this will be empty. Secondly, we also need a todo string array of the uncompiled const declarations. It will contain their names so long as they still need to processed. At the start all declarations will be placed into todo. (For our example this will be [main, print, write, syscall3])

Then, it loops through the todo array and calculates the "dependencies" of each item. Remember earlier when we were going to parse main and needed print? That info is calculated here. After that we then have the following information:

main: [print]
print: [write]
write: [syscall3]
syscall3: []

Note: write calls syscall3 because the convention is the direct call-path for x86_64 Linux, which is what this base example was made for.

syscall3 has no "dependencies" because the only call it makes is inline assembly and will just about be directly inserted into the binary with its arguments.

Now, it will make the IR conversion for all AST objects whose calculated dependencies are all elements of the completed map object and remove them from todo.

For our example, that will be syscall3. syscall3 will then get sent to the LLVM Go library and placed into completed. The compiler will then repeat this loop until todo is empty and finally exit.

Future Improvements

This approach worked because the example was very direct and there are no imports or anything like that. Additionally, it's not very type-safe.

I'm conflicted on whether I want to eventually allow name overloading, but I want it to be able to throw an error if print(string) exists but you try and call print(int), etc.

On top of that, tree shaking for the benefit of smaller binary outputs would be great. This could be achieved by starting at main and diving into each dependency until it parses all that have none. It would then work backwards and only compile the functions/objects referenced through the main pathway. This would have the added benefit of giving me dead code analysis for free.

Wrap up

I hope this explains the internals a lot better than the last article and I'll try my best to leave out jarring wholes next time. Blogging is very much its own skill and kudos to those who do this regularly.

The project can still be found here https://github.com/nektro/slate and I can found here on DEV, Github, and Twitter all @nektro .

Thanks for reading, and I'll see you next time ❤

The Making of Slate [Part 2]

Meghan (she/her) — Wed, 09 Sep 2020 08:21:00 +0000

This is an update article of progress made since this one.

The Making of a Programming Language: Slate [Part 1]

Meghan (she/her) ・ Jul 10 '18

#webassembly #slate #slatedev #computerscience

Wew! And quite a lot has changed indeed. This is still at the phase where it is still only an exercise in learning about compilers and languages, but the architecture has changed completely in many parts.

The process of taking source text and converting it into code is largely the same as I described in the previous post, but the stack I am using and the methods I used to get there are completely different now.

First off, before the idea is that it was going to be largely based on JS/TS and then compile direct to WebAssembly1. However, at the time of me writing the last post, Wasm was significantly less featureful and so wanting to make a full language in that state was not the best move. On top of that, Wasm's ISA2 is based on a stack machine, which makes the output text fairly different from your typical assembly code.

Nowadays, Slate runs on the desktop but I am still keeping the scripting domain in mind. For now, it uses the LLVM3 compiler backend which takes a custom IR4 text format, which is like a pseudo-assembly, and converts that into the actual machine code and binary format5 for your system. This output you can then run as an actual program.

Also, the program is written in Go6, but the end goal is to eventually get it to a point where it could compile7 itself8.

Lexing

This is done the same as before, and the code involved can be found here: https://github.com/nektro/slate/blob/master/pgk/slate/lexer.go

This process is abstracted away and the actual lexing in done in another supporter package. Languages in this step only have to provide the list of keywords, a list of valid symbols, and characters used for strings. The lex package will then take this info and read the source text for a Slate file and return an array of tokens if it is valid.

This step is where simple errors such as illegal characters or words will be risen.

Parsing

This step is architecturally done completely different than before and the source can be found here: https://github.com/nektro/slate/blob/master/pgk/slate/parser.go

Trying to figure this stage out was a huge source of struggle and stopped work on this project for quite a while. But then, one day I was watching a stream by Andrew Kelley, the creator of the wonderful Zig language.

The a-ha I had whilst watching this stream was that you can take advantage of grammar to make the parser and get a bunch of really cool features for free. For instance, in English if you have a q you can practically guarantee that there is going to be a u after it. You can take advantage of this in programming language parsing as well. You know that if you run into the if keyword, then there is going to be a ( after it, because that's how you defined the language. So in the parser you can codify that and then produce really nice errors if you didn't see it.

Similar to rust errors

error: in if statement, expected `(` saw `pub`
  --> src/main.slate:11:5
   |
11 |     if pub {
   |        ^^^

Compiling

If the parser has completed its job and no errors have been produced, then we now have a single AST6 object that represents the entirety of our program that we can work with.

Using another helper package, this AST object gets output to LLVM IR text which we can then pass to LLVM to generate our program.

Where We're At Now

The basic program it can compile is located here: https://github.com/nektro/slate/blob/master/tests/basics/01.slate

The LLVM output of which can be found here: https://gist.github.com/nektro/a61777bbcb818d99dddf579b031e9798

Going forward

Given how experimental this is, I really wanna try out some ideas that I've had about languages for a while and see where it goes. If it works out then great! I made something cool. If not, I still learned a ton.

I have some ideas about syntax changes and upgrades to the languages I'm familiar with. I want to try and take a more functional approach than the popular languages as of late and I need to add recursion and more type checking.

The project can be found here https://github.com/nektro/slate and contains all the source code and documentation up to this point, as well as links to my social medias and ways to support me if you so choose.

Centralization is rampant. And I'm not sure how we should fix it..

Meghan (she/her) — Wed, 13 Nov 2019 10:07:56 +0000

I love self-hosted apps and the idea of making the Web decentralized again.

I make multiple apps that are coming into maturity the more I work on them (and don't worry, will all get posts 👀), but recently started using Gitea for a few projects for a group I volunteer for. And I love it!

Once I'm able to, I'm tempted to run and/or move many of my projects to my own instance, but I'm worried about stigma/FOMO of not being on GitHub. This is very akin to the Twitter/mastodon conversation. Everyone's on GitHub. Developers are on GitHub, companies are on GitHub, non-profit organizations are on GitHub.

I was thinking about this, and realized it's tough because even if Gitea/other git server apps made federation work in a way where you could star/follow members from any instance, GitHub won't. Or at least would be very unlikely to. Integrating with federation is something I imagine investors would think is a bad idea, particularly this late in the game. GitHub's already at the top, so why help people leave the platform?

How do you think we should go about breaking the centralized bubble? Should we?

I saw "Ajax" was trending on Twitter...

Meghan (she/her) — Thu, 29 Aug 2019 19:42:17 +0000

And my first thought was definitely not soccer/football xD

Goal

@goal

Ajax and Chelsea have never played each other in Europe before 🤯

18:13 PM - 29 Aug 2019

Language Proposal: The 'Any' Switch Case

Meghan (she/her) — Thu, 18 Jul 2019 06:20:49 +0000

This feature proposal is language agnostic, as I have not seen this in any language before. If you know of an existing better way to do this, please let me know :)

The tagline for this feature is like if switch had a finally block, but only ran if the successful case found was not default.

Here's what it would look like in code:

switch (number) {
  case 1:
    ...
  case 2:
    ...
  case 3:
    ...
  any:
    ...
  default:
    ...
}

The any block would be ran in this case if and only if case 1, 2, or 3 were also caught.

As far as whether or not the block would be run before or after the given case block is up to debate, and I could see scenarios where either would be useful. So it would be ideal to allow both, but I don't have a clear idea on how you could separate the difference between a before any clause and an after any clause.

Something I've been pondering the past few days. Feedback welcome! 😃

Debugging Travis CI is very hard.

Meghan (she/her) — Tue, 16 Jul 2019 02:47:58 +0000

I had the "simple" goal of setting up automatic builds for my Go project. And it's been a nightmare.

https://github.com/nektro/andesite

https://travis-ci.org/nektro/andesite

The docs are hard to follow, the error messages aren't descriptive, and it has taken up most of my afternoon.

In other news, I switched from Windows to Debian on my main machine and I've been loving every second of it.

Captain Marvel's Official Website is Amazing

Meghan (she/her) — Sun, 10 Feb 2019 09:40:31 +0000

In a blast from the past, Marvel Studios brings us a fantastically designed, fast, and easy to use website to showcase Captain Marvel which comes to theaters March 8, 2019.

Check it out for yourself here: https://www.marvel.com/captainmarvel

Facebook.com is Down

Meghan (she/her) — Mon, 12 Nov 2018 18:04:05 +0000

https://www.facebook.com/

Thoughts on interpreted vs compiled languages?

Meghan (she/her) — Thu, 20 Sep 2018 18:12:57 +0000

Are there inherent advantages/disadvantages to either option? If a new language was to be made which would you prefer it to be?

I really don't like the reactions stickied at the bottom of the screen.

Meghan (she/her) — Thu, 13 Sep 2018 20:59:36 +0000

I feel really bad this is my first post in a while, life's been really busy lately, but I just got the update that puts the article reactions stickied to the bottom of the screen (at least on desktop, haven't checked on mobile) but I really don't like it and it feels like a very unnecessary and unnatural too. After the article but before the comments is exactly when the user should be sending those reactions. After reading the whole article. And when you're reading the comments the reactions take up space because by then you already left your reaction.

I'll delete this tomorrow probably but I wanted to get my thoughts out there and see if maybe I'm thinking about it all wrong.

💖 you DEV

Search/replace file names in Powershell

Meghan (she/her) — Tue, 10 Jul 2018 23:05:11 +0000

Posting a command I recently found that allows you find/replace file names in Powershell on Windows. <search> and <replace> both accept a regex or you can also add the -LiteralPath option to rename-item

get-childitem *.txt | foreach { rename-item $_ $_.Name.Replace("<search>", "<replace>") }

In a world where Denvercoder69 actually does come back and reply to that blog post.

The Making of a Programming Language: Slate [Part 1]

Meghan (she/her) — Tue, 10 Jul 2018 02:45:44 +0000

I made a new programming language! (((Note: right now it can only add number literals to each other and export constants, but it runs!!))

[Part 1] Introduction

Slate!

In the recent time, I've been working on making a brand new programming language and it's called Slate! This is going to be a series, more or less start to "finish" as I document my progress making the compiler, standard library, and maybe even some programs in Slate.

Slate is the (first?) programming language that compiles directly from source code to WebAssembly. Yes, that's why I've been asking about WASM for so long :). The syntax is largely inspired by JavaScript ES2015+ with other influences from Java, Kotlin, and more.

So what can it do right now? A new language? Why? How is it different? Can I use it?

Right now this is about all it does.

/**
 * https://github.com/nektro/slate/blob/master/tests/02.grammar/01.operators/001.slate
 */
//
export const expected = 80;
//
export function main(): i32 {
    return 48 + 32;
}

What do?

You can export integer constant literals and export a function that adds integer literals together. That's about it. But slate.js can fully parse this and export a WASM Module that does the same, albeit very literal for now.

Why make?

I really love JavaScript. This love stems from a broader love of the Web as a platform as a whole and JS is all we get. Until now! WebAssembly is the answer to the old question "is the Web getting any other languages other than JavaScript?". With WASM gains access ALL THE LANGUAGES[1]. So in part of a love for JS, part for a desire to make my own language because, and to try to implement features I've never seen before, I set out to make a language specifically for the Web through WASM.

[1]: provided aforementioned language has the proper toolchain

And through this series I'm going to document more or less the entire journey.

How different?

Slate is strongly typed. So that's one thing that's different. But I also want to add things like operator overloading, object extensions(Like adding onto <Object>.prototype but in a statically typed lang), and more.

Can I use it?

Technically yes! If you'd like to compile the program above and run it in your very own WebAssembly-supporting browser then you can do the following:

import * as slate from "https://rawgit.com/nektro/slate/master/src/slate.js"

const slate_program = `
/**
 *
 */
//
export const expected = 80;
//
export function main(): i32 {
    return 48 + 32;
}
`;

Promise.resolve(slate_program)
.then(x => slate.parse(x))
.then(x => x.instance)
.then(x => x.exports)
.then(x => {
    // `x` == { expected: 80, main: func() { [native code] } }
});

How did you do it?

Like any other language, there are a number of steps that are similar between making a compiler and all of them take place in Slate as well.

Lexical Analysis
Parser
Semantic Analyzer
Code Generation
Linker

1. Lexical Analysis

This step was made easy because the majority of code used for this part was already written when I made an HTML preprocessor and added to my basalt javascript library. The code for Slate's lexer can be found here.

Our lexer will take the text of our program and do some really handy things for us. It has to remove the comments, as well as convert the code into a list of tokens with data that we can then later pass onto the parser.

So with the lexer set up properly, basalt will turn our test program into something like the code below.

[
    Key("export"),
    Key("const"),
    Id("expected"),
    Symbol(=),
    Int(80),
    Symbol(;),
    Key("export"),
    Key("function"),
    Id("main"),
    Symbol("("),
    Symbol(")"),
    Symbol(":"),
    Id("i32"),
    Symbol("{"),
    Key("return"),
    Int(48),
    Symbol("+"),
    Int(32),
    Symbol(";"),
    Symbol("}")
]

2. Parser

This part of the process was also very involved when I made my HTML preprocessor so parsing is also a module in basalt. Basalt helps us build a parser but we still have to add all the magic. Slate's parser is here. Those familiar with the computer science here, we are attempting to create a formal language by means of a pseudo-context-free grammar. ANTLR is another big project in this space of creating a lexer/parser in a format much more similar to Backus–Naur form.

Simply put, we have to come up with a series of patters that can take our token list from before and compress it down into a single express that we can then analyze later to create our program.

After that process, our test program looks more like this:

Note: I'm skipping the code demo part because the output from the parser is very verbose and the next step we're going to condense it down a bit to show the same information but in a lot more useful format

3. Semantic Analyzer

This part is done in Slate by the "converter" which takes the very verbose output from the parser, verifies it, and generates the AST. The source for the Slate converter can be found here.

So now what does our program look like?

File(
    Export(
        Const(
            "expected"
            80
        )
    )
    Export(
        Function(
            "main"
            Parameters(
            )
            "i32"
            Block(
                Return(
                    Add(
                        48
                        32
                    )
                )
            )
        )
    )
)

4. Code Generation

Whew! Almost there! At this point we have a nice AST but now need to compile to WebAssembly so that it's able to run by WebAssembly.instantiateStreaming(), etc. Since I wanted to make this a little easier on myself, I decided to have my compiler generate WASM in the text format as opposed to the binary format and then to use wabt to convert the text to binary WASM. Trust me, I love WebAssembly and what it stands for, but even trying to figure out the text format has been difficult. There is very little docs on the formats currently and most of what I've going off is the WASM platform spec tests and output from various WASM playgrounds.

The code for generating WAST from our AST is actually attached to the objects sent out of the converter, so that code is here. After generation of said WAST we shoudld get the following:

(module
    (memory $0 1)
    (global i32 (i32.const 80) )
    (export "expected" (global 0) )
    (func (export "main") (result i32)
        (i32.add (i32.const 48) (i32.const 32) )
    )
)

Hooray! 🙌

5. Linker

For now we're actually done. Imports are not currently implemented and there is no standard library yet, so this phase will have to come later.

Thanks for reading! If you liked this, let me know what you'd like to see in the future of Slate and stay tuned for more!

Coming up in future installments:

Design ideas and long term goals
More operators
Type inference to add support for floating point numbers and objects
Variables
if, while, for, etc
Strings
More functions
Classes
Metaprogramming

Links to save you a scroll:

Follow Slate on GitHub! https://github.com/nektro/slate
Follow me on Twitter! https://twitter.com/nektro
Follow me on Dev! https://dev.to/nektro