DEV Community: Steven L

I've Been Laid Off. What Now?

Steven L — Tue, 27 Dec 2022 02:26:19 +0000

I have had this post in the pocket for a few months now, but I think I'm ready to share.

Tl;dr: I got laid off. Now I have new plans in mind.

The Context

Few months back I was looking for a new role. I got in talks with a recruiter, went into the stage 1 and stage 2 interviews and did decently. So much that they hired me. Hooray! Stage 1 was basic coding, while stage 2 was chatting about the company and getting a feel for the environment.

The company was full remote, and that was something I would totally go for as someone who was dreading going into work every day, putting miles on my car, and not being paid enough to do so. COVID-19 has more than once put a scare on my life by going into the workplace, and going remote was a real ray of light for me.

The first month was weird, and strange, and such a distant experience from all my previous work environments. I was training at an okay rate, but the code was quite frankly, a disaster across the place. No object-inheriting patterns, no code re-usage even if projects were literally doing similar things, and just in general it was across dozens of repositories and there were a ton of tickets in queue.

Going into month two I feel a little confident and I start beginning to plan my life around this job, because it seemed like something I could really grow into. I started making moving plans to move homes and have something of my own (finally!), other new financial ventures, and starting to feel like my life was getting on track at a decent rate.

The Tech Downturn

On the world outside, things were heating up. Elon Musk bought Twitter and severed the workforce from ~7,500 workers to a mere ~2,000 over the span of the next few months. Facebook/Meta laid off employees, Shopify laid off employees, everyone was getting destroyed. I think I read the total count of all laid off tech workers ended up being somewhere in the low 100k figure. Yikes.

It seemed that over the last two pandemic years, tech companies realized their profits/earnings weren't growing with hiring, which is a correct thing to assume given the numbers. It's true that by hiring more workers you don't always improve your cash income, sometimes work is more slow-going or sometimes it's not dependent on how many employees you have. Tech is not a field that always scales like that.

So, I imagine many companies caught onto that trend, and realized it almost within the same quarter. U.S. inflation was at it's absolute highest, and the Federal Reserve raised the interest rates to combat it. In turn, this affects how central banks give out and collect on loans and debt, and some places get shattered by sudden increases in this.

Increasing the interest rate decreases how many people are looking to spend, as the cost to borrow money from banks will go up. This in turn means less people with money will come in for borrowing. Some financial assets like stocks may also lose value, and this can cause a ripple on stocks as people pull out of markets, because the expected value of some assets will decrease.

In turn less people will be borrowing, less people will be buying, and less money will be flowing through the markets. Financial companies are going to panic over this as they try to re-align their books, and tech companies are going to need to figure out how to stabilize to pay off any potential debt they may have, like government bonds or bank investment loans.

So, tech workers got eviscerated. And I was part of that cut. The job I was barely two months into got cut, just like that.

The Emotions

It was a pretty devastating hit to hear, and a lot of emotions went through my head. "Was any of this my fault, and could it have been prevented?" I kept asking myself as I took the day off after hearing the news.

I got the news the day after my birthday, and I felt like crap. This was one of the biggest opportunities I had, and it felt a bit like a rug pulled from under me. My family was pretty disappointed to hear the news that I was suddenly out of a job that I just so happily took on. Also I suppose no one in my company cared to note it was my birthday recently.

The next few days I just did what I could: I brushed off my 2-month old resume, put a 2-month old job on it, and got in contact with as many recruiters and job contacts as I could. I even had to contact old companies I was in talks with to tell them I was now available, after having denied them (whoops). Not the greatest look, but it sucks.

The heartbreak came as I heard more about the layoffs. Everyone was getting cut, but now that just increased the temperature in the room for me. Suddenly there were a couple thousand workers also looking for work. Yikes. That doesn't look good for me. I'm no Facebook-level engineer for sure, so I'm probably not even going to stick out at all in the pond full of piranhas.

It's nearing the end of the year and I have go through my job still hoping to get my severance. At this point I'm completely burnt, I hate the code I'm working on, I don't want to go to daily stand-ups or weekly meetings to listen to people talk about their jobs and how "safe" they feel. I just got cut, and these guys are talking about football like nothing happened. No one on my team even spoke to me about for weeks.

So there I am; a remote worker, with no one to confide in other than my small group of friends/family. They don't really know how to comfort me so I try not to rely on anyone. Losing a job sucks, but I have to get over it eventually.

The Comeback

I don't have any job prospects at this point of writing. Every job position I apply for has well over 100+ applicants to it. I hear from recruiters, I do their stupid quizzes that they ask for, and I never hear back or get feedback. And at that point I am just too lazy to try to follow-up and see why no one responded to me. There's too many faces and too many names, and I already know why no one's responding to my applications: they just don't want me.

I realized that this could very well happen again. I could get laid off just as easily at the next place, and the place after that, in an infinite sequence of work. N job will hire me, then fire me, N+1 job will hire me, then fire me. So on and so forth. It's an infinite sequence of pain.

The only way I could make myself more attractive is to do things that no one else does. But if I want to future-proof myself, I need to build something that I can have each time something like this were to happen; I need to become a valuable worker, someone that someone wants to hire. But I also want to have something that is truly mine, that I can share and talk about with people instead of saying "I write code for X company".

I am the worker, I produce the value, and I do care about computers and computer science more than most. I love the academics behind programming and the theory of programming languages. Surely I can think of something to put my brain towards.

It's that reason why I started my own game studio, and will continue to work on it until I find a new job, and will probably also put time on it while employed at said new job.

The Future

I love video games as an entertainment source, and I think it does a world of good. I am not saying I am not trying to enter the big business of game development, but I like the casual, laid-back approach of the indie world. I don't think I'd ever be able to make something that's as great as things like Braid or Super Meat Boy, but those are things I do look up to and have a lot of respect for.

No, my approach is to hopefully build accessible games. Much of the gaming world is locked behind smartphones, expensive consoles, high-end gaming PCs, and so on. But it is possible to have games that are fun that don't require those things, and making things accessible is an idea I believe in. I don't want to lock people out of having a good time simply because they are not in the right economic situation.

Zig is the current language I have been using. It's a great systems-level language and is for the most part pretty relaxing to write in. It's not quite as complex as Rust, but it's a much different feeling. There's only so many people who write Zig, so sometimes getting better at it involves talking to others who also use it too. Sometimes you can even talk to the creator, Andrew Kelley, and he has a blog where he writes about Zig a lot, and goes to conferences to talk about software development as a whole spectrum.

Zig's potential is that it is pretty easy to write cross-platform code with it. Normally writing cross-platform code involves using a cross-compiler setup where you write code that targets another platform that's different than your host system. It requires using compilers that have ABIs of the target system, which is not something many have readily available. Sometimes you're often left having to do this yourself with Docker, or Nix, and other times you're going to have to use Ninja, Meson, or CMake to build out your applications create a lot of sketchy and weird build files.

So, that sounds like it sucks, but Zig supports cross-compilation out of the box pretty easily. It does this by having shims for different targets and providing information to LLVM's IR framework for code generation. I have already done cross-compilation in Zig with libsdl2, and it didn't cost me all of my life to do so as a solo-dev.

Zig also supports WebAssembly, which I have been learning a lot about by using Zig to create WebAssembly demos as part of my recent series on here. I can't say if my writing is beneficial, but I noticed a lack of writing in this category of Zig and WASM, so I figured I'd provide some info for others who were also interested in either WebAssembly or Zig (or both!).

I have also, up to this point in my life, never created what's called a Progressive Web App. I have been interested in installable web pages that act as apps, because the "app store" model is broken by design on both Android and iOS. They are (in my eyes) awful services that both fail the needs of the developers and the consumers. Too many apps that are scams get added to the store, too many legitimate, helpful and well-designed apps get taken down from the app stores for dumb rules that Google or Apple can randomly uphold and apply when they feel like.

The web is beautiful and open, and I think we should embrace that. I am not looking to make money off my really crappy games, I am just hoping some of my games can bring joy. I am not looking to make a quick buck, I am looking to build the best possible products I can while supporting them for a really long time. It sucks when games stop working because developer companies cease to exist. I think we should support people, not soulless gaming companies seeking to extract maximal value from consumers.

The Summary

Losing a job is a pain in the butt, and in a bad economic market where nobody is hiring, it can look devastating. But the key component is to be able to do something you care about, and spend as much time doing that and being able to present to others that "hey, I do this, and this is something I'm really passionate about".

Whether it's building a web development portfolio because you really enjoy designing coffee shop websites, or you really like data science and grinding up statistics and inference, or if you like designing toy games to share with others online. Do something you care about, do it a lot, and make it your passion.

And at times if you feel like you don't believe in yourself, I will stand up and say "I think you can do it". Everyone has the potential to be awesome, and I think you can be as great as you want to be.

For any companies/recruiters reading: I am open to work and more than happy to chat if you reach out. For anyone in a similar situation, if you need someone to reach out to, I am here as well! We should all support each other in times of need and it helps to have friends.

'Til next time where I write more about Zig and WebAssembly, or maybe something functional. Who knows.

WebAssembly With Zig, Pt. II

Steven L — Wed, 30 Nov 2022 18:54:54 +0000

Hello everyone, and welcome back to a longer-form series I am (attempting) to write. It has been a few months, but I am here again putting together the next iteration of this series.

This will be a very long, but hopefully reasonably fun dive into more Zig and WebAssembly and seeing what we can do this time around. Grab a drink or snack and let's start digging in.

What We Know So Far

From the last article, I covered the basics of what WebAssembly (or WASM) can do for you. In short, it's a specialized engine inside a browser that we can access by feeding a stream of bytes, and the browser interprets packets of data into instructions to create virtual machine you can interact with. The advantages are pretty clear; WASM is fast, performant, and easy to target for compilers. If you can use WASM, you should try it.

Implementing said things, however, can be tricky, and the community is kind of spread all over the place. Rust is popular, but involves a lot of crates and boilerplate code being strapped on. C/C++ is an easy way of targeting WASM using Emscripten, and I'm sure it works well, but I feel it fits in the same place as Rust where a lot needs to be strapped on to bring you up to production speed.

The Zig approach is what I am mostly interested in, as the Zig compiler includes shims for targeting WebAssembly. Since Zig can be used as a compiler for C/C++, this means you could skip the Emscripten part entirely and incorporate a build.zig file into your C/C++ project, allowing you to mix C/C++ libraries into a Zig project targeting a WebAssembly backend fairly easily.

I have a very large interest in video game entertainment as well as game development. Games make people happy, games are fun to share with friends, and overall, games are a great way of pushing the envelope for computer science in many categories dealing with visuals, audio, computational geometry, artificial intelligence, and so much more. My goal as such is to get somewhere close to a game that we could perhaps "play" it.

Where We Last Were

We've seen how it's possible to load a WASM bytefile into a JavaScript-powered page. However, in the time since my last article, I have been working on a repository called zigtoys where I deploy my tests and upload the WASM files. You can download this and try out some examples for yourself, but for the most part I have two demos online you can see for yourself right now.

life.zig, a small Conway's Game of Life simulator
mandelbrot.zig, a Mandelbrot fractal generator

Mandelbrot is a little simpler than Conway's, because the Game of Life simulator does a lot of grid-based checks and makes use of a statically-defined "world". This idea of a game world is what we're going to need to focus on for a bit to start working on a game.

In both of these apps' folders we have a JS file to begin the WASM loading process and bind function names to Zig functions. This way we can use Zig from inside a JavaScript program, offloading difficult and long computations to the WASM engine instead of relying on the JavaScript engine.

However, we haven't gone backwards much, have we? I haven't really touched upon that much, because well... For better or worse, there are some parts I'm still not sure about on that front. JavaScript functions typically use things like i32 or dynamically-allocated strings, so it's hard for me to gauge how that can be used within Zig. It's slightly easier for me to judge how things interact when I focus on Zig->JavaScript instead of JavaScript->Zig.

In life.zig, a game world is explicitly declared, and that is kept throughout the lifetime of the web page being opened. This makes it very simple; the game world is static, but mutable, so this is what we could use for a jumping-off point to start creating small games in Zig for WASM purposes.

The goal would be to create a game that can update it's internal state with a simple function, and expose the data to JavaScript so it can be presented easily. We might have to mix and match some JavaScript here and there, but for a trade-off of performance and using Zig on the web, I'll take it for now.

Building a World

Building a world is one thing, it's no different than Conway's Game of Life. Declare an array and fill it with numbers representing the different entities you wish it to have. In Conway's case, it was simply some cells describing a square, and whether that square was dead or alive.

const State = struct{
    cells: [10*10]u8,
};

var WORLD = State{
    .cells = undefined, // auto-zero the array
};

Here I am creating a new struct called State which we will want to contain information about our world inside of. Then afterwards, I declare a new variable called WORLD, which we will update and occasionally ask for information about the world. I use uppercase here to denote it's importance, but you can use what you wish.

The problem is it doesn't seem like storing u8's will be a great idea, and we should maybe use an enum type to describe the types of actors in our little world. Let's shift that over to an enum now and create a micro-world with which we can move around in.

const Entity = enum{
    Empty,
    Player,
    Wall,
};

const State = struct{
    cells: [W_SIZE]Entity,
};

var WORLD = State{
    .cells = undefined, // auto-zero the array
};

The new Entity enumeration describes what kind of actors our world will have. This is a pretty basic approach, but for a tiny sample game, this is fine for me. Type-aliasing this info away from a raw number is better and facilitates better naming conventions in code, ie. currEntity == Entity.Player versus currEntity == 5. Less confusion the better.

We're not quite out of the woods yet, as it might be a good idea to define constants at the top of our file here to indicate a maximum world size and obtain information for bounds checks.

const WIDTH: u32 = 10;
const HEIGHT: u32 = 10;
const W_SIZE: u32 = WIDTH * HEIGHT;

const Entity = enum{
    Empty,
    Player,
    Wall,
};

const State = struct{
    cells: [W_SIZE]Entity,
};

var WORLD = State{
    .cells = undefined,
};

Like in my Game of Life sample, it's a good idea to use some constants to customize your world so you can adjust upwards or downwards as you see fit, or design some crazy stuff. Maybe you can parameterize it later so your world is resizable on the fly instead of statically like this, but that might be a different feature for later.

We haven't defined any functions still, so compiling this would lead to the most bare-bones WASM file imaginable. We're going to start working on a function called update(), which updates the state of the world by mutating the data it contains. We still need some helper functions to help us communicate with the cells of the world itself, since it is a linear 1-dimensional array. It might be convenient for some calculations from a 2D space using some helper functions to do the math.

fn calc_pos(x: u8, y: u8) usize {
    return x + (y * HEIGHT);
}

export fn get_pos(x: u8, y: u8) Entity {
    var index = calc_pos(x, y);
    if (index < W_SIZE)
        return WORLD.cells[index];
    return .Empty;
}

export fn set_pos(x: u8, y: u8, v: Entity) bool {
    var index = calc_pos(x, y);
    if (index < W_SIZE) {
        WORLD.cells[index] = v;
        return true;
    }
    return false;
}

The calc_pos(x,y) function will help us to calculate an index from two Cartesian co-ordinates, while get_pos(x,y) and set_pos(x,y,v) are the general interface for changing the game. All the updates should be done through those two functions when it comes to changing the cells.

It might help if we have a thing in place to set the world to some initial state, so maybe let's define an init() function which will give us some default world we can toy around in.

export fn init() void {
    _ = set_pos(4, 4, Cell.Wall);
    _ = set_pos(4, 3, Cell.Wall);
    _ = set_pos(4, 5, Cell.Wall);
    _ = set_pos(3, 4, Cell.Wall);
    _ = set_pos(5, 4, Cell.Wall);
    return;
}

Granted, since I want set_pos to return a Boolean of whether or not it could place an entity on the grid, we don't really need it in our tiny example here, so I use the _ = <expr> syntax to discard the result and ignore whatever gets thrown back. Those fixed numbers work for a 10x10 grid, but since I'm not writing them with a variable in mind, it could fail if your grid was too small, or look strange on a board with odd dimensions.

Displaying Our World

Next let's focus back on the JavaScript aspect, and go back and create a file that loads our WASM and binds our functions here and will actually start rendering our game world. We'll use a basic grid on a <canvas> element to portray our little game here.

First, let's look at how we originally loaded our WASM. This is done on a local HTTP server spun up with python -m http.server, which is frankly the fastest way I can think of to get something going for this. If you don't have Python, maybe try nodejs or a Docker HTTP server runtime of your choice.

request = new XMLHttpRequest();
request.open('GET', 'game.wasm');
request.responseType = 'arraybuffer';
request.send();

request.onload = function() {
    var bytes = request.response;
    WebAssembly.instantiate(bytes, {
        env: {}
    }).then(result => {
        // store references to Zig functions
        get_pos = result.instance.export.get_pos;
        // repeat etc ...
    });
};

Curiously enough, I never really looked at what is really inside the result.instance.exports object we seem to get. Let's try now.

Build your Zig first into a WASM (I keep forgetting what this command is so I'm really doing this for myself).

$ zig build-lib game.zig -target wasm32-freestanding -dynamic -O ReleaseFast

This outputs a game.wasm file (assuming you also named your file game.zig). Then we load it in our browser by pointing our browser to localhost:8000 (or wherever your local HTTP server is bound), and browsing to our web page, which we haven't made yet.

<!doctype html>
<html>
  <head>
    <title>Game Demo</title>
  </head>
  <body>
    <h3>Game Demo</h3>
    <canvas id="game_canvas" width="100" height="100">
      Browser does not support HTML5 canvas element
    </canvas>
  </body>
  <script src="game.js"></script>
</html>

After that, let's load into our web page with the local HTTP server running, and wedge in a tiny console.log statement on the results object to see what we're working with. The output looks like this:

Object { memory: WebAssembly.Memory, get_pos: 0(), set_pos: 1(), init: 2(), update: 3() }
get_pos: function 0()
init: function 2()
memory: WebAssembly.Memory { buffer: ArrayBuffer }
set_pos: function 1()
update: function 3()

Interestingly enough, it seems like my way of manually binding variables to these functions may have been redundant as we have that mapping already created for us. Now we can reduce some cognitive load a bit by assigning this to a namespace we can recognize. Before the WASM loading, let's create an object with some properties filled out.

var Game = {
    'init': null,
    'get_pos': null,
    'set_pos': null,
    'update': null,
    'loaded': false;
    'running': false,
};

If the WASM fails to load for some reason, we can use this as a means of declaring "empty" functions that will simply error out instead of leaving us with dead, undefined function errors. Maybe we can change this later to actually be functions that display better errors, but for right now I think this is okay.

Then all we do (during the WASM load phase) is bind this variable to the WASM export.

Game = result.instance.exports;

This might pose risks as the memory binding is tied in as well. However, it seems like in the WASM loading process, an ArrayBuffer given to the memory is not shared and as such is completely locked down. This might not be full-proof, and I'll come back if there's something leaky about this.

Now to test it, we can call get_pos() and set_pos() on it.

>>> Game.set_pos(4, 4, 2)
1 // indicator of it worked
>>> Game.get_pos(4, 4)
2 // the value we set!

This shows us that it's actually working and yields a Boolean indicative of whether it worked or not. Awesome stuff so far.

We need a way of displaying our game world, so let's try to do that quickly with a little canvas JS.

var canvas = window.document.getElementById("game_canvas");
var ctx = canvas.getContext('2d');

var main = function() {
    console.log("Main function started");

    var loop = function() {
    ctx.fillStyle = "white"; // clear the canvas
    ctx.fillRect(0, 0, 100, 100);
    for(var x = 0; x < 10; x++) {
        for(var y = 0; y < 10; y++) {
        var cell = Game.get_pos(x, y);
        if (cell == 1)
            ctx.fillStyle = "red";
        else if (cell == 2)
            ctx.fillStyle = "grey";
        else
            ctx.fillStyle = "white";
        ctx.fillRect(x*10, y*10, (x*10)+10, (y*10)+10);
        }
    }

        // loop to next frame
    if (Game.running)
        window.requestAnimationFrame(loop);
    };
    loop();
};

This will render a small game world where walls are grey, empty space is white, and a mysterious player is red. However, our player is nowhere to be found, because we never defined how he would move just yet. That will come later. If everything is done correctly up to here, you will see a canvas with this showing.

I am going to make a quick change to our WASM loader to change it from .instantiate to .instantiateStreaming, as mentioned by @rofrol in my last post (thanks!!). This makes use of an asynchronous promise via the fetch() function assuming your browser supports it, otherwise you will have to go back to the old XMLHttpRequest way. The WASM code will look like this now.

window.document.body.onload = function() {
    var env = { env: {} };
    WebAssembly.instantiateStreaming(fetch("game.wasm"), env).then(result => {
        console.log("Loaded the WASM!");
        Game = result.instance.exports;
        Game.loaded = true;
        Game.init(); // initialize world
        main(); // begin
    });
};

With that, it's time we move onto the next section of how we actually interact with this as a player.

Playing The Game

There isn't really a clear-cut objective here. We have a supposed Player that walks around this small grid of empty walls, with no real gameplay. Kind of sucks, right? Not much of a game.

We can gamify it with a few changes in mind.

make it so we can walk the player entity around
have a goal in mind, perhaps a destination
add a score, make it fun
randomization! or levels.

The first is to add controls, so we can move a player around. The second is adding a new entity called Goal where the player must step on it to win. The third is adding a tracker to the game state. The fourth is... Well, that's something else entirely I would say, and maybe a little complex a topic for right now.

The first goal in mind is to add the concept of a Player into the game. Naturally by having a cell entity be the Player is the first thing in our mind, but we need a little help tracking his position. Each time the player moves, his index in the array gets updated, so anytime you want to move the player, you have to find him again.

Naturally, you can add a variable into the world state, or maybe two, just to make calculations less annoying. The horizontal and vertical position can be stored, and this will help us to calculate his movements.

const State = struct{
    playerx: u8,
    playery: u8,
    cells: [W_SIZE]Entity,
};

We store playerx and playery as u8 types for now, because our grid is quite literally a 10x10 grid. We cannot go smaller with Zig custom bit-types like u4, because the C translation needs C conformant types, and the lowest is one byte, or eight bits (u8 in Zig).

Now, any time update() is called (which it isn't being called ever), we can update our world by adjusting the playerx and playery attributes reflecting our changes. So now we can map a value to reflect moving.

export fn update(dir: u8) void {
    if (dir == 0)
        WORLD.playery -= 1;
    if (dir == 1)
        WORLD.playery += 1;
    if (dir == 2)
        WORLD.playerx -= 1;
    if (dir == 3)
        WORLD.playerx += 1;
    return;
}

Okay that seems fine... Except wait, this is probably better reserved for an enum type so Zig can check it much easier (and safer). Let's move the direction into it's own enum.

const Direction = enum(u8) {
    Up,
    Down,
    Left,
    Right,
};

export fn update(dir: Direction) void {
    switch (dir) {
        .Up => { WORLD.playery -= 1; },
        .Down => { WORLD.playery += 1; },
        .Left => { WORLD.playerx -= 1; },
        .Right => { WORLD.playerx += 1; },
        else => {},
    }
    return;
}

This is also prone to fail since we're not doing bounds check, and this can lead to integer underflow/overflow or just plain going off the map entirely. We need to provide checks to assure we're still going to be on the map after an update.

export fn update(dir: Direction) void {
    if ((dir == .Up) and (WORLD.playery > 0))
        WORLD.playery -= 1;
    if ((dir == .Down) and (WORLD.playery < HEIGHT))
        WORLD.playery += 1;
    if ((dir == .Left) and (WORLD.playerx > 0))
        WORLD.playerx -= 1;
    if ((dir == .Right) and (WORLD.playerx < WIDTH))
        WORLD.playerx += 1;
    return;
}

We lose our switch case, but that happens when complex logic starts being required. This moves our player around, but doesn't actually change any of the cells, so nothing really happens other than two numbers being pushed around.

This also doesn't really do anything for us quite yet either, as it isn't contextually aware of our world, nor does it mutate the cells to reflect our new state. We should store new co-ordinates that represent our target cell that we want to move to, and apply logic based on what is in that cell. If the goal cell is invalid, we don't do anything and exit early.

/// Update our world by attempting to move the player somewhere
export fn update(dir: Direction) void {
    var goalx = WORLD.playerx;
    var goaly = WORLD.playery;
    if ((dir == .Up) and (goaly > 0))
        goaly -= 1;
    if ((dir == .Down) and (goaly < HEIGHT - 1))
        goaly += 1;
    if ((dir == .Left) and (goalx > 0))
        goalx -= 1;
    if ((dir == .Right) and (goalx < WIDTH - 1))
        goalx += 1;

    var goalpos = calc_pos(goalx, goaly);
    if (goalpos == calc_pos(WORLD.playerx, WORLD.playery))
        return;

    var dest_ent = WORLD.cells[goalpos];
    switch (dest_ent) {
        .Empty => {
            _ = set_pos(WORLD.playerx, WORLD.playery, .Empty);
            _ = set_pos(goalx, goaly, .Player);
        },
        else => {},
    }
    return;
}

This is a lot of ground to cover, but let's go through it slowly.

First, we need to calculate a "goal" position to move the player from. This is based on it's original co-ordinates in the world state, so we grab that. Then we apply checks based on the given Direction value we receive, and apply a range check if we're going to move outside of our world or not. Then we modify the goal co-ordinates if it's a valid move.

Next, we check if the goal position we want to move to is valid, and not the one we're currently standing on. If it's the one we're standing on, exit early, because there's no additional work to do.

Last, we do a switch over the Entity value that's in our goal index position. If it's an empty space, we move our player, and then we set an empty space where we previously were. To make it easier, I made an update to set_pos() to include moving the player if we're setting a player entity, this way there can only ever be one player in the world.

export fn set_pos(x: u8, y: u8, v: Entity) bool {
    var index = calc_pos(x, y);
    if (index < W_SIZE) {
        switch (v) {
            .Empty => { WORLD.cells[index] = .Empty; },
            .Player => {
                WORLD.playerx = x;
                WORLD.playery = y;
                WORLD.cells[index] = .Player;
            },
            .Wall => { WORLD.cells[index] = .Wall; },
        }
        return true;
    }
    return false;
}

We need to update the init() function to include an original player position.

export fn init() void {
    _ = set_pos(4, 4, Entity.Wall);
    _ = set_pos(4, 3, Entity.Wall);
    _ = set_pos(4, 5, Entity.Wall);
    _ = set_pos(3, 4, Entity.Wall);
    _ = set_pos(5, 4, Entity.Wall);

    // set the player
    _ = set_pos(0, 0, Entity.Player);
}

This works, and we can try it out by manually punching commands into the console.

>>> Game.update(1) // moves player down
>>> Game.update(0) // moves player up
>>> Game.update(2) // moves player left
>>> Game.update(3) // moves player right

Here's what it should look like now since we already drew the player in the JS earlier.

If you move him around, you'll see we don't collide and move into grey wall blocks. Why? Because we checked the target cell, and we never wrote logic for when a wall is encountered. Therefore, if a wall is found, we just don't do anything.

Binding the Controls

It would be very convenient if we could not use the JS console to move our player, and instead maybe use keyboard keys like the arrow keys, or WASD instead. This is something we'll have to patch on the JS side, so let's go back to that JS code and find a way.

Adding a key event to a web page involves adding a reactionary hook function to the DOM model. When keys are pressed down (or any other kind of user I/O event occurs), JavaScript can be executed if bound through an event "listener" function. Let's attempt to bind a listen to the canvas element itself.

window.document.body.addEventListener('keydown', function(evt){
    console.log(evt);
});

This will first tell us what kind of events we're receiving (if any) for the keydown type of events. Using console.log, we can see what events we're getting exactly, so let's grab some sample output.

keydown { target: body, key: "a", charCode: 0, keyCode: 65 }
keydown { target: body, key: "d", charCode: 0, keyCode: 68 }
keydown { target: body, key: "f", charCode: 0, keyCode: 70 }
keydown { target: body, key: "ArrowUp", charCode: 0, keyCode: 38 }

It's pretty basic; we get an object file that shows the key name and the key raw code value. We can map any group of these using either string names or integer values, but for now I will go with string values since it's easier to read at a glance, instead of questioning what each keycode is.

window.document.body.addEventListener('keydown', function(evt){
    if (!Game.loaded)
        return;
    if (evt.key == "w" || evt.key == "ArrowUp")
        Game.update(0);
    if (evt.key == "s" || evt.key == "ArrowDown")
        Game.update(1);
    if (evt.key == "a" || evt.key == "ArrowLeft")
        Game.update(2);
    if (evt.key == "d" || evt.key == "ArrowRight")
        Game.update(3);
});

First we check if the game even loaded (which I toggle to true when the WASM is initiated earlier), then we run through the different string names of all the keys. I want it to be WASD and arrow keys to move around so it's intuitively easy to understand. This unfortunately doesn't work on mobile devices, so that will be a whole different category of inputs we're not equipped for just yet.

...and it still doesn't work. Here's a JavaScript gotcha we didn't prepare for - the Game.loaded variable is completely undefined. Why? Look at this snippet of code.

Game = result.instance.exports;
Game.loaded = true;
Game.running = true;
Game.init();
main(); // begin

Seems valid, but it isn't. The WASM object we get from result.instance.exports seems to be completely immutable, but JavaScript didn't throw an error for it. The re-assignment changes and even remove old variables we tried binding in our original Game object definition, so these no longer hold true or store data. Oops!

No matter, we'll use another object to store metadata which I'll just call AppState at the top level near our original Game object. That original Game object is still handy for error handling in the case the WASM file doesn't load at all.

var Game = {
    'init': null,
    'get_pos': null,
    'set_pos': null,
    'update': null,
};

var AppState {
    'loaded': false,
    'running': true,
};

window.document.body.addEventListener('keydown', function(evt){
    if (!AppState.loaded)
        return;
    if ((evt.key == "w") || (evt.key == "ArrowUp"))
        Game.update(0);
    if ((evt.key == "s") || (evt.key == "ArrowDown"))
        Game.update(1);
    if ((evt.key == "a") || (evt.key == "ArrowLeft"))
        Game.update(2);
    if ((evt.key == "d" || evt.key == "ArrowRight"))
         Game.update(3);
});

After switching all the things over, it is now working much better.

Adding More Gameplay

This game is still really boring. I did mention we could add a goal to move to, but that would still be considerately boring. Maybe we can try to add a level of fun by adding a different type of goal.

This game shares resemblance to top-down 2D games like The Legend of Zelda, so it makes sense to me that perhaps we could include a bit of gameplay from those style of games.

A re-occurring theme in these top-down style games is the ability to "move" something else in the world dynamically. For instance, a moving block to solve a puzzle. We can make it a bit like Klotski, where the block must hit a goal to win.

First, we need new entities.

const Entity = enum(u8) {
    Empty,
    Player,
    Wall,
    Block,
    Goal,
};

Now we need new statements to render these new entities.

var main = function() {
    console.log("Main function started");

    var loop = function() {
    // clear the background
    ctx.fillStyle = "white";
    ctx.fillRect(0, 0, 100, 100);
    for(var x = 0; x < 10; x++) {
        for(var y = 0; y < 10; y++) {
        var cell = Game.get_pos(x, y);
        if (cell == 1)
            ctx.fillStyle = "red";
        else if (cell == 2)
            ctx.fillStyle = "grey";
        else if (cell == 3)
            ctx.fillStyle = "blue"; // push block
        else if (cell == 4)
            ctx.fillStyle = "green"; // goal
        else
            ctx.fillStyle = "white";
        ctx.fillRect(x*10, y*10, (x*10)+10, (y*10)+10);
        }
    }

    // loop to next frame
    if (AppState.running)
        window.requestAnimationFrame(loop);
    };
    loop();
};

Now we need to update our init() to include these new entities.

export fn init() void {
    _ = set_pos(4, 4, Entity.Wall);
    _ = set_pos(4, 3, Entity.Wall);
    _ = set_pos(4, 5, Entity.Wall);
    _ = set_pos(3, 4, Entity.Wall);
    _ = set_pos(5, 4, Entity.Wall);

    // set the player and the goal
    _ = set_pos(0, 0, Entity.Player);
    _ = set_pos(5, 5, Entity.Goal);

    // set a block to push
    _ = set_pos(3, 1, Entity.Block);
    return;
}

Now we have a new game world we can walk around in and check out. We cannot move anything or walk into the goal, which is fine and testament to some sturdy logic we wrote.

Adding the logic to the push block will add a new layer of logic we need to account for. Now we need to calculate another new destination, then see if we can move the block to that. Annoying, but could be refactored later down the line if needed.

// .. inside our player moving logic
.Block => {
    // move this block somewhere
    var newblockx = goalx;
    var newblocky = goaly;
    if ((dir == .Up) and (newblocky > 0))
        newblocky -= 1;
    if ((dir == .Down) and (newblocky < HEIGHT - 1))
        newblocky += 1;
    if ((dir == .Left) and (newblockx > 0))
        newblockx -= 1;
    if ((dir == .Right) and (newblockx < WIDTH - 1))
        newblockx += 1;

    var block_dest = calc_pos(newblockx, newblocky);
    if (block_dest == goalpos)
        return;
    var block_dest_ent = WORLD.cells[block_dest];
    switch (block_dest_ent) {
        .Empty => {
            _ = set_pos(WORLD.playerx, WORLD.playery, .Empty);
            _ = set_pos(goalx, goaly, .Player);
            _ = set_pos(newblockx, newblocky, .Block);
        },
        .Goal => {
            _ = set_pos(WORLD.playerx, WORLD.playery, .Empty);
            _ = set_pos(goalx, goaly, .Player);
        },
        else => {},
    }
}

And now we can push the block around a bit to try it out.

But this isn't without it's flaws, because this can happen.

A simple change to the event listener, we can trigger a re-initialization of the world by calling init() again. This can hilariously lead to errors, but we're gonna worry about that later.

window.document.body.addEventListener('keydown', function(evt){
    if (!AppState.loaded)
    return;
    if ((evt.key == "w") || (evt.key == "ArrowUp"))
    Game.update(0);
    if ((evt.key == "s") || (evt.key === "ArrowDown"))
    Game.update(1);
    if ((evt.key === "a") || (evt.key === "ArrowLeft"))
    Game.update(2);
    if ((evt.key === "d" || evt.key === "ArrowRight"))
    Game.update(3);
    if (evt.key == "r")
    Game.init();
});

By pressing r after moving the block, you end up with a funny "error" (it's not really an error, more unexpected behavior).

Winning

To close out this series, we're going to add one last thing - a win condition. You've already "won" by removing the blue block entirely, but how does the JS know when it's all said and done?

To do this we need a new flag, so we're going to modify the world state struct definition one last time.

const State = struct{
    playerx: u8,
    playery: u8,
    victory: bool,
    cells: [W_SIZE]Entity,
};

var WORLD = State{
    .playerx = 0,
    .playery = 0,
    .victory = false,
    .cells = undefined, 
};

A Boolean flag now exists to tell us when the game is effectively over. Pressing r will be how you reset the game back, and can start playing again. This isn't public, so we need to write a function to exposed it to JavaScript.

export fn is_won() bool {
    return WORLD.victory;
}

Now we can call it from JavaScript to write some text on screen when the game is over.

// in our rendering code
if (Game.is_won()) {
    ctx.font = '18px serif';
    ctx.fillStyle = "Black";
    ctx.fillText('You won!', 10, 15);
}

Lastly, let's update our update() function to change the victory flag when the game ends, and we can also change init() to reset the victory flag to start over.

export fn init() void {
    WORLD.victory = false;
    _ = set_pos(4, 4, Entity.Wall);
    _ = set_pos(4, 3, Entity.Wall);
    _ = set_pos(4, 5, Entity.Wall);
    _ = set_pos(3, 4, Entity.Wall);
    _ = set_pos(5, 4, Entity.Wall);

    // set the player and the goal
    _ = set_pos(0, 0, Entity.Player);
    _ = set_pos(5, 5, Entity.Goal);

    // set a block to push
    _ = set_pos(3, 1, Entity.Block);
    return;
}

// inside the update() function near the end
.Goal => {
    _ = set_pos(WORLD.playerx, WORLD.playery, .Empty);
    _ = set_pos(goalx, goaly, .Player);
    WORLD.victory = true; // you win!
},

And with all that, we've finally achieved real victory.

Plugging the Leaks

There's a few things to go over last, and it's due to the differences in how Zig creates a WASM file, but interactions with JavaScript can be flaky because they don't certain abstractions with each other.

First off, we have an enum type called Entity, which is a range of values conforming to a singular type. These enumerated names all have differently bound values that Zig understands, but JavaScript does not. For instance, check this out.

>>> Game.set_pos(4, 4, 1) // valid, sets a player
1
>>> Game.set_pos(4, 4, 255) // ?? what is this??
1
>>> Game.set_pos(4, 4, 20000) // not even 8-bit now
1
>>> Game.get_pos(4, 4)
32 // ????

Because the signature of set_pos is set_pos(u8, u8, Entity), it's C-language definition looks more like set_pos(u8, u8, u8), meaning it can support any value of eight bits. There isn't a restriction at this level, so using 255 as an Entity value is completely valid to the program, but invalid to us as game developers. This is a bit of a hole that should be plugged for safety.

We can define some level of safety by ensuring our input is an Entity value or not. This involves a Zig-level switch to check for all possible Entity values.

export fn set_pos(x: u8, y: u8, v: Entity) bool {
    var index = calc_pos(x, y);
    if (index < W_SIZE) {
        switch (v) {
            .Empty => { WORLD.cells[index] = .Empty; },
            .Player => {
                WORLD.playerx = x;
                WORLD.playery = y;
                WORLD.cells[index] = .Player;
            },
            .Wall => { WORLD.cells[index] = .Wall; },
            .Goal => { WORLD.cells[index] = .Goal; },
            .Block => { WORLD.cells[index] = .Block; },
            else => { return false; },
        }
        return true;
    }
    return false;
}

This might look reasonable, except this is actually a compiler error! Since we have five elements in Entity, this enum is considered 100% checked and safe in Zig's eyes. But for us, translating Zig into C-level code for WASM, this is not concrete.

There are two trains of thought here, and I am going to pick the one of least resistance after a bit of explaining. Our choices are:

bind all possible values of Entity on the JS side, making it easier to use set_pos() by using binded values instead of raw integers.
create a .Null property in the Entity enum, don't match against it in the switch, fallback to an else clause, and let Zig view that as "valid".

The lesser of two evils here is using the latter, and tricking Zig. Some may take issue with this, but I think it's a better work-around than attempting to fix the JS side, where there are less restrictions. An Entity.Null doesn't take up any extra space since the Entity enum is only a u8, and those addresses are already defined, so it's not like we're adding major overhead. This is simply a precaution I'm willing to make in the name of stricter safety.

With this set of code, Zig no longer errors on compile and we have a new layer of safety in our interface.

const Entity = enum(u8) {
    Empty,
    Player,
    Wall,
    Block,
    Goal,
    Null, // goes last, otherwise is the default value
};

fn set_pos(x: u8, y: u8, v: Entity) bool {
    var index = calc_pos(x, y);
    if (index < W_SIZE) {
        switch (v) {
            .Empty => { WORLD.cells[index] = .Empty; },
            .Player => {
                WORLD.playerx = x;
                WORLD.playery = y;
                WORLD.cells[index] = .Player;
            },
            .Wall => { WORLD.cells[index] = .Wall; },
            .Block => { WORLD.cells[index] = .Block; },
            .Goal => { WORLD.cells[index] = .Goal; },
            else => { return false; },
        }
        return true;
    }
    return false;
}

Now using set_pos() should start showing some safer results.

>>> Game.set_pos(1, 1, 20000)
0
>>> Game.set_pos(1, 1, 1)
1

This helps to further solidify the abstraction of the game to make it less leaky and error prone. The user might get a little crazy and try to nullify the entire map! We can secure our code more by making set_pos() not a publicly exported function as well, and since our JS doesn't manipulate the board, the user shouldn't be affected by this at all.

The other issue that can occur is what happens when the game resets and is re-initialized. In no capacity does the world get initialized to an empty state, and we cannot rely on the undefined keyword to do this for us. Simply setting something to undefined doesn't act as a means of setting an array full of values, unless it's the first time being initialized.

We can fix this by changing init() to fill values in the array all the way with empty values instead using a simple while loop.

export fn init() void {
    var index = 0;
    while (index < W_SIZE) : (index += 1) {
        WORLD.cells[index] = .Empty;
    }
    WORLD.victory = false;
    _ = set_pos(4, 4, Entity.Wall);
    _ = set_pos(4, 3, Entity.Wall);
    _ = set_pos(4, 5, Entity.Wall);
    _ = set_pos(3, 4, Entity.Wall);
    _ = set_pos(5, 4, Entity.Wall);

    // set the player and the goal
    _ = set_pos(0, 0, Entity.Player);
    _ = set_pos(5, 5, Entity.Goal);

    // set a block to push
    _ = set_pos(3, 1, Entity.Block);
    return;
}

Now the extra blue block glitch will go away after each reset. I am sad to see it go, personally.

Wrap-Up

We're slowly inching forward in better understanding how to use Zig for WebAssembly purpsoses. This was an important write-up for me because I now feel a little more confident in writing WASM with Zig, and can now start possibly focusing on developing more games with Zig targeting WASM.

The next write-up I do, however long it may take, I would like to delve into more complex things like using asset images and bringing that into JavaScript to use on a Canvas. This project could be extended to load levels or a score counter, but I will implement those on the side, as they would be slightly trivial to do. Check out my GitHub repo for more developments on it.

If you've made it this far, thank you for reading. I apologize if it seems too long, but hopefully the length is what will make it able to provide resourceful info for those looking to delve into WebAssembly with Zig.

Environment Variables with Python

Steven L — Wed, 21 Sep 2022 04:29:50 +0000

A common pattern that is seemingly not mentioned much in the Python space is how to manage environment variables reasonably. Frankly, for how often we use environment variables, there is no great solution to this.

What am I talking about? When a program is invoked, it is operated in a space where the user can define variables within the computer host environment that the program can refer to. These would be known as "environment variables", where the user can hold password secrets or configurations like file directories to modify a program.

Why this is good is that we no longer have to pass in passwords hard-coded or store passwords in our Git projects. In DevOps, environment variables can be set up within the Deploy or Continuous Integration areas so that the programs can receive variables safely and in a secure manner.

However, environment variables aren't actually fun to load, and lead to serious code bloat and other unsightly things that are tedious to dig through and maintain.

An example; let's say you want to import a set of environment variables crucial to your application. You might create a file like this, which gets loaded in with your program in the CI process (more likely than not this file is treated as a plain Bash script and gets source'd).

PATH_TO_DATA=/var/data
DB_URL=myapp.us.abc.s3.aws.amazon.com
DB_PASS=cupcake_sprinkles12345

These variables now exist in the program's host environment. If you didn't create this file and instead wanted to set environment variables, you can create any variable you want by doing this in the command line terminal.

$ MY_ARBITRARY_VARIABLE=5 python my_project.py

Getting these values into the Python program space is done by using the os library, which deals with all operating system-level functions and interactions. Using the os.getenv function, you can simply try to 'get' a variable.

import os
path_to_data = os.getenv("PATH_TO_DATA")

But as you can see, if this variable were ever to change, we're kind of stuck changing it not once, not twice, but three times in total - environment file, Python variable, and reference string.

Annoying, and just wait until there's multiple variables. But, there's a different way too. All the environment variables set by your running session can be loaded into Python via a globalized dictionary of sorts. Using os.environ, you can see all variables your profile has set, whether it's via manual definition or shell profile definitions.

import os
print(os.environ)
{"PATH_TO_DATA" : "/var/data", ... }

That's a little bit easier to pass around, but unpacking all of those key-value pairs is a bit of a pain. If you wanted only a few out of a large sum of environment variables, you're still left holding the bag having to do a lot of name definition.

(as a side-note: if you've guessed it already, os.getenv is simply a wrapper for dict.get, but requires a keyword argument to be supplied to provide a fall-back value when a key doesn't exist. Minor annoyance, but why move logic that already existed to a new keyword argument? Seems silly)

Automatic Variable Environment Loading

If your project requires environment variables to be loaded in across multiple modules, it might be annoying to have to do this all the time. And at the same time, it might be annoying to have to pass in a gigantic dictionary of environment variables through your modules, especially if not all of them are not even required.

An idea that occurred to me here is to employ the dataclasses library, which is a way of defining a sort of namedtuple (if anyone remembers those) for various data that does not need to be treated like a class object.

from dataclasses import dataclass

@dataclass 
class MyEnv:
    PATH_TO_DATA: str
    DB_URL: str
    DB_PASS: str

Now that looks like something interesting, and it uses typing annotations for better mypy analysis (... an environment dictionary is only strings... this is silly... anyways...), but the fundamental problem here is that the dataclass needs to be populated with information still. This alone isn't good enough to get the job done.

def init_env():
    return MyEnv(
          os.environ["PATH_TO_DATA"],
          os.environ["DB_URL"],
          os.environ["DB_PASS"]
    )

Yuck, is what you should be saying. I already hate it. Granted it's not the worst way to initiate something that can be shared and has formal rules, it's still tedious to initiate like this. Lots of unnecessary duplicate function calls.

The clever code-golfer you might be already sees the issue here - the repeated os.environ call. Why not just lift that into a multi-argument unpack and use a list comprehension?

def init_env():
    return MyEnv(
      *[os.environ[v] for v in
         ["PATH_TO_DATA", "DB_URL", "DB_PASS"]]
    )

Congratulations, you just annoyed everyone in your project with your clever code-golf. Yes, it's more functional, yes it's a code reduce and it will be fast, but still, I think we could probably do better than this, right?

The problem with environment variables is that it can lead to some weird code choices - you have a known list of variables you want to read, but somehow you need to get it into code. Maybe you create constants which hold the names of the variables you want to read, then read them later. Or maybe, you use a list or a dict of mappings to organize how it's going to come in easier.

The problem I see is that it creates too many pain-points of having to update code across a large project. You change one environment variable name, suddenly you're changing a ton of files just to update for that one reference. This isn't really a problem that can be aided by any software tricks.

If you want to bind variables to a collection like an Enum or a dataclass, then you need to share that object across multiple files again, and it all sort of leads back to a single source of pain.

Let me rewind a bit with a different path that maybe we can try and tackle. All of our environment variables are loaded into a dictionary in os.environ.

import os
os.environ["PATH_TO_DATA"] # /var/data

A variable to this would be defined as something like:

import os
PATH_TO_DATA = os.environ["PATH_TO_DATA"]

The variable name is the same as the environment variable. Wouldn't it be easy if we could simply have those environment variables turned into variables automatically for us?

Via the hidden method __setattr__, we can actually do something like this very easily. For this we will need a new class to generate instances of environment info for us.

import os

class EnvInfo(object):
    def __init__(self, args=[]):
        for k in args:
            self.__setattr__(k, os.environ.get(k, ""))

# test it out
e = EnvInfo(["MY_VAR"])
print(e.MY_VAR)

Now try it out by doing

$ MY_VAR=5 python EnvInfo.py
5

It works by using __setattr__(), which is the internal method for Python objects to set methods (each time you do a self.var declaration, it's using __setattr__() to bind it). The EnvInfo class itself has no bound variables, so it's an open book for setting variables.

By handling a list of arguments and looping through them, you can create an environment variable dictionary-like object that is easy to pass through around your codebase. And by providing a list of variables, you are still limiting what variables are out-bound for your code, so no accidental slip-ups of important secret variables.

We could even go a step further and provide a dictionary of functions to invoke when an environment variable is loaded too. Let's say you wanted to convert things to numbers, or lists, or whatever you fancy.

import os

class EnvInfo(object):
    def __init__(self, args=[], kwargs={}):
        for k in args:
            for k in args:
                self.__setattr__(k, os.environ.get(k, ""))
            for key, func in kwargs.items():
                self.__setattr__(key, func(os.environ.get(key, "")))

e = EnvInfo([], {"MY_VAR": lambda x: int(x)})
print(type(e.MY_VAR))

Try it out again:

$ MY_VAR=5 python EnvInfo.py
<class 'int'>

This provides a way of better error-reporting or doing post-load transformations of variables. You might want to do string splits, integer conversions, URL checking, hashing, or any sort of testing. This kind of system helps to better improve testing and program correctness and makes it relatively simple to implement.

Now, is this an improvement on where we were before with environment variable loading? For the most part, I would say absolutely. In this small example, a main entry point program is responsible for passing environment variables as arguments to some functions.

import os

PATH1 = os.environ["PATH1"]
KEY1 = os.environ["KEY1"]

PATH2 = os.environ["PATH2"]
KEY2 = os.environ["KEY2"]

run_program(PATH1, KEY1)
run_program(PATH2, KEY2)

It's a lot of boilerplate, and as the functions grow in complexity, more environment variables may be needed. Extending the function signature is simply too much, because it would turn into a run-on code line. With our new system in place, it's a little bit easier. By modifying the run_program signature to take in an EnvInfo object instead:

import os

JOB1 = EnvInfo(["PATH1", "KEY1"])
JOB2 = EnvInfo(["PATH2", "KEY2"])

run_program(JOB1)
run_program(JOB2)

It's been simplified quite a bit and looks a lot cleaner. Extending this to support more variables is as simple as appending more strings to the input lists, and nothing else is required. Since the variables aren't concrete to the EnvInfo implementation, some linters and language server protocols may not be entirely happy with you, but hey, it works.

Thanks for reading and hope you enjoyed!

General-Purpose Python Type Contracts

Steven L — Sun, 18 Sep 2022 20:29:05 +0000

For a while now at my new job, I have been staring at a lot of Python code. It's nice and easy, but I wonder where improvements can be made. A lot of improvements would simply come straight down to actually adding type signatures to functions and using mypy to cover the fuzzy testing.

I, however, have problems, as that seems like a half-baked problem in itself. mypy enables smarter code coverage, but I don't feel like it helps the runtime or the programmer development part much. Also, the fact that mypy is completely optional as well doesn't help much. In fact, simply looking at the first sentence in the typing docs shows us:

Note: The Python runtime does not enforce function and
variable type annotations. They can be used by third party
tools such as type checkers, IDEs, linters, etc.

Python is a dynamic language, and as such there is no such thing as "static typing", unless you employ tools (like mypy) to force Python and your development projects to simply act like it is a static language.

One project I wrote a long time ago attempted to fix this, and I did it by using function decorators to "fix" the language. I was influenced a lot by Haskell, and still am to this day, and love the idea of functional languages like Haskell or OCaml that simply do away with dynamic-language problems by making it much easier to write in a static-typed language. They do this because they made their compilers just... smarter. That's it.

Decorator Patching

Using decorators to fix problems is the easiest, lowest barrier fix we could make, as it simply overlays on top of a plain Python function. The decorator absorbs basic information and passes it along to the target, and once executed, outputs the output of it's target function.

There's two parts to this we can implement: an expects() function, and a outputs(). The expects() function takes in a list of arguments describing the parameters of our goal function, while the outputs() function takes in a list of arguments describing the type of the final value returned. You can use neither, you can use both, or you can choose to use only one of them. It depends on how you want to further solidify your code.

def expects(*types):
    def func_in(fn):
        def in_wrap(*args, **kwargs):
            if len(types) != len(args):
                raise SyntaxError(f"Expected {len(types)}, got {len(args)}.")
            for t, v in zip(types, args):
                if not isinstance(v, t):
                    raise TypeError(f"Value '{v}' not of type '{t}'")
            return fn(*args, **kwargs)
        return in_wrap
    return func_in 

def outputs(*types):
    def func_out(fn):
        def in_wrap(*args, **kwargs):
            finalv = fn(*args, **kwargs)
            if not hasattr(finalv, '__iter__'):
                if not isinstance(finalv, types):
                    raise TypeError(f"Value '{finalv}' not of type '{types}'")
            else:
                for t, v in zip(types, finalv):
                    if not isinstance(v, t):
                        raise TypeError(f"Value '{v}' not of type '{t}'")
            return finalv
        return in_wrap
    return func_out

If this looks like a lot of things going on, I can explain.

The goal behind a decorator is to treat plain-code functions as first-class citizens, allowing us to pass around the reference to the function itself. By doing this, we can take in functions, call them, and return functions from other Python functions.

A decorator returns a function that takes in a function, and does something with it. In some cases, they exist to modify the arguments to the function in some meaningful way, and helps us to modify logic of functions without changing much of their definition itself, or by re-using decorators to share them across a codebase. Instead of logging the inputs and outputs of function with gross print statements, we can write decorators that help us to debug the input and output to functions, all by decorating them easily.

def logger(fn):
    def handler(*args, **kwargs):
        print(f"{fn.__name__}({args}, {kwargs})")
        v = fn(*args, **kwargs)
        print(f"=> {v}")
        return v
    return handler

@logger
def f(x, y):
    return x*y

So long as you carry the arguments through until the final function call, these decorators do their job right. You might be concerned with double-function invokes since @expects and @outputs are separate decorators and can be used freely from one another. If using both, you might be concerned that the target function activates twice, but no, it doesn't, since it works by passing a decorated function into another, it invokes safely, as you can see with this example.

@expects(int) 
def do_with_int(x):
    print("Hi, I handled an int")

@outputs(str) 
def output_str():
    print("Hello, returning a string")
    return "Hi, I'm a string"

@expects(int, int)
@outputs(int) 
def multiply(x, y):
    print("I don't get activated twice")
    return x * y

Flat Contracts

Using decorators gets us a jump-start on better type checking, which is nice. It solidifies code and produces runtime errors which can almost act like they're compile-time errors. But the true power of languages like Haskell and OCaml comes from the idea of extending the type system in a way that allows functions to produce values that can be of any arbitrary type within a certain range.

Take for example this function, which adds one to a number and returns it.

@expects(int) 
@outputs(int) 
def add1(x):
    return x+1

add1(5) # 6
add1(5.1) # exception thrown, not an int

The + operator, shortcut for operator.add in the operator library, overloads different numeric types that implement __add__(). float and int both overload that method to prove that to the operator.add method. However, our crude type checker doesn't have a way of binding multiple types for one input variable, which is kind of unfortunate and would lead to duplicate code for this case.

A better way is to come up with a system that can can be used to define a range of type-checking and other useful facts about values. Sure we have the int function to type-compare and construct values, but there is no contextual association in this, it's purely a number. How do we restrict numbers to a range? How do we check if it's positive? Or if it's odd or even?

Asserting if a condition is satisfied is what we could call a "contract" on that type, that it must satisfy some bounds that we define in order to consider it valid. The contractual agreement between the values and functions would have to be satisfied, that way the functions can operate within that scope, leading to less run-time errors due to silly things like out-of-bounds, off-by-one or even nulled-out values or misformed data.

Let's start with what I will call a "flat" contract. It exerts minimal effort in asserting if a value is valid. A flat contract is normally one that is easy to prove based on it's bounds and the inputs. Let's pretend we have a contract that takes one Python type, and checks if value(s) given to it all match that one type.

def flat_contract(t):
    def inner_check(*vals):
        for x in vals:
            if not isinstance(t, x):
                return False
        return isinstance(x, t)
    return inner_check

is_int = flat_contract(int)
is_float = flat_contract(float)

print(is_int(3)) # true
print(is_float(3)) # false
print(is_float(3.1)) # true

We can see that is_int and is_float are valid flat contracts that are pretty simple to prove, and that an int cannot actually pass as a float in this system, so Python doesn't do any coercion.

The next step is providing tools to check if inputs belong to a family of types, satisfying the number overloading issue we were having. If we can write a contract that checks if a value matches a range of types, it will help us write better, more re-usable, generic Python code. For this we need to write contracts that implement similar logic to that of the any and all functions.

def or_contract(*types):
    def inner_check(*vals):
        for x in vals:
            res = [isinstance(x, t) for t in types]
            if not any(res):
                return False
        return True
    return inner_check

is_num = or_contract(int, float)

print(is_num(3, "seven")) # false
print(is_num(3.1, 4.1, 700)) # true
print(is_num("3.1", "300")) # false

Here we force logic similar to that of a Boolean or operator, where one value on the left or right-hand side must satisfy the predicate to count as truthful. For this we check each value, and run it across the list of types, and if not a single match occurs, we consider it invalid. In our example, we put int and float into an or_contract, which will help us to validate incoming values.

The more complicated contract is the and_contract, which is implemented only by changing a few things around.

def and_contract(*types):
    def inner_check(*vals):
        for x in vals:
            res = [isinstance(x, t) for t in types]
            if not all(res):
                return False
        return True
    return inner_check

is_num = and_contract(int, int)

print(is_num(3, 3.1)) # false
print(is_num(3.1, 4.1, 700)) # false
print(is_num(3, 4, 5, 6)) # true

It's the same core logic as or_contract, but the reason why I say this is complex, is because very few types in Python will ever satisfy this ruling unless a lot of object-oriented inheritance occurs. Some types may qualify as others, but very rarely. For instance, a float can't be an int, but very few cases require instances where one type may need to adhere to the type rules of another, maybe for instance a dict type might need to satisfy multiple bounds like defaultdict.

>>> d = {}
>>> isinstance(d, dict)
True
>>> isinstance(d, defaultdict)
False
>>> d2 = defaultdict()
>>> isinstance(d, defaultdict)
False # dict() does not qualify as defaultdict()
>>> isinstance(d2, defaultdict)
True # qualifies as a defaultdict
>>> isinstance(d2, dict)
True # also qualifies as a dict, can be and_contract'd

Now, all we need to help verify contracts is a way to bind it to functions, via what's called a function contract. It's a lot like expects() and outputs(), but we can't use those, since flat contracts are functions and not strictly types. For this we'll use the names contract_in() and contract_out().

def contract_in(*contracts):
    if not all([callable(c) for c in contracts]):
        raise TypeError("All types must be callable contracts")
    def fn_wrap(fn):
        def arg_wrap(*args, **kwargs):
            if len(contracts) != len(args):
                raise SyntaxError(f"Expected {len(contracts)} inputs, got {len(args)}")
            for con, val in zip(contracts, args):
                if not con(val):
                    raise TypeError(f"Expecting a value to satisfy {con.__name__}, got {type(val)}")
            return fn(*args, **kwargs)
        return arg_wrap
    return fn_wrap


def contract_out(*contracts):
    def func_out(fn):
        def in_wrap(*args, **kwargs):
            finalv = fn(*args)
            if not hasattr(finalv, '__iter__'):
                for con in contracts:
                    if not con(finalv):
                        raise TypeError(f"Expecting value to satisfy {con.__name__}, got '{type(finalv)}'")
            else:
                for con, val in zip(contracts, finalv):
                    if not con(val):
                        raise TypeError(f"Expecting value to satisfy {con.__name__},  got '{type(val)}'")
            return finalv
        return in_wrap
    return func_out

Seems similar right? That's because it's almost exactly the same code as expects() and outputs(). This one operates on contract functions instead of simple type constructors, allowing for new predicate checks to occur at both the input and output stages of functions.

is_num = or_contract(int, float)

@contract_in(is_num) 
@contract_out(is_num) 
def square(x):
    return x * x

square(500) # passes
square(True) # passes?
square("string") # fails

Seems like we're all good here! Except wait, why is Boolean passing the is_num contract? Well...

>>> isinstance(True, int)
True

Oops! Python thinks True, a Boolean value, is also a number. It even has an __add__() method bound for integral addition, but not floats. This is kind of okay, because a Boolean is considered a number (True=0, False=1/anything else) in traditional mathematics. Normally this would be fine, but since Boolean isn't a helpful number for basic addition, we might want to come up with a better alternative, and I think we can all blame isinstance() for this fun little bug.

It might be more beneficial in some instances to compare class names instead of the isinstance() function. isinstance() will follow the inheritance of object-oriented programming, and as such some classes may inherit properties you do not expect. It may be beneficial to have a strict_contract() function to compare exact class types instead.

def strict_contract(t):
    def inner_wrap(val):
        return val.__class__ == t
    return inner_wrap

This now compares the exact class constructor function against the class used to construct the given value. We could extend or_contract and and_contract to also use this, or we can add a keyword toggle to change behavior and add a strict mode to the contracts, but I will leave that as an exercise for the reader.

Overlaying with Type Annotations

Since type annotations are a direct Python library, one is left to wonder how they actually work. Ever since the release of them a few versions back, all functions come with a hidden variable now, which you can see for yourself here:

>>> def fn(x): return x
>>> fn.__annotations__
{}
>>> from typing import *
>>> def gn(x: Any) -> Any: return x
>>> gn.__annotations__
{'x': typing.Any, 'return': typing.Any}

The information is attached to the function object, and this is pretty much how all third party tools do work with Python to gather this type information. However, can we use this ourselves to our benefit? This is where I think it can get tricky, and unfortunately would still end up looking like our current contract system.

Take for example this code:

def fn(x: int) -> str:
    return f"{x}"

It's annotations:

>>> fn.__annotations__
{'x': <class 'int'>, 'return': <class 'str'>}

This would be a relatively simple process to check this with a decorator we could call enforce(), which would enforce the types it used to annotate the function.

def enforce(fn):
    def in_wrap(*args, **kwargs):
        annotes = fn.__annotations__
        for vname, val in zip(fn.__code__.co_varnames, args):
            if not val.__class__ == annotes[vname]:
                raise SyntaxError("Mis-matched annotation type")
        finalv = fn(*args, **kwargs)
        if not finalv.__class__ == annotes['return']:
            raise SyntaxError("Mis-matched annotation type")
        return finalv
    return in_wrap

@enforce
def fn(x: int) -> str:
    print("activated")
    return "YO"

fn(500)
fn("oops")

At a primitive glance, this is completely fine. It does similar stuff to what we have defined earlier with the contract systems. It checks types, and raises errors if a value is not put in properly. I am lazy so I didn't quite do this for the **kwargs, but again, another exercise for the reader. This code actually digs into the Python code object itself to scan for the local variable names to compare against the inputs and check for validity. However, this does not carry over for code like this:

import typing

def fn(x: int) -> typing.List[int]:
    return [y*y for y in range(x)]

With it's annotations:

>>> fn.__annotations__
{'x': <class 'int'>, 'return': typing.List[int]}

The point behind typing was to add a way to define functions in a more abstract manner, but since the typing library doesn't provide any enforcement, this is just work left to the plugin developers to figure it out and do all the busy work. That is something I find to be annoying. typing.List describes some iterable object that should be a list, but since lazy generation exists, this doesn't really do a good job of saying "should my output be a flat list, or a lazily-generated one that yields values, and is actually a function?"

I expect that the Python developers expect us, the programmers, to come up with a solution for the gigantic amount of work they left for us by providing the typing library. If you do dir(typing), you can see all the names defined, and they are mostly enumerated names or special syntax names to wrap other types.

Summary

Should we define an enforce() function to enforce the types of everything in the typing library, you would have a long road ahead of you to provide logic and gigantic switch statements to prove the validity of everything part of the typing library. And for most of the Python community, that's probably fine, because it's a shared goal they can all strive for to help others. The mypy library is important for that reason, because it is a great static analysis tool that others can join in and help with, and needs minimal effort to start using.

However, I think I might be more leaning on the side of my contract system. I am probably the only one who might think a clever contract system like this would do better to constrain programs. mypy doesn't actually help Python developers make stronger code, it tells them what they're doing wrong with their code currently. Whereas a contract system helps to enforce code validity and cements the abstractions in place.

However you go about it, hopefully this is an enlightening piece on how to improve the general smartness of your Python code. A few examples of more contracts you could easily write to make code smarter:

integers within a certain range
odd or even numbers
strings being non-zero, or being a certain length
checking if a list contains a uniform type
restricting the values of a dictionary
checking whether certain values have bound methods
automatic regex checking for valid input
add Python doc strings to aid developers and users

Thank you for reading and hope you learned a new thing or two.

A Simple Caching Racket Macro

Steven L — Sun, 11 Sep 2022 16:34:05 +0000

A project I am working on to build a custom website involves working with a third party API to receive a lot of information. The API associates users with products tied to their account, and the API is able to give you more information about each product in separate API calls.

To fully build this project, I need information about all users, the products they own, and then additional info about each product to produce an informative website. We have N users and M products and there are a number of connections between the users and products.

To start bootstrapping this project, we iterate over the users, then iterate over all the products they own.

(for ([user all-users])
  (define users-products ($fetch-products (user-id user)))
  (for ([pid users-products])
    (define product-info ($fetch-product pid))
    (do-something-with product-info)))

This isn't good though, because if we have 500 users and each user has 1000 products, we would be making somewhere around 500,000 network calls. I don't think the API provider would care much for that. Instead, we must take note that while user information can change regularly, product info often times does not.

Let's instead store product information on our file system as flat files. This way, when we do new builds of our website publishing, we have the products already set aside, and we can still fetch user live data from the API as that can change frequently.

(for ([user all-users])
  (define users-products ($fetch-products (user-id user)))
  (for ([pid users-products])
    (define cached (build-path "cache" (format "~a.txt" pid)))
    (define product-info
      (if (file-exists? cached)
          (file->string cached)
          (let ([resp ($fetch-product pid)])
            (call-with-output-file cached #:exists 'replace
              (lambda (out)
                (write resp out)))
             resp)))
    (do-something-with product-info)))

This is kind of a mouthful. This program as it stands isn't very modular, as caching is now tied with our main program logic. Making changes to the main program would mean ducking in and out of caching code, and doesn't reflect our true intent very well. We should move the caching logic into a separate function for modularity and readability.

(define (Cachify path-check code-to-run)
  (if (file-exists? path-check)
      (file->string path-check)
      (let ([resp code-to-run])
        (call-with-output-file cached
          #:exists 'replace
          (lambda (out)
            (write resp out)))
         resp)))

This looks right to me. We check a file path, and if there is no file, we execute our code payload. Seems right so far.

But herein lies our next problem: because of how Racket evaluates code, trying to use this would still result in us executing a network request each time, regardless of whether or not it was saved to file or not.

> (define product-info (Cachify "product.txt" ($fetch-product 5))
; ... still runs the network request ...
> "{\"pid\":\"5\"}"

This obviously doesn't work for us. We need to shift this into a language-level macro, where Racket will evaluate the macro as if it were a normal Racket function, but treats the arguments as literal values instead of sub-expressions to evaluate. Fortunately it doesn't look much different.

(define-syntax-rule (Cachify path-check code-to-run)
  (if (file-exists? path-check)
      (file->string path-check)
      (let ([resp code-to-run])
        (call-with-output-file path-check
          #:exists 'replace
          (lambda (out)
            (write resp out)))
         resp)))

This macro works by performing a literal translation on the code it receives and substituting variables appropriately. If we attempt to run this macro, we can see it will substitute the code in a way that it will only run our network request if it isn't on the file system first, and not before it has a chance to check. We can view the macro transformation using expand and syntax->datum to get a better picture.

> (syntax->datum (expand '(Cachify "1.txt" (println "Hello"))))
'(if (#%app file-exists? '"1.txt")
   (let-values (((temp16) '"1.txt"))
     (if (#%app
          variable-reference-constant?
          (#%variable-reference file->string101))
       (#%app file->string 'binary temp16)
       (#%app file->string101 temp16)))
   (let-values (((resp) (#%app println '"Hello")))
     (let-values (((string:5:8) call-with-output-file36)
                  ((temp17) '"1.txt")
                  ((temp18) 'replace)
                  ((temp19) (lambda (out) (#%app write resp out))))
; ...

It looks a bit garbled, but this is how it looks inside the Racket VM. It first checks if a file exists with the (if (#%app file-exists? '"1.txt"), and if that fails, it moves down to run the expression we gave it on the (let-values (((resp) (#%app println '"Hello"))) line.

Not only will this work with network requests, but frankly any computation you do, you can use with this macro. The nice part is you can adjust the macro itself to support better, modern technology, like say if you wanted to use a database or something like redis as a memory store. You can modify the macro and continue to use it without having to change the caching implementation all over your program. Now let's go build our website.

(for ([user all-users])
  (define users-products ($fetch-products (user-id user)))
  (for ([pid users-products])
    (define product-info
      (Cachify (build-path "cache" (format "~a.txt" pid))
               ($fetch-product pid)))
    (do-something-with product-info)))

On all incremental builds, you will have saved your API host thousands of calls, so long as their product info doesn't change. If it ever does, you might just have to invalidate your cache by cleaning out old files or some such, or creating an update cycle where the cache will validate itself only on certain days or intervals. But that's up to you, of course.

Thanks for reading!

WebAssembly with Zig, Part 1

Steven L — Fri, 05 Aug 2022 15:34:48 +0000

I am, by no means, an expert with WebAssembly. I feel like I still barely understand it, and how to properly use it. But I am going to change that.

WebAssembly is a new hot technology that involves loading a bunch of code that resembles the classic Assembly language. How this differs from JavaScript, which the browser already understands natively, is that the browser itself is capable of interpreting WebAssembly code much faster.

The reasons why WebAssembly is so much faster, would boil down to a couple basic reasons:

WebAssembly is inherently smaller, meaning it can retrieve code over the internet faster
WebAssembly is already parsed and optimized, so it avoids those steps as well
WebAssembly is compiled, there is no compilation phase when running the instructions in the browser
There is no garbage collection within WebAssembly, and it is all managed manually

WebAssembly performs better than JavaScript in many areas. So why hasn't it taken off for many people yet?

Generating WebAssembly

The first step in developing a WebAssembly application is actually finding a way to create it - it is ill advised that you find a 1000-page textbook and start learning how to write WebAssembly yourself. Unless you want to build the next Roller Coaster Tycoon.

WebAssembly is chosen as a target for cross-compilation in a few compilers. It is possible to write JavaScript to target WebAssembly, but JavaScript might not be entirely suitable for handling memory manually.

Instead, we often look to things like Rust or TypeScript to target WebAssembly. Rust has many language facilities which suit WebAssembly perfectly, namely that the Rust compiler strongly enforces memory correctness and prevents many spatial and temporal memory exploits through the syntax alone.

However, I am not super interested in Rust, it's a little too complex for my liking. Rust is a language that's broader than C++ and packs too many hammers into one toolbox. Abstract data types, macros, pattern matching, lifetime elision, traits, ownership and borrowing, asynchronous programming - enough topics to fill out two semesters of computer science courses.

A simpler language exists, named Zig. Zig is a newish language that still hasn't hit a 1.0 milestone release quite yet, it's currently at 0.9.1. But Zig is, in my eyes, a very good language. It's plain, it's simple, and you can read through the documentation pretty easily. It lends itself to classic C, and borrows ideas from other languages in a few places.

The reason why I also don't like Rust as much, is that it heavily depends on generating WASM bindings for you through a package called wasm_bindgen. This is again, a place where the magic just "happens" in the Rust background and you have to be okay with that and assume it's always correct.

The beauty behind Zig is that it supports many cross-platform targets by using shims from LLVM, and almost any platform that LLVM supports, Zig will support. Let's try to write some Zig code now to add two signed integer numbers.

// add_two.zig
pub fn add_two(a: i32, b: i32) i32 {
    return a + b;
}

This is what it's like to write Zig, and it looks vaguely similar to that of Rust with the type annotations. A pub keyword tells Zig to expose this function to the rest of the Zig program space that you write code in.

For when we want to write Zig code to be turned into a WASM program, we build it as a library, which is part of the command set for compiling Zig code via build-lib.

$ zig build-lib add_two.zig -target wasm32-freestanding -dynamic

That command will create for you a WASM file that we can use in a browser. We can verify the size of it to be certain.

$ ls -alh add_two.wasm
-rwxr-xr-x 1 steve steve 42 Aug  4 16:47 add_two.wasm*
                          ^- size is 42 bytes

Wait... Forty-two bytes?! That seems a little small. Did we do something wrong? There's no way a WASM can be that small. Well, unfortunately we wrote Zig the wrong way, but that's okay, we're learning.

Re-write our code, except instead of using pub, we are going to use export.

// add_two.zig
export fn add(a: i32, b: i32) i32 {
    return a + b;
}

Compile it again with:

$ zig build-lib add_two.zig -target wasm32-freestanding -dynamic

Verify it's contents with:

$ ls -alh add_two.wasm
-rwxr-xr-x 1 steve steve 329 Aug  4 16:49 add_two.wasm*

Much better, 329 bytes! The reason why we needed to use export is to tell Zig we are looking to create what is effectively a shared library object, which exports definitions and code. Before, we never indicated we were trying to create a library, so Zig saw no exportable bindings and exported a WASM file devoid of any meaning.

Eventually, it'd be nice to be able to build a Zig project without needing to write out the zig command yourself. I will probably cover that later on, but for now it helps to understand what certain flags are.

Importing the WASM

Now here's the section worth going over. We have a blob, but how do we bring it into our web page and get it running and all that?

The first step is that we have to somehow bring in the code altogether. This can be done via one of two ways: either by a network request, or by converting it to a base-64 format and embedding it in your web page. Both have their advantages and disadvantages, but I will use the network method first, as that will be the more common-case.

Back in the day, JavaScript got a feature called XMLHttpRequest, which is the ability to perform a network call from your current web page and arbitrarily retrieve or execute some function remotely. This was the turning point for most, if not all, modern websites currently. XMLHttpRequest was the king for a time, until newer methods started surfacing much later.

The new Fetch API is a common browser implementation to be smarter than XMLHttpRequest and be more elegant, but I'm a bit of an oldie. XMLHttpRequest is simple, so I'm going to stick with that right now.

First, edit your web page to look like the following:

<!doctype html>
<html>
  <head>
    <title>WASM Demo</title>
  </head>
  <body>
    <h3>WASM Demo</h3>
  </body>
  <script src="loader.js"></script>
</html>

Then we need to edit loader.js next. We put the <script> tag after all the other HTML in the page, because in JavaScript terms, it's easier to work with DOM elements if they actually exist, instead of depending on waiting for their existence to be defined first. It's less annoying this way. But you can put it wherever you want depending on your application.

Inside loader.js, we need to load our WASM blob through a special initialization process through the WebAssembly module called initialize. This is the main process for loading WASM, although a newer function exists to support a streaming buffer, but we're gonna go classic right now.

// loader.js
request = new XMLHttpRequest();
request.open('GET', 'add_two.wasm');
request.responseType = 'arraybuffer';
request.send();

request.onload = function() {
  var bytes = request.response;
  WebAssembly.instantiate(bytes, {
    env: {}
  }).then(result => {
    // do wasm things here!
    var add_two = result.instance.exports.add_two;
    console.log(add_two(3, 5));
  });
};

Voila! WASM code loaded into our website and JavaScript is able to call it.

The instantiate function is the way to take in a byte array representing our WASM code, which we requested in as an array on line 3. The follow-up is that there's an event we define to run after the WASM code has been successfully loaded, which is the .then() function. This is where we should be operating, as this code will only run when the WASM is fully ready to use.

The issue is you need a running HTTP server to serve this. Most browsers have strict CORS settings that forbid you from loading files locally, so we need to spin up a quick HTTP server. If you have Python installed, you can do the following and navigate to 0.0.0.0:8000 and find your index file wherever you placed it.

$ python -m http.server

With all that, we now have a working WASM application. Well, if you could call adding two numbers a working application.

Importing JavaScript Functions

When we write code in Zig, we have limited access to functionality that only exists elsewhere in the browser. For example, we can't use the Canvas API or WebAudio API. Zig has no bearing on what these APIs are, and without some kind of Zig code to back these APIs, our Zig compiler is to be left in the dust.

However, all is not lost. Let's start with the real question: how do we print text from inside Zig?

The traditional means of printing text in Zig is by using the std.io library, but that library needs some kind of output stream to write text to, to which we don't really have that. Instead it might be better to import a reference to the JavaScript namespace by using the env field of the instantiate function used to start the WASM code.

That env field can be used to bring in external references into Zig (or whatever your language is), so you can you JavaScript native functions in your Zig program.

First let's modify add_two.zig.

// add_two.zig
extern fn print(a: i32);

export fn add(a: i32, b: i32) i32 {
    print(a + b); // try it here
    return a + b;
}

Here we make use of the extern keyword to refer to a function from outside our program scope, which is to be brought in by the Zig process (or some other process). Usually this is important when targeting C libraries that there is no native Zig code for, but we can still use based on it's object file during the build phase.

Now we have to bring in a reference via the env section to some kind of printer function. We could import console.log, but I'll create a wrapper just in case.

// loader.js
request = new XMLHttpRequest();
request.open('GET', 'life.wasm');
request.responseType = 'arraybuffer';
request.send();

request.onload = function() {
  var bytes = request.response;
  WebAssembly.instantiate(bytes, {
    env: {
      print: function(x) { console.log(x); }
    }
  }).then(result => {
    // do wasm things here!
    var add_two = result.instance.exports.add_two;
    console.log(add_two(3, 5));
  });
};

It doesn't hurt to always bring in some kind of print function like this for further debugging. Once your code turns into a WASM file, the browser debugger will only get you so far before you start running into logic issues or whatever else comes up. Bring in a print function for your own sanity.

Release Modes

The last part I will go over now is the release modes available in Zig. For almost all compilers, there are different build modes you use to release your project. Some of my smaller WASM projects ended up compiling to thousands of bytes, which ordinarily I wouldn't complain about, because that's still pretty good on average.

However, I wasn't using the right release mode for this. Zig has several flags you can put into the compiler to optimize your release binary.

Debug, fast compile, safety checks on, slow runtime performance, very large size
ReleaseFast, fast performance, safety checks disabled, slow compilation, large size
ReleaseSafe, medium runtime performance, safety checks enabled, slow compilation, large size
ReleaseSmall, medium performance, safety off, small binary output, smallest size

The debug mode is perfect for when you're still prototyping, but for releasing, you would probably want one of the three after that. ReleaseSmall is very tempting, but if you want some good performance, you might want to consider ReleaseFast instead. In safety critical applications, you should probably aim for using only ReleaseSafe.

In one of my other small demos making a Mandelbrot Set generator, I build the Zig file with a normal build-lib command.

$ zig build-lib mandelbrot.zig -target wasm32-freestanding -dynamic

It's original size comes out to:

$ ls -alh mandelbrot.wasm
-rwxr-xr-x 1 steve steve  915 Aug  5 10:22 mandelbrot.wasm*

Not bad, but then after doing ReleaseFast:

$ zig build-lib mandelbrot.zig -target wasm32-freestanding -dynamic -O ReleaseFast
$ ls -alh mandelbrot.wasm
-rwxr-xr-x 1 steve steve 232 Aug  5 10:25 mandelbrot.wasm*

It comes out to a whopping 232 bytes. Pretty cool!

In the next post, I will go over some more advanced topics using Zig with WASM, introduce more Zig features like enums or structs, and try to create more programs using Zig and WASM together.

It is my hopes to shed some more light on Zig and WASM, as I find the current resources to be a little scarce. I have an interest in this area and I'm having fun exploring it currently with Zig.

Thanks for reading!

(Warnings: Zig is still in it's beta stages, so things may not be the same in the future. It's still fun none the less.)

Making Structs Fun and Interesting

Steven L — Mon, 18 Jul 2022 17:40:04 +0000

Organizing data into easy-to-use structures is one of the biggest underpinnings of programming. An organized, named structure of data helps us to write cleaner and more efficient code that is shareable with others.

However, I also think structs are ugly, and require too much mental overhead. Remembering the way to initialize each and every struct is a pain, and most of the time we would prefer to use smart constructors to execute the logic we would want anyway.

Or we might choose to use self-mutating methods to build up a struct over time, which is the result of method chaining. This is a much better method in my eyes because you don't have to fill in all the details of a struct at the struct's initiation point.

As a primary Racket user, I want to share structs in other languages and stack it up against Racket, and share the rabbit hole of struct initialization and how to improve it.

Structs in Rust

A struct in Rust is no different than that of C or C++ or even D, really. A struct is a list of fields containing some data, and we apply names that we wish to bind the fields to such that it's easy to access them later on. This is basically attaching named fields to an array of data, but this array of data doesn't have to be the same type.

struct MyData {
  pub a: i16,
  pub b: u32,
  pub c: f32,
}

To use this, we have to define a variable binding and create the struct and initialize all it's fields.

let my_point = MyData { a: 5, b: 77, c: 3.5 };

But if the struct had a lot more fields, this would start to run on quite a bit.

let my_long_struct = MyLongExample {
  a: 0, b: 1, c: 2,
  d: 3, e: 4, f: 5,
  g: 6, // ...
};

It would be at this point you would want to consider a smart constructor method that would fill in fields that you don't need immediately, like say you only wanted one or two fields to be filled while the rest get defaulted to their zero-type states (numbers are zeroed, Booleans are false, strings are empty, etc).

A smart constructor has a number of uses, but filling in un-needed values is one, but another can be satisfying an invariant rule in a larger data structure. Sometimes on struct initialization you need to execute some logic, so it's best off resting in a constructor method that is easy to use for all.

impl MyData {
  pub fn new(a: i16, b: u32) -> MyData {
    return MyData { a: a, b: b, c: 0.0, };
  }
}

While it's an improvement, long constructor method calls can still become a headache, and after some time, you may decide it's better to create a more method-chain-oriented way of creating structures. Instead of stuffing all your data into one function call, you can instead create several methods which pack and store the data for you, so it's less of an eye sore.

impl MyData {
  pub fn new(a: i16, b: u32) -> MyData { ... }

  pub fn empty() -> MyData {
    return MyData{ a: 0, b: 0, c: 0 };
  }

  pub fn set_a(mut self: MyData, a: i16) -> MyData {
    self.a = a;
    return self;
  }

  pub fn set_b(mut self: MyData, b: u32) -> MyData {
    self.b = b;
    return self;
  }
}

Which can be used in the following way at the initialization site:

let mystruct = MyData::empty()
    .set_a(5)
    .set_b(7);

Now it's more clear what you're setting, and the logic can be further restrained or better defined within these setter methods. That way, users cannot break any rules with your structs later on, especially if your values have to be within certain ranges.

The limitation with using self variables inside a struct implementation is that the self value must always de-reference to the struct type itself, and thus it is cumbersome to add additional abstract typing like with Option<T> or Result<T,E> to the implementation. Even if you did, it's tricky to unpack all the layers of pattern matching that this could incorporate without it being inside another Result<T, E> function to encapsulate the ? operator.

I am a big fan of method chaining for struct initialization, as it gives a good compromise between satisfying the invariants of the program and making it less clumsy to build structs from your cool source code library.

Now I will look at Racket, and I will show you why I think it's clumsy in it's current given state.

Structs in Racket

Racket is a Lisp/Scheme family language that enables you to do so much more than most other languages. It can be functional, it can be object-oriented, it can be used to design GUI applications, games, or even backend servers to serve web pages and other content. Racket, in my eyes, is a very cool language and I love talking about it.

However, one thing that has bothered me for a long time is how boring the structs are. Structs are sort of like that of C/C++, D, Zig or Rust, but not really. Since everything in Racket is a type of linked list, a struct is a way of defining a group of functions to create that linked list, and automatically create the functions needed to traverse said list.

(struct Point (x y z) #:transparent)

(define p1 (Point 3 4 5))
(displayln p1)
; shows #(struct:Point 3 4 5)

The magic behind the struct macro is that it generates a bunch of boilerplate methods alongside your layout.

Looking at the above example, the values that get generated for us are:

Point, the constructor
Point-x, the x value getting function
Point-y, the y access
Point-z, the z access
Point?, the predicate true/false function

Since a struct's underlying data is a linked list (or heck, it might not be a linked list but instead could be a hash map), creating the getter methods to get the fields is a trivial macro to implement. Same as the Point? function, all trivial macro implementations.

My issues with this generic struct macro implementation is that it suffers from the same thing as other struct language issues: once the fields become too many, initializing structs becomes more tedious than anything, as you have to remember the exact order of the values you want, making it rather difficult. Even more so difficult in a Lisp-like language.

(struct VeryLong (a b c d e f g ...))

(define my-data
  (VeryLong 1 2 3 4 5 6 7 ...))

In this example, because it's not required to have the field names when creating a struct, it gets a little hard to memorize exactly what each field is doing. Even if you named all the fields, when you create the struct, the field names aren't required, making it more unclear what you're really building. You would then have to look back at the source code to figure out what the fields you're filling out are.

Then we might consider building a smarter constructor to help us further constrain the program invariants, which is a smart thing to do. You can even use named fields with the smart constructor so you can provide default values and mix in things as you want.

(define (Builder #:value1 [v1 0]
                 #:value2 [v2 "hi"]
                 #:value3 [v3 '(1 2 3 4 5)])
  (SomeStruct v1 v2 v3))

(define mydata (Builder #:value1 500 #:value3 '(4 5))
; #(struct:SomeStruct 500 "hi" (4 5))

This is a smart solution for defining a better interface for your library, but the code can feel a bit like too much, as the list of keyword arguments can run on quite long the further your struct goes.

Let's look at the method-chaining path from Rust used earlier - say we were given some struct, and we wanted to copy it so that one field would be changed. There exists a macro to do this called struct-copy, which will copy a struct, and given a pair list of IDs and new values, it will modify those fields with the supplied values.

(struct Point (x y z) #:transparent)

(define p1 (Point 3 4 5))
(define p2
  (struct-copy Point p1 [x 7] [z 9]))

(displayln p2)
; #(struct:Point 7 4 9)

struct-copy is a macro that generates code to create a new struct from the old one. If we were to do this without struct-copy, it would look a little... odd, I guess.

(define p1 (Point 3 4 5))

; copy the original values without struct-copy
(define p2
  (Point (Point-x p1)
         (Point-y p1)
         (Point-z p1))

; use the old struct, but supply new values
(define p3
  (Point (Point-x p2) 8 (Point-z p2)))

struct-copy is a handy macro for this situation because copying struct information is a boring chore to do.

To enable some kind of method-chaining, we have to write some function that will copy the old struct state and copy it into a new one via struct-copy. This can be written as a plain function.

(define (change-x old-state new-value)
  (struct-copy Point old-state [x new-value]))

Great, now we can use this.

(define p1 (Point 3 4 5))
(define p2 (change-x p1 700))
; #(struct:Point 700 4 5)

... Frankly I'm unsure if this changed anything. If we continued to write more methods to satisfy the other fields:

(define (change-y old-state new-value)
  (struct-copy Point old-state [y new-value]))

(define (change-z old-state new-value)
  (struct-copy Point old-state [z new-value]))

It's still not super useful because data has to be stored somewhere in-between each call. You have to define a new binding each time you want to use these functions, or you have to chain them all together in one very large composition.

(define p3
  (change-z
    (change-x
      (change-y p2 700)
      800)
    900))
; #(struct:Point 800 700 900)

What a mess. It would be better if we could somehow compose this into a cleaner way of modifying old structs. Granted, if we reversed the positions of the new value and the old struct, it would look different like:

(define p3
 (change-z 900
  (change-x 800
   (change-y 700 p2))))

Still, maybe there's a better way.

Folding Functions over State

Our means of composing struct-copy functions isn't very pleasant in it's current form, but maybe there's a way of changing that. In this section I will introduce a way of folding functions over state.

Miniature glossary going forward:

a Function is a piece of code that accepts a value and returns some output
State is some value representing the state of a system, ie a number or a collection of numbers/values
Folding is the idea of "folding" a function over a list of values, by reducing them into a final value (like folding paper until there's no way to fold it any further)

What we have here is an example of functions and state: our structs are our state, and we want to create some functions that can copy the old state and return some kind of new, updated state.

Given a function f and some state s, we can "run" a function by giving the state to the function. We can define this as the following:

(define (run-state f s)
  (f s))

It almost seems redundant, but it will become important in a bit.

The next part is we need to re-write our custom struct modifying methods. Instead of writing a function that accepts some state and a new value, let's change it to curried form where it accepts a value, then yields a function that then accepts some state, then applies struct-copy.

(define (change-x new-value)
  (lambda (old-state)
    (struct-copy Point old-state [x new-value])))

What this gives us is not a function that directly manipulates state, but instead gives us a function that will later manipulate state. It's a little different, but I think it's cool since it takes advantage of currying.

(define set-x-500 (change-x 500))
(define p1 (Point 3 4 5))
(define p2 (set-x-500 p1))
;#(struct:Point 500 4 5)

We can do the same for the other fields as well.

(define (change-y new-value)
  (lambda (old-state)
    (struct-copy Point old-state [y new-value])))

(define (change-z new-value)
  (lambda (old-state)
    (struct-copy Point old-state [z new-value])))

Now we're getting somewhere. We have an indirect method for creating setter functions on a given struct. Now we need a way to glue this all together. In it's current form, it would still look like this:

(define p1 (Point 3 4 5))

(define p2
  ((change-x 500)
   ((change-y 300) p1)))

Little weird, but we can re-write this using a folding function.

Just for an example, a fold is a way of reducing a list of values by supplying some initial value as the start, and we have a "reducer" function repeatedly apply to the values as it traverses a list.

(foldl + 0 '(1 2 3 4 5))
; 15
; same as
; (+ 5 (+ 4 (+ 3 (+ 2 (+ 1 0)))))

It starts with a call that looks like (+ 0 1), which is the first value of the list with the initial value given to the fold function (0 in this case). The way it works is that it uses a two-argument function to take in some value and apply a function with some "accumulated" value.

This function is what I will call the "reducer", as it takes in all these values and tries to reach some final value after a series of calls.

(define (reducer value acc)
  (+ value acc))

(foldl reducer 0 '(1 2 3 4 5))
; 15

Let's re-think this a bit: does the list of values have to be some list of atomic values like numbers? No! It can be anything, because Racket is a dynamic language that treats all values as just ordinary citizens. This means that list of values can actually be a list of functions, too!

; from earlier
(define (run-state f s)
  (f s))

; use the `add1` function 3 times
(foldl run-state 0 (list add1 add1 add1))
; 3

Since we changed our struct modifying functions into curried form, they return a function that changes some state, meaning we can actually re-write all of our struct-copy wrapper functions into a foldl function call.

(foldl run-state (Point 0 0 0)
  (list
    (change-x 500)
    (change-y 600)
    (change-z 700)))

; #(struct:Point 500 600 700)

That almost looks like method-chaining from Rust! To simply leave some fields as zero, you can take them out from the given list entirely, or you can put as many of them as you want. I wouldn't say this is ground-breaking stuff, but it certainly opens the doors for much cleaner and precise code that your users (and you) might fall in love with.

One thing that's lacking here is a better way of creating empty structs, which we did in the Rust section, so I'll fill that in now.

(define (init-point)
  (Point 0 0 0))

(foldl run-state (init-point)
  (list
    (change-x 500)
    (change-y 600)
    (change-z 700)))

Now you don't have to initialize a struct yourself and can just use the plain initializer to set it to some default struct.

At this point, we have created a unique way of defining our structs, and this can be helpful in creating libraries and constraining the data in the ways you need. But there's one last thing that I want to go over real quickly: creating a macro.

Macro'ing it All Together

A macro is a way of generating code in Racket, and most other languages. While regular functions operate on data, a macro operates on the actual code you write. A macro returns a list of code that is to then be evaluated by the Racket VM.

Macros simplify programming, and languages that support writing macros are fascinating because of all the ways you can factor out things you don't like writing, simplifying things and creating your own miniature domain-specific language for a problem.

I am not going to make it more complex, but right now, if you wanted to define a struct using our new functions, it's not exactly pleasant to have to manually write out the foldl call yourself, nor would it be fun to make your users do that each time.

Instead, it might be user to write a function that can do it for you.

(define (make-point . funs)
  (foldl run-state (init-point) funs))

(define p1
  (make-point
    (change-x 500)
    (change-y 600)))
; #(struct:Point 500 600 0)

That's better, and thanks to the way Racket functions can handle multiple values, the function list is binded to the funs argument, which is supplied to foldl easily.

Even still, looking at this it's still fairly exhausting. My criticism is that it's still too verbose. You know you want to create a point, so why not just get straight to the point? This is where a macro will kick in and help us to reduce the code.

To give us a brief primer on macros, let's just write a simple macro that will double a value by quite literally "doubling" it. To define macros we use define-syntax-rule.

(define-syntax-rule (double x)
  '(x x))

; (double 5)
; '(5 5)
; (double "hello")
; '("hello" "hello")
; (double it)
; '(it it)

Racket takes this rule and performs a language transformation by substitution. Whatever the value of x is in this macro is substituted with whatever we gave to the macro. It doesn't do variable expansion at this point, because it's just performing a literal translation. When we supply it, it doesn't look for any kind of variable binding for it, it turns it into a symbol due to the literal translation.

When writing macros, all you do is return plain Racket code. The code won't be evaluated until later, so we can use it to automatically generate code for us.

Let's look at a macro that handles many values and treats it as a list. This will involve a special name binding known as the ellipsis, or ..., which gives us a way of binding many values to one binding in the macro space. We can translate code into a begin sequence by simply attaching the list of code to the original begin macro.

(define-syntax-rule (my-begin code ...)
  (begin code ...))

(my-begin
  (displayln 1)
  (displayln 2))
; 1
; 2

Not very complex, but it acts as a way of translating code from one form to another. At this level, the define-syntax-rule macro is one of the simpler ways of writing macros, and it only lets us get so far, but it's good enough for right now.

My goal is to create a macro that will do the above (define (make-point ...) code for us, so we don't have to always do it that way. It seems redundant, so maybe let's write a macro to make it a bit easier.

(define-syntax-rule (define-point name code ...)
  (define name
    (foldl run-state (init-point) (list code ...))))

So instead of doing:

(define p1
  (make-point
    (change-x 500)
    (change-y 600)))

We can now do:

(define-point p1
  (change-x 500)
  (change-y 600))

By taking advantage of the ellipsis, we can translate a list of syntax into a list of code to be used by the foldl function, and supplying some of the basic values to it like run-state and init-point, it acts as a nice and clean way of creating our Point struct.

It saves a few lines of code, but the name of the macro makes it clear you are in fact trying to define a new Point struct, and supplying the variable name gives it to the original define macro, so really it's a nice way of skipping a few steps and making it easier to do whatever your library is trying to do.

Conclusion

Structs are great, but the more complex your struct gets, the harder it is to maintain the code surrounding it. Even adding one field to the struct can mean tons of changes to all the original call-sites.

Smart constructors and method-chaining are good ideas for offsetting the amount of tedious labor you would have to do to provide the correct changes. On top of that, method-chaining is a nice and interactive way of building up your data.

For Racket, these approaches aren't baked in, but it doesn't mean impossible. The magic of macros makes it easy to fill in the blanks and create a more optimized experience for yourself and your users. I have been employing this folding-function method for a little bit now and thought to share it with all.

Thanks for reading!

(Header image credit @jannerboy62 on Unsplash)