DEV Community: Ryosuke

Making Galaga in Rust with Bevy - Part 1

Ryosuke — Fri, 10 Feb 2023 00:59:21 +0000

I took some time recently building a Galaga clone to wrap my head around Bevy, a game engine for Rust. I documented my process of going from a blank screen to a whole game. It only took a half a day (mostly co-writing this article) and was pretty fun. I thought I’d share it as a blog post since it’s a pretty new library and new users could always use an extra hand.

I used the Breakout clone from the Bevy examples as a basis for a lot of this logic, and then branched off where needed for Galaga-like features. In this tutorial I’ll show you how to work with 2D sprites and meshes, setup custom shaders to make an animated space background, and how to add sound to your game.

You can find the complete game code on Github here.

What’s Bevy?

Bevy is a game engine framework used with Rust. It’s runs on wgpu, which is a Rust implementation of WebGPU, which basically means Bevy apps run not only natively — but on the web! It’s a standard low-level game engine, expect to work with an ECS system to compose your 2D or 3D scene. Bevy also has a few systems in place for things like primitive shapes, handling images/audio, and other nice utilities.

In this tutorial we’ll explore quite a few Bevy features, but you can also read the Bevy “book” or browse through the official Bevy examples to get a bigger and better picture. I also recommend reading the Bevy Cheatbook for more insight into specific features.

How does Bevy work?

Bevy is a low level game engine that uses an ECS architecture to structure it’s scenes. There are a few concepts you’ll hear me say and use frequently — like entities, systems, resources, etc. It’ll become more clear how all these pieces work individually and together as you work with them.

You’ll find yourself creating a lot of “systems” that get added to your Bevy app. Systems are Rust functions that run each frame, usually responsible for logic like moving your character or spawning enemies.

You can use “resources” to manage assets and state in your game. From media like textures and sounds - to data like game state (e.g. “is game paused?”) — you’ll use a lot of resources in Bevy. Bevy exposes a bit of it’s logic through resources, like keyboard and gamepad input (which get added by “plugins”…).

You can organize multiple systems and resources together into a “plugin”. It basically just groups together logic into a single module. It might make sense to make a GameAudio plugin that handles all the audio logic in a game. It also makes it easier to enable/disable game features. Bevy provides a few plugins out of the box, like a Window plugin for managing a native window, or an Input plugin for adding a resource to your game with synced user input.

We’ll also end up using “events” which are exactly what they sound like. It’s a way for systems to communicate with other systems using specific events, like a NewGameEvent event could play intro music and spawn enemies.

“Magic” Function Parameters

The magic of Bevy systems is that you can create function parameters for anything you need, and Bevy will automatically provide the system the necessary data.

For example, if you want to access a “resource”, you’d create a system that has a function parameter with the type Res<YourResourceHere>. In this code snippet, we ask for the Input and GameState resources separately.

fn move_player(
    keyboard_input: Res<Input<KeyCode>>,
    game_state: Res<GameState>,
) { }

You can learn more about this in the Bevy book.

Here’s an example of a system and the different parameters it can use.

Galaga time

Now that we have a rough understanding of Bevy, let’s make a game!

Project Setup

Made a new project using cargo new.

Installed bevy cargo add bevy.

Tried running the build — failed requiring Rust Compiler v1.6.4 or higher. I had 1.6.3. Went to Rust site, downloaded binary, installed — build ran successfully.

Added default Bevy plugins that create the window, adds keyboard input, and other core Bevy goodies. You can find a list of the default plugins here.

fn main() {
    App::new()
        .add_plugins(DefaultPlugins)

Now we have a blank window! Let’s keep going!

Roughing it out

First we need to create a system to “setup” our game. It’ll initialize any data, load any assets, and spawn any necessary stuff (like the player or enemies). I added the Commands to the parameter list to allow us to commands.spawn() our player and other “entities” in our ECS system.

fn main() {
    App::new()
        .add_plugins(DefaultPlugins)
        .add_startup_system(setup_game)
                .run();
}

// ...

fn setup_game(
    mut commands: Commands
) {
}

Spawned a 2D camera entity into the scene to start using 2D elements. This turned the blank window into a gray screen (likely from the renderer’s “clear color”).

// Camera
commands.spawn(Camera2dBundle::default());

Creating a “player” sprite

Time to create our player! We use Commands to spawn a player “sprite” (basically 2D object). This also required making a simple struct for the “components” for our Player and Collider. This allows us to “query” the ECS for each later, like finding the current player position, or checking for collisions between objects. If we didn’t add these components, we’d have to query for all the entities in a scene and create some logic to discern which is which (much messier/buggy).

// The Player object
#[derive(Component)]
struct Player;
// Signifies an object is collidable
#[derive(Component)]
struct Collider;

const PADDLE_SIZE: Vec3 = Vec3::new(120.0, 20.0, 0.0);
const PADDLE_COLOR: Color = Color::rgb(0.3, 0.3, 0.7);

fn setup_game(
    mut commands: Commands
) {
    // Spawn Player in initial position
    commands.spawn((
        SpriteBundle {
            transform: Transform {
                translation: Vec3::new(0.0, 0.0, 0.0),
                scale: PADDLE_SIZE,
                ..default()
            },
            sprite: Sprite {
                color: PADDLE_COLOR,
                ..default()
            },
            ..default()
        },
        Player,
        Collider,
    ));
}

Added a “set” of systems. This will allow us to run a sequence of systems (like our player input, or enemy spawning) - but also have it respect a certain order of operations using the .before() method. Later we’ll add a collision system, and we’ll want all of our systems to wait for this once to finish processing each frame before they run (so the player doesn’t run through a wall - the app checks for the collision and limits player movement if necessary).

fn main() {
    App::new()
        .add_plugins(DefaultPlugins)
        .add_startup_system(setup_game)
        .add_system_set(
            SystemSet::new()
                // .with_system(check_for_collisions)
                // .with_system(move_player.before(check_for_collisions))
                .with_system(move_player),
        )
        .run();
}

The first system to add is a player input (aka “move the player using keyboard presses”). We can create a new system called move_player and grab the Input ”resource”, and specifically the KeyCode struct for keyboards. Then we can use the .pressed() command on the resource to check for specific keys (using the KeyCode struct).

fn move_player(
    keyboard_input: Res<Input<KeyCode>>
) {
    if keyboard_input.pressed(KeyCode::Left) {
        println!("[KEYBOARD] Pressed left");
    }

    if keyboard_input.pressed(KeyCode::Right) {
        println!("[KEYBOARD] Pressed right");
    }
}

Now lets actually move the player based on the input. In Bevy’s systems you can Query<> for a mutable Transform (aka position) of the Player. Using that query, we can mutate the value to our new position.

Movement is a simple and clever calculation. We think of movement on a 2D graph using X/Y coordinates, which means moving left would make numbers negative — or right positive. Using this as the basis (left is negative, right is positive), we set a direction variable with a number that -1 or 1, then multiply that by how fast our player should move (PLAYER_SPEED). We’ll also multiply by the frame rate of the app (TIME_STEP) since the update method runs 60 times a second. Not sure if this is super necessary, cause you can just adjust the other values (like player speed) - but we’ll use this frame rate later to power a timer to keep things more consistent.

// Defines the amount of time that should elapse between each physics step
// in this case, 60fps
const TIME_STEP: f32 = 1.0 / 60.0;
const PLAYER_SPEED: f32 = 100.0;

fn move_player(
    keyboard_input: Res<Input<KeyCode>>,
    mut query: Query<&mut Transform, With<Player>>,
) {
    let mut paddle_transform = query.single_mut();
    let mut direction = 0.0;

    if keyboard_input.pressed(KeyCode::Left) {
        println!("[KEYBOARD] Pressed left");
        direction -= 1.0;
    }

    if keyboard_input.pressed(KeyCode::Right) {
        println!("[KEYBOARD] Pressed right");
        direction += 1.0;
    }

    // Calculate the new horizontal paddle position based on player input
    let new_paddle_position = paddle_transform.translation.x + direction * PLAYER_SPEED * TIME_STEP;

    paddle_transform.translation.x = new_paddle_position;
}

Now let’s make the player shoot projectiles. First we’ll make a system to handle detecting the shoot button (spacebar in this case), and spawn a projectile at the player’s current position. It’s a lot of copy/paste/modify from earlier parts - like detecting input or spawning a mesh.

// Above (truncated)
.with_system(shoot_projectile)

// Below
fn shoot_projectile(
    mut commands: Commands,
    mut meshes: ResMut<Assets<Mesh>>,
    mut materials: ResMut<Assets<ColorMaterial>>,
    keyboard_input: Res<Input<KeyCode>>,
    mut query: Query<&Transform, With<Player>>,
) {
    let player_transform = query.single_mut();

    if keyboard_input.pressed(KeyCode::Space) {
        // Spawn projectile
        commands.spawn((
            MaterialMesh2dBundle {
                mesh: meshes.add(shape::Circle::default().into()).into(),
                material: materials.add(ColorMaterial::from(PROJECTILE_COLOR)),
                transform: Transform::from_translation(player_transform.translation)
                    .with_scale(PROJECTILE_SIZE),
                ..default()
            },
            Projectile,
            Velocity(INITIAL_PROJECTILE_DIRECTION.normalize() * PROJECTILE_SPEED),
        ));
    }
}

Then we can make the projectiles move! Any projectile on the screen should be flying. Similar to above, we’ll use previous code and modify it a bit to get what we need.

// Above (truncated)
.with_system(move_projectiles)

// Below
fn move_projectiles(mut query: Query<&mut Transform, With<Projectile>>) {
    for mut collider_transform in &mut query {
        // Calculate the new horizontal player position based on player input
        let new_projectile_position = collider_transform.translation.y + 250.0 * TIME_STEP;
        // TODO: make sure player doesn't exceed bounds of game area

        collider_transform.translation.y = new_projectile_position;
    }
}

Couple things we need to handle. Projectiles should despawn once they’re off screen. The top of the screen is around 350, so we check if the collider’s translation (aka position) is greater than that. If it is, we “despawn” the “entity” (aka the projectile and all it’s components like the Collider) to remove the projectile from the game.

fn destroy_projectiles(
    mut commands: Commands,
    query: Query<(Entity, &Transform), With<Projectile>>,
) {
    for (collider_entity, collider_transform) in &query {
        if collider_transform.translation.y > 350.0 {
            commands.entity(collider_entity).despawn();
        }
    }
}

This all works - but if you noticed, we fire like…60 bullets per second 😅 To prevent this, let’s create a timer to keep track of the “reload” time of the weapon. This way, the user can only fire again once the timer has completed. It’ll be a short interval (0.3 seconds).

First let’s create that timer globally. We create a struct to represent our timer “resource”, and it’ll contain the actual timer as it’s first tuple property. Then we can insert the resource into our app, and initialize the timer immediately. We have control over the duration of the timer (0.3 seconds in this case) and if the timer repeats or not. Since we want the user to control the timing of their shot, we’ll do it Once — so we can reset each time another shot is fired.

// Timer used to limit player shooting every frame per second
#[derive(Resource)]
struct ProjectileTimer(Timer);

fn main() {
    App::new()
        .add_plugins(DefaultPlugins)
        .insert_resource(ProjectileTimer(Timer::from_seconds(0.3, TimerMode::Once)))

Let’s update the shoot_projectiles function to include a time and projectile_time parameter. Bevy gives you access to a Time resource (I’m assuming from CorePlugin), which is a constantly running timer - so we can keep track of total time elapsed in our game.

Then we have to increment the ProjectileTimer timer using the time.delta() (or time that passed between last “render”). If we don’t, the timer will never run, and logging out projectile_timer.elapsed() will be 0.

Finally, we can check if the projectile timer is finished using the finished() method. If it is, we can spawn our projectile. But before we do, we also reset the timer, to prevent the user from shooting again immediately.

fn shoot_projectile(
    time: Res<Time>,
    mut projectile_timer: ResMut<ProjectileTimer>,
    mut commands: Commands,
    mut meshes: ResMut<Assets<Mesh>>,
    mut materials: ResMut<Assets<ColorMaterial>>,
    keyboard_input: Res<Input<KeyCode>>,
    mut query: Query<&Transform, With<Player>>,
) {
    let player_transform = query.single_mut();

    if keyboard_input.pressed(KeyCode::Space) {
        // Check if player is allowed to shoot based on internal timer
        // We have to "tick" the timer to update it with the latest time
        if projectile_timer.0.tick(time.delta()).finished() {
            // Reset the timer
            projectile_timer.0.reset();

Now we can start setting up some “enemies” we can shoot at them with our projectiles. We’ll need to create a new struct to represent our enemies (Enemy), and we’ll change the startup code to spawn an entity with a Enemy and Collider components attached.

// The Enemy object
#[derive(Component)]
struct Enemy;

fn setup_game(
    mut commands: Commands,
    mut meshes: ResMut<Assets<Mesh>>,
    mut materials: ResMut<Assets<ColorMaterial>>,
    asset_server: Res<AssetServer>,
) {
    // Other stuff

    // Spawn enemies
    commands.spawn((
        MaterialMesh2dBundle {
            mesh: meshes.add(shape::Circle::default().into()).into(),
            material: materials.add(ColorMaterial::from(PROJECTILE_COLOR)),
            transform: Transform::from_translation(PROJECTILE_STARTING_POSITION)
                .with_scale(PROJECTILE_SIZE * Vec3::new(2.0, 2.0, 2.0)),
            ..default()
        },
        Enemy,
        Collider,
    ));
}

So we have enemies, we have our player shooting projectiles — let’s check if any of the projectiles hit (or “collide) any enemies!

We create a new system called check_for_collisions that loops through all the projectiles in a scene, then for each bullet it checks each object in the scene if they’ve collided (using a nice collide() method from Bevy’s 2D system).

Now this trick is nice - since Bevy can query entities for their components, we can also check for components attached to the entity using the Option<> enum. In this case, we check if the entity with the Collider is also an enemy (Option<Enemy>), and if it is, we destroy the enemy by de-spawning it’s associated entity.

fn main() {
    App::new()
        .add_plugins(DefaultPlugins)
        .insert_resource(ProjectileTimer(Timer::from_seconds(
            PROJECTILE_TIME_LIMIT,
            TimerMode::Once,
        )))
        .add_startup_system(setup_game)
        .add_system_set(
            SystemSet::new()
                .with_run_criteria(FixedTimestep::step(TIME_STEP as f64))
                .with_system(check_for_collisions)
                .with_system(move_player.before(check_for_collisions))
                .with_system(move_projectiles.before(check_for_collisions))
                .with_system(destroy_projectiles.before(check_for_collisions))
                .with_system(shoot_projectile.before(check_for_collisions)),
        )
        .add_system(bevy::window::close_on_esc)
        .run();
}

fn check_for_collisions(
    mut commands: Commands,
    projectiles_query: Query<(Entity, &Transform), With<Projectile>>,
    collider_query: Query<(Entity, &Transform, Option<&Enemy>), With<Collider>>,
) {
    // Loop through all the projectiles on screen
    for (projectile_entity, projectile_transform) in &projectiles_query {
        // Loop through all collidable elements on the screen
        // TODO: Figure out how to flatten this - 2 for loops no bueno
        for (collider_entity, collider_transform, enemy_check) in &collider_query {
            let collision = collide(
                projectile_transform.translation,
                projectile_transform.scale.truncate(),
                collider_transform.translation,
                collider_transform.scale.truncate(),
            );

            if let Some(collision) = collision {
                // If it's an enemy, destroy!
                if enemy_check.is_some() {
                    println!("Collided!");

                    // Enemy is destroyed
                    commands.entity(collider_entity).despawn();

                    // Projectile disappears too? Prevents "cutting through" a line of enemies all at once
                    commands.entity(projectile_entity).despawn();
                }
            }
        }
    }
}

📘 You’ll also notice I also added a FixedTimestep to our system set. This ensures it runs 60fps using the TIME_STEP constant we created earlier. This is an optional step.

Now let’s style things a bit so it looks less like Breakout and more like Galaga. I downloaded some Galaga sprites from here, took them into Figma and cut out the player ship, a projectile, and an enemy. Each were 15x16 pixels. Or you could download them here on Github.

Now that we have PNGs, we can apply them to our existing Sprite entities. First, let’s create an assets folder in the project root. Bevy will start from that folder when we load things later. Then we can add our PNGs to the folder - I created a folder called sprites to keep things organized.

We’ll use the AssetServer to load the PNG and use it as the sprite’s texture property. You can see a simple example here on the Bevy site. We’ll also remove the weird scaling we did to make it a rectangle. And since our sprite is so small, we’ll scale it to 200% it’s original size by using a Vec3(2.0, 2.0, 0.0).

const PLAYER_SIZE: Vec3 = Vec3::new(2.0, 2.0, 0.0);

fn setup_game(
    mut commands: Commands,
    mut meshes: ResMut<Assets<Mesh>>,
    mut materials: ResMut<Assets<ColorMaterial>>,
    asset_server: Res<AssetServer>,
) {
    // Camera
    commands.spawn(Camera2dBundle::default());

    // Spawn Player in initial position
    commands.spawn((
        SpriteBundle {
                        // 👇 Add your texture here
            texture: asset_server.load("sprites/player_default.png"),
            transform: Transform {
                translation: PLAYER_STARTING_POSITION,
                scale: PLAYER_SIZE,
                ..default()
            },
            sprite: Sprite {
                // color: PLAYER_COLOR,
                ..default()
            },
            ..default()
        },
        Player,
        Collider,
    ));

Now we can replace the placeholder MaterialMesh2dBundle for the enemy and projectile with PNG textures.

Backgrounds

The background is looking a bit plain. In Galaga, it’s a tiled space texture that scrolls to simulate the sense of flying through space.

There’s a few ways we can do it. If we tried to imitate what we did for the enemies or player ships by creating “sprites”, we’d have to spawn enough sprites to cover the entire screen — then also mutate each position each frame using a system.

To do this more efficiently, we’ll implement this logic inside a fragment shader. A fragment shader is part of the graphic rendering pipeline and is responsible for “painting” the pixels down on the screen based on instruction from the previous shaders (like vertex). Since shaders run on the GPU, they’re better suited for simple (and complex) math operations — like iterating over a screen and calculating the color of each pixel.

In Bevy, we can add a “shader” in a few ways. We could apply a shader to the entire pipeline, and change the way the entire scene is rendered. This would be good for post processing, like rendering 100 objects - then blurring all of them at the same time.

Or you could apply a “custom material” to a “mesh” in your scene. This allows us to apply a shader to a specific object, like a cube or 2D shape.

Let’s create a custom material. You’ll notice we use Material2d instead of Material because we’re rendering a 2D scene (not 3D). We also have to give each material a UUID using a macro. And inside the material struct, we add a uniform property for Color and a texture (aka optional Image Handle). Bevy seems to handle generating the sampler for us under the hood, so we only need to pass the image data.

impl Material2d for CustomMaterial {
    fn fragment_shader() -> ShaderRef {
        "shaders/custom_material.wgsl".into()
    }
}

// Background shader material
#[derive(AsBindGroup, TypeUuid, Debug, Clone)]
#[uuid = "f690fdae-d598-45ab-8225-97e2a3f056e0"]
pub struct CustomMaterial {
    #[uniform(0)]
    color: Color,
    #[texture(1)]
    #[sampler(2)]
    color_texture: Option<Handle<Image>>,
}

Create a new file at /assets/shaders/custom_material.wgsl. We define our custom material at the top, and grab our uniforms we defined (in this case - only color). Our texture is available through bind groups, and Bevy’s 2D pipeline puts the texture in the first slot (@binding(1)), and sampler in second slot (@binding(2)).

📘 If you try to change these bind group indexes you’ll get an error about bind groups not matching up — this is because of how render pipelines work. Bevy defines a specific pipeline configuration for 2D elements, and it assumes the texture will always be bound to slots 1 and 2. If you want to add additional bind groups, you’ll need to customize the render pipeline (example here).

struct CustomMaterial {
    color: vec4<f32>,
};

@group(1) @binding(0)
var<uniform> material: CustomMaterial;
@group(1) @binding(1)
var base_color_texture: texture_2d<f32>;
@group(1) @binding(2)
var base_color_sampler: sampler;

@fragment
fn fragment(
    #import bevy_pbr::mesh_vertex_output
) -> @location(0) vec4<f32> {
    return textureSample(base_color_texture, base_color_sampler, uv);
}

In the setup, let’s add a new mesh and apply a custom texture to it. The mesh is a created from a Quad (aka a rectangle) that we scale to the screen size using the transform property. The Vec3::splat() method just sets the X and Y values at the same time. We’ll use an existing texture (the player ship) for now as a test.

// Background
commands.spawn(MaterialMesh2dBundle {
    // mesh: meshes.add(shape::Plane { size: 3.0 }.into()).into(),
    mesh: meshes.add(Mesh::from(shape::Quad::default())).into(),
    transform: Transform::default().with_scale(Vec3::splat(SCREEN_EDGE_VERTICAL * 2.0)),
    // material: materials.add(ColorMaterial::from(Color::TURQUOISE)),
    material: materials.add(CustomMaterial {
        color: Color::BLUE,
        color_texture: Some(asset_server.load("sprites/player_default.png")),
    }),
    ..default()
});

📘 I left a commented out Plane mesh that also works, if you were trying to use different kinds of shapes. The syntax was really weird (.into() then another into()) — had to find this in another Bevy example somewhere. I also left a ColorMaterial there if you wanted to debug the mesh without the custom material.

Change the function signature of our setup_game() to use our CustomMaterial instead of ColorMaterial. If we don’t do this, the project won’t compile and it’ll complain about inserting a different material type into the materials stack.

fn setup_game(
    mut commands: Commands,
    mut meshes: ResMut<Assets<Mesh>>,
    mut materials: ResMut<Assets<CustomMaterial>>,
    asset_server: Res<AssetServer>,
) {

Go to where we define our app and let’s add a new plugin that initializes our shader (likely loading and initializing the shader and any textures).

fn main() {
    App::new()
        .add_plugins(DefaultPlugins)
        .add_plugin(Material2dPlugin::<CustomMaterial>::default())

📘 If you don’t add this, your app will fail with an obscure error about the Asset Server not finding the CustomMaterial. It might seem like the image or shader isn’t found on disk, but you can remove them and the error should still show.

Cool! Now we should be seeing our texture centered on the screen.

Now that we have the texture displaying, let’s tile it using the shader. Right now the image is being stretched across the entire screen because the mesh is that large — which also means the “UVs” of the mesh are scaled that size. So how do we tile the image?

The first thought is to divide or multiply the UVs. But this doesn’t work. Our image only shows once, gets scaled down, and the last pixels get stretched to the size of our mesh.

Explaining UV Math

What’s the problem here? Well let’s break down how UVs work for a second. UVs are usually 2D vectors (aka X and Y values) that go from 0 to 1. The top left of an image is (0,0) and the bottom right is (1,1). Our texture data is mapped assuming this, When we multiply the UV by say 4.0, we’re telling it to go from (0,4) now.

Similarly when we divide, we change the range from (0,1) to (0,0.25) . That’s what causes the image to look stretched, we’re only seeing the top left quarter corner of it.

So how do we get our shader to “repeat” the UV data? The fract() function. You can see an basic interactive example here. By using the fract() function, we force the number when it exceeds 1.0 to reset back to 0 and increment up again.

@fragment
fn fragment(
    #import bevy_pbr::mesh_vertex_output
) -> @location(0) vec4<f32> {
    var tiled_uv: vec2<f32>;
    tiled_uv = fract(uv * 10.0);
    return textureSample(base_color_texture, base_color_sampler, tiled_uv);
}

Now it should tile the image 10 times horizontally and vertically.

Let’s quickly update the size of the mesh to fit the width of the screen too.

// Background
commands.spawn(MaterialMesh2dBundle {
    // mesh: meshes.add(shape::Plane { size: 3.0 }.into()).into(),
    mesh: meshes.add(Mesh::from(shape::Quad::default())).into(),
    transform: Transform::default().with_scale(Vec3::new(
        1300.0,
        SCREEN_EDGE_VERTICAL * 2.0,
        0.0,
    )),
    // material: materials.add(ColorMaterial::from(Color::TURQUOISE)),
    material: materials.add(CustomMaterial {
        color: Color::BLUE,
        color_texture: Some(asset_server.load("sprites/player_default.png")),
    }),
    ..default()
});

📘 Ideally we should use the window size from the Window resource, but we’ll hardcode stuff for now for simplicity.

And one last thing about the tiling — it’s stretching the images.

If we separate the UV’s X and Y properties, we can scale them independently, then re-combine them. We’ll make sure to stretch the X more than Y to match the screen ratio. There’s definitely better math for this, we’re just doing this hardcoded for now.

@fragment
fn fragment(
    #import bevy_pbr::mesh_vertex_output
) -> @location(0) vec4<f32> {
    var tiled_uv: vec2<f32>;
    var tiled_uv_x: f32;
    var tiled_uv_y: f32;
    tiled_uv_x = fract(uv.x * 20.0);
    tiled_uv_y = fract(uv.y * 15.0);
    tiled_uv = vec2(tiled_uv_x,tiled_uv_y);
    return textureSample(base_color_texture, base_color_sampler, tiled_uv);
}

Now things look better.

Let’s swap the BG texture with a real space texture. I went over to OpenGameArt and found a seamless space background texture. I copied it to the assets folder, then swapped the reference in the custom material.

color_texture: Some(asset_server.load("textures/space/space.png")),

Now we have a space texture that covers the entire screen!

But you’ll notice another problem — the player, enemy, and projectiles no longer show! I think this is because we’re a mesh for the background, and sprites for everything else. You can try changing the “render order” of different functions in the Bevy systems, but it’ll render with the mesh on top.

If we change the Sprites to Meshes - they start to appear more consistently.

// Spawn Player in initial position
commands.spawn((
    MaterialMesh2dBundle {
        // mesh: meshes.add(shape::Plane { size: 3.0 }.into()).into(),
        mesh: meshes.add(Mesh::from(shape::Quad::default())).into(),
        transform: Transform {
            translation: PLAYER_STARTING_POSITION,
            scale: PLAYER_SIZE,
            ..default()
        },
        material: materials.add(CustomMaterial {
            color: Color::BLUE,
            color_texture: Some(asset_server.load("sprites/player_default.png")),
            tile: 0.0,
        }),
        ..default()
    },
    Player,
    Collider,
));

// Spawn enemies
commands.spawn((
    MaterialMesh2dBundle {
        // mesh: meshes.add(shape::Plane { size: 3.0 }.into()).into(),
        mesh: meshes.add(Mesh::from(shape::Quad::default())).into(),
        transform: Transform {
            translation: PROJECTILE_STARTING_POSITION,
            scale: PLAYER_SIZE,
            ..default()
        },
        material: materials.add(CustomMaterial {
            color: Color::BLUE,
            color_texture: Some(asset_server.load("sprites/enemy_green_bug.png")),
            tile: 0.0,
        }),
        ..default()
    },
    Enemy,
    Collider,
));

But now we have a new problem - our asset server only supports 1 material type. And our material is using our shader that tiles images. So by applying it to our Player - we’ll see our player 20x15 times instead of once at the right size.

Instead of adding a new material (which I couldn’t quickly find out how to do for the life of me — I assumed enum but nope) — we’ll modify the existing shader to enable or disable the tiling based on a shader uniform we pass.

To add a new uniform, go back to our custom material and add a new property under our color. We need to use the uniform() macro to define the bind group, in this case, we want the same one as our color. We also need to define a data type - in this case f32. I originally tried to use a bool - but the shader pipeline refused it, so we use a float instead and assume 0 = false and 1 = true.

// Background shader material
#[derive(AsBindGroup, TypeUuid, Debug, Clone)]
#[uuid = "f690fdae-d598-45ab-8225-97e2a3f056e0"]
pub struct CustomMaterial {
    #[uniform(0)]
    color: Color,
    #[uniform(0)]
    tile: f32,
    #[texture(1)]
    #[sampler(2)]
    color_texture: Option<Handle<Image>>,
}

In our shader file we need to update a couple of things. The CustomMaterial type at the type needs to have our new tile property added to it. Then to access that property, we use the material variable (which grabs the data from the bind group we assigned earlier — at index 0). Then we can do a quick conditional check if the tiling is enabled or not.

struct CustomMaterial {
    color: vec4<f32>,
    tile: f32,
};

@group(1) @binding(0)
var<uniform> material: CustomMaterial;
@group(1) @binding(1)
var base_color_texture: texture_2d<f32>;
@group(1) @binding(2)
var base_color_sampler: sampler;

@fragment
fn fragment(
    #import bevy_pbr::mesh_vertex_output
) -> @location(0) vec4<f32> {
    var tiled_uv: vec2<f32>;
    // Default to the mesh UV
    tiled_uv = uv;
    // If tiling is enabled, redefine the UVs
    if(material.tile > 0.0) {
        var tiled_uv_x: f32;
        var tiled_uv_y: f32;
        tiled_uv_x = fract(uv.x * 10.0);
        tiled_uv_y = fract(uv.y * 7.0);
        tiled_uv = vec2(tiled_uv_x,tiled_uv_y);
    }
    return textureSample(base_color_texture, base_color_sampler, tiled_uv);
}

Now everything should look a little better now.

But we still have one issue - our objects are flickering now.

This looked a little like z-fighting, I’m assuming between the background and elements on top. I tried changing the translation of the background mesh and making the Z negative — but that didn’t render in some cases (only the clear color) or just didn’t do anything different.

I popped onto the Bevy Discord and asked the community and they quickly confirmed it was a z-fighting issue, but I had to increment to positive Z values instead. I made the Z value for the character and enemy 1.0. This stopped the flickering! 🙌

const PLAYER_STARTING_POSITION: Vec3 = Vec3::new(0.0, -300.0, 1.0);
const PROJECTILE_STARTING_POSITION: Vec3 = Vec3::new(0.0, 20.0, 1.0);

📘 Normally in game engines, you define “layers” for rendering, like a layer for your scene and another layer for UI. In this case, we could do a rudimentary version of it by defining constants at the top for different layer presets and use them as the Z value when needed. For example LAYER_UI could be 3.0 so it sits on top of everything. I don’t think Bevy has a layer system implemented yet, but you can see some progress on that feature in this Github issue.

Animated scrolling background

The background is missing one thing that’ll really sell it - it needs to scroll down like Galaga or most vertical “shmups” (or shoot-em-ups).

To animate anything with shaders, your shaders need to have a sense of time. We’ll create a new uniform for our material called time, which will be a f32 representing time in seconds (e.g 0.0 to infinity). This will let us animated things using math. As the time number changes and gets bigger, we can use it in our shader algorithms to drive movement.

// Background shader material
#[derive(AsBindGroup, TypeUuid, Debug, Clone)]
#[uuid = "f690fdae-d598-45ab-8225-97e2a3f056e0"]
pub struct CustomMaterial {
    #[uniform(0)]
    color: Color,
    // Should we tile this material? 1 = true
    #[uniform(0)]
    tile: f32,
    // ⭐ New uniform for time
    #[uniform(0)]
    time: f32,
    #[texture(1)]
    #[sampler(2)]
    color_texture: Option<Handle<Image>>,
}

Let’s add a new system to our game that goes through all the materials in our scene and updates the time variable on them. When we loop over the materials, they’re arrays, with the first index being the material ID and the second index as the actual material data. I left a dbg!() macro in the function that you can enable to see the time change as a sanity check.

fn main() {
    App::new()
        .add_system(update_material_time)
}

fn update_material_time(time: Res<Time>, mut materials: ResMut<Assets<CustomMaterial>>) {
    materials.iter_mut().for_each(|material| {
        // dbg!(material.1.time);
        material.1.time = time.elapsed_seconds();
    });
}

Now that we have time pumped into our shader, let’s use it to scroll the background.

Since we want to scroll the screen vertically, we need to focus on the Y coordinate of the UVs. We already take the Y coordinate of the UVs and multiply it, then use fract() to make it repeat between 0-1. We want to stay between the 0-1 coordinates, which means we should put the time inside the fract(), so it keeps the “repeating grid” consistent. If we subtract the time, lets say the UV starts at 0, if we subtract time, the UV will be negative — which gets converted back to a positive 0-1 by the fract().

struct CustomMaterial {
    color: vec4<f32>,
    tile: f32,
    // ⭐ Time increments
    time: f32,
};

@group(1) @binding(0)
var<uniform> material: CustomMaterial;
@group(1) @binding(1)
var base_color_texture: texture_2d<f32>;
@group(1) @binding(2)
var base_color_sampler: sampler;

@fragment
fn fragment(
    #import bevy_pbr::mesh_vertex_output
) -> @location(0) vec4<f32> {
    var tiled_uv: vec2<f32>;
    tiled_uv = uv;
    if(material.tile > 0.0) {
        var tiled_uv_x: f32;
        var tiled_uv_y: f32;
        tiled_uv_x = fract(uv.x * 10.0);
        // ⭐ We "scroll" the background by subtracting time here
        tiled_uv_y = fract(uv.y * 7.0 - material.time);
        tiled_uv = vec2(tiled_uv_x,tiled_uv_y);
    }
    return textureSample(base_color_texture, base_color_sampler, tiled_uv);
}

Adding sounds

Bevy has a great system setup in place for handling and playing audio assets. Let’s add some quick sound effects to the game to make it more immersive.

We could play sounds directly in the same functions where we currently fire the projectile - but that function has a lot going on already. Instead lets use Bevy’s “event system” to fire off an EnemyDeathEvent when we fire the weapon, then we can check for that event inside other systems, like say a play_enemy_death_sound() function. We could also create another system/function later that checks for the same event and increases the score.

Let’s create some structs to represent these “events”.

// Projectile has been fired
#[derive(Default)]
struct ProjectileEvent;

Then we can add them to our Bevy app.

fn main() {
    App::new()
        .add_plugins(DefaultPlugins)
        .add_plugin(Material2dPlugin::<CustomMaterial>::default())
        .insert_resource(ProjectileTimer(Timer::from_seconds(
            PROJECTILE_TIME_LIMIT,
            TimerMode::Once,
        )))
        .add_startup_system(setup_game)
        .add_system(update_material_time)
        // ⭐ Add events here
        .add_event::<ProjectileEvent>()

Now we can update the shoot_projectile() method to add an EventWriter property for our ProjectileEvent. This lets us send events using the .send_default() method on the event writer variable. We use that method specifically so we don’t have to define any event details (see the send() method for more info) and it creates a “default” event.

fn shoot_projectile(
    time: Res<Time>,
    mut projectile_timer: ResMut<ProjectileTimer>,
    mut commands: Commands,
    mut meshes: ResMut<Assets<Mesh>>,
    mut materials: ResMut<Assets<CustomMaterial>>,
    keyboard_input: Res<Input<KeyCode>>,
    mut query: Query<&Transform, With<Player>>,
    asset_server: Res<AssetServer>,
    mut projectile_events: EventWriter<ProjectileEvent>,
) {
    let player_transform = query.single_mut();

    if keyboard_input.pressed(KeyCode::Space) {
        // Check if player is allowed to shoot based on internal timer
        // We have to "tick" the timer to update it with the latest time
        if projectile_timer.0.tick(time.delta()).finished() {
            // Reset the timer
            projectile_timer.0.reset();

            // ⭐ Fire off a ProjectileEvent to notify other systems
            projectile_events.send_default();

We’ve got the event firing off, but we still need to have a system listening for that event and actually playing the sound. So let’s do just that.

We’ll create a new function play_projectile_sound that has an EventReader property to read our ProjectileEvent events. We also grab a audio property that gives us access to Bevy’s audio system (which we’ll use to play the sound). And we’ll grab the specific sound we want.

We can use the is_empty() method on the events to check if the list is empty, and if it’s not, we first clear the events (so the sound plays only once per frame). Then we can play the sound using the Audio system and our sound (which we have to clone because Rust reasons 🦀).

fn play_projectile_sound(
    projectile_events: EventReader<ProjectileEvent>,
    audio: Res<Audio>,
    sound: Res<ProjectileSound>,
) {
    // Check for events
    if !projectile_events.is_empty() {
        // Clear all events this frame
        projectile_events.clear();
        println!("[AUDIO] Playing projectile sound!");

        audio.play(sound.0.clone());
    }
}

But we actually need some sound to play right? I searched for a Galaga projectile sound effect and found one (but any MP3 or OGG will do — this is prime territory for silliness).

Once you have a sound in place, we can create the struct we used earlier ProjectileSound that’ll store the Resource reference to our sound asset.

// Sounds
#[derive(Resource)]
struct ProjectileSound(Handle<AudioSource>);

We need to load the sound, and since this sound will be used all the time, we can load it during the startup system. We use the insert_resource() method on the Commands property to add our sound file to Bevy’s list of active resources. This let’s us query it like we did using the Res<ProjectileSound> earlier.

fn setup_game(
    mut commands: Commands,
    mut meshes: ResMut<Assets<Mesh>>,
    mut materials: ResMut<Assets<CustomMaterial>>,
    asset_server: Res<AssetServer>,
) {
    // Camera
    commands.spawn(Camera2dBundle::default());

    // ⭐ Load sound effects
    let projectile_sound = asset_server.load("sounds/projectile.mp3");
    commands.insert_resource(ProjectileSound(projectile_sound));

You’ll notice one problem though when you run the game. No sound! If you look at the Bevy logs when the game starts you’ll see an error about the asset server missing a mp3 module. This means we need to enable the mp3 feature for Bevy in the cargo.toml file. You can see a list of all Bevy features you can optionally enable on Github.

Open up the cargo.toml and you can add the feature like so, or run cargo install bevy --features mp3:

[dependencies]
bevy = { version = "0.9.1", features = ["mp3"]}

Now you should be able to re-compile the app using cargo run and hear sound.

What’s next?

There’s a lot you could add, that’s the beauty of a game! From a start screen, to a score, to random enemies, animations for enemies entering — the list goes on and on (as most game devs will know).

In the next blog I’ll show you how to add UI, a start screen, an intro sequence, and enemy spawning to the game.

I hope this inspires you to make some games with Bevy and shows how simple Bevy makes the process.

As always, if you have any questions or want to share your own game dev work, feel free to share with me on Mastodon or Twitter.

Stay curious,
Ryo

Resources

bevy/breakout.rs at latest · bevyengine/bevy

Learning DirectX 12 in 2023

Ryosuke — Mon, 30 Jan 2023 21:21:26 +0000

For the past couple months I’ve been slowly diving into DirectX 12. After learning other graphic backends like OpenGL or WebGPU, I was interested in trying something more advanced and industry standard. After a lot of reading documentation, doing tutorials, and playing with a lot of example code - I thought I’d share the multitude of resources I discovered. And I’ll share my experience with most of the content, to give you a better idea of what might work or not for you.

This is my journey to learning DirectX 12, and tips to help get you ready to learn it.

Normally there’s a TLDR here with a link or two, but this article is packed full of helpful links, so it’s definitely worth going through!

Why learn DirectX 12?

There’s definitely lots of reasons not to, like if you need to create a cross-platform renderer or mobile specific renderer. I wouldn’t feel pressured to dive into a new and very complicated API if you don’t have a good reason to already.

But if you’re even slightly interested in DirectX 12, you probably have a slight idea of how amazing it is. It’s provides developers with much greater access and control over resource and memory management in applications and games — allowing you to optimize them more than DirectX 11 or similar, simpler graphic APIs.

If you’re looking to create AAA games on PC or Xbox, or high fidelity 3D applications, you’ll definitely need to learn DirectX 12. Likewise, you’ll need it to use new 3D technology, like raytracing or compute shaders.

“Learn DirectX 11!”

The first thing everyone will tell you to do when learning DirectX 12 is to go back and learn 11 if you haven’t already. A lot of the APIs you’ll use in DX12 were used in DX11, and some are even just renamed.

But if you’re like me, you probably skipped this step. I figured as someone who’s used a few different graphic backends, I’d be alright. That wasn’t the case! But would have DX11 helped me? Not completely.

Most of the issues I had were understanding the dramatically different architecture for drawing (aka the Draw Command Lists, Vsync and Swap Chains, etc). These weren’t in DirectX 11, so learning that first wouldn’t have helped. What would have helped? Learning Vulkan first. If you have any experience with Vulkan, it’s a very similar architecture to DirectX 12.

⚠️ That being said, I wouldn’t learn DirectX 12 until you have a solid grasp of one of the other “simpler” graphic backends like OpenGL, DirectX 11, or even Web GPU.

“Before You DirectX 12” Checklist

I never like to say “you absolutely need to know X before Y”, but when you’re learning a new concept it’s easier when you know what’s what. Is this a DirectX 12 API or just C++ stuff? The more I rushed I found myself asking these questions. It’s not a wrong approach - though you’ll find it can become overwhelming quickly looking everything up.

The more knowledge you can come reinforced with will make your journey a breeze. So these are the topics you should brush up on prior to DirectX 12:

☑ Learn all basic C++ concepts like the basic types (int, vector, etc), classes, conditionals, loops — you’ll need a lot of it.
☑ Dig into advanced C++ concepts like I/O streaming or memory allocation (like stack vs heap). These concepts will be used heavily since graphics programming tends to be very low level and deal with streams of data inside “buffers”.
☑ Using libraries in C++. Like much graphics projects, you’ll find yourself linking to DLLs to access APIs from graphic drivers (like DirectX 12 itself) — or maybe other libraries like DirectXTK for utilities or imgui for UI. Most tutorials for OpenGL will cover this process, ideally you just need to learn how to do it in Visual Studio (instead of going down a CMake rabbit hole).
☑ Refresh on modern C++ standards from C++ 11, 17, and maybe 20. Tips like using auto to quickly define variables will help you work faster or understand code examples from tutorials. Don’t worry about going too deep here.
☑ Learn the basics of graphics programming. From transformation matrices, to vertices, to techniques like depth buffers or shadow maps — these are all things you’ll exercise in DirectX 12 and previous experience with a simpler API (like OpenGL) would make the process easier. I recommend following the tutorials on the Learn OpenGL website, it does a great job of going over each concept and explaining them with visual diagrams.
☑ Learn common design patterns and architectures. From the simple infinite game loop, to factories or observers — you’ll find these patterns repeated in a lot of codebases and tutorials without much explanation. I recommend reading over the Game Programming Patterns, and a couple introductory game development tutorials. If you have experience with another programming language like JavaScript you might already be familiar with these concepts.

Not required - but consider…

☑ Maybe read a Vulkan tutorial or two… Like I mentioned earlier, DirectX 12 is very low level, way more than OpenGL. There’s way more Vulkan stuff out there than DirectX 12, so you might have an easier time finding an article or video that helps “click” some concepts before you dive into DX12.
☑ Play around with Microsoft UWP or WinRT apps. Most DirectX 12 tutorials will use Windows APIs for windows management (aka creating a blank window for DX12 to render inside). These window APIs come from a library called WinRT, part of the Universal Windows Platform (UWP). This isn’t a requirement, since you can use open source and cross-platform libraries like GLFW to do similar operations — but most DX12 tutorials won’t use GLFW, so keep that in mind.

What will learn alongside DirectX 12?

This is probably where the “start with DirectX 11” supporters win a bit — there’s a quite a few legacy Microsoft libraries you’ll find yourself using alongside DirectX 12 to simplify development.

From SimpleMath for calculating matrix math to DirectXTex load and process textures, you’ll inevitably learn and use these libraries as you explore DirectX 12 deeper. And most of these libraries are not new to DirectX 12, if you’ve used DirectX 11 you probably have experience with them.

DirectXTK 12 (includes SimpleMath and other goodies)
DirectX Shader Compiler
DirectXTex
D3D12 Memory Allocator
Win32 window APIs

Getting started with DirectX 12

When you’re initially getting started with DirectX 12, there are a few ways to get off the ground. From the official template in Visual Studio, to Microsoft’s examples on Github, to plenty of tutorials online.

I outlined a bunch of the different methods I tried during my studies, and the pros/cons of using each one.

Visual Studio Template

When I initially started, I used the official DirectX 12 template that comes with Visual Studio. This gets you up and running with DirectX 12 rendering a spinning cube. Just install Visual Studio, make sure DirectX development is enabled, and you should find a DirectX 12 template.

It’s a great way to jump from nothing to a cube, but it also relies heavily on Windows APIs (for things like window management) and their custom framework abstraction. It’s a decent app to reverse engineer while you read the extensive DirectX docs.

Official Examples

I eventually moved on to the official DirectX 12 Graphic Samples, which had isolated examples that were easier to learn from incrementally. The “Hello World” sample is the best starting point to get the renderer setup with a triangle. This repo is a great reference when you need to verify process and structure (like the order of operation of APIs). Whenever I had a strange bug in my app, I’d go back to these examples to do a diff and see what I was missing.

Alain Galvan tutorial

Since the DirectX documentation was a bit hard to difficult to read through (not very human friendly - gets lost in it’s acronyms), I started looking for alternative and modern resources for learning. I stumbled onto this blog post Raw DirectX 12 by Alain Galvan (and his entire blog full of fantastic graphic programming tips) which breaks down the DX12 render pipeline with nice visual diagrams and quick code examples.

I’ll be honest, this also was nice - but didn’t do a good job of explaining relationships between parts of the pipeline. It’s likely because they also have a book you can buy that goes into more detail. I also had issues spinning up the directx12-seed repo. Since my PC / graphics card doesn’t support all the DX12 features (like raytracing), the app would error out instead of running. As someone new to DirectX 12, stuff like this is an immediate productivity killer.

Alex Tardiff tutorial

Still not satisfied, I kept searching for more tutorials. I stumbled on this “gentle” introduction to DirectX12 by Alex Tardif. This one was probably one of the more confusing tutorials as a beginner.

Most of the tutorial is done using a custom abstraction around DirectX 12 (named “D3D12Lite”), which simplifies many of the operations like creating buffers or managing memory. The issue is that the tutorial doesn’t cover what D3D12Lite is doing under the hood, so you end up having to reverse engineer it yourself. You end up learning their framework - more than D3D12.

This was better as a resource to come back to once I had established a foundation using a “hello world” tutorial/template, and progressed with that far enough to consider higher level abstractions. The architecture of D3D12Lite was easier to understand that something like say MiniEngine.

3Dgep tutorial

Then I finally found one the best tutorials I’d seen yet on 3dgep called Learning DirectX 12. The article series goes into the history of DirectX, each part of the pipeline, and how to setup everything from scratch. I didn’t actually follow the tutorial part because they focused on using Windows APIs for things like windows management, but it was a fantastic read compared to the official Microsoft docs.

Braynzarsoft tutorial

This one was another fantastic tutorial that also speaks human (and not acronyms), and gives clear examples for what things are - like a code example for root parameters. It was lacking visuals, but I appreciated complete code examples at the end of each post for clarity. Another great reference if one of the tutorials doesn’t work for you.

Video tutorials

There’s not a wealth of introductory DirectX 12 video content online, but this series by Olympus Mons Tutorials was great for starting from scratch and going step by step. It uses Microsoft APIs for window management, but beyond that it’s DX12.

Taking DX12 further

After the beginner tutorials and articles, you might ask yourself, where do I go?

MiniEngine

The best place to get started is MiniEngine, part of the DirectX Graphic Samples. It’s a 3D game engine written in DirectX 12 with a lot of features built out (resource management architecture, shaders like PBR, etc). The app itself can load any 3D model provided (or use a default one) and allow the user to preview it. It’s a great reference for creating a whole application around DirectX.

DirectXTK

Alongside MiniEngine, you’ll want to look into the DirectX Toolkit. This is a set of utilities by Microsoft that simplify graphics and game development. It contains libraries like DirectXMesh for parsing and optimizing meshes for DX12, or DirectXMath which handles 3D math operations like the OpenGL library glm. It also has utilities for gamepad input or sprite fonts. You can see a list of the headers here to get an idea of the features. You’ll definitely want to include this in your project if you don’t want to think about a lot of these solved problems (and don’t have to worry about cross-platform support).

The DXTK also has a great tutorial series on creating a game loop that breaks down the DX12 pipeline through the process. They also have a repo with Xbox game SDK samples (not games — but features you’d use in games like gamepad input).

Cauldron

Cauldron is a 3D render engine framework that uses DirectX 12 or Vulkan as the backend. It was built by AMD for internal prototypes and projects. You use it like a static library and import it into your existing C++ app to use as a framework. It was built first using DirectX 12, then Vulkan support added later, so great project to analyze from that perspective.

It features a lot of intermediate to advanced features you’d want to add to your own DirectX 12 engine, like a ring buffer or post processing effects (anti-aliasing, bloom, HDR skyboxes). And it’s fairly well structured and documented, so it’s easy to reverse engineer.

Game Engine Series

If you’re more of a video person, there’s not a lot of videos on YouTube for DirectX 12. Most of it is pretty high level or conceptual - like presentations from Microsoft developers. But I did find one video series that was very in-depth and impressive.

This series on creating a cross-platform game engine had a section on adding DirectX 12. It comes from the perspective of making a Unity/Unreal Engine-like game engine, using Windows APIs for UI, and multiple graphic backends (using DirectX12 as the primary one).

This series definitely isn’t for beginners of DirectX 12, it’s better when you’ve learned the basics and you want to take it to the next level and apply it in a realistic application.

Unreal Engine

If you sign up to become a Unreal Engine developer (which is free), you can get access to the engine’s source code on Github. This would probably be best saved for the end of your studies because this is like asking someone to learn by reading a book on rocket science right off the bat. There are much better and simpler resources that break down features inside Unreal Engine (like the video series above!).

Unreal Engine is very complicated and has heavy abstractions around graphic backends. Check out this blog post by Kostas Anagnostou on How Unreal Renders a Frame to get an idea of how complex it is. This post by Alain also has a nice GIF showing the render order of the UI.

Learning from AI

I also experimented with using Notion AI as a tool for learning various DirectX concepts. I prompted the AI to explain a concept, and even provide code samples. You can see the results of that here, and see how accurate — and inaccurate — the AI was at teaching. Here’s a link to the full notes if you’re curious.

I wouldn’t recommend this for beginners, since the AI tends to “invent” APIs that don’t exist. Without prior experience, you’ll have a harder time debugging and picking up bad practices.

Community

When the videos and tutorials fail and you need a helping hand, or you just want to connect with other people doing similar cool 3D renderer stuff — I’d definitely recommend checking out these Discords communities.

DirectX Discord - The official DirectX community.
Graphics Programming Discord - They have a DirectX 11 study group happening now that’s making an introductory tutorial series.
RenderDoc - Not directly DirectX, but if you need help debugging apps using RenderDoc.
Shadertoy - Also not directly DirectX, but a great resource for shader debugging and tips.

Just dive in

The most important thing to do is dive right in. Don’t get intimidated how much you think you have to learn, or the mountain of documentation and resources that you have to dig through. Like I mentioned in the beginning, I jumped in without DirectX 11 knowledge, and I’m doing fine.

There are plenty of entry points into DirectX 12, especially if you have any experience in similar fields (game dev, C++, etc). It’s also ok to walk away, learn other stuff, and come back. It’ll surprise you how much things become easier to learn once you’ve done it, and if it’s with OpenGL or Vulkan and not DirectX first — that’s ok.

As always, if you have any questions or want to share your own DirectX work, feel free to message me on Twitter or Mastodon.

Stay curious
Ryo

Letting AI write my notes and code

Ryosuke — Wed, 04 Jan 2023 01:04:02 +0000

I recently got access to the Notion AI alpha test, where Notion is integrating AI features into their note-taking app. I’ve seen a lot of people talking about AI recently on my feed, particularly “chat-bot” style AI programs (like ChatGPT) that take a “prompt” and return text results. I thought this would be a great way for me to experiment with the new(ish) technology others are using.

So I asked went into the experiment to answer a few questions:

What can you use AI for, particularly in a notes app?
How accurate is it?
What is the extent of the knowledge - does it know about me and my previous projects?
What are the limitations?

What is Notion AI?

Once you get access, the Notion command menu gets new AI commands.

Just looking at this list though, we get an idea of what kind of content we could be generating with AI. From poems to social media posts complete with hashtags — it’s a wide range.

These commands pop open a textbox and request a prompt.

The AI results are formatted according to the command you pick, like a list of short sentences for a “Outline”, or 5 paragraphs that obey the “essay” format of intro, detail, and conclusion. The first few commands encompass the other more exact commands. For example, the “Brainstorm ideas” was very similar to “Outline”.

The AI “types” the results out in chunks. Once the results are generated, you get the option to keep them, delete it, or try the prompt again.

Experiments

I setup a new page in my Notion to store all the AI experiments. Then for each experiment, I created a new page.

I picked a wide variety of topics I work with on a daily basis, from programming to music to my personal brand.

I ran different kinds of tests depending on the topic. For conference talks for example, I ran the same prompts on the “Help me write…” and “Outline” commands. Or for graphics programming, I asked the AI to write the same “app” or feature in different backends (OpenGL, DirectX, or Vulkan).

Surprisingly it handled every topic pretty well and was fairly accurate with each (albeit vague or short).

Does it steal information?

You bet. It doesn’t technically plagiarize content, but you can clearly see where the AI takes a certain source, copies it, and replaces a few words with synonyms. It’s not as bad as I make it sound, it does a better job of paraphrasing things, but sometimes it’s fairly clear that only 1 source had this info.

For this reason, combined with the fact the AI doesn’t give you reference links to where it sourced the data, I’d never use it directly for production content.

It’s a great tool for personal use, bootstrapping, etc — but never going directly from AI to public use. One could argue that if you release the information open source it’s not infringing, but it feels like a disservice to real humans who put in the work to discover and document this only to be forgotten. And besides any ego, it makes information useless. Who said this? Has this source been verified? The AI could be copying lies and passing it off as truth.

Is it accurate?

Nope. From asking about myself, to asking about specific topics, there are a lot of inaccuracies in the results.

I asked for an “article about Ryosuke Hana” and it gave me a 5 paragraph essay about how I’m a painter and sculptor and won awards I haven’t. I go and prompt it again with a completely different name (not an artist) and it returns almost the exact same result, but with the name swapped.

I asked the AI to compare some APIs and it started inventing API functions that didn’t exist.

// Detect dark theme var iframe = document.getElementById('tweet-1608761340470104070-873'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1608761340470104070&theme=dark" }
A similar experience shared by @nikitavoloboev

It seems to operate almost contextually, so if you ask it something and get results back, they’ll often “pollute” the next prompts — even if they’re significantly different.

Or it just didn’t perform as well as I wanted. I tried asking “Article listing the best C++ starter templates” and it was giving me fake links and projects (”One popular starter template is the CPPTemplate” — links to a Github that never existed). That’s really bad. I expected it to just grab the top 10 templates from an “awesome C++ README” or something, but it was just inventing fallacies.

So what’s it useful for?

I found a couple use cases that make the AI fairly useful, and something that actually positively impacts my workflow.

Solving bugs
Comparing APIs + generating useful code snippets
Learning new topics
Generating listicles and word banks

Solving bugs

I had an issue with a wgpu renderer I was writing in Rust and I had a bug I couldn’t figure out. I asked the AI generally about the topic and it gave me a code snippet that gave me context I needed to solve the problem.

Did it solve the bug for me? No. Did I type the bug directly in as a prompt? Nope. It still took a my own investigation and critical thinking to work around the issue, but the AI proved to be an invaluable source, especially after using search engines and having StackOverflow fail on me.

I could almost liken it to the experience of pair programming and being able to ask someone (possibly more knowledgeable than you) how things work. It lets you check yourself and gives you more personalized insight (rather than digging through docs or something).

Comparing APIs

In graphics programming, we have to render 2D and 3D graphics on a lot of different devices and platforms. This means there’s often a lot of “duplicate” code for implementing certain features in different backends (like doing shadows in OpenGL, DirectX, and Vulkan). Each graphics backend has it’s own API and set of functions for interacting with the graphics drivers and hardware. This can be a lot to keep track of, particularly because of how verbose all the functionality tends to be.

I ran a test where I asked the AI to show me how to do different 3D graphic techniques in different backends.

Create a fog effect in [insert backend here] with C++
Create a PBR shader in [insert backend here] with C++
Level of distance rendering in [insert backend here] with C++
Particles rendering in [insert backend here] with C++

I ran each prompt 3 times, one for each backend (OpenGL, DirectX, and Vulkan).

Here were the results for the “fog effect”:

OpenGL

// Create a fog effect in OpenGL with C++

// Set up the fog parameters
float FogDensity = 0.05f;
float FogStart = 10.0f;
float FogEnd = 100.0f;

// Set the fog color
glClearColor(0.5f, 0.5f, 0.5f, 1.0f);

// Enable fog
glEnable(GL_FOG);

// Set the fog parameters
glFogi(GL_FOG_MODE, GL_LINEAR);
glFogf(GL_FOG_START, FogStart);
glFogf(GL_FOG_END, FogEnd);
glFogf(GL_FOG_DENSITY, FogDensity);

// Set the fog color
glFogfv(GL_FOG_COLOR, glClearColor);

DirectX 12

// Create a fog effect in DirectX 12 with C++

// Set up the fog parameters
float FogDensity = 0.05f;
float FogStart = 10.0f;
float FogEnd = 100.0f;

// Create a fog color
XMVECTORF32 FogColor = {0.5f, 0.5f, 0.5f, 1.0f};

// Set up the fog color
D3D12_RENDER_TARGET_BLEND_DESC RTBlendDesc = {};
RTBlendDesc.BlendEnable = true;
RTBlendDesc.SrcBlend = D3D12_BLEND_SRC_ALPHA;
RTBlendDesc.DestBlend = D3D12_BLEND_INV_SRC_ALPHA;
RTBlendDesc.BlendOp = D3D12_BLEND_OP_ADD;
RTBlendDesc.SrcBlendAlpha = D3D12_BLEND_ONE;
RTBlendDesc.DestBlendAlpha = D3D12_BLEND_ZERO;
RTBlendDesc.BlendOpAlpha = D3D12_BLEND_OP_ADD;

// Set up the blend state
D3D12_BLEND_DESC BlendDesc = {};
BlendDesc.RenderTarget[0] = RTBlendDesc;

// Set up the fog parameters
D3D12_RASTERIZER_DESC RasterDesc = {};
RasterDesc.FillMode = D3D12_FILL_MODE_SOLID;
RasterDesc.CullMode = D3D12_CULL_MODE_NONE;
RasterDesc.FrontCounterClockwise = FALSE;
RasterDesc.DepthBias = D3D12_DEFAULT_DEPTH_BIAS;
RasterDesc.DepthBiasClamp = D3D12_DEFAULT_DEPTH_BIAS_CLAMP;
RasterDesc.SlopeScaledDepthBias = D3D12_DEFAULT_SLOPE_SCALED_DEPTH_BIAS;
RasterDesc.DepthClipEnable = TRUE;
RasterDesc.MultisampleEnable = FALSE;
RasterDesc.AntialiasedLineEnable = FALSE;
RasterDesc.ForcedSampleCount = 0;
RasterDesc.ConservativeRaster = D3D12_CONSERVATIVE_RASTERIZATION_MODE_OFF;
RasterDesc.FogMode = D3D12_FOG_MODE_LINEAR;
RasterDesc.FogStart = FogStart;
RasterDesc.FogEnd = FogEnd;
RasterDesc.FogColor = FogColor;

// Set the state
ID3D12GraphicsCommandList* pCommandList;
pCommandList->RSSetState(pRasterState);

Vulkan

// Create a fog effect in Vulkan with C++

// Set up the fog parameters
float FogDensity = 0.05f;
float FogStart = 10.0f;
float FogEnd = 100.0f;

// Create a fog color
VkClearValue FogColor = {0.5f, 0.5f, 0.5f, 1.0f};

// Create the render pass
VkRenderPassCreateInfo RenderPassInfo = {};
RenderPassInfo.sType = VK_STRUCTURE_TYPE_RENDER_PASS_CREATE_INFO;
RenderPassInfo.attachmentCount = 1;
RenderPassInfo.pAttachments = &FogColor;

// Set up the blend state
VkPipelineColorBlendAttachmentState ColorBlendAttachment = {};
ColorBlendAttachment.blendEnable = VK_TRUE;
ColorBlendAttachment.srcColorBlendFactor = VK_BLEND_FACTOR_SRC_ALPHA;
ColorBlendAttachment.dstColorBlendFactor = VK_BLEND_FACTOR_ONE_MINUS_SRC_ALPHA;
ColorBlendAttachment.colorBlendOp = VK_BLEND_OP_ADD;
ColorBlendAttachment.srcAlphaBlendFactor = VK_BLEND_FACTOR_ONE;
ColorBlendAttachment.dstAlphaBlendFactor = VK_BLEND_FACTOR_ZERO;
ColorBlendAttachment.alphaBlendOp = VK_BLEND_OP_ADD;

// Set up the pipeline state
VkPipelineRasterizationStateCreateInfo RasterInfo = {};
RasterInfo.sType = VK_STRUCTURE_TYPE_PIPELINE_RASTERIZATION_STATE_CREATE_INFO;
RasterInfo.cullMode = VK_CULL_MODE_NONE;
RasterInfo.depthBiasEnable = VK_FALSE;
RasterInfo.depthClampEnable = VK_TRUE;
RasterInfo.fogMode = VK_FOG_MODE_LINEAR;
RasterInfo.fogStart = FogStart;
RasterInfo.fogEnd = FogEnd;
RasterInfo.fogColor = FogColor;

// Set the state
VkCommandBuffer CommandBuffer;
vkCmdSetFog(CommandBuffer, &RasterInfo);

You can see that in OpenGL, we have functions available that simplify creating a fog effect (like glFog). But in DirectX and Vulkan (which look vary similar because of their architectural similarities), we have to create a new render pass and add it to the command buffer.

I could have used a search engine and prompted it for each of these, searched through the results, found a good StackOverflow code snippet and copied that into my notes — or I could just ask the AI to cut all that work out.

However we quickly start to see the limitations of the AI if we inspect the code closely, the DirectX 12 snippet features a D3D12_FOG_MODE_LINEAR constant that doesn’t exist. It might have been pulled from a public DirectX project - or just made up. So the results kind of fall apart in some places, since I’d have to do more work to find the bug and fix it.

Learning new topics

I’m constantly trying to learn new things and apply them to my work. In graphics programming for example, there’s lots of techniques used with almost scientific sounding names that can be difficult to wrap your head around. The wiki pages can often read like a high level encyclopedia that drown you in a lot of academic terminology you either know or spend more time researching.

And to make things even more complicated, these graphics techniques are often explained in an abstract way. Then you have to figure out how to actually implement the concept in code using specific libraries or backends.

For one of the experiments, I prompted the AI to teach me about a topic, then show me a code example. From “Two Pass Occlusion Culling” to “Mesh Cluster Occlusion”, I asked the AI to explain the topics to me in terms of Unreal Engine or “game development” in general, so I’d have a context I’d be closer to. This didn’t work as intended, but it still generated helpful results:

## Two Pass Occlusion Culling Graphics Programming in Unreal Engine

Unreal Engine is an industry-leading game engine that is used by many developers to create stunning graphics, immersive gameplay, and realistic worlds. One of the key areas of graphics programming in Unreal Engine is two-pass occlusion culling. This technique is used to render only the objects that are visible in the player's view, making the game run more efficiently and look better.

In two-pass occlusion culling, the first pass is used to determine the visibility of objects in the view. This is done by calculating the distance between the player's view and each object in the scene. If the object is too far away, it is not rendered. The second pass is used to optimize the rendering of the scene. It takes into account the visibility of objects that are close to each other and attempts to render them together. This helps to reduce the number of draw calls and improves performance.

Two-pass occlusion culling is an important part of graphics programming in Unreal Engine. It helps to improve the performance of the game and make the graphics look better. It is also important for developers to understand how two-pass occlusion culling works and how to optimize it for their games.

Here was a code example generated:

// Initialize two-pass occlusion culling
for (int32 MeshIndex = 0; MeshIndex < Meshes.Num(); MeshIndex++)
{
    // Get the mesh
    const FMesh& Mesh = Meshes[MeshIndex];

    // Calculate the distance between the player's view and the mesh
    float Distance = Mesh.BoundingBox.GetDistanceToPoint(PlayerViewPosition);

    // If the distance is too great, skip the mesh
    if (Distance > MaxDistance)
        continue;

    // Otherwise, render the mesh
    RenderMesh(Mesh);
}

// Iterate through all meshes
for (int32 MeshIndex = 0; MeshIndex < Meshes.Num(); MeshIndex++)
{
    // Get the mesh
    const FMesh& Mesh = Meshes[MeshIndex];

    // Get the mesh's bounding box
    const FBox& Box = Mesh.BoundingBox;

    // Iterate through all meshes in the scene
    for (int32 OtherMeshIndex = 0; OtherMeshIndex < Meshes.Num(); OtherMeshIndex++)
    {
        // Skip the current mesh
        if (MeshIndex == OtherMeshIndex)
            continue;

        // Get the other mesh
        const FMesh& OtherMesh = Meshes[OtherMeshIndex];

        // Check if the meshes are close to each other
        if (Box.Intersect(OtherMesh.BoundingBox))
        {
            // Render both meshes
            RenderMesh(Mesh);
            RenderMesh(OtherMesh);
            break;
        }
    }
}

You can see that this is a lot of “pseudo” code almost, because we don’t have the context of a lot of the variable/class types or functions used. But if you look at what’s happening structurally (with the conditional statements and named method like Interesect()) — you can infer enough to understand the topic (at least for me).

Generating listicles and word banks

One of the hardest parts of writing is remembering all the structure of a topic, and recalling the numerous words or phrases related to the topic. For technical topics, it helps to have a “cheat sheet” of technical terminology and acronyms.

The AI was pretty good at generating lists or outlines. For my conference talk experiments, I would ask about creating a presentation about a topic and it’d give me a very solid structure. And if I used the longer-form AI results, I’d get more technical terms I could refer to.

I don’t think I’d use this immensely because most of my content is very contextual, so it’s hard to prompt an AI with enough information (e.g. give me a graphics programming presentation from the perspective of someone who just started 2 years ago and works at PlayStation…). For that reason this would only be a supplement of the draft or review process to quickly check if I’ve covered everything or am missing anything specific.

Would I use it again?

Probably. Not something I’d pay a lot for, but if it was available to me for free or low cost, I would utilize it alongside my other tools and practices.

If they improved the AI to:

Add footnotes with source references
Not lie / invent / get “creative”? Or a toggle for it when needed.
Have the ability to “clear” context to reset

It’d start to get to a point where it might replace some of my current processes.

As always, if you have any questions or want to share your own AI findings, feel free to message me on Twitter or Mastodon.

GLTF Animations in wgpu and Rust

Ryosuke — Wed, 14 Dec 2022 17:24:33 +0000

Liquid syntax error: Unknown tag 'endraw'

Render Pipelines in wgpu and Rust

Ryosuke — Sat, 19 Nov 2022 02:33:48 +0000

I finished this great wgpu tutorial that gets you up and running with WebGPU and Rust using the wgpu crate. It teaches you to setup the graphic context and render pipeline, import and render a textured model, and even add lighting. You can see the end result in this video:

// Detect dark theme var iframe = document.getElementById('tweet-1578653432872767488-6'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1578653432872767488&theme=dark" }

Afterwards though, you end up with a few modules, but most of the code is in one giant file. In a real application, you’d probably split up a lot of the functionality happening into reusable modules. I started to ask myself: how would I render multiple models with different textures — or an entire scene?

I thought I’d share my approach as I take the wgpu tutorial code and refactor it. We’ll split the functionality out and build in an architecture to allow for a flat scene of 3D models. I’ll even sprinkle in a little bit of nested scene architecture. This process helped me (and hopefully you) attain a deeper understanding of the render pipeline and how each component works together.

TLDR?: Here’s the final code.

This is part of a series on wgpu in Rust. Find more posts in the #wgpu tag.

Research

Before I set off on the journey of refactoring, I looked for other wgpu projects to see how they structured their app architecture. I found 2 great examples that I primarily referenced - both with “b” names funnily enough: baryon and bevy.

baryon is a lightweight toy renderer for prototyping 3D applications in Rust. It uses wgpu to render 3D elements, hecs for an ECS system (to make a scene with “entities” like 3D models), and winit for handling cross-platform window management (just like the wgpu tutorial). It also allows for setting different “render passes”, like a Phong (”cartoony”/fast) vs PBR (”realistic”) pass.

Bevy is an entire game engine for creating 2D or 3D games in Rust. It uses wgpu for rendering everything, and a few other dependencies - mostly their own crates. It’s not “fully featured” like Unity or Godot since it’s so early — but it’s jam-packed with a lot of great functionality (like most recently — compute shaders).

I ended up referring to baryon more, because the codebase was much smaller and simpler. But bevy was a great alternate reference for specific functionality (like figuring out how to render primitives - or soon, parsing and rendering different 3D file formats).

Breaking down baryon

I won’t go too deep here, but I want to quickly overview the architecture of Baryon to see what we can takeaway for our own app.

The best place to start is the library’s simplest example to understand the shape of the API and kind of functions we’ll see run.

A Window is created and built (likely using the builder pattern) (L#9)
Context is initialized using a reference to the window, and we use pollster to block the main thread from stopping (L#10)
A Scene is created. (L#11)
A Camera is created with initial data (like a position and clear color)
Then the scene is populated with a light and 2 primitive entities.
We create a “render pass” - in this case a Phong pass (L#42)
Now the magic happens — we run the Window method, which creates a infinite loop that we can render inside by matching the Event::Draw enum.
Inside the draw event, we use the Context to present (or likely ”render” our scene)

That’s it! Not too bad. But there’s a bit happening behind the scenes to make this code look so short and elegant. You can also see some similar modules from the wgpu tutorial, like the Camera.

My goal was to use baryon as inspiration, but not just copy paste. I wanted to truly refactor the app and experience why and how baryon author got to their code (or end up on my own path if needed). It’ll be hard to convey everything I learned, so I highly recommend doing something like this as an exercise, especially if you want to learn more advanced Rust (like me).

The Process

We’ll break down the app into a few different parts:

Window
Context
Render Pass

I tried to keep the commits fairly organized, so feel free to cruise through the commits to get a clearer picture if needed.

The “gimmes”

The wgpu tutorial did a great job of modularizing some of the code already, like the Camera or Instance structs. We can take those and separate them into their own modules to clean up the library file a bit. Here are the commits for Camera and Instance. Great practice for basic understanding Rust modules.

Window

This was probably the easiest module to pull out. I grabbed all the initialization methods at the top of the app and put them into a Window struct.

use winit::{
    event::*,
    event_loop::{ControlFlow, EventLoop, EventLoopWindowTarget},
    window,
};

pub struct Window {
    pub event_loop: EventLoop<()>,
    pub window: window::Window,
}

impl Window {
    pub fn new() -> Self {
        // TODO: Add size
        let event_loop = EventLoop::new();
        let window = window::WindowBuilder::new()
            .with_title("ryos wgpu playground")
            .build(&event_loop)
            .unwrap();

        Self { event_loop, window }
    }
}

Then in the app, we could initialize the new Window struct and use the window reference we store inside:

let window = Window::new();
// Later in the app
window.window.set_inner_size(PhysicalSize::new(450, 400));

This worked great, but we get left with our event loop code (window.event_loop) outside the Window struct. And there’s a lot of boilerplate code happening that could be moved into the Window struct. So let’s do that (commit for reference).

First we have to create some window “events” to match inside our render loop. These are just winit events we pass down, like the window resizing, or keyboard events:

pub enum WindowEvents {
    Resized {
        width: u32,
        height: u32,
    },
    Keyboard {
        state: ElementState,
        virtual_keycode: Option<VirtualKeyCode>,
    },
    Draw,
}

I tried not to create new keyboard events like baryon, and just keep it simple by using the types provided by winit - like VirtualKeyCode.

Now we need a method to “run” our window loop. And it needs to accept a callback function that gets the window event enums we just created. I looked up how to do a callback function parameter in Rust and it was recommend to use the Fn() or FnMut() types to wrap the result you return (the enum in this case WindowEvents). Rust did not like this…

// 🚫 Compiler error!
pub fn run(self, mut callback: FnMut(WindowEvents) -> ()) {

I ended up doing essentially the same function signature as the baryon project because the Rust compiler suggested it 😅 — using the impl 'static + prefix:

pub fn run(self, mut callback: impl 'static + FnMut(WindowEvents) -> ()) {

Once I figured that out, it was just a matter of copying/pasting the giant event loop in the Window.run() method and adding the callback param.

Now in the app, the window loop looked much cleaner ✨:

window.run(move |event| match event {
    WindowEvents::Resized { width, height } => {
        state.resize(winit::dpi::PhysicalSize { width, height });
    }
    WindowEvents::Draw => {
        state.update();
        state.render();
    }
    WindowEvents::Keyboard {
        state,
        virtual_keycode,
    } => {}
});

Now that we have a window setup, let’s move on to the graphical context.

Context

This was was fairly simple (at first). Since we’d be splitting our app into a context and a “render pass”, the logic for the context itself is very short.

We basically initialize the context (the “surface” we draw on, the “device” we use, the “queue” we schedule draws with, and the “config” of the surface) and keep it around using the struct. More of the good ol’ copy paste basically, and making sure the GraphicsContext struct types were correct.

use crate::{texture, window::Window};

pub struct GraphicsContext {
    // Graphic context
    pub surface: wgpu::Surface,
    pub device: wgpu::Device,
    pub queue: wgpu::Queue,
    pub config: wgpu::SurfaceConfiguration,
}

impl GraphicsContext {
    pub async fn new(window: &Window) -> GraphicsContext {
        let size = &window.window.inner_size();

        // The instance is a handle to our GPU
        // BackendBit::PRIMARY => Vulkan + Metal + DX12 + Browser WebGPU
        let instance = wgpu::Instance::new(wgpu::Backends::all());
        let surface = unsafe { instance.create_surface(&window.window) };
        let adapter = instance
            .request_adapter(&wgpu::RequestAdapterOptions {
                power_preference: wgpu::PowerPreference::default(),
                compatible_surface: Some(&surface),
                force_fallback_adapter: false,
            })
            .await
            .unwrap();

        // Select a device to use
        let (device, queue) = adapter
            .request_device(
                &wgpu::DeviceDescriptor {
                    label: None,
                    features: wgpu::Features::empty(),
                    // WebGL doesn't support all of wgpu's features, so if
                    // we're building for the web we'll have to disable some.
                    limits: if cfg!(target_arch = "wasm32") {
                        wgpu::Limits::downlevel_webgl2_defaults()
                    } else {
                        wgpu::Limits::default()
                    },
                },
                // Some(&std::path::Path::new("trace")), // Trace path
                None,
            )
            .await
            .unwrap();

        // Config for surface
        let config = wgpu::SurfaceConfiguration {
            usage: wgpu::TextureUsages::RENDER_ATTACHMENT,
            format: surface.get_supported_formats(&adapter)[0],
            width: size.width,
            height: size.height,
            present_mode: wgpu::PresentMode::Fifo,
        };
        surface.configure(&device, &config);

        GraphicsContext {
            surface,
            device,
            queue,
            config,
        }
    }
}

For the time being, I also copied the create_render_pipeline function into the module, since it seemed like a good place (in hindsight, better alternative might be the render pass module).

Here’s the full commit for reference.

This is so easy! lol. Not exactly… But we’re moving past a lot the copy + paste stuff now. Time to figure out the render pass.

Render pass

This was the most puzzling part to figure out. What exactly is a render pass in wgpu terms? Why do you need different kinds? Where does it start and end (depth texture, pipeline, buffers, etc)? How would I dynamically swap between one render pipeline and another? There were lots of basic questions that I had to research a bit before being able to answer.

What is a render pass?

A render pass usually represents a single “pass” of the renderer, using a specific rendering pipeline. This article does a great job of breaking it down.

In most 3D engines, there are often several render passes that combine or “composite” into the final image. You can see in the Unreal Engine documentation that they do separate render passes for lighting, shadows, reflections, and the unlit models. Or in the Godot docs, you can see post processing as a render pass.

This video breaks down the rendering process for 10 PS2 games - if you go to Metal Gear Solid or Okami section you can see that they render the all the models and lighting, then spend a few passes on post processing effects. If you want to learn more about the process of breaking down the rendering pipeline, I recommend checking out RenderDocs.

It gets even more complex when you go into the parts of a render pass, like depth stencils, but for now we’ll focus on the high level.

The render pass came from inside the house

But let’s take a few steps back. During the wgpu tutorial, we actually created 2 separate render passes whether you realized or not.

If you look at the tutorial code, we draw the lights - then the 3D models. This process happens by setting a specific render pipeline (lighting or 3D models), doing the draw calls, then rinse repeat with another pipeline (3D models in this case).

// Setup lighting pipeline
render_pass.set_pipeline(&self.light_render_pipeline);
// Draw/calculate the lighting on models
render_pass.draw_light_model(
    &self.obj_model,
    &self.camera_bind_group,
    &self.light_bind_group,
);

// Setup 3D model pipeline
render_pass.set_pipeline(&self.render_pipeline);
// Draw the models
render_pass.draw_model_instanced(
    &self.obj_model,
    0..self.instances.len() as u32,
    &self.camera_bind_group,
    &self.light_bind_group,
);

Why does this matter though (beyond giving us “stacked” effects like lighting, shadows, or post-FX)? As you can see in the example above, the render pipeline is also important to this process. With 2 different pipelines, we get 2 shaders running (the light.wgsl and the main shader).

What is the render pipeline?

The render pipeline is in charge of understanding the pipeline layout (aka any “uniforms” or data we pass to shaders), the shaders we’ll use (vertex, fragment, compute, etc) - as well as the shader code to run (.wgsl files). It also has configurations for how to render the elements, like the cull_mode which can avoid rendering the “back” of models the camera would never see (to save rendering time).

The big thing we can takeaway from the render pipeline is that it defines the uniforms (all the variables we send to shader) — and the actual shader itself.

Why does the render pipeline matter?

So let’s say we want to render our 3D scene with a different shader, like a “toon” shader. What if the shader also required new input (or “uniforms”), like a color or positional data (to help calculate an outline)?

![From Cel Shader wiki

We’d need a whole new pipeline to accomplish this, since our current pipeline doesn’t accommodate for the new uniforms (color + position), and we need to use a different shader file to instruct the pipeline differently.

Sometimes you might even want to do this as a stacked effect (e.g. for post processing). The ability to be able to modularly define a render pass and a pipeline is essential for this.

Creating render passes

In this example I’ll focus on creating a “Phong” pass. Later you could create a PBR (physically based rendering) pass.

Keep in mind though, this won’t actually be a Phong shader… I’m just using that as a placeholder name. We’ll be migrating the existing render pipeline and shader from the wgpu tutorial.

To start, I basically took all the relevant initialization code and shoved it in a new struct called PhongPass. Inside each pass struct we’d store:

Depth texture
Bind groups (the uniform “structure”)
Render pipelines
Buffers for uniform data
Uniform data (optional - could be app-level)

We still need to migrate the render/draw method with the actual render pass.

Because we want to make multiple render passes, we need to create a Trait to define some shared functionality they can all implement. In the pass/mod.rs file, I created a Pass trait that has a draw function. We provide it all the parameters it needs (mostly from the GraphicsContext and the obj_model we’ll be rendering).

pub trait Pass {
    fn draw(
        &mut self,
        surface: &Surface,
        device: &Device,
        queue: &Queue,
        obj_model: &Model,
    ) -> Result<(), wgpu::SurfaceError>;
}

And you guessed it - for now lets just throw most of the render code in there from our [lib.rs](http://lib.rs) file:

impl Pass for PhongPass {
    fn draw(
        &mut self,
        surface: &Surface,
        device: &Device,
        queue: &Queue,
        obj_model: &Model,
    ) -> Result<(), wgpu::SurfaceError> {
                let output = surface.get_current_texture()?;
        let view = output
            .texture
            .create_view(&wgpu::TextureViewDescriptor::default());

        let mut encoder = device.create_command_encoder(&wgpu::CommandEncoderDescriptor {
            label: Some("Render Encoder"),
        });

        {
            let mut render_pass = encoder.begin_render_pass(&wgpu::RenderPassDescriptor {
                label: Some("Render Pass"),
                color_attachments: &[Some(wgpu::RenderPassColorAttachment {
                    view: &view,
                    resolve_target: None,
                    ops: wgpu::Operations {
                        // Set the clear color during redraw
                        // This is basically a background color applied if an object isn't taking up space
                        load: wgpu::LoadOp::Clear(wgpu::Color {
                            r: 0.1,
                            g: 0.2,
                            b: 0.3,
                            a: 1.0,
                        }),
                        store: true,
                    },
                })],
                // Create a depth stencil buffer using the depth texture
                depth_stencil_attachment: Some(wgpu::RenderPassDepthStencilAttachment {
                    view: &self.depth_texture.view,
                    depth_ops: Some(wgpu::Operations {
                        load: wgpu::LoadOp::Clear(1.0),
                        store: true,
                    }),
                    stencil_ops: None,
                }),
            });

                        // ...truncated

Nice! Now we have a good structure to work with. Our app’s render function becomes as simple as:

self.pass.draw(
    &self.ctx.surface,
    &self.ctx.device,
    &self.ctx.queue,
    &self.obj_model,
);

This app should work exactly the same as before — but a lot of our functionality has been split into it’s own module. Now we can do the cool stuff, like rendering more than one model.

So what’s missing from rendering multiple models now? We currently have one buffer for our instance data (instance_buffer), so if we tried to use self.pass.draw() with another 3D model, they’d share the exact same positions (and number of instances).

In order to have different positions, we need to have a buffer setup for each object and use that as a vertex buffer when we render each model.

We also currently do things like pass the camera and light uniform data every time we render an object. Instead, we could batch them into a combined uniform (or bind group) and define them only once (”globally”).

Local vs global uniforms

This is a concept that may be familiar to you if you use other 3D engines. There are shader uniforms that are used by all the shaders (like the camera position), and there are other uniforms that are “local” to the object (like it’s position, color, normals, etc).

It’s a little confusing in our app though, because we setup instancing for models, so our “local” uniform data (like position) is stored there instead. But we do have “local” uniforms in the form of our texture data.

So my goal was to take the camera bind group and make it a “global” bind group. I’d also include the texture sampler in the global uniforms (instead of locals, like we do now), so we don’t waste buffer space on duplicate data.

We’ll also create a “local” bind group. This will contain our object data. You might think we don’t need a position uniform here (since our instances each have a position) — but we could use this position to “offset” all our instances (like a <group> in ThreeJS or a GameObject in Unity that have child objects).

#[repr(C)]
#[derive(Clone, Copy, bytemuck::Pod, bytemuck::Zeroable)]
struct Globals {
    view_position: [f32; 4],
    view_proj: [[f32; 4]; 4],
    ambient: [f32; 4],
}

#[repr(C)]
#[derive(Clone, Copy, bytemuck::Pod, bytemuck::Zeroable)]
struct Locals {
    position: [f32; 4],
    color: [f32; 4],
    normal: [f32; 4],
    lights: [f32; 4],
}

One note: We store the lights in Locals here (I think baryon does similar) - but might be better to lift them to Globals.

Setting up the global + local uniforms

Now to use these uniforms we need to create a bind group layout to describe the data structure, a buffer to transmit data, and a bind group using the layout and buffer.

Since we want to support multiple objects, we’ll also need a way to store multiple buffers and bind groups (since every object will have unique data - like position or color). To do that, we use HashMap to store the bind groups and buffers, and keep track using the object’s “ID” (aka a number or usize).

Note: In this commit/version, I only have 1 uniform buffer, because we don’t have multiple objects yet. We’ll do something special later to handle that.

pub struct PhongPass {
    // Uniforms
    pub global_bind_group_layout: BindGroupLayout,
    pub global_uniform_buffer: wgpu::Buffer,
    pub global_bind_group: wgpu::BindGroup,
    pub local_bind_group_layout: BindGroupLayout,
    local_uniform_buffer: wgpu::Buffer,
    local_bind_groups: HashMap<usize, wgpu::BindGroup>,
        // Other properties
}

impl PhongPass {
    pub fn new(
        phong_config: &PhongConfig,
        device: &wgpu::Device,
        queue: &wgpu::Queue,
        config: &wgpu::SurfaceConfiguration,
        camera: &Camera,
    ) -> PhongPass {
        // Other stuff

        // Initialize global uniforms
        let global_size = mem::size_of::<Globals>() as wgpu::BufferAddress;
      let global_bind_group_layout =
          device.create_bind_group_layout(&wgpu::BindGroupLayoutDescriptor {
              label: Some("[Phong] Globals"),
              entries: &[
                  // Global uniforms
                  wgpu::BindGroupLayoutEntry {
                      binding: 0,
                      visibility: wgpu::ShaderStages::VERTEX | wgpu::ShaderStages::FRAGMENT,
                      ty: wgpu::BindingType::Buffer {
                          ty: wgpu::BufferBindingType::Uniform,
                          has_dynamic_offset: false,
                          min_binding_size: wgpu::BufferSize::new(global_size),
                      },
                      count: None,
                  },
                  // Lights
                  wgpu::BindGroupLayoutEntry {
                      binding: 1,
                      visibility: wgpu::ShaderStages::FRAGMENT,
                      ty: wgpu::BindingType::Buffer {
                          ty: wgpu::BufferBindingType::Uniform,
                          has_dynamic_offset: false,
                          min_binding_size: wgpu::BufferSize::new(light_size),
                      },
                      count: None,
                  },
                  // Sampler for textures
                  wgpu::BindGroupLayoutEntry {
                      binding: 2,
                      visibility: wgpu::ShaderStages::FRAGMENT,
                      ty: wgpu::BindingType::Sampler(wgpu::SamplerBindingType::Filtering),
                      count: None,
                  },
              ],
          });

      // Global uniform buffer
      let global_uniform_buffer = device.create_buffer(&wgpu::BufferDescriptor {
          label: Some("[Phong] Globals"),
          size: global_size,
          usage: wgpu::BufferUsages::UNIFORM | wgpu::BufferUsages::COPY_DST,
          mapped_at_creation: false,
      });
        let global_bind_group = device.create_bind_group(&wgpu::BindGroupDescriptor {
                label: Some("[Phong] Globals"),
                layout: &global_bind_group_layout,
                entries: &[
                    wgpu::BindGroupEntry {
                        binding: 0,
                        resource: global_uniform_buffer.as_entire_binding(),
                    },
                    wgpu::BindGroupEntry {
                        binding: 1,
                        resource: light_buffer.as_entire_binding(),
                    },
                    wgpu::BindGroupEntry {
                        binding: 2,
                        resource: wgpu::BindingResource::Sampler(&sampler),
                    },
                ],
            });
    }
}

We add the bind group layout to our pipeline layout:

// Setup the render pipeline
let pipeline_layout = device.create_pipeline_layout(&wgpu::PipelineLayoutDescriptor {
    label: Some("[Phong] Pipeline"),
    bind_group_layouts: &[&global_bind_group_layout, &local_bind_group_layout],
    push_constant_ranges: &[],
});

And then in our draw method and render pass, we can set the globals first, then the locals. Later, we’ll loop over all objects in the scene and create bind groups and buffers for each object.

render_pass.set_bind_group(0, &self.global_bind_group, &[]);

self.local_bind_groups.entry(0).or_insert_with(|| {
    device.create_bind_group(&wgpu::BindGroupDescriptor {
        label: Some("[Phong] Locals"),
        layout: &self.local_bind_group_layout,
        entries: &[
            wgpu::BindGroupEntry {
                binding: 0,
                resource: self.local_uniform_buffer.as_entire_binding(),
            },
            wgpu::BindGroupEntry {
                binding: 1,
                resource: wgpu::BindingResource::TextureView(
                    &obj_model.materials[0].diffuse_texture.view,
                ),
            },
        ],
    })
});

And we’ll also need to update our shader to accept the Globals now from the correct bind group binding index:

// Define any uniforms we expect from app
struct Globals {
    view_pos: vec4<f32>,
    view_proj: mat4x4<f32>,
    ambient: vec4<f32>,
};
// We create variables for the bind groups
@group(0) @binding(0)
var<uniform> globals: Globals;

You can see the full commit here.

Writing to the buffer

So how do we update the uniforms? The answer is to use the Queue’s write_buffer() method to update a specific buffer (like global_uniform_buffer) with the new data. We also need to cast the data into a buffer friendly data format using bytemuck.

In our app file we can update the camera position like so:

self.ctx.queue.write_buffer(
    &self.pass.global_uniform_buffer,
    0,
    bytemuck::cast_slice(&[self.pass.camera_uniform]),
);

See the full commit here.

Multiple models

We finally made it. It took a lot of setup to get here, but I promise, it was worth it. Now it should be much easier to change our system and render 2 models (or more!).

In our app, let’s change our obj_model property to models and make it a Vec. This will let us store as many models as we need, and even change the size of the vector dynamically throughout the app (to add or remove models).

models: Vec<model::Model>,

Then in our app initialization, we can copy paste our obj_model code using the load_model() function. Make sure to set a different .obj model.

let obj_model = resources::load_model("banana.obj", &ctx.device, &ctx.queue)
    .await
    .expect("Couldn't load model. Maybe path is wrong?");
let cube_model = resources::load_model("cube.obj", &ctx.device, &ctx.queue)
    .await
    .expect("Couldn't load model. Maybe path is wrong?");

let models = vec![obj_model, cube_model];

And in the render method, let’s update the render pass draw call to use the models instead of obj_model:

self.pass.draw(
    &self.ctx.surface,
    &self.ctx.device,
    &self.ctx.queue,
    &self.models,
);

And make sure to change that function signature in the Pass trait and the PhongPass implementation of the trait and function.

pub trait Pass {
    fn draw(
        &mut self,
        surface: &Surface,
        device: &Device,
        queue: &Queue,
                // 👇 Use the vector of models here
        models: &Vec<Model>,
    ) -> Result<(), wgpu::SurfaceError>;
}

And in the PhongPass draw method, we can loop through the models and render each of them. But before we draw them - we need to create the bind group for each object. And when we create the bind group, we also assign each objects texture.

Note: We do this in a separate loop because of Rust mutability, but you could probably get away with 1 loop and wrapping the bind group assignment in a block.

let mut model_index = 0;
for model in models {
    self.local_bind_groups
        .entry(model_index)
        .or_insert_with(|| {
            device.create_bind_group(&wgpu::BindGroupDescriptor {
                label: Some("[Phong] Locals"),
                layout: &self.local_bind_group_layout,
                entries: &[
                    wgpu::BindGroupEntry {
                        binding: 0,
                        resource: self.local_uniform_buffer.as_entire_binding(),
                    },
                    wgpu::BindGroupEntry {
                        binding: 1,
                        resource: wgpu::BindingResource::TextureView(
                            &model.materials[0].diffuse_texture.view,
                        ),
                    },
                ],
            })
        });

    model_index += 1;
}

model_index = 0;
for model in models {
    // Draw the models
    render_pass.draw_model_instanced(
        &model,
        0..*&self.instances.len() as u32,
        &self.local_bind_groups[&model_index],
    );

    model_index += 1;
}

See the full commit here.

This should get multiple objects rendering in the scene. But you’ll notice a couple problems. Both objects are in the same positions! This is because we don’t set an instance buffer per-object, we do it once and then render all objects (so they use the same one). We also haven’t wired up our locals yet, and if you look close, we’re using the same buffer for all objects there too.

Multiple instance buffers

So let’s add the ability to have multiple instance buffers. It’ll look very similar to our bind group setup.

The first thing we need to do is define a new type to combine our Model and Instances types. We can name it Node (but Element, Entity, etc are all good). I created a separate file for it (since we might use it across the app). The parent property is for creating nesting later.

pub struct Node {
    pub parent: u32,
    // local: Matrix?
    pub model: model::Model,
    pub instances: Vec<Instance>,
}

Now instead of having a Vec<Model> - we have a Vec<Node>. And when we initialize our models, we need to provide separate instance data:

let cube_instances = (0..2)
    .map(|z| {
        let z = SPACE_BETWEEN * (z as f32);
        let position = cgmath::Vector3 { x: z, y: 1.0, z };
        let rotation = if position.is_zero() {
            cgmath::Quaternion::from_axis_angle(cgmath::Vector3::unit_z(), cgmath::Deg(0.0))
        } else {
            cgmath::Quaternion::from_axis_angle(position.normalize(), cgmath::Deg(45.0))
        };
        Instance { position, rotation }
    })
    .collect::<Vec<_>>();

let banana_node = Node {
    parent: 0,
    model: obj_model,
    instances: banana_instances,
};

let cube_node = Node {
    parent: 0,
    model: cube_model,
    instances: cube_instances,
};

let models = vec![banana_node, cube_node];

Then in our PhongPass render pass, we use a HashMap to store buffers for each instance.

let mut model_index = 0;
for node in nodes {
        // Bind group code here...

        // Find the instance buffer for this model, or create o ne
    self.instance_buffers.entry(model_index).or_insert_with(|| {
        // We condense the matrix properties into a flat array (aka "raw data")
        // (which is how buffers work - so we can "stride" over chunks)
        let instance_data = node
            .instances
            .iter()
            .map(Instance::to_raw)
            .collect::<Vec<_>>();
        // Create the instance buffer with our data
        let instance_buffer =
            device.create_buffer_init(&wgpu::util::BufferInitDescriptor {
                label: Some("Instance Buffer"),
                contents: bytemuck::cast_slice(&instance_data),
                usage: wgpu::BufferUsages::VERTEX,
            });

        instance_buffer
    });

    model_index += 1;
}

Then when we draw our model, we use the model’s specific instance buffer by using the set_vertex_buffer method before the draw:

model_index = 0;
for node in nodes {
    render_pass.set_vertex_buffer(1, self.instance_buffers[&model_index].slice(..));

And just like that, we have multiple models in our app, each with unique instancing!

// Detect dark theme var iframe = document.getElementById('tweet-1592692904996151296-77'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1592692904996151296&theme=dark" }

Using Local uniforms

We still haven’t fully setup local uniforms in the app yet. The bind group layout and bind group are there, but we don’t have unique buffers for each object. We also don’t have any local data to pass yet, so we need to make some.

Lets update our Node struct to accept a locals property:

use crate::{instance::Instance, model, pass::phong::Locals};

// This represents a 3D model in a scene.
// It contains the 3D model, instance data, and a parent ID (TBD)
pub struct Node {
    // ID of parent Node
    pub parent: u32,
    // Local position of model (for relative calculations)
    pub locals: Locals,
    // The vertex buffers and texture data
    pub model: model::Model,
    // An array of positional data for each instance (can just pass 1 instance)
    pub instances: Vec<Instance>,
}

Now when we initialize the nodes, we need to provide local data. We can just use our Locals struct from our render pass file and initialize it with dummy data. Here I provide a blue color to both objects (so we can check for it later in the shader):

// Create the nodes
let banana_node = Node {
    parent: 0,
    locals: Locals {
        position: [0.0, 0.0, 0.0, 0.0],
        color: [0.0, 0.0, 1.0, 1.0],
        normal: [0.0, 0.0, 0.0, 0.0],
        lights: [0.0, 0.0, 0.0, 0.0],
    },
    model: obj_model,
    instances: banana_instances,
};

let cube_node = Node {
    parent: 0,
    locals: Locals {
        position: [0.0, 0.0, 0.0, 0.0],
        color: [0.0, 0.0, 1.0, 1.0],
        normal: [0.0, 0.0, 0.0, 0.0],
        lights: [0.0, 0.0, 0.0, 0.0],
    },
    model: cube_model,
    instances: cube_instances,
};

We have local data, now we need to create a new buffer for each object. We could do basically what we did for the instances and create the buffers in the render loop using a HashMap to store them. But after looking at baryon, I noticed they use a “uniform pool” to handle this.

It basically does the same thing as our instance buffer code, but instead of using a HashMap, we use a Vec. This UniformPool struct will also help us add “helper methods” like update_uniform() to simplify writing to the buffers.

/// Uniform buffer pool
/// Used by render passes to keep track of each objects local uniforms
/// and provides a way to update uniforms to render pipeline
pub struct UniformPool {
    label: &'static str,
    pub buffers: Vec<wgpu::Buffer>,
    size: u64,
}

impl UniformPool {
    pub fn new(label: &'static str, size: u64) -> Self {
        Self {
            label,
            buffers: Vec::new(),
            size,
        }
    }

    pub fn alloc_buffers(&mut self, count: usize, device: &Device) {
        // We reset the buffers each time we allocate
        // TODO: Ideally we should keep track of the object it belongs to,
        // so we can add/remove objects (and their uniform buffers) dynamically
        self.buffers = Vec::new();

        for _ in 0..count {
            let local_uniform_buffer = device.create_buffer(&wgpu::BufferDescriptor {
                label: Some(&self.label),
                size: self.size,
                usage: wgpu::BufferUsages::UNIFORM | wgpu::BufferUsages::COPY_DST,
                mapped_at_creation: false,
            });
            self.buffers.push(local_uniform_buffer);
        }
    }

    pub fn update_uniform<T: bytemuck::Pod>(&self, index: usize, data: T, queue: &Queue) {
        if &self.buffers.len() > &0 {
            queue.write_buffer(&self.buffers[index], 0, bytemuck::cast_slice(&[data]));
        }
    }
}

Now in our PhongPass struct we can use our UniformPool instead of our local_uniform_buffer.

And in our draw method, we first initialize the buffers for each object:

// Allocate buffers for local uniforms
if (self.uniform_pool.buffers.len() < nodes.len()) {
    self.uniform_pool.alloc_buffers(nodes.len(), &device);
}

Then when we loop over our nodes and create the bind groups for each object, we can assign each object’s unique buffer:

// Loop over the nodes/models in a scene and setup the specific models
// local uniform bind group and instance buffers to send to shader
// This is separate loop from the render because of Rust ownership
// (can prob wrap in block instead to limit mutable use)
let mut model_index = 0;
for node in nodes {
    let local_buffer = &self.uniform_pool.buffers[model_index];

    // We create a bind group for each model's local uniform data
    // and store it in a hash map to look up later
    self.local_bind_groups
        .entry(model_index)
        .or_insert_with(|| {
            device.create_bind_group(&wgpu::BindGroupDescriptor {
                label: Some("[Phong] Locals"),
                layout: &self.local_bind_group_layout,
                entries: &[
                    wgpu::BindGroupEntry {
                        binding: 0,
                                                // 👇 We use the buffer from UniformPool here
                        resource: local_buffer.as_entire_binding(),
                    },
                    wgpu::BindGroupEntry {
                        binding: 1,
                        resource: wgpu::BindingResource::TextureView(
                            &node.model.materials[0].diffuse_texture.view,
                        ),
                    },
                ],
            })
        });

And since we passed our local bind groups to our draw method previously like so:

// Draw all the model instances
render_pass.draw_model_instanced(
    &node.model,
    0..*&node.instances.len() as u32,
    &self.local_bind_groups[&model_index],
);

We can start using the local uniforms in our shader! Let’s open up the shader.wgsl file and add our local uniforms:

// Define any uniforms we expect from app
struct Globals {
    view_pos: vec4<f32>,
    view_proj: mat4x4<f32>,
    ambient: vec4<f32>,
};
struct Locals {
    position:  vec4<f32>,
    color:  vec4<f32>,
    normal:  vec4<f32>,
    lights:  vec4<f32>,
}
// We create variables for the bind groups
@group(0) @binding(0)
var<uniform> globals: Globals;
@group(1) @binding(0)
var<uniform> locals: Locals;

And in our fragment portion of our shader, we can use the locals.color to pass the color from our node all the way to the shader. We can use it directly:

return locals.color;

Let’s use it to colorize our texture to a certain hue.

return locals.color * vec4<f32>(result, object_color.a);

You should see your texture with a blue hue applied!

Updating local uniforms

So how do we update a local uniform, like moving an object while the app is running? It’s very similar to the process for globals: write the new data to the appropriate buffer.

In our app’s update() method, let’s loop over our nodes and change the local data, then send that to the buffer using our UniformPool.update_uniform() method:

// Update local uniforms
let mut node_index = 0;
for node in &mut self.nodes {
    node.locals.color = [
        node.locals.color[0],
        (node.locals.color[1] + 0.001),
        (node.locals.color[2] - 0.001),
        node.locals.color[3],
    ];
    &self
        .pass
        .uniform_pool
        .update_uniform(node_index, node.locals, &self.ctx.queue);
    node_index += 1;
}

This should animate the color of your 3D object to go from blue to green (since we’re updating the RGBA values and adding/subtracting from the GB parts).

// Detect dark theme var iframe = document.getElementById('tweet-1593409473393045504-409'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1593409473393045504&theme=dark" }

You still here? 👀

This was quite the long and arduous process just to get another 3D model running, but now we should have a nice architecture in place to do cooler stuff. We could create a cartoon or PBR render pass, or start to create a scene with nested objects with relative positioning to their parents. And did I mention all this code runs native and on the web? The potential is endless (or as much as the WebGPU spec allows for anyway).

Want to share your progress or ask any questions? Feel free to reach out to me on Twitter or Mastodon. Also make sure to check out the Rust Game Development group’s Discord channel, there’s lots of cool people on there that work on wgpu too. And thanks again to the author of Baryon for such a great resource on advanced wgpu architecture.

References

Parsing JSON with Rust

Ryosuke — Fri, 04 Nov 2022 18:59:12 +0000

As I’ve been learning Rust, I’ve been looking for easy and practical projects to get my head around basic concepts and increase my syntax muscle memory. One of the things I find myself doing everyday in most languages is loading data from different sources, like text files, or more structured sources like JSON or YAML.

So it got me thinking, how would parsing JSON look like in Rust? Not from scratch though — we’re not interested in building a parser here. Ideally we’d use either internal Rust methods or a 3rd party crate.

In this article I’ll go over how I used serde and serde-json to read, parse, and serialize JSON into Rust structs to use inside Rust apps. You can find the final source code on Github.

Setting up a new project

Let’s create a new Rust app to code inside. You could also clone this branch and skip this step.

cargo new json-parser

Once you navigate inside, install serde and serde_json. We’ll also add the derive feature for serde, which we’ll use later.

cargo add serde --features derive
cargo add serde_json

Now let’s make our parser! Create a new file at src/parser.rs that’ll hold all our parsing logic.

If you look at the serde_json README file, you can find some fantastic examples on how to use the library out of the box. Let’s try one of those to see if we installed everything correctly:

use serde_json::{Result, Value};

pub fn untyped_example() -> Result<()> {
    // Some JSON input data as a &str. Maybe this comes from the user.
    let data = r#"
        {
            "name": "John Doe",
            "age": 43,
            "phones": [
                "+44 1234567",
                "+44 2345678"
            ]
        }"#;

    // Parse the string of data into serde_json::Value.
    let v: Value = serde_json::from_str(data)?;

    // Access parts of the data by indexing with square brackets.
    println!("Please call {} at the number {}", v["name"], v["phones"][0]);

    Ok(())
}

Now we can use this function inside of our src/main.rs

// "Import" anything public in the parser module
pub mod parser;

fn main() {
    println!("Hello, world!");

    // Parse the JSON
    let result = parser::untyped_example();

    // Handle errors from the parser if any
    match result {
        Ok(result) => (),
        Err(error) => print!("{}", error),
    }
}

Try running this with cargo run. You should see the text “Please call John Doe at…” in your console. If not, we also added an error handling match that should return any errors from serde.

You can see from this example the library API is fairly easy to use. They expose a from_str() method that parses JSON from a string, which you can provide inline (like we did above) - or load from a local or remote file (we’ll do this later). Once the JSON is parsed, you can access the data through the properties or keys in the JSON (e.g. v["name"] grabs from { "name": "John Doe" }).

Let’s see what else we can do with the library now that we have it integrated into our app.

Loading JSON from local file

In order to load JSON, we need a JSON file. So let’s create one. At the root of the project create a folder called data. Inside it, create a test.json file.

{
    "name": "John Doe",
    "age": 43,
    "phones": [
        "+44 1234567",
        "+44 2345678"
    ]
}

To load data, we can use the FileSystem API from Rust’s standard library. We’ll do this in our main.rs file and pass the data (aka a long string of JSON) to the parser.

// src/main.rs
use std::fs;

pub mod parser;

fn main() {
    println!("Hello, world!");

    // Grab JSON file
    let file_path = "data/test.json".to_owned();
    let contents = fs::read_to_string(file_path).expect("Couldn't find or load that file.");

    parser::untyped_example(&contents);
}

And we’d need to modify the parser to accept data now. You can also delete the data variable with the inline JSON since we don’t need it anymore:

// src/parser.rs
pub fn untyped_example(json_data: &str) -> Result<()> {
    let v: Value = serde_json::from_str(json_data)?;

Try running this with cargo run and you should get the same result as before (the “Please call…” message).

Typed data

But what if we know what the structure of our data is before we parse it? For example, we might want to parse a theme using the System UI / Styled System specification, like Chakra UI.

This would let us access our data using strictly typed structs - so instead of using v["name"] to access the name, we could get autocompletion in our IDE when we type v. and it’d finish to v.name. This is a much better developer experience, and creates more safety nets against using properties that don’t exist.

The serde_json README also provides a great example for handling typed data. We can copy paste this entirely into the [parser.rs](http://parser.rs) file.

use serde::{Deserialize, Serialize};
use serde_json::Result;

#[derive(Serialize, Deserialize)]
struct Person {
    name: String,
    age: u8,
    phones: Vec<String>,
}

pub fn typed_example() -> Result<()> {
    // Some JSON input data as a &str. Maybe this comes from the user.
    let data = r#"
        {
            "name": "John Doe",
            "age": 43,
            "phones": [
                "+44 1234567",
                "+44 2345678"
            ]
        }"#;

    // Parse the string of data into a Person object. This is exactly the
    // same function as the one that produced serde_json::Value above, but
    // now we are asking it for a Person as output.
    let p: Person = serde_json::from_str(data)?;

    // Do things just like with any other Rust data structure.
    println!("Please call {} at the number {}", p.name, p.phones[0]);

    Ok(())
}

Then replace the untyped_example function with the typed_example function in your main.rs file.

pub mod parser;

fn main() {
    println!("Hello, world!");

        // You can keep the file system stuff, I removed it for simplicity
        // we'll use it later in the tutorial

    parser::typed_example();
}

How to handle object types?

So the first question that came to mind after looking at the examples — how do you handle an object with keys and properties? It seems you can use a HashMap<> type and provide a type for the object key (usually a String) and the object value (anything — sometimes a String - maybe a i32 for numbers).

So say I had a JSON theme that looked like this:

{
  "animation": {
    "default": "400ms ease-in",
    "fast": "300ms ease-in"
  },
  "breakpoints": {
    "mobile": "320px",
    "tablet": "768px",
    "computer": "992px",
    "desktop": "1200px",
    "widescreen": "1920px"
  },
  "colors": {
    "text": "#111212",
    "background": "#fff",
    "primary": "#005CDD",
    "secondary": "#6D59F0",
    "muted": "#f6f6f9",
    "gray": "#D3D7DA",
    "highlight": "hsla(205, 100%, 40%, 0.125)",
    "white": "#FFF",
    "black": "#111212"
  },

  "fonts": {
    "body": "Roboto, Helvetiva Neue, Helvetica, Aria, sans-serif",
    "heading": "Archivo, Helvetiva Neue, Helvetica, Aria, sans-serif",
    "monospace": "Menlo, monospace"
  },
  "font_sizes": [12, 14, 16, 20, 24, 32, 48, 64, 96],
  "font_weights": {
    "body": 400,
    "heading": 500,
    "bold": 700
  },
  "line_heights": {
    "body": 1.5,
    "heading": 1.25
  },
  "space": [0, 4, 8, 16, 32, 64, 128, 256, 512],
  "sizes": {
    "avatar": 48
  },
  "radii": {
    "default": 0,
    "circle": 99999
  },
  "shadows": {
    "card": {
      "light": "15px 15px 35px rgba(0, 127, 255, 0.5)"
    }
  },
  "gradients": {
    "primary": "linear-gradient()"
  }
}

You could structure that type to look like this:

use std::collections::HashMap;

use serde::{Deserialize, Serialize};
use serde_json::Result;

#[derive(Serialize, Deserialize)]
struct Theme {
    colors: HashMap<String, String>,
    space: Vec<i32>,
    font_sizes: Vec<i32>,
    fonts: HashMap<String, String>,
    font_weights: HashMap<String, i32>,
    line_heights: HashMap<String, f32>,
    breakpoints: HashMap<String, String>,
    animation: HashMap<String, String>,
    gradients: HashMap<String, String>,
}

You can see I use String for any string based values, i32 for numbers, and specifically f32 for any “floats” aka numbers with decimals.

When the JSON is parsed, a HashMap is returned, so you can access the data inside using the get() method - or loop over all the values using for loop:

// Get a single value
println!("Black: {}", p.colors.get("black"));

// Loop over all the colors
for (key, color) in p.colors {
    // Create the custom property
    let custom_property = format!("--colors-{}", key);
    let css_rule = format!("{}: {};", &custom_property, color);

    // @TODO: Export a CSS stylesheet file (or return CSS)
    println!("{}", css_rule);
    stylesheet.push(css_rule);

    // Add the custom property
    theme_tokens.colors.insert(key, custom_property);
}

This works pretty well! You can see here we’re able to loop over the colors and even generate CSS custom properties based on the key and value from the HashMap (aka the color name and value).

Handling optional / multiple types

But what if we have optional types? Or multiple types for the same property (like a “size” unit that could be an number 10 or a string like 10px)? In Typescript we’d be able to just create a type like this type Size = string | number. In Rust, the equivalent of this would be an Enum.

After researching a bit I found that serde supports Enum types if you pass the untagged macro to them:

#[derive(Serialize, Deserialize, Debug)]
#[serde(untagged)]
enum Colors {
    Name(String),
    Number(i32),
}

#[derive(Serialize, Deserialize)]
struct Theme {
    test: Colors,
}

// ... after parsing
let p: Theme = serde_json::from_str(json_data)?;
println!("{:#?}", p.test);

Add the following property to your JSON:

{
    "test": 200
}

And serde will grab from the Number(i32) Enum property and return that — you’ll need to do a match statement to figure out what it is + get value back:

match p.test {
    // We don't want the name so we do nothing by passing empty tuple
    Name(val) -> (),
    Number(num) -> println!("Theme Color is number: {}", num),
}

This works great too! You can easily create some “dynamic” types and still have fairly strict handling of them based on their type.

Putting the eyy back in JSON

I hope this gives you a good idea of how to parse JSON in Rust using the serde crate, and how to handle different use cases and data types. There’s lot of cool apps you can create using JSON (or other file types - serde supports lots like YAML, TOML, and more).

As always, you can find the full code for this tutorial up on my Github.

If you have any questions or want to share what you’ve been working on, feel free to hit me up on Twitter.

Cheers,
Ryo

Primitive Geometry in wgpu and Rust

Ryosuke — Sat, 15 Oct 2022 00:27:47 +0000

I recently found the Learn WGPU tutorial and finished it, which teaches you how to write a custom 3D rendering engine using the wgpu library to interface with WebGPU.

At the end of the tutorial you essentially have a grid of instanced 3D .obj models, a camera, light, and a couple shaders. Since the tutorial did such an excellent job breaking everything down, I wanted to practice a bit more and figure out how to add primitive 3D objects to the scene (like a cube, plane, or sphere).

// Detect dark theme var iframe = document.getElementById('tweet-1578653432872767488-920'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1578653432872767488&theme=dark" }

After plenty of helpful error codes (and a bit of frustration with non-existent errors), I was finally got a simple example working. I thought I’d share my findings for other fellow graphic programmers.

⚠️ This tutorial assumes you’ve followed and completed the “Learn wgpu” tutorial. You could also use my repo here as a starting point - but it will require Rust and Trunk setup in your local dev environment. You can also checkout the final code here for reference.

Research

Before I dove too deep into the process, I was curious how other libraries handled their primitive geometry in Rust. I looked into Bevy and Baryon, the former being a well established (and huge) game engine and the latter being a smaller, easier to digest library.

Bevy Primitives

First things first, Bevy is a workspace made up of a lot of sub crates (aka “monorepo with Yarn workspaces” in JS). You can find the logic for primitives in the bevy-render crate under the mesh shape module. I found this by searching “Cube” on GitHub, scrolling past all the examples, and finding the library it used. You could also trace it back in your Rust IDE if you have a Bevy project setup (e.g. using “Go to Definition” in VSCode).

The method for creating the cube (or “Box” in this case) is done using an implementation (impl) that extends the Mesh struct to accept a Box struct. This is similar to a builder pattern, where you have a struct that handles accepting user configuration (in this case, the box sizing), and then that gets turned into the true product: a Mesh.

You can see they create buffers for vertices and indices using a “min” and “max” size property:

impl From<Box> for Mesh {
    fn from(sp: Box) -> Self {
        let vertices = &[
            // Top
            ([sp.min_x, sp.min_y, sp.max_z], [0., 0., 1.0], [0., 0.]),
            ([sp.max_x, sp.min_y, sp.max_z], [0., 0., 1.0], [1.0, 0.]),
            ([sp.max_x, sp.max_y, sp.max_z], [0., 0., 1.0], [1.0, 1.0]),
            ([sp.min_x, sp.max_y, sp.max_z], [0., 0., 1.0], [0., 1.0]),
            // Bottom
            ([sp.min_x, sp.max_y, sp.min_z], [0., 0., -1.0], [1.0, 0.]),
            ([sp.max_x, sp.max_y, sp.min_z], [0., 0., -1.0], [0., 0.]),
            ([sp.max_x, sp.min_y, sp.min_z], [0., 0., -1.0], [0., 1.0]),
            ([sp.min_x, sp.min_y, sp.min_z], [0., 0., -1.0], [1.0, 1.0]),
            // Right
            ([sp.max_x, sp.min_y, sp.min_z], [1.0, 0., 0.], [0., 0.]),
            ([sp.max_x, sp.max_y, sp.min_z], [1.0, 0., 0.], [1.0, 0.]),
            ([sp.max_x, sp.max_y, sp.max_z], [1.0, 0., 0.], [1.0, 1.0]),
            ([sp.max_x, sp.min_y, sp.max_z], [1.0, 0., 0.], [0., 1.0]),
            // Left
            ([sp.min_x, sp.min_y, sp.max_z], [-1.0, 0., 0.], [1.0, 0.]),
            ([sp.min_x, sp.max_y, sp.max_z], [-1.0, 0., 0.], [0., 0.]),
            ([sp.min_x, sp.max_y, sp.min_z], [-1.0, 0., 0.], [0., 1.0]),
            ([sp.min_x, sp.min_y, sp.min_z], [-1.0, 0., 0.], [1.0, 1.0]),
            // Front
            ([sp.max_x, sp.max_y, sp.min_z], [0., 1.0, 0.], [1.0, 0.]),
            ([sp.min_x, sp.max_y, sp.min_z], [0., 1.0, 0.], [0., 0.]),
            ([sp.min_x, sp.max_y, sp.max_z], [0., 1.0, 0.], [0., 1.0]),
            ([sp.max_x, sp.max_y, sp.max_z], [0., 1.0, 0.], [1.0, 1.0]),
            // Back
            ([sp.max_x, sp.min_y, sp.max_z], [0., -1.0, 0.], [0., 0.]),
            ([sp.min_x, sp.min_y, sp.max_z], [0., -1.0, 0.], [1.0, 0.]),
            ([sp.min_x, sp.min_y, sp.min_z], [0., -1.0, 0.], [1.0, 1.0]),
            ([sp.max_x, sp.min_y, sp.min_z], [0., -1.0, 0.], [0., 1.0]),
        ];

        let positions: Vec<_> = vertices.iter().map(|(p, _, _)| *p).collect();
        let normals: Vec<_> = vertices.iter().map(|(_, n, _)| *n).collect();
        let uvs: Vec<_> = vertices.iter().map(|(_, _, uv)| *uv).collect();

        let indices = Indices::U32(vec![
            0, 1, 2, 2, 3, 0, // top
            4, 5, 6, 6, 7, 4, // bottom
            8, 9, 10, 10, 11, 8, // right
            12, 13, 14, 14, 15, 12, // left
            16, 17, 18, 18, 19, 16, // front
            20, 21, 22, 22, 23, 20, // back
        ]);

        let mut mesh = Mesh::new(PrimitiveTopology::TriangleList);
        mesh.insert_attribute(Mesh::ATTRIBUTE_POSITION, positions);
        mesh.insert_attribute(Mesh::ATTRIBUTE_NORMAL, normals);
        mesh.insert_attribute(Mesh::ATTRIBUTE_UV_0, uvs);
        mesh.set_indices(Some(indices));
        mesh
    }
}

They ultimately return a Mesh type, which contains the buffers and attributes (like position, normal, etc). Then when you use it in your app, the Mesh is placed inside their ECS system (kinda like a node tree) so it can be rendered.

Baryon Primitives

Baryon’s system was very similar. They use a Geometry struct that acts as the overarching parent for the primitive types. Then the primitives extend it, and implement (impl) their own methods for generating geometry (like cuboid() for creating a cube).

In this case, the size of the cube is also dynamic with a vector you can pass of X,Y,Z scaling. The function returns the vertex and index buffers, as well as normal attributes as a Geometry struct.

use std::iter;

impl super::Geometry {
    pub fn cuboid(streams: super::Streams, half_extent: mint::Vector3<f32>) -> Self {
        let pos = |x, y, z| {
            crate::Position([
                (x as f32) * half_extent.x,
                (y as f32) * half_extent.y,
                (z as f32) * half_extent.z,
            ])
        };

        // bounding radius is half of the diagonal length
        let radius = (half_extent.x * half_extent.x
            + half_extent.y * half_extent.y
            + half_extent.z * half_extent.z)
            .sqrt();

        if streams.contains(super::Streams::NORMAL) {
            let positions = vec![
                // top (0, 0, 1)
                pos(-1, -1, 1),
                pos(1, -1, 1),
                pos(1, 1, 1),
                pos(-1, 1, 1),
                // bottom (0, 0, -1)
                pos(-1, 1, -1),
                pos(1, 1, -1),
                pos(1, -1, -1),
                pos(-1, -1, -1),
                // right (1, 0, 0)
                pos(1, -1, -1),
                pos(1, 1, -1),
                pos(1, 1, 1),
                pos(1, -1, 1),
                // left (-1, 0, 0)
                pos(-1, -1, 1),
                pos(-1, 1, 1),
                pos(-1, 1, -1),
                pos(-1, -1, -1),
                // front (0, 1, 0)
                pos(1, 1, -1),
                pos(-1, 1, -1),
                pos(-1, 1, 1),
                pos(1, 1, 1),
                // back (0, -1, 0)
                pos(1, -1, 1),
                pos(-1, -1, 1),
                pos(-1, -1, -1),
                pos(1, -1, -1),
            ];

            let normals = [
                crate::Normal([0.0, 0.0, 1.0]),
                crate::Normal([0.0, 0.0, -1.0]),
                crate::Normal([1.0, 0.0, 0.0]),
                crate::Normal([-1.0, 0.0, 0.0]),
                crate::Normal([0.0, 1.0, 0.0]),
                crate::Normal([0.0, -1.0, 0.0]),
            ]
            .iter()
            .flat_map(|&n| iter::repeat(n).take(4))
            .collect::<Vec<_>>();

            let indices = vec![
                0u16, 1, 2, 2, 3, 0, // top
                4, 5, 6, 6, 7, 4, // bottom
                8, 9, 10, 10, 11, 8, // right
                12, 13, 14, 14, 15, 12, // left
                16, 17, 18, 18, 19, 16, // front
                20, 21, 22, 22, 23, 20, // back
            ];

            Self {
                radius,
                positions,
                normals: Some(normals),
                indices: Some(indices),
            }
        } else {
            let positions = vec![
                // top (0, 0, 1)
                pos(-1, -1, 1),
                pos(1, -1, 1),
                pos(1, 1, 1),
                pos(-1, 1, 1),
                // bottom (0, 0, -1)
                pos(-1, 1, -1),
                pos(1, 1, -1),
                pos(1, -1, -1),
                pos(-1, -1, -1),
            ];

            let indices = vec![
                0u16, 1, 2, 2, 3, 0, // top
                4, 5, 6, 6, 7, 4, // bottom
                6, 5, 2, 2, 1, 6, // right
                0, 3, 4, 4, 7, 0, // left
                5, 4, 3, 3, 2, 5, // front
                1, 0, 7, 7, 6, 1, // back
            ];

            Self {
                radius,
                positions,
                normals: None,
                indices: Some(indices),
            }
        }
    }
}

Baryon also used an ECS system to contain multiple “objects” (or Geometry) and render them through the pipeline.

Cool, it seems simple enough, get some vertices and indices - maybe some normals, and we’re good to go…right?

Primitives in wgpu

The wrong way

Early in the “Learn WGPU” tutorial you end up making a very simple vertex buffer to make a triangle, and eventually an octagon (using the index buffer). I initially started copying that code as the basis, but I quickly found that I was duplicating a lot of the code I already had (like creating a new Vertex implementation — PrimitiveVertex).

I created a PrimitiveMesh struct that would hold our buffers and allow us to instantiate multiple objects. Then I created a PrimitiveVertex to contain the position and color and extended the Vertex trait. I put this in it’s own core module, and then I created another module to contain the cube code.

// src/primitives/mod.rs
use crate::Vertex;
use std::ops::Range;
use wgpu::util::DeviceExt;
pub mod cube;

/// General methods shared with all primitives
#[repr(C)]
#[derive(Copy, Clone, Debug, bytemuck::Pod, bytemuck::Zeroable)]
pub struct PrimitiveVertex {
    pub position: [f32; 3],
    pub color: [f32; 3],
}

impl Vertex for PrimitiveVertex {
    fn desc<'a>() -> wgpu::VertexBufferLayout<'a> {
        use std::mem;
        wgpu::VertexBufferLayout {
            array_stride: mem::size_of::<PrimitiveVertex>() as wgpu::BufferAddress,
            step_mode: wgpu::VertexStepMode::Vertex,
            attributes: &[
                // Position
                wgpu::VertexAttribute {
                    offset: 0,
                    shader_location: 0,
                    format: wgpu::VertexFormat::Float32x3,
                },
                // Color
                wgpu::VertexAttribute {
                    offset: std::mem::size_of::<[f32; 3]>() as wgpu::BufferAddress,
                    shader_location: 1,
                    format: wgpu::VertexFormat::Float32x3,
                },
            ],
        }
    }
}
pub struct PrimitiveMesh {
    pub vertex_buffer: wgpu::Buffer,
    num_vertices: u32,
}

impl PrimitiveMesh {
    pub fn new(device: &wgpu::Device, vertices: &[PrimitiveVertex]) -> Self {
        let num_vertices = vertices.len() as u32;
        let vertex_buffer = device.create_buffer_init(&wgpu::util::BufferInitDescriptor {
            label: Some("Primitive Vertex Buffer"),
            contents: bytemuck::cast_slice(vertices),
            usage: wgpu::BufferUsages::VERTEX,
        });

        Self {
            num_vertices,
            vertex_buffer,
        }
    }
}

The cube code was very simple. I imported the new PrimitiveVertex type and created an array of them, initializing each with a triangle’s position:

// src/primitives/cube.rs
use super::PrimitiveVertex;

pub const CUBE_VERTICES: &[PrimitiveVertex] = &[
    PrimitiveVertex {
        position: [0.0, 0.5, 0.0],
        color: [1.0, 0.0, 0.0],
    },
    PrimitiveVertex {
        position: [-0.5, -0.5, 0.0],
        color: [0.0, 1.0, 0.0],
    },
    PrimitiveVertex {
        position: [0.5, -0.5, 0.0],
        color: [0.0, 0.0, 1.0],
    },
];

I realized I needed a custom draw method to handle the new mesh type (PrimitiveMesh vs ModelMesh) so I copied the structure from the model.rs and removed things like the index buffer (since we didn’t need it yet).

// src/primitives/mod.rs

// ...Code from above...

pub trait DrawPrimitive<'a> {
    fn draw_primitive(
        &mut self,
        mesh: &'a PrimitiveMesh,
        camera_bind_group: &'a wgpu::BindGroup,
        light_bind_group: &'a wgpu::BindGroup,
        play_bind_group: &'a wgpu::BindGroup,
    );
    fn draw_primitive_instanced(
        &mut self,
        mesh: &'a PrimitiveMesh,
        instances: Range<u32>,
        camera_bind_group: &'a wgpu::BindGroup,
        light_bind_group: &'a wgpu::BindGroup,
        play_bind_group: &'a wgpu::BindGroup,
    );
}

impl<'a, 'b> DrawPrimitive<'b> for wgpu::RenderPass<'a>
where
    'b: 'a,
{
    fn draw_primitive(
        &mut self,
        mesh: &'b PrimitiveMesh,
        camera_bind_group: &'b wgpu::BindGroup,
        light_bind_group: &'b wgpu::BindGroup,
        play_bind_group: &'a wgpu::BindGroup,
    ) {
        self.draw_primitive_instanced(
            mesh,
            0..1,
            camera_bind_group,
            light_bind_group,
            play_bind_group,
        );
    }

    fn draw_primitive_instanced(
        &mut self,
        mesh: &'b PrimitiveMesh,
        instances: Range<u32>,
        camera_bind_group: &'b wgpu::BindGroup,
        light_bind_group: &'b wgpu::BindGroup,
        play_bind_group: &'a wgpu::BindGroup,
    ) {
        self.set_vertex_buffer(0, mesh.vertex_buffer.slice(..));
        // self.set_index_buffer(mesh.index_buffer.slice(..), wgpu::IndexFormat::Uint32);
        self.set_bind_group(1, camera_bind_group, &[]);
        self.set_bind_group(2, light_bind_group, &[]);
        self.set_bind_group(3, play_bind_group, &[]);
        self.draw(0..mesh.num_vertices, instances);
    }
}

With all this code in place, we could toss it into our render flow. It’d require a bit of setup though.

Initialize the cube struct, which also initializes the buffers.
Save the cube in our app state (so we can use it during the render)
Add an extra draw call for the cube in our render method

And it should work! But it didn’t.

I kept getting errors about the pipeline not supporting my vertex type. This led me down a rabbit hole.

Added the PrimitiveVertex to the render pipeline, this didn’t solve it.
Created a new render pipeline called primitive_render_pipeline for the PrimitiveVertex. I initialized it, added it to the app state, applied it before my primitive draw call and…new errors.
I thought I might have needed to implement the index buffer, so I created a very simple index buffer (vec![0,1,2]), initialized it, and added it to the PrimitiveMesh struct. That didn’t seem to solve things.
I realized that the render pipeline also defines the shader used for the mesh. Because I was using the same shader as the OBJ model, it was expecting much more data from the buffers and bind groups. At this point I could have created a specific shader to accomdate the simpler PrimitiveVertex - but I figured users would want to do things like texture their primitives (which make the current shader work fine — if I gave it the right data…)

Ultimately I started to get some code smells so I stopped exploring this branch, but you can find an error-free (but still broken) version here.

The right way

Initially the triangle was simple because out render pipeline was also simple. Now our render pipeline was expecting a very specific shape of data, like ModelVertex or bind groups like a texture — all things we got for “free” by importing an .obj model.

In order to render a primitive in this pipeline, we need to create a similar structure of data. So I started looking at the .obj parser that imports models into the app (inside resources.rs). I noticed they were doing a very similar process to the one I was trying — creating vertex + indices buffers (and bind groups in their case).

pub async fn load_model(
    file_name: &str,
    device: &wgpu::Device,
    queue: &wgpu::Queue,
    layout: &wgpu::BindGroupLayout,
) -> anyhow::Result<model::Model> {
        // Load the obj file into a buffer
    let obj_text = load_string(file_name).await?;
    let obj_cursor = Cursor::new(obj_text);
    let mut obj_reader = BufReader::new(obj_cursor);

        // Use tobj crate to load the obj from the file buffer we created
    let (models, obj_materials) = tobj::load_obj_buf_async(
        &mut obj_reader,
        &tobj::LoadOptions {
            triangulate: true,
            single_index: true,
            ..Default::default()
        },
        |p| async move {
            let mat_text = load_string(&p).await.unwrap();
            tobj::load_mtl_buf(&mut BufReader::new(Cursor::new(mat_text)))
        },
    )
    .await?;

        // Loop over the materials , create bind groups, and add it to a vec
    let mut materials = Vec::new();
    for m in obj_materials? {
        let diffuse_texture = load_texture(&m.diffuse_texture, device, queue).await?;
        let bind_group = device.create_bind_group(&wgpu::BindGroupDescriptor {
            layout,
            entries: &[
                wgpu::BindGroupEntry {
                    binding: 0,
                    resource: wgpu::BindingResource::TextureView(&diffuse_texture.view),
                },
                wgpu::BindGroupEntry {
                    binding: 1,
                    resource: wgpu::BindingResource::Sampler(&diffuse_texture.sampler),
                },
            ],
            label: None,
        });

        materials.push(model::Material {
            name: m.name,
            diffuse_texture,
            bind_group,
        })
    }

        // Loop over models, create vertex and index buffers, and push all meshes into vec
        // This assumes there may be multiple objects in a single OBJ file
    let meshes = models
        .into_iter()
        .map(|m| {
            let vertices = (0..m.mesh.positions.len() / 3)
                .map(|i| model::ModelVertex {
                    position: [
                        m.mesh.positions[i * 3],
                        m.mesh.positions[i * 3 + 1],
                        m.mesh.positions[i * 3 + 2],
                    ],
                    tex_coords: [m.mesh.texcoords[i * 2], m.mesh.texcoords[i * 2 + 1]],
                    normal: [
                        m.mesh.normals[i * 3],
                        m.mesh.normals[i * 3 + 1],
                        m.mesh.normals[i * 3 + 2],
                    ],
                })
                .collect::<Vec<_>>();

            let vertex_buffer = device.create_buffer_init(&wgpu::util::BufferInitDescriptor {
                label: Some(&format!("{:?} Vertex Buffer", file_name)),
                contents: bytemuck::cast_slice(&vertices),
                usage: wgpu::BufferUsages::VERTEX,
            });
            let index_buffer = device.create_buffer_init(&wgpu::util::BufferInitDescriptor {
                label: Some(&format!("{:?} Index Buffer", file_name)),
                contents: bytemuck::cast_slice(&m.mesh.indices),
                usage: wgpu::BufferUsages::INDEX,
            });

            model::Mesh {
                name: file_name.to_string(),
                vertex_buffer,
                index_buffer,
                num_elements: m.mesh.indices.len() as u32,
                material: m.mesh.material_id.unwrap_or(0),
            }
        })
        .collect::<Vec<_>>();

    Ok(model::Model { meshes, materials })
}

With this structure in mind, I revisited my module I created for primitives. Instead of using the PrimitiveVertex and PrimitiveMesh — I used ModelVertex and model::Model. This simplified my code a bit.

The only major issue I had immediately was that the Model struct required a texture. I tried omitting it at first, but the shader kept complaining about undefined data. So I put in a placeholder texture for the time being. I feel like there’s a better way to handle this though, maybe with an empty buffer.

// src/primitives/mod.rs
use crate::{
    model::{self, Material, ModelVertex},
    resources::load_texture,
    texture::Texture,
    Vertex,
};
use std::ops::Range;
use wgpu::util::DeviceExt;
pub mod cube;
pub struct PrimitiveMesh {
    pub model: model::Model,
}

impl PrimitiveMesh {
    pub async fn new(
        device: &wgpu::Device,
        queue: &wgpu::Queue,
        layout: &wgpu::BindGroupLayout,
        vertices: &[ModelVertex],
        indices: &Vec<u32>,
    ) -> Self {
        let primitive_type = "Cube";

        println!("[PRIMITIVE] Creating cube materials");
        // Setup materials
        // We can't have empty material (since shader relies o n bind group)
        // And it doesn't accept Option/None, so we give it a placeholder image
        let mut materials = Vec::new();
        let diffuse_texture = load_texture(&"banana.png", device, queue)
            .await
            .expect("Couldn't load placeholder texture for primitive");
        let bind_group = device.create_bind_group(&wgpu::BindGroupDescriptor {
            layout,
            entries: &[
                wgpu::BindGroupEntry {
                    binding: 0,
                    resource: wgpu::BindingResource::TextureView(&diffuse_texture.view),
                },
                wgpu::BindGroupEntry {
                    binding: 1,
                    resource: wgpu::BindingResource::Sampler(&diffuse_texture.sampler),
                },
            ],
            label: None,
        });

        materials.push(model::Material {
            name: primitive_type.to_string(),
            diffuse_texture,
            bind_group,
        });

        println!("[PRIMITIVE] Creating cube mesh buffers");
        let mut meshes = Vec::new();

        let vertex_buffer = device.create_buffer_init(&wgpu::util::BufferInitDescriptor {
            label: Some(&format!("{:?} Vertex Buffer", primitive_type)),
            contents: bytemuck::cast_slice(&vertices),
            usage: wgpu::BufferUsages::VERTEX,
        });
        let index_buffer = device.create_buffer_init(&wgpu::util::BufferInitDescriptor {
            label: Some(&format!("{:?} Index Buffer", primitive_type)),
            contents: bytemuck::cast_slice(&indices),
            usage: wgpu::BufferUsages::INDEX,
        });

        meshes.push(model::Mesh {
            name: primitive_type.to_string(),
            vertex_buffer,
            index_buffer,
            num_elements: indices.len() as u32,
            material: 0,
        });

        let model = model::Model { meshes, materials };

        Self { model }
    }
}

Using it inside the app was very similar. I initialized the cube (and subsequently, the buffers and bind groups). I call 2 functions here, cube_vertices() and cube_indices() which generate the cube positional data.

// src/lib.rs

// Load primitives
let cube = PrimitiveMesh::new(
    &device,
    &queue,
    &texture_bind_group_layout,
    &cube_vertices(),
    &cube_indices(),
)
.await;

And when I need to render the primitive, I can use the same draw function as the OBJ model:

// src/lib.rs
render_pass.draw_model_instanced(
  &self.cube.model,
  0..self.instances.len() as u32,
  &self.camera_bind_group,
  &self.light_bind_group,
  &self.play_bind_group,
);

Here are the cube functions that contain the data. I had to make them functions because Rust didn’t like creating a const / static Vec with pre-initialized values. I also let the user pass in a “scale” to change the size of the cube (not as much control as Baryons XYZ scaling - but quick and dirty):

// src/primitives/cube.rs
use crate::model::ModelVertex;

pub fn cube_vertices(scale: f32) -> Vec<ModelVertex> {
    vec![
        // Front face
        ModelVertex {
            position: [-scale, -scale, scale],
            normal: [0.0, 0.0, scale],
            tex_coords: [0.0, 0.5],
        },
        ModelVertex {
            position: [scale, -scale, scale],
            normal: [0.0, 0.0, -scale],
            tex_coords: [0.0, 0.5],
        },
        ModelVertex {
            position: [scale, scale, scale],
            normal: [scale, 0.0, 0.0],
            tex_coords: [0.0, 0.5],
        },
        ModelVertex {
            position: [-scale, scale, scale],
            normal: [-scale, 0.0, 0.0],
            tex_coords: [0.0, 0.5],
        },
        // Back face
        ModelVertex {
            position: [-scale, -scale, -scale],
            normal: [0.0, scale, 0.0],
            tex_coords: [0.0, 0.5],
        },
        ModelVertex {
            position: [-scale, scale, -scale],
            normal: [0.0, -scale, 0.0],
            tex_coords: [0.0, 0.5],
        },
        ModelVertex {
            position: [scale, scale, -scale],
            normal: [0.0, 0.0, scale],
            tex_coords: [0.0, 0.5],
        },
        ModelVertex {
            position: [scale, -scale, -scale],
            normal: [0.0, 0.0, -scale],
            tex_coords: [0.0, 0.5],
        },
        // Top face
        ModelVertex {
            position: [-scale, scale, -scale],
            normal: [scale, 0.0, 0.0],
            tex_coords: [0.0, 0.5],
        },
        ModelVertex {
            position: [-scale, scale, scale],
            normal: [-scale, 0.0, 0.0],
            tex_coords: [0.0, 0.5],
        },
        ModelVertex {
            position: [scale, scale, scale],
            normal: [0.0, scale, 0.0],
            tex_coords: [0.0, 0.5],
        },
        ModelVertex {
            position: [scale, scale, -scale],
            normal: [0.0, -scale, 0.0],
            tex_coords: [0.0, 0.5],
        },
        // Bottom face
        ModelVertex {
            position: [-scale, -scale, -scale],
            normal: [0.0, 0.0, scale],
            tex_coords: [0.0, 0.5],
        },
        ModelVertex {
            position: [scale, -scale, -scale],
            normal: [0.0, 0.0, -scale],
            tex_coords: [0.0, 0.5],
        },
        ModelVertex {
            position: [scale, -scale, scale],
            normal: [scale, 0.0, 0.0],
            tex_coords: [0.0, 0.5],
        },
        ModelVertex {
            position: [-scale, -scale, scale],
            normal: [-scale, 0.0, 0.0],
            tex_coords: [0.0, 0.5],
        },
        // Right face
        ModelVertex {
            position: [scale, -scale, -scale],
            normal: [0.0, scale, 0.0],
            tex_coords: [0.0, 0.5],
        },
        ModelVertex {
            position: [scale, scale, -scale],
            normal: [0.0, -scale, 0.0],
            tex_coords: [0.0, 0.5],
        },
        ModelVertex {
            position: [scale, scale, scale],
            normal: [0.0, 0.0, scale],
            tex_coords: [0.0, 0.5],
        },
        ModelVertex {
            position: [scale, -scale, scale],
            normal: [0.0, 0.0, -scale],
            tex_coords: [0.0, 0.5],
        },
        // Left face
        ModelVertex {
            position: [-scale, -scale, -scale],
            normal: [scale, 0.0, 0.0],
            tex_coords: [0.0, 0.5],
        },
        ModelVertex {
            position: [-scale, -scale, scale],
            normal: [-scale, 0.0, 0.0],
            tex_coords: [0.0, 0.5],
        },
        ModelVertex {
            position: [-scale, scale, scale],
            normal: [0.0, scale, 0.0],
            tex_coords: [0.0, 0.5],
        },
        ModelVertex {
            position: [-scale, scale, -scale],
            normal: [0.0, -scale, 0.0],
            tex_coords: [0.0, 0.5],
        },
    ]
}

pub fn cube_indices() -> Vec<u32> {
    vec![
        0, 1, 2, 0, 2, 3, // front
        4, 5, 6, 4, 6, 7, // back
        8, 9, 10, 8, 10, 11, // top
        12, 13, 14, 12, 14, 15, // bottom
        16, 17, 18, 16, 18, 19, // right
        20, 21, 22, 20, 22, 23, // left
    ]
}

This got the cubes rendering!

You can find the full final code here on Github.

The Recap

So basically, to render primitive geometry we need to:

Understand the current structure of the render pipeline and adapt or assimilate our primitive geometry to it. If you have a render pipeline, you probably have a Vertex type to use, as well as defined bind groups (like any shader uniforms your app might use).
Or create a new render pipeline that supports our primitive geometry (if we need to pass less or different properties). This presumes your existing render pipelines might be too complex for simple primitives (like too many bind groups). In this case you’ll want to create a new Vertex and Mesh type, a new pipeline to support them, and an appropriate shader that’s structured to the new pipeline. Often though for primitives, you’ll want to keep them consistent with other models in your app (since the user will often need similar attributes, like normals or texture coordinates).

You can see an example in the Bevy repo of how they handle render pipelines - it seems like they have 2, one for “rendering” (probably 99% of everything) and one for “computing” (aka compute shaders).

I hope this was informative and helped enlighten you a bit more on how low level render pipelines work. If you have any questions or comments, feel free to hit me up on Twitter.

Stay curious,
Ryo

Create a Markdown Editor with Rust and React

Ryosuke — Thu, 13 Oct 2022 21:57:24 +0000

I’ve been looking for easy projects to jump into Rust with as a beginner and I thought, why not a Markdown app? Previously I’d built a Markdown-based text editor using Electron, and I was wondering if I could replicate that using Rust. It’d be a simple app, where the user can write Markdown and see a “live preview”. And with any WSIWYG, there might be buttons to help the user write Markdown, like text formatting (bold, italics, etc).

But how would you start to do this in Rust…? I’ll break it down in this post, piece by piece. First we’ll start with figuring out a way to just parse Markdown in Rust, probably using a 3rd party library (so we don’t have to manually parse it ourselves). Then we can build a UI for the user to see the Markdown, edit it, and send changes to Rust to update our “preview”.

Here’s what our app will look like:

You can find the complete code for this post in my tauri-markdown-editor repo.

⚠️ This article assumes you have a basic understanding of the Markdown syntax, and even intermediate aspects like frontmatter. You’ll also see the use of “CommonMark” in this article, this refers to the specification for the Markdown syntax.

Parsing Markdown…in JavaScript

Before we dive into Rust, let’s see how we’d do this process in JavaScript - a language people might be more familiar with.

Normally in JavaScript we’d use a library like marked to take our text that contains Markdown and parse it into an AST (or abstract syntax tree).

import marked from "marked";

marked.parse("# Marked in the browser\n\nRendered by **marked**.");

The AST acts as a representation of the Markdown broken up into related chunks (like paragraphs, or bold text). An AST in JavaScript is usually just a giant object made up of Node objects that contain Markdown data. It’ll probably be a flat array, not with nested children, since you’ll inevitably “join” it together in order. This is pseudo-code of what an AST might look like:

const ast = {
    children: [
        Node {
            id: 0,
            type: "string",
            content: "Who is Ryo?",
        }
    ]
}

We could then convert that AST to HTML to export to a website. For NextJS blogs, they do this process in the getStaticProps server-side function, which generates HTML for the frontend to use.

In Rust, the process will look very similar. We’ll take a giant string of Markdown text and use a library to parse that into an AST, then eventually into another format (HTML or back into Markdown).

Parsing Markdown in Rust

In Rust, we can use the comrak crate to parse our Markdown. This is a Rust port of a C library that, you guessed it, also parsed Markdown (aka “CommonMark”).

Project setup

Let’s create a fresh environment and get things setup.

Run cargo new to create a new Rust project
Go into the directory using cd your-project
Add comrak as a dependency cargo add comrak
We’ll also need Markdown! Create a sample [README.md](http://README.md) in the root folder (same level as Cargo.toml). Maybe fill it with docs on getting the project running 😉
Open up the [main.rs](http://main.rs) file and let’s add the example code from the comrak Github README

use comrak::{markdown_to_html, ComrakOptions};

fn main() {
        println!("Hello, world!");

    let markdown = markdown_to_html("Hello, **世界**!", &ComrakOptions::default());

    println!("{markdown}");
}

In Rust we run code using cargo run. This should print out the HTML version of the sample string. Here’s the commit on Github for reference.

Now lets parse the actual [README.md](http://README.md) we created.

Parsing Markdown files

This is a pretty straightforward process, because comrak expects a &str or &String (basically a pointer of 1 long string with all the Markdown). In Rust, we can conveniently load local files directly into strings using std::fs::read_to_string(). You can find the Rust docs for that function here.

use comrak::{markdown_to_html, ComrakOptions};
use std::fs;

fn main() {
    println!("Opening README \n");

    let file_path = "README.md".to_string();
    let contents = fs::read_to_string(file_path).expect("Couldn't read file");

    let markdown = markdown_to_html(&contents, &ComrakOptions::default());

    println!("Markdown parsed into HTML \n");
    println!("{markdown}");
}

If you run this, you should see the contents of the README file you created — but converted to HTML! Here’s the commit on Github for reference.

Parsing Markdown to AST

Now that you’ve parsed your Markdown, let’s say you want to do stuff to it before it gets converted to HTML. We could edit the Markdown string, but with long text files this becomes inefficient. And if we’re looking for things like bold text, or links, we’d need to use some sort of regular expression (or regex) to find what we need. Instead, we can parse the Markdown into an AST - which is basically an array of all the elements from the Markdown.

For example, one of the most common things you often need to extract from Markdown is “frontmatter”. This is metadata you can embed in your Markdown files, like a blog title or post date.

---
title: "README"
url: https://github.com
---

It’s actually not part of the CommonMark spec, so you’ll often need a 3rd party library to parse it out on top of your Markdown parser. In JavaScript we use gray-matter which converts frontmatter into a JS object we can more easily use.

Luckily, comrak offers frontmatter support out of the box! Kinda. We’ll still have to additionally parse it 😅

Let’s check out the AST parsing example from the comrak Github README and add a case to detect frontmatter:

use std::fs;
extern crate comrak;
use comrak::nodes::{AstNode, NodeValue};
use comrak::{parse_document, Arena, ComrakOptions};

fn main() {
    println!("Opening README \n");

    let file_path = "README.md".to_string();
    let contents = fs::read_to_string(file_path).expect("Couldn't read file");

    println!("Markdown parsed into AST \n");

    // The returned nodes are created in the supplied Arena, and are bound by its lifetime.
    let arena = Arena::new();

    // Change any default options here for Comrak (parser, extensions, etc)
    let mut options = &mut ComrakOptions::default();
    // Enable frontmatter detection
    options.extension.front_matter_delimiter = Some("---".to_string());

    // The "root" node, which we parse our markdown into
    let root = parse_document(&arena, &contents, options);

    // Iterate through the nodes (and their children) recursively
    // We pass the node to the callback provided as the second function param
    fn iter_nodes<'a, F>(node: &'a AstNode<'a>, f: &F)
    where
        F: Fn(&'a AstNode<'a>),
    {
        f(node);
        for c in node.children() {
            iter_nodes(c, f);
        }
    }

    // Call the iterate nodes function
    iter_nodes(root, &|node| {
        // Check the value of the node data
                // This is basically a giant "switch" statement (for my JS peeps)
        match &mut node.data.borrow_mut().value {
                        // Here we check for the "FrontMatter" enum type
            NodeValue::FrontMatter(ref mut block) => {
                                // The block is returned as a utf8 integer array (basically letters = numbers)
                                // So we use a method to convert that to a String so we can read it
                println!(
                    "Frontmatter: {}",
                    String::from_utf8(block.to_vec()).unwrap()
                );
            }
                        // Other enum types you can play with. Just comment out the `println!`
            // Got text?
            &mut NodeValue::Text(ref mut text) => {
                let orig = std::mem::replace(text, vec![]);
                // println!("{}", String::from_utf8(orig).unwrap());
                // *text = String::from_utf8(orig).unwrap().replace("my", "your").as_bytes().to_vec();
            }
            _ => (),
        }
    });

}

If you add some frontmatter to your [README.md](http://README.md) and run this, you should see the frontmatter appear in your shell as a single string (with line breaks and whatnot).

Try playing around and adding more NodeValue enum types to the match statement to find different data types (like NodeValue::Link for links). You could even “mutate” or change these values, like finding and replacing text - or appending text like code snippets.

⚠️ If we wanted to use each property, we’d still need to parse this further, but that starts to reach out of the scope of this beginner tutorial.

Going back to Markdown!

We can convert our Markdown to an AST and HTML — but how do we go from an AST back to Markdown? The comrak library has our back with this one too and offers a format_commonmark method. This converts the AST back to “CommonMark” - aka Markdown.

use std::fs;
extern crate comrak;
use comrak::nodes::{AstNode, NodeValue};
use comrak::{format_commonmark, format_html, parse_document, Arena, ComrakOptions};

fn main() {
    println!("Opening README \n");

    let file_path = "README.md".to_string();
    let contents = fs::read_to_string(file_path).expect("Couldn't read file");

    println!("Markdown parsed into AST \n");

    // The returned nodes are created in the supplied Arena, and are bound by its lifetime.
    let arena = Arena::new();

    // Change any default options here for Comrak (parser, extensions, etc)
    let mut options = &mut ComrakOptions::default();
    // Enable frontmatter detection
    options.extension.front_matter_delimiter = Some("---".to_string());

    // The "root" node, which we parse our markdown into
    let root = parse_document(&arena, &contents, options);

    // Iterate through the nodes (and their children) recursively
    // We pass the node to the callback provided as the second function param
    fn iter_nodes<'a, F>(node: &'a AstNode<'a>, f: &F)
    where
        F: Fn(&'a AstNode<'a>),
    {
        f(node);
        for c in node.children() {
            iter_nodes(c, f);
        }
    }

    // Call the iterate nodes function
    iter_nodes(root, &|node| {
        // Check the value of the node data
        match &mut node.data.borrow_mut().value {
            NodeValue::SoftBreak => {
                // println!("Soft break")
            }
            NodeValue::LineBreak => {
                // println!("Line break")
            }
            NodeValue::Item(ref mut blocks) => {
                // dbg!(blocks);
            }
            NodeValue::FrontMatter(ref mut block) => {
                println!(
                    "Frontmatter: {}",
                    String::from_utf8(block.to_vec()).unwrap()
                );

                let raw_frontmatter = String::from_utf8(block.to_vec())
                    .expect("Couldn't parse frontmatter into string.");
            }
            &mut NodeValue::CodeBlock(ref mut block) => {
                let orig = std::mem::replace(&mut block.literal, vec![]);
                // println!("Code Block: {}", String::from_utf8(orig).unwrap());
            }
            &mut NodeValue::Link(ref mut link) => {
                let orig = std::mem::replace(&mut link.url, vec![]);
                // println!("Link: {}", String::from_utf8(orig).unwrap());
            }
            &mut NodeValue::Strong => {
                // println!("Bold text: ");
            }
            // Got text?
            &mut NodeValue::Text(ref mut text) => {
                let orig = std::mem::replace(text, vec![]);
                // println!("{}", String::from_utf8(orig).unwrap());
                // *text = String::from_utf8(orig).unwrap().replace("my", "your").as_bytes().to_vec();
            }
            _ => (),
        }
    });

    // Output "CommonMark" (aka Markdown) format
    let mut output = String::new();
    let mut buffer = File::create("test.txt").expect("Couldn't create output file.");
    format_commonmark(&root, &options, &mut buffer);
}

Now we have all the power we need! We can take Markdown, change it (using the AST), and we can return the Markdown or HTML back to the user.

Let’s make a frontend app that’ll let the user write Markdown!

Markdown Editor using Tauri

At first I considered using a Rust GUI library for creating my app, but I found they were all very unstable and fairly undocumented. Although I’ll admit Yew it a solid consideration in the future.

I settled on using Tauri, an Electron alternative written in Rust. It’d let me write Rust code for the “backend” (aka IPC layer in Electron), and the frontend could be in any web-based language I prefer (in this case - React and Typescript).

Project Setup

The Tauri project setup is very simple thanks to the fantastic quick start CLI.

yarn create tauri-app

Once the Tauri app was setup with React, we could also add comrak as a dependency. To do that, we need to open up src-tauri/Cargo.toml and add comrak as a dependency (you could also just cd src-tauri && cargo add comrak):

[package]
name = "tauri-markdown-editor"
version = "0.0.0"
description = "A Tauri App"
authors = ["you"]
license = ""
repository = ""
edition = "2021"
rust-version = "1.57"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[build-dependencies]
tauri-build = { version = "1.1", features = [] }

[dependencies]
serde_json = "1.0"
serde = { version = "1.0", features = ["derive"] }
tauri = { version = "1.1", features = ["api-all"] }
# Add comrak here
comrak = "0.14"

[features]
# by default Tauri runs in production mode
# when `tauri dev` runs it is executed with `cargo run --no-default-features` if `devPath` is an URL
default = [ "custom-protocol" ]
# this feature is used used for production builds where `devPath` points to the filesystem
# DO NOT remove this
custom-protocol = [ "tauri/custom-protocol" ]

Here’s the commit on Github for reference.

Once that’s setup, we can create our UI for the user to write.

Simple text editor

I opened up the src/App.tsx and added a <textarea> and synced any text changes to React state.

import { useState } from "react";
import reactLogo from "./assets/react.svg";
import { invoke } from "@tauri-apps/api/tauri";
import "./App.css";

function App() {
  const [markdown, setMarkdown] = useState("");

  return (
    <div className="container">
      <h1>Welcome to Tauri!</h1>

      <div className="row">
        <a href="https://vitejs.dev" target="_blank">
          <img src="/vite.svg" className="logo vite" alt="Vite logo" />
        </a>
        <a href="https://tauri.app" target="_blank">
          <img src="/tauri.svg" className="logo tauri" alt="Tauri logo" />
        </a>
        <a href="https://reactjs.org" target="_blank">
          <img src={reactLogo} className="logo react" alt="React logo" />
        </a>
      </div>

      <p>Click on the Tauri, Vite, and React logos to learn more.</p>

      <div className="row">
        <div>
          <textarea
            id="greet-input"
            onChange={(e) => setMarkdown(e.currentTarget.value)}
          />
        </div>
      </div>
    </div>
  );
}

export default App;

Now that we have a textbox we can write Markdown inside, let’s convert that to HTML to preview.

Creating a Tauri command

We’ll need to use Tauri commands to communicate between the frontend and backend - and vice versa (similar to the IPC layer in Electron).

Tauri comes with an example Tauri command called greet() already setup, so let’s use that and change it to parse Markdown. Open up src-tauri/src/main.rs and change it to the following:

#![cfg_attr(
    all(not(debug_assertions), target_os = "windows"),
    windows_subsystem = "windows"
)]
// Import comrak
use comrak::{markdown_to_html, ComrakOptions};

// Learn more about Tauri commands at https://tauri.app/v1/guides/features/command
#[tauri::command]
fn greet(markdown: &str) -> String {
    // format!("Hello, {}! You've been greeted from Rust!", name)

        // Parse the markdown text to HTML
    let html = markdown_to_html(&markdown, &ComrakOptions::default());

    println!("Markdown parsed into HTML \n");
    println!("{html}");

        // We return the HTML to the frontend (in Rust, we return by omitting the `;`)
    html
}

fn main() {
    tauri::Builder::default()
                // Here is where we add our Tauri commands to our app
        .invoke_handler(tauri::generate_handler![greet])
        .run(tauri::generate_context!())
        .expect("error while running tauri application");
}

And in our frontend, let’s create a <button> we can press to run the Tauri command. We use the invoke function provided by Tauri that’ll call a Tauri command with the same name.

import { useState } from "react";
import reactLogo from "./assets/react.svg";
import { invoke } from "@tauri-apps/api/tauri";
import "./App.css";

function App() {
  const [html, setHtml] = useState("");
  const [markdown, setMarkdown] = useState("");

  async function greet() {
    // Learn more about Tauri commands at https://tauri.app/v1/guides/features/command
    setHtml(await invoke("greet", { markdown }));
  }

  return (
    <div className="container">
      <h1>Welcome to Tauri!</h1>

      <div className="row">
        <a href="https://vitejs.dev" target="_blank">
          <img src="/vite.svg" className="logo vite" alt="Vite logo" />
        </a>
        <a href="https://tauri.app" target="_blank">
          <img src="/tauri.svg" className="logo tauri" alt="Tauri logo" />
        </a>
        <a href="https://reactjs.org" target="_blank">
          <img src={reactLogo} className="logo react" alt="React logo" />
        </a>
      </div>

      <p>Click on the Tauri, Vite, and React logos to learn more.</p>

      <div className="row">
        <div>
          <textarea
            id="greet-input"
            onChange={(e) => setMarkdown(e.currentTarget.value)}
          />
          <button type="button" onClick={() => greet()}>
            Convert to HTML
          </button>
        </div>
      </div>
      <p>{html}</p>
    </div>
  );
}

export default App;

If you type some Markdown into the text box and click the button, it should print out raw HTML representing your Markdown. Here’s the commit on Github for reference.

But how do we preview the HTML?

If we wanted to preview the HTML, we could use React’s dangerouslySetInnerHTML to inject the HTML inside our app:

const createMarkdownMarkup = () => ({
  __html: html
})

<div dangerouslySetInnerHTML={createMarkdownMarkup()} />

And ideally, this isn’t as “dangerous” as it sounds, because Comrak sanitizes the Markdown output for any malicious code before converting to HTML 👍

We could also update the onChange function to run our conversion process, instead of waiting for a button press, so we “instantly” see a live preview. In my case, I do it in a useEffect with a refresh boolean — but it’s all good. Note, this is a little risky, since the user might be able to break the app, so in production it’d be best to create a cache of the last “working” preview just in case one fails.

import { useEffect, useRef, useState } from "react";
import reactLogo from "./assets/react.svg";
import { invoke } from "@tauri-apps/api/tauri";
import "./App.css";

function App() {
  const [html, setHtml] = useState("");
  const [markdown, setMarkdown] = useState("");
  const [refreshCheck, setRefreshCheck] = useState(false);

  async function greet() {
    setRefreshCheck(true);
  }

  useEffect(() => {
    const parseMarkdown = async () => {
      setHtml(await invoke("greet", { markdown }));
      setRefreshCheck(false);
      console.log("new markdown");
    };
    console.log("refreshed");
    if (refreshCheck) {
      parseMarkdown();
      console.log("parsing!");
    }
  }, [refreshCheck]);

  const createMarkdownMarkup = () => ({
    __html: html,
  });

  const handleTextArea = (e) => {
    setMarkdown(e.currentTarget.value);
    setRefreshCheck(true);
    console.log("need new markdown!");
  };

  return (
    <div>
      <div className="row">
        <div>
          <textarea id="greet-input" onChange={handleTextArea} />
          <button type="button" onClick={() => greet()}>
            Convert to HTML
          </button>
        </div>
      </div>
      <div dangerouslySetInnerHTML={createMarkdownMarkup()} />
    </div>
  );
}

export default App;

Here’s the commit on Github for reference.

The world is your Markdown

Now you can parse Markdown using Rust, there’s lots of cool stuff you can do! From CLIs to UI apps, you can make tools that help people use Markdown. And ideally, this will run faster than most other implementations (like Electron apps) because it’s running in Rust (unless you can write Markdown tooling in C or something).

I hope this helped you understand some Rust fundamentals and new techniques to use down the line.

As always, if you have any questions or make something using this guide, tag me on Twitter. I’d love to help or see what cool stuff you’re hacking on.

You can find the complete code for this post in my tauri-markdown-editor repo.

Stay regular!
Ryo

Migrating my Gatsby MDX blog to AstroJS (and why you shouldn't)

Ryosuke — Fri, 07 Oct 2022 02:33:53 +0000

It’s that time again. The initial commit on the last version of my Gatsby-based site was Mar 30, 2018, and the last major redesign was October 14th 2019. It’s been 3 years since I last touched the design of the site, so as a designer you can imagine I’ve been reeling waiting to change things up.

But I’ve also been quite busy lately between work and life, so I haven’t had the time to invest in the wild and ambitious redesign I’ve been envisioning for the past year or so. To keep things moving, I opted to do a mini redesign or a “refresh” where I kept things as simple as possible.

My goal was to try a few new libraries and get experience under my belt (cause isn’t that what dev blogs are for?). I’d swap out Gatsby for **Astro,** and Styled Components + Styled System for **Vanilla Extract**. I was interested in trying Astro as it just hit 1.0, and I’ve been experimenting for a while with Vanilla Extract as a way to write Typescript powered styles and export to CSS.

In case you’re looking at an archived version of this post in the distant future, here’s a preview of what the redesign looked like:

I’ll break down a bit of my redesign process and some of the issues that tripped me up along the way.

The Process

I started initially in code. Then designed a rough and finalized version in Figma. Then brought the design to life in Astro, React, and Vanilla Extract.

Gatsby to Astro

I had a framework from my Gatsby website to work with, and I needed to figure out how to migrate that content to another framework without too much effort. All my content is in MDX files, mostly Markdown with a few files using some custom React components (like a <Box> for layout).

At first, I wasn’t sure I was going to use NextJS or Astro. But Astro won me over once I saw how plug and play the Markdown and MDX support was with a couple of plugins (even frontmatter! which lots of stuff struggles with).

I learned a bit about how Astro works while prototyping a very rough and unstyled version of the blog. I started with a single blog post to see how things carried over. Things seemed good, so I proceeded with the design phase in Figma.

Figma Design

This was probably the chillest part of the process. I had 2 mood boards I’d been accruing over months, one for an ambitious redesign and one for a simpler one. I took a look at those, and dug around for a bit more inspo.

The first round didn’t deviate too far from where we ended up. I had a fairly clear vision of what I wanted, but the minor elements just needed to fall into place.

I got a little distracted trying to incorporate texture into the website…

But I eventually landed on using a 3D animation as the primary background element. I found an abstract 3D render on a stock image website and figured I could recreate something similar in ThreeJS / R3F.

With the design in place, I moved back over to the code to develop the design system and components.

Vanilla Extract with Sprinkles please

I started to develop some of the components in isolation using Storybook and Vanilla Extract. I used one of my previous projects as the basis, Gelato UI, a design system I created using Nx monorepo.

After finishing all the components here, I copied them over to my Astro website.

It actually wasn’t that simple initially though — when I was working on the design system, Astro didn’t support Vanilla Extract (even as a pre-compiled 3rd party library). Luckily someone updated the Vanilla Extract Vite plugin (which Astro uses under the hood).

Time to surf 3D waves

For this part it was pretty straightforward since I had a 3D design I was using as the basis. I opened up CodeSandbox and forked one of my old R3F sandboxes that had a custom shader material.

I created a cube (or box) and formed it into a long rectangle (like strips of paper). I made sure to increase the number of “segments” in the box so it’d animate a bit smoother.

import React, { useMemo, useRef } from "react";
import { MeshProps, useFrame } from "@react-three/fiber";
import { Line } from "@react-three/drei";
import "./material";

type WaveProps = MeshProps & {
  offset: number,
};

// The box/cube parameters
const WAVE_SIZE = {
  width: 0.5,
  height: 10,
  depth: 0.05,
  // How "subdivided" the mesh is (aka bigger number = more polygons)
  widthSegment: 2,
  heightSegment: 32,
};

function Wave({ offset, ...props }: WaveProps) {
  const geom = useRef();
  useFrame((state) => {
    // Send (and update) "uniforms" or data to the shader (like the time)
    geom.current.material.uniforms.time.value = state.clock.getElapsedTime();
    geom.current.material.uniforms.offset.value = offset;
    // geom.current.geometry.verticesNeedUpdate = true
  });

  return (
    <mesh
      ref={geom}
      rotation={[-Math.PI / 2, 0, 0]}
      castShadow
      receiveShadow
      {...props}
    >
      <boxBufferGeometry
        args={[
          WAVE_SIZE.width,
          WAVE_SIZE.height,
          WAVE_SIZE.depth,
          WAVE_SIZE.widthSegment,
          WAVE_SIZE.heightSegment,
        ]}
      />
      <waveMaterial />
      {/* <meshBasicMaterial wireframe /> */}
    </mesh>
  );
}

export default Wave;

Now that I had a wave, I just had to create a component that makes a bunch more. I make an empty array of 9 items and map over it with each <Wave /> component.

import React from "react";
import Wave from "../Wave/Wave";

const Waves = () => {
  const waves = new Array(9)
    .fill(0)
    .map((_, index) => (
      <Wave key={index} position={[index * 0.6, 0, 0]} offset={index} />
    ));

  return waves;
};

export default Waves;

Then the magic (or animation) happened in the shader. Essentially the boxes animate using time as a variable (so as time progresses, the variable changes). We take that time and use a sin() calculation to make it fluctuate between 0 and 1 (instead of going from 0 to Infinity). This makes the math a bit easier for stuff since it loops from 0 to 1 and back to 0 looping infinitely.

We apply the animation to the z index, so it only animates vertices in the z direction (up and down in this case). And it’s offset using an offset variable, so all the waves don’t animate at the same pace. That variable is the array index we loop over to make waves (aka 0-9).

float z = position.z + sin(position.y + time + offset);

Here is a version with the wireframe enabled as a visual aid:

After the scene was laid out and animated, I took some time to make to look better. This included lighting the scene, placing the camera, and adding post processing effects (like a bokeh).

For the post processing I used @react-three/post-processing components, and to get the look just right I created a debug panel using Leva to adjust some values (like bokeh blur).

Custom shadow shader

Adding shadows (castShadow and receiveShadow) to all the objects helped give some depth — but I immediately noticed that my custom shader didn’t apply any shadows to the objects. After a bit of research, I found that the standard ThreeJS materials (like MeshPhongMaterial) have shadow calculations built into them. But when you create a custom shader, it’s a completely blank slate.

In order to add shadows, I had to copy and paste the shadow related code from the default ThreeJS materials (or shaders really).

import * as THREE from "three";
import { extend } from "@react-three/fiber";
import { mergeUniforms } from "three/src/renderers/shaders/UniformsUtils.js";
import { UniformsLib } from "three/src/renderers/shaders/UniformsLib.js";

class WaveMaterial extends THREE.ShaderMaterial {
  constructor() {
    super({
      transparent: true,
      // Enable fog/light/dithering maps to get passed to shader
      fog: true,
      lights: true,
      dithering: true,
      // wireframe: true,

      // Merge in the light and fog uniforms with any of your custom ones
      uniforms: mergeUniforms([
        { time: { value: 1 }, offset: { value: 0 } },
        UniformsLib.lights,
        UniformsLib.fog,
      ]),
      vertexShader: `
      // Include the shadow scripts
    #include <common>
    #include <fog_pars_vertex>
    #include <shadowmap_pars_vertex>

      uniform float time;
      uniform float offset;
      attribute float size;
      void main() {

          // Include the shadow scripts
        vec3 objectNormal = vec3( normal );
        vec3 transformedNormal = normalMatrix * objectNormal;
        #include <begin_vertex>
        #include <project_vertex>
        #include <worldpos_vertex>
        #include <shadowmap_vertex>
        #include <fog_vertex>

                // Do what you need (in this case, makes wave animation)
        float x = sin(position.x);
        float y = position.y;
        float id = position.z;
        float z = position.z + sin(position.y + time + offset);
        vec3 pos = vec3(position.x, position.y, z);
        gl_Position = projectionMatrix * modelViewMatrix * vec4(pos, 1.0 );
      }`,
      fragmentShader: `
      // Include the shadow scripts
      #include <common>
      #include <packing>
      #include <fog_pars_fragment>
      #include <bsdfs>
      #include <lights_pars_begin>
      #include <shadowmap_pars_fragment>
      #include <shadowmask_pars_fragment>
      #include <dithering_pars_fragment>

      uniform float time;
      void main() {
        // CHANGE THAT TO YOUR NEEDS
        // ------------------------------
        vec3 finalColor = vec3(0.95, 0.95, 0.95);
        vec3 shadowColor = vec3(0, 0, 0);
        float shadowPower = 0.5;
        // ------------------------------

        // it just mixes the shadow color with the frag color
        gl_FragColor = vec4( mix(finalColor, shadowColor, (1.0 - getShadowMask() ) * shadowPower), 1.0);
        #include <fog_fragment>
        #include <dithering_fragment>
      }`,
    });
  }
}

extend({ WaveMaterial });

This introduced a “banding” to my objects, which is technically incorrect I’m sure, but it looked kinda cool so I was ok with it.

You can find the final sandbox here for reference.

Light vs Dark mode

In the final site I also had to add a shader uniform to let the shader know if it was light or dark mode on the site.

// Get theme value from Zustand store
const { mainNav, theme } = useStore();

// Change color based on theme
const WAVE_PROPERTIES = {
  color: {
    light: new Color({
      r: 0.95,
      g: 0.95,
      b: 0.95,
    }),
    dark: new Color("#000"),
    hovered: new Color("#001BD8"),
  },
};
const themeColor =
  theme === "light" ? WAVE_PROPERTIES.color.light : WAVE_PROPERTIES.color.dark;

useFrame((state) => {
  geom.current.material.uniforms.color.value = themeColor;
});

Hover interaction

Once I had the 3D in place, I wanted to include some sort of interaction with the user. So when the user hovers over a navigation item, it “highlights” a specific wave by changing it’s color. This was pretty simple, I just set some onMouseEnter and onMouseLeave props to sync which nav item was active to the store.

const { mainNav, setMainNav } = useStore();

const clearHover = () => {
  setMainNav("none");
};

return (
  <MainNavItem
    href="/blog"
    onMouseEnter={() => setMainNav("writing")}
    onMouseLeave={clearHover}
  >
    Writing
  </MainNavItem>
);

Then inside the 3D I grab the menu items and use a little math to pick a wave to color:

const NAV_TO_INDEX: Record<MainNavNames, number> = {
  none: 0,
  writing: 1,
  work: 2,
  resources: 3,
  playlist: 4,
};

const { mainNav, theme } = useStore();

// There's 18 waves, we want to do nearest ones and skip between
// Take the NAV_TO_INDEX and multiply by 2 to adjust for spacing
// And we add 8 to offset it enough to be in camera view
const isColored =
  mainNav !== "none" && offset === NAV_TO_INDEX[mainNav] * 2 + 8;

useFrame((state) => {
  if (isColored) {
    geom.current.material.uniforms.color.value = WAVE_PROPERTIES.color.hovered;
  } else {
    geom.current.material.uniforms.color.value = themeColor;
  }
});

This effect was pretty simple and made the site a bit more fun to use.

Astro, blast off!

With a final design and components in place, I started to lay out the various pages of the site. I had already roughly created the blog page, but I finished styling it, and moved on to the index page, then archive pages (blog and tags), and finally a playlist page.

This process was full of frustration as I learned the Astro ecosystem and adapted to it. There were a lot of really simple things I wanted to do (like wrapping components in React Context providers) that would break the entire app. Most of these issues are described below. A lot of the issues I had were definitely a little niche in some cases, but I do think that a lot of users will expect to do most if not everything I did.

Astro-nomical Issues

Vanilla extract support

As I was creating the blog initially, there was no Vanilla Extract support. I mentioned it on Twitter and in the same day someone mentioned they had just patched the Vanilla Extract Vite plugin to support Astro.

Updated my dependency and it worked like a charm.

Although it has been a bit buggy? Occasionally pages will load with none or incorrect styles, and require a reboot to the dev server to correct. This is probably something that’ll get patched down the line since everything is so fresh hot 👍

MDX Provider not supported out of box

Had to find a custom Astro plugin script to do it. It essentially is a middleware for the MDX and it goes in does a simple “find and replace” on the exported JS. This works well, but it tends to crash the dev server randomly with a “maximum call stack exceeded” error 🤷‍♂️

// astro.config.js
import { defineConfig } from "astro/config";
import react from "@astrojs/react";
import { vanillaExtractPlugin } from "@vanilla-extract/vite-plugin";

import mdx from "@astrojs/mdx";

// Swap out elements in MDX file with React components
// e.g. `<p>` becomes `<Text>`
const mdxProvider = () => {
  return {
    name: "mdx-components-provider",
    enforce: "post",
    transform(code, id) {
      if (!id.endsWith(".mdx")) return;
      code = `import { components } from '@/components/MDXComponents/MDXComponents';\n${code}`;
      code = code.replace(
        "export default MDXContent;",
        `
        export default function (props) {
          const newProps = {
            ...props,
            components
          }
          return MDXContent(newProps);
        };`
      );
      return code;
    },
  };
};

// https://astro.build/config
export default defineConfig({
  markdown: {
    shikiConfig: {
      theme: "material-palenight",
    },
  },
  // Enable React to support React JSX components.
  integrations: [react(), mdx()],

  vite: {
    // Example: Add custom vite plugins directly to your Astro project
    plugins: [vanillaExtractPlugin(), mdxProvider()],
  },
});

// MDXComponents.tsx
import Box from "../Box/Box";
import Text from "../Text/Text";

export const components = {
  h1: (props) => (
    <Box mt={7}>
      <Text as="h1" fontFamily="heading" fontSize={5} {...props} />
    </Box>
  ),
  div: (props) => <Box {...props} />,
};

Can’t use render props inside `.astro` files

Tried to create a component that accepts another React component as a prop (<Accordion title={<Text>Title</Text>}>) and the compiler did not like it, crashing the site.

Had to convert to a regular string-based prop.

React Context doesn’t work for components in Astro pages

Lets say you have a about.astro page. You also have a <PageLayout> React component that wraps all the page content, and it has a few context providers inside (theme, shared app state, etc). You write your page markup in the about.astro file — cause why not?

<DefaultPage>
  {" "}
  // Has context providers
  <AnotherReactComponent>
    {" "}
    // Doesn't have access to context - despite being nested
    <h1>About Page</h1>
    <MoreReactStuff /> // Also can't access context
  </AnotherReactComponent>
</DefaultPage>

All the nested React components inside the Astro file will not get access to the context.

The solution? Create your page entirely inside React, then import that whole page component into your Astro page.

In my case however, this didn’t work. I had react-three-fiber components that needed a special client:load tag in Astro to load them. And when I added that tag to the entire page component, the page rendered — but the R3F component didn’t.

My workaround? I ended up using zustand to create a store outside of “React-land” and using that between components. But this still was a major issue with a lot of wrapper components that needed access to context (like a theme provider).

Same with React `children`

I had issues using children when it was content from an Astro page and the React component was client:load:

<SomeComponent client:load>
    <h1>Content inside Astro file</h1> // SomeComponent can't load this

This has to do with the way they isolate code into “islands”. If you inspect the HTML, you’ll see that the component we labeled with client:load is in an isolated island, but the HTML template it uses doesn’t included the nested children content.

Utility Props Problem

A lot of my components are “dynamic” where they have Vanilla Extract’s Sprinkles API working to swap out classnames based on the props I pass to the component. For Astro, it’d initially render these components, but they’d lose their “interactivity” without a client:load directive. So I couldn’t see style changes when I swapped props.

This got a little tedious, since a lot of components use or extend from a component that uses utility props.

A little slow and clunky

I have archive pages that need to query all of my blog posts — every single one. This ends up being an intensive process, taking a few minutes for it to spin up a page (~75 blog posts). This doesn’t get cached between pages, so it re-runs.

It’s smart enough to cache if I update other code, but the cache actually works against you? Sometimes it’ll cache broken code, and I’m forced to delete massive chunks to trigger a proper reload. It causes a lot of false positives when coding, which can be frustrating when debugging complex (or even trivial) issues.

What makes it worse is there’s no built in cache clearing option? I think they’d benefit from an astro clean command to give you a clean slate sometimes.

Markdown images broken?

One thing I didn’t notice until the website was built and in production on Netlify — all my markdown images were broken. Which was weird, because they worked fine in development.

I literally went back and double checked and the development server was using:

https://whoisryosuke.com/blog/2022/2022-05-17_182533_-_Windows_PowerShell_Setup.png

While the production server used:

https://whoisryosuke.com/blog/2022/leveling-up-windows-powershell-with-oh-my-posh/2022-05-17_182533_-_Windows_PowerShell_Setup.png

I’m not sure what causes the difference between what the markdown parser shows in development vs production. Luckily though, because I was already migrating my content from my previous Gatsby blog, all I had to do was go back and change my migration script to use the production URLs.

Speaking of which, here’s what that migration script looked like. It goes through all my blog folders, edits Markdown to add the Astro layout, and copies images over to a separate folder (to copy into /public):

const fs = require("fs");
const path = require("path");

// Loop through all years
// Go into each folder

// Get MDX file
// Add template to frontmatter
// layout: "@/layouts/BlogLayout.astro"

// Found images in folder with MDX?
// Copy images to public/blog/YEAR folder

const ROOT = "./content/blog/";
const IMAGE_ROOT = "./images/";

const years = fs.readdirSync(path.join(ROOT));

years.forEach((year) => {
  const blogPostFolders = fs.readdirSync(path.join(ROOT, year));

  // Go into each folder
  blogPostFolders.forEach((blogPostFolder) => {
    const blogImagePath = path.join(IMAGE_ROOT, year);
    const blogPostFolderContents = fs.readdirSync(
      path.join(ROOT, year, blogPostFolder)
    );

    // Create folder if it doesn't exist
    if (!fs.existsSync(blogImagePath)) {
      fs.mkdirSync(blogImagePath);
    }

    blogPostFolderContents.forEach((file) => {
      const filePath = path.join(ROOT, year, blogPostFolder, file);
      const copyPath = path.join(IMAGE_ROOT, year, file);
      // Get MDX file
      // Add template to frontmatter
      // layout: "@/layouts/BlogLayout.astro"
      if (file.includes(".md")) {
        const blogPostMdx = fs.readFileSync(filePath, "utf-8");
        console.log("mdx", blogPostMdx);

        // Find frontmatter closing tag (second `---`)
        const splitPost = blogPostMdx.split("---");
        console.log();
        splitPost[1] = `${splitPost[1]}
layout: "@/layouts/BlogLayout.astro"
`;
        const joinPost = splitPost.join("---");

        // Save post
        fs.writeFileSync(filePath, joinPost);
      }

      if (
        file.includes(".png") ||
        file.includes(".jpg") ||
        file.includes(".jpeg") ||
        file.includes(".gif") ||
        file.includes(".svg")
      ) {
        fs.copyFileSync(filePath, copyPath);
        fs.rmSync(filePath);
      }
    });
  });
});

That fixed it, but now when I look at the site in development, none of my blog images work 😓 I can probably create a plugin to fix it in development, but I feel like this is a bug that’ll get resolved down the line.

Would I use Astro again?

Maybe. Probably not though. If I had a project that needed to merge multiple frontend frameworks, this would be a great pick. If I had a simple website that didn’t use too many dynamic elements, you’d avoid many of the headaches I had.

I also like the island architecture, despite being at odds with it in this project and framework. I think if Astro could support it in a more clear way that didn’t require the user to read docs and run into walls.

But if I had to create a Markdown based site, I’d definitely pick another solution (or wait until these bugs get worked out — cause it is easy to use compared to other frameworks).

The whole experience honestly had me appreciating NextJS a bit. It was missing a few things that Astro had (like easy MDX imports and frontmatter support), but the minimalism and freedom NextJS provided make it more attractive for prototyping and even production projects.

Grain of new salt

As someone who’s very new to the framework and didn’t bother to ask around on the Discord before complaining in this blog posts, this is just sharing my initial experience and critiques. Like I mentioned before, many of these issues will be resolved in the (often near) future. And I have no excuse to contribute cause it’s open source 😅

Are you using Astro in your project and enjoying it? Did I get something wrong about Astro? Let me know on Twitter. Always interested in hearing other’s experiences.

So when’s the redesign?

Soon.

And expect wild things like gamepad input 🎮

I’ve said too much already 😉

See you next time!
Ryo

References

Three.js no shadows on ShaderMaterial

A basic example of a ThreeJS (r108) ShaderMaterial with shadows, fog and dithering support.

How to add shadows to custom shaders using ShaderMaterial as basis
Basically need to include a bunch of scripts ThreeJS uses in other materials (like MeshLambertMaterial).
This is similar to the THREE.ShaderChunk["common"] process in older/vanilla stackoverflows

https://github.com/zadvorsky/three.bas/issues/7

Was finally able to import shader “chunks” (aka #include scripts from ThreeJS) — but was getting error about transformedNormal not existing. Had to define it myself.

react-three-fiber by example

Three.js Shaders Tutorial (part 2/2) | GLSL Shaders with Uniforms and Varying

Ditch the DOM with SolidJS and Skia CanvasKit

Ryosuke — Fri, 05 Aug 2022 23:03:01 +0000

Recently I’ve become fascinated with cross-platform rendering of UI and approaching the problem from a lower level layer (usually some graphics API, ideally speaking to a GPU). I discovered Skia’s CanvasKit, which is a WASM module for using their Skia API on the web.

If you haven’t heard of Skia, it’s used in Chrome and Android for rendering graphics (like SVGs). It’s a C++ library that you can use to “draw” shapes, text, and other elements to a “canvas” (like the DOM’s <canvas> or even a virtual canvas for rendering server-side in Node).

I’d become more interested in Skia when Shopify created a React Native integration for it - React Native Skia. It was cool to see you could take a C++ library like that and create bindings for different platforms like Android or iOS based on their graphic engines - like OpenGL vs Vulkan.

It got me thinking — what if you could create a library that uses Skia under the hood to render directly to the canvas? Depending on the render target/platform, you could use different “bridges” to the Skia API. On native, you’d use React Native Skia (or a similar implementation). And on web, you’d use Skia’s CanvasKit WASM module.

My goal was to have an library that the developer can write in a JSX syntax and instead of rendering to the DOM (like React for web), we’d render to a <canvas> element.

Want to see the code or try the demo? Check out the Github repo for the final code here: solid-canvaskit-renderer.

Beat to it

After a little research, I found that someone has already beat me to it for React — react-canvaskit. This implementation used react-reconciler to create a custom renderer for React. This is the same process used by **react-three-fiber** (aka R3F) to bring ThreeJS elements to the React VDOM/lifecycle (e.g. when you use <mesh>).

Basically what react-reconciler does is let you create custom elements that are lowerCamelCase (like <ambientLight> or <skParagraph>). When these components render, they call a function or class you specify. In react-canvaskit, they essentially call the CanvasKit APIs.

The author of react-canvaskit did a great job of knocking most of the core architecture down (and even major components). There didn’t feel like much to contribute there. But I still wanted to get my hands dirty and glean something from digging, so I pivoted my research a bit.

SolidJS universal renderer

I had some time to dive into SolidJS a couple weeks ago, and it seems like a great and even much faster alternative to React. It’s basically the same as React — it uses JSX and hooks, with a couple minor differences.

What really attracted me to SolidJS was the Universal renderer. It’s like being able to create your own version of React DOM.

By default, SolidJS will target the web, and use the DOM API to render your app. So when you write a <div> - it knows to actually say document.createElement('div'). This is similar to React if you’ve ever looked at the compiled version of a React app.

Instead of using the DOM, with the universal renderer you can target anything! You could create a “virtual” DOM that you can parse through with a testing framework or server-side render. Or in this case, you could target the <canvas> directly.

We’re still technically using the DOM, since we’re writing HTML and JavaScript (and ultimately rendering that code in a browser). But we cut out the overhead of the DOM, and any time it took the browser to create those new DOM elements (as well as mutate them as we change update our Solid component props).

When I first was looking into the universal renderer, I found **solid-three,** which is a universal renderer for ThreeJS. It’s actually based on react-three-fiber and features a similar architecture and API (using Zustand to store Scene context, or even a useFrame). This would prove as an excellent resource to help craft my own renderer that ditched the DOM.

Initial experiments

I started by doing some initial prototypes that verified that certain processes worked. These were just quick sanity checks, like going through the CanvasKit getting started guide and trying to use the library on my own.

To do this, I spun up a quick NextJS app, installed the CanvasKit WASM module from NPM, and tried using the API inside some components. Here’s the commit with that initial attempt. This worked great — after I figured out some issues with NextJS and WASM modules (I basically tapped out at some point and just used a CDN version of the WASM module).

Once it was working, I abstracted the initialization process to a React Context provider and hook (useCanvasKit). This let me grab the context in any component and “draw” to it.

You can find the complete code for this initial prototype here.

Solid implementation

I had previously done some explorations into SolidJS and universal rendering where I tried to create a ThreeJS version (before I realized one existed). That project was basically a monorepo with a package for the universal renderer, and a “demo” Vite app to test the renderer. I basically just created a new Vite app using their SolidJS template. And for the monorepo — I just copied from my mountain of previous monorepo code.

With this basis in place, I started to create the renderer. It was honestly a very simple process with the API they expose:

// example custom dom renderer
import { createRenderer } from "solid-js/universal";

const PROPERTIES = new Set(["className", "textContent"]);

export const {
  render,
  effect,
  memo,
  createComponent,
  createElement,
  createTextNode,
  insertNode,
  insert,
  spread,
  setProp,
  mergeProps
} = createRenderer({
  createElement(string) {
    return document.createElement(string);
  },
  createTextNode(value) {
    return document.createTextNode(value);
  },
  replaceText(textNode, value) {
    textNode.data = value;
  },
  setProperty(node, name, value) {
    if (name === "style") Object.assign(node.style, value);
    else if (name.startsWith("on")) node[name.toLowerCase()] = value;
    else if (PROPERTIES.has(name)) node[name] = value;
    else node.setAttribute(name, value);
  },
  insertNode(parent, node, anchor) {
    parent.insertBefore(node, anchor);
  },
  isTextNode(node) {
    return node.type === 3;
  },
  removeNode(parent, node) {
    parent.removeChild(node);
  },
  getParentNode(node) {
    return node.parentNode;
  },
  getFirstChild(node) {
    return node.firstChild;
  },
  getNextSibling(node) {
    return node.nextSibling;
  }
});

My first order of action was to take each method and console.log out the parameters. It’d give me an idea of when things ran, and what was needed when.

createElement(string) {
    console.log(string);
},

I quickly discovered my monorepo wasn’t configured appropriately, and I needed to define a custom Babel config that pointed Solid to my custom renderer. Shoutout to the SolidJS Discord community for the protip:

import { defineConfig } from "vite"
import solidPlugin from "vite-plugin-solid"

export default defineConfig({
  plugins: [
    solidPlugin({
      solid: {
        generate: "universal",

        renderers: [
          {
            name: "universal",
            moduleName: "solid-canvaskit-renderer",
            elements: []
          }
        ]
      }
    })
  ]
})

Without this config, SolidJS kept using the DOM, so when I created <testelement> - the DOM actually made one (instead of calling my createElement method in my custom renderer).

Once that was resolved, I was quickly able to wire up the CanvasKit API to the createElement process. I created a <skCanvas> and <skGradient> component. The Canvas class went through the CanvasKit initialization process (aka running init() on CanvasKit, wiring it up to a <canvas> element, picking a backend like OpenGL, etc). The Gradient class rendered a gradient from the documentation samples.

I created a component that used these new custom elements:

function App() {
  return (
      <skCanvas>
        <skGradient />
      </skCanvas>
  );
}

This process didn’t work…but kinda did? When the app initially renders, nothing happens (no errors, only a blank canvas). But when you update a component, the app rendered the components to canvas correctly.

After inspecting the logs, I found that SolidJS creates elements (createElement), then inserts them into your “tree” (insertBefore). I had the canvas initialize and each component render during the createElement stage. I moved the component rendering to insertBefore to try and give the canvas time to initialize — but this didn’t work either.

The problem lies in the CanvasKit init() method. It’s a Promise - meaning it needs to be await-ed in order for the app to run sequentially.

I tried to move out the initialization as a separate method so the user could call it before rendering — but this felt wrong? It didn’t seem like a good practice to hold rendering up for initialization? Ideally the tree should form and wait to render until the canvas is ready (that way things don’t “pause” and do a few things concurrently - like initializing their elements).

To solve the issue, I had to take my brain out of the renderer and back into Solid-land for a bit. I kept trying to find a way to get the renderer to acknowledge the canvas initialization — but there weren’t any methods that made it easy. Instead, I created a <Canvas> SolidJS component that handles the initialization (instead of a custom element like <skCanvas>). This way, I could leverage Solid’s children prop, and suspend rendering until the initialization was complete.

export const Canvas = ({children}: Props) => {
  const [initialized, setInitialized] = createSignal(false);

    console.log('[CANVAS] children', children);

    /**
     * If there's only 1 child, make it an array so we can loop over it
     * @param childrenCheck 
     * @returns 
     */
    const checkChildren = (childrenCheck: ResolvedChildren) => {
      if(!Array.isArray(childrenCheck)) return [childrenCheck];
      return childrenCheck;
    }

    /**
     * Initialize CanvasKit and connect `<canvas>` element
     */
    createEffect(async () => {
      if(!initialized()) {
                // This is the "await" I mentioned earlier
        await initializeCanvas()
        setInitialized(true);
      }
    })

    /**
     * Run the render method on all children (aka "drawing" all child elements)
     */
    const memo = solidChildren(() => children);
    createEffect(() => {
      // We should only be rendering when the canvas is initialized!
      // Otherwise nothing renders unless props change
      if(initialized()) {
        console.log('[CANVAS-C] RENDERING CHILDREN!');
        const realChildren = memo();
        // @ts-ignore Not sure where to wire this up...but this does return SkNode, not JSXElement
        let childrenMap = checkChildren(realChildren) as SkNode[];

        childrenMap?.forEach((c: SkNode) => c.render())
      }
    })
  return (
    <></>
  )
}

export default Canvas;

This solution worked great, and allowed the app initialize appropriately.

Wiring up props

The one thing I didn’t understand about Solid’s renderer — how do props get passed to elements? I noticed the renderer had a setProperty method that used the DOM’s setAttribute API to assign the property to the DOM element (very Vue or Web Component-like). I assumed the DOM element would have access to it’s own attributes (aka “props”).

I didn’t mention earlier, the way my custom renderer works, it uses a special “Node” class I created. This acts as a placeholder for the DOM’s Element API. But in my case, I only need to keep initialize and render instances of CanvasKit classes I’ve created (like a SkText). So my SkNode base class only had initialize() and render().

In order to keep track of props, I created a props property on the SkNode class:

export class SkNode implements SkBase {
    initialize() {};
    render() {};

    // Component props (aka React/Solid props)
    props: Map<string, any> = new Map();
    setProp(name: string, value: any) {
        this.props.set(name, value);
    };
    getProp(name: string) {
        this.props.get(name);
    };
}

Very simple getter/setter methods using a Map to store the props. And with that, I was able to set properties with the renderer setProperty method, and then access them in the component class using this.props.get(propName):

import store from "../store";
import { SkNode } from "./SkNode";

export default class SkGradient extends SkNode {

    render() {
                // ...code cut out...

        const colors = this.props.get('color') ?? ['RED', 'GREEN', 'BLUE'];

                // ...code cut out...
        }
}

And the magic that helped it work — I also call the component’s render method after changing a property. So ideally, only the changed component re-draws.

setProperty(node: SkNode, name: string, value: any) {
  log('Setting prop', node, name, value);
  node.setProp(name, value);

  // Re-render component
  node.render();
},

What’s next?

This was a POC to explore the possibility of this kind of “stack”. I’d like to run some performance benchmarks to see if this is truly more performant than classic DOM. And I’d also like to try writing an entire application using this system to understand the kind of app and component architecture it’d take to organize everything.

Tips and tricks

You don't need to implement all the renderer methods! Most are kinda optional, especially if you don't have things like parent/child relationships between components or component props.
Similarly, you don't necessarily need a (virtual or not) DOM/node tree. If your graphics API is a flat scene graph, you don't need to worry about all the "parent"/"child" methods. If you do need a virtual DOM, I have a VNode class in my template that has most methods you'll need.
SolidJS goes through the component tree as a giant call stack of functions, first starting with createElement, then insertBefore, etc.
Apparently browsers are missing some features that force the CanvasKit to use some JS-based APIs (instead of the WASM code) - which inevitably can contribute to a bottleneck.

Universal everything!

Interested in making your own SolidJS universal renderer? I created a template based off an early iteration of my CanvasKit renderer (basically at the beginning console.log phase). This should be a clean*-ish* slate to get you started to plug in any API you’d like! Clone and try the code here on Github.

Kanpai,
Ryo

References

Leveling up Windows Powershell with Oh My Posh

Ryosuke — Thu, 19 May 2022 23:26:03 +0000

I was sitting on YouTube the other day and looking for a quick productivity win, and I stumbled across Scott Hanselman’s video on making the ultimate Windows Terminal. I’ve recently started doing more web and game development on my Windows machine, so I thought I’d give it a shot and see how much better my shell experience could get.

After watching 15 minutes of his video, I was hooked and already tweaking and customizing my theme to my exact needs.

I thought I’d write about my process and the steps I took to get to my final Terminal setup. The process was similar to “oh my zsh” on Mac if you’re familar with that.

Getting Started

First things first, we need PowerShell. This will work with other shells, but to make this process simple, we’ll start with PowerShell.

Windows 10 comes with a “Windows Powershell”, but it’s not the same as Powershell. You’ll need to download a separate shell if you haven’t already:

Download and install Powershell

⚠️ It’s recommended you use Windows Terminal to manage different shells. That’s a separate download, but it automatically picks up any shells you have (from Windows PowerShell, to PowerShell, to even the WSL (aka Linux) ones. I had to set PowerShell as default in the Windows Terminal settings.

Oh My Posh

Like oh my zsh for Mac, but for PowerShell. Styles your Powershell and displays extra info (like current Github branch).

Windows | Oh My Posh

How to make the ultimate Terminal Prompt on Windows 11 - This video is LONG and WORDY and DETAILED

Getting Started

Make sure Powershell is installed (see top of this page)
Install oh-my-posh. I used the manual method.
Restart your Powershell (or Windows Terminal).
You should see it! But with broken fonts and missing glyphs…
Install special fonts from NerdFont (basically fonts + glyphs like FontAwesome).
- You can usually find your favorite coding font (like Fira Code or Jet Brains Mono)
Go to Windows Terminal settings, select Powershell settings, go to Appearance, and change font to the one you downloaded - it will end in NF (e.g. JetBrainsMono NF)
- You’ll also want to do this for Visual Studio Code if you plan to use the integrated shell/console.
Now things should look correct! 🎆

🎨 Themes

Browse all the themes here or run Get-PoshThemes. The default themes come pre-installed with oh-my-posh in the $env:POSH_THEMES_PATH.

Install the theme by opening your $PROFILE (or .bashrc for WSL) and changing the oh-my-posh initialization we added earlier with your theme config. Here you can see we’re installing one of the default themes - Night Owl:

oh-my-posh init pwsh --config $env:POSH_THEMES_PATH\night-owl.omp.json | Invoke-Expression

⚠️ You can use echo $PROFILE to see where the profile is, and open it with whatever text editor you prefer.

Protip: I use the code command to open things in VSCode. You can activate this by going to the Command Prompt (CMD + Shift + P) and typing “bash” and installing it from there.

Customize | Oh My Posh

My Theme

I created a theme based on powerlevel10k_modern and took some modules from night-owl like the CPU monitor, Node version, and PHP version.

Oh My Posh - My Custom Theme (based on powerlevel10k + night-owl)

3rd Party Plugins

These are extra sprinkles on top of Windows Terminal that help enhance the experience (like adding icons or autocomplete to commands).

Autocomplete using history

Predictive intellisense based off your shell history - like a smart search. So if you go to certain folders often, they’ll pop up in the list as you type.

Install the prelease of the module (requires 2.2.0-beta1 or greater as of 5/17/2022): Install-Module -Name PSReadLine -AllowPrerelease -Scope CurrentUser -Force -SkipPublisherCheck
Add this to your $PROFILE:

# Add auto complete (requires PSReadline 2.2.0-beta1+ prerelease)
Set-PSReadLineOption -PredictionSource History
Set-PSReadLineOption -PredictionViewStyle ListView
Set-PSReadLineOption -EditMode Windows

This wasn’t as simple as the commands sometimes (go figure with prerelease lol) - but these resources helped get things working.

GitHub - PowerShell/PSReadLine: A bash inspired readline implementation for PowerShell

Set-PSReadLineOption : A parameter cannot be found that matches parameter name 'PredictionSource' · Issue #2189 · PowerShell/PSReadLine

Adding Predictive IntelliSense to my Windows Terminal PowerShell Prompt with PSReadline

Terminal Icons

These are nice icons for the different folders and files you have in development projects. If you’ve used VSCode and installed any icon extensions, you’re probably familiar with this.

Run this command to install the module: Install-Module -Name Terminal-Icons -Repository PSGallery
Add this to your $PROFILE: Import-Module -Name Terminal-Icons
Refresh your PS config by running: . $PROFILE
Try running ls or dir to see the icons!

Thanks again to Scott Hanselman for the tip!:

Take your Windows Terminal and PowerShell to the next level with Terminal Icons

My Final Configuration

For reference, here is the final configuration file for my Windows Terminal (aka $PROFILE). I also added some comments that go over the installation of plugins/modules I mentioned above (just in case).

PowerShell Config - Includes Oh My Posh, Terminal Icons, and Autocomplete with History (requires module see below)

What’s your shell look like?

Now that I’ve dipped my toe in the shell customization water, I’m definitely looking for more inspiration to improve my setup and make it even better. Hit me up on Twitter and share your shell setup, I’d love to see it.

React Three Fiber and NextJS Starter Template

Ryosuke — Thu, 19 May 2022 23:18:45 +0000

I’ve been really getting into 3D and game development, from Blender and Maya to Unity and Unreal Engine to even custom programmed C++ game engines using OpenGL or DirectX. The deeper you go, the more you tend to work with languages like Python or C. But what if there was a way to do a lot of the things you do in a software like Unity — but on the web...with JavaScript?

This is where WebGL, ThreeJS, and React Three Fiber come into play. WebGL is the web implementation of the OpenGL specification (aka brings 2D and 3D rendering to web). ThreeJS is a JavaScript library that uses WebGL to create a low-level 2D and 3D graphics API (like drawing a canvas, creating a “camera”, vertex and pixel/fragment shaders, rendering vertex buffers from 3D meshes, etc). And finally — react-three-fiber is a React library that integrates ThreeJS more efficiently into the React ecosystem with easy to use functions and components.

React Three Fiber has been a fantastic way to quickly prototype 3D apps and games, and even efficient enough to carry them through to production. The problem? It’s can be a lot of setup. Just from that last paragraph, you see the string of dependencies R3F uses, and it gets even more complicated from there when you familiarize yourself with the current ThreeJS ecosystem (see: three-stdlib).

I tried to find a template that just worked out of the box, and I was able to find a few, but I wanted a little more. So I did the inevitable — scrapped together my own template: r3f-next-starter.

This template gives you a solid foundation out of the box: NextJS site with R3F + DOM setup, accessibility (aria elements for 3D), quick “toggles” using Leva, and a fantastic shader setup with syntax highlighting.

Using the template

Clone the repo: git clone https://github.com/whoisryosuke/r3f-next-starter.git
Install dependencies: yarn
Run the development server: yarn dev
See some 3D 👀 at http://localhost:3000/

What’s in it?

♻ Easily mix DOM and 3D Canvas
🎨 Fragment and Vertex Shaders (with syntax highlighting + autocomplete)
🎛 Leva panel for debugging
♿ R3F A11y
🏪 Zustand store
📁 Relative paths (@/components/)
📴 Offline mode
🍱 Bundle Analyzer
☑ Typescript

Where I started

I was able to find an example in the NextJS repo that covers the barebones setup for ThreeJS (aka using React Three Fiber). This was great, but not enough.

I dug a little deeper and I discovered create-r3f-app. This CLI didn’t work completely for me on my Windows setup (crashed during a step). Though it did have a partial output which allowed me modify the template to a working point. It also came with Tailwind setup, so I removed any references to it.

The create-r3f-app template was a great starting point, and basically makes up a large portion of my template. It had a great setup for managing R3F Canvas vs DOM elements. And a nice Webpack setup for importing shader files (.frag and .vert). I go into more detail on all the features below.

I went through and started to convert it completely into Typescript, add any goodies I needed (Leva, a11y, etc), and I was basically good to go. I go over these down further down below.

Feature Breakdown

There were a lot of nice things happening in this template (mostly from create-r3f-app) and I thought it’d be cool to go over a few of them in a bit of detail.

DOM vs R3F

When you create a new page, you don't need to wrap anything in a canvas. This automatically happens in the _app.jsx file. The partition script in _app.jsx checks for <Dom> and <R3f>, and wraps your R3F components in a canvas.

import type { AppProps } from 'next/app'
import { useRouter } from 'next/router'
import useStore from '@/helpers/store'
import { useEffect } from 'react'
import Header from '@/config'
import Dom from '@/components/layout/dom'
import partition from '@/helpers/partition'
import dynamic from 'next/dynamic'
import '@/styles/index.css'

// We dynamically import the canvas (required of any R3F component in NextJS-land)
const LCanvas = dynamic(() => import('@/components/layout/canvas'), {
  ssr: false,
})

// This component takes it's children and finds the R3F and DOM
// and splits them up into their own space
const Balance = ({ child }) => {
  const [r3f, dom] = partition(child, (c) => c.props.r3f === true)

  return (
    <>
      <Dom>{dom}</Dom>
      <LCanvas>{r3f}</LCanvas>
    </>
  )
}

// The wrapper around the entire app
// Standard NextJS process/API
function App({ Component, pageProps = { title: 'index' } }: AppProps) {
  // We grab the children whatever page we're rendering
  // and make sure we run any initialProps from NextJS to hydrate the component
  const child = Component(pageProps).props.children

  return (
    <>
      <Header title={pageProps.title} />
      <Balance child={child} />
    </>
  )
}

export default App

Each page should export <DOM /> and <R3f r3f /> components in a JSX fragment (so you can have 2 children without a parent):

import dynamic from 'next/dynamic'

const Shader = dynamic(() => import('@/components/canvas/Shader/Shader'), {
  ssr: false,
})

// dom components goes here
const DOM = () => {
  return (
    <>
      <p>420 blaze it</p>
    </>
  )
}

// canvas components goes here
const R3F = ({ r3f = true }) => {
  return (
    <>
      <Shader />
    </>
  )
}

// The page component that gets "split up" by the `_app.js` file above
const Page = () => {
  return (
    <>
      <DOM />
      <R3F r3f />
    </>
  )
}

export default Page

export async function getStaticProps() {
  return {
    props: {
      title: 'Index',
    },
  }
}

It makes composing pages simpler, mixing DOM and R3F easy, and no fuss with wrapping the canvas every time.

Shader Support

This is actually pretty simple to implement if you’re already using Webpack. You essentially use the raw-loader to load the “raw” shader file and glsify-loader to parse the GLSL shader.

In NextJS, you can extend the Webpack rules to add the plugins:

// next.config.js
config.module.rules.push({
  "test": /\.(glsl|vs|fs|vert|frag)$/,
  "exclude": /node_modules/,
  "use": ["raw-loader", "glslify-loader"]
})

If you prefer a different file format, you can add one here to have it get picked up. For instance, some projects prefer .pixel or pshader for Fragment shaders.

When you browse the shaders (.frag and .vert) in VSCode (or just open the project really), you should see a popup to install a plugin that handles the syntax highlighting and autocompletion.

The suggestion when you open a project is done using the VSCode Workspaces feature and using the Recommended Extensions config (.vscode\extensions.json).

The Changes

Typescript

The first major change was converting everything to Typescript. The create-r3f-app template had TS support — but most of the files were still in .js format. This unveiled a couple of issues with shader files (.frag), whenever they were imported in a file I saw an error saying they couldn’t be found. I needed to create a new TS definition file shader.d.ts file (see this StackOverflow):

// shader.d.ts
declare module '*.vtx' {
  const content: string
  export default content
}

declare module '*.frg' {
  const content: string
  export default content
}

CSS Reset

Since I removed Tailwind, I also had removed their “base/reset” CSS file that helps “normalize” the style output across browsers. I opted to integrate the classic Normalize.css and call it a day.

a11y Support

One of the major things missing from the template was the new react-three-a11y library. This lets you wrap your 3D components in an <A11y> component that does a few things:

Creates an aria-compliant HTML element in the DOM for your canvas element
Lets you add logic for focus events (e.g. onFocus) in the 3D layer

This one is really easy to integrate, so I just installed it, added a component to “announce” things to screen reader, and wrapped the existing components in accessibility wrappers. Done! No excuse not to make your app or game more accessible.

Quick toggles with Leva

One of the most powerful tools during development and prototyping is the ability to alter properties using a GUI. For example, using sliders to change an objects position or shader properties like colors is incredibly satisfying.

This was another quick and easy add. Just install and wire up some component properties to the panel using their useControl hook.

CodeSandbox optimization

One of the first things that you’ll notice when you work in R3F and an online code editor like CodeSandbox — it crashes way too much. This usually happens because of 2 things: the file constantly hot reloading in bug-prone areas (like the useFrame hook), and the hot reloading messing with the canvas state ThreeJS has vs the R3F/React side (canvas often just drops out / crashes).

The way to avoid this is by adding a configuration file to let CodeSandbox know not to reload on each keystroke, and avoid infinite re-render when possible:

// sandbox.config.json
{
  "infiniteLoopProtection": true,
  "hardReloadOnChange": false,
  "view": "browser"
}

What’s next?

I’m looking to add some more examples to the template, just things that I find myself repeating often - like the proper way to type a R3F React component props. Is it Vector3 or number[3]? Answers like that should just be a copy/paste away instead of spending time Googling or digging through the R3F Discord.

Anything that’s too intense or might bloat the template is going to end up in my new r3f-experiments repo.

But what can you do?

Lots of cool stuff — from award winning app experiences, to complete 2D/2.5D/3D games in your browser!

I took the template, cloned it, and created a mini Asteroids inspired game. All I had to do was add a player “model” (aka a cone), add some asteroids (aka cubes) to the scene, and rig up the player to keyboard input (WASD for movement). Oh and figuring out how the move the camera with the player model was a little tricky. But you can see from the commits it didn’t take a lot of code to create this whole experience.

I hope this inspires you (and make it easier!) to create your own 3D prototypes and experiments. As always you can reach out to me on Twitter if you want to chat about anything or ask a quick question.