DEV Community: Lucas Damian Johnson

I Published My First Game To Steam Using Electron, BabylonJS, & Divine Voxel Engine

Lucas Damian Johnson — Fri, 09 Feb 2024 16:11:10 +0000

Hello my name is Luke and I've been on an interesting journey the last few years building an open source voxel engine and finally publishing my first game that uses the engine.

The game is marked as "coming soon" until the 20th of February but you can play it online for free until then. This is just Steams rules.

What The Game Is

Watch Trailer

Crystalline Bliss is a unique atmospheric 3D puzzle game that takes places in various voxel worlds. The basic idea of the game is to match enough same colored crystals together to clear them from the board.

Competitive mode adds a lot of variety since there are different types of crystals that can do different things. For instance there are crystals that cause good things to happen to the player that popped it and bad things to happen to another random player. There are also random effect crystals that can stack creating either very helpful effects or devastating ones for the player.

Development

The game is shipped to Steam as an Electron app.
The game itself though is built with Divine Voxel Engine and BabylonJS. The game also does use a few other libraries that I wrote. I could go on about the development of the game. If people want to know I will make a longer post.

Future Plans

I plan on making the game fully multi-player over network. So you and 3 of your friends can play together.
I also plan on adding more levels and content.
Really though I will also have to see what people want most and focus on that.

What I've Learned

This was a massive undertaking just to build this one project. I think you kinda have to be a little crazy or ignorant to build a lot of this from scratch like I did.

If you can find good libraries use them and don't get caught up in the weeds of writing everything from scratch if you don't have to.

But that being said my engine does a few things most other available JavaScript based engines do not. Such as Minecraft style sun light and voxel light. So there was some things I was going to have to write from scratch anyway.

Also, monorepos are awesome for big JavaScript projects like this. I just use npm workspaces and it links everything together beautifully.

Links

Steam Link
Link To Play Online
Link To Voxel Engine

Compress & Decompress An ArrayBuffer Client Side in JS

Lucas Damian Johnson — Mon, 28 Nov 2022 16:36:56 +0000

The Compressions Streams API is new and up and coming way to compress and decompress data in the browser. Please note the support for the API is not very wide so use at your own risk.

I am writing this because I did not find any good examples of just compressing/decompressing an ArrayBuffer which is what I needed the API to do. The following two code snippets show how to do exactly that.

Compress An Array Buffer

const compressArrayBuffer = async (input: ArrayBuffer) => {
  //create the stream
  const cs = new CompressionStream("gzip");
  //create the writer
  const writer = cs.writable.getWriter();
  //write the buffer to the writer 
  writer.write(input);
  writer.close();
  //create the output 
  const output: Uint8Array[] = [];
  const reader = cs.readable.getReader();
  let totalSize = 0;
  //go through each chunk and add it to the output
  while (true) {
    const { value, done } = await reader.read();
    if (done) break;
    output.push(value);
    totalSize += value.byteLength;
  }
  const concatenated = new Uint8Array(totalSize);
  let offset = 0;
  //finally build the compressed array and return it 
  for (const array of output) {
    concatenated.set(array, offset);
    offset += array.byteLength;
  }
  return concatenated;
};

Deeompress An Array Buffer

const  decompressArrayBuffer = async (input: ArrayBuffer): Promise<Uint8Array> =>{
  //create the stream
  const ds = new DecompressionStream("gzip");
  //create the writer
  const writer = ds.writable.getWriter();
  //write the buffer to the writer thus decompressing it 
  writer.write(input);
  writer.close();
  //create the output
  const output: Uint8Array[] = [];
  //create the reader
  const reader = ds.readable.getReader();
  let totalSize = 0;
  //go through each chunk and add it to the output
  while (true) {
   const { value, done } = await reader.read();
   if (done) break;
   output.push(value);
   totalSize += value.byteLength;
  }
  const concatenated = new Uint8Array(totalSize);
  let offset = 0;
  //finally build the compressed array and return it 
  for (const array of output) {
   concatenated.set(array, offset);
   offset += array.byteLength;
  }
  return concatenated;
 }

Using Binary Data In JavaScript

Lucas Damian Johnson — Sat, 26 Nov 2022 15:32:51 +0000

This article discusses working with raw binary data in JavaScript.

The Basics
Bitwise Operations
Encoding And Decoding Data
Buffers
Typed Arrays
DataViews
Applications

The Basics

Binary data in JavaScript comes in a few forms:

Numbers
- A single number variable can be seen as binary data.
Buffers
- There are two types of buffers ArrayBuffer and SharedArrayBuffer. Both are essentially fixed arrays of binary data.
- Node has an object called Buffer which is like ArrayBuffer but slightly different.
Blobs
- You can extract an ArrayBuffer from a Blob.

To work with this binary data we have a few options:

Bitwise Operations
- We can use binary math to extract and encode meaningful data.
TypedArrays
- We can use TypedArrays to read the data as fixed number arrays.
DataViews
- We can use DataViews to work with and extract specific bytes.

Bitwise Operations

Bitwise operations allow us to change bits of numbers and create new numbers from two inputs. They are useful for encoding and decoding data.

Limitations
When using bitwise operations on numbers all numbers get converted into 32 bit signed integers even though all numbers are stored as 64 bit floats. However, you can use the BigInt number type and work with all 64 bits of the number.

const bigInt = (2n ** 64n) >> 32n;
4294967296n
const normalNumber = (2 ** 64) >> 32;
0

AND

The & operation sets a bit to 1 if only the bit it is being compared to is also 1.

const v1 = 0b1111;
const v2 = 0b1011;
const result = v1 & v2;
0b1011

OR

The | operation sets a bit to 1 if one of the bits being compared is 1.

const v1 = 0b1111;
const v2 = 0b1011;
const result = v1 | v2;
0b1111

XOR

The ^ operation sets a bit to 1 if only one of the bits being compared is 1 otherwise it sets it to 0.

const v1 = 0b1111;
const v2 = 0b1011;
const result = v1 ^ v2;
0b0100

NOT

The ~ operation flips all bits.

const result = ~0b1011;
0b0100

Zero Fill Left Shift

The << operation shifts bits to the left by a given amount.

const result = 0b0011 << 2;
0b1100

Signed Right Shift

The >>> operation shifts bits to the right by a given amount and preserves the sign if the number is negative.

const result = -5 >> 2;
-2

This operation can also be used for flooring a number:

const result = 1.2 >> 0;
1

Zero Fill Right Shift

The >>> operation shifts bits to the right by a given amount.

const result1 = -5 >>> 2;
1073741822
const result2 = 0b1100 >>> 2;
0b0011

Encoding And Decoding Data

Here is a basic example showing how you could use a single number to store 8 numbers with a range of 0 to 15.

const mask = 0xf;
const bitSize = 4;
const getValue = (data: number, index: number) => {
    index *= bitSize;
    return ((mask << index) & data) >>> index;
}
const setValue = (data: number, index: number, value: number) => {
    index *= bitSize;
    return (data & ~(mask << index)) | ((value & mask) << index)
}

So, we have two functions here. The getValue function will take the encoded number and return the value at the given index. While the setValue function will take the encoded number and encode the provided value at the given index and then return the new encoded number.

Here is a basic example of using it:

let v = 0;
v = setValue(v,4,12);
//v is now 786432
getValue(v,4);
//value is 12

Now, I am going to break down what is exactly going on line by line.

First thing we do is define as mask and bit size:

//0xf = 0b1111 = 15
const mask = 0xf;
const bitSize = 4;

This mask here is in the hex number format. We are defining the mask as 4 bits that are all 1. This will define the range of the numbers stored as 0 though 15. We also declare bitSize so we can access the values stored in the number more like an array.

Then we define the getValue function. Here is the logic of the function expanded out:

const getValue = (data: number, index: number) => {
    //get the bit index
    index *= bitSize;
    //create a new mask by right shifting by the bit index
    const newMask = mask << index;
    //remove all other data expect the data at the given index
    const value = newMask & data;
    //right shift the number to get the actual value
    return value >>> index;
}

And finally we define the setValue function. Here is the logic of the function expanded out:

const setValue = (data: number, index: number, value: number) => {
    index *= bitSize;
    //create a new mask by right shifting by the bit index
    let newMask = (mask << index);
    //invert the new mask
    newMask = ~newMask;
    //remove the data at the given index
    data = data & newMask
    //remove all bits that do not fit within the mask
    value = value & mask;
    //left shift the value to the index it will be stored at
    value = value << index;
    //or the data and the value thus encoding the value at the index
    return data | value;
}

Buffers

JavaScript uses buffers or "byte arrays" to work with raw binary data. You can not directly access the data. You need to use another object such as a Typed Array to work with the bytes of the buffer.

Buffers are useful because they can be sent to and shared with other threads without copying the data. They can also be easily sent and received through web sockets, compressed, and saved to the file system.

ArrayBuffer

An ArrayBuffer object is a fixed length array of bytes. You can make one like this:

const buffer = new ArrayBuffer(4);

In this case we set the length of the buffer to be 4 bytes. So, if we created a TypedArray that used 1 byte per number we could store four numbers in that buffer.

SharedArrayBuffer

A SharedArrayBuffer is basically the same as the ArrayBuffer but it can be shared by multiple threads.

const buffer = new SharedArrayBuffer(4);

Please check out the MDN Docs about the SharedArrayBuffer to learn how to properly get it working.
Also, if you plan to use one you may run into race conditions. You can use Atomics to aid with fixing that.

When an ArrayBuffer is transferred to another thread the thread that sent it losses access to it. The SharedArrayBuffer then basically acts as a pointer to the same buffer.

Typed Arrays

Typed Arrays are fixed length arrays that act as a view over a buffer. They can either be created with or without a buffer.
Here are the typed arrays:

//1 byte per number | range: -128 - 127
const int8 = new Int8Array();
//1 byte per number | range: 0 - 255
const uInt8 = new Uint8Array();
//1 byte per number | range: 0 - 255
const clampedInt8 = new Uint8ClampedArray();
//2 bytes per number | range: -32768 - 32767
const int16 = new Int16Array();
//2 bytes per number | range: 0 - 65535
const uInt16 = new Uint16Array();
//4 bytes per number | range: -2147483648 - 2147483647
const int32 = new Int32Array();
//4 bytes per number | range: 0 - 4294967295
const uInt32 = new Uint32Array();
//4 bytes per number | range: -3.4E38 - 3.4E38
const float32 = new Float32Array();
//4 bytes per number | range: -1.8E308 - 1.8E308
const float64 = new Float64Array();
//4 bytes per number | range: -2^63 - 2^63 - 1
const int65 = new BigInt64Array();
//4 bytes per number | range: 0 - 2^64 - 1
const uInt65 = new BigUint64Array();

You can create a typed array in a few ways:

//Creates a typed array with the given length
const ex1 = new Uint16Array(4);
//Creates a typed array from an array
const ex2 = new Uint16Array([1,2,3,4]);
//Creates a typed array from an ArrayBuffer
const byteSize = 4 * 2;
const buffer = new ArrayBuffer(byteSize);
const ex3 = new Uint16Array(buffer);
/* Creates a typed array from an ArrayBuffer 
at a given index and byte length */
const buffer2 = new ArrayBuffer(byteSize * 2);
const ex4 = new Uint16Array(buffer,byteSize,byteSize);
//Creates a typed array from a SharedArrayBuffer 
const SAB = new SharedArrayBuffer(byteSize)
const ex5 = new Uint16Array(SAB);

You can use a Typed Array by indexing it like a normal array:

const data = new Uint16Array(4);
data[0] = 300;
data[1] = 1000;
data[2] = 10_000;
data[3] = 5_000;

DataViews

The DataView object lets you set and get sets of bytes from buffers.

Here are its functions:

const buffer = new ArrayBuffer(16);
const dv = new DataView(buffer);
//1 byte signed int
const int8 = dv.getInt8(0);
dv.setInt8(0,int8);
//1 byte un-signed int
const uInt8 = dv.getUint8(0);
dv.setUint8(0,uInt8);
//2 byte signed int
const int16 = dv.getInt16(0);
dv.setInt16(0,int16);
//2 byte un-signed int
const uInt16 = dv.getUint16(0);
dv.setUint16(0,uInt16);
//4 byte signed int
const int32 = dv.getInt32(0);
dv.setInt32(0,int32);
//4 byte un-signed int
const uInt32 = dv.getUint32(0);
dv.setUint32(0,uInt32);
//4 byte float
const float32 = dv.getFloat32(0);
dv.setFloat32(0,float32);
//8 byte float
const float64 = dv.getFloat64(0);
dv.setFloat64(0,float64);
//8 byte signed big int
const int64 = dv.getBigInt64(0);
dv.setBigInt64(0,int64);
//8 byte un-signed big int
const uInt64 = dv.getBigUint64(0);
dv.setBigUint64(0,uInt64);

You can create a DataView object in a few ways:

const buffer = new ArrayBuffer(10);
//Create a DataView from an ArrayBuffer
const dv = new DataView(buffer);
//Create a DataView from part an ArrayBuffer with a given offset and length
const dv2 = new DataView(buffer,5,5);
//Create a DataView from a SharedArrayBuffer
const SAB = new SharedArrayBuffer(10);
const dv3 = new DataView(SAB);

Here is a basic example of using a 3 byte ArrayBuffer and a DataView object to update different parts of the buffer.

const buffer = new ArrayBuffer(3);
const dv = new DataView(buffer);
dv.setUint8(0,10);
dv.setUint16(1,300);
//value is 10
dv.getUint8(0);
//value is 300
dv.getUint16(1);

Applications

My voxel engine which I previously wrote about makes extensive use of binary data encoding, the SharedArrayBuffer and the DataView object. You can see what I wrote about it here:
How I made a multi-threaded voxel engine in TypeScript

I also wrote another library called Divine Binary Object which lets you encode and decode data from buffers.

However, there are actually a lot of other applications for binary data in JavaScript:

Web Audio API
- The API uses an raw audio buffers to create audio source nodes. Check out: BaseAudioContext.decodeAudioData
Web GL API
- You can use Typed Arrays to create mesh data.
Web Sockets
- You can communicate with raw binary data using web sockets.
Compression Streams API
- You compress and decompress raw binary data.
- You can also do this in Node.
Transferable objects
- You can transfer ArrayBuffers and TypedArrays to other threads without copying them.

How I made a multi-threaded voxel engine in TypeScript

Lucas Damian Johnson — Thu, 10 Nov 2022 15:11:33 +0000

Hello, I am Luke the creator of Divine Voxel Engine.

Divine Voxel Engine (DVE) is a truly multi-threaded voxel engine written in TypeScript that uses Babylon.Js.

In this post I just want to go over on a high level how it works.

Chunk Data And Shared Array Buffers

The engine stores the data for voxels in chunks. Chunks are segments of the world. By default they are defined as 16mx16mx16m areas. They are stored in a multi-level hash map. The first level being a region (512mx512mx512m area), then the next being a column (same width and depth as a chunk but as tall as a region).
The chunk data itself is just a SharedArrayBuffer. The engine uses the DataView object to read and write data to it. The buffer has a few headers such as its' position and height map.
The buffer is shared with all threads (workers) that need it. This way chunk building, water flow, light updates, and much more can be done in parallel.

Data Indexing

Since the data for the chunk is basically stored in a flat array we need to get the index for it ourselves.

The next code snippets are part of objects in the engine.

The first thing we do is get the chunk position:



 getChunkPosition(x: number, y: number, z: number) {
  this.__chunkPosition.x = (x >> 4) << 4;
  this.__chunkPosition.y = (y >> 4) << 4;
  this.__chunkPosition.z = (z >> 4) << 4;
  return this.__chunkPosition;
 }

Then we get the relative voxel position in the chunk:



 getVoxelPositionFromChunkPosition(
  x: number,
  y: number,
  z: number,
  chunkPOS: Position3Matrix
 ) {
  this.__voxelPosition.x = Math.abs(x - chunkPOS.x);
  this.__voxelPosition.z = Math.abs(z - chunkPOS.z);
  this.__voxelPosition.y = Math.abs(y - chunkPOS.y);
  return this.__voxelPosition;
 }

Then we get the index of the voxel in the chunks data:



const bounds = {

  x : 16,

  y : 16,

  z : 16

};

 export function getIndex(x: number, y: number, z: number) {

  return x + y  bounds.x + z  bounds.z * bounds.y;

 }

Voxel Data

Voxel data is stored in 4 bytes.
The first two bytes are for light data. Light is stored as 4 bits for each light type. Light types include sun light and RGB light.
The next two bytes are for storing the voxel numeric id. After that the last bytes are for storing voxel shape state and voxel level as well as secondary voxel id for dual voxels (think of a plant underwater).

Mesh Building

All mesh building is done in the Constructor thread. A chunk is sent into a processor which will produce a template. The template is simply a list of all the exposed faces, the lighting gradient (and other gradients), texture uvs, and the actual voxel shapes themselves.
This template is then sent to a mesher which produces the actual mesh data for the chunk. Once it is done it just sends it to the render thread.
If you want to see how the light gradient and ambient occlusion is calculated check this out:
CalculateVoxelLight

Light Updates

Lighting for the voxel engine uses a simple breadth first search algorithm. If you want to see the code directly here it is:
RGB light
Sun Light
Sun light is a little different than RGB light. Since sun light does not deplete as it goes downward through air while RGB light does.

Inter Thread Communication

The engine is stitched together with another library I wrote called ThreadComm. It makes it easy to communicate between many threads and is set up to work in both the browser and the node environment.
ThreadComm allows you to create tasks queues which are basically functions in other threads which are executed from another thread and have a call back once all the tasks for that queue are finished.
The engine does not use one big queue for every update. It splits updates into regions. So, updates far away from one another will not be affected.

Conclusion

This was just a very high level overview of how DVE works. If you would like to know more or have specific questions let me know. DVE is licensed under MIT so you can use it for any project. However, all assets for the test worlds are owned and copyrighted by me.