DEV Community: Techie Kho

Bag: A header-only, simplistic, C++20, zipping & unzipping library.

Techie Kho — Fri, 10 Jan 2025 06:13:39 +0000

Hello! It has been a while since the last time I did something for entertainment and learning purposes. So I take up a fairly simple C++ project that hopefully could be useful for other fellow C++ devs.

Problem Statement

As I play around with raylib, the small, easy, and exceptionally cool game library, I met a small issue. There is no elegant way of packing files together. When I use a texture or audio, or any external resources, I either need to convert it into a C++ header that declares a byte array filled with that resource, or making sure the the user having the resource at the particular place. The first approach is no good when comes to loading new resource after compilation, while the latter approach is kind of awkward and too easy to break.

I'm quite fond of the embedded zip approach. Zip files having a quirky file structure in which the header and metadata is at the end of the file in contrast with other file structure, such as executables, which is at the beginning of the file. This quirk gives a possibility, which is appending the zip file at the end of the executable. This makes the file behave like an executable, and a zip file at the same time.

I believe there is definitely other zip library that did what I requested. But I would like to have some fun in C++20. So let's get started.

Overview of bag

With the aforementioned reasoning in mind, I wrote libbag, a header-only library for simple zipping and unzipping. It also includes the zipping (bag) and unzipping (unbag) utilities.

By itself, it is sort of like a serialiser and deserialiser for key-value pairs, with key being a simple C-style string and value as byte array. The bag file structure is fairly simple:

|------------------------|
| key 0 (C-style string) |
|------------------------|
| Content 0 (Byte array) |
|------------------------|
|       ......           |
|------------------------|
| key N (C-style string) |
|------------------------|
| Content N (Byte array) |
|------------------------|
| indices                |
|------------------------|
| metadata               |
|------------------------|

So the key-value pairs are tightly packed together, and some data is appended at the end of the pack. Metadata specifies the whole bag's byte count (to ignore any excess byte in front), position and length of indices, and lastly some magic bytes to indicate it is a bag.

From the length and position of indices specified in metadata, we can query the memory to get the indices slice in bag. There it resides a list of position and length for each of the key-value pairs saved.

To get the keys, just treat each key-value pairs as a C-style string since there is a null-terminator that marks the end of the key string. To query the content, just skip 'till the first null-terminator (basically the whole key), and then you get the content begin position, with some pointer arithmetic using the position and length from the indices, the end position of the content can be calculated.

Here is some example usage.

#include<libbag.hpp>
#include<sstring>


int main() {
    std::map<libbag::key_type, libbag::content_type> input_map;
    /// Add items into `input_map`.
    std::basic_ostreamstring<libbag::unit_type> packed_stream; // Can be replace with std::basic_ofstream to output to file.
    libbag::pack(input_map, packed_stream); // Pack.

    libbag::unit_string_type packed = packed_stream.str();
    std::map<libbag::key_type, libbag::content_type> result;
    libbag::unpack_all(libbag::bag_type(packed.begin(), packed.end()), std::inserter(result, result.end())); // Unpack.
    /// Do something about the unpacked data.
}

The API depends on iterators, stream and inserters to reduce allocations.

I also wrote a simple CLI to zip and unzip files, which is great for referencing as well:

bag
unbag

React website sample for portfolio

Techie Kho — Thu, 11 Apr 2024 15:18:51 +0000

TL;DR: I make a website about kitties for portfolio, and here is the source code

Introduction

Hello people! So I have been trying to do a little freelance as a front-end web developer. Programming is my passion, but it has yet to feed me. I always heard that front-end web developer is the most common job in the market; many start-ups need a website, right(?). I've realised that I don't really have a web project to show off my web-making skill. My portfolio website doesn't count as it is made using vanilla javascript. So I guess it is time to make a random website using modern and popular framework like React.

The tech stack is very simple:

Vite for project setup and bundling.
React (of course)

I was about to use Material UI as they give a bunch of cool reusable components that is already stylised. But, it feels too "Google" and I am a C programmer (having an obsession over reinvent the wheel).

Setting up the project

Vite projects are VERY easy to setup to the point that I feel like it is too good to be true. The last time I play around with web, vite is yet to exist and configuring webpack took me 30 minutes of scanning through the docs only to be forgotten once the setup is completed.

For vite, it really just these magic word:

npm create vite@latest

Do some command line inputs and it is done, amazing.

Working the design

Alright, I'll going to be honest here: I kind of want to finish this ASAP in the weekend so I made things up along the way instead of planning it in Figma.

I first checking out any good fonts on Google font that fits the theme of the website. I select the Nunito as I could feel the playful vibe behind it.

Then, I get some color palette from this color palette generator. I have been using it since the very beginning of web dev journey, a good color palette could better describe the theme of the website.

CSS

Writing CSS is much a creative process here since I don't have a mockup, thus it is kind of describe it in detail. But I will still share things that I usually do in CSS.

reset.css
I will always use the reset stylesheet. It is because different web browser has its own default stylesheet that could mess up my code. So I just copy the code and put it in the project.
CSS variables
I love CSS variables, I use it to for the color palette.

:root {
  --light: #f5f5f5;
  --grey: #796c9f;
  --dark: #392f5a;
  --primary: #2ea9c4;
  --primary-light: #a1d8eb;
  --secondary: #ff9f1c;
  --secondary-light: #ffbf69;
}

It avoids hardcoding colors. Then, I only need to change the value of the variable instead of searching through the codebase for that particular color code.

Javascript

Since it is React, I need to do something special here.
I decided to use the Cataas API to fetch cat pictures in my code to display the random cats.

But behind the scene, it is just useState and useEffect hook.

  const [catImage, setCatImage] = useState(initialCatImage);

  useEffect(() => {
    const updateCatImage = () =>
      (async () => {
        const response = await fetch(
          `https://cataas.com/cat?width=250&height=250`
        );
        setCatImage(URL.createObjectURL(await response.blob()));
      })();

    const id = setInterval(updateCatImage, 5000);
    return () => {
      clearInterval(id);
    };
  });

And that is all the special stuff in the website...

Deployment

I'm using pgs.sh to host my static site. It uses ssh (secure shell) and rsync (a file transfer protocol) to do all the stuff. It doesn't need to create account or whatsoever. Very quick to setup, very cool.

Conclusion

Well, I got a website to show-off and maybe get a freelance project. And, you get to see all my source code and learn something. It is beneficial to both of us. Thank you for reading this late night posting and have a nice day/night.

Running pre-trained ML models in Godot

Techie Kho — Thu, 18 Jan 2024 12:21:04 +0000

So I have been developing this GDExtension called iree.gd. It is mission to embed IREE, another cool project that compiles and runs ML models, into Godot. It took me quite a while, but finally It has reached alpha.
Hope you guys could check it out the sample.

I can't upload video to here, but you still can watch a small clips of iree.gd doing its magic here

Overview

Currently, it is still having a very thin abstraction layer, and some manual stuff need to be done in prior.

Preparing the ML model

Before doing anything, you'll need to download your trained ML model from the internet. You could find some from Kaggle. Then, you'll need to preprocess them with python via this detailed documentation. You'll need to set the HAL backend following this table depending on your targeted platform. As you generate the .vmfb byte code, you are ready to proceed to the next step.

Use the ML model

.vmfb files are automatically imported into the project as you drag and drop 'em into it. You could then use them in your script by loading them.

var model : IREEModule = preload("res://your_model.vmfb")
@export var model_2 : IREEModule # Drag and drop in editor

Then, you'll need to prepare your inputs with IREETensor, then execute the ML model.

var input := IREETensor.from_bytes(image.get_data(), [1, 50, 50, 3]) # Remember to consider the input type.
var outputs := module.run_module("module.main", [input])
for output in outputs:
    pass # Do something with the `output`.

The input and function name vary models to models. You could inspect the .vmfb file using iree-dump-module from the IREE packages you installed during preparing the ML model.

Here is one section of the dump of the ML model used:

...
Exported Functions:
  [  0] main(!vm.ref<?>) -> (!vm.ref<?>)
        iree.abi.declaration: sync func @main(%input0: tensor<1x50x50x3xf32> {ml_program.identifier = "input_0"}) -> (%output0: tensor<1x200x200x3xf32> {ml_program.identifier = "Identity"})
  [  1] __init() -> ()
...

You could kind of guess it, it has main function that takes a tensor of float32 (1x50x50x3xf32) and outputs another tensor of float32 (1x200x200x3xf32).

Sample

There is a readily available sample of esrgan for you to try in the release page.

The sample only work on windows and linux.

After download and open iree-gd-sample-*.zip with Godot 4.2+, you will need to do these:

set res://scenes/main/main.tscn to be loaded on start.
Drag and drop appropriate byte code in res://bytecodes/esrgan/esrgan.*.vmfb into UI/TextureRect's module export.
Run it.

Afterword

Hopefully, in the future, Godot developer could utilise this piece of technology and create something extraordinary and magical. Thank you for reading this.

Minky scripting language

Techie Kho — Wed, 15 Nov 2023 12:38:29 +0000

Edit: It has been rebranded to Blinklet, here is the repo

So I have written a simple scripting language in Rust, the obscure language for a lot of devs. It is called "Minky".

Here is the link

The whole idea of this language stems from this thought:

What if we remove parentheses from LISP?

Essentially, Minky is my version of WISP.

Syntax

Let's jump into the syntax:
In a typical C-derived language, a function call usually look like this:

function(arg1, arg2, arg3)

In Minky, since we disdain parentheses, we replace all the separating symbols with whitespace

function arg1 arg2 arg3

But... what if I want to inline a function call as an argument like this?

function(arg1, arg2, another_function())

Well, in Minky, you just indent it!

function arg1 arg2
    another_function # This will call

If you don't indent it, another_function will be passed as value.

But! What if I want to pass value after some indents?

function arg1
    another_function 
    arg3 # I don't want this to get call!

Well, you just prefix it with | to cancel out the call

function arg1
    another_function
    | arg3 # Pass as value!

Some features

String interpolation
Closure
Loose Object-Oriented Programming

String interpolation

To do string interpolation:

some-command 'Text `identifier`'

The identifier is the variable name, it will substitute with the variable's string representation.

Closure

All the user-defined functions are closure. It has access to the lexically scoped variable. In simple term, the scope that the closure is declared in lives with the closure until the closure itself dies, and the closure can access the scope.

Here is an example:

var make-counter
    closure
        var counter 0 # counter will live with the returned closure even after `make-counter` complete execution.
        return
            closure
                set counter # access parent counter.
                    add counter 1
                println 'counter: `counter`'

var counter
    make-counter

counter # 1
counter # 2
counter # 3

Loose Object-Oriented Programming

In normal class-based object-oriented programming, we have class and object, in which the object's structure is based on the class. In Minky, all of the classes and objects are table. Tables are not linked together in anyway, but you can make a copy of a table (instancing) and extends on top of it (inheritance). Table is also act as scope, too!

To make a person class:

var Person
    table
        var name 'Bazinga'
        var age 25
        var say-hi
            closure # aka. function but has access to parent scope.
                println 'Hi! I am `name`. Nice to meet you!'

To make an instance:

var peter
    Person
        set name 'Peterson'
        set age 90

# Call the method
peter
    say-hi

# Access values
var peter-age
    peter
        return age

To extend:

var Police
    Person
        set name 'Policeman'
        set age 15
        var gun 'AK15'

Conclusion

It is my attempt on making a scripting language. There are more feature to be discussed, and to be added. You can fetch the code at my GitHub repo. Thank you for reading and have a nice day.

Piko-piko OS. A homemade 16-bit x86 toy operating system for fun.

Techie Kho — Wed, 07 Dec 2022 11:33:18 +0000

So I made a 16-bit x86 toy OS in pure assembly. 3 months ago, I found a very fun tutorial on github that is about Operating system development. I read the first few chapter and from there I made a very simple, extensible (?) toy operating system that could run on hardware (yes, it is madness).

Piko-piko OS overview

As the title stated, it is a toy os so itself doesn't do much. Still, the language is still turing complete. Here is an example that prints Hello world 10 times.

00000 set 0 10
00001 say n "Hello world"
00002 sub 0 $0 1
00003 cmp $0 0
00004 june 1

The language is more or less like an interpreted assembly language without label. It can read user input, print to output, basic arithmetic, save & load data and conditional jumps. Here is all the commands included. I'll write a more user-friendly tutorial to kick start your Piko-piko OS coding journey, but right now you can only

In Piko-piko OS, you can either enter the command to the prompt to run the command directly or using = command to save lines of command to buffer and run it all together with run command. By default, there is 3 buffer. Each buffer can only holds 2560 lines of text.

Piko-piko OS is self-contained, which means that it does not rely on any other code and won't effect other files except itself. When saving & loading data, it does read and write to itself only.

Practicality?

Yes, it is not practical but it is a very educational resource to learn lower-level programming and really knowing how machine work. Still, you can use it to:

Show off
Stop your child from watch too much youtube by locking them up in Piko-piko OS.

It is a very cool experience to be able to make an operating from scratch. I might be able to share the experience of developing Piko-piko OS on the next blog post.

Don't write HTML page, I will tell you why.

Techie Kho — Mon, 11 Oct 2021 10:07:03 +0000

Instead of encourage people, I am here to discourage you a guy, who started / want to start web development.

The start

At the start your journey, you install either sublime or vscode or notepad++, then you start to make a website using HTML and know that HTML is a HyperText Markup Language. You feel proud of yourself as your HTML page is displayed in google chrome. Then, you color your page with CSS and make your page interactive with alert() and document.write().

framework hell

Slowly, you think you are very good, then you heard of React or Angular and you immediately jump into it. In the blink of an eye, you are overwhelmed. What the hell is Webpack? What is JSX? What is Nodejs? What is typescript? What is npm? What is Sass? Why a simple website need so much technology with weird names?

server-side programing

At the verge of giving up, you heard of database and server. You install XAMPP or WAMPP and then start poking around with PHP. You also play around with expressjs. Then, you are once again overwhelmed. What is composer? What is a session? What is phpMyAdmin and MySQL?

game development

After you learn one or two things, you think desktop application and game development is cool, so you install Unity. Oh my, Object-Oriented Programming in C# (pain in general). You think that ElectronJS is easier and want to give it a shot but because everyone critisize that it is less performant so you give up.

linux

Then, you heard that people keep telling you "I use Arch, btw" and "Vim is the best editor". You watch a few shitpost from Mental Outlaw (on youtube) and decided to install gentoo linux. You are once again overwhelmed. Why so much command (cd, touch, ls, emerge, time, ifconfig, vim and etc.)? Systemd called as soystemd? OpenRC? Init system? GNU and Linux is different? Display server? X.org vs Wayland? dwm vs bloated Gnome?

lower-level programming

Then as you get used to the tty and know how to edit some text in vim. You start to do some C and C++ programming. Although the OOP from C# or Java helps, you still feel overwhelmed. CMake, Ninja, gcc, tcc, clang, llvm and etc.
Maybe you also step on some assembly language and suffers.

Ok I will stop now. All these painful learning + failing + imposter syndrome + mental illness is all because of you decided to install sublime / vscode to create a HTML page.

In conclusion, don't create your first HTML page.

Here is my website without fancy stuff like React or Angular or Vue

Techie Kho — Sat, 09 Oct 2021 01:47:46 +0000

A website that I created using only HTML, CSS, and JS : https://rechiekho.github.io/RechieKho/index.html

Yes, its content is really stupid because I can't come out a new one.