DEV Community: Hetarth Shah

Cells, Queries, and Chaos: The Game of Life in SQL!

Hetarth Shah — Sun, 02 Nov 2025 13:55:50 +0000

Github Repo: https://github.com/Hetarth02/game-of-life-sql

👾History

I have always been fascinated with cellular automaton, it’s so cool seeing how life can emerge from such simple rules. To give you a brief context, many people think Conway was the first person to coin this theory but it is not correct while Conway was the reason this branch got so famous but he was not the first. The idea of Cellular Automata first took birth around late 1950s inside the minds of Stanislaw Ulam and John von Neumann. While working together on a particular project both of these created the first automaton. It would be in the 1970s that mathematician John Conway would invent his now famous Game of Life.

🎮What’s the game?

Alright, now that we know a bit of history let’s talk briefly talk about the game.

It is a zero-player game meaning that its evolution is determined by its initial state, requiring no further input. One interacts with the Game of Life by creating an initial configuration and observing how it evolves.

Source: Game of Life WIkipedia

🗒️Rules

You start with an infinite 2D grid in which each cell will have two states either 0(dead) or 1(alive). Every cell interacts with its eight neighbours, which are the cells that are horizontally, vertically, or diagonally adjacent.

At each update, a new generation of cells are born, continue to live or die based on the following rules:

Any live cell with fewer than two live neighbours dies, as if by underpopulation.
Any live cell with two or three live neighbours lives on to the next generation.
Any live cell with more than three live neighbours dies, as if by overpopulation.
Any dead cell with exactly three live neighbours becomes a live cell, as if by reproduction.

And that’s it, that’s the entire game. You gave it an initial configuration and it will play itself.

😃So why it’s called “Game of Life”?

Even though it is this simple, the game when played mimicks “Life” itself in a way. Cells get born, die, depend on their neighbours.Moreover, it has another fantastic property which make this game unique, which is 🤖Turing Completeness. Yup, so basically you can create computers inside game of life and people have done so and if you want to have mindf*ck moment, you can simulate “Game of Life” inside “Game of Life”.🤯

There are many more generalised forms of Game of Life which come pretty close behaving like organic life itself and have been known as digital life or artifical life or algorithmic life.

I very highly recommend you give these videos a watch!

💻Why SQL and what’s the twist?

Just for the sake of it! It’s easy to find the implementation in any other languages and I just wanted to have a fun challenge. The twist is, usually when you implement this game you have a two storages, one which stores the current generation and another which stores the next generation well; I will not only use just a single table, but only a singular column too.(I could have also gone for just using a single DB cell but DBs usually have limitations on how much you can store in a single cell.)

What followed next was late night debugging sessions, bit ops and me pulling my hair out on calculations!!!

#️Creating the Grid

Alright, let’s get into the code.

The very first step is to create the world. Ok, so usually you would create a 2D array and call it a day(or night) not me though! We already have rows inside postgres. I see you asking, “We also have columns”. I mean yeah, but I ain’t gonna store each cell in single columns…let alone you need to name those columns and have to dynamically create them everytime.

Instead what we will do is, use bitstrings. Bitstrings are basically binary sequence of 0s and 1s. Ok, so 2 columns one for current gen and another for next gen right? Nope! As said earlier, we will use only one! Here’s how we will manage the game state,

Each game cell will be assigned 2 bits, one for next gen(left/odd) and one for current gen(right/even).
We will calculate next gen using the right bit which is current gen.
We will store the value of next gen in the left bit .
Once, all calculations are done for all current gen(right/even bits) cells, we will drop their values and assign them the value of the next gen(left/odd bits)
Reset all next gen bits.

So, for a 5 × 5 world, there will be 1 column and 5 rows each with 10 bits of data(because each cell stores 2 states so 5 cells × 2 bits = 10 bits).

We have our world ready. Next task is to generate an initial random state for the world, so we write a random number generator. I have made all the code as much read-friendly as possible.

drop function if exists generate_random_int;

create or replace function generate_random_int(min_num integer, max_num integer) returns integer
as $$
begin
    -- explanation: https://stackoverflow.com/questions/62981108/how-does-math-floormath-random-max-min-1-min-work-in-javascript
    return (select floor(random() * (max_num - min_num + 1)) + min_num);
end
$$ language plpgsql;

But we store bitstrings inside the columns, hence we gotta create a bitstring for each row with all left bits 0 because initially the next gen will not have any data.

create or replace function generate_random_bitstring(str_length integer) returns text
as $$
declare
    str text := '';
begin
    for i in 1 .. str_length loop
        -- Concate str with a '0' attached
        str := str || 0 || generate_random_int(0,1);
    end loop;
    return str;
end
$$ language plpgsql;

Alright, everything is going smooth as butter. Next up is creating the grid(table) and inserting the initial state to into it.

drop function if exists create_grid;

create or replace function create_grid(grid_rows integer, grid_cols integer) returns void
as $$
declare
    data_str text := '';
begin
    drop table if exists grid;

    execute 'create table if not exists grid (cell_values bit(' || 2 * grid_cols || '));';

    for row_counter in 1 .. grid_rows loop
        data_str := generate_random_bitstring(grid_cols);
        data_str := 'insert into grid (cell_values) values (' || E'b\'' || data_str || E'\'' || ');';
        execute data_str;
    end loop;
end
$$ language plpgsql;

You are probably wondering why do we need that weird E’b\’’ thing?🤔

So, E marks the starts of an string in which we can escape characters. The b part is us saying to postgres that this is a binary string and not just a string, otherwise postgres will store the binary value of the string and not what we supply it.

Now, we just make a small function which fetches the current table and displays all the records to use.

drop function if exists current_generation;

create or replace function current_generation() returns table (cells text)
as $$
begin
    return query (select cell_values::text from grid);
end
$$ language plpgsql;

To execute we do the the following,

do $$
begin
    -- Executes the function, discards the return value
    perform create_grid(5, 5);
end;
$$ language plpgsql;

select current_generation() as cells;

It will look something like this,

📐Debugging and Maths

All right now comes the hardest part, usually it’s not hard to calculate neighbours because you have nice grid structure in a 2D array and with arrays come certain perks like there’s not -1 index, etc. But for us, we have two bits representing single cell and it’s just a single column hence a fake(believed) 2D array.

So, at first I searched if anyone did this kind of thing within any language but couldn’t find it…forget SQL. Also, AI is not that helpful…(I personally don’t like using AI because I want to learn by myself whether its by making mistakes, print debugging or reading through reddit comments)

Ok, didn’t have much help in directly finding the answer back to board…so, I took a pen and paper and started jotting down a matrix numbered each cell and did all the maths for it. I didn’t wanted to call each row every time inside a loop so I just fetched all data at once and calculated total_rows and total_cols using it which is essentially just what dimensions our matrix(grid) was, don’t forget to divide by 2 else the maths gets crazy in iterating rows and columns.

So, at first I tried many different complex and smart approaches to calculate neighbours and especially dealing with the boundaries, edge cases and everything else…but at the end to preserve my sanity I just went the simple route.

Simplicity is the best complexity.

Ok, so my main problem was calculating boundaries of the world when iterating through the top/bottom row/column and those pesky corners.

Simply put, I started with 3 variables top_row, current_row and bottom_row. Always initialise those rows to be entirely dead(0’s) and then check if the indexes were out of bounds or not. If the indexes are out of bounds then do nothing, else we use our current row_index to find out top_row, current_row and bottom_row and put our index at the start and fetch the substring from the entire data string using total_cols.

We now have our rows, let’s move forward with columns. First, we get the current_value and then go with the same approach as we did during row, initialise all the neighbouring values as dead → check if column are within bounds → if they are fetch their values.

You need to double everything coz we have 2 bits per cell.

After having all 8 neighbours it’s just a matter of summing them up, and using the game of life rules!

We now have old and new generations in a single string new_all_data. Here comes the fun part, we now need to swap the new(left bits) and old generation(right bits) then reset the new generation(left bits) back to dead(0), so how do we do that? The answer is…

Bitmasking!!!

For those who don’t know what it is it’s super simple; bitmasking is nothing but hiding of unwanted bits using clever masks(we will see how we can create these).

Bitmasking uses AND operation to achieve masking because the logic for AND gate is if one of the inputs is 0 then the output will be 0.

In our case, we don’t care about the right(even) bits. So, we will use a mask which hides all the even bits i.e. 1010101010. We will create this mask for each row with the length of total_cols and AND it with the substrings of each row. After that we just we will have a bitstring for each row with all even bits 0 so, we will just do a bit-wise right shift by 1 to shift all odd bits to even positions and we don’t even need to reset the odd bits because it happens automatically!🎉

Now, all that’s left is to clear the current grid and insert the new rows into the table.

drop function if exists next_generation;

create or replace function next_generation() returns void
as $$
declare
    all_data text := '';
    new_all_data text := '';

    total_cols integer := 0;
    total_rows integer := 0;

    top_row text := 0;
    current_row text := 0;
    bottom_row text := 0;

    current_value integer := 0;

    top_left integer := 0;
    top_middle integer := 0;
    top_right integer := 0;

    middle_left integer := 0;
    middle_right integer := 0;

    bottom_left integer := 0;
    bottom_middle integer := 0;
    bottom_right integer := 0;

    live_neighbours integer := 0;

    bitmask text := '';
begin
    select string_agg(grid.cell_values::text, '') into all_data from grid;
    select bit_length(grid.cell_values) / 2 into total_cols from grid limit 1;
    total_rows := length(all_data)::integer / (2 * total_cols);

    for row_index in 1 .. total_rows loop
        top_row := repeat('0', total_cols * 2);
        if row_index - 1 != 0 then
            top_row := substr(all_data, ((row_index - 1) * total_cols * 2) - (total_cols * 2) + 1, total_cols * 2);
        end if;

        current_row := substr(all_data, (row_index * total_cols * 2) - (total_cols * 2) + 1, total_cols * 2);

        bottom_row := repeat('0', total_cols * 2);
        if row_index + 1 <= total_rows then
            bottom_row := substr(all_data, ((row_index + 1) * total_cols * 2) - (total_cols * 2) + 1, total_cols * 2);
        end if;

        for col_index in 1 .. total_cols loop
            -- We store the current value in right bit
            current_value := substr(current_row, (col_index * 2), 1)::integer;

            top_left := 0;
            middle_left := 0;
            bottom_left := 0;

            top_middle := 0;
            bottom_middle := 0;

            top_right := 0;
            middle_right := 0;
            bottom_right := 0;

            if col_index - 1 != 0 then
                top_left := substr(top_row, (col_index * 2) - 2, 1)::integer;
                middle_left := substr(current_row, (col_index * 2) - 2, 1)::integer;
                bottom_left := substr(bottom_row, (col_index * 2) - 2, 1)::integer;
            end if;

            top_middle := substr(top_row, (col_index * 2), 1)::integer;
            bottom_middle := substr(bottom_row, (col_index * 2), 1)::integer;

            if col_index + 1 <= total_cols then
                top_right := substr(top_row, (col_index * 2) + 2, 1)::integer;
                middle_right := substr(current_row, (col_index * 2) + 2, 1)::integer;
                bottom_right := substr(bottom_row, (col_index * 2) + 2, 1)::integer;
            end if;

            live_neighbours := 0;
            live_neighbours := top_left + top_middle + top_right + middle_left + middle_right + bottom_left + bottom_middle + bottom_right;

            -- We store the next value in left bit, hence only -1
            if current_value = 1 and (live_neighbours = 2 or live_neighbours = 3) then
                current_row := overlay(current_row placing '1' from (col_index * 2) - 1 for 1);
            elsif current_value = 0 and live_neighbours = 3 then
                current_row := overlay(current_row placing '1' from (col_index * 2) - 1 for 1);
            else
                current_row := overlay(current_row placing '0' from (col_index * 2) - 1 for 1);
            end if;
        end loop;

        new_all_data := new_all_data || current_row;
    end loop;

    execute 'truncate grid;';

    for i in 1 .. (total_cols * 2) loop
        if i % 2 = 1 then
            bitmask := bitmask || 1;
        else
            bitmask := bitmask || 0;
        end if;
    end loop;

    bitmask := E'b\'' || bitmask || E'\''; 

    for row_counter in 1 .. total_rows loop
        current_row := substr(new_all_data, (row_counter * total_cols * 2) - (total_cols * 2) + 1, total_cols * 2);
        current_row := E'b\'' || current_row || E'\'';
        current_row := '(' || current_row || ' & ' || bitmask || ') >> 1';
        current_row := 'insert into grid (cell_values) values (' || current_row || ');';
        execute current_row;
    end loop;
end
$$ language plpgsql;

You can run it using same method as before, just need to repeat these for infinity that’s all!😄

do $$
begin
    -- Executes the function, discards the return value
    perform next_generation();
end;
$$ language plpgsql;

select current_generation() as cells;

And that’s all!!!

🗒️Conclusion

Honestly, there might be more optimisations and improvements which can be done to this code but for now I am happy with what I have and I honestly had so much fun implementing this. Figuring out the mathematics, learning bitmasking everything was so fun and I hope you the reader will also try doing this or any other quirky project just for the sake of it!

Have a nice day!❤️

If you’d like to know more about my journey, please feel free to reach out or follow me!

Github: Hetarth02

LinkedIn: Hetarth Shah

Website: Portfolio

💖Thank you for joining me on this journey, and look forward to more interesting articles!

I made my own JSON Parser

Hetarth Shah — Sat, 15 Jun 2024 15:04:57 +0000

✏️Preface

I was always interested in how programming languages worked. To be specific, how what we write in a human readable way is interpreted my the machine. I did have an bird's eye view of what would happen, but it was and still kind of is like black magic to me. So, now I have decided to learn these magic verses and this is my first step towards it.

ℹ️First Steps

To not overwhelm myself, I wanted to start with something easy and which is used almost everyday. This way, I will be able to understand, solve it and actually have motivation to move forward rather than just learning amidst the hype is going on in my mind and then when it shimmers down I would eventually drop it.

So, I decided to write a JSON parser.

The goal was simple:

Input a string
Output should be JSON or Error (depending on the input)

I wanted to start with golang but eventually chose to go with javascript, the reason being simply that I do not wanted to divide my focus on learning many things at once. Hence, javascript it is.

⚠️I had no idea

Well, at first I tried doing it myself without having any idea of how to do. Pretty quickly, I was deep into for loops and regex territory. My intention was never to be able to solve this on my own but to actually learn where and what things I get stuck on. This way, when I actually refer something I can make a bridge from my thoughts and what the actual logic is.

After struggling for a while I started to notice what problems needed to be solved.

I needed some way to proper differentiate different kinds of words/symbols/values.
After differentiating them, I needed to validate each of them and put them back together.

👾Wow, this is interesting as hell!

Ok, now I had a basic understanding of what I needed to do and my mind was warmed up enough. Alright, let's start digging up the internet. After a while, I stumbled upon an article from Oz named Write Your Own JSON Parser with Node and Typescript.

This article does a fantastic job to decode the logic in a very step by step manner. I encourage you to give this one a read.

So basically, we need a tokenizer and parser . Tokenizer will identify the different types of token. Hey, this solves one of my problems. Ok, so rather than using different terms like words, symbols, characters we call them tokens. Seeing an example json there are many different kinds of tokens which we need to identify.

{
  "id": "647ceaf3657eade56f8224eb",
  "index": 0,
  "something": [],
  "boolean": true,
  "nullValue": null
}

There is { (Open Brace), } (Close Brace), [ (Open Bracket), ] (Close Bracket), strings like "id" , : colon, , commas and much more.

Alright, now if we were to tokenize this we should have information for basically two things,

Type of Token
Value of Token

We would store this array of tokens and then feed this to the parser which will then take it forward. So our end result should look something like this,

[
  { type: "BraceOpen", value: "{" },
  { type: "String", value: "id" },
  { type: "Colon", value: ":" },
  { type: "String", value: "647ceaf3657eade56f8224eb" },
  { type: "Comma", value: "," },
  { type: "String", value: "index" },
  { type: "Colon", value: ":" },
  { type: "Number", value: "0" },
  { type: "Comma", value: "," },
  { type: "String", value: "something" },
  { type: "Colon", value: ":" },
  { type: "BracketOpen", value: "[" },
  { type: "BracketClose", value: "]" },
  { type: "Comma", value: "," },
  { type: "String", value: "boolean" },
  { type: "Colon", value: ":" },
  { type: "True", value: "true" },
  { type: "Comma", value: "," },
  { type: "String", value: "nullValue" },
  { type: "Colon", value: ":" },
  { type: "Null", value: "null" },
  { type: "BraceClose", value: "}" },
];

🏷️Tokenizing

We will make a function named tokenizer which will take a string as an input and give us an array of token objects. We will iterate the string character by character and classify it based on some logic.

Oz's article does a very nice job of explaining the logic for token classification. However, it still skips out on somethings. The main fun which I had in making this was tackling the issues which the article didn't handled.

But let's start with the basics here, first we need a variable to track our position in the string and an array to store different token objects.

/**
 * JSON Tokenizer
 *
 * @param { String } string
 * @returns { any[] }
 */
function tokenizer(string) {
    let current = 0;
    const tokens = [];

    while (current < string.length) {
        let char = string[current];
        current++;
    }

    return tokens;
}

Now we will add some basic conditions to extract of tokens like { , }, etc.

/**
 * JSON Tokenizer
 *
 * @param { String } string
 * @returns { any[] }
 */
function tokenizer(string) {
    let current = 0;
    const tokens = [];

    while (current < string.length) {
        let char = string[current];

        if (char === "{") {
            tokens.push({ type: "BraceOpen", value: char });
            current++;
            continue;
        }

        if (char === "}") {
            tokens.push({ type: "BraceClose", value: char });
            current++;
            continue;
        }

        if (char === "[") {
            tokens.push({ type: "BracketOpen", value: char });
            current++;
            continue;
        }

        if (char === "]") {
            tokens.push({ type: "BracketClose", value: char });
            current++;
            continue;
        }

        if (char === ":") {
            tokens.push({ type: "Colon", value: char });
            current++;
            continue;
        }

        if (char === ",") {
            tokens.push({ type: "Comma", value: char });
            current++;
            continue;
        }

        // If it whitespace, ignore it.
        if (/\s/.test(char)) {
            current++;
            continue;
        }

        // For all other characters throw error
        throw new Error("Unexpected character: " + char);
    }

    return tokens;
}

All right, let's now tackle the trickiest things. Strings, Numbers, Boolean and Null values.

Let's start with Strings. So, in json each string starts and ends with " . This is the key part in our logic. When we encounter a " , we then start iteraing and storing all the values after it until we encounter another " . Then we store that as a value and increment our current position in the string.

if (char === '"') {
    let value = "";
    char = string[++current];
    while (char !== '"') {
        value += char;
        char = string[++current];
    }

    current++;
    tokens.push({ type: "String", value: value });
    continue;
}

For Boolean and Null values the logic is almost the same, just the condition for entering and breaking the loop is different.

// Test if it starts with a character
if (/[a-zA-z]/.test(char)) {
    let value = "";

    // Loop until it is not a character
    while (/[a-zA-Z]/.test(char)) {
        value += char;
        char = string[++current];
    }

    if (value === "true") {
        tokens.push({ type: "True", value });
    } else if (value === "false") {
        tokens.push({ type: "False", value });
    } else if (value === "null") {
        tokens.push({ type: "Null", value });
    } else {
        throw new Error("Unexpected value: " + value);
    }

    continue;
}

Now, for the last type Number . This is was the most fun part for me because the article which I was referring only dealt with Int numbers, if I tried a Float , or -ve number it didn't work. For recognizing something as a number there are some rules in json .

Basically, 69, -69 , 6.9 , -6.9 , 6e9 , -6e9 , 6.e9 , -6.e9 are all valid numbers. We will first see if the current character is - or a number. If it is then we will again repeat the same thing as we did for strings, boolean and null. We will start capturing the value, unless it is not a digit, not an e , - , . , + . But I hear you say, would this be wrong, this could parse 1.1.1 as number. You are right! So we will then add a check if it truly is a Number or not.

if (char === "-" || /\d/.test(char)) {
    let value = "";
    while (
        char === "+" ||
        char === "-" ||
        char === "e" ||
        char === "." ||
        /\d/.test(char)
    ) {
        value += char;
        char = string[++current];
    }

    if (isNumber(value)) {
        tokens.push({ type: "Number", value });
    } else {
        throw new Error("Unexpected value: " + value);
    }

    continue;
}

/**
 * Checks for Number
 *
 * @param { String } value
 * @returns { Boolean }
 */
function isNumber(value) {
    return !isNaN(Number(value));
}

The final version of tokenizer function looks like this.

/**
 * Checks for Number
 *
 * @param { String } value
 * @returns { Boolean }
 */
function isNumber(value) {
    return !isNaN(Number(value));
}

/**
 * JSON Tokenizer
 *
 * @param { String } string
 * @returns { any[] }
 */
function tokenizer(string) {
    let current = 0;
    const tokens = [];

    while (current < string.length) {
        let char = string[current];

        if (char === "{") {
            tokens.push({ type: "BraceOpen", value: char });
            current++;
            continue;
        }

        if (char === "}") {
            tokens.push({ type: "BraceClose", value: char });
            current++;
            continue;
        }

        if (char === "[") {
            tokens.push({ type: "BracketOpen", value: char });
            current++;
            continue;
        }

        if (char === "]") {
            tokens.push({ type: "BracketClose", value: char });
            current++;
            continue;
        }

        if (char === ":") {
            tokens.push({ type: "Colon", value: char });
            current++;
            continue;
        }

        if (char === ",") {
            tokens.push({ type: "Comma", value: char });
            current++;
            continue;
        }

        if (char === '"') {
            let value = "";
            char = string[++current];
            while (char !== '"') {
                value += char;
                char = string[++current];
            }

            current++;
            tokens.push({ type: "String", value: value });
            continue;
        }

        if (char === "-" || /\d/.test(char)) {
            let value = "";
            while (
                char === "+" ||
                char === "-" ||
                char === "e" ||
                char === "." ||
                /\d/.test(char)
            ) {
                value += char;
                char = string[++current];
            }

            if (isNumber(value)) {
                tokens.push({ type: "Number", value });
            } else {
                throw new Error("Unexpected value: " + value);
            }

            continue;
        }

        if (/[a-zA-z]/.test(char)) {
            let value = "";

            while (/[a-zA-Z]/.test(char)) {
                value += char;
                char = string[++current];
            }

            if (value === "true") {
                tokens.push({ type: "True", value });
            } else if (value === "false") {
                tokens.push({ type: "False", value });
            } else if (value === "null") {
                tokens.push({ type: "Null", value });
            } else {
                throw new Error("Unexpected value: " + value);
            }

            continue;
        }

        if (/\s/.test(char)) {
            current++;
            continue;
        }

        throw new Error("Unexpected character: " + char);
    }

    return tokens;
}

🖨️Parsing

The parser is where we make sense out of our tokens. Now we have to build our Abstract Syntax Tree (AST). The AST represents the structure and meaning of the code in a hierarchical tree-like structure. It captures the relationships between different elements of the code, such as statements, expressions, and declarations.

Every language or format you can think of uses some form of AST based on grammar rules of the programming language or data format being parsed.

Alright, so we will code for both things, to output the AST representation from the tokens and also an actual json object.

The logic again is kinda similar, we will iterate through the tokens and map it based on their type one by one.

/**
 * Parser + AST for tokens
 *
 * @param { any[] } tokens
 * @returns { any }
 */
function parser(tokens) {
    if (!tokens.length) {
        throw new Error("Nothing to parse. Exiting!");
    }

    let current = 0;

    function advance() {
        return tokens[++current];
    }

    function parseValue() {
        const token = tokens[current];

        // Gives ASTNode
        switch (token.type) {
            case "String":
                return { type: "String", value: token.value };
            case "Number":
                return { type: "Number", value: Number(token.value) };
            case "True":
                return { type: "Boolean", value: true };
            case "False":
                return { type: "Boolean", value: false };
            case "Null":
                return { type: "Null" };
            case "BraceOpen":
                return parseObject(); // To be implemented
            case "BracketOpen":
                return parseArray(); // To be implemented
            default:
                throw new Error(`Unexpected token type: ${token.type}`);
        }
    }

    return parseValue();
}

Alright, but whenever we see { or [ we need to handle the nested objects and arrays recursively which means calling parseValue() again from parseObject() and parseArray() .

function parseObject() {
    const node = { type: "Object", value: {} };

    let token = advance();

    // Iterate through the tokens until we reach a BraceClose '}'
    while (token.type !== "BraceClose") {
        if (token.type === "String") {
            const key = token.value;
            token = advance();

            if (token.type !== "Colon") {
                throw new Error("Expected : in key-value pair");
            }
            token = advance(); // Skip ':'

            const value = parseValue(); // Recursively parse the value
            node.value[key] = value;
        } else {
            throw new Error(`Unexpected key. Token type: ${token.type}`);
        }

        token = advance(); // Increment one step
        if (token.type === "Comma") token = advance(); // Skip ','
    }

    // Gives ASTNode
    return node;
}

function parseArray() {
    const node = { type: "Array", value: [] };
    let token = advance(); // Skip '['

    while (token.type !== "BracketClose") {
        const value = parseValue();
        node.value.push(value);

        token = advance();  // Increment one step
        if (token.type === "Comma") token = advance();  // Skip ','
    }

    // Gives ASTNode
    return node;
}

Now our parser is ready, here is what the final implementation looks like.

/**
 * Checks for Number
 *
 * @param { String } value
 * @returns { Boolean }
 */
function isNumber(value) {
    return !isNaN(Number(value));
}

/**
 * JSON Tokenizer
 *
 * @param { String } string
 * @returns { any[] }
 */
function tokenizer(string) {
    let current = 0;
    const tokens = [];

    while (current < string.length) {
        let char = string[current];

        if (char === "{") {
            tokens.push({ type: "BraceOpen", value: char });
            current++;
            continue;
        }

        if (char === "}") {
            tokens.push({ type: "BraceClose", value: char });
            current++;
            continue;
        }

        if (char === "[") {
            tokens.push({ type: "BracketOpen", value: char });
            current++;
            continue;
        }

        if (char === "]") {
            tokens.push({ type: "BracketClose", value: char });
            current++;
            continue;
        }

        if (char === ":") {
            tokens.push({ type: "Colon", value: char });
            current++;
            continue;
        }

        if (char === ",") {
            tokens.push({ type: "Comma", value: char });
            current++;
            continue;
        }

        if (char === '"') {
            let value = "";
            char = string[++current];
            while (char !== '"') {
                value += char;
                char = string[++current];
            }

            current++;
            tokens.push({ type: "String", value: value });
            continue;
        }

        if (char === "-" || /\d/.test(char)) {
            let value = "";
            while (
                char === "+" ||
                char === "-" ||
                char === "e" ||
                char === "." ||
                /\d/.test(char)
            ) {
                value += char;
                char = string[++current];
            }

            if (isNumber(value)) {
                tokens.push({ type: "Number", value });
            } else {
                throw new Error("Unexpected value: " + value);
            }

            continue;
        }

        if (/[a-zA-z]/.test(char)) {
            let value = "";

            while (/[a-zA-Z]/.test(char)) {
                value += char;
                char = string[++current];
            }

            if (value === "true") {
                tokens.push({ type: "True", value });
            } else if (value === "false") {
                tokens.push({ type: "False", value });
            } else if (value === "null") {
                tokens.push({ type: "Null", value });
            } else {
                throw new Error("Unexpected value: " + value);
            }

            continue;
        }

        if (/\s/.test(char)) {
            current++;
            continue;
        }

        throw new Error("Unexpected character: " + char);
    }

    return tokens;
}

/**
 * Parser + AST for tokens
 *
 * @param { any[] } tokens
 * @returns { any }
 */
function parser(tokens) {
    if (!tokens.length) {
        throw new Error("Nothing to parse. Exiting!");
    }

    let current = 0;

    function advance() {
        return tokens[++current];
    }

    function parseValue() {
        const token = tokens[current];

        // Gives ASTNode
        // switch (token.type) {
        //     case "String":
        //         return { type: "String", value: token.value };
        //     case "Number":
        //         return { type: "Number", value: Number(token.value) };
        //     case "True":
        //         return { type: "Boolean", value: true };
        //     case "False":
        //         return { type: "Boolean", value: false };
        //     case "Null":
        //         return { type: "Null" };
        //     case "BraceOpen":
        //         return parseObject();
        //     case "BracketOpen":
        //         return parseArray();
        //     default:
        //         throw new Error(`Unexpected token type: ${token.type}`);
        // }

        // Gives value
        switch (token.type) {
            case "String":
                return token.value;
            case "Number":
                return Number(token.value);
            case "True":
                return true;
            case "False":
                return false;
            case "Null":
                return null;
            case "BraceOpen":
                return parseObject();
            case "BracketOpen":
                return parseArray();
            default:
                throw new Error(`Unexpected token type: ${token.type}`);
        }
    }

    function parseObject() {
        const node = { type: "Object", value: {} };

        let token = advance();

        while (token.type !== "BraceClose") {
            if (token.type === "String") {
                const key = token.value;
                token = advance();

                if (token.type !== "Colon") {
                    throw new Error("Expected : in key-value pair");
                }
                token = advance();

                const value = parseValue();
                node.value[key] = value;
            } else {
                throw new Error(`Expected String key in object. Token type: ${token.type}`);
            }

            token = advance();
            if (token.type === "Comma") token = advance();
        }

        // Gives ASTNode
        // return node;

        // Gives Value
        return node.value;
    }

    function parseArray() {
        const node = { type: "Array", value: [] };
        let token = advance();

        while (token.type !== "BracketClose") {
            const value = parseValue();
            node.value.push(value);

            token = advance();
            if (token.type === "Comma") token = advance();
        }

        // Gives ASTNode
        // return node;

        // Gives Value
        return node.value;
    }

    return parseValue();
}

let output = parser(
    tokenizer(
        JSON.stringify({
            products: [
                {
                    id: 1,
                    title: "Essence Mascara Lash Princess",
                    category: "beauty",
                    price: 9.99,
                    discountPercentage: 7.17,
                    rating: 4.94,
                    stock: 5,
                    tags: ["beauty", "mascara"],
                    brand: "Essence",
                    sku: "RCH45Q1A",
                    weight: 2,
                    dimensions: {
                        width: 23.17,
                        height: 14.43,
                        depth: 28.01,
                    },
                    warrantyInformation: "1 month warranty",
                    shippingInformation: "Ships in 1 month",
                    availabilityStatus: "Low Stock",
                    reviews: [
                        {
                            rating: 5,
                            comment: "Very satisfied!",
                            date: "2024-05-23T08:56:21.618Z",
                            reviewerName: "Scarlett Wright",
                            reviewerEmail: "scarlett.wright@x.dummyjson.com",
                        },
                    ],
                    returnPolicy: "30 days return policy",
                    minimumOrderQuantity: 24,
                    meta: {
                        createdAt: "2024-05-23T08:56:21.618Z",
                        updatedAt: "2024-05-23T08:56:21.618Z",
                        barcode: "9164035109868",
                        qrCode: "https://dummyjson.com/public/qr-code.png",
                    },
                    images: [
                        "https://cdn.dummyjson.com/products/images/beauty/Essence%20Mascara%20Lash%20Princess/1.png",
                    ],
                    thumbnail:
                        "https://cdn.dummyjson.com/products/images/beauty/Essence%20Mascara%20Lash%20Princess/thumbnail.png",
                },
            ],
            total: 194,
            skip: 0,
            limit: 30,
        })
    )
);

console.log(JSON.stringify(output, "", 2));

🔭Conclusion

This wasn't even that time consuming and I really learned some useful concepts in during this. Now, since I have a understanding how this works; I can take up any new language which I want to learn and implement this as a proof of concept in it.

If you’d like to know more about my journey, please feel free to reach out or follow me!

Github: Hetarth02

LinkedIn: Hetarth Shah

Website: Portfolio

💖Thank you for joining me on this journey, and look forward to more interesting articles!

Credits:

Cover Image from, Image Source.

Article Images from, json.org.

References:

Write Your Own JSON Parser with Node and Typescript

How 8 bytes broke production!

Hetarth Shah — Wed, 27 Mar 2024 05:04:40 +0000

✏️Preface

This blog is one of the many in a series which I hope to write in the future about a project which I was assigned in my professional career. This project is in the edtech domain and is used by many of the big players, let's call this "Revolutionary Education Knowledge Technology" a.k.a "REKT" .

There are many stories which I would like to share about project REKT, but let me start with this one. This is a story of how a mere of 8 bytes broke the entire production.

📖A Little bit of Backstory

It was the day before the production deployment was scheduled and if I remember correctly it was also a day before a major content update for a game which I was playing. I can't go into much/actual details about the project but a quick overview of the project was basically imagine udemy but with proprietary domain specific content for companies. There were many things which I had to implement in REKT and one of them was making an admin/content management panel to manage educative modules. One of the features was to implement a CRUD for content system which can be attempted by the user in modules. Just like in udemy, a module contains many lectures and content for the user to attempt it, it was the same here.

Well, I was not the first developers in this project. Before me there was a group of developers who had developed the entire "architecture" and chosen the tech stack for REKT.

Now, I will tell my programming journey in another post but a bird's eye view is I believe in grasping the fundamentals rather than going and learning the language because according to me how to solve a problem is more important rather than knowing the syntax for solving it. A simple example, if I need to do something repetitively I would need like a loop. Here loop is a solution to my problem, as for the ways to implement it are different in different programming languages.

// For loop in javascript
for (let i = 0; i < 10; i++) {
  console.log(i);
}

# For loop in python
for x in range(0, 10):
  print(x)

// For loop in PHP
for ($x = 0; $x <= 10; $x++) {
  echo $x;
}

Hence, I focus on developing the problem solving mindset rather than focusing on the language specific ways and/or syntax to solve a problem.

Well, coming back to the point, I at the time had somewhat idea of frontend, backend and server from a MVC and monolith perspective. I wasn't new to the component driven frontend aspects, fortunately I learnt it from SvelteJs❤️ and unfortunately I hadn't had the chance to delve into frameworks like ReactJs and VueJs. And guess what the tech stack for the project REKT was...frontend in ReactJs backend in Fastify and a tightly coupled aws ecosystem dependency and to put a cherry on top it was all serverless and all of this managed by terraform. At the time, all I knew about serverless was that it basically spins up a server like environment runtime when the request comes.

Also, there was very less or close to no documentation with what the entire architecture looked like and what services were used.

🎮Deployment and Gaming

Past aside, fast forward to the day of the deployment of the feature. I and the QA had extensively tested the feature with almost all the scenarios which we could have come up with. Everything was greenlit, and with the approval of the manager we pushed it to the production in the start of the day. After the deployment the QA started testing in live environment and call it my overconfidence, naivety or lack of experience, I without any worries installed the update for my game and started playing it. Our manager saw this, but since everything was going fine they didn't bat much eye to it. The first half of the day went very smoothly, everything was working perfectly in the production we had verified and tested it several times and we told that it was ready for client usage.

⚡Getting REKTed

Then comes the second half of the workday. I just came after lunch and was planning to take up other tickets, suddenly a message popped up in the teams chat.

"The questions aren't being added when I save the content." - Client

Thoughts started flooding in my mind,

How can this be possible? We tested everything multiple times on dev, staging and production too? Surely, this must be a misunderstanding.

"Can you tell them to clear the browser cache or hard refresh and try again?" - Me

"We are already running this incognito mode." - Client

Well fu*k, I thought! Instantly, I opened up the dev deployment and inspected it. I created a lesson, added a content(quiz) with a few questions and options, clicked on the submit button and Voila!🪄 it saved perfectly.

I immediately tried it again, but it worked perfectly everytime. I screen recorded the video and send it in the teams chat.

"It works when I do it." - Me

And boom, client also sends a screen recording with the issue. At this point in time, me and the QA have going through each and every scenario on all three different environments and everything worked perfectly for us.

The clock kept ticking away, time flew but we weren't getting a breakthrough at all. In search for the bug of why the content wasn't getting saved for the client me and the QA found another major bugs in production. Everything was fine until the first half of the day, I was enjoying my game and everything seemed going smoothly and now here we were 11:00 pm in the night with more than one production breaking bugs.

We were too tired to think of anything at that point, I tried adding some patches but they didn't work we were starting to get desperate.

"We aren't getting anywhere. Let's call it a day, come back tomorrow and look at it with a fresh mind" - QA

He was right, we had spent all our day in the same thing over and over and over again we were getting tunnel visioned.

💪Chad Senior Dev

Before we called it a day, something interesting happened which eventually paved the way for me to solve this clusterfu*k of problems. A senior dev, had some time left after completing his work and offered to help us debugging this. We tried our best to sum up the situation, but even we didn't have any idea why the content wasn't getting saved. (We kept other bugs between myself and QA!😉)

The senior dev already had extended his work time and now was helping us, we felt bad and somewhat relieved at the same time. Well, the senior dev did nothing extraordinary or out of the world. He just opened up the dev tools, asked me to enter the data and save it. It saved perfectly. Then he asked me to reproduce what the client was facing and I gladly did.

"I think here lies your problem..." - Chad Senior Dev

He showed us the request payload size for both cases,

Passing Case: ~8,192 bytes
Failing Case: ~8,200 bytes

"Heh, that can't be the case." - Me

"Surely that can't be the case, it is just 8kb. It has to be something else." - Me

With this thought in mind, we wrapped up the day.

🤬Next time write some documentation!

For the most part, developers frown upon when asked to write documentation but at the very least take 5 minutes and think about the future person who would maintaining your code, I mean do you really want them to write a ~2k word article saying how frustrated they are when someone doesn't write something basic down.

Alright, back to the incident. The next day, we came early around 7:00 AM to the office with a fresh mind. One thing was stuck with me, that the 8kb payload size is bullshit(No offense to the Chad senior). So, I tried extending the POST body size in fastify, redeployed the backend but the same issue persisted. Maybe it was due to CORS or some shit, allow everything for the time being and redeploy. Nothing worked.

I thought let me leave this for a bit and work on other bugs before they are found out by the client. So, I put off this and solved other bugs. They were not easy but they weren't a black box, I could easily trace through the code and was able to find a tiny flaw in the logic. I fixed each of the other bugs. Even though the main thing was left, I finally had some hope. Sometimes little victories help your state of mind and change its thinking direction. The same happened for me.

Up until now, I have been looking through the logs for an error in both the frontend and the backend but there was no sign of error logs anywhere. It was truly a black box for me. We could reproduce the bug but it wasn't leaving any trails to follow.

💡Thinking with a different perspective

Wait, why aren't I getting any logs for this particular request? It is getting fired from the frontend, and it sometimes reaches the backend too. Let's assume that the 8kb thing is right, so it means that something is blocking the request from reaching the backend. What could it be, though? We don't have any server...

"We should revert back the allow all CORS it is not secure." - QA

Aha!! Secure, security, do we have something like security group for routes or anything which has some rules. I logged onto the AWS dashboard went to the security groups but couldn't find anything. Even google searching didn't helped much. Then it struck to me, that to that since the request isn't getting to the backend then there must be something protecting it and since it was serverless, I thought that security groups can't be the one. What other thing could be there, I searched on google "aws website security products" and then looking for something which blocks requests in the transport/network layer and here it was the culprit of this clusterfu*k...

AWS Web Application Firewall(WAF)

Then everything started falling into place, I then searched "aws waf body size limit" and I find out this.

"We also use AWS WAF?" - Me, QA, Manager, maybe client

This not only came a surprise for me but the entire team. Well then, fixing it was as easy as it could get. Just logged into the AWS console, changed the WAF rules to allow over-sized payload for request and then tested again.

Everything Works!🥂

Redeployed after cleaning some code and tested again...it failed again.

The hell?! I thought.

Ah yes, since the entire infrastructure was maintained by terraform it reset the settings for the WAF rules. Well, just change it again manually for the time being and it should work! Automating it is a problem for future me.

🔭Conclusion

After all this settled down, I took around an hour or so. Documented the entire incident with guides and my thought processes and links to the aws docs and manual solution with an in-depth step by step tutorial to help anyone in the future encountering similar kind of things in project REKT!

If you’d like to know more about project REKT or my journey, please feel free to reach out or follow me!

Github: Hetarth02

LinkedIn: Hetarth Shah

Website: Portfolio

💖Thank you for joining me on this journey, and look forward to more interesting articles!

Credits:

Cover Image from, Image Source.

Where It All Starts and Ends

Hetarth Shah — Fri, 10 Nov 2023 12:36:38 +0000

As the sun sets on my GSoC journey with Julia, I find myself reminiscing about the incredible experiences of the past few months. The final two weeks have been a whirlwind of collaboration, coding, and personal growth. In this blog post, I'll take you through my journey of iterating on the valuable feedback from the Julia community and the preparations leading up to my final submission.

🔄️Iterating on Awesome Feedback:

Imagine receiving a treasure map - that's what the feedback from the community felt like to me. Their insights were the guiding light that illuminated the path toward a better search UI.

Amidst the flurry of coding excitement, I had unintentionally omitted a close button for shutting down the search modal on mobile devices. How did I miss something as basic as that? It was like setting up a grand dinner party and forgetting to invite the guest of honour!

Thankfully, my oversight was caught by the vigilant eyes of the community, who promptly pointed out this rather conspicuous absence. It's moments like these that remind us that coding is as much about attention to detail as it is about grand designs. This also reinforced the age-old wisdom that a fresh perspective can work wonders. Amidst chaos and lines of code, it's remarkably easy to become 'code blind', missing the forest for the trees. And that's where the community steps in as a friendly lifeguard, ready to pull you out of the code sea when you're adrift. It's not just about code review; it's about having a fresh pair of eyes to spot the inconspicuous, question the obvious, and keep the ship sailing in the right direction.

Well, I added that much-needed close button, grateful for the watchful community that helped me steer clear of my oversight in this journey. And with some code cleaning, I made my final push to the PR. And here is what the finished product looks like in comparison to before...

Search UI Before

Search UI After

Farewell, But Not Goodbye!

As the final submission date looms, a bittersweet feeling settles in. It's like bidding farewell to a beloved TV series - you're excited to see how it all concludes, yet sad that the journey is ending. These weeks have been a blend of challenges, triumphs, and late-night coding sessions fueled by equal parts determination and caffeine.

While I'll soon hit the "submit" button with a mix of relief and anticipation, what truly warms my heart is the camaraderie I've found within the Julia community. The people who were once mere usernames on a forum are now friends and mentors who have shaped me into a better coder.

So here's to the last two weeks of GSoC - a chapter filled with lines of code, laughter, and growth. As I wrap up my project, I can't help but look back and smile at how far I've come. As this chapter closes, I'm reminded that every farewell is simply a prelude to a new adventure.

If you're interested in following my progress or connecting with me, please feel free to reach out!

Github: Hetarth02
LinkedIn: Hetarth Shah
Website: Portfolio

Thank you for joining me on this journey, and look forward to more updates as I continue contributing to the Open Source community.

Relax, Code, Repeat

Hetarth Shah — Fri, 10 Nov 2023 11:56:03 +0000

Hey there! Welcome back to my ongoing adventure in the GSoC program with Julia. If you've been following my journey, you'll remember that I was cruising ahead of schedule and had just wrapped up the mid-term evaluations. So, let's jump right in and explore what weeks 7 to 10 brought to my journey!

Breathing Room after Mid-Term Madness:

Ah, the sweet taste of being ahead of the game! With the mid-term evaluations in the rearview mirror, the next few weeks were a welcome breather. No frantic coding sessions, no last-minute debugging marathons - just the gentle hum of productivity. It was the ideal time to catch my breath and recharge for the exciting tasks ahead.

Modal UI: Unveiling the Magic

The heart of these weeks was dedicated to bringing the search to life. Imagine interacting with your code in a more intuitive and user-friendly way - that's exactly what the new UI aims to achieve. I dived headfirst into this task, determined to craft an experience that would make searching inside Julia an absolute delight.

Seeking Wisdom: Feedback Loop

One of the coolest aspects of the GSoC journey is the collaborative nature of the project. With the new UI taking shape, I reached out to the Julia community for feedback. It was exhilarating to see the community engage with my work, sharing their insights, suggestions, and experiences. This feedback loop was not just about improving the UI, but also about building connections and learning from experienced developers.

Refining the Rough Edges

As the weeks progressed, I poured over the feedback received and focused on refining the Modal UI. It's amazing how a fresh pair of eyes can spot things you might have missed. The iterative process of coding, testing, and incorporating feedback was immensely satisfying. The UI began to take on a polished form, and with each passing day, it grew more user-friendly and powerful.

Conclusion: Taking It All in Stride

And there you have it, the laid-back adventure of Weeks 7 to 10 in my GSoC journey with Julia. Who would have thought that a relaxed phase could be so rewarding? Building the Modal UI, gathering feedback, and refining the user experience has been a journey of growth, creativity, and camaraderie. The journey continues, and I can't wait to see where the final leg of GSoC takes me.

The next article would be second to the last article containing what the work on feedback was and what the final finished product looks like!

Until next time, fellow developers and readers, keep coding, keep exploring, and remember - taking it easy once in a while might just lead to some of your most ingenious creations!

If you're interested in following my progress or connecting with me, please feel free to reach out!

Github: Hetarth02
LinkedIn: Hetarth Shah
Website: Portfolio

Thank you for joining me on this journey, and look forward to more updates as I continue to contribute to the Julia community during GSoC.

Credits:
Cover Image from,

Today marks a very exciting day for the channel, as we release a brand new 24/7 livestream on YouTube! 🎉

Introducing a brand new character in the Lofi universe, we bring to you the genre of Synthwave - beats to chill/game to! 🌌 pic.twitter.com/wlG9Awc4S2
— Lofi Girl (@lofigirl) April 11, 2023

Transitioning from Lunr.js to Minisearch.js

Hetarth Shah — Fri, 10 Nov 2023 11:49:32 +0000

As I embarked on weeks five and six of my Google Summer of Code (GSoC) project, a pivotal shift took place in client-side searching. To optimize the initial load times, I transitioned from using Lunr.js to Minisearch.js. This change brought about a host of improvements and refinements that made a more efficient search functionality. In this blog post, I'll delve into the intricacies of this transition and explore key concepts of this transition.

Current Implementation:

lunr.tokenizer.separator = /[\s\-\.]+/

lunr.trimmer = function (token) {
  return token.update(function (s) {
    return s.replace(/^[^a-zA-Z0-9@!]+/, '').replace(/[^a-zA-Z0-9@!]+$/, '')
  })
}

lunr.Pipeline.registerFunction(lunr.stopWordFilter, 'juliaStopWordFilter')
lunr.Pipeline.registerFunction(lunr.trimmer, 'juliaTrimmer')

var index = lunr(function () {
  this.ref('location')
  this.field('title', { boost: 100 })
  this.field('text')
  documenterSearchIndex['docs'].forEach(function (e) {
    this.add(e)
  }, this)
})

tokens.forEach(function (t) {
  q.term(t.toString(), {
    fields: ['title'],
    boost: 100,
    usePipeline: true,
    editDistance: 0,
    wildcard: lunr.Query.wildcard.NONE,
  })
  q.term(t.toString(), {
    fields: ['title'],
    boost: 10,
    usePipeline: true,
    editDistance: 2,
    wildcard: lunr.Query.wildcard.NONE,
  })
  q.term(t.toString(), {
    fields: ['text'],
    boost: 1,
    usePipeline: true,
    editDistance: 0,
    wildcard: lunr.Query.wildcard.NONE,
  })
})

As you can see, we currently have many moving components in the search logic. There is a tokenizer, a trimmer function, we are boosting some fields and an editDistance parameter which is used during the search. We will learn what each of these is step by step.

Tokenization: The Foundation of Search

lunr.tokenizer.separator = /[\s\-\.]+/;

The process of tokenization involves breaking down the search input into individual tokens or words. This step is crucial as it lays the groundwork for the search engine to efficiently match the tokens against the indexed documents. Each token is subsequently treated as a unique identifier that allows for accurate and targeted search results. Here, we have regex which tokenizes the word using white-space or . because there are results such as Documenter.Anchors.add! which would not trigger anything if we do not tokenize it properly.

Trimmer Function: Polishing the Search Input

One of the challenges you encounter is dealing with special characters and unwanted characters that might hinder the search process. To address this, a trimmer function that sanitizes the search input is used. This function identifies and removes extraneous characters, ensuring that the search query remains clean and focused. This significantly contributes to the accuracy of search results. However, there might be certain characters that are useful to us and hence we again make use of regex to not remove those characters.

// custom trimmer that doesn't strip @ and !
lunr.trimmer = function (token) {
    return token.update(function (s) {
      return s.replace(/^[^a-zA-Z0-9@!]+/, "").replace(/[^a-zA-Z0-9@!]+$/, "");
    });
  };

Stopwords: Filtering out the Noise

Stopwords are common words such as and, the, is, etc., that hold little semantic value and can be safely excluded from search queries. By identifying and filtering out stopwords, the search engine can allocate more attention to meaningful keywords and ultimately improve the relevance of the search results. This is particularly important when aiming for precision in search queries.

  // List of stop words used
  [ "a","able","about","across","after","almost","also","am","among","an","and","are","as","at","be","because","been","but","by","can","cannot","could","dear","did","does","either","ever","every","from","got","had","has","have","he","her","hers","him","his","how","however","i","if","into","it","its","just","least","like","likely","may","me","might","most","must","my","neither","no","nor","not","of","off","often","on","or","other","our","own","rather","said","say","says","she","should","since","so","some","than","that","the","their","them","then","there","these","they","this","tis","to","too","twas","us","wants","was","we","were","what","when","who","whom","why","will","would","yet","you","your" ]

Field Boosting: Spotlighting Priority Fields

In scenarios where certain fields within a document hold higher importance, field boosting comes into play. This technique assigns higher relevance scores to specific fields, effectively prioritizing them in the search results. For instance, if you're searching for products, boosting the "product name" field might yield more accurate outcomes than boosting other less relevant fields.

var index = lunr(function () {
    this.ref("location");
    // Boost score when title matches
    this.field("title", { boost: 100 });
    this.field("text");
    documenterSearchIndex["docs"].forEach(function (e) {
      this.add(e);
    }, this);
});

Fuzzy Searching: Embracing Variability

Fuzzy searching is an indispensable feature that accounts for slight variations and typos in search queries. Edit distance, a metric used to measure the similarity between two strings, is a fundamental component of fuzzy searching. When users inadvertently mistype a word or opt for a slightly different variant, fuzzy searching leverages edit distance to identify potential matches. This ensures that users aren't penalized for minor errors such as typos or when a cat walks over their keyboard and are still presented with relevant results.

Transitioning to MinisearchJs

Well, the above might seem daunting but minisearch is very flexible and is extremely developer friendly. Below is the minisearch implementation.

let index = new minisearch({
  fields: ["title", "text"], // fields to index for full-text search
  storeFields: ["location", "title", "text", "category", "page"], // fields to return with search results
  processTerm: (term) => {
    let word = stopWords.has(term) ? null : term;
    if (word) {
      // custom trimmer that doesn't strip @ and !, which are used in julia macro and function names
      word = word
        .replace(/^[^a-zA-Z0-9@!]+/, "")
        .replace(/[^a-zA-Z0-9@!]+$/, "");
    }

    return word ?? null;
  },
  // add . as a separator, because otherwise "title": "Documenter.Anchors.add!", would not find anything if searching for "add!", only for the entire qualification
  tokenize: (string) => string.split(/[\s\-\.]+/),
  // options which will be applied during the search
  searchOptions: {
    boost: { title: 100 },
    fuzzy: 2,
    processTerm: (term) => {
      let word = stopWords.has(term) ? null : term;
      if (word) {
        word = word
          .replace(/^[^a-zA-Z0-9@!]+/, "")
          .replace(/[^a-zA-Z0-9@!]+$/, "");
      }

      return word ?? null;
    },
    tokenize: (string) => string.split(/[\s\-\.]+/),
  },
});

Was the switch worth it?

In one word, Yes! It improved cold start times by a certain magnitude but it mainly reduced the size of the index by 16%.

Implementation of Design Mockups and Static Build:

Leveraging insights from prior research on search UIs, I crafted interactive and user-friendly interfaces. These designs were aimed at enhancing the visual and functional aspects of the search modal. This involved ensuring seamless navigation, clean typography, and optimal placement of UI elements. What followed was the creation of a static build housing an impressive array of ten or more UI variations for the search modal. This diverse set of UIs allows for a comprehensive exploration of different visual and interactive possibilities.

If someone wants to test out the different options then head here, Mock Search UIs

⚡An Unexpected Development:

With the JuliaCon 2023 coming up, my mentor asked if I could improve the current search listing with something that can be presentable enough. Well, we quickly set up a meeting to discuss how we could improve the current listing in a way that is fast enough and doesn't require much time. In the end, we decided to go with keeping the listing as it is because integrating a full-blown modal UI would require time and iterations based on feedback. Hence, we agreed to update the results using the new UI as a reference. And since I already had all the required data at my hands coming from minisearch it was just a matter of updating the DOM structure and some CSS magic🪄. I quickly put together the elements and this was the result.

📜Mid-Term Evals:

With the search engine changed and the mockups built, I was pretty much ready for the mid-term evaluation. This is an optional evaluation for both the mentors and contributors where each leaves feedback for the other and answers some basic questions. But, if a mentor fails you in the evals you won't be able to continue. This fact had me feeling nervous, but since I was ahead of the tasks I passed with flying colours.

📎Sharing My GSoC Proposal:

For those who are curious to delve deeper into the foundations of my GSoC project and explore the roadmap that led to all this, I invite you to take a look at my GSoC proposal. It outlines the initial concepts, goals, and strategies that have guided these weeks of progress. Feel free to explore the proposal here📝. Your interest in the journey is greatly appreciated, and I'm excited to share this transformative process with you.

If you're interested in following my progress or connecting with me, please feel free to reach out!

Github: Hetarth02
LinkedIn: Hetarth Shah
Website: Portfolio

Thank you for joining me on this journey, and look forward to more updates as I continue to contribute to the Julia community during GSoC.

Credits:
Cover Image from Minisearch Repo.

Weeks 3 and 4 of GSoC Journey: Steps towards Crafting An Enhanced Search Experience

Hetarth Shah — Sun, 20 Aug 2023 17:55:49 +0000

These weeks have been a fusion of research, analysis, and design brainstorming, propelling the pursuit of a more user-friendly experience. So, gear up as we embark on this virtual expedition, exploring the fascinating progress I've made in revamping search functionality.

Exploring the Research Path:

Research is like a compass guiding innovation, and I eagerly jumped into studying different ways people make search work, some major inspirations were from the big players like Tailwind CSS, Bootstrap, Laravel and many more. Most of these had search implemented by Algolia, and the most obvious part which stood out was that the search was easily accessible everywhere thanks to their modal UI.

Tailwind's Search UI

While it was clear as day that there was a need for a better search in Documenter, the current is a bit too simplistic. One other major flaw was that the search was only accessible through a single Search page which the user had to visit every time they had to search something.

Currently generated UI

Generating Issues:

Well, after the weekly meeting with my mentor, we decided to open up two issues as the first step in the upcoming process. One for the UI improvements and another for changing the search engine.

The GitHub issue for UI: Updating the search UI

The GitHub issue for Changing Search Engine: Changing the search engine

Code Action:

Now, for the truly fascinating part – coding! As much as I enjoyed immersing myself in research, there's an unmatched thrill in turning ideas into lines of code.

Speaking of which, let me share a glimpse of what's been cooking in the coding kitchen. Inspired by the quest for a speedier search experience, I began crafting a preliminary script for pre-building the search index. Imagine this as a way to give our search engine a head start – much like a car's engine warming up before hitting the road. This script is destined to play a pivotal role in our CI/CD (Continuous Integration and Continuous Deployment) process. By automating the index creation in this way, we're not only boosting search speed but also ensuring a more consistent and reliable experience for all users.

// data.json
[    
      {
        "location": "stdlib/CRC32c.html#CRC32c",
        "page": "CRC32c",
        "title": "CRC32c",
        "text": "",
        "category": "section"
      },
      {
        "location": "stdlib/CRC32c.html",
        "page": "CRC32c",
        "title": "CRC32c",
        "text": "Standard library module for computing the CRC-32c checksum.",
        "category": "page"
      }
]

// Rudimentary script for index generation
let fs = require('fs')
let gz = require('zlib')
let data = require('./data.json')
const Minisearch = require('./minisearch.js')

data = data.map((x, key) => {
  x['id'] = key
  return x
})

const stopWords = new Set([
  'a',
  'able',
  'about',
  'across',
  'after',
  'almost',
  'also',
  'am',
  'among',
  'an',
  'and',
  'are',
  'as',
  'at',
  'be',
  'because',
  'been',
  'but',
  'by',
  'can',
  'cannot',
  'could',
  'dear',
  'did',
  'does',
  'either',
  'ever',
  'every',
  'from',
  'got',
  'had',
  'has',
  'have',
  'he',
  'her',
  'hers',
  'him',
  'his',
  'how',
  'however',
  'i',
  'if',
  'into',
  'it',
  'its',
  'just',
  'least',
  'like',
  'likely',
  'may',
  'me',
  'might',
  'most',
  'must',
  'my',
  'neither',
  'no',
  'nor',
  'not',
  'of',
  'off',
  'often',
  'on',
  'or',
  'other',
  'our',
  'own',
  'rather',
  'said',
  'say',
  'says',
  'she',
  'should',
  'since',
  'so',
  'some',
  'than',
  'that',
  'the',
  'their',
  'them',
  'then',
  'there',
  'these',
  'they',
  'this',
  'tis',
  'to',
  'too',
  'twas',
  'us',
  'wants',
  'was',
  'we',
  'were',
  'what',
  'when',
  'who',
  'whom',
  'why',
  'will',
  'would',
  'yet',
  'you',
  'your',
])

let search_index = new Minisearch({
  fields: ['title', 'text'], // fields to index for full-text search
  storeFields: [
      'location',
      'title',
      'text',
      'category'
  ], // fields to return in results
  tokenize: (string, _fieldName) => string.split(/[\s\-\.]+/),
  processTerm: (term, _fieldName) => {
    let word = stopWords.has(term) ? null : term.toLowerCase()
    if (word) {
      word = word.replace(/^[^a-zA-Z0-9@!]+/, '')
                 .replace(/[^a-zA-Z0-9@!]+$/, '')
    }

    return word ?? null
  },
  searchOptions: {
    boost: { title: 100 },
    fuzzy: 0.2,
    tokenize: (string, _fieldName) => string.split(/[\s\-\.]+/),
  },
})

search_index.addAll(data)

// To write a gzip json file
// To read a gzip file on client side use pako
// gz.gzip(Buffer.from(JSON.stringify(search_index)), (_, result) => {
//   fs.writeFileSync('../assets/search_index.json.gz', result)
// })

fs.writeFileSync('../assets/search_index.json', JSON.stringify(search_index))

So, while the coding journey is just beginning, the potential impact is already looking promising. Stay tuned for more technical insights and the adventures that await as I transform lines of code into meaningful enhancements for all.

Next time, I will be going through the actual internal workings of the searching logic with the semi-improved UI for the current search and what the mid-term evals are in GSoC. And for those curious minds who've been inquiring about my GSoC proposal, I've got a treat for you! I will also be linking my proposal, where the initial seeds of this exciting journey were sown. Stay tuned for more updates as I continue to shape the future of Julia's search functionality. Until next time, keep coding and dreaming big!

If you're interested in following my progress or connecting with me, please feel free to reach out!

Github: Hetarth02
LinkedIn: Hetarth Shah
Website: Portfolio

Thank you for joining me on this journey, and look forward to more updates as I continue to contribute to the Julia community during GSoC.

Credits:

Cover Image by Kevin Granger

Two Weeks of Coding

Hetarth Shah — Sun, 16 Jul 2023 18:55:09 +0000

Welcome, coding aficionados! It's time for a recap of the first two weeks of my GSoC journey with Julia. Now, don't get too excited, as the start of the project is often a slow one. These initial weeks were all about getting familiar with the project, diving into the codebase, and understanding the existing search functionality. So, grab your caffeinated beverage of choice, sit back, and let's embark on this slightly leisurely adventure!

During the initial weeks of GSOC, I mainly engaged in insightful discussions with my mentor, which proved to be incredibly valuable. These conversations allowed us to align our expectations for the new UI and dive deep into the intricacies of the search functionality. Together, we examined the existing flaws and brainstormed approaches to address them.

As we delved deeper into the search functionality, we realized the need for a smoother developer experience and a more maintained solution. It led us to explore the possibility of shifting from Lunr.js to MiniSearch.js. This shift to MiniSearch.js held the promise of streamlining our development workflow and reducing the overall maintenance burden. With its efficient indexing algorithms and smaller index sizes, we could deliver a snappier search experience to our users.

These first two weeks of GSoC in the world of Julia might have been a bit slow and steady, but that doesn't mean we weren't making progress. From discussions about expectations and flaws to proposing new ideas for a search engine, we are setting the stage for exciting improvements. So, let's raise our virtual glasses to the future, where a revamped search UI will bring joy and efficiency. Stay tuned for more thrilling developments as this GSOC adventure unfolds!

If you're interested in following my progress or connecting with me, please feel free to reach out!

Github: Hetarth02
LinkedIn: Hetarth Shah
Website: Portfolio

Thank you for joining me on this journey, and look forward to more updates as I continue to contribute to the Julia community during GSoC.

Cracking GSoC for Julia

Hetarth Shah — Fri, 19 May 2023 17:32:28 +0000

Today, I share my thrilling experience of cracking Google Summer of Code (GSoC) for the Julia programming language. In this blog post, I'll take you through my journey of preparing for GSoC 2023, selecting the right organisation, and my activities during the community bonding week. Let's dive in!

Starting Early and Researching Organisations:

As soon as GSoC 2023 was announced, I began researching various organisations planning to participate in the program. I cherry-picked organisations that not only aligned with my skills and interests, but would also allow me to make a meaningful impact in the community. It felt like a treasure hunt, except instead of gold💎, I was seeking the perfect match for my skills and interests.

Reaching Out to Potential Mentors:

With my research done, I put on my networking hat and reached out to potential mentors to initiate communication and express my interest in contributing to their projects. This initial contact allowed me to establish a connection, gather more information about the projects, and discuss my potential contributions. The mentor's guidance was invaluable in helping me understand project requirements, refining my ideas and ultimately developing a strong project proposal.

Submission and Selection:

After putting in meticulous effort, I submitted my GSoC application with confidence. The anticipation during the selection period was palpable, and when the acceptance email arrived, it was a moment of pure joy🎊. Getting selected for GSoC validated my dedication and hard work, motivating me for the exciting journey ahead.

My GSoC Project📃

Week 0 (Community Bonding):

With the community bonding period going on, we had google organised events like QnA session and contributor summits. I also had a video call with my mentor to discuss the project in detail and plan the upcoming weeks. We went over the project milestones and expectations, ensuring a clear understanding of what was expected from me.

Conclusion:

Cracking GSoC for Julia has been an incredible journey, starting from thorough research and selecting the right organisation to the exciting community bonding period. I'm thrilled to be a part of the Julia community and look forward to contributing meaningfully to the project during GSoC 2023.

Remember, early preparation, effective communication, and proactive engagement are key to succeeding in GSoC. Best of luck to all future GSoC aspirants, and may your journey be as rewarding as mine!

If you're interested in following my progress or connecting with me, please feel free to reach out!

Github: Hetarth02
LinkedIn: Hetarth Shah
Website: Portfolio

Thank you for joining me on this journey, and I look forward to sharing more updates as I continue to contribute to the Julia community during GSoC.