DEV Community: Kinanee Samson

Build a markdown parser from scratch (Tokenizer)

Kinanee Samson — Mon, 14 Jul 2025 07:52:51 +0000

How many of you have ever had the desire to build a parser from scratch? Since you started using your favorite programming language, it must have occurred to you at some point. You have asked yourself countless times, How is this jargon I'm writing being compiled or interpreted?

You may know a thing or two about compilers and interpreters, but you haven't taken the time to implement or build one of your own. Well, look no further because in today's video, I will take my time to teach you how to make a parser for the markup language (Markdown).

In this video, you'll discover numerous amazing and useful concepts that will 100% make you a 10X developer; heck, you might even create your programming language after you're done with this video. You will learn things like

Regular Expressions (RegEx) for Pattern Matching.
Object-Oriented Programming (OOP)
Fundamentals of a Compiler/Interpreter Pipeline (Lexical Analysis (Tokenization), Syntactic Analysis (Parsing), and Code Generation (Compilation))
Recursive Parsing and Compilation

If you prefer watching video content, then you can watch the video on YouTube.

Let’s create a type TokenType which will be a union type of a HEADING, PARAGRAPH, NEWLINE, TEXT and SPACE. Next, I will create an interface Token that will serve as the interface for valid tokens. It will have a type property, which will be a TokenType. It will also have a value property, which is a string; it will have an optional level property, which is a number; an optional alt property, which is a string; an optional url property, which is a string; it will also have an optional indent property, which is a string. This interface will represent an identified token, while the tokenType is self-explanatory.

export type TokenType =
  | "HEADING"
  | "PARAGRAPH"
  | "NEWLINE"
  | "TEXT"
  | "SPACE";

export interface Token {
  type: TokenType;
  value: string;
  level?: number;
  alt?: string;
  url?: string;
  indent?: number;
}

Next, I will define an interface ASTNode, which will represent an identified node. It will have a type property, which is of tokenType, and it will have a children's property, which is an array of ASTNode, and it will have an optional value property, which is a string, and a level property, which is a string. It will also have an optional level property, which is a string; it will also have an optional alt property, which is a string, and it will also have an optional indent property, which is a number, and finally, it will have an optional lang property, which is also a string.

export interface ASTNode {
  type: TokenType;
  children?: ASTNode[];
  value?: string;
  level?: number;
  alt?: string;
  url?: string;
  indent?: number;
  lang?: string;
}

We need to implement a tokenizer; it is the job of the tokenizer to go over the text string we want to parse and identify meaningful tokens inside it. Imagine you have a long piece of text, like a sentence or a paragraph. The tokenizer is like a specialized reader that goes through this text and breaks it down into individual,
meaningful "words" or "chunks." These chunks aren't English words; rather,
they're defined by specific rules. In programming, we call these chunks tokens.

Algorithm for Tokenizer

Start with an empty list called tokens (this is where we'll store all the identified tokens).
Set a position pointer to the beginning of the input string.
Create a predefined list of patterns for the types of elements you want to recognize.
Each pattern consists of A regular expression (regex): The search rule to find the element. A token type: What to call this element if it's found (e.g., "HEADING", "LINK", "TEXT"). An optional handler function: A small piece of code that will extract specific details from the matched text (like the heading level or a link's URL) and put them into the token.
Keep looping as long as the position pointer hasn't reached the end of the Input string.
Inside the loop, for the current position, try to match each predefined pattern (regex) against the remaining part of the input string, starting exactly at the position.
Reset the regex's internal lastIndex before each attempt to ensure it always starts matching from the correct spot.
If a pattern successfully matches at the current position
- Get the full text that was matched.
- Use the optional handler function (if it exists) to extract any special data for this type of token (e.g., level for headings, url for links, alt for images). If no handler, just use the matched text as the token's value.
- Create a new Token object with the correct type and the extracted data.
- Add this new Token to your tokens list.
- Advance the position pointer by the length of the matched text, so the next search starts right after the token you just found.
- Stop checking other patterns for this position and go back to step 3 (Start the loop again for the new position.)
If no pattern matches at the current position:
- This means the character (or sequence of characters) isn't recognized by any of your predefined rules.
- For the tokenizeInline method, it typically creates a "TEXT" token for the single character it couldn't match.
Advance the position pointer by one character, effectively skipping over the unrecognized part (or treating it as plain text if that's the fallback behavior).
Once the loop finishes (meaning you've gone through the entire input string)
Return the tokens list. This list is now ready for the next stage: the parser.

class Tokenizer {
  tokenize(input: string): Token[] {
    const tokens: Token[] = [];

    let pos = 0;

    const blockPatterns: [RegExp, TokenType, Function?][] = [
      // * HEADING
      [
        /^#{1,6}\s[^\n]*(?:\n|$)/,
        "HEADING",
        (match: string) => ({
          level: match.trim().split(" ")[0].length,
          value: match.replace(/^#+\s/, "").trim(),
        }),
      ],
      //*  NEWLINE
      [/\n+/, "NEWLINE"],

      // * TEXT
      [/^(?!#|!|\[|-)[^\n]+/, "TEXT", (match: string) => ({ value: match })],

      // * Links
      [
        /^\[([^\]]+)\]\(([^)]+)\)/,
        "LINK",
        (match: string, text: string, url: string) => ({
          value: text,
          url,
        }),
      ],
      // * Images
      [
        /^!\[([^\]]+)\]\(([^)]+)\)/,
        "IMAGE",
        (match: string, alt: string, url: string) => ({
          alt,
          url,
          value: match,
        }),
      ],

      // * LIST_ITEM
      [
        /^(\s*)([-*+])\s+([^\n]*)(?:\n|$)/,
        "LISTITEM",
        (match: string, indent: string) => ({
          value: match.replace(/^\s*[-*+]\s+/, "").trim(),
          indent: indent.length,
        }),
      ],
    ];

    while (pos < input.length) {
      let matched = false;

      for (const [regexp, type, handler] of blockPatterns) {
        regexp.lastIndex = 0;
        const match = regexp.exec(input.slice(pos));
        if (match && match.index === 0) {
          const tokenData = handler ? handler(...match) : { value: match[0] };
          tokens.push({
            type,
            ...tokenData,
          } as Token);

          pos += match[0].length;
          matched = true;
          break;
        }
      }

      if (!matched) {
        pos++;
      }
    }
    return tokens;
  }
}

Then I will create a class Tokenizer. Inside this token class, I will define a tokenize function that accepts an input, which is a string, and returns an array of tokens. Don't worry, I will
fix this bug shortly. Inside the function, I will create a variable tokens, which will be an array of tokens. All identified tokens will be stored inside this array. I will create a variable pos, which will be equal to 0. This will serve as a cursor that will go over the text and identify tokens.

I will create a variable blockPattern, which will be an array of tuples. The first item will be a regular expression, the second item will be the type for the token, and the third argument will be an optional function. This will be equal to an empty array. The first item inside this array will be an array
itself, and the first item will be a regular expression that matches a markdown heading The second item, which is the type, will be a heading, and the last item will be a function that accepts a match, which is a string, and inside it we return an object with a value property that will match the actual
heading and a level property, which replaces the number of headings for that particular heading element.

We will add another array to the parent array, and this will allow us to identify a new line token from the string. We will attempt to only parse a heading and a newline for now. I will now add a while loop that will run as long as we have not gotten to the end of the input string. Then I will create a variable matched, which will be false.

Then I will loop through the block patterns array, extracting the regex, type, and handler function from each array inside, then I will set the lastIndex property on the regex to be equal to zero, then we will attempt to extract a token from the text by first creating a variable match, which stores the result of calling the exec method on the regexp, we will ensure that we slice the input starting from the position of the cursor.

Then I will check that there is a match and match.index is equal to zero, and if this is the case, I will create a tokenData variable, the value of this variable will be conditional. First, we will check if the handler function exists. If it does, I will call the handler function with the result of the match, but if the handler does not exist, then the tokenDatawill be equal to an object with a value property whose value will be the first item in the match array, and this stores the values that were matched against the regular expression.

Then I will push the token inside the tokens array and increment the pos(the cursor) by the length of the matched item, set matched to true, and break out of this inner for loop. After the for loop, we will just increment the cursor by one if there is no match, and after the while loop, I will return the tokens array.

This tokenizer is responsible for block-level tokenization. The method (tokenize) focuses on identifying larger, structural pieces of your text, like entire headings, links, or images. It processes these "blocks" line by line or by significant patterns. It uses a list of predefined rules, each containing a regular expression (regex). When it finds a match, it grabs that chunk of text, determines its type (e.g., "HEADING", "LINK"), and extracts any specific information related to it (like the heading level or the link's URL). It then moves past that matched chunk and continues searching in the rest of the text.

const tokenizer = new Tokenizer()
console.log(tokenizer.tokenizer(`
## Hello World
`))

// [
//   { type: "NEWLINE", value: "\n" },
//   { type: "HEADING", value: "Hello World", level: 2 }
// ]

So let's test out this class by creating a variable tokenizer which stores the result of instantiating the tokenizer class, then I will attempt to tokenize a basic markdown string with just a two-level heading, and logging out the tokens array to the console.

You can observe that there are two tokens in the array, one corresponding to a new line, while the second is a heading where the level property is two. The next thing would be to tokenize inline characters. We need to split the value obtained from a heading into individual tokens, and this is the job of the tokenizeInline method. This method will not be called directly; rather, it will be called by the parser.

class Tokenizer {
  tokenize(input: string){
    // cont'd
  }

  tokenizeInline(input: string): Token[] {
    let tokens: Token[] = [];
    let cursor = 0;

    const inlinePatterns: [RegExp, TokenType, Function?][] = [
      [/\s+/, "SPACE"],
      [/[^\s*_![\n]+]/, "TEXT"],
      // Links
      [
        /^\[([^\]]+)\]\(([^)]+)\)/,
        "LINK",
        (match: string, text: string, url: string) => ({
          value: text,
          url,
        }),
      ],
      // images
      [
        /^!\[([^\]]+)\]\(([^)]+)\)/,
        "IMAGE",
        (match: string, alt: string, url: string) => ({
          alt,
          url,
          value: match,
        }),
      ],

      // Bold
      [
        /(\*\*|__)([^\n]+?)\1/,
        "BOLD",
        (match: string, _: any, content: string) => ({
          value: content,
        }),
      ],
      // Italic
      [
        /(\*|_)([^\n]+?)\1/,
        "ITALIC",
        (match: string, _: any, content: string) => ({
          value: content,
        }),
      ],
    ];

    while (cursor < input.length) {
      let matched = false;

      for (const [regexp, type, handler] of inlinePatterns) {
        regexp.lastIndex = 0;
        const match = regexp.exec(input.slice(cursor));

        if (match && match.index === 0) {
          const tokenData = handler ? handler(...match) : { value: match[0] };

          tokens.push({
            type,
            ...tokenData,
          } as Token);

          cursor += match[0].length;
          matched = true;
          break;
        }
      }

      if (!matched) {
        tokens.push({ type: "TEXT", value: input[cursor] });
        cursor++;
      }
    }
    return tokens;
  }
}

Now let's implement the tokenizeInline method, which will accept an input argument, which is a string, and it will return an array of tokens. Inside this method, I will create a variable tokens, which is an array of tokens; this variable will store tokens extracted from the input. I will create another
variable cursor, which will be equal to 0.

I will now create an inlinePatterns variable, which is similar to the blockPatterns variable we created when implementing the tokenize method. The inlinePatterns will be an array of tuples, where the first item in the tuple is a regular expression, the second item is the type of the token, while the third item is an optional function.

The first tuple will identify white spaces, while the second tuple will identify and extract text tokens after the inlinePatterns array. I will add a while loop that will run until we have consumed all of the tokens from the input. Inside this while loop, first, I will create a variable matched that tracks whether or not a token was matched, and then create a for loop that goes over the inlinePatterns array, extracting the regular expression, the type, and the handler function.

Inside the for loop, I will set regexp.lastIndex equal to 0, then I will try to identify a token. This will be done by creating a variable match, which is equal to the result of calling the exec method on the regexp I will pass the input string as an argument to it, but I will ensure that we slice the input string with the cursor to ensure that we don't continue parsing tokens we have already consumed.

If there's a match and the index of the match is equal to zero, then I will create a variable tokenData, which will be equal to the result of calling the handler function with the match, if there's a handler function, otherwise I will just return an object with a value property whose value is the first item in the match array.

I will push a new token inside the tokens array, the type for that token will be the matched type, while I will spread the tokenData into the rest of the token, then I will increment the cursor by the length of the value matched which is stored in the first index inside the match array, I will also set matched to be true and break out of this inner for loop.

If matched is false, then we want to push a text token into the tokens array and increment the cursor by one. I will exit the while loop and ensure that I return the identified tokens at the end of the function.

Understanding Garbage Collection: How Modern Programming Languages Manage Memory

Kinanee Samson — Mon, 07 Jul 2025 08:09:48 +0000

You just had a brand new idea for an awesome project, you believe that the idea for this project is going to take the world by storm. Users and developers alike are not prepared for the next big thing in tech. So you open your favorite editor, im guessing VS Code, except of course you are already on the vibe train and you're using one of those over hyped VS Code AI forks (which is still VS Code by the way just with a lot of AI packed into it).

Let's assume you decide to use a mordern programming language like JavaScript or Go, you go about building your awesome new project you are going to abandon in the next 2 weeks when you get another awesome idea. You create variables, arrays and objects making sure to suck every available memory location from your CPU. Your program works or barely just, you don't know how or why it works but you do know that it works.

But let's pause for a moment and ask ourselves how come I am not using up all the available memory in my CPU despite creating so many variables and objects, somehow it seems that your app runs smoothly with little to no perfomance overhead, you ask yourself again how come the objects in my code are not using up all the available memory? Well this is because of the very awesome Garbage collector that sits at the heart of your favorite mordern programming language and this is going to be our focus for today's video.

We are going to look at garbage collection, what it is, how it works from top to bottom and why it is so useful and found in almost all mordern programming langauge. If you think that makes sense then absolutely smash the like button, ensure you subscribe to the channel to get access to all of our videos in the future as they drop. Make sure you stick around to the end cause I have a challenging question for you.

If you prefer watching video content, then you can watch the video on YouTube.

Garbage collection (GC) is a form of automatic memory management that involves automatically reclaiming memory in a computer program that is no longer being used. It frees developers from manually tracking and releasing memory, reducing the risk of memory leaks.

The garbage collector attempts to reclaim memory that was allocated by the program, but is no longer referenced; such memory is called garbage. Garbage collection was invented by American computer scientist John McCarthy around 1959 to simplify manual memory management in Lisp.

Before the introduction of the garbage collector a developer was tasked with hunting down objects that have gone out of scope in a program and de-allocating that object from memory like a caveman. Garbage collection is not all sweet and no sour as it may take a significant proportion of a program's total processing time, and affect performance as a result.

We have already established that most mordern programming languages incoorperates a garbage collector either as part of the language specification (e.g., Java, C#, Go, and JavaScript) or effectively for practical implementation and these are said to be garbage-collected languages.

Languages, such as C and C++, were designed for use with manual memory management Some languages, like Ada, Modula-3, and C++ allow both garbage collection and manual memory management to co-exist in the same application by using separate heaps for collected and manually managed objects.

Garbage Collection Strategies

We will talk about two major types of garbage collectors namely Tracing Garbage Collectors and Reference Garbage Collectors.

Tracing

Tracing garbage collection is the foundation of modern JavaScript garbage collectors. It's a strategy that identifies and reclaims memory by determining which objects are still reachable from known root references. This is the most common type of garbage collection – so much so that "garbage collection" often refers to the tracing method.

Tracing collectors are so called because they trace through the working set of memory. These garbage collectors perform collection in cycles. It is common for cycles to be triggered when there is not enough free memory for the memory manager to satisfy an allocation request. But cycles can often be requested by the mutator directly or run on a time schedule.

Reachability

When talking about Tracing Garbage Collectors we need to talk about Reachability. Reachability determines which objects stay in memory and which gets garbage collected. An object is reachable if it is referenced by at least one variable in the program, either directly or through references from other reachable objects. Objects can be reachable in only two ways:

-These are global variables and objects that are referenced from anywhere in the call stack (that is, all local variables and parameters in the functions currently being invoked).

Anything referenced from a reachable object is itself reachable; more formally, reachability is a transitive closure.

Algorithms Using Tracing

Mark and Sweep

Mark-and-sweep is the core algorithm behind JavaScript's garbage collection. It solves the critical flaw of reference counting by handling circular references naturally. In the naive mark-and-sweep method, each object in memory has a flag (typically a single bit) reserved for garbage collection use only. This flag is always cleared, except during the collection cycle.

Phase 1: Marking (Identifying Reachable Objects)

The first stage is the mark stage which does a tree traversal of the entire 'root set' and marks each object that is pointed to by a root as being 'in-use'. All objects that those objects point to, and so on, are marked as well, so that every object that is reachable via the root set is marked.

Phase 2: Sweeping (Reclaiming Memory)

In the second stage, the sweep stage, all memory is scanned from start to finish, examining all free or used blocks; those not marked as being 'in-use' are not reachable by any roots, and their memory is freed. For objects which were marked in-use, the in-use flag is cleared, preparing for the next cycle.

This method has several disadvantages, the most notable being that the entire system must be suspended during collection; no mutation of the working set can be allowed. This can cause programs to 'freeze' periodically (and generally unpredictably), making some real-time and time-critical applications impossible.

Tri-color marking

This is a specific technique used within the mark phase of mark-and-sweep. It categorizes objects into three colors. Because of the performance problems associated with a naive mark-and-sweep GC, most modern tracing garbage collectors implement some variant of the tri-color marking abstraction. Three sets are created – white, black and grey;

The white set, or condemned set is the set of objects are no longer being used and require clean up .
The grey set contains objects that have been discovered but their children haven't been fully scanned yet.
The black set is the set of Objects that have been visited and all their children have also been visited, so they are confirmed as live.

In many algorithms, initially the black set starts as empty, the grey set is the set of objects which are directly referenced from roots and the white set includes all other objects. Every object in memory is at all times in exactly one of the three sets.

The algorithm proceeds as following:

Pick an object o from the grey set
Move each white object that o references to the grey set. This ensures that neither this object nor any object it references can be garbage-collected.
Move o to the black set
Repeat the last three steps until the grey set is empty.

When the grey set is empty, the scan is complete; the black objects are reachable from the roots, while the white objects are not and can be garbage-collected.
Since all objects not immediately reachable from the roots are added to the white set, and objects can only move from white to grey and from grey to black, the algorithm preserves an important invariant – no black objects reference white objects. This ensures that the white objects can be freed once the grey set is empty. This is called the tri-color invariant.

The tri-color method has an important advantage – it can be performed "on-the-fly", without halting the system for significant periods of time. The system can perform garbage collection periodically, rather than as needed, and the need to touch the entire working set on each cycle is avoided.

Non Moving vs Moving Systems
The garbage collector may simply discard the unreachable objects and leave other parts of the system the way it is and this type of Garbage collector is known as a non-moving garbage collector.

In other systems the garbage collector might move some or all of the reachable objects into a new area of memory and update all references to the objects and this is known as a moving garbage collector.

Why the distinction? It seems on paper that the moving GC may appear to the doing too much when compared to the non-moving GC, however the moving algorithm leads to several performance advantages, both during the garbage collection cycle itself and during program execution, some of which are

The space that was freed up when the moving GC moved the reachable objects into a new memory is now considered free space without the system doing any extra work. In contrast a non-moving GC will need to ensure that it checks each memory occupied by each object and ensure that it is now free.
This gives the GC an ability to rapidly allocate new objects because the ALG naturally creates large swabs of free memory, all the system needs to do is to increase a free memory pointer to get access to more memory. A non-moving ALG will eventally lead to a fragmented heap and this will in turn require very expensive requesting of the list of free available blocks of memory in order to allocate new objects.
Objects can be moved very close to the objects they refer to in memory, increasing the chance that they will be located in the same cache line or virtual memory page. This can significantly speed up access to these objects through these references.

It is not all rosy with moving systems because their implementation does not give much room for pointer arithmetics and all references to objects are managed by the garbage collected environment. This is because any pointers to objects will be invalidated if the garbage collector moves those objects

Reference Counting

Reference counting is an alternative approach to automatic memory management where each object tracks how many references point to it. When references drop to zero, the object is immediately collected. The main advantage of the reference counting over tracing garbage collection is that objects are reclaimed as soon as they can no longer be referenced, and in an incremental fashion.

In real-time applications or systems with limited memory, this is important to maintain responsiveness. Reference counting performance does not deteriorate as the total amount of free space decreases. Reference counting has three main disadvantages over the tracing garbage collection;

The frequent updates it involves are a source of inefficiency. Reference counting requires every memory-managed object to reserve space for a reference count. In tracing garbage collectors, this information is stored implicitly in the references that refer to that object, saving space.
Reference counting can't handle reference cycles, an object which refers directly or indirectly to itself. A mechanism relying purely on reference counts will never consider cyclic chains of objects for deletion, since their reference count is guaranteed to stay nonzero. Methods for dealing with this issue exist but can also increase the overhead and complexity of reference counting.
In a concurrent setting, all updates of the reference counts and all pointer modifications must be atomic operations, which incurs an additional cost.

Implementation Strategies

Weighted reference counting

In weighted reference counting, each reference is assigned a weight, and each object tracks not the number of references referring to it, but the total weight of the references referring to it. A newly created object has a large weight say 1000 and when ever there's a new reference to that object is the weight of the original reference is split in two and one half points to the new reference.

Since the total weight does not change, the object's reference count does not need to be updated. Destroying a reference decrements the total weight by the weight of that reference. When the total weight becomes zero, all references have been destroyed. Weighted reference counting was independently devised by Bevan[12] and Watson & Watson[13] in 1987.

Deferred (Indirect) referrence counting

DRC is an optimization for reference counting (RC) that minimizes its core weakness: by delaying the update of reference counts for certain objects, particularly those stored on the stack.

Instead of updating counts for every reference change, it postpones these updates until a designated "reconciliation phase". This can significantly improve performance, especially in languages where stack references are common.

Some drawbacks of deferred reference counting are:

It Requires compiler support to identify and track references, especially those stored in registers and on the stack.
This strategy cannot reclaim objects with cyclical references (where objects refer to each other in a loop).
May require a stop-the-world reconciliation phase, which can pause execution

Async the Programming: How Concurrency Works in JavaScript

Kinanee Samson — Sun, 29 Jun 2025 11:03:03 +0000

Picture this: you are trying to make a network request; let's say you are building the next big AI startup, but you don't have the funds and technical knowledge required for LLMs, and you shamelessly use the OpenAI API because that's what a normal developer would do. You give barely any attention to how the request is made; yes, you write the code, and I'm betting all the money I have, which is none, that you are using async/await.

We are all accustomed to the syntax of async/await or Promises, but rarely do we fully understand how this simple yet powerful feature enables us to build non-blocking applications. One of the most satisfying parts of working with JavaScript is the ability to handle asynchronous tasks. Some of you might have a basic understanding of how asynchronous programming works in JavaScript, but on a deeper level, do you fundamentally understand how JavaScript, which is inherently synchronous and will run on a single thread executing code line by line, will allow you to write code that can be executed asynchronously

In today's post, I will take my time to explain in detail what asynchronous programming is and how it is implemented in JavaScript. If you prefer watching video content, then you can watch the video on YouTube.

What is asynchronous programming?

Asynchronous programming is a technique that allows a program to execute tasks without blocking the main thread. Instead of waiting for a time-consuming task to complete, the program continues running other operations. When the task finishes, its result is processed. This approach is essential for efficient I/O operations (e.g., network requests, file reading) and scalable systems. Asynchronous programming in JavaScript is achieved via callbacks, promises, and the event loop, and is single-threaded by design.

We have certain terms in that explanation that we need to split apart to understand more of what's happening, first on the CPU level and then work our way back up to the software level.

What is a thread?

A thread is the smallest unit of execution within a process. (Hold on now, we've got another term, process, which I will further expand on shortly; for now, it's back to threads). A thread is a sequence of programmed instructions that can be managed independently by an operating system scheduler (this is another term that I will also touch on shortly). Threads enable concurrent execution within a single program, allowing multiple tasks to make progress simultaneously.

What is a process?

A process is an independent instance of a running program with its dedicated resources, managed by the operating system (OS). A process represents a program in execution, complete with code, data, memory space, and system state.

Key Characteristics Of Processes

One process cannot directly access another's memory.
The crash of one process doesn’t affect others (unless designed to communicate).
Processes can spawn child processes (e.g., a terminal launching a text editor).

Each process has a Process Control Block (PCB) stored in kernel memory:

Process ID (PID): Unique identifier for each process (e.g., 4043).
Program Counter: Next instruction to execute.
Memory Limits: Code, data, heap, and stack boundaries associated with that process.
I/O Status: Open files, network connections.
Permissions: User/group privileges.

Threads and processes are fundamental units of execution managed by an operating system, but they serve different roles and interact in specific ways. Here's a clear breakdown of their relationship:

A process is a container for resources (memory, files, CPU time), while threads are workers inside the container that execute tasks. Every process has at least one thread (the main thread). A process can spawn multiple threads to work concurrently.

Threads share the process's resources (memory, files), but each thread runs independently with its own:

Stack (local variables, function calls).
Program counter (next instruction).
Register state (CPU context)

Process = A factory (with its building, supplies, and budget).
Threads = Workers in the factory (share the factory's resources but perform separate tasks).

Key Differences

Processes are heavyweight (slow; need OS setup), while threads are lightweight (fast; share existing resources).
Processes are crash-safe (fail independently), while a single thread crash kills an entire process.
Threads are complex and require Inter-Process Communication (IPC), while threads are simple with direct shared memory access.

How They Interact

There is a single-threaded process, where one thread handles all tasks (e.g., a simple CLI tool). In this situation, blocking one task blocks the entire process.

Multi-Threaded Process where multiple threads run concurrently within one process: Example: A web browser. Thread 1: Downloads images, Thread 2: Renders the page, Thread 3: Handles user clicks; threads share heap memory but not stack memory.

Multi-process applications with separate processes run independently (e.g., Chrome uses multiple processes for tabs). Communications via IPC (pipes, sockets, shared memory).

Operating system scheduler
The OS scheduler is a core component of an operating system that decides which process/thread runs next on the CPU. It ensures efficient resource use, prevents starvation, and balances responsiveness across all tasks.

Why do we need an OS scheduler?
It is important for

CPU Utilization: Keeps the CPU busy (e.g., switches to a ready task when one waits for I/O).
Fairness: Prevents any single task from monopolizing resources.
Throughput: Maximizes tasks completed per time unit.

Each browser tab is typically its process in modern browsers (Chrome, Edge, Firefox), but a single tab can also use multiple processes for advanced isolation.
JavaScript runs on threads within these processes, with one main thread per renderer process that handles DOM, events, and your JS (single-threaded!). Let's assume that a browser tab is using a single process for its execution. JavaScript is fundamentally single-threaded, meaning it uses only one main thread for execution within a browser tab's process. T

This design simplifies programming but requires careful handling of blocking operations. Here's how synchronous execution works with threads and processes:

Each browser tab runs in its renderer process; within this process, JavaScript executes on a single main thread. All tasks (DOM updates, event handling, JavaScript code) run sequentially on this thread.
JavaScript executes code in sequence without interruption
Each operation must finish before the next starts.

Take a simple JavaScript program.

// 1. Synchronous code runs immediately
console.log("Start"); 

// 2. Synchronous blocking operation
for (let i = 0; i < 1e9; i++) {} // Freezes the tab for 3 seconds

// 3. Next line waits for the loop to finish
console.log("End");

The main thread runs console.log("Start"). The main thread is blocked by the CPU-intensive loop. After 3 seconds, console.log("End") runs. During this time, the tab freezes (no UI updates, no event handling).

For JavaScript to behave asynchronously, there is an event loop, which:

Manages asynchronous operations (e.g., setTimeout, fetch) without multi-threading.
Delegates I/O tasks to browser APIs (which use background threads), then queues callbacks. JavaScript uses the browser's multi-threaded capabilities behind the scenes.

Take a simple asynchronous code,

console.log("Start"); 

// Asynchronous: Delegated to the browser's timer thread
setTimeout(() => console.log("Timeout"), 0); 

console.log("End");

Start  
End  
Timeout

Why? setTimeout is handled by a browser timer thread (outside JavaScript's main thread). The callback () => console.log("Timeout") is queued in the callback queue after 0ms. The event loop moves it to the main thread only when the call stack is empty.

Callback Queue, Heap, and The Call Stack?

Since we have already established that every JavaScript program is a self-isolated process, and the process provides a queue, a term we will look at shortly, meanwhile, each thread has its stack. Stack another term we will look at, and it's heap. The heap is a largely unstructured area of memory allocated to all the threads currently running in a process. Let's think of the heap as an area where messengers on horseback wait before they get to deliver their messages.

When it comes to the turn of a messenger, they unmount and move to the queue. This area they are attended to in a first in first out manner, when each messenger dispatches their message, there is usually a reaction to that message, which in our case is a function call, for every message in the queue there is a function associated with it, that function is called when the message is processed out of the queue.

Each function call creates a stack frame that contains the statement and expression in the function. When that function returns a value or void, its frame is then popped out, and the next function will begin executing. If we call a function inside another function, a frame will be created for each.

The frame for the nested function sits on top of the frame for the function that called it. When the nested function is done executing, it will return and get popped off, and the main function will continue executing or return and get popped off. The items in the stack are treated in a last-in in first-out format.

The stack is a data structure that holds the frame for each function. We can deduce that this is a synchronous process, so how is concurrency achieved with the stack and the queue? In other words, the call stack is just a list of instructions in the program.

Event Loop

The event loop is simply a loop that iterates through the queue and processes any message if one is in the queue. Since we are in a JavaScript development environment, messages could also be added to the queue as a result of events happening in the DOM (Main thread).

The event loop does not care; its job is to process the messages in the queue. It is interesting to remember that a stack frame, which is in essence a function call, can emit an event that adds a new message to the queue, or it can directly add a message to the queue. So when the result of an expression might take a long time, some APIs allow us to add that result as a message when it is available to the queue, and we go on processing other things without waiting.

This is the basis of callback-based code. This is also how setTimeout and setInterval add messages asynchronously to the queue. When we write a setTimeout function, a message is added to the queue after the specified delay in milliseconds.

getSomeData(place, action)
// do something with the place
 let result = { data : place } // something
 setTimeout(() => {
   action(result)
 }, 0)
}

getSomeData("london", console.log)
console.log("hey")

From the example above when the first function is executed a new stack frame is created, we create a variable and then use setTimeout to call the function passed in. as the second argument and give it the variable we created earlier when, if the first function has to take some time before completing the action would have to wait, but our code does not have to wait and it moves on to processing the next statement, when the result is ready action is called with the result passed in as an argument to the function.

A new stack frame is created for it, and the next message in the queue is processed if any. The above process, the way the event loop is described above, is synchronous; the event loop is usually expressed in terms of a while loop.

while(queue.waitForMessage()){
 queue.processNextMessage()
}

To take advantage of the event loop, initially JavaScript used callback-based code, then we got promises, and finally we have async/await.

Callbaclk Based Code

Callback-based code is usually the first solution to asynchronous programming, and it involves passing a function as an argument to another function; the function we passed as an argument will delay execution until the initial function has finished running, then the function we passed as a callback will run.

let request = function(url, cb){
  let XHR = new XMLHttpRequest();
  XHR.open('GET', url, true)
  XHR.send(null)
  XHR.onload = function(){
    if(this.status === 200){
      cb(undefined, XHR.response)
    }
    else if(XHR.status !== 200){
      let err = { message: 'Error fetching resource', status: XHR.status}
      cb(err, undefined)
    }
  }
}

request('jsonplaceholder', (err, data) => {
  if(!err){
    console.log('request completed', data)
  }
  else{
    console.log('request completed', err)
  }
)

Promises
Promises represent a cleaner way of performing asynchronous tasks. A Promise basically will return the result of an asynchronous process, and you can access that using a then method to handle the data, or a catch method to handle errors. Let's see the basic syntax of a promise.

console.log('before myProm called')
let myProm = new Promise((resolve, reject) => {
  if(1 < 2) resolve(true)
})
console.log('myProm defined')

myProm.then(data => console.log('got data back', data))

A Promise is declared using the Promise constructor, which takes a function as an argument, and that function we pass as an argument to the promise takes in two parameters, resolve and reject.

We use resolve to return a value from the promise if everything is okay, and we call reject to return an error if something is wrong. The data that is resolved can be accessed using the then method, which takes in an argument argument represents the data that is resolved by the promise, and in the example above, we just log it to the console. We didn't handle failures in our above example, but if there was a failure, we would use the reject parameter and reject a value with it; the data that is returned by the reject method is made available on the catch method.

Async/Await
Async/Await is a new feature of JavaScript, and it makes handling asynchronous code easy. We can mark a function to be asynchronous using the async keyword, and then we use the await keyword to await some asynchronous task and continue writing other logic inside our function. Async/await is a much-improved way of dealing with promises.

When we use the async keyword to mark a function as asynchronous meaning at some point we will do some form of asynchronous work and then inside the function we use the await keyword before the asynchronous operation and in our case it is the request function, now this is what will happen; the function will start executing and when it encounters the await keyword it will suspend the function and move on to the next thing, when the value from the request is available it continues with the async function and we see our data logged out to the console. The await keyword simply waits for a promise to evaluate and return a value or an error, and then we continue with the rest of the code.
Here are the major takeaways from this topic:

Asynchronous programming is a technique that allows a program to execute tasks without blocking the main thread. Asynchronous programming in JavaScript is achieved via callbacks, promises, and the event loop.
Threads are the smallest unit of execution within a process; it is a sequence of programmed instructions that can be managed independently.
A process is an independent instance of a running program with its dedicated resources; it represents a program in execution.
The JavaScript code you write runs on a single main thread, and asynchronous behaviour can only be achieved using the Event Loop, the call stack, and the Queue.

Node.js Backend Security: Password Hashing & JWT From Scratch (DIY)

Kinanee Samson — Sat, 21 Jun 2025 00:21:38 +0000

Node.js Backend Security: Password Hashing & JWT From Scratch (DIY)Securing user data in Node.js backend applications is paramount, especially where authentication and authorization are critical.

This article focuses on symmetric encryption, demonstrating how to implement robust security measures for password hashing and JSON Web Tokens (JWTs) using Node.js's built-in crypto module, avoiding external libraries. This DIY approach offers deep understanding and control over your application's security.

Secure Password Hashing

Password hashing converts a password into an irreversible string, safeguarding it even if the database is compromised. A basic hash with createHash is insufficient, as it yields identical results for identical inputs, making it vulnerable to rainbow table attacks. Initially:

import { createHash } from "crypto"

const password = "Test1234"

const hash = (text: string) => createHash('sha256')
  .update(text)
  .digest('hex')

// console.log('hash', hash(password))

To counter this, a "salt"—a unique, random string—is added to the password before hashing, ensuring distinct hashes even for identical passwords.

import { createHash, randomBytes } from "crypto"

const password = "Test1234"

const salt = randomBytes(16).toString('hex')

const hash = (text: string) => createHash('sha256')
  .update(text + salt)
  .digest('hex')

// console.log('salted hash', hash(password))

For robust password hashing and verification, Node.js's pbkdf2Sync and timingSafeEqual are preferred. pbkdf2Sync is a computationally intensive key derivation function that makes brute-force attacks impractical through numerous iterations. timingSafeEqual prevents timing attacks by ensuring comparison operations take a constant amount of time, regardless of input values.

Hashing with PBKDF2:

import { randomBytes, pbkdf2Sync } from "crypto";

const hashPassword = (password: string) => {
  const salt = randomBytes(16).toString('hex')
  return pbkdf2Sync(
    password,
    salt,
    100000, // Iterations
    64,     // Key length
    'sha512'
  ).toString('hex') + `.${salt}`
}

// console.log("Hashed Password:", hashPassword("Test1234"))

Verifying with PBKDF2 and timingSafeEqual:

import { pbkdf2Sync, timingSafeEqual } from "crypto";

const verifyPassword = (password: string, hashedPassword: string) => {
  const salt = hashedPassword.split(".")[1]
  const newHash = pbkdf2Sync(
    password,
    salt,
    100000,
    64,
    'sha512'
  ).toString('hex')

  return timingSafeEqual(
    Buffer.from(`${newHash}.${salt}`),
    Buffer.from(`${hashedPassword}`)
  )
}

// Example usage
// const myHashedPass = hashPassword("Test1234");
// console.log("Password match:", verifyPassword("Test1234", myHashedPass));

Crafting JSON Web Tokens (JWT) From Scratch

JWTs are used to securely transmit information. A JWT comprises a header, Payload, and Signature. JWT Creation:

import { createHmac } from "crypto";

type Payload = Record<string, any>;
type Header = {
  alg: string;
  type: string;
};

// Function to create the signature part of the JWT
const createSignature = (header: string, secret: string, payload: string) =>
  createHmac('sha256', secret)
    .update(`${header}.${payload}`)
    .digest('base64url');

// Function to sign and create the full JWT
const sign = (secret: string, payload: Payload) => {
  const header: Header = {
    alg: "HS256", // Hashing algorithm
    type: "JWT"
  };

  // Encode Header and Payload to Base64URL
  const encodedHeader = Buffer.from(JSON.stringify(header)).toString('base64url');
  const encodedPayload = Buffer.from(JSON.stringify(payload)).toString('base64url');

  // Create the signature
  const signature = createSignature(encodedHeader, secret, encodedPayload);

  // Combine parts to form the JWT
  return `${encodedHeader}.${encodedPayload}.${signature}`;
};

// Example Usage
// const myJwt = sign("Test1234", { _id: "user123", role: "admin" });
// console.log("Generated JWT:", myJwt);

Verifying and Decoding JWTs

Verifying a JWT involves re-creating its signature and comparing it to the token's original signature. If they match, the token is valid. Decoding the payload simply requires base64-decoding the payload segment. JWT Verification:

import { createHmac } from "crypto"; // Ensure createHmac is imported

// Function to verify a JWT
const verify = (token: string, secret: string): boolean => {
  const parts = token.split('.');
  if (parts.length !== 3) {
    throw new Error('Invalid token format');
  }

  const [encodedHeader, encodedPayload, encodedSignature] = parts;

  // Recreate the signature
  const newSignature = createHmac('sha256', secret)
    .update(`${encodedHeader}.${encodedPayload}`)
    .digest('base64url');

  // Compare the new signature with the provided signature
  if (newSignature !== encodedSignature) {
    throw new Error('Invalid signature');
  }

  // To decode the payload:
  // try {
  //   const decodedPayload = JSON.parse(Buffer.from(encodedPayload, 'base64url').toString('utf8'));
  //   console.log("Decoded Payload:", decodedPayload);
  // } catch (e) {
  //   console.error("Failed to decode payload:", e);
  // }

  return true; // Token is valid
};

// Example Usage with a generated JWT
// const myGeneratedJwt = sign("Test1234", { _id: "user123", role: "admin" });
// try {
//   console.log("JWT Verified:", verify(myGeneratedJwt, "Test1234"));
// } catch (e) {
//   console.error("JWT Verification failed:", e.message);
// }

Building security features directly with Node.js's crypto module offers deep understanding and control over your application's data protection. This DIY approach provides valuable insights into symmetric encryption, crucial for any backend developer. Remember, security is an ongoing process, and continuous learning is key to safeguarding user data effectively.

10 NPM packages all React Developers should know and use

Kinanee Samson — Fri, 13 Jun 2025 15:42:00 +0000

React, ah, doesn’t he get tired of it? No, I don’t. If you love using React, this is for you. If you don’t, hold my beer and take a seat because we are going in for quite the ride.

Now that we have that out of the way, I am going to talk about 10 absolutely amazing npm packages every React developer should know and use. A huge thumbs up to you if you already know them, and I would love to know how many you have used personally in your projects via the comment section.

If you'd rather watch video content, then here is the video for you on YouTube.

Framer Motion

Framer Motion is an animation library built for React. Framer Motion allows you to produce amazing animations with seamless effort. The animation API is very intuitive, and there is no shortage of excellent examples of what you could achieve in their documentation.

Framer Motion Animate elements via props like animate, initial, and exit. Values update automatically, with smart default transitions. There is built-in support for hover, tap, drag, and pan interactions: whileHover, whileTap for interactive states. Drag with constraints, momentum, and elastic boundaries. It also allows you to automate size/position changes with layout or layoutId for shared-element transitions. You can define reusable animation states and synchronize parent-child transitions.

I18next

I18next is a robust internationalization (i18n) library for JavaScript, widely used in React applications for multi-language support. It handles translations, plurals, date/number formatting, and more. The react-i18next package provides React-specific utilities for seamless integration.

Its core features are translation management by mapping keys to language-specific strings. Pluralization & Interpolation. It also supports dynamically inserted variables (e.g., {{count}} items). I18n can help us format dates, numbers, and currencies. It features a modular design by loading translations on demand (e.g., via HTTP, filesystem). It is also extensible and has plugins for language detection (browser, cookies), caching, etc.

React Apex Charts

React ApexCharts is a React wrapper for ApexCharts.js, enabling seamless integration of interactive, responsive charts into React applications. It supports 20+ chart types with extensive customization via SVG rendering, making it ideal for dashboards and data visualizations.

React ApexCharts charts feature diverse chart types. It supports line, bar, pie, donut, heatmap, radialBar, and candlestick charts, with mixed-chart combinations. It is highly interactive with built-in gestures like tooltips, zoom/pan, and event handlers (e.g., dataPointSelection for click events). It auto-renders charts when series or options props change, enabling real-time data sync. Charts adapt to container size; supports width="100%" for fluid layouts.

React-toastify

React-Toastify is a lightweight, MIT-licensed library for adding elegant toast notifications to React apps. It supports customizable, non-intrusive messages for user feedback (e.g., success/error alerts) with minimal setup.

There are predefined styles for default, success, error, warning, and info toasts, each with intuitive icons and color coding. It allows flexible positioning with six positions to choose from: top-left, top-center, top-right, bottom-left, bottom-center, bottom-right. Rich Customization with themes: light, dark, or colored (matches notification type). CSS Control: Style via inline styles, CSS classes, or SCSS variables (e.g., toastClassName, progressClassName). You can replace default icons or disable them. It also enables advanced interactions with swipe-to-close: draggable in any direction.

React Select

React Select is a highly customizable dropdown library for React, replacing native <select> elements with enhanced features like search, multi-select, async loading, and accessibility compliance. It simplifies complex selection workflows while offering deep customization via props and hooks.

It enables selection of multiple values with tags, search filtering, and customizable UI for added/removed options. You can fetch data dynamically from APIs (e.g., user lists), ideal for large datasets or real-time filtering, allowing developers to override default UI parts (options, dropdowns) with React components.

Xlsx

SheetJS (xlsx) in React Ecosystem SheetJS (xlsx) is a robust JavaScript library for reading, manipulating, and writing spreadsheet files (e.g., XLSX, CSV) in React apps. It supports 20+ file formats, enables data conversion between JSON/HTML/Excel, and works client-side or server-side.

Moment

Moment.js is a legacy JavaScript library for parsing, validating, manipulating, and formatting dates and times. Though now in maintenance mode (with modern alternatives like date-fns or Luxon recommended), it remains widely used for date handling in React apps due to its simplicity and rich feature set.

Moment allows us to convert strings/objects to dates with strict validation, display dates in 200+ locales, add/subtract time units with human-readable differences, work with time spans, and compare dates.

React Dropzone

React Dropzone is a lightweight, accessible library for handling file drag-and-drop and selection in React apps. It provides a seamless interface for file uploads with robust customization options, supporting both browser-native drag/drop and programmatic control.

With its awesome features, React Dropzone creates drop zones with visual feedback (e.g., border highlights) during file interactions. You can validate files by type (e.g., accept: "image/*"), size, or custom logic; you can extract metadata (name, size, type); and you can preview files (images, PDFs).

React Calendar Timeline

React Calendar Timeline is a powerful, customizable library for rendering interactive timeline views (e.g., Gantt charts, resource schedulers) in React. It supports drag-and-drop resizing, grouping, and time range navigation, ideal for project management, booking systems, and historical data visualization.

It allows us to display events on horizontal timelines with zoomable scales (minutes to years) and customizable time zones. Organize items into vertical lanes (e.g., teams, resources) with collapsible groups. Reschedule events via drag interactions (with callback support for validation), and you can render custom components per timeline item (e.g., tooltips, progress bars).

React Grid Gallery & React Image Lightbox

React Grid Gallery is a responsive image gallery component that displays images in a masonry-style grid, while React Image Lightbox provides a customizable lightbox viewer for image previews. Together, they create a seamless gallery and lightbox experience with features like zooming, navigation, and responsive layouts.

You can auto-adjust columns based on container width with customizable breakpoints, add tags to images and filter them dynamically, control aspect ratio, spacing, and hover effects, connect with lightbox libraries for image previews, and lastly, it supports lazy loading and virtualization for large galleries.

Now that was quite the ride. You have 10 packages all devoted React worshipers should absolutely use for their next projects. I’d love to hear your thoughts on the video and how many of them you’ve used before; do so using the comment section.

Why Go is finally poised to be the best backend Programming Language

Kinanee Samson — Mon, 09 Jun 2025 20:48:55 +0000

Go, the next level, dawg, in backend development. Go is arguably one of the best, if not the best, backend frameworks in my opinion. Go has got it all to be the OG framework for backend developers. However, it is not currently receiving the level of worship it deserves.

Ever since Go was released, there has been very little to no programming language that has offered features in a game-changing way that Go has done. Some developers will hear this and say, Maybe Rust is better.

Go has got it all to be the best thing that ever happened to backend development, and in today's post, I will go over 5 reasons why I believe that Go is the ultimate programming language for backend developers.

Go is incredibly easy to get started with: If you are a programmer with enough experience using at least one other programming language, you will find out that using Go for the first time is very easy. You just have to download and install the Go executable to begin building apps with Go. All this can be done in less than 5 minutes (I have a full video on that if you’d like to get started using Go). This makes it very easy for beginners to get started using the programming language. Even if it’s your first time building a backend, trust me, it will not be so much of a bumpy ride.
Elegant Syntax: Go has an angelic syntax that ensures that code is written in an imperative and procedural style, ensuring that code is written step by step. Go’s design centers on explicit control flow: loops, conditionals, direct state manipulation, and explicit error handling, a blessing and a curse. It also supports type inference, giving you an overall minimalistic and elegant code.
Speed: Go has incredible compile and run time speed. The reason for Go's speed is its minimalist language design. There is no complex inheritance or exceptions. The dependency management ensures that imports are resolved directly from source code (no header files) with a fast dependency graph resolution via the go.mod` file. Its compiler architecture is a single-pass compiler with limited abstract syntax tree (AST) rewrites. It focuses on fast compilation over aggressive runtime optimizations (by default). Go compiles to standalone native executables with an embedded garbage collector, scheduler, and runtime with no external dependencies (unless using CGO).
Concurrency Model: Go’s concurrency model is one of its game-changing features. Concurrency is not unique to Go; we already know this, but the way Go manages concurrency makes other languages drool. Go's concurrency model is indeed one of its most revolutionary features—a core design philosophy that fundamentally sets it apart from traditional threading models. The pillars of Go's concurrency are goroutines (not threads), which are lightweight: 2KB initial stack (grows/shrinks dynamically) vs. 1-2MB per OS thread compared to other traditional programming languages, and are managed by the Go runtime scheduled on OS threads (GOMAXPROCS workers), not the OS kernel. Then there are channels for communication with a shared memory. Go routines synchronize by passing data through type-safe pipes, which can be a bidirectional channel where send/receive operations synchronize execution automatically.
Real-World Application and Big Tech Support: There is no shortage of applications built with Go that are currently being used today; most notably are

Docker is a containerization platform. Which chose Go because of its native networking and concurrency support for efficient container orchestration.
Kubernetes is a container orchestration system that chose go because of how it manages distributed clusters at scale and simplifies complex orchestration logic.
Hugo, a static site generator that chose Go for its blazing-fast build times (e.g., 1ms per page) for large content sites. Netflix integrates Go for backend services to reduce streaming latency. Dropbox migrated performance-critical components from Python to Go for better scalability. SendGrid processes 500M+ daily emails using Go’s concurrency model .

No programming language comes close to what this bad boy can do. Is it the way it is easy to set up and get started writing Go code, or is it its beginner-friendly approach that makes it a breeze to start building applications with the language, or its unique syntax that has the benefits of years of developer experience? Show some love to Go and leave your thoughts below in the comment section

Implementing JWTs in Go

Kinanee Samson — Tue, 16 Jul 2024 07:51:30 +0000

Go is our favorite programming language for now and that’s for a good reason if you agree with me. Others might not think so but don’t worry it isn’t easy being a full-stack developer who knows how to use multiple programming languages especially one designed and developed by Google.

We can remember Angular and Google due to their unnecessary complexity, but trust me, Google learned from their mistake. Thus, Go is a very easy-to-use, elegant, and modern systems programming language. In today's post, we examine how we can implement JWT authentication and RBAC (Role Based Access Control) in our API as such we will cover the following

Project setup
Creating a token
Verifying a token

Project step up

Let’s set up a basic Go project, to do that we’ll need to create a new folder inside our projects directory.

mkdir go_jwt

Let’s navigate into that project directory and create a Go mod file.

cd go_jwt && go mod init ./go_jwt

Now we have our project set up correctly let’s install Gorilla Mux to help us set up a basic server.

go get -u github.com/gorilla/mux

Now we have gorilla mux installed let’s create a basic server

package main

import (
    "net/http"
    "github.com/gorilla/mux"
)

func main () {
    r := mux.NewRouter()
    http.handle("/", r)
    http.ListenAndServe(":8080", nil)
}

Create Token

We will implement a simple RBAC where only users who are admins can create a project and update its status. Meanwhile, other users can only view existing projects.

package main

// cont’d

type User struct {
    Type string    `json:"type"`
}
// cont’d

Now we have a struct to serve as a user let’s install the package to help us with JWTs.

go get -u github.com/golang-jwt/jwt/v5

Let’s go ahead and set up a helper function for creating a token.

package main

import (
    // cont’d 
    "github.com/golang-jwt/jwt/v5"
)

// cont’d

var secretKey = []byte("Test1234")

func CreateToken(type string) (string, error) {
    claims := jwt.NewWithClaims(jwt.SigningMethodHS256, jwt.MapClaims{
             // Issuer
        "aud": user_type,                        // Audience (user role)
        "exp": time.Now().Add(time.Hour).Unix(), // Expiration time
        "iat": time.Now().Unix(),                // Issued at
    })
    tokenString, err := claims.SignedString(secretKey)

    if err != nil {
        fmt.Printf("Error in creating token: %v\n", err)
        return "", err
    }

    return tokenString, err
}

The function creates a new JSON Web Token (JWT). The function accepts a string argument named type. This argument represents the type of token being created (e.g., "admin", "user"). The jwt.NewWithClaims function creates a JWT with certain claims for us. The claims specify the following information:

aud: This represents the audience for the token, set to the user type.
exp: This defines the expiration time for the token, set to one hour from the current time.
iat: This specifies the time at which the token was issued.

The function uses the SignedString method to sign the JWT claims with a secret key (secretKey). This signing process ensures the integrity of the token. If there's an error during signing, the function prints an error message and returns an empty string along with the error. If everything is successful, the function returns the signed JWT token string as well as a nil error value.

Now we'll create a new token whenever we create a new user account

package main

import(
    "fmt"
)

func CreateUser (type string) func(http.ResponseWriter, *http.Request) {
    return func (w http.ResponseWriter, r *http.Request) {
    // Create user somehow 
    result := db.Create(&user)
        if result.Error != nil {
            http.Error(w, result.Error.Error(), http.StatusInternalServerError)
            return
        }
        token, err := CreateToken(type)
        if err != nil {
            http.Error(w, err.Error(), http.StatusInternalServerError)
            return
        }
        fmt.FPrintf(w, token);
     }
}

The code snippet above builds upon the previous CreateToken function by defining a new function called CreateUser within the main package. The CreateUser function relies on the CreateToken function to generate a token after a user is created. The CreateUser function returns another inner function that handles HTTP requests. Upon successful user creation, it calls the CreateToken function (from the previous codebase) to generate a token for the newly created user. The type argument passed to CreateUser is forwarded to CreateToken. If there's an error creating the token, it returns an internal server error response to the client. If both user creation and token generation are successful, it writes the generated token (from CreateToken) to the HTTP response.

Verifying a token

Now let’s see how we can verify a token to ensure that the user with the right role can take the right action;


func VerifyToken(tokenString string) (*jwt.Token, error) {
    // Parse the token with the secret key
    token, err := jwt.Parse(tokenString, func(token *jwt.Token) (interface{}, error) {
        return secretKey, nil
    })

    // Check for verification errors
    if err != nil {
        return nil, err
    }

    // Check if the token is valid
    if !token.Valid {
        return nil, fmt.Errorf("invalid token")
    }

    // Return the verified token
    return token, nil
}

func IsUserAdmin(token *jwt.Token) bool {
    // get token from request headers
    claims := token.Claims

    audClaims, audienceErr := claims.GetAudience()

    if audienceErr != nil {
        return false
    }

    userRole := audClaims[0]

    res := userRole == "admin"

    return res
}

We have added two helper functions one for verifying our token and another for checking if the user is an admin or not. functions VerifyToken and IsUserAdmin work together to validate and extract information from tokens generated by the CreateToken function.

VerifyToken: This function takes a token string (received from a client) and verifies its authenticity using the same secret key used for signing in CreateToken. If the token is valid, it returns the parsed token object. If the token is invalid or there are errors during parsing, it returns an error.

The IsUserAdmin: This function assumes a verified token is provided as input (from VerifyToken). It extracts the audience information (claims) from the token and checks if the user role (audience) is set to "admin". If the user role is "admin", it returns true. Otherwise, it returns false.

In essence, these functions ensure that only authorized users (with valid tokens) can access specific functionalities (like admin actions) within your application.


func CreateProject() func(http.ResponseWriter, *http.Request) {
    return func(w http.ResponseWriter, r *http.Request) {
        tokenString := r.Header.Get("Authorization")
        token, err := helper.VerifyToken(tokenString)

        if err != nil {
            http.Error(w, err.Error(), http.StatusUnauthorized)
            return
        }

        isAdmin := helper.IsUserAdmin(token)

        if !isAdmin {
            http.Error(w, "Unauthorized", http.StatusUnauthorized)
            return
        }
        // create project somehow
        result := db.Create(&project)

        if result.Error != nil {
            http.Error(w, result.Error.Error(), http.StatusInternalServerError)
            return
        }

        json.NewEncoder(w).Encode(project)
    }
}

The code snippet above defines a function named CreateProject that serves as a route handler for creating projects. The function returns another function that acts as the actual handler for project creation requests. The inner function retrieves the authorization token from the request header then It calls the VerifyToken function to validate the retrieved token string. If the token is invalid or there are errors during verification, it returns an unauthorized error message (HTTP status code 401) to the client. If the token is valid, it calls the IsUserAdmin function (likely also from the helper package) to check if the user making the request has admin privileges based on the token's audience information.

If the user is not an admin, it returns an unauthorized error message (HTTP status code 401) to the client. If both token verification and admin checks pass, it attempts to create a new project. If there's an error creating the project, it returns an internal server error message (HTTP status code 500) to the client. Upon successful project creation, it encodes the newly created project object as JSON and sends it back in the HTTP response.

Let’s add another function that allows any user with a token to retrieve a list of projects.


func GetProjects() func(http.ResponseWriter, *http.Request) {
    return func(w http.ResponseWriter, r *http.Request) {
        tokenString := r.Header.Get("Authorization")
        _, err := helper.VerifyToken(tokenString)
        if err != nil {
            http.Error(w, err.Error(), http.StatusUnauthorized)
            return
        }

        // get project some how
        db.Find(&projects)

        json.NewEncoder(w).Encode(projects)
    }
}

Manage Your Go Database Like a Pro with GORM

Kinanee Samson — Fri, 12 Jul 2024 12:08:43 +0000

No matter how much we say SQL is better than NoSQL, we can all agree that writing SQL by hand is not much fun, especially where you have to do joins and the rest. The need to solve this annoying issue has led to the development of several different ORMs (Object Relational Mapper) that sit as an abstraction layer on top of our database.

They provide developers a way to interact with data typically an SQL-compatible database using Objects and code instead of writing a bunch of SQL code. All programming languages have their ORM but in today's post we will take a brief look at GORM an ORM that is developed for Go or Go Lang as it is commonly called hence we are going to consider the following talking points;

Features of GORM
Installation and setup
Models
Add data
Read Data
Update Data
Delete Data

Features of GORM

GORM is a Full-Featured and developer-friendly ORM built with out-of-the-box support for all types of associations, and hooks that you can listen to extend the behavior of your models further. GORM is written in Go and supports multiple SQL-based databases like MySQL, SQL-Lite, Postgre SQL, and OracleDB. GORM has built-in support for Transactions, nested transactions, save points, and rollback to saved points. GORM allows us to run Batch Insert, FindInBatches, Find/Create with Map amongst others, let's see how we can install and setup GORM in a Go project.

Installation and Setup

To install GORM you need to run the following command;

go get -u gorm.io/gorm

Next, you need to install a driver compatible with your SQL database and as such we will install the driver for MySQL with the following command.

go get -u gorm.io/driver/mysql

Now we have GORM and a compatible SQL driver installed let's connect to a MySQL database.

package main

import (
  "gorm.io/driver/mysql"
  "gorm.io/gorm"
)


func main () {
    db, err := gorm.Open(mysql.New(mysql.Config{
        DSN: "root@tcp(localhost:3306)/test?charset=utf8mb4&parseTime=true",
    }))

    if err != nil {
        panic("failed to connect database")
    }
}

Models

Models serve as a way to map a struct in your Go program to a database table. Each Model corresponds to a table in your database, while the keys on the struct correspond to columns on that table. While the values for each key serve as a row in your database. Let's create a User struct that will correspond to a user table.

import "gorm.io/gorm"

type User struct {
    gorm.Model,
    Name string `json:"name"`
    Email string `json:"email"`
}

You can observe that each key on the struct has a type and that type will correspond to the type for that particular value on our SQL table. We can use other types like uint and bool. We can do things like specify foreign keys and unique columns.

import "gorm.io/gorm"

type User struct {
    gorm.Model,
    Name string `gorm:"type:varchar(20)" json:"name"`
    Email string `gorm:"unique;not null" json:"email"`
}

You can embed the gorm.Model struct into your model struct. This adds commonly used fields like:

ID: A unique identifier for each record (usually the primary key).
CreatedAt: Automatically set to the current time when a record is created.
UpdatedAt: Automatically updated whenever a record is modified.
DeletedAt (optional): Used for soft deletes (marking records as deleted without physically removing them).

Add data

Let’s see how we can add data to our database using the model we just created.


// cont’d

func CreateUser (db *gorm.DB, name string, email string) User {
    newUser := User{
        Name: name,
        Email: email,
    }
    result := db.Create(&newUser)
    if result.Error != nil {
        panic("error creating user")
    }
    return newUser
}

user := CreateUser(db, "sam","sam@gmail.Com")

The code above defines a function called CreateUser that takes a database connection (db), a name (name), and an email (email) as arguments and returns a User model. We can also create multiple users easily.


// cont’d

func CreateUsers(db *gorm.DB) []User {
    newUsers := []*User{
        {  Name: “jay”, Email: “jay@gmail.Com”  },
        {  Name: “joe”, Email: “joe@gmail.Com”  },
    }
    result := db.Create(&newUsers)
    if result.Error != nil {
        panic("error creating user")
    }
    return newUsers
}

users := CreateUsers(db)

Read Data

Now we’ve created some data let’s see how we can read those data we just added.

// get single record

func GetUser(db *gorm.DB, id uint) User {
    user = User{
        ID: id
    }

    result := db.First(&user)
    if result.Error != nil {
        panic("Error fetching data")
    }
    return user
}

user := GetUser(db, 1)

The code snippet above shows how to retrieve a single user, what if we are interested in getting multiple users?

// get single record

func GetUsers(db *gorm.DB) []User {
    var users []User
    result := db.Find(&users)
    if result.Error != nil {
        panic("Error fetching data")
    }
    return users
}

users := GetUsers(db);

Update Data

Now we’ve seen how to read our data let’s see how we can update our data.

func UpdateUser(db *gorm.DB, id uint, name string) user {
    user := User{
        ID: id,
    }
    result := db.Find(&users).Update("name", name)
    if result.Error != nil {
        panic("Error fetching data")
    }
    return user
}

users := UpdateUser(db, 1, "sam");

If we ware interested in updating multiple columns for a particular data we can use the following approach;

func UpdateUser(db *gorm.DB, id uint) user {
    user := User{
        Name: "phill"
        Email: "phill@gmail.com"
    }
    result := db.Where("id = ?", id).Updates(&user)
    if result.Error != nil {
        panic("Error fetching data")
    }
    return user
}

users := UpdateUser(db, 1);

Delete Data

Now let’s see how we can delete a record or multiple records from our database


// delete single record

func DeleteUser(db *gorm.DB, id uint) {
    result := db.Delete(&User{}, id)
    if result.Error != nil {
        panic("Error fetching data")
     }
}

DeleteUser(db, 1)

// delete multiple records

func DeleteUsers(db *gorm.DB, ids []int{}) {
    result := db.Delete(&User{}, ids)
    if result.Error != nil {
        panic("Error fetching data")
     }
}
DeleteUsers(db, {1, 4, 10})

That’s going to be all for this one guys, I hope you found this useful leave your thoughts in the comments section below.

Building a Basic Auth System With Go and MySQL

Kinanee Samson — Tue, 09 Jul 2024 06:18:31 +0000

Go is one of the few languages that have caught my attention in the past. So in our quest to do more, we will build a simple auth system using Go. I am starting to like writing Go because the syntax is quite elegant and helps to reduce the overall amount of code I have to write.

Go is also very fast and as such it would make so much sense for us to build a backend service with Go, in today’s post we will explore how to build a simple authentication system using Go and as such we will cover the following talking points;

Project Setup
Integration with MySQL and GORM
Create Account
Login

Project Setup

The first thing we need to do is to set up a new project and for that, we’ll need to create a new directory “go_auth_app”. We’ll navigate into and run the go init command inside the directory.

mkdir go_auth_app

Now we need to navigate into the go_auth_app and run the go mod init command

cd go_auth_app && go mod init

Let’s install the dependencies that we will need for this project starting with Gorilla Mux Http amongst others.

go get -u github.com/gorilla/mux

Now let's create a server.go file in our project root. This file will contain our server so let's create a basic server.

package main

import (
  "net/http"
  "github.com/gorilla/mux"
)

func main () {
  r := mux.NewRouter()
  http.Handle("/", r)
  http.ListenAndServe(":8080", nil)
}

Integration with MySQL and GORM

Now we need to install an ORM that will simplify the process of interacting with our database and for that, we will use GORM. Let's go ahead and add that dependency to our project.

go get -u gorm.io/gorm

Now we have GORM installed we need to install a compatible SQL driver.

go get -u gorm.io/driver/mysql

Now let's create a model for a user. To do that we need to create a new folder. models and inside this folder, we will create a new file user.go let's add the following code to the user.go.

// go_auth_app/models/user.go

package models

import (
  "gorm.io/gorm"
)

type User struct {
  gorm.Model
  Name string
  Age uint
  Email string
  Phone string
  Password string
}

Create User

Now that we have our model set up, we need to create a controller package to export some functions for interacting with our data. Let's make a folder called controllers and inside we'll create a new go file user.go.


// go_auth_app/controllers/user.go

package controller

import (
  "net/http"
  "encoding/json"

  "go_auth_app/helper"

  "gorm.io/gorm"
  "golang.org/x/crypto/bcrypt"
)

func ReturnCreateUser(db *gorm.DB) func(http.ResponseWriter, *http.Request) {
    return func(w http.ResponseWriter, r *http.Request) {
        user, err := helper.ParseRequestBody(r)

        if len(err) > 1 {
            http.Error(w, err, http.StatusBadRequest)
            return
        }

        toHash := []byte(user.Password)
        hashedPassword, hashErr := bcrypt.GenerateFromPassword(toHash, 14)
        if hashErr != nil {
            {
                http.Error(w, "Failed to hash password", http.StatusInternalServerError)
                return
            }
        }

        createdUser := models.User{Name: user.Name, Age: user.Age, Email: user.Email, Password: string(hashedPassword)}
        result := db.Create(&createdUser)

        if result == nil {
            http.Error(w, "Failed to create user", http.StatusInternalServerError)
            return
        }
        json.NewEncoder(w).Encode(createdUser)
    }
}

The code snippet above implements a function called ReturnCreateUser in the controller package. Here's a breakdown of what the code does: This function is designed to handle creating a new user in the system. It takes the database connection (db of type *gorm.DB) as input and returns another function that acts as the actual route handler.

Inside the route handler function, we call ParseRequestBody on the helper package. The ParseRequestBody function is used to extract user data (name, age, email, password) from the request body (likely sent as JSON data). If there's an error parsing the request body (length of err string is greater than 1), then we send a bad request (400) error response.

Next, we hash the user's Password by converting the user's password (assumed to be a string) to a byte slice ([]byte). We use the bcrypt.GenerateFromPassword function to securely hash the password. Bcrypt is a popular hashing algorithm for passwords. The function takes the password bytes and a cost factor (14 in this case) to control the hashing intensity. If there's an error during hashing (hashErr is not nil), it sends an internal server error (500) response with a message and exits. Run the following command to install the bcrypt library.

go get golang.org/x/crypto/bcrypt

Then we create a new models.User object by populating the user object with the parsed data (name, age, email) and the hashed password converted back to a string. Then we save the user to the Database using the database connection (db). We call the Create method on the database connection, passing the createdUser object as an argument. If the Create operation fails (result is nil), it sends an internal server error (500) response with a message and exits. If successful, we use the json.NewEncoder to encode the newly created user object (including the ID generated by the database) back into JSON format and sends it as the response. Now we need to implement the helper package and the ParseRequestBody function.


// go_auth_app/helper/helper.go

package helper

import (
    "encoding/json"
    "io"
    "net/http"
)

type Payload struct {
    Name     string `json:"name"`
    Age      int    `json:"age"`
    Email    string `json:"email"`
    Password string `json:"password"`
}

func ParseRequestBody[S string, P Payload](req *http.Request) (P, S) {
    body, err := io.ReadAll(req.Body)

    var payload P
    if err != nil {
        return payload, "Error reading request body"
    }
    err = json.Unmarshal(body, &payload)

    if err != nil {
        return payload, "Invalid JSON format in the request body"
    }

    return payload, ""

}

The code snippet above defines a package named helper containing functionalities for parsing request bodies in a Go web application. The package imports the necessary libraries:
encoding/json for working with JSON data.
io for reading data from the request body.
net/http for accessing request information.

Then we define a struct named Payload. This struct represents the expected format of the data sent in the request body. It has fields for Name, Age (int), Email, and Password. The json tag specifies the corresponding JSON field names for each struct field during marshaling and unmarshalling. This function is designed to parse the request body and extract the user data. It takes an http.Request object as input and returns two values: P An instance of the Payload struct populated with the parsed user data (name, age, email, password). S: A string representing any error message encountered during parsing.

Now let's put all of this together in our server.go file


// go_auth_app/server.go

package main

import (
    "log"
    "net/http"

    "github.com/gorilla/mux"
    "gorm.io/driver/mysql"
    "gorm.io/gorm"

    "go_auth_app/controller"
    "go_auth_app/models"
)

func main() {
    r := mux.NewRouter()

    db, err := gorm.Open(mysql.New(mysql.Config{
        DSN: "root@tcp(localhost:3306)/test?charset=utf8mb4&parseTime=true",
    }))

    if err != nil {
        panic("failed to connect database")
    }

    // Migrate the schema
    db.AutoMigrate(&models.User{})

    r.HandleFunc("/user", controller.ReturnCreateUser(db)).Methods("POST")
    r.HandleFunc("/login", controller.ReturnLoginUser(db)).Methods("POST")

    http.Handle("/", r)
    log.Println("server started on port 8080")
    http.ListenAndServe(":8080", nil) // Start server on port 8080
}

Login

The code snippet in the main package ties together the previously explained controller and helper packages to create a functional web server application. Now let's add a new handler function to enable the user to log in to their account. We'll edit the controller/user file to add the handler function to enable the user login.

// go_auth_app/controller/user.go

package controller

// cont'd

func ReturnLoginUser(db *gorm.DB) func(http.ResponseWriter, *http.Request) {
    return func(w http.ResponseWriter, r *http.Request) {
        payload, err := helper.ParseRequestBody(r)

        if len(err) > 1 {
            http.Error(w, err, http.StatusBadRequest)
            return
        }

        var user models.User
        db.Where("email =?", payload.Email).First(&user) // Find user by email
        // if the user with that email does not exist throw an error
        if user.Email == "" {
            http.Error(w, "User not found", http.StatusNotFound)
            return
        }

        // compare their passwords
        compareErr := bcrypt.CompareHashAndPassword([]byte(user.Password), []byte(payload.Password))
        if compareErr != nil {
            http.Error(w, "Invalid credentials", http.StatusUnauthorized)
            return
        }
        // if the passwords match, return the user
        json.NewEncoder(w).Encode(user)
    }
}

That's going to be all for now, I hope you found this useful and leave your thoughts on the article and building a web server with Go in the comment section below.

Building a basic HTTP server with Go

Kinanee Samson — Wed, 26 Jun 2024 21:10:47 +0000

Go or Go lang as it’s commonly referred to is a wannabe system’s programming language originally designed and developed at Google, it’s aim was to be the ultimate replacement for C/C++. Go incorporates modern programming language features like type inference and generics while presenting itself in an effortless and minimalist style, Go has borrowed concepts from Typescript, Javascript, and Python.

Go has been used to buy some awesome projects like the open-source Pocketbase server amongst others and in today’s post we will expo how to set a basic REST server with Go and as such we will cover the following talking points;

Project Setup
Create a basic Server with net/http
Parsing request headers
Parsing request query params
Parsing request body

Project Setup

The first thing to do is to ensure you have the Go compiler installed on your machine, if you don’t have it installed then you can download and install the Go executable and ensure that the GoPath is correctly configured. Now let’s set up a Go project, first navigate into your projects directory to create a Go project.

mkdir go_server && go mod init

This will create a go package for us, now we need to navigate into the newly created folder and create a new go file, index.go. This file will house the content of our server.

cd go_server

Create a basic Server with net/http

Now we have our project all set up let's add the following code to our index.go file,

package main

import (
  "fmt"
  "log"
  "net/http"
)

func sayHello(w http.ResponseWriter, req *htt.Reques) {
  fmt.FPrintf(w, "Hello World")
}

main () {
  http.handleFunc("/", sayHello);
  log.Println("Server listening on port", 8080);
  http.ListenAndServer(":8080", nil);
}

The code snippet above demonstrates how to set up a basic HTTP server with Go, we need to declare a package for the current Go module, and then we import a bunch of packages from the standard Go library, the fmt package is responsible for formatted printing and input/output operations. The log package in the Go standard library provides basic logging functionality. The net/http package in the Go standard library provides a powerful and comprehensive set of tools for building both HTTP servers and clients in Go.

The sayHello function is a route handler function, it accepts two arguments, the first is the response object, while the second is the request object. We use the FPrintf function to write a message to the response body, this also effectively ends the request. Inside our main function, we register the sayHello function as a handler for the / route. Then we use the Println function from the log package to notify us when our server starts up. Then we use the ListenAndServe method on the http package to create a server on port 8080.

The ListenAndServe method accepts two arguments, The first is a string that specifies the network address and port on which the server will listen for incoming HTTP requests. It's typically formatted as "hostname:port" or just ":port". The second argument is an optional interface that can be used to customize the overall request-handling behavior of the server. In most cases, you'll leave this argument as nil and rely on specific handler functions registered using http.HandleFunc or other routing mechanisms.

Parsing request headers

There are times when you are interested in parsing the request and that is standard when building an API so in the next snippet we'll see how we can do just that;

package main

// cont'd

var headers map[string]string

func sayHello(w http.ResponseWriter, req *http.Request) {

    for key, values := range req.Header {
        headers[key] = values[0]
    }
        log.Println(headers) // do h
    fmt.Fprint(w, "Hello, World!")
}

// cont'd

main () {
  headers = make(map[string]string)
  // cont'd
}

First, we create a new map object header to store the contents of the request headers, inside our main function we initialize the headers to be an empty map object. Inside the sayHello function we loop through the request headers and for each key on the request header we create a similar key on the headers map object and its value is the value of the key on the request header then we print the headers object to the console.

Parsing Request Query Parameters

To parse the request query parameters we just need to add a few extra lines of code to our handler function;

// cont'd

func sayHello(w http.ResponseWriter, req *http.Request) {

    // cont'd

    // Get all query parameters as a map
    queryParams := req.URL.Query()

    // Access specific parameters
    name := queryParams.Get("name")

    log.Println(name)

    // cont'd
}

// cont'd

We use the req.URL.Query() method of the http.Request object to access the query string parameters as a url.Values map. Then queryParams.Get("name") retrieves the value for the "name" parameter as a string or an empty string if not present.

Parsing request body

package main

import (
  "encoding/json"
  "io"
  // cont'd
)

type Payload struct {
    Id string
}

// cont'd

func sayHello (w http.ResponeWriter, req http.Request) {
        // cont'd
      body, err := io.ReadAll(req.Body)

    if err != nil {
        log.Println("Error reading request body:", err)
        w.WriteHeader(http.StatusBadRequest)
        fmt.Fprintf(w, "Error reading request body")
        return
    }

    var payload Payload

    err = json.Unmarshal(body, &payload)

    if err != nil {
        log.Println("Error parsing JSON body:", err)
        w.WriteHeader(http.StatusBadRequest)
        fmt.Fprintf(w, "Invalid JSON format in request body")
        return
    }

    log.Println("payload", payload.Id)

       // cont'd

}

// cont'd

We have imported two new packages encoding/json and io. Next, we create a struct to serve as a type for the request body. Inside our handler function we call io.ReadAll and pass the req.Body as an argument to the ReadAll method we just called.

This method returns two values we can destructre out, the request body and an error object. If there's an error we notify the user and end the request, otherwise we create a payload variable of type Payload and call json.UnMarshall and pass the body as the first argument to it and a pointer to the payload variable we just created. If there’s an error while trying to convert the body of the request we notify the user that their JSON is invalid then we end the request as a bad request, otherwise, we just log out the Id of the payload and continue with the rest of the code.

File Upload with Google Cloud Storage and Node.js

Kinanee Samson — Thu, 20 Jun 2024 23:26:02 +0000

With the data collection rate currently going through the roof, the chances that you will build an app requiring users to upload one or multiple files will also be through the roof. There are many solutions to this problem, there are tons of services out there focused on making this process as smooth as possible. We have names like Firebase storage, Supabase storage, and Cloudinary.

However, in today's post, we will consider Google Cloud Storage (GCS), a cloud storage service provided by Google via GCP, spoiler alert, this is what powers Firebase Storage underhood, Google Cloud Storage is a Cloud managed service for storing unstructured data. You can store any amount of data and retrieve it as often as you like

In today's post, we will see how we can handle file uploads to Google Cloud Storage using Node.js, and here are the following talking points;

• Project Setup
• Build an express server
• processing uploaded files with Multer
• Handling file upload with GCS

Project Setup

The first thing you need to do is to head over to Google Cloud and register a new account if you do not already have one. If you do then you're part of the bandwagon of developers with unfinished projects. If you have registered successfully or you have an account the next thing is to navigate to the cloud storage on your Google cloud console. We need to create a new bucket, take note of the name of this bucket we'll use it later. Now we need to set up a Node.js project open up your terminal and navigate into your projects directory, create a folder to serve as the current project directory say, gc_node and navigate into the newly created folder.

Now we need to generate a package.json file.

npm init --y

Now we are going to install some dependencies for the project with the following command.

npm i express multer @google-cloud/storage cors nodemon

These are the dependencies we'll need to get our server up and running now let's create a basic express server. This will serve as the entry point for processing files stored on GCS. Inside the gc_node directory let's make a new folder, src which will house all of our working files.

gc_node
├── src                  
│   ├── index.ts

Now let's open up our index file and edit it accordingly;

const express = require('express');
const cors = require('cors');

const app = express()

app.use(cors())

app.listen(3000, () => {
  console.log('server running on port 3000');
});

Let's add the following script to our package.json file.

{
"scripts": {
    "dev": "nodemon ./src/index.js",
    "start": "node ./src/index.js"
  },
}

from the terminal, we need to run the dev command to serve our project locally;

npm run dev

You'll see that our project is running on port 3000 if everything is done correctly. Now let up a route for processing uploaded files.

//  gc_node/index.ts

const multer = require('multer');
const upload = multer({ storage: multer.memoryStorage() });

// cont'd

app.post('/upload', upload.array("images"), (req, res) => {
  const imageMimeTypes = ["image/png", "image/jpeg", "image/jpg", "image/svg"];

  const files = req.files as any[];

  files.forEach((file) => {
    const mimeType = imageMimeTypes.find((mT) => mT === file.mimetype);

    if (file && !mimeType) {
      return res.status(400).json({ message: "Only images allowed!" });
    }
  });

// more on here later

})

We have used the multer middleware to parse any files sent along with the request and we are expecting to receive an array of files. We'll define a helper function that will process all uploaded files before we send them off to Google.

//  gc_node/index.ts
boostrapFile(file) {
    const key = crypto.randomBytes(32).toString("hex");

    const [, extension] = file.originalname.split(".");

    const uploadParams = {
      fileName: `${key}.${extension}`,
      Body: file.buffer,
    };

    return { uploadParams, key, extension };
  }

// .... continued

In summary, the function above takes a file object, generates a unique key, creates a new filename with the key and original extension, prepares the file data for upload, and returns all the necessary information for upload processing. Let's upload our file now;

//  gc_node/index.ts

import { Storage } from "@google-cloud/storage";

const storage = new Storage({
  projectId: process.env.GOOGLE_PROJECT_ID,
  keyFilename: path.join(__dirname + "../key.json"),
});

async function uploadFile(
    bucketName: string,
    buffer: Buffer | string,
    destFileName: string
  ) {
    const file = await storage.bucket(bucketName).file(destFileName);
    await file.save(buffer);
    const message = `file uploaded to ${bucketName}!`;
    const [publicUrl] = await file.getSignedUrl({
      expires: new Date("12/12/3020"),
      action: "read",
    });
    return { message, publicUrl };
  }

// cont'd

The code snippet above defines an asynchronous function called uploadFile that uploads a file to Google Cloud Storage (GCS) and potentially returns a publicly accessible URL. First, we import the Storage class from the @google-cloud/storage library, which provides functionalities to interact with GCS. Then we create a new Storage instance using configuration options. It retrieves the projected for the project on Google Cloud from the environment variable process.env.GOOGLE_PROJECT_ID and uses the path path.join(__dirname + "../key.json") to specify the location of a service account key file. This key file is required for GCS authentication.

The uploadFile function accepts three parameters, the first is the name of the bucket we want to upload the file to, and the second is the array buffer which stores the content of the file we want to upload, the third argument is the name for the file. Inside the function, we create a reference to the file object within the specified bucket using the destFileName. Then we call await file.save(buffer); This asynchronous call uploads the file data (either from the buffer) to the GCS file object. Next, we create a success message indicating the file was uploaded to the specified bucket. The next line retrieves a publicly accessible URL for the uploaded file. It uses the getSignedUrl method with configuration options; expires sets an expiration date for the URL (here, a very distant date "12/12/3020"), and action specifies the allowed action on the URL, which is set to "read" for public read access.
Then return an object containing the success message and the public URL (if generated). We will use this function inside our route handler function to help us upload the file to GCS.

//  gc_node/index.ts


// cont'd

app.post('/upload', upload.array("images"), (req, res) => {
  // cont'd
  const files = req.files as any[];
  const responses = []

  // cont'd
  for (const file of files) {
      const {
        uploadParams: { Body },
        extension,
        key,
      } = storageService.boostrapFile(file);
      const response = await uploadFile(
        process.env.BUCKET_NAME,
        Body,
        `/photos/${extension}.${key}`
      );
      response.push(response)
    }
    console.log(responses);
    return res.json(responses);
})

This route handler allows uploading multiple files, prepares data for each file using the boostrapFile function, and then uploads each file to GCS using the uploadFile function. Finally, it returns a JSON response containing information about each uploaded file.

You can go ahead and use this next time you want to set up uploads to GSC, what are your thoughts on the post? Do you personally use GCS for handling file storage? Or have you at any point in time used GCS? What was the experience like and what are your thoughts on using GCS? I would like to know all these and more, so leave your thoughts below using the comment section. I hope you found this useful and I will see you in the next one.

Running Advanced MongoDB Queries In TypeORM

Kinanee Samson — Wed, 12 Jun 2024 00:22:17 +0000

TypeORM is a Javascript/Typescript ORM that works well with SQL databases and MongoDB, which is built to run in the browser, on the server aka Node.JS, Expo, and every single execution context that utilizes Javascript. TypeORM has excellent support for Typescript while allowing newbies to use it with Javascript.

TypeORM sits as a layer on top of our application and allows us to build our application using Object-oriented programming principles when interacting with our data you can check the documentation to get more information about what TypeORM is. In today's post, we will go over how we can run some complex/advanced queries with TypeORM while working with MongoDB as our Database, and as such we will cover the following;

Find Where Object property is equal to value
Find where the field is in an array
Find where value is a regexp
Find where field A and field B is equal to a value
Find where field A is equal to value or field B is equal to another value
Paginating your queries.

Find where object property is equal to a value

Let's say we have an Entity with the following structure

@Entity()
class User {
  @Column()
  first_name: string;
  @Column()
  last_name: string;
  @Column()
  address: string;
  @Column()
  standIn: {name: string;phone: string};
  @Column()
  roles: string[]
}

Let's say we want to fetch all the users who have a standIn with the name of John. This is a simple query to run, so we would write the query as thus;

import {AppDataSource } from '../app-data-source';
import {User} from '../entity/user';

const getUsersByStandIn = (standIn) => {
  return AppDataSource.mongoManager.find(User, {
    where: {
      "standIn.name": {$eq: standIn}
    }
  })
}

Find where the field is in an array

Let's say we want to run a query to get all the users with a certain role, say all the contractors.

const getUserByPrevillage = (role) => {
  return AppDataSource.mongoManager.find(User, {
    where: {
      role: {
        $in: [previllage]
      }
    }
  })
}

Running Regular Expression Checks

Sometimes we might want to run a regular expression check, say we want to find a user by their street address, however, we are not sure that the input string will be the complete address, the user might only enter their street Name or house number so this is what the query will look like;

const findUserByAddress = (address) => {
  const regexp = new RegExp(address);
  return AppDataSource.mongoManager.find(User, {
    where: {address}
  })
}

And that's all it takes to run regular expression checks.

Find where field A and field B is equal to a value

Sometimes we might want to run a query where we check that a particular field is equal to a value and another field is equal to another value, say we want to get a user by their email and phone number, the query we will write will look like this;

const findUserByEmailAndPhone = (email, phone) => {
  return AppDataSource.mongoManager.find(User, {
    where: {
      email: {$eq: email},
      $and: [{phone: {$eq: phone }}]
    }
  })
}

Find where field A is equal to value or field B is equal to another value

There are also scenarios where we want to check that a field is equal to a value or another field is equal to another value, say we want to fetch a user by their email or phone number, and the query for such an operation would look like this;

const findUserByEmailOrPhone = (email, phone) => {
  return AppDataSource.mongoManager.find(User, {
    where: { 
      email: {$eq: email },
      $or: [{phone: {$eq: phone }}]
      },
  })
}

Paginating your queries.

Lastly, let's look at how we can paginate our queries, this is very important for most applications because fetching all the documents in a collection might be computationally expensive and as such we will need to reduce the amount of data we are trying to retrieve at each point in time. This is my approach to pagination;

const getUsers = (limit = 5, page = 1) => {
  return AppDataSource.mongoManager.find(User, {
    take: limit,
    skip: (page - 1) * limit
  })
}

This is a very simple but very powerful approach, as it handles both forward-tracking and backward-tracking. That's going to be it for this post, what are your thoughts on the queries we addressed, do you have a different way of doing things, I would gladly appreciate your feedback and your experience working with TypeORM and MongoDB. Use the comment section to express your thoughts and I hope you found this useful.