DEV Community: Daniel Boll

Unlocking the Power of Native Modules in JavaScript: An Introductory Guide

Daniel Boll — Fri, 29 Sep 2023 02:29:35 +0000

You may also read this blog post in my blog for a better experience

Disclaimer ⚠️
The topic of creating native Node.js modules and working with C++ bindings is intricate,
and while great care has been taken to ensure the accuracy of this blog post, there might be some
inadvertent errors or oversights. The world of native modules is complex, and the documentation can sometimes be heavy to digest.
If you notice any inaccuracies or have suggestions for improvement, please feel free to reach out to me directly or address it
in the comments section below. Your input is highly valuable, not just for me but for everyone else looking to expand their knowledge in this area.
🙇‍♂

Introduction 🚀

Welcome to the first installment of our series, "Going Native on JavaScript!" Whether you're a seasoned developer or
just getting started with JavaScript, you've likely encountered modules — reusable pieces of code that can be imported into your projects.
But have you ever heard of native modules?

Native modules are special types of modules that allow you to execute code written in languages like C++ directly within your Node.js application.
These modules can offer significant performance benefits and access to lower-level APIs that are not readily available in JavaScript.
So why aren't native modules as commonly discussed as their pure JavaScript counterparts?

In this post, we'll delve deep into the world of native modules. We'll explore what they are, why they're critical for certain applications,
and how you can get started with writing your own. Whether you're looking to optimize performance or access unique system-level functionalities,
native modules have a lot to offer.

What are Native Modules? 🤔

You may be wondering, "What exactly are native modules, and how do they differ from the JavaScript modules I'm familiar with?"
Great questions! Native modules are essentially extensions for Node.js that enable execution of lower-level code, typically written in languages
like C++ or Rust, directly in your Node.js environment.

Definition and Key Characteristics 🔍

A native module is a compiled piece of code that acts as a bridge between Node.js and functionalities written in other programming languages.
These modules are generally written in C++ and are loaded dynamically at runtime, allowing them to be used just like any regular JavaScript module.

How Do They Differ From JavaScript Modules? 🔄

Performance: Native modules often offer performance benefits, especially when it comes to CPU-intensive tasks,
as they run closer to the metal, so to speak.
Capabilities: While JavaScript is a powerful language, there are operations it simply cannot perform or can perform only inefficiently.
Native modules fill these gaps by offering functionalities that are not natively available in JavaScript.
Language Interoperability: Native modules can be written in multiple languages, giving you the flexibility to integrate codebases in
languages like C++ and Rust.

In a Nutshell 🥜

Native modules serve as a gateway, opening up new possibilities beyond what is achievable using only JavaScript.
They act as a bridge between the high-level, dynamically-typed world of JavaScript and the lower-level, statically-typed realm of
languages like C++ or Rust.

🎯 Why Use Native Modules?

Native modules are a powerful feature that can dramatically affect how you design and implement Node.js applications.
While they add complexity, the trade-offs can often be worth it, especially in performance-critical applications or
those that require low-level system access. In this section, we dive deeper into why you might consider using native modules in your next project.

🚀 Performance Benefits

One of the most compelling reasons to use native modules is the significant performance boost they offer for CPU-bound tasks.
Let's look at some key technical aspects:

Compiled Languages

Native modules often utilize compiled languages like C++ or Rust. Compiled languages generally outperform interpreted languages like
JavaScript because the code is transformed into machine code, which is directly executable by the computer's CPU.

Direct Memory Access

Unlike JavaScript, which relies on garbage collection and automatic memory management, languages like C++ and Rust allow for direct memory access.
This enables you to optimize memory usage manually, leading to faster execution times, especially in memory-intensive operations.

System-Level Optimizations

The closer you are to the hardware, the more room you have for optimization. Native modules enable more efficient CPU instruction sets, cache
optimizations, and other system-level performance improvements that are not accessible or are inefficient in a higher-level language like JavaScript.

In this diagram, you'll notice two distinct pathways from your JavaScript code to the System-Level APIs.
One goes through the Node.js Runtime, while the other proceeds directly through a Native Module.

JavaScript Code to Node.js Runtime

When your JavaScript code needs to perform an action like reading a file or sending network requests, it calls a function that interacts with the Node.js
Runtime. This runtime environment is a multi-layered construct, consisting of several components that each serve a specific purpose but also add
complexity and overhead.

Event Loop: The heart of Node.js's non-blocking I/O capability. It enables asynchronous operations by queuing up tasks to be executed later.
However, it can also be a bottleneck, especially for CPU-bound tasks that can't be offloaded and must be executed sequentially.

V8 Engine: This is where your JavaScript code actually gets executed. While V8 is highly optimized, it still can't match the speed of
running precompiled code for specific tasks in various cases, adding a layer of latency.

Garbage Collection: The built-in memory management mechanism can, paradoxically, introduce delays. Though it relieves you from manual
memory management, the process of identifying and clearing unused memory can stall the execution flow.

These layers cumulatively make the route through the Node.js Runtime slower, especially for performance-critical tasks.

Node.js Runtime to Native Module

The second pathway moves from the Node.js Runtime directly to a Native Module, bypassing the complexities of the runtime environment. Native modules interact closely with system-level APIs and are usually written in compiled languages like C++ or Rust.

When you initiate a function call to a native module, you are essentially creating a shortcut, bypassing the event loop, the V8 engine, and garbage collection. This results in what the diagram illustrates as "Fast API Calls", leading to noticeably quicker interactions with system-level functionalities.

The Ultimate Destination: System-Level APIs

Regardless of the route taken, the ultimate objective is to interact with System-Level APIs to execute tasks like file manipulation, network requests, or other I/O operations. The difference lies in the efficiency and speed of reaching this layer. Native modules provide a more direct, unobstructed path, making them a highly advantageous choice for performance-intensive applications.

By understanding the architecture outlined in this diagram, it becomes evident why native modules can be a game-changer for certain Node.js applications. They offer a more efficient pathway to System-Level APIs, bypassing the latency-inducing layers present within the Node.js Runtime.

💡 Access to Low-Level APIs

Native modules give you unparalleled access to system-level APIs, offering functionalities that are simply not available or inefficient in Node.js.
Here's why this is crucial:

File Systems and I/O Operations

While Node.js provides basic file system APIs, they may not be suitable for all use-cases. For instance, you might need to access specific
file attributes or use low-level I/O operations that the Node.js APIs don't expose.

Networking Protocols

Need to implement a custom networking protocol or use a less common one that isn't supported by Node.js? Native modules allow you to do just that.

Hardware Interactions

From accessing USB ports to communicating with IoT devices, native modules can provide the low-level access you need to interface directly
with hardware components.

🌍 Real-World Applications and Examples

The theoretical benefits of native modules are compelling, but let's ground this discussion with some real-world applications where going
native makes sense.

Data Science and Machine Learning

Traditional data science languages like Python have robust ecosystems for statistical analysis and machine learning. However, for real-time
data processing in a Node.js application, native modules can leverage optimized C++ or Rust libraries to perform complex calculations much faster
than a native JavaScript implementation.

Video and Audio Processing

Real-time video and audio processing require high performance and low latency, something that native modules are particularly good at.
By tapping into low-level APIs and system resources, native modules can handle tasks like video encoding and decoding, noise reduction, and more.

Gaming Backends

Highly interactive, real-time gaming backends require the utmost efficiency in handling numerous simultaneous connections and rapid data exchange.
Native modules can optimize networking protocols and data serialization techniques far beyond what's possible in plain JavaScript.

By understanding the benefits and real-world applications of native modules, you can make a well-informed decision on whether they are the right
choice for your project. They offer a potent combination of performance and functionality, albeit with added complexity. However, in scenarios
where performance, low-level access, or specialized functionalities are paramount, native modules can be a game-changing addition to your tech stack.

The Anatomy of a Native Module 🛠️

Understanding a native module's inner workings is paramount to appreciate its advantages fully. In this section, we'll dissect a native module, looking at its core components and how files are typically organized within it.

Core Components of a Native Module ⚙️

Native modules are primarily written in lower-level languages like C++ or Rust and serve as a bridge between your Node.js code and system-level functionalities. Here are the core components you'll often encounter:

Binding Layer: This is where the native module interacts with the Node.js runtime. It defines the functions, objects, and properties that your JavaScript code can access. Typically implemented using Node's N-API or libraries like node-gyp or Neon for Rust.
Native Code: The bulk of the native module, written in a compiled language. This is the part that carries out heavy computations or interacts directly with system-level APIs. Due to being precompiled, this code is highly optimized for performance.
Package Configuration: Files like binding.gyp for C++ or Cargo.toml for Rust specify how the module should be built. These files contain metadata and build instructions for the native module.
Exported Functions: Functions that are exposed to your JavaScript code, enabling interaction between the native module and your application.

Understanding these components allows for a deeper grasp of how native modules function and how they can be fine-tuned for optimal performance.

File Structure and Organization 📂

A well-organized file structure is crucial for efficient development and maintenance of a native module. Although the specific organization can vary depending on the programming language and other requirements, you'll generally find directories for source code, package configuration, and more.

For a detailed exploration of how package configurations like binding.gyp play a vital role in the build process, be sure to check
out my next blog post: "Mastering Native Node.js Addons with node-addon-api: A Comprehensive Guide"

Brewing a Full-Stack Breakfast with ExpressoTS, Bun, and Elysia

Daniel Boll — Mon, 11 Sep 2023 17:34:08 +0000

Combine the power of ExpressoTS with Bun's Elysia framework in TypeScript for a mouth-watering full-stack experience.

Introduction

Welcome to a hands-on guide focused on integrating ExpressoTS, Bun, and Elysia into a full-stack TypeScript application. Let's cut to the chase and see what ingredients we'll be working with.

Our Ingredients

ExpressoTS

This TypeScript framework is designed for server-side applications. It offers flexibility in project structure and supports multiple
architectural patterns, including MVC. If you like your applications like your coffee — robust and versatile — ExpressoTS is your go-to.

You can also check my guide on a first project with ExpressoTS

Bun

An all-in-one toolkit that brings speed and efficiency to your JavaScript and TypeScript projects.
Think of Bun as the wholesome bread holding all your stack's ingredients together.
It's an entire toolbox in one neat package, and it's faster than you can say "bun in the oven."

ElysiaJS

A Bun web framework that promises performance, simplicity, and flexibility. Designed with TypeScript in mind, Elysia is like that final
touch of seasoning that takes your project from good to great.

So, whether you're into French toast or a classic Eggs Benedict, the aim of this blog post is to show you how to whip up a dish that's not just full of flavor, but also robust and scalable.

Getting Started: Your First Bite of Elysia

Now that we've set the table with the ingredients, let's start cooking. We'll begin by setting up a basic Elysia application using Bun.

1. Install Bun

First, you'll need to download and install Bun. Open your terminal and run the following command:



curl -fsSL https://bun.sh/install | bash

This will download and install Bun on your machine, equipping you with a fast and efficient JavaScript runtime.

2. Create Your Elysia Project

Once Bun is installed, it's time to create your Elysia project. Run:



bun create elysia breakfast-ts

You should see a message like this:



Created elysia project successfully

# To get started, run:

  cd breakfast-ts
  bun run src/index.ts

3. Run Your Application

Follow the instructions in the terminal message:



cd breakfast-ts
bun run src/index.ts

And just like that, you should see the message, instantaneously 🌪:



🦊 Elysia is running at localhost:3000

Voila! You've just whipped up your first Elysia app, hot and fresh.

4. Verify Your Application

To verify Elysia's functionality, you can perform a health check using the provided endpoint. While I'll be using httpie for its straightforward API, you're welcome to use curl if you prefer.



http :3000



HTTP/1.1 200 OK
Content-Length: 12
Date: Sun, 10 Sep 2023 20:47:46 GMT
content-type: text/plain;charset=utf-8

Hello Elysia

Extending Elysia: Crafting a Simple User CRUD

With the basic setup out of the way, let's extend our Elysia app by adding some CRUD functionality for users.
This will serve as a good starting point before we dive into more complex operations in future posts.

1. Run Your Application in Watch Mode

First, stop the server if it's running. Then leverage Bun's watch mode, which will automatically reload the application
when changes are made to the source code:



bun run --watch src/index.ts

2. Add User CRUD Routes

Now, let's define some basic CRUD operations. Open src/index.ts and add to your existing code the following:



import { Elysia } from "elysia";

// Create a user route group
const user = new Elysia().group("user", (app) =>
  app
    .get("/", () => "All users")
    .get("/:id", (context) => `Hello, ${context.params.id}`)
    .post("/", () => "Create User")
    .put("/:id", (context) => `Update User ${context.params.id}`)
    .delete("/:id", (context) => `Delete User ${context.params.id}`)
);

// Main app
const app = new Elysia()
  .get("/", () => "Hello Elysia")
  .use(user)
  .listen(3000);

console.log(
  `🦊 Elysia is running at ${app.server?.hostname}:${app.server?.port}`,
);

Here, we've used Elysia's group<Prefix extends string = string>(prefix: Prefix, run: (group: Elysia<...>)) method to bundle all user-related routes together. We've defined routes for getting all users, getting a single user by ID, creating a user, updating a user, and deleting a user.

3. More Advanced Features

Note: You can extend this basic setup in various ways. Elysia supports both simple state management within the server and more advanced
dependency injection techniques for robust applications. Check out Elysia's documentation on
State Decorate and
Dependency Injection if you're interested.

For the sake of this blog post, we're keeping it simple. If you are willing to expand your horizons, however, you are encouraged to check out the
Call to Action section at the end of this post, where you may help the ExpressoTS community in the bun integration.

4. Test Your User CRUD Endpoints

After running your server, you can test each endpoint to make sure they're working as intended.

Get All Users



http :3001/user



HTTP/1.1 200 OK
Content-Length: 9
Date: Sun, 10 Sep 2023 20:47:46 GMT
content-type: text/plain;charset=utf-8

All users

Get a Single User

Replace {id} with the ID of the user you're interested in.



http :3001/user/42



HTTP/1.1 200 OK
Content-Length: 11
Date: Sun, 10 Sep 2023 20:47:46 GMT
content-type: text/plain;charset=utf-8

Hello, 42

Create a User



http POST :3001/user



HTTP/1.1 200 OK
Content-Length: 11
Date: Sun, 10 Sep 2023 20:47:46 GMT
content-type: text/plain;charset=utf-8

Create User

Update a User

Again, replace {id} with the ID of the user you want to update.



http PUT :3001/user/42



HTTP/1.1 200 OK
Content-Length: 16
Date: Sun, 10 Sep 2023 20:47:46 GMT
content-type: text/plain;charset=utf-8

Update User 42

Delete a User

And one last time, replace {id} with the ID of the user you want to delete.



http DELETE :3001/user/42



HTTP/1.1 200 OK
Content-Length: 16
Date: Sun, 10 Sep 2023 20:47:46 GMT
content-type: text/plain;charset=utf-8

Delete User 42

Now you've successfully tested all of your user CRUD operations. It's like a full-course breakfast — everything is present and accounted for!

Diving into the Heart of the Matter: Reflection and Decorators

In programming, reflection is a mechanism that allows you to inspect and manipulate program elements like classes and objects at runtime. In other words, reflection enables a program to observe its own structure, similar to how you can observe your own reflection in the mirror. This allows for greater dynamic behavior, enabling more powerful and flexible constructs like decorators.

Dive deeper @ daniel-boll.me

If you are interested in the stuff you can achieve with reflection make sure to check out the full blog at daniel-boll.me for much more content.

A first project with ExpressoTS

Daniel Boll — Sun, 27 Aug 2023 17:35:21 +0000

Introduction

ExpressoTS is a new developer-friendly TypeScript framework for Server-Side Applications. It's currently built on top of Express, easy to get a new developer on board, light and easy project structure, two different project structure supported (opinionated and non opinionated version), supports MVC, non MVC, architecture. The opinionated template was built on top of clean architecture, you will see concepts embedded in the opinionated template such as entities, useCases, repositories and providers.

Setting Up Your First ExpressoTS Project

Ah, the first step to any great journey in the world of coding—a proper setup. Trust me, nobody wants to get lost before even starting the adventure. So, let's cut the chit-chat and dive right in.

Installation of ExpressoTS CLI

You've got two routes here, and the destination is the same, a functional ExpressoTS environment. Your first option is to install the ExpressoTS CLI globally:

pnpm i -g @expressots/cli

Got limited commitment issues? No worries, you can also use dlx to run the CLI without installing it globally. Ah, the wonders of a one-night stand with a CLI!

pnpx @expressots/cli new expressots-first-project

Configuration Wizard

After running the command, you're greeted by a friendly (albeit text-based) wizard 🧙‍♂️.

[🐎 Expressots]

? Project name expressots-first-project
? Package manager pnpm
? Select a template Non-Opinionated :: A simple ExpressoTS project.
? Do you want to create this project? Yes

Fill out the form wisely but don't overthink it. For this tutorial, I'm going with the "Non-Opinionated" template because, it will give us a shallower learning curve in the start.

Getting Comfy in Your New ExpressoTS Home 🏡

Navigating to Your Project

So you've set up your new ExpressoTS project. Awesome! Time to get inside the engine room and take a look under the hood.

cd expressots-first-project

You're now in the root folder, and if you're curious about what comes packed by default, run a quick tree command.

@expressots/tests/expressots-first-project via  v16.19.0
❯ tree -I node_modules -a
.
├── .eslintrc.js
├── expressots.config.ts
├── .gitignore
├── jest.config.ts
├── package.json
├── pnpm-lock.yaml
├── .prettierrc
├── README.md
├── src
│   ├── app.container.ts
│   ├── app.controller.ts
│   ├── app.module.ts
│   ├── app.usecase.ts
│   └── main.ts
├── test
│   └── app.usecase.spec.ts
├── tsconfig.build.json
└── tsconfig.json

3 directories, 16 files

There you go! These are the files you'll be living with.

Customizing Your Prettier Preferences

You've got your own coding style—don't we all? You can tweak the .prettierrc file to your heart's content.

{
  "singleQuote": false,
  "trailingComma": "all",
  "endOfLine": "auto",
  "tabWidth": 2
}

Let's Take a Quick Tour of package.json

A glance at the scripts section in package.json tells you all you need to know to get things up and running.

❯ cat package.json | jq ".scripts"
{
  "prebuild": "rm -rf ./dist",
  "build": "tsc -p tsconfig.build.json",
  "dev": "tsnd ./src/main.ts",
  "prod": "node ./dist/main.js",
  "test": "jest",
  "test:watch": "jest --watchAll",
  "test:cov": "jest --coverage",
  "format": "prettier --write \"src/**/*.ts\" \"test/**/*.ts\"",
  "lint": "eslint \"src/**/*.ts\" --fix"
}

Here, you've got your usual suspects: build, dev, test, and more.

Format All The Things!

Before diving into development, let's keep it neat. The format script will handle that.

pnpm format

Running in Dev Mode

And finally, the moment you've been waiting for—fire up the dev server and see your app come to life!

@expressots/tests/expressots-first-project via  v16.19.0
❯ pnpm dev

[INFO] ts-node-dev ver. 2.0.0 (using ts-node ver. 10.9.1, typescript ver. 4.9.5)
Application version not provided is running on port 3000 - Environment: development

Making Your First Request in ExpressoTS 🌐

Time for a Little Hello!

You've been through the house tour, you've feng shui'd your .prettierrc and got your scripts all figured out. Now what? Well, how about actually seeing your app in action?

Using HTTPie for Your First GET Request

For this demonstration, we're going to use HTTPie — the cURL for the 21st century. But like, way more readable. Here's how to send your first GET request to http://localhost:3000.

@expressots/tests/expressots-first-project via  v16.19.0
❯ http :3000

Bam! You should see a response like:

HTTP/1.1 200 OK
...
Hello Expresso TS!

Congrats! You just made your server say hello.

Anatomy of the "Hello Expresso TS!" 🦴

Alright, let's dissect that "Hello Expresso TS!" message. How does this simple string pass through layers of TypeScript files to make it to your browser? Spoiler: It's not magic; it's just well-structured code.

The Starting Point: src/main.ts

This is where the baton is picked up. Here, we import the essential parts from ExpressoTS and set up the initial application instance. Simple enough.

import "reflect-metadata";
import { AppInstance, ServerEnvironment } from "@expressots/core";
import { container } from "./app.container";

async function bootstrap() {
  AppInstance.create(container);
  AppInstance.listen(3000, ServerEnvironment.Development);
}

bootstrap();

The Container: src/app.container.ts

Think of this as the backstage, where everyone gets ready for the show. It's where the application container is configured with your custom modules.

import { AppContainer } from "@expressots/core";
import { AppModule } from "./app.module";

const appContainer = new AppContainer();
const container = appContainer.create([AppModule]);

export { container };

The Module: src/app.module.ts

Modules in ExpressoTS group related functionalities. Here, it's as simple as importing the AppController.

import { CreateModule } from "@expressots/core";
import { AppController } from "./app.controller";

const AppModule = CreateModule([AppController]);

export { AppModule };

The Controller: src/app.controller.ts

This is the conductor of our orchestra. It's responsible for handling HTTP requests and directing traffic. In this case, it's just saying, "Hello Expresso TS!".

import { BaseController, controller, httpGet, response } from "inversify-express-utils";
import { Response } from "express";
import { AppUseCase } from "./app.usecase";

@controller("/")
class AppController extends BaseController {
  constructor(private appUseCase: AppUseCase) {
    super("app-controller");
  }

  @httpGet("/")
  execute(@response() res: Response) {
    return res.send(this.appUseCase.execute());
  }
}

The Use Case: src/app.usecase.ts

Here's where the actual "Hello Expresso TS!" lives.

import { provide } from "inversify-binding-decorators";

@provide(AppUseCase)
class AppUseCase {
  execute() {
    return "Hello Expresso TS!";
  }
}

"Hello, Who?" Changing the Default Response 🖊️

Got bored of the usual "Hello Expresso TS!"? Let's give it a personal touch. To do that, we only need to venture into the src/app.usecase.ts file. See, this is the beauty of a well-structured codebase; you don't have to jump through hoops to make a simple change.

The Tweak: src/app.usecase.ts

One line change. That's it. Here we just swap out the text to "Hello from <Daniel::Boll>".

@provide(AppUseCase)
class AppUseCase {
  execute() {
    return "Hello from <Daniel::Boll>";
  }
}

The Reload: Auto-refresh FTW

As soon as you save that change, the development server detects this update and reloads itself. No manual effort required. Ah, the joys of modern development.

[INFO] Restarting: /../@expressots/tests/expressots-first-project/src/app.usecase.ts has been modified
Application version not provided is running on port 3000 - Environment: development

The Result: Let's Talk to the Server Again

Run the command, and voila! The updated greeting is now served hot, right from your server.

Hello from <Daniel::Boll>

Dive deeper

If you are interested in more stuff you can do in this simple ExpressoTS project make sure to check out the full blog at daniel-boll.me.