DEV Community: Rogue Paradigms

FS-Routing for nodejs: a low config approach

Rogue Paradigms — Sat, 24 Jun 2023 23:48:53 +0000

Welcome, fellow developers, to a journey through the fascinating world of routing. In this blog post, we'll take a practical and easy-going approach to routing for nodejs api frameworks inspired by nextjs routing. I'm using express as my base framework, however, this code can be easily adapted to use in other popular frameworks.

Whether you're a beginner or an intermediate developer, I've got you covered. Routing can be a bit tricky, but fret not! I'm here to guide you through the process step by step. Say goodbye to the hassle of manual route registration and complex configurations. With this approach, you'll learn how to create a router module that will simplify route management in Express.js. I'll even introduce you to the awesome File System (fs) module and the handy Path module (if those are new to you). By the time we're done, you'll have a fully functional router module that automagically handles route mapping, supports parameterized routes, and makes your code modular and reusable. So, are you ready to embark on this exciting journey with me? Let's simplify route handling and build some amazing web applications using Express.js together!

Goal

Our goal is to create a router to handle the following routes

GET: /home
GET: /authors
POST: /authors
GET: /books
POST: /books
GET: /books/[bookid]
POST: /books/[bookid]

By defining our route handlers in the following directory structure

./home.js
./authors/GET.js
./authors/POST.js
./books/index.js
./books/[bookid].js

Each of these files represents a different route and HTTP method combination that we'll handle in our application.

Setup

To kick off our adventure, let's bootstrap our project by following a few simple steps. Open up your command line and execute the following commands:


mkdir fsrouter
cd fsrouter
npm init
touch readme.md
touch fsrouter.js # CONTAINS MODULE CODE
npm install express
touch index.js # server code/entry point

With these commands, we create the necessary files and directories for our routing experiment.

You can generate the controller files for this exercise by running following shell commands:

mkdir routes
cd routes

# ./routes
touch home.js

# ./routes
mkdir authors
cd authors
touch GET.js
touch POST.js
cd ..

# ./routes
mkdir books
cd books
touch index.js
touch "[bookid].js"
cd ..

Now, let's dive into the contents of these files. Within each handler module, we define the logic to handle the corresponding route. Here's an example:

// handler modules that are named by HTTP methods such as GET,POST,PUT etc. mush default export a handler functtion
// authors/GET.js, authors/POST.js
module.exports = (req, res) => res.send("Response");

AND

// handler modules that match a route path must export an object
// home.js, books/index.js, books/[bookid].js
module.exports = {
    GET: (req, res) => res.send(req.params),
    PUT: (req, res) => res.send("Response")
};

In this approach, we define the handlers as simple functions or objects with HTTP method-specific functions. The goal is to keep things straightforward and focus on the functionality.

Now, let's integrate our custom routing solution into our Express server. Open up the index.js file and add the following code:


const express = require("express");
const fsrouter = require("./fsrouter");
const app = express();
app.use(fsrouter('./routes'));
app.listen(5001, () => {
    console.log("Server is up and running on port 5001!");
});

With this code in place, our server is ready to handle the routes we defined earlier. By utilizing the fsrouter middleware, we unleash the power of our custom routing solution without being constrained by traditional frameworks.

Walkthrough of fsrouter.js

So, how does this routing actually work? Inside the fsrouter.js module, a clever mechanism traverses the routing structure we created. It dynamically matches the incoming requests to their corresponding handlers, allowing us to handle various HTTP methods and even dynamic routes.

Let's dive into the code and explore how fsrouter.js simplifies routing in Node.js applications:

Importing Required Modules

The first step in the fsrouter.js file is importing the necessary modules. It requires the fs and path modules from Node.js, which allow us to work with the file system and manipulate file paths effectively.


const fs = require('fs');
const path = require('path');

These modules play a crucial role in scanning the file system and mapping routes to handlers.

Defining Supported HTTP Methods

Next, fsrouter.js declares an array of supported HTTP methods. These methods include 'GET', 'POST', 'DELETE', 'PATCH', and 'PUT'. By having a predefined list of methods, fsrouter.js can identify and map each route's corresponding HTTP method correctly.


const httpMethods = ['GET', 'POST', 'DELETE', 'PATCH', 'PUT'];

Mapping Route Files to HTTP Methods

The getMethodFromName function is a utility function that extracts the HTTP method from a route file's name. It checks if the file name matches any of the supported HTTP methods using a regular expression. If there is a match, it returns the HTTP method; otherwise, it returns null.


function getMethodFromName(filename) {
    const isNamedByMethod = httpMethods.some(name => (new RegExp(`^${name}.(t|j)s$`)).test(filename))
    if (isNamedByMethod) return filename.replace(/.(t|j)s$/, '');
    return null;
}

This function comes into play when scanning the file system and mapping route files to their corresponding HTTP methods.

Reading Controllers and Building Route Mapping

The readControllers function is the heart of fsrouter.js. It reads the controllers and builds a mapping of routes to their respective handlers.


function readControllers(basePath, urlPath) {
    const fsPath = path.resolve(basePath, urlPath)
    const entities = fs.readdirSync(fsPath);
    return entities.map(entity => {
        const entitypath = path.resolve(fsPath, entity);
        const stat = fs.lstatSync(entitypath);

        if (stat.isFile()) {
            // Handle file-based routes
            const methodName = getMethodFromName(entity);
            if (methodName)
                return { [urlPath]: { [methodName]: require(entitypath) } }

            // Handle route files with multiple HTTP methods
            const { GET, POST, DELETE, PATCH, PUT } = require(entitypath);
            const entityUrlSegment = entity.replace(/.(t|j)s$/, '');
            const pathToMatch = entityUrlSegment === "index" ? urlPath : path.join(urlPath, entityUrlSegment);
            return { [pathToMatch]: { GET, POST, DELETE, PATCH, PUT } }
        } else if (stat.isDirectory()) {
            // Recursively handle subdirectories
            const mapping = readControllers(basePath, path.join(urlPath, entity))
            return mapping;
        }
    }).reduce((aggr, val) => {
        Object.entries(val).forEach(([path, handlers]) => {
            const commonHandlers = aggr[path] || {};
            aggr[path] = { ...commonHandlers, ...handlers }
        });
        return aggr;
    }, {})
}

This function takes in the base path and URL path, resolves the file system path, and reads the entities (files and directories) within that path. It then iterates through each entity, distinguishing between files and directories.

For files, it checks if they represent a single HTTP method or multiple methods. If it's a single method, the mapping is straightforward. If it's multiple methods, the function extracts the corresponding handlers for each HTTP method.

For directories, the function recursively handles subdirectories, building the complete route mapping structure.

Matching Routes and Handling Requests

The findHandler function is responsible for matching routes and retrieving the appropriate handler for a given HTTP method and path. It also extracts dynamic route params from url.


function findHandler(routeMap, method, path) {
    const matchedPath = Object.keys(routeMap).find(pathDefinition => {
        const pathDefSegments = pathDefinition.split('/').map((segment) => {
            const [_, param] = segment.match(/^\[(.*)\]$/) || [];
            if (!param) return segment;
            return { paramName: param }
        });
        const pathSegments = path.split('/').filter((seg, pos) => seg || pos); // filterout 1st null segment. This is caused be leading '/' 
        if (pathDefSegments.length != pathSegments.length) return false;
        return pathDefSegments.every((definedSegment, pos) => {
            if (typeof definedSegment === "object") return definedSegment.value = pathSegments[pos]
            if (pathSegments[pos].toLowerCase() === definedSegment.toLowerCase()) return true;
        })
    })
    if (!matchedPath) return null;
    return {
        handler: routeMap[matchedPath][method], params: pathDefSegments.reduce((aggr, seg) => {
            if (typeof seg === "string") return aggr;
            return { ...aggr, [seg.paramName]: seg.value }
        }, {})
    };
}

This function takes a routing map, an HTTP method, and a URL path as input. It searches for a matching route in the routing map based on the method and path. If a match is found, it returns an object containing the matched handler function and any URL parameters extracted from the path.

Initialize the router

function loadRoutesFrom(routesControllerDir) {
    const routeMap = readControllers(routesControllerDir, './');
    return (method, path) => findHandler(routeMap, method, path);
}

This function takes the path to routes directory as input. It calls readControllers to generate the routing map based on the directory structure. It then returns a function that takes an HTTP method and a URL path as arguments and calls findHandler to find the corresponding handler function and URL parameters.

Creating the fsrouter Middleware

Finally, fsrouter.js exports the loadRoutesFrom function, which creates the fsrouter middleware.


module.exports = routesControllerDir => {
    const router = loadRoutesFrom(routesControllerDir);
    return (req, res, next) => {
        const match = router(req.method, req.path);
        if (!match) return next();
        const { handler, params } = match;
        req.params = params;
        return handler(req, res, next)
    }
};

This function takes the controller directory as input and returns a router function. The router function, when used as middleware in an Express application, intercepts incoming requests, matches the routes, and executes the corresponding handlers.

Conclusion

Stay tuned for future blog posts where we'll delve deeper into the intricacies of routing, experiment with new ideas, and uncover even more ways to optimize our development process.

NB: some code sections in this post can be explained in more depth. I'm skipping that to keep it from being an unreadably long post. I'd love to elaborate in the comments.

The Unhinged Pursuit: Unraveling the Mad Scientist's Motivation

Rogue Paradigms — Sat, 24 Jun 2023 18:20:35 +0000

Welcome, fellow seekers of the extraordinary! Step into the chaotic laboratory of innovation known as Rogue Paradigms. I, your mad scientist of coding, am here to unleash mind-bending utilities and modules that defy the ordinary.

In the realm of software development, where conformity reigns supreme, I emerge as a whirlwind of unconventional brilliance. Why settle for the mundane frameworks and industry standards when we can concoct our own elixirs of technological marvels? Behold, as we sprint through the corridors of core programming capabilities, leaving a trail of groundbreaking solutions in our wake!

This blog is the sanctuary for those who crave the thrill of uncharted territories. Forget the familiar script and embrace the exhilaration of experimentation. From manipulating intricate data structures to summoning the powers of intricate algorithms, we shall dare to tread where others fear to code.

Whether you're an audacious novice seeking to bend the rules or an experienced maestro ready to shatter the status quo, Rogue Paradigms shall be your catalyst for audacious innovation. Through my ramblings, you shall unlock the secrets of unseen potential, unveiling the true essence of programming's untamed wilderness.

No time for trivialities, my comrades! Together, we shall redefine the very fabric of possibility. Let us surge forward, tearing through barriers with our swift strokes of brilliance, sculpting software that not only echoes the industry but transcends it.

Join me on this electrifying quest as we seize the true essence of coding's core capabilities. Brace yourself for an adrenaline-fueled journey, where inspiration and madness entwine, and creativity knows no bounds.

Now, haste awaits us! Together, let's scribble this introduction and embark on an extraordinary adventure, for greatness favors those who fearlessly push the boundaries.

Onward, my fellow mad scientists of the digital realm! Let chaos reign supreme!

Part 2: Creating a web framework

Rogue Paradigms — Wed, 16 Mar 2022 06:24:17 +0000

Like I promised in my previous post, I spent more time in this project to create a usable frontend framework with a javascript view controller.

The main objective of this exercise is to create a low level view controller. This should be flexible and modular and programmable. If I write down a check list, it would look like this.

No transpiling/template-rendering (custom syntaxes)
No bundlers. Apps should be able to run straight in browser
Use low level apis dom apis, create custom-elements with shadow-root and maintain isolation

The decision to keep the api minimal/low-level is so that it will be possible to use other existing legacy libraries while maintaining isolation. For example, even though promax doesn't have a template system, one could bring their own template system and use within promax

This is the api I came up with:

index.html. Here p-frame is like iframe, but it loads another html file into a shadow dom.

<script src="./module/.bundle/script.js"></script>
<p-frame src="./src/app.html"></p-frame>

You can use this without the rest of promax too, just to embed any html with its own contained style and scoped scripts. (Limiting scope is what the rest of promax is about)

./src/app.html

<!-- UI -->
<button id="button" onclick="pscope.clickListener(event)">
  This text will be replaced by initialState value
</button>

<!-- view controller -->

<script>
  promax
    .initState({ buttonText: "starting" })
    .setRenderer(({ root, state }) => {
      root.getElementById("button").innerHTML = state.buttonText;
    })
    .attachScope(({ getState, patchDom }) => {
      let clickCount = 0;
      return {
        // returns scope object
        clickListener: () => {
          clickCount++;
          patchDom({
            buttonText: "Clicked " + clickCount,
          });
        },
      };
    });
</script>

This is a click counter button. At first rendering, button text is set to "starting". Then we attach an event listener to count clicks. Butten text updates to "Clicked #count" after each click

Breakdown of `./src/app.html`

Notice the onclick=pscope.clickListener(event). pscope is a global proxy that know how to look up the component scope object at run time. It only works for events with target field. It will throw error if a promax scope is not defined.

<!-- UI -->
<button id="button" onclick="pscope.clickListener(event)">
  This is the welcome page!! This is also an html file. :)
</button>

<!-- view controller -->
<script>

promax is a global variable injected by p-frame component.
Note that promax.initState().setRenderer().attachScope() works only in this fixed order.

  promax

initState sets first state and creates a closure around it to attach renderer.

    .initState(
      { buttonText: "starting"  }
    )

setRenderer sets a renderer function. This is the only place you touch dom. asynchronous dom modifications are prohibited. This will also invoke renderer immediately after setting. This is also why we start with initState

    .setRenderer((
      {root,state }
    ) => {

      root.getElementById("button").innerHTML = state.buttonText;
    })

attachScope is where you inject values to component scope. Event listeners are defined through scope object. Currently this is the only use-case, but it has other uses.

    .attachScope((
      { getState, patchDom }
    ) => {
      let clickCount = 0;
      return {
        // returns scope object
        clickListener: () => {
          clickCount++;
          patchDom({
            buttonText: "Clicked " + clickCount
          });
        },
      };
    });

</script>

That is basically it for this exercise. The above code is a click counter button, if you didn't guess it already.

Next episode

I am debating myself on adding a prop system to pass javascript objects, or stick with html attributes. I really don't like to add custom syntax. Most likely to stick with attributes.

I haven't figured out how to render lists and conditional views yet. But the fact that we have a view controller is giving me great confidence.

I rencently renamed the project to romax. I know this is not an objectively better name than promax, but at least it sounds less lame to me.
Github: /bwowsersource/romax

To Do

Props
Dynamic rendering lists
Conditional rendeing
script scope
promax.renderFromString and promax.renderFromTemplate

Creating a web framework in under an hour!!

Rogue Paradigms — Sun, 06 Mar 2022 00:48:02 +0000

That was sort of a clickbait ;) But let me explain. If the following line of html gets you interested, keep reading.

<include src="/snippet.html" />

This post is partly an opinion piece and a toy project report. The project is still a work in progress. My intention for writing this post is to pique the interest of fellow mad men

At the end of this exercise, my index.html look like

<script src="script.js"></script>
<include src="./src/app.html"></include>
<script> htmlPromax.start(); </script>

script.js contains all htmlPromax code. My project folder look like this

<rant>

Skip this section if you get bored

I, quite often, find myself overwhelmed by the amount of setup required (even though it is mostly automated) to get started with a minimal front-end project. Then the way you write frontend code... It is no where close to what browsers were designed to do. This is while the fact remains that a simple index.html file could be your front-end just fine.

So I set out on a search to find out a close to native html framework that would make the effort little easier, while work out of the box in a browser. To be clear, my end goal is not to eliminate bundling completely, rather to write code in html and pure embedded javascript, or as close to it as possible. In my book, eliminating as much bundling steps as possible gets me half way close to the objective

Polymer, NO. It is now in maintenance mode. Lit is the new alternative they suggest. Both of them use web-components instead of a custom component system like react. This is good, but the way you write code feels more like a react spin-off.

If you got here looking for a native web-components based framework, please do check out Lit. I haven't used it myself before. I will be considering use of Lit for this project in future.

I didn't find any other projects. (Please leave a comment if you know any.)

Then I decided to find out what browsers can do in its native form today.

Module imports. Actually this is a good idea, especially if you use it along with Polymer. However I'm not sure if Lit can be used in this way, because of the use of decorators. So that is a tall stone wall. Besides, I'm not happy with this classed approach.

<rant/>

Next I looked for ways to import html snippets into dom. That lead me to this snippet of code
This method is part of w3.js library. The simplicity of this code got me interested.

I started a project of my own. I call it htmlPromax, coz why not?!
I copied this method, modified it little bit. The modifications can be summarized as follows.
I changed the include method to recursively include files with depth-limitting ( it can go 99 includes deep before erroring).
I then changed the query selector/element-identifier used to tagName:include (This will be changing soon, but for the time being it looks cool and reads like #include fileName.h).
I also added a unique id generator to tag each included component. This is a foundation to next steps of this project.

That was 1 hour

By the end of 1 hour, I had setup a very minimal purely html based web framework that does nothing beyond importing html snippets.

But, I am not stopping there. It would be very irresponsible of me to copy a method from w3.js, change few attributes and call it a web framework of my own. However, I was able to write frontend code in pure html, organize the code into folders and maintain a project structure. I could simply load the index.html and everything just worked. So this one hour got me excited enough to keep working on it.

Beyond 1 hour

At this point I wouldn't blame you if you think I am a caveman who think static html websites are cool. I'm not. Next step is to add Javascript capabilities to drive a dynamic web app.

Isolation of code run in a component context is a major requirement. But I'd also like to keep it unobtrusive and as close to native as possible.

I have some ideas. I am trying to substitute <include> tag with a custom component and shadowDom, thus solving the lifecycle events and isolation problems. I'll add a global ctx variable that is available only inside the shadowDom of an <include>. This will change the include code substantially. We will not need to recursively check for includes, the component constructor will take care of it for us.
This part is still a work in progress. Wish me luck and watch out for an update post.
git: https://github.com/bwowsersource/promax

I created a basic node module and it does the job!!

Rogue Paradigms — Sun, 20 Feb 2022 20:47:52 +0000

First of all, I have created another node module 5 years ago, so this is not my first time. I published it because I can. This one isn't that different... but, let me tell the story.

Package name: 'jsonian'
version: '0.0.2' 
// I had to correct the readme, otherwise it is same as 0.0.1

So what does it do?

It lets you persist javascript object to a json file.

...Am I getting Eyeballs yet?

Yeah, there are bunch of other modules that let you do it. I didn't like the way they expose the api. Most ones I found use a getter/setter approach.
Nothing wrong with that, may be it is even better for certain application.

I am working on a jupyter notebook and I wanted a way to keep my js objects and json files to be in sync without changing the syntax too much.

I'll share a bit of Readme.md here

Usage

const jsonian = require('jsonian');

const $table = jsonian('./testTable.json'); // file must already exist

$table is proxy to a root object. You can use it to get or set values as with a usual object.

// './testTable.json' 
{
    "name": "Students"
}

// to read value
var name = $table.name; // "Students"

// to write value to file
$table.name = "New_table_name"; // replaces "Students" in file './testTable.json' 
$table.rows = ["name","class"]; // adds new field "rows" in file './testTable.json'

The $ sign in the begining of $table is an indicator that it is a pure json object (jsonian proxy). I find it useful, to maintain a readable code. If you already use $ for something else feel free to come up with your own convention.

That is it. Your object will always be persisted in file.

The uglys

It is using javascript Proxy object to capture get/set actions. So assume all the gotchas associated with it.
The project is still in an alpha state. It is untested for the most part.
This is particularly not recommended for web servers (scalability won't be great with this kind of file write load + I can't think of any usecases) and browsers as long as they don't have filesystem access.
The good part is it all sits in 31 lines of index.js.

If you read this far, I think I've earned the right to share git repo https://github.com/bwowsersource/jsonian

Ok how does it work?

When we load a file, it reads the file and parse json to a rootObj.
The loader then creates a Proxy around this rootObj. The proxy intercepts get and set operations on rootObj.
sets will trigger a JSON.stringify of rootObj followed by a write-to-file.
On the other hand get will return the value on rootObject only if it is not an object. If the value being accessed is another object, it will create another proxy around this child object, thus allowing trapping of nested objects.